Sleep Mode & Auto Delete for Virtual Clusters

Our platform provides two powerful features to reduce Kubernetes costs:

• Sleep Mode which puts virtual clusters to sleep when nobody is using them, i.e. purging all pods while keeping all resources inside the virtual clusters during periods of inactivity.
• Auto-Delete which deletes virtual clusters that have been idle for a while.

Both of these feature rely on vCluster Platform's inactivity detection.

Working with Sleep Mode

With sleep mode, you can put virtual clusters to sleep which means that vCluster Platform will set replicas: 0 for all replica-controlled resources such as Deployments and StatefulSets inside the virtual cluster. This means that Kubernetes will delete all pods but the entire configuration of resources within the virtual cluster is still there.

Sleep mode can be:

• Invoked manually

• Triggered by an inactivity timeout (no one has run a kubectl command or accessed an ingress in this virtual cluster for X minutes)

• Scheduled using a CRON syntax

Manual + Automatic Sleep / Wake-up

Start Sleep (manual)

UI

1. Click on Virtual Clusters to find the virtual cluster that you want to put to sleep.

2. Hover over the row of the virtual cluster that you want to put to sleep.

3. Click on the vertical ellipsis button and select Sleep to put the virtual cluster to sleep.

4. Notice how the Status column shows that the virtual cluster is now sleeping.

Automatic Wakeup

Note that the virtual cluster will automatically wake up again, once you run a kubectl command or send a Kubernetes request via another tool to the virtual cluster.

CLI

To pause a vCluster instance using the CLI, run:

vcluster pause [name]

Automatic Resume

Note that the vCluster instance will automatically resume, once you run a kubectl command or send a Kubernetes request via another tool to the vCluster instance.

Automatic Sleep Mode (individual virtual cluster)

UI

Manual Support only

This only works for vCluster instances without a template, if you want to change the sleep mode configuration for a vCluster instance created by a template, it must be configured in the template.

1. Click on Spaces to find the vCluster instance that you want to put to configure automatic sleep mode for.

2. Hover over the row of the vCluster instance that you want to configure

3. Click on Edit on the vCluster instance that you want to edit.

4. Use the Sleep After Inactivity (minutes) field to specify the time to wait before putting the vCluster instance to sleep if there is no more user activity in this vCluster instance.

5. Click on the button to save the changes.

CLI

1. To configure the platform to put your virtual cluster to sleep after 5 minutes of inactivity, add this to the vcluster.yaml for your virtual cluster:

   external:
     platform:
       autoSleep:
         # in seconds
         afterInactivity: 300

2. vcluster platform create [vcluster-name] -f vcluster.yaml

Updating a vCluster instance

If you have already created the vCluster instance and need to change the vcluster.yaml values add the --use flag to the vcluster platform create command if you used the platform driver, or --upgrade if the vCluster instance was created with the helm driver.

Wake up virtual cluster

UI

1. In the Virtual Clusters view, hover over the virtual cluster that you want to wake up.

2. Click on the vertical ellipsis button and select Wake Up to wake up the virtual cluster.

3. Notice how the Status column shows that the virtual cluster is now running again after a Pending state.

CLI

To resume a paused virtual cluster using the CLI, run:

vcluster resume [name]

Scheduled Sleep / Wake-up

Scheduled Sleep & Wake-Up (individual space)

Manual Support only

This only works for virtual clusters without a template, if you want to change the sleep mode configuration for a virtual cluster created by a template, it must be configured in the template.

1. In the Virtual Clusters view, hover over the virtual cluster that you want to put to configure automatic sleep mode for.

2. Click on the button.

3. In the Definition section, under the Sleep Mode section. Use the Sleep Schedule field and/or the Wake-Up Schedule field to specify the Cronjob Times when the respective virtual cluster should be put to sleep or woken up.

4. [Optional] Input a Schedule Timezone for the timezone to apply the schedule.

5. Click on the button to save the changes.

Working with Auto-Delete

The platform lets you configure an auto-delete for virtual clusters that have not been used for a certain period of time (inactivity).

Configure Auto-Delete Timeout

Manual Support only

This only works for virtual clusters without a template, if you want to change the sleep mode configuration for a virtual cluster created by a template, it must be configured in the template.

1. In the Virtual Clusters view, hover over the virtual cluster that you want to configure auto-delete for.

2. Click on the button.

3. Under the Sleep Mode section. Use the Delete After Inactivity (minutes) field to specify the time to wait before deleting the virtual cluster if there was no more user activity within this virtual cluster.

4. Click on the button to save the changes.

Inactivity Detection

All requests that are made through the platform count as activity in the virtual cluster.

If your kube-context points to the platform's API server as a proxy before the actual connected cluster's API server, every kubectl request will be an activity and reset the inactivity timeout.

Ingress Requests

For ingress-nginx based ingresses, activity detection also works automatically. Other ingress controllers are currently not supported. For nginx based ingresses, the platform will add a special annotation to each ingress that will track activity and reset the timer as soon as a request is made to that ingress.

Ingress Wakeup

vCluster Platform supports waking up a virtual cluster through an ingress independent of the underlying ingress controller. Just navigate to the ingress host and you should see a wakeup page for the virtual cluster.

Advanced Configuration

Exclude Resources From Sleep

The platform allows you to specify resources that should not sleep within a virtual cluster by providing the annotation sleepmode.loft.sh/exclude: 'true' on either a Deployment, StatefulSet, ReplicaSet or Pod. For example the following Deployment would not sleep, if the virtual cluster is sleeping:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
  annotations:
    sleepmode.loft.sh/exclude: "true"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
        - name: busybox
          image: busybox
          command:
            - sleep
            - "3600"

Long-Living Connections

It's possible that certain requests such as kubectl exec or kubectl port-forward keep an active connection open to the virtual cluster that prevents it from sleeping. This is in most cases wanted since the virtual cluster shouldn't start sleeping when someone is still clearly using it, however there are cases where the connection might be idle (someone left the laptop open, but is not using it anymore) in which you want to terminate such connections and put the virtual cluster to sleep.

There are multiple ways to approach this problem of active connections preventing a virtual cluster from sleeping:

1. You can tell the platform to timeout idle streaming connections (such as kubectl exec, kubectl port-forward etc.) after a certain time period with the annotation loft.sh/streaming-connection-idle-timeout: '3600' on a cluster. With this annotation set, the platform will close connections automatically that are idle after the given seconds. By default, the platform will not timeout any connections. This will only apply to new opened connections to that cluster and not affect already running connections.
2. Tell the platform to ignore all active connections for determining virtual cluster activity via the annotation sleepmode.loft.sh/ignore-active-conntections: 'true'. This will put a virtual cluster to sleep even though there still might be open connections such as kubectl exec or kubectl port-forward.
3. Configure your kubelets with the flag --streaming-connection-idle-timeout duration(see Kubernetes docs). This behaves essentially as option 1, however this configuration is independent of the platform.