Multi-Region Mode

Running vCluster Platform in Multi-Region Mode provides a way to avoid the central vCluster Platform instance and directly connect to a connected vCluster Platform cluster. By default, all Kubernetes contexts that are created through vCluster Platform will route all Kubernetes traffic (such as kubectl get pods etc.) through the central vCluster Platform instance. This allows vCluster Platform to handle authentication, authorization, auditing, sleep mode activity etc.

However, if you have multiple connected clusters in a vCluster Platform instance that are spread across the globe, the traffic redirection through the central vCluster Platform gateway can increase request delay. A solution to this are direct cluster endpoints, which avoid rerouting traffic to the central vCluster Platform instance and rather handle authentication, authorization etc. directly within the connected cluster through the installed vCluster Platform agent.

Enabling Multi-Region Mode

Enabling Multi-Region Mode can be achieved by exposing Direct Cluster Endpoints. These are bundled into the automatically installed vCluster Platform agent and can be enabled via an annotation on the connected cluster object.

In order for the direct cluster endpoint to work correctly, you need to expose the vCluster Platform agent either via an Ingress or a LoadBalancer / NodePort service first. The easiest way is to create an ingress.yaml like this and apply it via kubectl:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: loft-agent-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-read-timeout: "43200"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "43200"
    nginx.ingress.kubernetes.io/proxy-buffers-number: "8 32k"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "32k"
    nginx.org/websocket-services: loft-agent
    cert-manager.io/cluster-issuer: my-cluster-issuer
spec:
  rules:
    - host: my-cluster-endpoint.com
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: loft-agent
                port:
                  number: 80
  tls:
    - hosts:
        - my-cluster-endpoint.com
      secretName: name-of-tls-secret

Then apply the ingress via kubectl:

kubectl apply -n vcluster-platform -f ingress.yaml
ingress.networking.k8s.io/loft-agent-ingress created

tip

As kubectl requires an HTTPS connection, the direct cluster endpoint has to be reachable by HTTPS as well. By default, the ingress will use a self-signed certificate, however this is insecure and it is recommended to use either cert-manager to issue a Let's Encrypt certificate or to use a trusted self signed certificate.

After creating the ingress, please make sure that the direct cluster endpoint is reachable:

# Omit --insecure if have a valid TLS certificate
curl https://my-cluster-endpoint.com/healthz -I --insecure

HTTP/2 200
date: Mon, 30 Aug 2021 08:40:20 GMT
access-control-allow-headers: *
access-control-allow-methods: POST, GET, OPTIONS, PUT, DELETE, PATCH
access-control-allow-origin: *
cache-control: no-cache, private
strict-transport-security: max-age=15724800; includeSubDomains

Enable a Direct Cluster Endpoint

In order to tell vCluster Platform to use a direct cluster endpoint for a connected cluster, you will need to add an annotation to the cluster object.

apiVersion: management.loft.sh/v1
kind: Cluster
metadata:
  name: test
  annotations:
    # Domain of the direct cluster endpoint
    loft.sh/direct-cluster-endpoint: my-direct-cluster-endpoint.com
    #
    # Optional option to tell vCluster Platform the endpoint is insecure
    #
    #loft.sh/direct-cluster-endpoint-insecure: 'true'
    #
    # Optional option to use this base64 encoded ca cert for the endpoint
    #
    #loft.sh/direct-cluster-endpoint-ca-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tC...
spec:
  config: {}

The following annotations can be specified:

• loft.sh/direct-cluster-endpoint: This is the domain name that should be used to connect to the cluster directly. This usually corresponds to the ingress host that you have specified earlier. As soon as this annotation is set, the direct cluster endpoint for the connected cluster is turned on and will be preferred in future over the central vCluster Platform instance
• loft.sh/direct-cluster-endpoint-insecure (Optional): If this is true, vCluster Platform will create Kubernetes contexts with insecure-skip-tls-verify: true for this endpoint
• loft.sh/direct-cluster-endpoint-ca-data (Optional): Base64 encoded ca cert of the direct cluster endpoint

Using a Direct Cluster Endpoint

If a direct cluster endpoint is enabled via the loft.sh/direct-cluster-endpoint annotation on a connected cluster, the vCluster Platform commands vcluster pro space use, vcluster pro cluster use, vcluster pro use, vcluster pro space create and vcluster pro create will now use the direct cluster endpoint instead of the central vCluster Platform gateway.

The CLI will also notify you when running one of the above commands if a direct cluster endpoint is used:

vcluster pro space use my-space --cluster my-connected-cluster
[info]   Using direct cluster endpoint at https://my-cluster-endpoint.com
[done] √ Successfully updated kube context to use space my-space in cluster my-connected-cluster

You can then check which server is contacted by running:

kubectl get ns -v 6
I0624 15:53:30.287997   70943 loader.go:379] Config loaded from file:  /xxx/.kube/config
I0624 15:53:30.307530   70943 round_trippers.go:445] GET https://my-cluster-endpoint.com/kubernetes/cluster/api/v1/namespaces?limit=500 200 OK in 13 milliseconds
NAME              STATUS   AGE
default           Active   164m
kube-node-lease   Active   164m
kube-public       Active   164m
kube-system       Active   164m

As you can see, kubectl now routes the traffic directly to the direct cluster endpoint instead of the central vCluster Platform instance.