Skip to main content

Metrics And Monitoring

Logging​

By default, vCluster Platform will print out structured logs to the console with a log level of info.

These logs will include constant log messages, a human-readable timestamp, log levels, the file and line number of the source caller, and with variables in the body:

2023-07-11 09:20:56     INFO    controller-runtime.metrics      metrics/listener.go:44  Metrics server is starting to listen    {"component": "loft", "addr": "127.0.0.1:12000"}
2023-07-11 09:20:56     INFO    initialize/context.go:68        Initialize...   {"component": "loft"}
2023-07-11 09:20:56     INFO    initialize/context.go:72        Ensure certificates...  {"component": "loft"}
2023-07-11 09:21:02     INFO    initialize/context.go:80        Ensure crds...  {"component": "loft"}

You can change the log level by setting the helm value logging.level to one of the following values: debug, info, error, and upgrading your vCluster Platform release with the following command:

helm upgrade loft vcluster-platform --repo https://charts.loft.sh/ \
  --namespace vcluster-platform \
  --reuse-values \
  --set logging.level=error # or debug or info

JSON Output Encoding​

In addition to the console output, vCluster Platform also supports printing logs encoded in JSON format.

This output format is helpful for structured logging, which allows you to search and filter logs easily.

You can change the log output by setting the helm value logging.encoding to one of the following values: json, and upgrading your vCluster Platform release with the following command:

helm upgrade loft vcluster-platform --repo https://charts.loft.sh/ \
  --namespace vcluster-platform \
  --reuse-values \
  --set logging.encoding=json

These JSON-encoded logs will include constant log messages with different variables in the body, like the default console output:

{"level":"info","ts":1689067388.209614,"logger":"controller-runtime.metrics","caller":"metrics/listener.go:44","msg":"Metrics server is starting to listen","component":"loft","addr":"127.0.0.1:12000"}
{"level":"info","ts":1689067388.2099726,"caller":"initialize/context.go:68","msg":"Initialize...","component":"loft"}
{"level":"info","ts":1689067388.2100549,"caller":"initialize/context.go:72","msg":"Ensure certificates...","component":"loft"}
{"level":"info","ts":1689067389.678155,"caller":"initialize/context.go:80","msg":"Ensure crds...","component":"loft"}

The JSON encoded logs will contain at least the following fields:

  • level: The log level of the message
  • ts: The timestamp of the message (floating-point number of seconds since the Unix epoch)
  • caller: The file and line number of the message
  • msg: The message itself
  • component: The vCluster Platform component that emitted the message

Metrics​

vCluster Platform exposes several prometheus conformant metrics that can be scraped.

Create the Prometheus ServiceMonitor​

The metrics can be scraped with the included ServiceMonitor in the vCluster Platform chart, which can be deployed with helm. In order for this to work, make sure you have installed a prometheus operator into your cluster. You'll also need to upgrade your vCluster Platform release with the following command to create the ServiceMonitor:

# Make sure you change serviceMonitor.namespace if your prometheus installation requires this.
helm upgrade loft vcluster-platform --repo https://charts.loft.sh/ \
  --namespace vcluster-platform \
  --reuse-values \
  --set serviceMonitor.enabled=true \
  --set serviceMonitor.namespace=loft

Access the metrics endpoint directly without a ServiceMonitor​

vCluster Platform will expose its internal metrics at https://my-loft-subdomain.my-url.com/metrics. By default vCluster Platform will require an Authorization http header field with a kubernetes bearer token that has access to the non resource url /metrics in the kubernetes cluster where vCluster Platform is installed. If the header is not provided, vCluster Platform will deny the request.

Disable metrics authentication

If you wish to scrape metrics without authentication, you can disable it via the environment variable INSECURE_METRICS=true in the vCluster Platform helm chart.

If you have kubernetes service account token that has the appropriate rights, you can access the metrics via curl:

# Add an optional --insecure, if your vCluster Platform instance is using an untrusted certificate
$ curl https://my-vcluster-platform-subdomain.my-url.com/metrics -H "Authorization: Bearer eyJhbGci..."

# HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
...

Which metrics are exposed?​

vCluster Platform collects metrics from its internal components and merges them together into a single metrics endpoint at /metrics. The following component metrics are merged together:

  • API gateway metrics: vCluster Platform exposes metrics for its internal api gateway such as the total amount of requests (apigateway_ENDPOINT_request_total), request latency (apigateway_ENDPOINT_request_duration_seconds) and request sizes (apigateway_ENDPOINT_response_sizes) for the different endpoints. The metrics have different labels depending on the endpoint. The following endpoints and metrics are available:
    • ui: all requests that target vCluster Platform UI assets (metrics are apigateway_ui_request_total, apigateway_ui_request_duration_seconds, apigateway_ui_response_sizes)
    • auth: all requests that target vCluster Platform authentication endpoints (such as login, token refresh etc.)
    • kubernetes: all forwarded requests that target a kubernetes cluster. The targeted backend is provided by a label and can be either management (the integrated vCluster Platform kubernetes api server), cluster (a request forwarded to a connected cluster) or vcluster (a virtual kubernetes cluster deployed by vCluster Platform). Further labels are provided as well such as kubernetes resource, api version, api group, subresource etc.
    • oidc: all requests that target the internal OIDC server
  • Integrated Kubernetes API server metrics: all internal kubernetes api server metrics are exposed. An incomplete list can be found in the kubernetes repository
  • Integrated Controller manager metrics: all vCluster Platform management cluster controller metrics are exposed.
  • Go runtime metrics: some general metrics about the golang runtime and resource usage are exposed.