Skip to main content

Monitoring

Monitor your self-hosted Primary Site using Prometheus.

Configure Prometheus​

The Primary Site services expose various application metrics that Prometheus can scrape and ingest.

The first option is to use one of the major cloud providers' managed integrations:

Alternatively, deploy Prometheus to your cluster directly using the directions below.

Create Helm repositories and namespace​

Add Prometheus Helm repositories:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update

Create a Kubernetes namespace for Prometheus:

$ kubectl create namespace prometheus

Install Prometheus​

Install Prometheus to your Kubernetes cluster:

$ helm install prometheus prometheus-community/prometheus -n prometheus

Configure pods for scraping​

Locate the streamService, siteController, and inboxListener sections in your values.yaml file.

For each section, add a pod annotation under deployment instructing Prometheus to scrape the pod:

siteController:
  deployment:
    podAnnotations:
      prometheus.io/scrape: true

Upgrade your deployment with the new configuration. You may need to alter this command with your own namespace, Helm release name, and file path:

$ helm upgrade primary-site -f values.yaml --namespace foxglove

Prometheus will now scrape the metrics endpoints of the deployed services. To confirm this is working, forward the port of the Prometheus UI to view collected metrics:

$ kubectl -n prometheus port-forward service/prometheus-server 9090:80

Visit http://localhost:9090 and enter the query {app="site-controller"}. Executing that should show a list of metrics scraped from the edge controller.

Install custom metrics adapter​

Finally, install the Prometheus adapter for the Kubernetes custom metrics API using Helm.

This assumes you have installed Prometheus into the "prometheus" namespace. If you are using a different namespace, replace the second component of the URL accordingly.

$ helm install prometheus-custom-metrics-adapter prometheus-community/prometheus-adapter -n prometheus --set prometheus.url=http://prometheus-server.prometheus.svc.cluster.local

After a couple minutes, you should see custom metrics:

$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1

If the output contains metrics, you are ready to create an autoscaler on custom metrics.

Monitor metrics​

Once Prometheus is configured, the services expose useful application metrics for monitoring.

Stream service​

  • stream_sync_ms - Time between reception of request and establishment of connections to all immediately-required objects in storage (histogram)
  • time_to_first_message_ms - Time between reception of request and first message written out (histogram)
  • files_request_latency_ms - Time to request list of data files from Foxglove API (histogram)

Inbox listener​

  • import_success_count - Number of imports successfully processed (counter)
  • import_quarantine_count - Number of imports quarantined (counter)
  • import_input_size_bytes - Size of input files in bytes (histogram)
  • import_output_size_bytes - Size of output files (processed data files) in bytes (histogram)
  • import_processing_time_seconds - Processing time for imports in seconds (histogram)
  • output_file_count - Number of output files per import (histogram)
  • input_message_count - Number of messages per import (histogram)

Site controller​

  • unleased_pending_import_count - Number of backlogged pending imports for processing (gauge)
  • oldest_unprocessed_pending_import_age_secs - Age of oldest unprocessed pending import in seconds (gauge)