Prometheus, Grafana and Loki Installation (Kubeadm)

This guide walks through installing Prometheus, Grafana, and Loki on a Kubeadm-based Kubernetes cluster. These components provide metrics collection, visualization, and centralized log aggregation for monitoring your Capacity Private Cloud deployment.

By the end of this guide you will have a fully functional monitoring stack with persistent storage, ingress-based access to Grafana, and a pre-built dashboard tailored to the platform.

Prerequisites

Before proceeding, ensure your Kubeadm cluster is operational and that you have Helm 3 and kubectl configured with cluster-admin access. You will also need an NGINX Ingress Controller installed if you plan to expose Grafana externally.

Setup Metrics Server

The Kubernetes Metrics Server provides resource usage data (CPU, memory) used by kubectl top and the Horizontal Pod Autoscaler. Install it first, as Prometheus will scrape these metrics.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl edit deploy metrics-server -n kube-system

Add the following lines under the container args section. The --kubelet-insecure-tls flag is typically required in Kubeadm environments where kubelet certificates are self-signed:

- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP

System Configuration

Increase Open Files Limit

Prometheus and Loki can open a large number of file descriptors. Increase the system limit to avoid issues under load:

sudo vi /etc/security/limits.conf

Add the following lines at the bottom of the file:

* hard nofile 1000000
* soft nofile 1000000

Increase inotify Limits

Promtail uses inotify to watch log files. The default limits are often too low for production clusters:

sudo vi /etc/sysctl.conf

Add the following lines at the end of the file:

fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=524288

Apply the changes:

sudo sysctl --system

Persistent Storage Setup

All three components (Prometheus, Grafana, and Loki) require persistent storage to retain data across pod restarts. The steps below create local host-path volumes suitable for single-node or development clusters.

Create Directories

sudo mkdir -p /data/prometheus
sudo mkdir -p /data/grafana
sudo mkdir -p /data/loki

Set Permissions

sudo chmod -R 777 /data/prometheus
sudo chmod -R 777 /data/grafana
sudo chmod -R 777 /data/loki

Create Namespace and Storage Class

kubectl create ns monitoring

Create a file named storageClass.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

kubectl apply -f storageClass.yaml -n monitoring

Create Persistent Volumes

Create a file named persistentVolumes.yaml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/prometheus
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ubuntu-k8s-01 # Replace with your node name

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/grafana
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ubuntu-k8s-01 # Replace with your node name

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: loki-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/loki
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ubuntu-k8s-01  # Replace with your node name

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-alert-pv
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/prometheus
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ubuntu-k8s-01  # Replace with your node name

kubectl apply -f persistentVolumes.yaml -n monitoring

Create Persistent Volume Claims

Create a file named persistentVolumeClaims.yaml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/prometheus
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ubuntu-k8s-01 # Replace with your node name

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/grafana
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ubuntu-k8s-01 # Replace with your node name

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: loki-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/loki
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ubuntu-k8s-01  # Replace with your node name

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-alert-pv
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/prometheus
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ubuntu-k8s-01  # Replace with your node name

kubectl apply -f persistentVolumeClaims.yaml -n monitoring

Install Monitoring Stack via Helm

Add Helm Repositories

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Install Prometheus

Create a file named prometheus-values.yaml. The scrape interval below is set to 10 seconds for near-real-time metrics; adjust this based on your performance and storage requirements:

server:
  global:
    scrape_interval: 10s     # Default is 1m - set to your desired interval
    scrape_timeout: 7s
    evaluation_interval: 10s
  persistentVolume:
    enabled: true
    existingClaim: "prometheus-server"
    storageClass: "local-storage"
    size: 10Gi
alertmanager:
  persistentVolume:
    enabled: true
    existingClaim: "storage-prometheus-alertmanager-0"
    storageClass: "local-storage"
    size: 2Gi

helm upgrade --install prometheus prometheus-community/prometheus -f prometheus-values.yaml -n monitoring

Install Grafana

Create a file named grafana-values.yaml. This configuration pre-provisions both Prometheus and Loki as data sources so they are available immediately after installation:

persistence:
  enabled: true
  existingClaim: "grafana"
  storageClassName: local-storage
  size: 5Gi
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.monitoring.svc.cluster.local
      access: proxy
      isDefault: true
    - name: Loki
      type: loki
      access: proxy
      url: http://loki.monitoring.svc.cluster.local:3100/

helm upgrade --install grafana grafana/grafana -f grafana-values.yaml -n monitoring

Install Loki

Create a file named loki-values.yaml. This deploys Loki in single-binary mode, which is appropriate for small to medium clusters. For larger deployments, consider the microservices deployment mode:

deploymentMode: SingleBinary

loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1  
  schemaConfig:
    configs:
      - from: "2020-10-24"
        store: boltdb-shipper
        object_store: filesystem
        schema: v11
        index:
          prefix: index_
          period: 24h
  storage:
    type: filesystem
  limits_config:
    allow_structured_metadata: false
    volume_enabled: true

singleBinary:
  replicas: 1
  persistence:
    enabled: true
    existingClaim: "storage-loki-0"
    storageClass: local-storage
    size: 10Gi
  extraArgs: []

canary:
  enabled: false
gateway:
  enabled: false

read:
  replicas: 0
write:
  replicas: 0
backend:
  replicas: 0

helm upgrade --install loki grafana/loki -f loki-values.yaml -n monitoring

Install Promtail

Promtail is the log shipping agent that sends logs from your cluster nodes to Loki. Create a file named promtail-values.yaml:

config:
  clients:
    - url: http://loki:3100/loki/api/v1/push
  positions:
    filename: /data/positions.yaml
  scrape_configs:
    - job_name: varlogs
      static_configs:
        - targets:
            - localhost
          labels:
            job: varlogs
            __path__: /var/log/*.log
# Optional: persistent volume for positions file
persistence:
  enabled: true

helm upgrade --install promtail grafana/promtail -f promtail-values.yaml -n monitoring

Expose Grafana via Ingress

To access Grafana from outside the cluster, create an Ingress resource. This requires an NGINX Ingress Controller already running in your cluster.

Create a file named grafana-ingress.yaml. Replace grafana.testmachine.com with your actual hostname:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana-ingress
  namespace: monitoring
spec:
  ingressClassName: nginx
  rules:
    - host: grafana.testmachine.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: grafana
                port:
                  number: 80

kubectl apply -f grafana-ingress.yaml -n monitoring

Access Grafana

Retrieve the auto-generated admin password:

kubectl get secret grafana -o jsonpath="{.data.admin-password}" -n monitoring | base64 -d ; echo

Open your browser and navigate to http://grafana.testmachine.com (or whatever hostname you configured in the Ingress). Log in with:

• Username: admin
• Password: the value returned by the command above

Import the Monitoring Dashboard

A pre-built Grafana dashboard is available that provides visibility into your Capacity Private Cloud cluster health, resource utilization, and service metrics.

Step 1: Download the dashboard JSON file from:
https://lumenvox-public-assets.s3.amazonaws.com/third-party/grafana-dashboards/LumenVox-KubeAdm-Grafana-Dashboard.json

Step 2: In Grafana, click Home then Dashboards.

Step 3: Click + Create Dashboard and select the Import Dashboard option.

View full size

Step 4: Upload the LumenVox-KubeAdm-Grafana-Dashboard.json file, select Prometheus as the data source, and click Import.

Once imported, you will see a comprehensive dashboard displaying cluster and application metrics:

View full size

Related Articles

• Prometheus Metrics Available
• Installing KEDA (Kubernetes Event-driven Autoscaler)