Federated Monitoring with Prometheus in Kubernetes

In this guide, we will set up Prometheus federation for monitoring Kubernetes clusters. This includes deploying a local Prometheus instance inside the cluster and a global Prometheus instance outside the cluster.

Deploying Prometheus Inside the Cluster

Install Node Exporter

First, we need to deploy node-exporter to collect system metrics from all nodes.

kubectl apply -f node-exporter.yaml

Here's the DaemonSet configuration:

    {`apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: kube-system
  labels:
    k8s-app: node-exporter
spec:
  selector:
    matchLabels:
      k8s-app: node-exporter
  template:
    metadata:
      labels:
        k8s-app: node-exporter
    spec:
      containers:
      - image: prom/node-exporter
        name: node-exporter
        ports:
        - containerPort: 9100
          protocol: TCP
          name: http
      hostNetwork: true
      hostPID: true
      hostIPC: true
      restartPolicy: Always`}

Deploy Prometheus

Create a ConfigMap with the Prometheus configuration:

kubectl apply -f configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: kube-system
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    scrape_configs:
    - job_name: 'kubernetes-apiservers'
      kubernetes_sd_configs:
      - role: endpoints
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https`}

Apply RBAC permissions:

kubectl apply -f rbac.yaml

Deploy Prometheus:

kubectl apply -f prometheus_deploy.yml
kubectl apply -f prometheus_svc.yml

Verify deployment:

kubectl get deployment -o wide -n kube-system

kubectl get svc -o wide -n kube-system

Install Grafana

To visualize the collected data:

kubectl apply -f grafana_deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana-core
  namespace: kube-system
  labels:
    app: grafana
    component: core
spec:
  selector:
    matchLabels:
      app: grafana
      component: core
  replicas: 1
  template:
    metadata:
      labels:
        app: grafana
        component: core
    spec:
      containers:
        - image: grafana/grafana:4.2.0
          name: grafana-core
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 100m
              memory: 100Mi
            requests:
              cpu: 100m
              memory: 100Mi
          env:
            - name: GF_AUTH_BASIC_ENABLED
              value: "true"
            - name: GF_AUTH_ANONYMOUS_ENABLED
              value: "false"
          readinessProbe:
            httpGet:
              path: /login
              port: 3000
          volumeMounts:
            - name: grafana-persistent-storage
              mountPath: /var
      volumes:
        - name: grafana-persistent-storage
          emptyDir: {}

apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: kube-system
  labels:
    app: grafana
    component: core
spec:
  type: NodePort
  ports:
    - port: 3000
  selector:
    app: grafana
    component: core

Deploying the Global Prometheus Instance

For cross-cluster monitoring, deploy a global Prometheus instance outside Kubernetes using Docker:

Test access:

Troubleshooting

If Prometheus cannot scrape kube-proxy, check if metrics are bound to 127.0.0.1:10249:

netstat -lntp | grep 10249

For systemd installations, modify kube-proxy.conf:

--metrics-bind-address=<your-node-ip>