Deploy OpenTelemetry Collector + Prometheus on Kubernetes: Observability Stack
Tags:
OpenTelemetryKubernetesmetricslogsPrometheus
Created by:
Siddiqui
28 uses
Ingest multi-signal telemetry through an OpenTelemetry Collector pipeline and pull processed metrics directly into a Prometheus server. This setup configures OTLP ingestion over gRPC/HTTP, Kubernetes metadata enrichment, and a Prometheus scraping target inside an isolated namespace.
| Component | Type | Port | Role |
|---|---|---|---|
| otel-system1 | Namespace | N/A | Provides logical isolation for all telemetry stack resources |
| otel-collector-sa | ServiceAccount | N/A | Authorizes the OpenTelemetry Collector to query Kubernetes API metadata |
| otel-collector-role | Role | N/A | Grants read permissions for pod and namespace resources |
| otel-collector-rolebinding | RoleBinding | N/A | Binds otel-collector-sa to otel-collector-role |
| otel-collector-config | ConfigMap | N/A | Defines OTLP receivers, k8sattributes processors, and pipeline exporters |
| otel-collector | Service | 4317, 4318, 8889, 13133 | Exposes OTLP gRPC/HTTP ingest, Prometheus metrics, and health probes |
| otel-collector | Deployment | 4317, 4318, 8889, 13133 | Runs the OpenTelemetry Collector Contrib instance |
| prometheus-config | ConfigMap | N/A | Configures Prometheus scrape jobs for self-monitoring and OTel endpoints |
| prometheus | Service | 9090 | Exposes the Prometheus web UI and API internally |
| prometheus | Deployment | 9090 | Runs the Prometheus server time-series database instance |
Applications push traces and metrics to otel-collector via OTLP gRPC on port 4317 or HTTP on port 4318. The Collector uses otel-collector-sa to query the local Kubernetes API, enriching incoming metrics with pod and namespace metadata via k8sattributes. Processed metrics route to an internal Prometheus exporter listening on port 8889. The prometheus Deployment periodically scrapes otel-collector:8889 over the cluster network and stores time-series data locally on an emptyDir volume, accessible through prometheus:9090.
otel-collector.kubectl get pods -n otel-system1 to confirm otel-collector and prometheus pods display 1/1 in the Ready column.kubectl port-forward svc/prometheus 9090:9090 -n otel-system1 and open http://localhost:9090 in your browser.otel-collector job status shows UP.This template provisions a complete namespace-scoped observability stack on Kubernetes containing an OpenTelemetry Collector pipeline, RBAC metadata enrichment, and a Prometheus scraping server.