Building an Event-Driven System Using Kafka on Kubernetes (Manually)

In this article, we'll walk through the manual setup of an event-driven architecture on Kubernetes using Apache Kafka. We will also highlight the time, complexity, and error-prone nature of doing it manually, followed by how KubeKanvas can simplify the process significantly.

Why Kubernetes for Event-Driven Architectures: A Fast and Portable Approach

We want to build a simple, portable system using Kubernetes, which excels at orchestrating distributed, container-based workloads. This includes a Kafka cluster, a producer service that generates events, a consumer service that processes them, and secure communication using Ingress and network policies. Why Kubernetes? Because it enables fast deployments, seamless service discovery, and scalable architecture out-of-the-box.

For a foundational deep dive into how Kubernetes orchestrates workloads at scale, read Kubernetes Architecture Series - Part 1: From Containers to Cloud-Native Orchestration.

Configuration Drift, Secrets, and Why Manual YAML Slows You Down

Writing Kubernetes manifests by hand introduces the risk of configuration drift—especially when managing secrets and network policies manually. Kubernetes YAML is verbose, and each CI/CD pipeline becomes harder to maintain due to unnecessary duplication. YAML’s whitespace sensitivity makes it prone to breaking builds. Adding security elements like Ingress and NetworkPolicies also requires domain expertise. Kubernetes has a low feedback loop for errors, often revealing mistakes only after deployment, which slows down fast pipelines and increases runtime failures.

If YAML has ever frustrated you (as it does most of us), you’ll enjoy our deeper breakdown: Why is Kubernetes YAML So Hard? (And How to Make it Easier).

Manual Deployment: Step-by-Step

Step 1: Deploy Kafka using Bitnami Helm Chart

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install kafka bitnami/kafka --set replicaCount=1

Estimated Time: 10–15 minutes

Step 2: Securing Deployments with Secrets and Namespaces

apiVersion: v1
kind: Namespace
metadata:
  name: kafka-demo
---
apiVersion: v1
kind: Secret
metadata:
  name: kafka-credentials
  namespace: kafka-demo
  labels:
    app: kafka-demo
stringData:
  username: user
  password: pass

Estimated Time: 5 minutes

Step 3: Create the Kafka Producer Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-producer
  namespace: kafka-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-producer
  template:
    metadata:
      labels:
        app: kafka-producer
    spec:
      containers:
        - name: producer
          image: myorg/kafka-producer:latest
          env:
            - name: KAFKA_BROKER
              value: kafka.kafka.svc.cluster.local:9092
            - name: TOPIC
              value: events

Estimated Time: 15–20 minutes

Step 4: Create the Kafka Consumer Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-consumer
  namespace: kafka-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-consumer
  template:
    metadata:
      labels:
        app: kafka-consumer
    spec:
      containers:
        - name: consumer
          image: myorg/kafka-consumer:latest
          env:
            - name: KAFKA_BROKER
              value: kafka.kafka.svc.cluster.local:9092
            - name: TOPIC
              value: events

Estimated Time: 15–20 minutes

Step 5: Define Services for Communication

apiVersion: v1
kind: Service
metadata:
  name: kafka-producer
  namespace: kafka-demo
spec:
  selector:
    app: kafka-producer
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-consumer
  namespace: kafka-demo
spec:
  selector:
    app: kafka-consumer
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

Estimated Time: 10 minutes

Step 6: Configure Ingress for External Access

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kafka-ingress
  namespace: kafka-demo
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - host: kafka.example.com
      http:
        paths:
          - path: /producer
            pathType: Prefix
            backend:
              service:
                name: kafka-producer
                port:
                  number: 80
          - path: /consumer
            pathType: Prefix
            backend:
              service:
                name: kafka-consumer
                port:
                  number: 80

Estimated Time: 10–15 minutes

Step 7: Define Network Policies for Security

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-producer-to-kafka
  namespace: kafka-demo
spec:
  podSelector:
    matchLabels:
      app: kafka
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: kafka-producer
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-kafka-to-consumer
  namespace: kafka-demo
spec:
  podSelector:
    matchLabels:
      app: kafka-consumer
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: kafka

Estimated Time: 15–20 minutes

Total Time Estimate (Manual Setup)

To see how Kubernetes supports full-stack content platforms as well, check out our guide on CMS deployment: Building a Modern Blog with Strapi & Kubernetes: Your First CMS in the Cloud.

CI/CD-Ready Deployments with KubeKanvas and Monitoring Tools

With KubeKanvas, your architecture is CI/CD-ready, and configurations are schema-validated to prevent configuration drift. Helm charts, GitOps-ready folder structures, and visual workflows simplify deployments. Built-in support for monitoring tools ensures visibility across the stack, allowing teams to build, observe, and iterate confidently from a single pane of glass.

KubeKanvas significantly reduces setup time, in some cases offering up to 80% time savings. It eliminates syntax-related issues through auto-validation and introduces built-in security by abstracting the configuration of network policies and Ingress into intuitive visual workflows. Teams benefit from improved collaboration as visual modeling removes ambiguity and promotes shared understanding of system architecture.

Curious how this visual-first approach holds up in machine learning workflows? Don’t miss our hands-on comparison: Real-Time ML Inference with Redis Using KubeKanvas vs. Manual YAML.

Conclusion

Manually setting up an event-driven architecture on Kubernetes is a detailed, error-prone process that can consume over 1.5 hours even for experienced engineers. Each step, from creating deployments and services to configuring secure communication with Ingress and network policies, involves intricacies that are difficult to automate without specialized tooling. With KubeKanvas, teams accelerate development, reduce configuration errors, and ensure consistent security—all from an intuitive, visual interface.

If you're exploring other ways to manage events within Kubernetes itself, don't miss our post on native event handling: How to Manage Events with a Dynamic Informer in Kubernetes.

Embrace KubeKanvas to modernize your Kubernetes workflows and ship faster with confidence.