Kubernetes

Real-time Data Dashboard

Kafka-to-ClickHouse streaming analytics handling 100K events/second with sub-300ms query response.

Real-time Data Dashboard — ProjectsHub
100K+
Events/Second
<300ms
Query Latency
100M+
Rows in ClickHouse
4.8★
Client Rating
Overview

What is this project?

A fully cloud-native streaming analytics stack built for a FinTech client needing sub-second visibility into transaction events. Apache Kafka ingests 100K+ events per second across 12 topics. Spark Structured Streaming consumers apply windowed aggregations (1-min, 5-min, 1-hour), detect anomalies using a streaming z-score algorithm, and land results into ClickHouse. ClickHouse materialized views serve the Superset dashboards with query response times under 300ms even at 100M+ row tables.

The entire stack runs on Kubernetes with Helm charts. Kafka and ClickHouse are horizontally scaled via custom HPA policies triggered by consumer lag and query throughput respectively. A GitOps workflow (ArgoCD) handles deployments, making environment promotion from staging to production a one-click operation.

Process

Build Timeline

Phase 01 Jun 2024
Infrastructure & Kafka Setup
Provisioned Kubernetes cluster on GKE, deployed Kafka via Helm with 12-topic configuration, and load-tested to 150K events/second.
Phase 02 Aug 2024
Spark Streaming Consumers
Built Spark Structured Streaming jobs for 1-min, 5-min, and 1-hour windowed aggregations with exactly-once semantics and a streaming anomaly detector.
Phase 03 Sep 2024
ClickHouse & Superset
Designed ClickHouse schema with materialized views for common aggregation patterns. Built 8 Superset dashboards with drilldown filters for the client BI team.
Phase 04 Oct 2024
GitOps & Autoscaling
Set up ArgoCD for GitOps deployments, configured HPA policies, and handed over runbooks for on-call operations to the client team.
Get This Project

What's Included

₹6,999
one-time payment · lifetime access
  • Full source code (Kafka producers, Spark jobs, Helm charts)
  • ClickHouse schema + materialized view definitions
  • ArgoCD GitOps pipeline configuration
  • Superset dashboard JSON exports
  • 90-minute video walkthrough
  • 60-day email support
Get Source Code
More Work

Related Projects

View all
Finance Agent — Invoice & Document Intelligence — AI project by ProjectsHub
Django, RAG, Document AI, OCR, Analytics
Finance Agent — Invoice & Document Intelligence
A finance document agent that extracts invoices into structured, chartable data and reads statements, P&L and reports s…
 Customer Churn Pipeline — AI project by ProjectsHub
XGBoost, MLflow, Prefect, PostgreSQL, Grafana, Gr…
Customer Churn Pipeline
End-to-end MLOps pipeline for churn prediction with automated retraining, data drift detection, and Grafana monitoring …