Skip to main content

Analytics Service 📊

The Analytics Service is responsible for collecting, processing, and exposing business metrics in ShopVerse using an event-driven analytics pipeline.

It provides near real-time insights without impacting core transactional services.

🎯 Responsibilities​

The Analytics Service handles:

  • Consuming domain events from Kafka
  • Processing events using stream processing
  • Storing analytics data in ClickHouse
  • Exposing aggregated metrics via APIs
  • Supporting business dashboards and reporting

🧠 Why a Separate Analytics Service?​

Analytics workloads are:

  • Read-heavy
  • Aggregation-focused
  • Computationally expensive

Separating analytics ensures:

  • No performance impact on transactional services
  • Independent scaling
  • Technology freedom (OLAP databases)
  • Clean separation between operations and insights

🏗️ High-Level Architecture​

📡 Event Ingestion Flow​

📦 Events Consumed​

Event TypeSource ServicePurpose
USER_REGISTEREDAuth ServiceUser growth
ORDER_PLACEDOrder ServiceOrder metrics
PAYMENT_SUCCESSPayment ServiceRevenue tracking
PRODUCT_CREATEDProduct ServiceCatalog insights

All events follow a standard analytics schema.

🧩 Analytics Event Model​

{
  "eventType": "PAYMENT_SUCCESS",
  "service": "PAYMENT_SERVICE",
  "userEmail": "user@example.com",
  "entityId": "ORD123",
  "amount": 2499.00,
  "timestamp": "2026-01-21T11:45:00",
  "metadata": {
    "paymentMode": "UPI"
  }
}

🗄️ Data Storage (ClickHouse)​

ClickHouse is used because:

  • Columnar storage
  • Extremely fast aggregations
  • High write throughput
  • Ideal for OLAP workloads

Example Metrics Stored:​

  • Total users
  • Orders per day
  • Revenue per day
  • Revenue by payment method
  • Service-level event counts

🌐 Analytics APIs​

MethodEndpointRoleDescription
GET/api/analytics/dashboardADMINGet dashboard metrics
GET/api/analytics/revenueADMINRevenue trends
GET/api/analytics/ordersADMINOrder statistics

These APIs are read-only.

🔐 Security Model​

  • All APIs accessed via API Gateway

  • Only ADMIN role allowed

  • Identity verified using:

    • X-User-Email
    • X-User-Role

Analytics data is never exposed to customers.

🔄 Stream Processing Strategy​

Apache Flink handles:

  • Event ingestion
  • Transformation
  • Aggregation
  • Windowing (daily, hourly metrics)

Benefits:​

  • Near real-time analytics
  • Fault tolerance
  • Exactly-once semantics
  • Horizontal scalability

🛡️ Reliability & Fault Tolerance​

  • Kafka retains events on failure
  • Flink checkpoints state
  • ClickHouse ensures data durability
  • Queries are idempotent
  • No data loss during restarts

⚠️ Failure Scenarios​

❌ Kafka Lag​

  • Analytics delayed
  • Core services unaffected

❌ Flink Job Failure​

  • Job restarts from checkpoint

❌ ClickHouse Down​

  • Events buffered until recovery

⚙️ Key Components​

  • AnalyticsConsumer – Kafka consumer / Flink job
  • AnalyticsEvent – Standard event model
  • AnalyticsQueryService – ClickHouse queries
  • AnalyticsController – REST APIs
  • JdbcTemplate – ClickHouse integration
  • GatewayHeaderAuthenticationFilter – Security

📈 Scalability Considerations​

  • Kafka partitions enable parallel ingestion
  • Flink scales horizontally
  • ClickHouse supports distributed clusters
  • Analytics Service APIs are stateless

🧪 Testing Strategy​

  • Unit tests for query logic
  • Mock Kafka events for Flink jobs
  • Validation of aggregations
  • Security tests for admin-only access

📌 Summary​

The Analytics Service provides:

  • Real-time business insights
  • Zero impact on core workflows
  • Scalable stream processing
  • Production-grade analytics architecture

It turns ShopVerse operational data into actionable intelligence.