Skip to main content

Idempotency Strategy ♻️

This document explains how ShopVerse handles duplicate events and retries using idempotent consumer design.

Idempotency is critical in event-driven systems.

🎯 Why Idempotency Is Needed​

In Kafka:

  • Messages can be delivered more than once
  • Consumers can restart
  • Network failures can occur

Without idempotency: ❌ Duplicate notifications
❌ Incorrect analytics
❌ Corrupted state

🧠 Core Idea​

Processing the same event multiple times should produce the same result.

🛡️ Idempotency Strategy in ShopVerse​

1️⃣ Unique Event Identification​

Each event has a unique combination of:

  • eventType
  • entityId
  • timestamp

Optionally:

  • Explicit eventId (future-ready)

2️⃣ Consumer-Side Checks​

Before processing:

  • Check if event already processed
  • Skip if duplicate
  • Proceed only once

Example:

  • Notification already exists → skip
  • Analytics row already inserted → skip

🔁 Offset Management​

  • Kafka offsets are committed after successful processing
  • Prevents partial processing
  • Ensures at-least-once delivery with safe handling

🧾 Example: Notification Service​

If the event is replayed:

  • Duplicate detected
  • Processing skipped

📊 Example: Analytics Service​

  • Aggregations are idempotent
  • Re-processing produces same result
  • ClickHouse handles duplicate-safe inserts

⚠️ Failure Scenarios​

ScenarioHandling
Consumer crashKafka replays event
Duplicate eventSafely ignored
Partial failureOffset not committed

❌ Why Not Exactly-Once Everywhere?​

Exactly-once:

  • Adds complexity
  • Not required for all domains

ShopVerse uses:

  • At-least-once delivery
  • Idempotent consumers

This is the industry-standard tradeoff.

🧠 Interview One-Liner​

“Kafka guarantees at-least-once delivery, so we design consumers to be idempotent to safely handle duplicates.”

📌 Summary​

ShopVerse idempotency ensures:

  • Correct behavior under retries
  • Safe event reprocessing
  • Reliable async workflows

It is a must-have design for production event-driven systems.