Project: a clickstream pipeline
Request
The Cedar Trail product lead needs session metrics for a marketplace funnel.
Data
20,000 events/s, 500 bytes/event, retries within 24 hours, offline uploads up to 20 minutes late, and device clocks up to four minutes wrong. Workload figures are fictional exercise inputs.
Done
Deliver a durable ingestion graph, raw-volume estimate, retry and late-event policies, a versioned result model, and a checked replay recommendation.
Cedar Trail runs a fictional outdoor marketplace. Its product team needs page views and completed checkouts by session, with provisional results within one minute and corrected daily totals. You'll keep one design through seven steps. Each step adds a requirement the earlier diagram doesn't yet meet.
Browser events retry when a response is lost. Some phones remain offline for 20 minutes, and their clocks can be wrong by four minutes. Preserve an immutable input record before applying session rules. Then separate processing time, event time, and correction time in the model. A queue that delivers each stored message once would still leave duplicate browser submissions to handle. At the end, your board must explain both the fast result and the route that can rebuild it when the session definition changes.
Browser: event_id, user_id, session_id, occurred_at
Collector: received_at, schema_version
Raw archive: immutable event payload
Session result key: session_id + session_rule_version
Separate the event identity from the session result identity. A new session rule can rebuild the same raw events.