Tidewater Cloud asks for a telemetry pipeline for 20,000 hosts producing 200 MB/s of logs plus metrics. Deliver the scope, a log volume estimate, a time series table, the pipeline, stream retention, a downsampling policy and a cardinality guard.
How interviewers level this design
- Junior: puts a durable stream between hosts and stores and uses different stores for metrics and logs.
- Mid-level: sizes volume, sets retention to cover consumer lag and downsamples metrics.
- Senior: controls cardinality, plans backpressure and sampling, and estimates cost per day of retention.
Tidewater Cloud collects metrics and logs from 20,000 hosts. Engineers need the last hour of metrics in under a second during an incident, and logs must stay searchable for 30 days. The data never stops arriving, and when a store slows down the hosts must not lose data or fall over trying to resend it.
A durable event stream in the middle absorbs bursts and lets each consumer go at its own pace. Metrics are aggregated into time series and downsampled for long retention; logs are indexed for search and archived raw. You will size the log volume, define the time series table, draw the pipeline, set stream retention, choose a downsampling policy and stop one careless label from exploding the metric store.
Buffer telemetry in a stream, store metrics and logs in fit-for-purpose stores and control cardinality and retention.