Stage 1 of 7 · Requirements analysis
Find the real requirement
Task brief & requirements
Marcus (Finance): "Just make the dashboard real-time. Everything else is fine."
Decision4 options
What should you do first?
Lesson & team handoff3 notes
- Stakeholders describe symptoms; engineers must find the underlying requirement.
- Double-counted points to duplicate events — a correctness problem, not a latency one.
- Senior engineers write requirements as measurable outcomes: e.g. " matches exchange within 0.1%" and "data freshness < 15 min".
Glossary context
Key terms in this lesson (3)
Tell us how familiar you are and we'll tailor the explanation. Open full glossary
Stakeholder
Anyone affected by or with authority over the work — finance, product, ops, customers.
Spend
The money an advertiser has paid (or owes) for ads that ran.
Invoice
A bill from a supplier stating what is owed; finance treats it as the source of truth.
Specification & architecture maps
Project documentation
Delivery contract
| Requirement | Acceptance target |
|---|---|
| Freshness | Less than 15 minutes |
| Spend accuracy | 99.9% |
| Replay safety | No additional spend after replaying a batch |
| Deduplication key | (exchange_id, bid_id) |
Data contract
exchange_id, bid_id, ts and spend are required fields. Keep the latest timestamp for each natural key. Decide and document a deterministic tie-breaker for equal timestamps.
event = {
"exchange_id": "exchange_a",
"bid_id": "bid_042",
"ts": "2026-01-01T10:00:00Z",
"spend": 0.12,
}
Acceptance checklist
- Replayed events do not double-count spend.
- A five-minute micro-batch completes within the freshness budget.
- Invalid events are isolated and can be inspected.
- Row counts, rejected events and freshness are observable.
Architecture decision record
Decision: at-least-once delivery with idempotent warehouse writes.
Trade-off: micro-batching reduces operational complexity for a nine-person team. Revisit streaming when verified consumer needs cannot tolerate the agreed freshness window.
Failure recovery: retry the same batch with the same natural keys; only commit the consumed offsets after a successful load.