Engineering glossary
Rate how familiar you are with each term — the explanation adapts to you. Your ratings are saved on this device.
Your concept progress
Requirements analysis · Find the real requirement
Architecture decisions · Choose the ingestion design
Sprint planning · Slice the epic into 2-point stories
Coding task · Write an idempotent dedup function
Integrations · Load with MERGE into the warehouse
Testing · Write the pytest that proves idempotency
Deployment · Ship it safely
New · Conceptual · Applied Back to the simulation
Side-by-side comparisons
At-least-once vs exactly-once
At-least-once + idempotent writes
- Duplicates possible on retry
- Simple offset handling
- Dedupe at the sink (MERGE)
for batch in consumer:
merge_into_warehouse(dedupe(batch))
consumer.commit() # after writeExactly-once processing
- No duplicates by design
- Transactions across every hop
- Higher latency and ops cost
producer.init_transactions() producer.begin_transaction() producer.send(out, rec) producer.send_offsets_to_transaction(...) producer.commit_transaction()
Verdict: Most teams choose at-least-once + idempotent writes: same end result, far less complexity.
Streaming vs micro-batching
Micro-batching
- Latency: minutes
- Scheduled job, easy retries
- Cheap; small teams can run it
# every 5 min (Airflow)
load_new_files('s3://bids/', since=last_run)
run_sql('MERGE INTO fact_bids ...')Stream processing
- Latency: seconds
- Always-on, stateful jobs
- Needs 24/7 on-call expertise
env.from_source(kafka) .key_by(lambda b: (b.exchange_id, b.bid_id)) .window(Tumbling.of(10s)) .reduce(latest).sink_to(warehouse)
Verdict: If the SLA is in minutes, micro-batching usually wins on cost and reliability.
Sign in to suggest a term or phrase for the glossary.
All terms
26 of 26 terms. Lessons only show the terms used in their own text.
Stakeholder
Anyone affected by or with authority over the work — finance, product, ops, customers.
Exactly-once delivery
Guarantee that every event is processed one time only — never lost, never duplicated.
At-least-once delivery
Every event arrives, but some may arrive more than once (e.g. after retries).
Idempotency
Running an operation twice gives the same result as running it once: f(f(x)) == f(x).
Micro-batching
Processing data in small, frequent batches (every few minutes) instead of event-by-event streaming.
Stream processing
Handling each event continuously as it arrives, with latency in seconds.
SLA (Service Level Agreement)
A committed target for a service, e.g. freshness, uptime or response time.
Natural key
An identifier that comes from the data itself and uniquely identifies a real-world record.
MERGE (upsert)
A SQL statement that updates matching rows and inserts new ones in a single step.
Data warehouse
A database optimised for analytics over large volumes (Snowflake, BigQuery, Redshift, Synapse).
Partitioning
Splitting a large table into chunks (usually by date) so queries only scan what they need.
Story points
A relative estimate of effort, complexity and risk for a user story in Scrum.
Vertical slicing
Splitting work into thin pieces that each deliver end-to-end value.
Acceptance criteria
Testable conditions a story must meet to be considered done.
Type hints
Python annotations declaring expected types, checked by tools like mypy.
O(n) complexity
Big-O notation: how runtime grows with input size. O(n) grows linearly.
Shadow mode
Running a new system alongside the old one on real traffic without affecting users.
Sandbox / staging / production
Separate environments to experiment, validate, then serve real users.
On-call
A rota of engineers responsible for responding to production incidents.
AI agent
A program that uses a language model to decide which tools to call, in a loop, until a goal is met.
Large language model (LLM)
A model trained on large amounts of text that predicts the next token; used for chat, summarising and code.
Retrieval-augmented generation (RAG)
Fetching relevant documents first and giving them to the model so answers are grounded in your data.
CPM
Cost per mille — what an advertiser pays for 1,000 ad impressions.
Spend
The money an advertiser has paid (or owes) for ads that ran.
Reconciliation
Checking that two records of the same money (e.g. your data vs an invoice) agree.
Invoice
A bill from a supplier stating what is owed; finance treats it as the source of truth.