NVIDIA Nemotron 3

Latency is the time a single order takes from ticket to plate; throughput is how many orders the kitchen finishes per hour.
A kitchen with one chef (low parallelism) has low latency — each dish gets the chef’s full attention — but low throughput. Adding more chefs (or a larger grill) raises throughput because many tickets are cooked simultaneously, yet each chef now shares the stove, prep space, and ingredients. Contention for the grill, limited burners, and queueing for the pass increase the time any single dish spends waiting, so latency grows.

# Simplified model
def serve(orders, chefs):
    # each chef pulls next order, but all share one grill
    for order in orders:
        chef = chefs.next()
        chef.cook(order)          # may wait for grill

When chefs > grill capacity, orders queue for the grill, raising per‑order latency even though total orders per hour (throughput) rises.

Use an idempotency key generated before processing. Store the result in a table with the key as a unique constraint. On retry, attempt to insert the result; if a duplicate key error occurs, fetch and return the existing result.

def process_job(job_id, idempotency_key): try: result = do_work(job_id) db.execute( "INSERT INTO job_results (idempotency_key, result) VALUES (?, ?)", idempotency_key, result ) except UniqueViolation: result = db.query( "SELECT result FROM job_results WHERE idempotency_key = ?", idempotency_key ) return result

Consumer

idempotency_key = generate_uuid() try: result = process_job(job.id, idempotency_key) queue.ack(message) except Exception: queue.nack(message) # retry with same key

The first successful write commits the result. Subsequent retries detect the existing key and reuse the original output, preventing duplicate side effects while preserving exactly-once semantics.

Design: Eager Loading (Batch Fetching) Replace the loop executing a query per row with a single WHERE IN query fetching all related data upfront.

Code (Conceptual ORM Example):

# Before (N+1): 1 query for parents + 30 queries for children
parents = db.query(Parent).limit(30).all()
for p in parents:
    print(p.children)  # Triggers lazy load per row

# After: 2 Queries Total
parents = db.query(Parent).limit(30).all()
parent_ids = [p.id for p in parents]
children_map = db.query(Child).filter(Child.parent_id.in_(parent_ids)).all()
# Map children to parents in memory (O(N))

Query Count: 2 queries (1 for parents, 1 for all children), regardless of row count.

Measurement:

1. Query Log/Profiler: Enable DB logging (e.g., SET log_statement = 'all' in Postgres, Django Debug Toolbar, Laravel Telescope) and count SELECT statements before/after.
2. APM Tools: Use Datadog/New Relic to compare Database Time and Query Count metrics for the endpoint.
3. Load Test: Run wrk or locust against the endpoint; measure latency (p99) and throughput (RPS) improvement under load.