Case study: Race condition & architecture refactor

Players on trials.gg intermittently lost challenge progress. I traced it to a race condition in concurrent progress updates, then led a staged refactor of the affected TypeScript services — fixing the concurrency model and leaving behind an automated regression suite.

Context

trials.gg rewards competitive gamers for completing challenges. Progress is computed from a stream of gameplay events; wrong progress means wrong rewards — a direct trust and payout problem. The team was small and remote, shipping ~5 times a week. I owned the backend: diagnosis, the refactor plan, implementation with the team, test strategy, deployment, and production monitoring afterward.

Technical problem

Two workers could process events for the same player concurrently. Both read the same progress row, computed independently, and the slower write silently overwrote the faster one — a classic lost-update. It only surfaced under bursts, so reports looked random and unreproducible.

The constraints: no downtime for live challenges; no big-bang rewrite the lean team couldn’t absorb; event throughput and costs had to stay flat.

Investigation

Correlated affected players against event timestamps in the analytics pipeline; the anomalies clustered where events for one player landed < 200 ms apart. A load test replaying bursty event traffic against staging reproduced the loss reliably — turning a ghost into a failing test.

Data flow

game clientAPI
fastifySQS FIFO
group = playerworker
idempotentpostgres
row lock

before — parallel workers, shared rows, last write wins · after — per-player ordering, transactional apply, dedup on event id

Options considered

Optimistic locking

Version column + retry. Least change, but burst retries amplify load and reordering stays possible.

Serialize in Postgres

SELECT … FOR UPDATE everywhere. Simple, but pushes contention into the DB and hides ordering bugs.

Per-player ordering ✓

FIFO queue grouped by player + idempotent, transactional handlers. Removes the race by construction.

Chose per-player ordering with idempotent handlers, accepting slightly lower per-player throughput (bounded by FIFO group) in exchange for correctness we could prove. Row locks remained as a second line of defense — belt and suspenders where money is involved.

Implementation

Staged over several releases: extract progress logic into a pure, testable TypeScript module; introduce the queue behind a feature flag; dual-run old and new paths and diff outcomes on live traffic; then cut over player cohorts incrementally. No downtime, reversible at every step.

Testing strategy: unit tests over the pure progress module; property-style tests asserting order-independence of final state; the burst-replay load test promoted into CI as a regression gate; integration tests per handler in Jest.

Shipped through the existing CodePipeline flow (~5 deploys/week), with new metrics for dropped/duplicate events, per-group queue lag, and progress-diff alarms wired into Grafana dashboards from the analytics pipeline.

Result

Lost-progress reports stopped after cutover; the platform held 99.9% availability through the migration. The regression suite has since caught reintroductions before production.

Lessons learned

Reproduce before you fix — the failing load test did more than any amount of reading code. Concurrency bugs are architecture bugs; patching call sites just relocates them. Today I’d start from an event-sourced progress model with a transactional outbox, so ordering and replay are first-class instead of retrofitted.