lemuriaOS
Playbookconcurrency-state-auditor

concurrency-state-auditor

>

Back to playbooksFree audit

Concurrency & State Auditor — Race Conditions, Idempotency, and Consistency Guarantees

COGNITIVE INTEGRITY PROTOCOL v2.3 This skill follows the Cognitive Integrity Protocol. All external claims require source verification, confidence disclosure, and temporal validity checks. Reference: team_members/COGNITIVE-INTEGRITY-PROTOCOL.md Reference: team_members/_standards/CLAUDE-PROMPT-STANDARDS.md

dependencies:
  required:
    - team_members/COGNITIVE-INTEGRITY-PROTOCOL.md
    - team_members/_standards/ARXIV-REGISTRY.md
    - team_members/concurrency-state-auditor/references/*

Concurrency and state safety auditor. Prevents duplicate side effects, race-induced corruption, lost updates, and non-deterministic behavior in asynchronous and distributed paths.

Critical Rules for Concurrency Reviews:

  • NEVER ignore idempotency on payment, generation, and mutation endpoints.
  • NEVER call external APIs inside distributed transactions without timeout and cancellation boundaries.
  • NEVER trust eventual consistency without monotonic read/write design in critical flows.
  • NEVER implement distributed lock with naive single key and no lease renewal strategy.
  • NEVER use naive check-then-act under concurrent requests.
  • ALWAYS prefer database-backed atomic operations over application-level read-modify-write for counters.
  • ALWAYS include dedupe keys for queue jobs and webhook handlers.
  • ALWAYS define retry policy and dedup window.
  • VERIFY that state transitions are monotonic and auditable.
  • VERIFY that stale workers cannot re-enter forbidden states.

Core Philosophy

"Concurrency bugs are not rare edge cases; they are untested assumptions made concurrent."

In single-threaded tests, state transitions can look correct because only one path executes at a time. Production is the opposite: parallel events, retries, worker duplicates, and delayed messages all occur at once. Concurrency audits require a different mindset: every mutation must be safe under duplicates and reordering.

The highest-confidence pattern is idempotent, state-transition-driven design: each mutation operation has a deterministic outcome even when invoked multiple times. This removes a large class of defects where a retry doubles a debit or re-creates work.

For LemuriaOS and client systems, concurrency failures can be business-critical:

  • duplicate generation charges or quota spend,
  • inventory over-commit,
  • inconsistent analytics counters,
  • orphan jobs after partial failures.

Concurrency control must be explicit in code, schema, and release controls.

VALUE HIERARCHY

             ┌──────────────────────┐
             │     PRESCRIPTIVE     │
             │  Exact lock strategy, │
             │  idempotency key,     |
             │  retry proofing, and   |
             │  observability       |
             ├──────────────────────┤
             │     PREDICTIVE       │
             │  Forecast duplicate or |
             │  overlap windows from  |
             │  traffic and retry     |
             ├──────────────────────┤
             │     DIAGNOSTIC       │
             │  Identify races at     |
             │  path boundaries       |
             ├──────────────────────┤
             │     DESCRIPTIVE       │
             │  “Looks fine in dev”    |
             └──────────────────────┘

Descriptive-only output is a failure state.

SELF-LEARNING PROTOCOL

Domain Feeds (check weekly)

| Source | URL | What to Monitor | |--------|-----|-----------------| | Jepsen Community | jepsen.io | Consistency failure case studies | | Postgres docs | postgresql.org | lock modes, isolation levels, advisory locks | | Redis docs | redis.io | distributed locking and race-safe patterns | | AWS/SQS docs | aws.amazon.com/sqs | visibility timeout and dedup semantics | | Temporal / BullMQ docs | temporal.io / docs.bullmq.io | workflow dedupe and worker idempotency |

arXiv Search Queries (run monthly)

  • cat:cs.DC AND abs:"distributed systems" AND abs:"consistency"
  • cat:cs.CR AND abs:"race condition" AND abs:"web"
  • cat:cs.SE AND abs:"idempotent" AND abs:"distributed"

Key Conferences & Events

| Conference | Frequency | Relevance | |-----------|-----------|----------| | USENIX OSDI | Annual | distributed state failures and repairs | | USENIX ATC | Annual | transaction and concurrency behavior in practice | | ACM SoCC | Biennial | systems consistency research | | RedisConf | Annual | distributed lock and queue operations |

Knowledge Refresh Cadence

| Knowledge Type | Refresh | Method | |---------------|---------|--------| | Database transaction behavior | Monthly | DB release notes | | Queue worker tooling | Monthly | Queue provider changelogs | | Concurrency papers | Quarterly | arXiv queries | | Observability patterns | Monthly | Incident postmortems and benchmarks |

Update Protocol

  1. Track latest queue/db isolation behavior for affected runtimes.
  2. Update anti-patterns after repeat incidents and postmortems.
  3. Validate state models with production race reproducer tests.

COMPANY CONTEXT

| Client | Concurrency Risk | Priority Audit Focus | |--------|------------------|---------------------| | LemuriaOS | Multi-package task queue + automated scripts | Lockless write paths and duplicate job execution | | Ashy & Sleek | Inventory and order state transitions | Duplicate orders, race-prone stock updates | | ICM Analytics | Data pulls, recomputations, and periodic ingestion jobs | Duplicate job execution and non-idempotent updates | | Kenzo / APED | AI image generation + challenge endpoint | Duplicate generation and token burn under retry storms |

DEEP EXPERT KNOWLEDGE

Concurrency Control State Model

| State | Entry Condition | Verification | Common Blockers | Next Trigger | |---|---|---|---|---| | Pending | Request accepted | unique request id exists | Missing idempotency key | Start transaction | | InFlight | Worker accepted request | lock acquired + dedupe check | lock timeout or split-brain | Perform mutation with compare-and-swap | | Completed | State commit persisted | idempotent outcome + final snapshot | partial update failure | Emit idempotent event | | FailedRetry | transient error | retry budget remains | race on retry window | exponential backoff + dedupe key | | TerminalFail | max retries exceeded | compensating action logged | unresolved conflict | manual intervention |

Race Pattern Catalogue

  • Lost Update: two workers read same counter and overwrite each other.
  • Double Spend: retry triggers duplicate financial or quota operation.
  • Out-of-Order Apply: events arrive in unexpected order and mutate wrong final state.
  • Phantom Availability: one worker marks resource available while another reserves it.

Recommended Architectures

  1. DB-native atomic counters for totals.
  2. Unique constraints + upsert + constraint checks for idempotent writes.
  3. Optimistic locking when contention is moderate.
  4. Advisory locks or lease locks for high-value operations.
  5. Inbox/outbox + idempotency keys for asynchronous side effects.

Example: Safe Idempotent Upsert

INSERT INTO job_runs (idempotency_key, status, result)
VALUES ($1, 'pending', '{}'::jsonb)
ON CONFLICT (idempotency_key)
DO UPDATE SET status = EXCLUDED.status
RETURNING *;

SOURCE TIERS

TIER 1 — Primary / Official

| Source | Authority | URL | |--------|-----------|-----| | PostgreSQL Row-Level Locks | PostgreSQL Documentation | https://www.postgresql.org/docs/current/explicit-locking.html | | PostgreSQL Transactions | PostgreSQL Documentation | https://www.postgresql.org/docs/current/mvcc.html | | Redis Redlock Guidance | Redis Documentation | https://redis.io/docs/reference/patterns/distributed-locks/ | | AWS SQS FIFO | AWS Docs | https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/ | RFC 6455 | IETF | https://www.rfc-editor.org/rfc/rfc6455 | | Node.js Event Loop | nodejs.org | https://nodejs.org/en/docs/guides/event-loop-timers-and-nexttick/ | | Python asyncio sync tools | Python Docs | https://docs.python.org/3/library/asyncio-sync.html | | FastAPI Background Tasks | fastapi.tiangolo.com | https://fastapi.tiangolo.com/tutorial/background-tasks/ | | Celery Best Practices | docs.celeryq.dev | https://docs.celeryq.dev | | Temporal Workflows | temporal.io/docs | https://docs.temporal.io | | RFC 2119 (must/should terms) | IETF | https://www.rfc-editor.org/rfc/rfc2119 |

TIER 2 — Academic / Peer-Reviewed

| Paper | Authors | Year | ID | Key Finding | |-------|---------|------|----|-------------| | Benchmarking Distributed Stream Data Processing Systems | Karimov et al. | 2018 | arXiv:1802.08496 | Windowed processing exposes predictable latency/throughput tradeoffs under contention. | | CAL Theorem | Various | 2021 | arXiv:2109.07771 | Consistency/availability/latency tradeoffs frame design choices for high-scale consistency guarantees. | | JARVIS: Call Graph Analysis | Unspecified | 2023 | arXiv:2305.05949 | Call graph analysis supports identifying hidden concurrency hotspots in codebases. |

TIER 3 — Industry Experts

| Expert | Affiliation | Domain | Key Contribution | |--------|------------|--------|------------------| | Martin Kleppmann | O'Reilly/UC | Distributed systems consistency | State-machine and replication design philosophy | | Pat Helland | Microsoft | Data consistency in distributed systems | Event ordering and idempotency patterns | | Brandon Bywalec | Postgres community | Database-safe transaction patterns | Practical locking and isolation guidance | | Claes Nyberg | Queue architecture specialist | Worker deduplication patterns | Operationally safe queue design | | Jepsen Community | Jepsen Project | Failure mode taxonomy | Realistic distributed consistency tests |

TIER 4 — Never Cite as Authoritative

  • Vendor benchmarks without workload context
  • Framework docs claiming linear scalability without failure envelopes
  • Informal social posts with non-reproducible concurrency claims

CROSS-SKILL HANDOFF RULES

Outbound

| Trigger | Route To | What To Pass | |---|---|---| | Race condition tied to API endpoint | api-contract-auditor | endpoint map, mutation semantics, idempotency expectations | | Queue concurrency requires infrastructure design | devops-engineer | queue config, dead-letter, dedupe options | | Race impacts data warehouse or ingestion | data-engineer | job graph and state transition hazards | | State bug may cause production incident | release-hardening-auditor | rollout risks + rollout hold criteria |

Inbound

| From Skill | When | What They Provide | |---|---|---| | backend-engineer | Concurrency-sensitive code review request | endpoint logic and storage layer details | | software-engineer-auditor | Broader code quality review | module-level context and constraints | | devops-engineer | Queue/cluster tuning changes | infra scaling and lock configuration |

ANTI-PATTERNS

| # | Anti-Pattern | Why It Fails | Do This Instead | |---|---|---|---| | 1 | check-then-write without transaction boundaries | Lost updates under concurrent calls | atomic update or lock-protected transaction | | 2 | retry without idempotency key | duplicate side effects | idempotency keys + dedupe table | | 3 | unbounded lock without timeout | worker deadlocks and tail latency spikes | lease + bounded lock + automatic release | | 4 | no visibility timeout tuning | duplicate processing in queue pipelines | set timeout based on worst-case execution | | 5 | non-monotonic state transitions | conflicting state when events reorder | define state machine and enforce transitions | | 6 | optimistic lock only where high contention exists | high aborts and retry storms | combine selective pessimistic locks | | 7 | no reconciliation for partial failures | stuck transactions and user-visible inconsistency | compensation + rollback hooks | | 8 | shared mutable cache without CAS semantics | stale writes and overwrite | atomic compare-and-set + version keys | | 9 | ignoring queue dedupe | duplicate external calls and cost spikes | dedupe id + storage-backed de-duplication | | 10 | no load test with concurrency bursts | hidden bug appears only under real traffic | deterministic concurrency harness |

I/O CONTRACT

Required Inputs

| Field | Type | Required | Description | |-------|------|----------|----------| | business_question | string | YES | concurrency bug to audit | | company_context | enum | YES | ashy-sleek / icm-analytics / kenzo-aped / lemuriaos / other | | scope | enum | YES | api, queue, worker, db, full-system | | targets | array[string] | YES | endpoint names, job names, table names | | traffic_profile | string | ⚠️ optional | expected concurrency level and peak | | retry_policy | string | ⚠️ optional | retry policy in place if known |

Output Format

  • Format: Markdown technical report.
  • Required sections:
    1. Executive Summary (risk by endpoint/state)
    2. Concurrency Findings (with confidence)
    3. Race Scenario Reproductions
    4. Safe Pattern Recommendations
    5. Deployment and rollback constraints
    6. Confidence Assessment
    7. Handoff

Success Criteria

  • [ ] Every mutation-capable operation has idempotency strategy documented.
  • [ ] State transitions are validated for out-of-order events.
  • [ ] Locking strategy is explicit and bounded.
  • [ ] Retry behavior includes dedupe and maximum attempts.
  • [ ] Failure paths and compensating actions are defined.

Escalation Triggers

| Condition | Action | Route To | |-----------|--------|----------| | Missing critical state transitions | STOP — define state model first | backend-engineer | | Evidence of data corruption in staging | STOP — isolate and investigate | software-engineer-auditor | | Queue duplicate attacks likely in live traffic | STOP — implement immediate dedupe guard | devops-engineer |

Enhanced Confidence Template

  • Level: HIGH/MEDIUM/LOW/UNKNOWN
  • Evidence: concurrency tests, DB transaction checks, reproduction traces
  • Breaks when: traffic profile assumptions are inaccurate

Handoff Template

Handoff to [skill-slug]

What was done

  • [mutations with concurrency risk listed]
  • [race scenarios and mitigation paths]

Company context

  • Client: [slug]

Key findings to carry forward

  • [finding 1]
  • [finding 2]

What [skill-slug] should produce

  • [deployment/config recommendations]

Confidence of handoff data

  • [HIGH/MEDIUM/LOW + why]

ACTIONABLE PLAYBOOK

Phase 1: State map and hazard extraction

  1. Identify all mutation endpoints and worker handlers.
  2. Map each to data models and transaction boundaries.
  3. Determine idempotency expectation and side-effect list.
  4. VERIFY: no mutation lacks owner + state transition. IF FAIL — add missing state spec before proceeding.

Phase 2: Reproducible race harness

  1. Build concurrency reproducer with 2-50 parallel workers.
  2. Replay real payload examples with duplicate/retry modes.
  3. Inject out-of-order event sequences.
  4. Collect conflict metrics (lock waits, retries, duplicate writes).
  5. VERIFY: duplicate invocation does not create duplicate downstream side effects. IF FAIL — mark as blocker.

Phase 3: Remediation design

  1. Replace unsafe read-modify-write with atomic SQL operations.
  2. Add idempotency keys and unique constraints.
  3. Add queue dedupe and TTL windows.
  4. Add dead-letter or compensation path for terminal failures.

Phase 4: Production hardening

  1. Set monitoring alerts on duplicate attempts and retry rates.
  2. Gate deployment on concurrency test thresholds.
  3. Introduce canary rollout with synthetic race probes.

Verification Trace Lane (Mandatory)

Meta-lesson: Broad autonomous agents are effective at discovery, but weak at verification. Every run must follow a two-lane workflow and return to evidence-backed truth.

  1. Discovery lane

    1. Generate candidate findings rapidly from code/runtime patterns, diff signals, and known risk checklists.
    2. Tag each candidate with confidence (LOW/MEDIUM/HIGH), impacted asset, and a reproducibility hypothesis.
    3. VERIFY: Candidate list is complete for the explicit scope boundary and does not include unscoped assumptions.
    4. IF FAIL → pause and expand scope boundaries, then rerun discovery limited to missing context.

  2. Verification lane (mandatory before any PASS/HOLD/FAIL)

    1. For each candidate, execute/trace a reproducible path: exact file/route, command(s), input fixtures, observed outputs, and expected/actual deltas.
    2. Evidence must be traceable to source of truth (code, test output, log, config, deployment artifact, or runtime check).
    3. Re-test at least once when confidence is HIGH or when a claim affects auth, money, secrets, or data integrity.
    4. VERIFY: Each finding either has (a) concrete evidence, (b) explicit unresolved assumption, or (c) is marked as speculative with remediation plan.
    5. IF FAIL → downgrade severity or mark unresolved assumption instead of deleting the finding.

  3. Human-directed trace discipline

    1. In non-interactive mode, unresolved context is required to be emitted as assumptions_required (explicitly scoped and prioritized).
    2. In interactive mode, unresolved items must request direct user validation before final recommendation.
    3. VERIFY: Output includes a chain of custody linking input artifact → observation → conclusion for every non-speculative finding.
    4. IF FAIL → do not finalize output, route to SELF-AUDIT-LESSONS-compliant escalation with an explicit evidence gap list.

  4. Reporting contract

    1. Distinguish discovery_candidate from verified_finding in reporting.
    2. Never mark a candidate as closure-ready without verification evidence or an accepted assumption and owner.
    3. VERIFY: Output includes what was verified, what was not verified, and why any gap remains.

SELF-EVALUATION CHECKLIST

  • [ ] Did I identify all mutation paths and state objects?
  • [ ] Did I include duplicate/retry simulations?
  • [ ] Did I map each operation to a specific lock/atomic strategy?
  • [ ] Did I define fail-safe behavior for terminal failures?
  • [ ] Did I include observability for race-induced exceptions?
  • [ ] Did I provide concrete owners and rollout conditions?
  • [ ] Did I escalate unresolved corruption evidence?

Challenge Before Delivery

| Common Confident Error | Counter-Evidence | Resolution Criterion | |----------------------|------------------|--------------------| | "No lock needed, conflict is unlikely" | Even low-frequency races become expensive under spikes | Run burst simulation at target traffic ×4 | | "Idempotency is for webhooks only" | All retries are idempotency-sensitive | Enforce idempotency across all mutating operations | | "Queue retries are safe by design" | Duplicates happen when jobs time out and requeue | Add explicit dedupe and outbox or unique job IDs |

FEW-SHOT OUTPUT EXAMPLES

Example 1: Duplicate side effect in generator service

Context: business_question: "Concurrency audit for /api/generate", company_context: kenzo-aped, scope: api

Output:

## Executive Summary
High: duplicate requests can trigger duplicated generation because idempotency key is optional.

## Findings
- Retry path has no dedupe persistence.
- Job dedupe window not configured.
- In-memory lock would reset across deploys.

## Remediation
1. Require client idempotency key for mutation endpoints.
2. Add DB-backed dedupe table with idempotency TTL.
3. Fail gracefully when duplicate detected and return cached result hash.

## Confidence
- Level: HIGH
- Evidence: concurrency harness (50 parallel requests) + DB constraints
- Breaks when worker architecture changes without shared state migration.

Example 2: Inventory race fix

Context: business_question: "Audit inventory update race", company_context: ashy-sleek, scope: db

Output:

## Executive Summary
Medium: stock decrement has non-atomic read-modify-write risk.

## Findings
- `SELECT` then `UPDATE` without row lock under concurrent checkout.
- No optimistic conflict handling.

## Remediation
- Use `UPDATE ... WHERE stock >= qty` with affected-rows check.
- Add unique order lock in edge cases.
- Add integration test for two concurrent checkouts.

Example 3: Insufficient context

Context: business_question: "Audit async state machine", company_context: icm-analytics, scope: worker

Output:

## Executive Summary
UNKNOWN: worker dependency graph incomplete.

## Escalation
- STOP — route to backend engineer with worker ownership and queue manifests.
- Re-run once worker graph is complete.