Migrated from Notion TEST coverage recommendations (2026-09-24) — https://app.notion.com/p/3e51f8eaa3ad8165b2cad8f82cc08805
Callout Shareable audit for teammates. Read-only review of
reservation-api-serverdevelop +reservation-browser-testsmain + GitHub Project 10. Class-bound rule: new outlet of an existing class = zero new tests (do not invent SKUs). Audit time: 2026-09-24 ~23:00 IST (Asia/Calcutta) — re-verified; tips unchanged since Sep 23 audit API tip:94f64f5(merge PR #1019, 2026-09-23 ~19:14 IST) Browser tip:4a53eab(2026-09-08 IST) — still no newer commits on main Related: Test approach · Scenario matrix · Project 10 board
Top five (start here)¶
- P0 — Fix Playwright nightly. BA Midway failed ~6 days straight (outlet “Select location”). Epic A/C Done evidence is stale until
suite=adminis green. - P0 — Re-verify Buyout + gate S-IDOR in CI. Run
suite=admin-buyoutfor#140. Addtests/security/toci-quality(today onlytests/unit/). - P1 — Fail-closed CI (
#153/#154).ci-integrationstill soft-fails with|| true; coverage still commented out / no--cov. - P1 — Desk gap evidence (
#141/#143/#144). Pay Later and slot-full specs exist but aren’t on the nightly schedule; SA multi-location still missing. - P1 — Epic F holes (
#145/#146). ADR-0008 confirmed-edit (patch vs lock-replace) still untested; golden create smokes (Midway / Super Pass / ULTIMATE) still missing.
A. What changed on develop since ~2026-09-18¶
| PR | Merged (UTC) | Test / CI impact |
|---|---|---|
| #974 | 2026-09-18 | Karate API + k6 1VU packs + sys005-karate-k6-manual.yml (Epic H / #166–#169) |
| #790 / #776 | 2026-09-18 | Security baseline docs/scripts — not SYS-005 PR gates |
| #1014 | 2026-09-22 | tests/unit/test_venue_resource_dedup.py (capacity double-count) |
| #1016 / #1019 | 2026-09-23 | Overnight lock + calendar category; limited SYS-005 matrix expansion |
| Still true from prior audit | ||
- ci-integration.yml ends bench with || true → #153 open |
||
- Coverage block commented out; ci-quality runs pytest tests/unit/ without --cov → #154 open |
||
| - Golden create smokes missing → #146 Backlog | ||
| - ADR-0008 unit gap open → #145 Backlog | ||
| - Epic H packs on develop; board cards #166–#169 still Backlog | ||
| --- | ||
| # B. Current coverage map (strengths) | ||
## Unit + oracle (reservation-api-server) |
||
- SYS-005 oracle tests/sys005_oracle/ — offline fixtures; sys005-oracle.yml enforces ≥30 tests; Midway / Super Pass / ULTIMATE goldens |
||
| - S-NOTIMESLOT — offline + mocked API (#162 Done) | ||
- S-IDOR tests/security/test_idor_get_booking.py (#163 Done) — not in ci-quality path yet |
||
- Unit suite tests/unit/ — pricing, locks, cart, venue dedup (#1014), etc. |
||
| - CI Quality — ruff + unit fail-closed on PR (path filters) | ||
- Bench bookingzone/tests/ — soft-fail under ci-integration |
||
| ## Karate | ||
- tests/karate-api-tests/ — 19 features via #974 |
||
- Concurrency/exhaust tagged @manual-only (correctly not PR-gated) |
||
| - Manual workflow: docs-only / k6-inspect / karate-dryrun only | ||
| ## k6 | ||
- tests/k6-load-tests/booking-one.js — 1 VU × 1 iteration; destructive flags documented |
||
## Playwright (reservation-browser-tests) |
||
- Tip still 4a53eab (Sep 8) |
||
| - Specs: admin Midway lifecycle, buyout (#140), pay-later (#143), slot-full (#141), venuplus Datacap (#142 gated) | ||
- Schedule: daily suite=admin only |
||
| ## What actually blocks today | ||
| - Hard: ruff/syntax/unit, oracle | ||
- Soft: bench || true, post-deploy smoke, Karate/k6 dry-run |
||
| - Playwright is nightly on browser-tests, not an api-server PR gate | ||
| --- | ||
| # C. Gaps vs Project 10 + classic matrix | ||
| ## Board (live 2026-09-24 — same open set as Sep 23) | ||
| Done: #129–#139, Epic A #155, Epic B #156, #162, #163 | ||
| Open | ||
| - Epic C #157 — #140 Buyout Todo; nightly BA Midway red (outlet select; tip still Sep 8) | ||
| - Epic D #158 — #141 / #143 / #144 evidence missing or planned | ||
- Epic E #159 — #153 / #154 unchanged (ci-quality still tests/unit/ only; no tests/security/) |
||
| - Epic F #160 — #145–#152 Backlog | ||
| - Epic G #161 — #142 Datacap manual-only | ||
| - Epic H #166–#169 — packs landed (#974); need operator evidence, not new scenarios | ||
| ## Classic matrix | ||
| ID | Intent | Status |
| --- | --- | --- |
| S-NOTIMESLOT | 200 + [] | Covered (#162) |
| S-IDOR | cross-guest get_booking | Covered (#163); orphaned from ci-quality |
| S-GOLDEN-CREATE | live create vs oracle | Missing (#146) |
| S-ADR-0008 | Confirmed edit = patch | Missing (#145) |
| N-CAPACITY-RACE / N-EXHAUST-DAY | concurrency / exhaust | Packs present, manual-only |
| P-K6-1VU | latency twin | Pack present; needs approved DEV run (#169) |
| --- | ||
| # D. Prioritized recommendations | ||
| ## P0 — stop the bleeding | ||
| # | Outcome | Repo |
| --- | --- | --- |
| P0-1 | Fix nightly BA Midway outlet-select; restore green suite=admin |
browser-tests |
| P0-2 | Re-verify Buyout (suite=admin-buyout); close #140 when green |
browser-tests |
| P0-3 | Include tests/security/ in ci-quality unit job |
api-server |
| ## P1 — gates + desk gaps on the board | ||
| # | Outcome | Repo |
| --- | --- | --- |
| P1-1 | Remove \|\| true from ci-integration after canary green; require bench |
api-server |
| P1-2 | Enable --cov + modest fail_under; branch-protect unit+oracle+lint |
api-server |
| P1-3 | Nightly/weekly matrix: buyout, pay-later, slot-full | browser-tests |
| P1-4 | SA multi-location (#144) — class-bound second outlet | browser-tests |
| P1-5 | ADR-0008 unit/API: Confirmed edit = safe patch | api-server |
| P1-6 | Golden create smokes Midway / Super Pass / ULTIMATE | api-server |
| P1-7 | Epic H board hygiene: evidence + move #166–#169 (packs already landed) | api-server |
| ## P2 — deepen without SKU explosion | ||
- #149 — real mocked deposit_source() calls (not source greps) |
||
| - #147 — US golden parity | ||
| - #148 — Magensa never selected for ATP/US goldens | ||
| - #150–#152 — cart expiry, buffer in availability, addon reprice | ||
| - #142 — PWA Datacap only with explicit approval | ||
| - Optional staff-session override unit for #1016 | ||
| --- | ||
| # E. What NOT to do | ||
| 1. Do not invent SKUs / outlets / packages — class-bound (#138 Done). | ||
2. Do not PR-gate @manual-only Karate or raise k6 above 1 VU until slot distribution exists. |
||
3. Do not enable live Datacap / destructive flags on pull_request. |
||
| 4. Do not treat Epic H as needing new scenarios — finish evidence + board Status. | ||
| 5. Do not PR-gate Guest PWA Datacap (#142) without approval. | ||
| 6. Do not mark Epic C/D Done while nightly BA Midway is red. | ||
7. Do not wire aspirational quality_gates.py theater before #153/#154. |
||
| --- | ||
| # Appendix | ||
- API develop tip: 94f64f5 (PR #1019) — re-checked 2026-09-24 IST, still HEAD |
||
- Browser main tip: 4a53eab (2026-09-08 IST) — re-checked, still HEAD |
||
- sys005-karate-k6-manual.yml present on develop; ci-quality.yml unit job = pytest tests/unit/ only |
||
| - Project 10 | ||
| - Karate+k6 land PR #974 | ||
- Playwright fail sample: Actions run 35833999404 (2026-09-23) — outlet select for Arnot Mall |