You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Recovery cancellation bypasses Store and crashes after privacy fencing on Postgres #23761
Shared old/new recovery conformance reproduces a cancellation gap on current main (4f52db8b3b), with synthetic owners only. No production observations or private data are involved.
After a held unit has a prepared recovery intent, add an account-deletion marker either before recovery or during audio download. RecoveryStore.prepare/start correctly raises BatchPrivacyBlocked. The coordinator catches it and calls cancel_fenced_recovery, which unconditionally uses store.client.collection(...). SimStore.client is None and PostgresStore has no client attribute, so the caller raises AttributeError (NoneType.collection on sim, missing PostgresStore.client on Postgres) and the durable intent is not marked cancelled. Firestore completes both cases.
Reproduction uses backend/tests/integration/test_recovery_admission_conformance.py::test_pl072_account_deletion_cancels_recovery_before_paid_work_or_upload in #23763. Before marking expected failures, the complete strict matrix ran 23 passed / 4 failed / zero skips, on real loopback Firestore and real local Postgres plus the simulator. The four failures are both deletion phases on sim/Postgres; all Firestore cases pass. Production modules are unchanged from the main baseline.
Required invariant (PL-072): deletion cancels a prepared/running recovery before another paid call or result upload; duplicate deliveries remain cancelled and do no provider/artifact IO; unrelated owners keep progressing. Preserve owner, privacy-generation and exact-intent fences. Physical privacy erasure is a separate gate.
Recommendation: A supplies a bounded, revocation-only cancellation operation on the shared Store surface; C uses it instead of an unconditional Firestore-client write. Do not attach a fake Firestore client to Postgres or permit ordinary fenced writes. The conformance PR records strict expected failures limited to these two backends and the reproduced AttributeError; they do not count as completed parity.
Action class: local code/test and issue publication. No merge, deploy, production data access, paid retry or configuration change.
Extended the shared Store protocol (backend/database/batch_pipeline_store.py) with cancel_fenced_recovery(self, segment_id: str) -> bool.
Implemented FirestoreStore.cancel_fenced_recovery delegating to cancel_fenced_recovery_firestore.
Implemented PostgresStore.cancel_fenced_recovery executing a serializable revocation-only write via write_doc directly upon catching BatchPrivacyBlocked, without triggering ordinary flush() privacy rechecks.
Implemented SimStore.cancel_fenced_recovery updating {**job, 'recovery_state': 'cancelled'} when customer.blocked(...) is true.
Updated cancel_fenced_recovery(store, segment_id) in backend/database/batch_pipeline_quarantine_recovery.py and backend/testing/pipeline_sim/sim.py to delegate to store.cancel_fenced_recovery(segment_id) on the shared Store surface.
Added comprehensive unit test suite in backend/tests/unit/test_recovery_cancellation_store_parity.py testing SimStore, PostgresStore, and FirestoreStore recovery cancellation before and after account deletion, plus coordinator cancellation handling and protocol conformance.
Activity
test(backend): port recovery and admission parity invariants
fix(backend): provide bounded cancel_fenced_recovery on Store surface (…
fix(backend): invoke mirror_refresh in FirestoreStore.cancel_fenced_r…
fix(jobstore): add revocation-only cancel_fenced_recovery to shared S…
fix(jobstore): add revocation-only cancel_fenced_recovery to shared S…
fix(jobstore): add revocation-only cancel_fenced_recovery to shared S…
fix(jobstore): add revocation-only cancel_fenced_recovery to shared S…
Dimank92 commentedon Oct 28, 2026
I have resolved this issue in PR #23893.
Summary of Fix
Storeprotocol (backend/database/batch_pipeline_store.py) withcancel_fenced_recovery(self, segment_id: str) -> bool.FirestoreStore.cancel_fenced_recoverydelegating tocancel_fenced_recovery_firestore.PostgresStore.cancel_fenced_recoveryexecuting a serializable revocation-only write viawrite_docdirectly upon catchingBatchPrivacyBlocked, without triggering ordinaryflush()privacy rechecks.SimStore.cancel_fenced_recoveryupdating{**job, 'recovery_state': 'cancelled'}whencustomer.blocked(...)is true.cancel_fenced_recovery(store, segment_id)inbackend/database/batch_pipeline_quarantine_recovery.pyandbackend/testing/pipeline_sim/sim.pyto delegate tostore.cancel_fenced_recovery(segment_id)on the shared Store surface.backend/tests/unit/test_recovery_cancellation_store_parity.pytestingSimStore,PostgresStore, andFirestoreStorerecovery cancellation before and after account deletion, plus coordinator cancellation handling and protocol conformance.Unified payout wallet:
0xC345C07229307d4f1c7935aD2cAe48e390215808test(backend): port recovery and admission parity invariants
fix(backend): provide bounded cancel_fenced_recovery on Store surface (…
fix(backend): invoke mirror_refresh in FirestoreStore.cancel_fenced_r…