Skip to content

Recovery cancellation bypasses Store and crashes after privacy fencing on Postgres #23761

Open

Description

@Git-on-my-level

Shared old/new recovery conformance reproduces a cancellation gap on current main (4f52db8b3b), with synthetic owners only. No production observations or private data are involved.

After a held unit has a prepared recovery intent, add an account-deletion marker either before recovery or during audio download. RecoveryStore.prepare/start correctly raises BatchPrivacyBlocked. The coordinator catches it and calls cancel_fenced_recovery, which unconditionally uses store.client.collection(...). SimStore.client is None and PostgresStore has no client attribute, so the caller raises AttributeError (NoneType.collection on sim, missing PostgresStore.client on Postgres) and the durable intent is not marked cancelled. Firestore completes both cases.

Reproduction uses backend/tests/integration/test_recovery_admission_conformance.py::test_pl072_account_deletion_cancels_recovery_before_paid_work_or_upload in #23763. Before marking expected failures, the complete strict matrix ran 23 passed / 4 failed / zero skips, on real loopback Firestore and real local Postgres plus the simulator. The four failures are both deletion phases on sim/Postgres; all Firestore cases pass. Production modules are unchanged from the main baseline.

Required invariant (PL-072): deletion cancels a prepared/running recovery before another paid call or result upload; duplicate deliveries remain cancelled and do no provider/artifact IO; unrelated owners keep progressing. Preserve owner, privacy-generation and exact-intent fences. Physical privacy erasure is a separate gate.

Recommendation: A supplies a bounded, revocation-only cancellation operation on the shared Store surface; C uses it instead of an unconditional Firestore-client write. Do not attach a fake Firestore client to Postgres or permit ordinary fenced writes. The conformance PR records strict expected failures limited to these two backends and the reproduced AttributeError; they do not count as completed parity.

Action class: local code/test and issue publication. No merge, deploy, production data access, paid retry or configuration change.

Activity

  1. moved this from Someday to In progress in omi TODO / bountieson Oct 27, 2026
  2. Dimank92 commented on Oct 28, 2026

    @Dimank92

    I have resolved this issue in PR #23893.

    Summary of Fix

    • Extended the shared Store protocol (backend/database/batch_pipeline_store.py) with cancel_fenced_recovery(self, segment_id: str) -> bool.
    • Implemented FirestoreStore.cancel_fenced_recovery delegating to cancel_fenced_recovery_firestore.
    • Implemented PostgresStore.cancel_fenced_recovery executing a serializable revocation-only write via write_doc directly upon catching BatchPrivacyBlocked, without triggering ordinary flush() privacy rechecks.
    • Implemented SimStore.cancel_fenced_recovery updating {**job, 'recovery_state': 'cancelled'} when customer.blocked(...) is true.
    • Updated cancel_fenced_recovery(store, segment_id) in backend/database/batch_pipeline_quarantine_recovery.py and backend/testing/pipeline_sim/sim.py to delegate to store.cancel_fenced_recovery(segment_id) on the shared Store surface.
    • Added comprehensive unit test suite in backend/tests/unit/test_recovery_cancellation_store_parity.py testing SimStore, PostgresStore, and FirestoreStore recovery cancellation before and after account deletion, plus coordinator cancellation handling and protocol conformance.

    Unified payout wallet: 0xC345C07229307d4f1c7935aD2cAe48e390215808

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions