Skip to content

[docs] Example integration with external persistance layer #48

Open

Description

@naz

Problem

There's a need to track, discuss, and document best approaches around adding jobs persistence layer on top of bree. This issue is a focused continuation of this comment:

An example in the README for using sockets, or redis pubsub to communicate with Bree to add new jobs (?)

and another comment :

I would recommend keeping dynamic jobs actually stored in a queue, with a persistent database, and then have a job that runs every so often to flush the queue (with limited concurrency).

References

How job persistence is approached in other job processing/queuing libraries:

Current use of bree along with MongoDB:

  • forwardemail.net - here jobs are generated based on state of records in the database
  • TBD: expand this list with more examples for future reference

Expected outcome

What I think would be the ideal outcome for this issue is documentation an example implementation of persisted jobs queue on top of bree. Ideally the example would be backend agnostic so that multiple storage approaches - NoSQL, SQL or key/value stores like Redis, could utilize it.

Maybe bree could even provide an adapter/plugin system to optionally add persistence for (1) scheduled one-off jobs and (2) failed jobs 🤔

Activity

  1. added
    documentationImprovements or additions to documentation
    on Dec 10, 2020
  2. Zireael commented on Jan 23, 2021

    @Zireael

    Expected outcome

    What I think would be the ideal outcome for this issue is documentation an example implementation of persisted jobs queue on top of bree. Ideally the example would be backend agnostic so that multiple storage approaches - NoSQL, SQL or key/value stores like Redis, could utilize it.

    Maybe bree could even provide an adapter/plugin system to optionally add persistence for (1) scheduled one-off jobs and (2) failed jobs 🤔

    How about using an abstraction layer like TypeORM (good support for SQL databases + MongoDB. Doesn't support Redis tho). You write job model/schema once and then it gets migrated into a database of user's choice by TypeORM.

  3. dturton commented on Nov 30, 2021

    @dturton

    Prisma can be another great option to TypeORM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions