Skip to content

Store Celery task events in a datastore like Redis #1192

Open

Description

@dmorina

Is your feature request related to a problem? Please describe.
Storing a large number of tasks e.g. 1M+ in memory makes Flower super slow. Additionally, given the fact that internally it uses an LRUCache whenever we schedule a ton of tasks (more than the in memory cache holds) we lose information about the tasks, old tasks that have been scheduled but not picked up by workers yet will be cleared out (this is common with lazy-queues that are consumed slowly).

Describe the solution you'd like
I think it would be great to shift the cache to a datastore like Redis, this would make Flower more responsive and allow for storing task events as long as they are needed.

Would you be open to something like that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions