Skip to content

Investigate missing asyncio support for Celery #21

@devraj

Description

@devraj

As part of this lab, we made a massive push to migrate our technology stack including the future of SQLAlchemy which is headed towards a 2.0 syntax (this is in line with SQLAlchemy 2.0 release, where there 1.4 branch provides a migration strategy). This aligns SQLAlchemy to work with the async mode that our asgi application based on FastAPI was already headed.

Celery our queue processor of choice was meant to support ayncio in the 5.x release, which is what we use in this stack. Until such time that Celery supports we have the following choices:

  • Use FastAPI background tasks wherever possible to process background tasks
  • Find a way to use async code in a sync fashion so we can use Celery without asyncio support
  • Provide a sync version of SQLAlchemy sessions and queries - this is the least preferred option as it will add unwanted code that will significantly add to the technical debt

At the moment our use cases for applications would be able to get away with using FastAPI background tasks, our current requirements are limited to sending SMS and email and processing webhook payloads from Stripe (these could take longer depending on what we are trying to do and would be more suited to Celery handled tasks).

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions