Django Tasks: Why I Skipped Celery for InboxToKindle

Every Django developer knows the moment. You need a background task. Send an email, process a file, call an API asynchronously. And someone says: "Just use Celery."

So you spend an afternoon setting up Redis, configuring a broker URL, writing a celery.py file, running a separate worker process, installing Flower so you can monitor it, and wondering why a simple background job now requires three running services and a new mental model.

I call this the Celery tax. It's real, and most projects pay it without questioning whether they need to.

Why I Built InboxToKindle

I've written before about wanting to reclaim my attention from the infinite scroll. I like reading newsletters, but reading them on a phone is a losing battle. You open the mail app, click a link, switch apps, and off you go.

So I built InboxToKindle, a service that converts newsletter emails to EPUB files and delivers them to your Kindle. Clean, distraction-free, on a screen designed for reading. It's the third product I've built at Tyn Studio, and it follows the same principle as the others: start simple, use what's already there, and only add complexity when the problem demands it.

That principle is also how I ended up skipping Celery.

This is the third post in my series on building InboxToKindle: the first post introduced the product, and the second post covered the architecture. This one goes deep on background task processing.

What Changed in Django 6.0

Django 6.0 introduced a built-in Tasks framework, a standard API for defining and running background work outside the request-response cycle. The idea had been floating around the Django community for years. Jake Howard first proposed it in 2021 for the Wagtail CMS, it became a formal Django Enhancement Proposal (DEP 0014) in 2024, and after nearly a year of review it finally landed in Django 6.0. Adam Johnson, who served on the Steering Council during the evaluation, wrote about it and called it one of the most exciting additions to the framework.

There's an important distinction here. Django 6.0 provides the framework itself (the @task decorator, the .enqueue() method, the TaskResult model, and the backend interface), but it only ships two backends out of the box. ImmediateBackend executes tasks synchronously, which is useful for development. DummyBackend does nothing, which is useful for testing. For production, you need a third-party backend.

The one I chose is django-tasks, which provides a DatabaseBackend that stores tasks in your existing PostgreSQL or SQLite database. The tasks live in the same database as everything else. No Redis or RabbitMQ required.

This isn't a new idea. Third-party packages like django-background-tasks and procrastinate have offered database-backed queues for years. But having the core framework in Django itself changes things. There's a standard interface now. Backends are swappable. The ecosystem will converge around this API rather than each package inventing its own.

When I read about it, I knew this was what I wanted for InboxToKindle. I'd been putting off the background task decision, and now I didn't have to make the hard choice. The simple choice was the right one.

How I Use It in InboxToKindle

Configuration

The settings are minimal:

# config/settings.py
TASKS = {
    "default": {
        "BACKEND": "django_tasks.backends.database.DatabaseBackend",
        "QUEUES": {
            "default": {
                "BACKEND": "django_tasks.backends.database.DatabaseBackend",
            }
        },
    }
}

# Task retry configuration
TASK_RETRY_ATTEMPTS = 3
TASK_RETRY_DELAY = 60  # seconds

That's the entire task infrastructure configuration. No broker URL, no result backend, no separate config file. The tasks use the same PostgreSQL database that stores users, newsletters, and delivery logs.

I remember staring at this settings block after writing it and thinking: that's it? Coming from projects where the Celery configuration alone was a dedicated file with dozens of options, this felt almost too simple. It wasn't. It just didn't need to be complicated.

Defining and Enqueuing Tasks

The @task() decorator turns any function into a background task. One thing to keep in mind: all arguments and return values must be JSON-serializable. No datetime objects, no model instances, just strings, numbers, lists, and dicts. I pass record IDs and look up the objects inside the task.

Here's the core of InboxToKindle's email processing pipeline, close to what runs in production:

# newsletters/tasks.py
from django_tasks import task

@task()
def process_inbound_email(email_id):
    """Parse and process a received email."""
    inbound = InboundEmail.objects.get(id=email_id)

    # Handle S3 vs webhook format
    if "parsed" in inbound.raw_data and inbound.raw_data.get("source") == "s3":
        parsed = inbound.raw_data["parsed"]
    else:
        parsed = parse_email(inbound.raw_data)

    # Find subscription, check limits, deduplicate
    subscription = SubscriptionEmail.objects.get(
        email_address=parsed["to_address"], is_active=True
    )

    if not can_deliver(subscription.user):
        inbound.processed = True
        inbound.save()
        return

    # Create newsletter and clean content
    newsletter = Newsletter.objects.create(
        subscription=subscription,
        user=subscription.user,
        subject=parsed["subject"],
        sender=parsed["from_address"],
        raw_content_html=parsed["html_body"],
        status="processing",
    )

    cleaned = clean_substack_content(parsed["html_body"])
    newsletter.cleaned_content = cleaned
    newsletter.status = "ready"
    newsletter.save()

    # Chain to next step
    generate_kindle_document.enqueue(str(newsletter.id))

The real version is longer. It handles confirmation emails, usage tracking, deduplication by Message-ID, and error recovery. But the shape is the same: a task does its work, then enqueues the next task in the pipeline.

The Pipeline

InboxToKindle has a multi-step pipeline, and each step is its own task:

1. process_inbound_email parses the raw email, validates the subscription, checks usage limits, cleans the HTML, and creates a Newsletter record.
2. generate_kindle_document converts the cleaned HTML to EPUB format, uploads it to S3, and checks whether the user wants instant delivery or a digest.
3. send_to_kindle sends the EPUB to the user's Kindle email via SES. If the file is too large for an attachment, it falls back to a download link.
4. compile_digest gathers undelivered newsletters for users who prefer daily or weekly digests and compiles them into a single EPUB.

There are also supporting tasks like notify_delivery_failed, notify_no_kindle_device, send_newsletter_to_email, and cleanup_old_inbound_emails. Each one is its own @task() function, self-contained and independently retryable.

I'll be honest: the first version of this pipeline was not this clean. I had four tasks chained together with barely any error handling, and I spent a weekend debugging why certain emails were being processed twice. The deduplication by Message-ID came out of that weekend. Sometimes the simplest-sounding architecture still takes a few iterations to get right.

The chaining itself is explicit. A task enqueues the next task when it finishes. No dependency framework, no DAG, just one function calling .enqueue() on another. When something fails, only that step fails. When I retry, only that step reruns. That granularity has saved me more than once.

How Tasks Are Stored

The DatabaseBackend stores tasks in a DBTaskResult table in your database. Each record has a task_name, a status (READY, CLAIMED, SUCCEEDED, or FAILED), the JSON-serialized arguments, a priority value, and a run_after timestamp for scheduling. There's also a claimed_by field that tracks which worker picked it up, and either a result or error field depending on the outcome.

I can inspect the task queue from the Django admin, query it with the ORM, and understand the state of the system without opening a separate monitoring tool. That alone is worth a lot. When something goes wrong at 11pm, I'd rather run a Django shell query than SSH into a worker and grep through Celery logs.

The Worker

This is the part that took the most thought. The django-tasks package provides a basic worker, but I ended up writing a custom management command that acts as both a worker and a scheduler:

# newsletters/management/commands/run_scheduler.py
class Command(BaseCommand):
    def handle(self, *args, **options):
        while True:
            now = timezone.now()

            # Fetch emails from S3 every 60 seconds
            # Process pending emails every 30 seconds
            # Generate EPUBs every 2 minutes
            # Process queued django-tasks every cycle
            # Check for scheduled digests every hour

            self.process_pending_tasks()
            time.sleep(interval)

The process_pending_tasks method is where the queue gets drained:

def process_pending_tasks(self):
    from django_tasks.backends.database.models import DBTaskResult

    now = timezone.now()
    pending_tasks = (
        DBTaskResult.objects.filter(status="READY")
        .filter(run_after__lte=now)
        .order_by("priority", "enqueued_at")[:10]
    )

    for task_result in pending_tasks:
        worker_id = str(uuid.uuid4())
        task_result.claim(worker_id)
        args = task_result.args_kwargs.get("args", [])
        kwargs = task_result.args_kwargs.get("kwargs", {})
        result = task_result.task.call(*args, **kwargs)
        task_result.set_succeeded(result, {})

It queries the database for tasks that are ready, claims them with a worker ID so no other worker picks them up, executes them, and marks them as succeeded or failed. That's the entire worker. A management command polling a database table.

I run it with:

python manage.py run_scheduler --interval 60

One process. It starts on deploy and runs continuously. No Redis, no Flower, no separate infrastructure to worry about.

Periodic Tasks: The Self-Rescheduling Pattern

django-tasks doesn't have a built-in scheduler like Celery Beat. The solution I use is simple: a task reschedules itself before it exits.

# newsletters/tasks_periodic.py
from django_tasks import task
from datetime import timedelta
from django.utils import timezone

@task()
def auto_fetch_s3_emails():
    """Fetch emails from S3, then reschedule for next run."""
    call_command("fetch_s3_emails", delete=True)

    # Queue each new email for processing
    for email in InboundEmail.objects.filter(processed=False):
        process_inbound_email.enqueue(str(email.id))

    # Reschedule self for next run
    next_run = timezone.now() + timedelta(minutes=1)
    auto_fetch_s3_emails.enqueue(run_after=next_run)

The run_after parameter tells the worker not to pick up the task until that time has passed. The task runs, does its work, and drops a new version of itself into the queue set to run one minute later.

I have three of these loops running continuously:

• auto_fetch_s3_emails every minute
• process_all_pending_emails every 30 seconds
• generate_and_send_pending_epubs every 2 minutes

To start them all, there's a bootstrap task:

@task()
def start_automation_scheduler():
    """Call once on deploy to start all loops."""
    auto_fetch_s3_emails.enqueue()
    process_all_pending_emails.enqueue()
    generate_and_send_pending_epubs.enqueue()

Call this once on deploy and the loops run indefinitely. Is this as precise as a cron scheduler? No. There's some drift as execution time adds to the interval. For InboxToKindle, where fetching emails every minute or two is perfectly adequate, it's not a problem worth solving.

I've found this pattern works for most of my projects. Whether it's processing newsletter emails or analyzing urban transport networks, the same principle applies: reach for the simplest tool that solves the problem. You can always add complexity later. You can rarely remove it.

What I Gained by Skipping Celery

The infrastructure cost alone is worth considering for early-stage projects. A Redis instance on a managed platform runs $15-25 per month before you add the worker dynos to run Celery itself. For InboxToKindle, which runs on a single server, that cost buys nothing I actually need.

What I appreciate most, though, is the operational simplicity. One process to monitor, one database to back up, one place to look when something goes wrong. When a task fails, I can query the DBTaskResult table directly, inspect the traceback, and understand what happened without SSH-ing into a worker or cross-referencing timestamps between services.

When You Should Still Use Celery

Celery exists for good reasons, and some projects genuinely need it. If you're processing thousands of tasks per second, database polling will become a bottleneck. If you need workflow primitives like chains, chords, and groups to coordinate parallel tasks, you'll want Celery's mature tooling. Same goes for specialized worker pools with different concurrency settings, or real-time task prioritization at scale.

These are real requirements. If you have them, database-backed tasks will eventually cause pain and you'll end up migrating to Celery anyway.

But most web applications don't have these requirements. They need to send a welcome email, process an uploaded image, generate a report, or convert an email to EPUB and deliver it. For this kind of work, django-tasks isn't a compromise. It's the right tool.

Start Simple, Scale When Needed

The instinct to reach for Celery before you need it comes from a good place. Nobody wants to migrate task infrastructure after the fact. But the cost of that insurance is real: more services to run, more things to monitor, more concepts for new developers to learn.

Django 6.0's task framework inverts the default. You start with something that works, costs nothing extra, and requires no new infrastructure. If you eventually outgrow the database backend, you swap it for something else without rewriting your tasks. The API stays the same.

That day may come. For most projects, including InboxToKindle at its current scale, it hasn't arrived yet.

If you're building a Django application and you're about to spin up Redis for Celery, stop for five minutes and ask whether django-tasks would get the job done. There's a good chance the answer is yes.

I'm curious: have you tried Django's new task framework? Or are you still on Celery and happy with it? I'd genuinely like to hear. Drop me a line at hi@tynstudio.com.

This is how we build at Tyn Studio: start simple, use what's already there, and only add complexity when the problem demands it. If you're working on a Django project and need help, whether it's background tasks, geospatial features, or a full production application, let's talk.

InboxToKindle is a service that delivers newsletters directly to your Kindle as EPUB files. Try it at inboxtokindle.com.