Hatchet: Run Background Tasks at Scale

Hatchet: Detailed System Description

Section 1: Quick Snapshot of Hatchet
Hatchet is a platform designed to run background tasks and orchestrate durable workflows on top of PostgreSQL. It bundles a durable task queue, observability, alerting, a real-time dashboard, and a command-line interface into a single cohesive platform.
It provides built-in support for chaining tasks into workflows, robust monitoring, and a user-friendly UI to observe progress, retries, and outcomes in real time.
Section 2: Visual and Reference Badges (for quick access and credibility)
[Docs badge]
[License badge]
[Go reference]
[NPM downloads]
[Discord]
[Twitter]
[GitHub stars]
Additional note: Hatchet Cloud and Documentation pages are available for exploring a running instance and full capabilities respectively.
Section 3: What Is Hatchet Used For?
Core purpose: offload work from your main web application by scheduling, retrying, and orchestrating background tasks.
Reliability: Hatchet uses a durable queue that persists task history and progress so that work continues or can be alerted upon even if the host application crashes.
Observability: built-in dashboards, logs, and alerting help you detect and diagnose issues quickly.
Extensibility: supports complex workflows through orchestration primitives, flow control, scheduling, and event-driven patterns.
Section 4: Getting Started Quickly
Hatchet emphasizes a fast path to a running instance via the Hatchet CLI. The typical steps are:
- Install the Hatchet CLI on macOS, Linux, or Windows Subsystem for Linux (WSL).
- Ensure Docker is installed locally, as Hatchet runs with a containerized backend.
- Start the Hatchet server with a single command sequence, for example using curl to fetch an installer script, then starting the server.
Quick-start summary (high level):
- Install and run the Hatchet server.
- Sign up for Hatchet Cloud if you want to explore without local hosting.
- Visit the documentation for self-hosting instructions and cloud usage guides.
Section 5: When Should You Use Hatchet?
Background tasks are essential to offload long-running or spike-prone work from your web processes.
Hatchet shines when:
- You need durable, observable queues that can survive worker/process crashes.
- You want to compose tasks into complex workflows (DAGs) with clear input/output handoffs.
- You require robust alerting, real-time dashboards, and precise control over retries and failures.
- You want built-in flow control, rate limits, and concurrency controls to protect system stability under load.
In contrast, simpler queues or ad-hoc scripts may be sufficient for trivial tasks, but Hatchet provides a more scalable and maintainable foundation for larger systems.
Section 6: Core Concepts and Features
6.1 Queues and Durable Execution
- Hatchet uses a durable queue that enqueues tasks and hands them to workers at a rate the workers can handle.
- Progress tracking and guarantees help ensure that work completes or triggers alerts when it cannot complete.
- This durability is especially valuable for preventing dropped user requests, absorbing traffic spikes, and decomposing large logic into smaller, reusable tasks.
6.2 Task Orchestration and Workflows
- Hatchet supports orchestration through workflows, allowing the creation of fans-out and fan-in patterns, as well as sequential progression through steps.
- DAGs (directed acyclic graphs) are a primary mechanism: you predefine how a workflow flows, automatically routing outputs from parent tasks to inputs of child tasks.
- Durable tasks store a full history of spawned tasks, enabling caching of intermediate results and more resilient execution.
- Language examples illustrate how to declare tasks, attach them to workflows, and invoke tasks from your application:
- Python example demonstrates defining a workflow, tasks, and the invocation pattern.
- TypeScript example shows registering tasks and running them via a worker instance.
- Go example outlines creating a workflow, adding tasks with dependencies, and triggering execution.
6.3 Flow Control for Per-User, Per-Tenant, and Per-Queue Throttling
- Flow control helps avoid overwhelming users or backends by applying concurrency limits and rate limits.
- Concurrency: you can set per-key limits (e.g., per-user) to cap how many tasks can run simultaneously.
- Rate limits: global or dynamic limits, such as per-user quotas per minute, ensure fair resource usage.
- Language-agnostic examples illustrate how to set a concurrency expression, max runs, and a rounding strategy, plus rate limit definitions on individual tasks.
6.4 Scheduling
- Hatchet supports cron-based and one-time scheduling as well as durable sleep (pausing execution for a duration).
- Scheduling lets you automate recurring pipelines, data refreshes, or notification workflows without manual triggers.
- Examples in Python, TypeScript, and Go show how to schedule a one-off task, add cron rules, and align scheduled runs with input data.
6.5 Task Routing and Scheduling Semantics
- Sticky assignment: spawn tasks can prefer or require execution on the same worker, which can improve cache locality and throughput.
- Worker affinity: workers are ranked to determine the best-suited worker for a given task, balancing load and reducing latency.
- Examples in Python, TypeScript, and Go demonstrate how to declare sticky behavior and how to target a specific worker or allow scheduling flexibility when the original worker is busy.
6.6 Event Triggers and Listeners
- Hatchet supports event-driven execution: tasks can pause until a specific external event fires, or trigger new workflows when an event occurs.
- Event-based patterns enable complex reactive flows where external conditions determine execution progress.
6.7 Real-Time Web UI
- The platform comes with a real-time dashboard that surfaces task statuses, queues, and workflow progress.
- Logging is integrated to correlate task failures with application logs.
- Alerting is available for task failures via Slack and email, with adjustable alerting windows.
- The UI features live updates and built-in alerting to help operators respond quickly to incidents.
Section 7: Practical Code Illustrations (Cross-Lamily Examples)
7.1 Python: A Simple Task and a Simple Workflow
- Demonstrates defining a simple input model, a task with a transformation, registering it on a worker, and invoking the task from an application.
- This example helps illustrate the end-to-end flow from task definition to execution and result retrieval.
7.2 TypeScript: A Simple Task and a Simple Workflow
- Shows how to declare a task input type, define a task function, register the task with a worker, and run the task from your app.
- Highlights the strong typing and clear separation between task definitions and invocation in a TypeScript codebase.
7.3 Go: A Simple Task and a Simple Workflow
- Presents a Go-based approach to declaring input types, creating a task with a transformation function, registering with a worker, and invoking execution.
- Demonstrates idiomatic patterns for Go developers seeking to integrate Hatchet with Go services.
Section 8: Task Orchestration Patterns in Depth
DAG-driven workflows
- Predefine the shape of work and automatic data flow by routing outputs to inputs.
- Supports complex dependencies and parallelism while preserving correctness.
Durable tasks and caching
- Each spawned task contributes to a verifiable history, enabling result reuse and robust caching strategies.
Fan-out and fan-in
- Break large workloads into parallel tasks when possible to shorten total processing time, then aggregate results in later steps.
Failure handling
- Retries, compensation steps, and alerting strategies help ensure resilience in the face of transient errors.
Section 9: Scheduling and Routing in Practice
Scheduling
- Cron-style schedules can orchestrate regular pipelines, while one-time schedules target future runs for ad-hoc data processing.
- Durable sleep allows defined pauses between steps, enabling time-based pacing and rate control.
Task routing and affinity
- Sticky assignment helps preserve locality for hot workloads or cache warmth.
- Worker affinity directs tasks to the best-suited worker, balancing load and optimizing throughput.
Section 10: Flow Control and Throughput Guarantees
Concurrency and rate limits
- Concurrency controls ensure per-user or per-tenant bottlenecks are avoided during peak usage, reducing cascading effects on the system.
- Dynamic rate limiting prevents spiky bursts from overwhelming downstream services or external dependencies.
Practical examples
- Real-world scenarios show how to configure per-user concurrency with a max run limit and a specific strategy (e.g., GROUPROUNDROBIN).
- Rate limits demonstrate per-user quotas to cap task consumption per minute, ensuring predictable behavior.
Section 11: Event Triggers, Listeners, and Reactivity
Event listening
- Tasks can pause and resume in response to external events or conditions, enabling complex workflows that hinge on real-world signals.
Event triggering
- External events can initiate new tasks or steps within a workflow, enabling reactive pipelines and event-driven architectures.
Section 12: Real-Time Monitoring and Observability
Real-time dashboards
- Live updates on task progress, queue health, and workflow status help operators detect anomalies quickly.
Logging
- Centralized logs tied to task execution simplify debugging and correlation with application logs.
Alerts
- Slack and email alerts notify teams promptly when tasks fail or meet defined failure thresholds.
Section 13: Documentation, Community, and Support
Documentation
- The most up-to-date information is available at https://docs.hatchet.run.
Community channels
- Discord: for developer support and discussion
- GitHub Issues: for bug reports
- GitHub Discussions: for in-depth technical conversations
- Email: contact@hatchet.run for Hatchet Cloud support and billing inquiries
Contribution
- The project welcomes contributors and encourages engagement through the #contributing channel on the Hatchet Discord server.
Section 14: Hatchet vs Other Platforms (A Comparative View)
Hatchet vs Temporal
- Hatchet aims to be a general-purpose orchestration platform with broad capabilities, including queueing, DAG-based workflows, and durable execution, all built on Postgres.
- Temporal focuses more narrowly on durable execution with a broader set of database backends and result stores for certain deployments.
- When to choose Hatchet: you want more control over underlying queue behavior, DAG-based workflows, and simplified self-hosting with a Postgres backbone.
- When to choose Temporal: you require non-Postgres result stores or workloads that align with Temporal’s durable execution model.
Hatchet vs BullMQ / Celery
- Hatchet is a durable queue capable of maintaining complete execution history, enabling robust UI support and observability.
- BullMQ and Celery are highly performant but rely on external tooling for persistence and monitoring; their durability and observability features are typically less integrated than Hatchet.
- When to choose Hatchet: you want persistent history, built-in observability, and a unified UI for monitoring tasks and workflows.
Hatchet vs DAG-based platforms (Airflow, Prefect, Dagster)
- Airflow, Prefect, Dagster are often heavier data engineering platforms with higher latency and cost, optimized for data pipelines with integrations and connectors.
- Hatchet is designed for high-volume application workloads requiring throughput, low latency, and strong runtime durability in a web-application context.
- When to choose Hatchet: you want a high-throughput, low-latency orchestrator tightly integrated with your application and Postgres-based durability.
- When to choose other DAG platforms: you want specialized data-ops features, connectors, and broader ecosystem integration out of the box.
Hatchet vs AI frameworks
- Many AI frameworks run in-memory with durability as an afterthought; Hatchet provides explicit durability, task orchestration primitives, and a robust UI to manage, monitor, and observe tasks and workflows.
- When to use Hatchet alongside an AI framework: you can rely on Hatchet for orchestration, fault tolerance, and monitoring while using AI models and calls within the defined tasks.
Section 15: Issues, Contribution, and Getting Involved
If you encounter bugs, submit issues on GitHub to help the maintainers triage and fix problems.
For contributions, engage with the community on Discord and discuss ideas or features in GitHub Discussions.
Hatchet welcomes new contributors and offers guidance on where your help is most needed.
Section 16: Visual Aids and Inline Media (References from the Input)
Hatchet’s branding and identity are reflected by the logo included at the top of this description.
Visual badges provide quick access to essential resources:
- Docs, License, Go reference, NPM downloads, Discord, Twitter, and GitHub stars.
Real-time UI and observability images (as referenced in the input) illustrate:
- A live dashboard with dashboards, metrics, and alert indicators.
- The logging and alerting visuals that aid in rapid debugging and incident response.
Note: The actual images cited in the content are embedded or linked to in the input, and their presence enhances comprehension of Hatchet’s UI and capabilities.
Section 17: Summary and Takeaways
Hatchet presents a unified platform for running background tasks and durable workflows with a PostgreSQL-based backbone, extensive observability, and a feature-rich orchestration surface.
Its core strengths lie in durability, real-time monitoring, and flexible orchestration patterns (DAGs, durable tasks, and event-driven flows).
With built-in flow control, scheduling, and sophisticated task routing, Hatchet helps maintain system stability under load while enabling complex, scalable architectures.
The platform is suitable for developers seeking strong guarantees around task execution, end-to-end visibility, and a self-hostable solution with cloud options.
By leveraging Hatchet, teams can simplify the construction and operation of long-running data pipelines, background processing jobs, and reactive systems, all while benefiting from a cohesive UI and a consistent developer experience across Python, TypeScript, and Go ecosystems.
Section 18: Accessibility and How to Read This Description
The description is organized into sections with clear headings and is peppered with bullet points and numbered sequences to guide you through concepts, patterns, and usage.
Key terms are highlighted through structured subsections, enabling quick navigation for developers, operators, and decision-makers.
Images from the input are included to provide visual context and branding references, making it easier to connect textual descriptions with the real Hatchet interface and assets.
Section 19: Final Note
Hatchet is designed to scale with your needs, from small teams prototyping background tasks to large-scale systems requiring durable workflows and real-time observability.
The combination of a durable Postgres-backed queue, robust orchestration primitives, and a user-friendly UI makes Hatchet a compelling option for modern backend infrastructure.
For more information and the latest updates, consult the official Hatchet documentation and join the community channels to participate in ongoing development and discussion.
Section 20: Images Embedded in Context (Direct References)
Logo: Hatchet Logo at the top of the document.
Docs badge: embedded as a badge reference near the section describing quick access to documentation.
License badge: embedded to communicate the MIT license clearly.
Go reference, NPM downloads, Discord, Twitter, and GitHub stars badges: included to provide quick access to tooling, packages, and community engagement.
Real-time UI visuals: inline media references that illustrate dashboards and logs, reinforcing the description of the real-time observability features.