An Airflow 3 deployment is a few separate services around a metadata database:
- The Dag processor parses Dag files on a loop and stores the serialized Dags. Import errors come from here; slow top-level code slows it down.
- The scheduler creates Dag runs from timetables and asset events, decides which task instances are ready, and hands them to the executor.
- The executor decides where tasks run:
LocalExecutor (processes on the scheduler's machine), CeleryExecutor (a pool of workers fed by a queue), KubernetesExecutor (a pod per task).
- Workers run tasks through the Task SDK. In Airflow 3 they talk to the API server, not straight to the database.
- The API server (
airflow api-server) serves the UI, the REST API and the API the workers use.
- The triggerer runs deferred tasks' async triggers, thousands at a time.
Knowing the split tells you where to look: a Dag missing from the UI is a parsing problem; tasks stuck in queued point at the executor or workers; deferred tasks that never resume point at the triggerer.
The services of a Celery deployment, each started on its own.