#APACHE AIRFLOW INTERVIEW QUESTIONS CODE#

#APACHE AIRFLOW INTERVIEW QUESTIONS CODE#

High performance: It optimizes query execution through various techniques like query optimization, code generation, and in-memory caching which results in faster data processing. Unified data processing: Spark SQL integrates relational processing (SQL queries) with Spark's distributed computing capabilities to enable seamless data processing across structured and unstructured data.

It allows developers to leverage the power of SQL and the expressiveness of programming languages like Scala, Java, Python, and R to perform data processing tasks. Spark SQL is a module in Apache Spark that provides a programming interface for working with structured and semi-structured data using SQL queries, DataFrame API, and Dataset API. Memory management: Spark's in-memory computation is based on resilient distributed datasets (RDDs), whereas Flink uses a combination of managed memory and disk-based storage to optimize the usage of memory. Flink provides a unified streaming and batch API called the DataStream API that offers fine-grained control over time semantics and event processing.įault tolerance: Spark achieves fault tolerance through RDD lineage, while Flink uses a mechanism called checkpointing that periodically saves the state of operators to enable recovery from failures. Flink offers built-in support for event time processing and provides more advanced windowing and streaming capabilities.ĭata processing APIs: Spark provides high-level APIs, including RDDs, DataFrames, and Datasets, which abstract the underlying data structures. Processing model: Spark is primarily designed for batch and interactive processing, while Flink is designed for both batch and stream processing. There are also features for letting you easily pre-configure access to a central resource, like a datastore, in the form of Connections & Hooks, and for limiting concurrency, via Pools.Apache Spark and Apache Flink are both distributed data processing frameworks, but they have some differences: TaskFlow API automatically passes data between tasks via implicit XComsĪirflow sends out Tasks to run on Workers as space becomes available, so there’s no guarantee all the tasks in your DAG will run on the same worker or the same machine.Īs you build out your DAGs, they are likely to get very complex, so Airflow provides several mechanisms for making this more sustainable - SubDAGs let you make “reusable” DAGs you can embed into other ones, and TaskGroups let you visually group tasks in the UI. Uploading and downloading large files from a storage service (either one you run, or part of a public cloud)

XComs (“Cross-communications”), a system where you can have tasks push and pull small bits of metadata. To pass data between tasks you have three options: By default, a task will wait for all of its upstream tasks to succeed before it runs, but this can be customized using features like Branching, LatestOnly, and Trigger Rules. These dependencies are what make up the “edges” of the graph, and how Airflow works out which order to run your tasks in. In the default Airflow installation, this runs everything inside the scheduler, but most production-suitable executors actually push task execution out to workers.Ī triggerer, which executes deferred tasks - executed in an async-io event loop.Ī webserver, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks.Ī folder of DAG files, read by the scheduler and executor (and any workers the executor has)Ī metadata database, used by the scheduler, executor and webserver to store state.įirst_task. A workflow is represented as a DAG (a Directed Acyclic Graph), and contains individual pieces of work called Tasks, arranged with dependencies and data flows taken into account.Ī DAG specifies the dependencies between Tasks, and the order in which to execute them and run retries the Tasks themselves describe what to do, be it fetching data, running analysis, triggering other systems, or more.Īn Airflow installation generally consists of the following components:Ī scheduler, which handles both triggering scheduled workflows, and submitting Tasks to the executor to run.Īn executor, which handles running tasks. Airflow is a platform that lets you build and run workflows.

Apache airflow interview questions