deep·tech·intuition
intermediate ·

Daft Deep Intuition

An experienced engineer's guide to Daft

1. One-Sentence Essence

Daft is a lazy, streaming query engine with a DataFrame skin, built so that a multi-megabyte image or a GPU model inference is just another column expression — treated by the optimizer exactly like col("a") + 1.

Everything else about Daft falls out of that sentence. It is not “Polars but distributed” and it is not “Spark but in Rust,” though it borrows heavily from both. The thing that makes Daft Daft is that it took the query-engine machinery the database world spent forty years perfecting — logical plans, predicate pushdown, cost-based optimization, streaming execution — and pointed it at workloads where a single “value” in a cell is a video file, an embedding vector, or a call to GPT-4o. Once you internalize that AI operations are first-class expressions inside a query plan, the rest of Daft becomes predictable.

2. The Problem It Solved

Picture the actual shape of a modern AI data pipeline in 2023. You have ten million PDFs in S3. You need to: download each one, parse it, chunk the text, run an embedding model on a GPU, and write the vectors to a vector database. Or: a million product images, each needs decoding, resizing, a classification model, and the labels written back to a table.

Before Daft, you had three bad options.

Option one: Spark. Spark is magnificent at SQL-style analytics over tabular data — joins, group-bys, aggregations over Parquet. But Spark’s world is rows of integers and strings. The moment your “row” contains a 4 MB JPEG that decodes to 80 MB in memory, Spark’s assumptions break. Its memory accounting doesn’t understand that a column inflates 20x on decode. Its JVM-based executors fight your Python ML code through serialization boundaries. Keeping GPUs fed is a constant battle because Spark was architected for CPU-bound shuffles, not for streaming binaries through a model. People did it anyway, and they spent their lives hand-tuning executor memory and partition counts and watching jobs OOM at hour three.

Option two: a pile of Python scripts. boto3 to download, multiprocessing or asyncio for concurrency, PyTorch DataLoader for batching, hand-rolled checkpointing so a crash at 80% doesn’t cost you the whole run. This works at small scale and becomes an unmaintainable swamp at large scale. You are now the query optimizer, the scheduler, and the memory manager — by hand.

Option three: Ray Data. Ray got the streaming and the heterogeneous CPU/GPU scheduling right, and it’s a real contender. But it historically lacked a sophisticated query planner — it would happily spill excessively to disk and leave you hand-tuning partition sizes and block sizes, doing the optimizer’s job yourself.

Daft was built by people who had lived this pain. The founders, Sammy Sidhu and Jay Chia, came out of self-driving (DeepScale, acquired by Tesla; Lyft Level 5). Self-driving is the original “petabytes of multimodal data — images, LiDAR, video — that has to flow through models” problem. They founded Eventual (YC W22, Series A in 2025), open-sourced Daft, and made one central bet: the abstraction that wins is the DataFrame, but the engine underneath has to be designed from scratch for multimodal data, in Rust, on Arrow, with the query optimizer treating model inference and URL downloads as expressions it can reason about.

That bet is the whole product. Keep it in mind through everything below.

3. The Concepts You Need

These are the words you need before the rest of the document will land. They cluster into four groups.

The data model

  • DataFrame — the top-level object: a table with named, typed columns. The API surface you touch. Borrowed conceptually from pandas/Polars/Spark, so select, where, with_column, groupby, join all mean what you expect.
  • Expression — the single most important concept in Daft. An expression is a description of a computation on columns, not the computation itself. col("a") + 1 is an Expression. col("url") passed to download() is an Expression. A call to a model via prompt(...) is an Expression. Crucially, an LLM call and an integer addition are the same kind of object to Daft — both are nodes in an expression tree the engine will later optimize and execute. This unification is the source of Daft’s power.
  • Series / Arrow arrays — under each column is an Apache Arrow array: a contiguous, columnar, language-agnostic memory layout. “Arrow-native” means Daft can hand a column to your Python UDF without copying or serializing it (zero-copy), which is why pushing Python code to the data is cheap.
  • DataType — Daft’s type system. The headline is that it includes types tabular engines don’t have: Image, Tensor, Embedding, Binary, and arbitrarily nested Struct/List. An Image[MIXED] column is a genuine first-class typed column, not a blob you smuggle through.

The execution model

  • Lazy evaluation — Daft does not execute when you write a transformation. df.select(...).where(...).with_column(...) builds up a plan and runs nothing. This is the behavior that surprises every newcomer and is responsible for half the “why is nothing happening” confusion. Forward reference: this is Mental Model #1.
  • LogicalPlan — the tree of operators (Source, Project, Filter, GroupBy, Join) describing what you want, built up lazily as you chain methods. It describes intent, not execution.
  • Materialization — actually running the plan and producing data. Triggered by specific terminal actions: .collect() (whole result into memory), .show(n) (first n rows), .write_parquet(...) (out to storage), .to_pydict(), etc. Until you call one of these, you have a recipe, not a meal.
  • Optimizer — between your plan and execution sits a two-pass optimizer (rule-based then cost-based) that rewrites your plan into a faster equivalent. You’ll see it via .explain().

The engine internals

  • Swordfish — the native, single-machine execution engine. A streaming engine written in Rust on the Tokio async runtime. When you pip install daft and run on your laptop, this is what executes your plan.
  • Flotilla — the distributed execution engine, built on Ray. Same Swordfish engine runs inside an actor on each worker node; a scheduler on the head node hands out work. The headline promise: the same code runs on Swordfish locally and Flotilla on a cluster, no rewrite.
  • Streaming execution — data flows through the operator graph in bounded batches, like a pipeline, rather than each stage fully completing before the next begins. This is what keeps memory bounded and GPUs fed. Forward reference: Mental Model #2.
  • Partition — the unit of distributed parallelism in Flotilla. A DataFrame is split into partitions (often one per input file); each partition is a task assigned to a worker. Distinct from batch, which is the finer-grained unit of streaming within a partition.

The AI layer

  • UDF (User-Defined Function) — your own Python/library code wrapped so Daft can call it as an expression over batches of a column. The @daft.udf decorator. This is how you bring arbitrary model inference into the plan.
  • AI Functions — Daft’s batteries-included expressions for the common AI operations: prompt() (LLM call, optionally with structured Pydantic output), embed_text()/embed_image(), classify(). They’re UDFs that ship in the box.
  • concurrency and batch_size — the two knobs on a UDF that control, respectively, how many parallel instances run and how many rows each invocation processes. These are your primary levers against OOM and your primary levers for GPU utilization. Forward reference: Section 8 and Section 10 live and die on these.

4. The Distilled Introduction

This section is the 10-hour tutorial, compressed. After it you can install Daft, build a real multimodal pipeline, and understand what each line is doing.

Setup

Daft needs Python 3.10+. Install with the extras you actually need:

pip install -U daft              # core
pip install -U "daft[openai]"    # + OpenAI AI functions
pip install -U "daft[ray]"       # + distributed execution on Ray
pip install -U "daft[aws]"       # + S3 connector

The extras pattern matters: Daft’s core is lean, and connectors (S3, Iceberg, Delta, Hugging Face) and AI providers are opt-in. Don’t install [all] reflexively; it pulls in a lot.

The first principle you must accept immediately

import daft

df = daft.read_parquet("s3://bucket/data/*.parquet")
df  # prints the SCHEMA, then: "(No data to display: Dataframe not materialized, use .collect())"

Printing a DataFrame shows you column names and types but runs nothing. Daft is lazy. This is not a quirk; it is the entire design (Mental Model #1). To see data you must materialize:

df.show(5)        # materialize + display first 5 rows — your everyday inspection tool
df.collect()      # materialize the ENTIRE result into memory — use deliberately

show() is cheap because the optimizer pushes the limit down and only computes what it needs. collect() computes everything. Reaching for collect() when you meant show() is the most common rookie mistake and the most common cause of a surprise OOM.

Reading data

Daft reads from almost anywhere, and the API is uniform:

df = daft.read_parquet("s3://bucket/*.parquet")
df = daft.read_csv("data.csv")
df = daft.read_json("events.jsonl")
df = daft.read_huggingface("calmgoose/amazon-product-data-2020")
df = daft.read_iceberg(iceberg_table)
df = daft.read_delta_lake("s3://bucket/delta-table")
df = daft.from_pydict({"a": [1, 2, 3], "b": ["x", "y", "z"]})  # from memory

Reading is lazy too. daft.read_parquet on a million files returns instantly — it has only resolved the schema and built a Source operator. The files aren’t touched until you materialize.

The core verbs

These are the DataFrame operations you’ll use constantly. They all return a new DataFrame (immutable, lazy) — they never mutate in place.

# SELECT columns
df = df.select("name", "price", "image_url")

# ADD or replace a column (the workhorse — you'll use this more than anything)
df = df.with_column("price_with_tax", col("price") * 1.2)

# FILTER rows (.where and .filter are the same thing)
df = df.where(col("price") > 100)

# LIMIT
df = df.limit(1000)

# SORT
df = df.sort(col("price"), desc=True)

# GROUP BY + aggregate
df = df.groupby("category").agg(
    col("price").mean().alias("avg_price"),
    col("price").count().alias("n"),
)

# JOIN
df = orders.join(customers, on="customer_id", how="inner")

# materialize
df.show(10)

col("x") is the canonical way to reference a column inside an expression; df["x"] is equivalent. Build expressions by chaining: col("price").is_null(), col("name").str.lower(), col("tags").list.contains("sale"). Each .str, .list, .dt, .float namespace holds the typed operations for that data type.

Expressions are where the real work happens

An expression composes into arbitrarily complex column logic, all of which the engine understands and can optimize:

from daft import col
from daft.functions import regexp_extract

df = df.with_column(
    "first_image_url",
    regexp_extract(col("Image"), r"^([^|]+)", 1)   # Rust-powered regex, runs natively
)

Reach for built-in expressions before you reach for a UDF. The built-ins run in native Rust code over Arrow arrays — vectorized, no Python interpreter in the loop. A UDF drops you into Python per-batch, which is slower and where memory pressure starts. The official guidance is explicit: if you can express it with Daft expressions, do that instead of writing a UDF.

The multimodal turn — what makes Daft Daft

Now the part no tabular engine does cleanly. Downloading, decoding, and running a model on images — as ordinary column operations:

from daft.functions import download, decode_image

df = (
    df
    .with_column("image_bytes", download(col("first_image_url"), on_error="null"))
    .with_column("image", decode_image(col("image_bytes"), on_error="null"))
)

download does highly concurrent async I/O (Tokio, under the hood) and produces a Binary column. decode_image turns those bytes into a typed Image column — which, in a Jupyter notebook, renders as actual thumbnails. The on_error="null" is important production hygiene: out of a million URLs, some will 404, and you want a null in that cell, not a dead pipeline.

AI inference as a column

from pydantic import BaseModel, Field
from daft.functions import prompt

class WoodAnalysis(BaseModel):
    is_wooden: bool = Field(description="Whether the product appears to be made of wood")

df = df.with_column(
    "wood_analysis",
    prompt(
        ["Is this product made of wood? Look at the material.", col("image")],
        return_format=WoodAnalysis,   # structured output, validated against the schema
        model="gpt-4o-mini",
        provider="openai",
    )
)

# the result is a struct column; pull a field out of it
df = df.with_column("is_wooden", col("wood_analysis")["is_wooden"])

df = df.collect()   # NOW everything runs: download, decode, inference, extraction

The thing to notice: a GPT-4o call sits in with_column exactly like col("price") * 1.2 did. Daft automatically batches the calls, parallelizes them across cores, handles concurrency, and — because of lazy evaluation — only runs inference on rows that survive any upstream filters. You wrote declarative pipeline logic; Daft handled the scheduling and the parallelism.

Your own models — UDFs

When the built-in AI functions don’t cover your model, wrap it yourself. The class-based UDF is the production pattern because it loads the model once per worker, not once per row:

import daft
from daft import col, DataType

@daft.udf(return_dtype=DataType.string(), concurrency=4, batch_size=16)
class Classifier:
    def __init__(self):
        # runs once per concurrent instance — load the model here
        self.model = load_my_model()

    def __call__(self, images_col):
        # receives a BATCH (an Arrow-backed series), returns a batch
        return [self.model.predict(img) for img in images_col]

df = df.with_column("label", Classifier(col("image")))

concurrency=4 means four model instances run in parallel; batch_size=16 means each __call__ gets 16 rows. On GPU UDFs you’d also pass num_gpus=1. These knobs are how you keep a GPU saturated without OOMing — Section 8 and Section 10.

Writing results

df.write_parquet("s3://bucket/output/", write_mode="overwrite")
df.write_iceberg(table)
df.write_deltalake("s3://bucket/delta/")

Writing is a materializing action — it triggers the whole plan. Daft writes multiple files with UUID names to avoid collisions and to parallelize the write.

Going distributed — the payoff

The same script runs on a cluster by changing the runner, not the logic:

import daft
daft.context.set_runner_ray("ray://head-node:10001")  # now Flotilla executes
# ... identical DataFrame code runs across the cluster ...

This “write once, scale from laptop to 1000 nodes” property is the promise that justifies adopting a whole engine instead of gluing scripts together.

SQL, if you prefer

Daft has a SQL interface over the same engine; daft.sql("SELECT ... FROM ...") produces the same LogicalPlan as the DataFrame API. Use whichever fits; they optimize identically.

5. The Mental Model

Four ideas. Internalize these and you can predict Daft’s behavior without the docs.

Core Idea 1: You are building a plan, not running code. Nothing happens until you materialize.

Every transformation method returns a new, lazy DataFrame carrying a LogicalPlan. Execution is deferred until a materializing action (collect, show, write_*, to_pydict). This is not laziness for elegance’s sake — it is what lets the optimizer see your entire pipeline at once and rewrite it before a single byte moves.

What this predicts:

  • Printing a DataFrame shows a schema and “not materialized,” and you should expect that, not be confused by it.
  • An expensive operation written early (say, an LLM call) might execute last, or not at all on filtered-out rows, because the optimizer reorders it.
  • A bug in your UDF won’t surface when you write the with_column line — it surfaces at collect(), sometimes hundreds of lines later. Debugging means thinking about where materialization happens.
  • Calling .collect() twice re-runs the whole plan twice unless you’ve materialized and held the result. There’s no implicit cache.

Core Idea 2: Data streams through the pipeline in bounded batches; it is not loaded all at once.

Swordfish (and Flotilla on each worker) is a streaming engine. Sources emit batches; intermediate operators transform a batch and immediately pass it on; only blocking operators (sort, aggregate, join) must accumulate. Memory stays bounded to “a few batches in flight” rather than “the whole dataset.”

What this predicts:

  • You can process datasets far larger than RAM, as long as you don’t force full materialization (collect() on the whole thing) or hit a blocking operator on data that doesn’t fit.
  • show(5) on a billion-row source is near-instant: the limit propagates upstream and the stream stops after five rows survive.
  • Memory blowups happen at inflationary and blocking points: image decode (20x inflation), explode, sort, aggregate, join. Knowing the streaming model tells you exactly where to look when memory spikes (Section 10).
  • GPUs stay busy because batches keep arriving rather than waiting for an entire stage to finish — this is the architectural reason Daft beats Spark on GPU workloads.

Core Idea 3: An AI operation is just an expression. The optimizer treats prompt(...) like col("a") + 1 — except it knows it’s expensive.

This is the unification that defines Daft. Model inference, URL downloads, image decoding, and Python UDFs all become nodes in the plan. But the optimizer has special knowledge: it isolates these expensive projections into dedicated nodes, refuses to push them into scans, and schedules them as late as correctness allows — so it never wastes a GPU call on a row that a later filter would discard.

What this predicts:

  • Putting a filter after an expensive prompt() in your code is fine — the optimizer will pull the filter earlier if it legally can, so inference runs on fewer rows. (But don’t rely on it blindly; see Things That Bite You.)
  • Expensive operators get independent batching, concurrency, and backpressure — which is why concurrency and batch_size are per-UDF knobs and not global settings.
  • The reason Daft can “keep the GPU fed” is that the plan understands which operator is the expensive bottleneck and streams work into it.

Core Idea 4: One engine, two runners. Your logic is decoupled from where it runs.

The LogicalPlan and the optimizer are runner-agnostic. Swordfish executes a plan on one machine; Flotilla executes the same plan across a Ray cluster by running a Swordfish instance inside an actor on each worker and scheduling partitions to them by data locality. You switch with one line.

What this predicts:

  • Develop and debug on your laptop with the exact code that runs in production. The “works on my machine” gap mostly closes.
  • Distributed concepts (partitions, shuffles) only become your problem at the scale where they matter; below that, the local runner just works.
  • Performance characteristics differ between runners (a shuffle is free-ish locally, expensive across the network), so the correctness transfers but the tuning sometimes doesn’t (a real downside — Section 11).

6. The Architecture in Plain English

Walk a single query from your keystrokes to bytes on disk.

You write DataFrame or SQL code. Each method call appends an operator to a LogicalPlan — a tree like Source → Filter → Project → GroupBy. Inside the Project operators live expression trees: col("a") + lit(1) is a small tree with a column-reference leaf, a literal leaf, and an addition node. A prompt(...) call is a node in exactly the same kind of tree. At this point nothing has executed; you hold a description of intent. You can inspect it with df.explain().

You call a materializing action (collect, show, write_parquet). This triggers the optimizer, which makes two passes:

  1. Rule-based pass. Classical rewrites with no runtime stats needed: push filters and projections and limits down toward the source (so you read less), prune columns you never use, fold and split projections, drop redundant repartitions, simplify expressions, unnest subqueries.
  2. Cost-based pass. Using statistics gathered from the sources, it reorders joins to the cheapest ordering via brute-force enumeration.

Layered on top is multimodal awareness: expensive projections (UDFs, model inference, URL downloads, image decode) are split out into dedicated logical nodes and deliberately not pushed into scans, so the engine can batch, schedule, and backpressure them independently — and it runs them as late as correctness permits, after joins and aggregations, to avoid wasting compute on discarded rows.

The optimized plan goes to a runner.

On Swordfish (single machine): the runner builds a graph of physical operators mirroring the plan. Operators connect via async channels and pass data up the graph in batches. Sources read from disk/object store and emit batches; intermediate operators (Project, Filter, UDF) transform each batch and forward it immediately; sinks accumulate. Streaming sinks like Limit can emit early and stop the upstream; blocking sinks like Aggregate and Sort must wait for all input. Each operator chooses its own parallelism and batch size, and the whole thing runs on a Tokio threadpool doing async I/O and multithreaded compute. State lives in the batches flowing through channels and in the accumulation buffers of blocking operators — there’s no central dataframe sitting in memory.

On Flotilla (distributed): a Ray cluster has one head node and many workers. Each worker runs one Ray actor hosting a full Swordfish engine. The Flotilla scheduler on the head node splits the DataFrame into partitions (typically one per input file), then assigns each partition as a task to a worker — choosing workers by data locality (run the compute where the data already is) and load balance. Each worker executes its partition with Swordfish, using the whole machine. Task outputs land in Ray’s object store, and for global operations (group-bys, joins) data is shuffled between workers through that object store. The key insight: distribution is “many Swordfish engines coordinated by a scheduler,” not a different execution model.

So: plan (intent) → optimize (rewrite) → stream through operators (one machine or many). State lives in flight, not in a monolithic in-memory table. That single sentence explains the memory behavior, the laziness, and the scaling story all at once.

7. The Things That Bite You

Each of these traces back to a mental model. That’s how you know they’re structural, not trivia.

1. “Nothing is happening” — because everything is lazy. You expect: writing transformations runs them. What happens: you chain ten operations, print the DataFrame, and see only a schema and “not materialized.” Nothing ran. Why: Mental Model #1. Handle it: use .show(n) to inspect during development; remember a materializing action is required to execute. Internalize that building the pipeline and running it are separate phases.

2. collect() when you meant show() — instant OOM. You expect: a quick peek. What happens: .collect() materializes the entire result into memory; on a large source it OOMs or spills hard. Why: collect is “compute everything,” show(n) is “compute just enough for n rows” (the limit pushes upstream — MM #2). Handle it: show() to inspect, collect() only when you genuinely need the full result in memory, write_* to stream results out without holding them.

3. Image decode (and explode/decompress) silently inflates memory ~20x. You expect: a column of small JPEGs is small. What happens: decode_image turns a 4 MB JPEG into ~80 MB of raw pixels; a batch that fit comfortably as bytes blows past memory once decoded. Why: decode is an inflationary operator in a streaming engine (MM #2) — the batch grows mid-pipeline. Handle it: shrink the batch before the inflation with df.into_batches(...), lower the decoding UDF’s batch_size, and decode as late as possible so filters cut rows first.

4. Re-materializing re-runs the whole plan. There is no implicit cache. You expect: after df.show(), the next action reuses the work. What happens: every materializing action re-executes the plan from the sources. Call collect() twice, download every image twice. Why: MM #1 — a lazy DataFrame is a recipe, and recipes don’t remember being cooked. Handle it: materialize once into a concrete result and reuse that (e.g. assign df = df.collect() and operate on the collected frame), rather than re-triggering the lazy chain.

5. UDF concurrency × batch_size is your real memory budget, and the defaults won’t always fit. You expect: Daft figures out memory for you. What happens: a memory-hungry model UDF at high concurrency runs many heavy instances at once and OOMs; workers get killed and restarted, which is slower than running fewer at once. Why: expensive operators get independent batching/concurrency (MM #3), and you own those knobs. Handle it: lower concurrency for memory-hungry UDFs (fewer parallel instances often beats OOM-kill-restart churn), lower batch_size so each call handles less, set num_gpus/num_cpus to reserve resources correctly.

6. Highly concurrent download() queues huge responses in memory. You expect: more concurrency is always faster. What happens: hundreds of concurrent downloads buffer large responses faster than downstream operators consume them; memory balloons. Why: download is async and aggressive by default; downstream backpressure isn’t infinite. Handle it: cap max_connections on download() to bound in-flight bytes. Slower-but-stable beats fast-then-dead.

7. UDFs are slower than expressions, and beginners reach for them first. You expect: “I’ll just write a Python function.” What happens: you drop out of vectorized native Rust into per-batch Python and lose a lot of speed. Why: built-in expressions run on Arrow arrays in native code with no interpreter in the loop; UDFs cross into Python. Handle it: the official rule — if it can be an expression, make it an expression. Reach for a UDF only for genuinely custom logic (your own model). When you do, use the class-based form so the model loads once per instance, not per row.

8. on_error defaults can stop a pipeline on one bad row. You expect: one corrupt image among a million is skipped. What happens: without on_error="null", a single decode/download failure can fail the operation. Why: fail-fast is the safe default for correctness. Handle it: explicitly pass on_error="null" on download, decode_image, etc. for large dirty datasets, then filter or inspect the nulls.

9. Execution-config tuning knobs are experimental and can change between releases. You expect: a config you set in daft.context.set_execution_config(...) is stable. What happens: batch-sizing and partition-target params are explicitly experimental and may shift between versions, silently changing your tuning. Why: the engine is young and evolving fast. Handle it: record every override you depend on and re-review it on every upgrade; don’t bury these in production without a note.

10. Local tuning doesn’t always transfer to distributed. You expect: what’s fast on Swordfish is fast on Flotilla. What happens: a group-by/join that’s cheap locally becomes an expensive cross-network shuffle through Ray’s object store. Why: MM #4 — correctness transfers across runners, but cost characteristics don’t. Handle it: think about partitioning and shuffle cost when you scale out; profile on a small cluster before assuming local intuitions hold.

8. The Judgment Calls

The decisions that separate someone who uses Daft from someone who knows it.

1. Expression vs. UDF. Expression when the logic is data manipulation Daft already supports (string ops, regex, math, list/struct access, dates). UDF only for genuinely custom code — chiefly your own model inference. Experienced engineers exhaust the built-ins first and treat every UDF as a small performance liability they accept knowingly. The signal: if you’re about to write a UDF that does string parsing or arithmetic, stop — there’s an expression for it.

2. show() vs collect() vs write_* during development. show(n) to inspect (cheap, limited). write_* when the result is large and headed for storage (streams out, never fully held). collect() only when you truly need the whole result resident in memory for further Python-side work — and then you’ve accepted the memory cost. The signal: if you find yourself collecting just to look, you wanted show.

3. concurrency vs batch_size when tuning a UDF. These trade differently. Raising batch_size improves throughput and GPU utilization (bigger batches per inference) but raises peak memory per call. Raising concurrency adds parallel instances (more throughput) but multiplies total memory and, for model UDFs, multiplies model copies in memory. The experienced move on a memory-hungry model: high-ish batch_size, low concurrency — saturate one GPU well rather than thrash several. The signal: OOM-kill-restart churn means you’ve over-set concurrency.

4. Swordfish (local) vs Flotilla (Ray) — when to scale out. Stay local far longer than instinct suggests. Swordfish on a fat single machine (lots of RAM, a GPU) handles surprisingly large streaming workloads because memory stays bounded. Go distributed when (a) the data genuinely exceeds what one machine’s I/O and compute can stream in acceptable time, or (b) you need many GPUs in parallel. The signal: you’re not memory-bound, you’re throughput-bound and a single machine’s GPUs are the ceiling. Don’t adopt Ray’s operational complexity before you need it.

5. Decode/inference placement — trust the optimizer or force the order? The optimizer pushes filters before expensive operators when it legally can. Mostly trust it. But when the dependency is subtle (a filter that depends on a UDF output can’t be pushed before that UDF), structure your pipeline so cheap, selective filters come first explicitly. The signal: check .explain() on expensive pipelines to confirm the plan filters before it spends GPU.

6. into_batches() — when to override default batching. Daft picks batch sizes, but ahead of an inflationary operator (decode, explode) the default can be too large and blow memory. Insert df.into_batches(small_n) right before the inflation point. The signal: memory spikes precisely at a decode/explode step — that’s where to shrink batches, not globally.

7. File format and connector choice. Parquet for general columnar storage; an open table format (Iceberg / Delta / Hudi) when you need ACID writes, schema evolution, and time travel over a lake; Lance when you’re storing vectors/embeddings for retrieval; Turbopuffer/vector DB when the output is a search index. The signal: if multiple writers or evolving schemas are in play, raw Parquet directories will hurt — reach for a table format.

8. AI Functions (prompt/embed/classify) vs. a hand-rolled UDF. Use the built-in AI functions for standard provider calls (OpenAI, etc.) — they handle batching, concurrency, retries, and structured output for you. Hand-roll a UDF when you’re running a local/custom model, need exotic batching, or must control the inference loop. The signal: if you’re about to reimplement provider plumbing inside a UDF, use the AI function instead.

9. Structured output vs. free-text parsing from an LLM. When using prompt, pass a Pydantic return_format so the output is a validated struct column you can index into — rather than parsing free text downstream. The signal: any time you’d write regex against an LLM’s prose output, you wanted structured output instead.

10. How much to lean on Daft vs. keep boundaries. Daft is excellent as the pipeline engine — ingest, transform, infer, write. It is not your serving layer, your transactional store, or your interactive BI cache. The experienced boundary: let Daft own the batch/streaming transformation, and let purpose-built systems own serving and storage. The signal: if you’re trying to make Daft answer low-latency point queries, you’ve pushed it past its design.

9. The APIs That Actually Matter

Grouped by task, with the why.

Reading

  • daft.read_parquet / read_csv / read_json — lazy source creation; resolves schema, touches no data.
  • daft.read_huggingface(...) — direct from the HF hub; great for getting started.
  • daft.read_iceberg / read_delta_lake(...) — table-format reads with the metadata/time-travel benefits.
  • daft.from_pydict(...) — build a DataFrame from in-memory Python, for tests and small inputs.

Transforming

  • with_column(name, expr) — the workhorse; add/replace one column from an expression.
  • select(...) — project columns (and rename via .alias); prunes early thanks to projection pushdown.
  • where(pred) / filter(pred) — row filter; push it early and make it selective.
  • groupby(...).agg(...) — grouped aggregation; a blocking operator (accumulates).
  • join(other, on=, how=) — relational join; the cost-based optimizer reorders multi-join plans.
  • sort(expr, desc=) — blocking; needs all input.
  • into_batches(n) — force a batch size, critically before inflationary ops.
  • explode(col) — turn a list column into rows; inflationary, watch memory.

Expressions

  • col("x") / df["x"] — reference a column.
  • Typed namespaces: .str (lower, contains, regex), .list (lengths, contains, get), .dt (dates), .float/numeric ops — vectorized native ops; prefer these over UDFs.
  • regexp_extract(col, pattern, group) — Rust-speed regex extraction.
  • .alias("name") — name the result; essential inside aggregations and selects.
  • ["field"] on a struct column — pull a field out (e.g. structured LLM output).

Multimodal & AI

  • download(url_col, on_error="null", max_connections=...) — concurrent fetch to a Binary column; cap connections to bound memory.
  • decode_image(bytes_col, on_error="null") — bytes → typed Image; inflationary.
  • prompt([instruction, col], return_format=Pydantic, model=, provider=) — LLM call as a column with structured output.
  • embed_text(...) / embed_image(...) / classify(...) — batteries-included AI expressions.
  • @daft.udf(return_dtype=, concurrency=, batch_size=, num_gpus=) — wrap custom code/models; class form loads the model once per instance.

Materializing & writing

  • show(n) — inspect first n rows cheaply.
  • collect() — full result into memory; use deliberately.
  • explain() — print the optimized plan; your first debugging tool.
  • write_parquet(path, write_mode=) / write_iceberg(...) / write_deltalake(...) — stream results out; triggers execution.

Scaling

  • daft.context.set_runner_ray(address) — switch from Swordfish to Flotilla; same code, distributed execution.

10. How It Breaks

Failure modes, what they look like, and how to think about them.

Out-of-memory / heavy disk spilling — the dominant failure. Symptoms: the job slows to a crawl, spills to disk, or workers get OOM-killed and restart (often masquerading as “the job is stuck”). Root cause: a streaming engine keeps memory bounded except at inflationary operators (decode, explode, decompress), blocking operators (sort, aggregate, join), memory-hungry UDFs, and over-concurrent downloads (MM #2, MM #3). Diagnose: identify which of those four patterns your plan contains; check .explain() for where the expensive/inflationary operators sit. Fix, in order: lower UDF batch_size; cap UDF concurrency; insert into_batches() before inflation; cap download max_connections; only then scale to bigger machines or more workers.

Pipeline fails on dirty input. Symptoms: a single bad URL or corrupt image kills the whole run. Root cause: fail-fast defaults. Diagnose: read the traceback to the offending operator. Fix: pass on_error="null" to download/decode_*, then filter or quarantine the nulls.

Surprise re-computation / “it ran the downloads twice.” Symptoms: the same expensive work happens more than once; costs double. Root cause: lazy plans re-execute on every materializing action; no implicit cache (MM #1). Diagnose: count how many materializing actions your code triggers on the same chain. Fix: materialize once and reuse the concrete result.

Idle GPU / poor throughput. Symptoms: GPU utilization low, job slower than expected. Root cause: batch too small to saturate the GPU, or too few concurrent instances, or an upstream bottleneck (slow downloads) starving the model. Diagnose: check whether the bottleneck is I/O (downloads) or compute (inference). Fix: raise batch_size for the inference UDF; ensure upstream download concurrency keeps the model fed; confirm num_gpus is set so the scheduler reserves the device.

Distributed job slower than local intuition predicted. Symptoms: scaling out didn’t help, or made it worse. Root cause: a global operation (group-by/join) became a network shuffle through Ray’s object store (MM #4); or partition count is wrong for the cluster. Diagnose: look for shuffles in the plan; check partition count vs. worker count. Fix: tune partitioning; minimize global operations; verify data locality is actually being exploited.

Tuning silently changed after an upgrade. Symptoms: a pipeline that was stable regresses after a version bump. Root cause: experimental execution-config defaults changed between releases. Diagnose: diff your pinned config against the new release notes. Fix: pin Daft versions in production and review execution-config overrides on every upgrade.

General debugging workflow: (1) df.explain() — read the optimized plan, find the expensive and blocking operators. (2) Classify your memory risk: inflationary? blocking? hungry UDF? over-concurrent download? (3) Reproduce on a small limit() slice locally on Swordfish before blaming the cluster. (4) Adjust batch_size / concurrency / into_batches / max_connections — in that order. (5) Only then scale hardware. (6) For dirty-data failures, add on_error="null" and inspect the nulls.

11. The Downsides / Disadvantages

The honest price of admission. None of these go away with experience.

1. It’s a young engine, and the tuning surface is explicitly unstable. Where it comes from: Daft is years old, not decades; the execution-config knobs that control batching and partitioning are labeled experimental and change between releases. What it costs you: production pipelines you tuned can regress on upgrade; you must pin versions and re-validate tuning each bump. Spark’s knobs, for all their ugliness, are stable and exhaustively documented across a decade of Stack Overflow. Daft’s are a moving target. Dealbreaker when: you need set-and-forget stability and can’t afford an upgrade-validation cycle. Livable when: you have engineers who own the pipeline and track releases.

2. The ecosystem and community are a fraction of Spark’s. Where it comes from: it’s new and from a single company (Eventual). What it costs you: fewer Stack Overflow answers, fewer pre-built integrations, fewer engineers who already know it, fewer third-party tools that speak Daft natively. When you hit an obscure bug at 2am, you may be filing a GitHub issue rather than finding an answer. Dealbreaker when: you need a deep hiring pool of people who already know the tool, or you depend on a long tail of niche integrations. Livable when: your team is comfortable being early and the core connectors you need exist (they cover the major ones).

3. Single-vendor open source — the governance risk is real. Where it comes from: Daft is developed and steered by Eventual, a VC-backed startup (Series A, 2025) that needs to monetize. What it costs you: the roadmap follows the company’s commercial interest; a future license change, a pivot, or an acquisition could change the terms under which you depend on it. This is the standard COSS (commercial open source) risk, and Daft carries it. Dealbreaker when: you need the multi-vendor, foundation-governed assurance that Apache projects offer. What people think mitigates it but doesn’t: “it’s open source, I can fork it” — forking and maintaining a Rust query engine is not a realistic plan for almost any team.

4. Distributed execution inherits Ray’s operational weight. Where it comes from: Flotilla is built on Ray. What it costs you: the moment you scale past one machine, you’re operating a Ray cluster — head node, workers, the object store, autoscaling, the failure modes of a distributed system. The “same code scales” promise is real for the logic; the operations are a genuine new burden. Dealbreaker when: you have no appetite for running distributed infrastructure and your data fits a big single machine anyway. Livable when: you already run Ray, or your scale genuinely demands a cluster.

5. Tuning intuition doesn’t transfer cleanly from local to distributed. Where it comes from: MM #4 — correctness is runner-agnostic but cost is not; a local group-by is cheap, a distributed one is a network shuffle. What it costs you: you’ll tune twice, and the “it was fast on my laptop” confidence is misleading at cluster scale. Dealbreaker when: rarely a dealbreaker, but a real time cost. Livable when: you profile at small cluster scale before trusting local numbers.

6. The lazy/streaming model has a learning curve that bites repeatedly. Where it comes from: MM #1 and MM #2 are genuinely different from pandas’ eager model, where most data people start. What it costs you: the first weeks are full of “why is nothing happening,” surprise OOMs from collect(), and double-execution from missing caching. It’s not harder than Spark’s model, but it is unfamiliar to the pandas majority. Dealbreaker when: your team is pandas-native and resistant to a mental-model shift. Livable when: the team reads this section and internalizes the four mental models up front.

7. It is a batch/streaming transformation engine — not a database, not a serving layer. Where it comes from: by construction, Daft is a query engine over files and tables, optimized for throughput. What it costs you: it cannot be your low-latency point-query store, your transactional system, or your interactive BI cache. Trying to make it one fights the design. Dealbreaker when: you wanted one tool to also serve queries — it won’t. Livable when: you keep clean boundaries: Daft transforms, purpose-built systems serve.

8. Benchmarks are largely vendor-published, so calibrate the claims. Where it comes from: the headline “2–7x faster than Ray Data, 4–18x faster than Spark” numbers come from Daft’s own (open-sourced, reproducible) benchmarks; Ray’s team published their own showing Ray Data ahead on some configurations. What it costs you: the truth is workload-dependent, and you should benchmark your pipeline rather than trust any vendor’s framing. The real differentiator is less “always fastest” and more “designed for multimodal so it doesn’t fall over.” Dealbreaker when: never — just don’t adopt on a marketing number. Livable when: you run a proof-of-concept on your actual data.

12. The Taste Test

What separates code written by someone who gets Daft from someone who learned the syntax.

Reaching for expressions, not UDFs. Beginner: @daft.udf wrapping a Python function that lowercases a string or does arithmetic. Experienced: col("name").str.lower(), regexp_extract(...), native math — UDFs reserved strictly for model inference and genuinely custom logic. A pipeline full of trivial UDFs is the clearest novice tell.

Class-based UDFs that load the model once. Beginner: a function UDF that loads the model inside the call (reloading per batch — catastrophic). Experienced: a class UDF with the model loaded in __init__, concurrency and batch_size and num_gpus set deliberately.

show() for inspection, collect() only when meant. Beginner: df.collect() scattered everywhere to look at data, plus surprise OOMs. Experienced: show(n) for peeks, write_* for large outputs, collect() rare and intentional, and the result materialized once and reused.

on_error="null" on dirty large datasets. Beginner: downloads and decodes with default error handling, pipeline dies on the first bad row in a million. Experienced: on_error="null" plus explicit handling of the nulls — production hygiene baked in.

Memory awareness at inflation points. Beginner: decode a million images in default batches and wonder why it OOMs. Experienced: into_batches() before the decode, capped max_connections on downloads, decode placed late after filters.

Structured LLM output. Beginner: prompt(...) returning prose, then regex to parse it. Experienced: a Pydantic return_format so the output is a typed struct column, indexed with col("x")["field"].

explain() before scaling. Beginner: throws the job at a cluster and hopes. Experienced: reads the optimized plan, confirms filters precede expensive operators, finds the shuffles, then decides whether distribution helps.

Knowing when not to use Daft. Beginner: tries to make Daft serve low-latency queries. Experienced: uses Daft for the transformation pipeline and hands serving to a system built for it.

13. Where to Go Deeper

  • The official Quickstart (docs.daft.ai/en/stable/quickstart) — the e-commerce/image walkthrough; do it hands-on once, it’s the fastest way to feel the lazy/multimodal model.
  • The Architecture page (docs.daft.ai/en/stable/architecture) — short and dense; the API/Planning/Execution layering and the Swordfish/Flotilla split. Re-read it after you’ve written a pipeline; it lands differently.
  • “Managing Memory Usage” (docs.daft.ai/en/stable/optimization/memory) — the four sources of memory pressure and the exact knobs. This is the page you’ll actually return to in production.
  • The Flotilla announcement (“Introducing Flotilla,” daft.ai blog) and the multimodal benchmark post (“Benchmarks for Multimodal AI”) — read these for the why behind the distributed design and the workload patterns it targets; read them skeptically as vendor material.
  • Anyscale’s counter-benchmark (“Benchmarking Multimodal AI Workloads on Ray Data”) — read alongside Daft’s numbers to calibrate. Seeing both sides is the fastest way to stop believing any single benchmark.
  • The UDF and Expressions API references (docs.daft.ai/en/stable/api) — the typed expression namespaces (.str, .list, .dt) are worth skimming end-to-end once so you know what’s available before you ever write a UDF.
  • The GitHub repo (github.com/Eventual-Inc/Daft) — issues tagged good first issue are a genuinely good way to learn internals; the release notes are mandatory reading before any production upgrade given the experimental config surface.

14. The Final Verdict

Here is the honest take after all of that. Daft is the first DataFrame engine that was designed — not retrofitted — for the workload that actually defines this era of data work: take a vast pile of images / audio / video / documents, push them through models, and write the results somewhere. Spark can be coerced into it and will punish you. A pile of Python scripts will do it until it won’t. Ray Data does it well but historically left you doing the query planner’s job by hand. Daft’s bet — make AI inference a first-class expression inside a real query optimizer, on a streaming Rust engine, with one codebase from laptop to cluster — is the right bet, and it’s increasingly clear it was the right bet.

What it gets profoundly right: three things. First, the unification — that prompt(image) and col("a") + 1 are the same kind of object means you get decades of query-optimizer machinery (pushdown, reordering, lazy planning) applied automatically to your GPU pipeline, for free, and you stop hand-tuning. Second, the streaming-by-default execution, which is why memory stays bounded and GPUs stay fed where Spark stalls and Ray spills. Third, the one-codebase-two-runners decoupling, which collapses the dev-to-prod gap that has plagued distributed data work forever.

What it costs you: youth and singularity. The engine is fast-moving, its tuning surface is explicitly unstable, its community is a fraction of Spark’s, and it is steered by one VC-backed company with the governance risk that implies. The shape of the regret, if you feel it, is an upgrade that quietly changes your tuning, or a 2am bug with no Stack Overflow answer, or a roadmap decision made for Eventual’s commercial reasons rather than yours.

Who should reach for it: teams building multimodal AI data pipelines — embedding pipelines, batch inference over images/audio/video/PDFs, training-data preprocessing — who have engineers willing to be early and to own their pipeline. If your nightly job is “run a model over millions of files and write the results,” Daft is close to the obvious choice. Who shouldn’t: teams doing pure tabular SQL analytics at scale where Spark’s maturity, ecosystem, and stability matter more than multimodal ergonomics; teams that need foundation-governed, set-and-forget infrastructure; anyone trying to use it as a serving layer or database.

What you should now believe. Believe that lazy + streaming + AI-as-expression is a genuinely better architecture for this workload, not marketing — the design reasons are real and you can trace every behavior back to them. Don’t believe any single benchmark number, in either direction; run your own. When someone says “Daft is just Polars/Spark but for AI,” understand they’ve missed the point — the point is that the optimizer treats a model call like arithmetic, and that’s the thing nothing else does cleanly.

The hard-won line: in Daft, your job is to describe the pipeline honestly and put your filters early; the engine’s job is everything else — and the day you stop fighting it and start trusting the plan is the day it gets fast.

The ideas are mine. The writing is AI assisted

#python #daft

Related reading

intermediate

Spark Deep Intuition

An experienced engineer's guide to Spark

intermediate

Kafka Deep Intuition

An experienced engineer's guide to Kafka

intermediate

Narwhals Deep Intuition

An experienced engineer's guide to Narwhals

intermediate

Airflow Deep Intuition

An experienced engineer's guide to Airflow