DuckDB

How JetBrains Built Cross-Database Queries into DataGrip with DuckDB

2026-11-05T00:00:00+00:00

When JetBrains set out to let developers join data across different database connections in a single query, it chose not to build its own cross-database query engine. Instead, DataGrip, its database IDE, embeds DuckDB to execute those queries, and exposes DuckDB’s SQL dialect directly to the developers writing them.

Quote

DuckDB turned out to be the simplest option to integrate.

— Ilia Epelbaum, DataGrip, JetBrains

Querying across Data Sources

DataGrip is JetBrains’ IDE for working with databases and writing SQL, developed since 2015. The same database functionality is also available as a plugin inside the other JetBrains IDEs, so the feature reaches well beyond DataGrip itself.

For a long time, users had been asking for one thing in particular: the ability to run a single query across more than one data source. The most common request was to join data from different database connections in one query. DataGrip could already execute queries against each connection, but nothing tied the results together, so the team needed an engine that could.

Choosing DuckDB over Building Its Own

JetBrains explored several approaches. It considered Apache Calcite, handling the queries itself using PSI (Program Structure Interface, the internal code representation JetBrains uses across its IDEs), and DuckDB.

DuckDB won on ease of integration. “DuckDB turned out to be the simplest option to integrate,” says Ilia Epelbaum, who works on the DataGrip team. The deciding feature was the ability to register a custom table function: it gave JetBrains a straightforward way to connect DuckDB to the query-execution functionality DataGrip already had. DuckDB was also preferred over lighter alternatives such as H2 because it brought more capabilities, including query optimizations like projection pushdown.

Maxim Sobolevskiy, Head of Database Ecosystem at JetBrains, puts the result in concrete terms. “With DuckDB and its support for custom table functions, cross-database queries have finally become practical in DataGrip,” he says. “Users can join data from PostgreSQL, MySQL, MongoDB, or any other supported database in one query, without setting up extra infrastructure.” The embedded alternatives the team evaluated could not do this, because “extending them required either native code or a complicated implementation.”

Epelbaum came to the project knowing the territory. He had been a database developer working on Trino before moving into database tooling, and already knew how flexible DuckDB was. That background is part of why the team reached for it rather than writing and maintaining a cross-database query engine of its own.

DuckDB in Plain Sight

Developers know DuckDB is there. Because a cross-source join runs through a DuckDB table function, the query itself is written in DuckDB’s SQL dialect, and DataGrip tells developers so. “We highlight that it’s DuckDB,” Epelbaum says, so that someone writing a join across connections knows they are writing a DuckDB query.

JetBrains has gone further and supports DuckDB as a first-class dialect in its own right, not only as the engine behind cross-source joins. Epelbaum contributed syntax highlighting for DuckDB, including the syntax introduced in DuckDB 2.0.

An Initial Solution, Still Evolving

This is still a first version. “We’ve built the initial solution, and we want to see how users actually use it in practice,” Epelbaum says. Only after that does the team plan to layer on more of DuckDB’s optimizations.

Two directions are on the roadmap. The first is to take fuller advantage of DuckDB’s query optimizations. The second is to use DuckDB for file-based data sources, reading files the way it already reads across connections. Epelbaum also singled out DuckDB’s friendly SQL as something the team values in day-to-day use.

For now, DuckDB sits at the core of a feature that JetBrains users had wanted for years, letting them join across their databases without JetBrains having to become a query-engine vendor to make it happen.

About JetBrains and DataGrip

JetBrains builds professional developer tools, including IntelliJ IDEA, PyCharm, and the Kotlin language, that have served millions of programmers for more than 20 years. DataGrip, its database IDE, helps developers and data engineers query, explore, and manage multiple databases from a single environment.

Bringing Local Analytics to the Quant's Desktop

2026-11-03T00:00:00+00:00

Quote

I like the simplicity of just a small thing running and working. And then you can get more complex.

— Ryan Hamilton, founder, TimeStored

Fifteen Years of Heavyweight Infrastructure

For about fifteen years, Ryan Hamilton has built large market-data platforms for banks and hedge funds. Most of that work was in kdb+, integrated with real-time messaging systems such as Solace. He built much of the platform at UBS and almost all of it at Citi, contributed to Morgan Stanley's, and most recently worked at a hedge fund.

His users are typically teams of 10 to 15 quants. They are deep specialists in finance who spend anywhere from a tenth to more than half their time on technology. Ryan's job has been to store their data efficiently so they can get on with their work. Sometimes he leads a team to do that. Sometimes he builds it himself. Sometimes he talks quants out of building infrastructure on their own.

Problems That Didn't Need a Heavyweight Answer

Alongside that work, startups, smaller tech firms, and smaller funds kept asking Ryan the same question: what should we run this on?

For years he had no good answer. He tried other systems but found that their license was prohibitively expensive, their setup didn't offer an embedded mode of operation, or they were much more operationally involved than he wanted at the time. For some clients, the honest answer was that he would write something in C for their narrow use case.

A Different Starting Point

DuckDB was different. It was small, embedded, reliable, and analytically capable, and a team could start with it inside an application without standing up a server.

“I like the simplicity of just a small thing running and working,” Ryan says. “And then you can get more complex.”

That let him give smaller firms a straight recommendation. DuckDB might not scale to everything, but it would run well and serve them for a long time. For roughly 90% of use cases, he reasoned, it would carry a team for years. By the time they outgrew it, they would understand their workload well enough to choose what came next. In many cases, DuckDB has expanded quickly enough that they never needed to move.

QStudio: A Local Analytical Sandbox

The clearest illustration is QStudio, the analytics IDE Ryan released about 13 years ago. It was originally built for kdb+ and now connects to around 15 databases. He leaves database administration to tools like DBeaver and focuses QStudio on analytics and charting.

DuckDB was the first deep integration, and it changed how quants work with expensive queries. The workflow runs like this. An analyst runs a heavy query against a remote database. Some of these take up to 20 minutes. QStudio then saves the result locally as Parquet through DuckDB, with a click in the UI.

The next day, the analyst queries that saved result directly, without rerunning the original workload:

SELECT
    symbol,
    avg(price) AS avg_price,
    sum(volume) AS total_volume
FROM 'results/trades_2026-11-10.parquet'
GROUP BY ALL
ORDER BY total_volume DESC;

The result is a personal sandbox where quants can build on their own earlier work. It proved popular, and Ryan found its performance close to kdb+'s.

Because QStudio is a Java Swing desktop application deployed inside banks, the integration depends on DuckDB's Java client. Ryan's early version relied on inefficient workarounds to get appending working. Once a dedicated Java developer joined and rebuilt the Appender properly, every kink was ironed out.

“It just worked,” Ryan says, “which is the highest praise I would give any engineering thing.”

That experience became the basis of his DuckCon talk. His argument was that an open-source engine already beat a far more exotic and expensive stack in some areas. DuckDB has moved well ahead since then.

Pulse: DuckDB by Default

QStudio was never a significant direct revenue generator, but it got a very small company onto the approved-vendor lists of some very large firms, which is hard for small vendors to do. When Ryan's company, TimeStored, launched Pulse, a real-time HTML visualization tool that is now its main product, that path into large firms already existed.

Ryan reached for DuckDB again. Pulse uses it to snapshot and cache data. What began as an integration experiment had become a repeated TimeStored architecture pattern. DuckDB sits in the embedded storage and query layer, and increasingly supports AI-assisted data exploration. Excel exports and similar user-facing features stay in the application's UI. That is partly to keep styling intact, and partly because bank users often can't install extensions at all.

PeachQ: Back to Q

Ryan's newest project is PeachQ, an open-source implementation of the Q language. Ryan loves Q as a language but thinks DuckDB is ahead as a database, so PeachQ separates the two.

PeachQ uses the Rayforce in-memory engine and DuckDB-backed historical storage to deliver a modern, open-source alternative to kdb+, bringing together real-time processing, powerful analytics, and efficient storage in a single system. Ryan's recent work on C extensions points the same way, along with experiments with newer formats such as Vortex and custom table formats. He credits DuckDB's extension API with letting people add features cleanly, unlike the fragile workarounds he and others once built into closed systems. Often, he notes, DuckDB later builds those features itself.

Learn More

Ryan's DuckCon talk gives a short tour of this work.

You can explore QStudio, now open source, and try PeachQ at peachq.org.

We would like to thank Ryan for the conversation. You can follow his work on DuckDB, Q, and TimeStored at timestored.com. And if you are reading this, are using DuckDB, and would like to share your story, drop us a line at geertjan@ducklabs.com.

From Spreadsheets to DuckLake: A Data Engineer's Road to DuckDB

2026-10-29T00:00:00+00:00

Quote

There actually is something out there that's easy to use and doesn't require a bunch of infrastructure overhead.

— Matt Martin

From Copy-Paste to Code

Matt's career started in 2008 the way many data careers do: with spreadsheets. As a fresh graduate, he spent his days copying and pasting between Excel, Microsoft Access, and SQL Server. Assembling a single report could take three hours, and one mistake meant starting over.

“There's got to be an easier way,” he remembers thinking. His dad taught him how to write VBA macros, and that was the first step. Next, he got access to the SQL Server and Oracle databases behind those spreadsheets and started querying the source directly instead of other people's workbooks. Macros turned into scheduled stored procedures, and stored procedures turned into automated pipelines.

When he started, the team published its 20 to 30 daily reports by 10 AM. By the time he had automated the process, the reports were out by 6 AM, before anyone had touched anything.

Teradata, BigQuery, and the Spark Years

From there, Matt moved into big data. Teradata gave him his first taste of massively parallel processing, along with lessons about data skew and distributing work evenly across nodes. Around 2016, at The Home Depot, he moved to Google BigQuery. BigQuery did not support stored procedures at the time, so Matt picked up Python, then pandas and eventually Apache Spark.

Spark was powerful, but it was also heavy. Even local testing meant aligning JVM and Spark versions, deciphering stack traces that ran for hundreds of lines, and waiting three to five minutes for the cluster to warm up before processing a few hundred files. “A lot of battle scars,” as Matt puts it.

He is also skeptical that most teams need that machinery. Outside a handful of domains such as aerospace, fraud detection, and high-frequency trading, he rarely sees workloads that truly need to churn through petabytes at speed. More often, it is the classic meme: someone insists on real-time data, and it turns out to be for a monthly report.

Meeting DuckDB

Matt first came across DuckDB around 2022. At first, he was not sure what to make of it: it was not pandas, not Spark, and not SQLite, yet it promised SQL for transforming data. Then he installed it and read a CSV file with two lines of code:

import duckdb
duckdb.sql("FROM 'sales.csv'").show()

“There actually is something out there that's easy to use and doesn't require a bunch of infrastructure overhead,” he says. No JVM, no knobs to turn. From local files, he moved on to files on Amazon S3, over the s3:// protocol:

CREATE SECRET (TYPE s3, PROVIDER credential_chain);

SELECT count(*) AS num_rows
FROM 's3://bucket/exports/*.parquet';

And from there, to mixing CSV, Parquet, and Iceberg tables in a single query:

INSTALL iceberg;
LOAD iceberg;

SELECT
    o.order_id,
    c.customer_name,
    p.product_name
FROM iceberg_scan('s3://bucket/warehouse/orders') AS o
JOIN 's3://bucket/customers.parquet' AS c
    ON o.customer_id = c.customer_id
JOIN 'products.csv' AS p
    ON o.product_id = p.product_id;

Two things made DuckDB a natural fit. First, simplicity: after nearly two decades in data engineering, Matt values a frictionless experience above almost anything else, as long as the SLAs are met. Second, SQL: after 15 years of writing advanced SQL, he found that DuckDB supported practically all of the syntax he already knew.

He also uses DuckDB for his personal finances, analyzing CSV exports of his bank transactions locally, the kind of query that takes a minute to write:

SELECT
    date_trunc('month', transaction_date) AS month,
    category,
    sum(amount) AS total
FROM 'transactions_*.csv'
GROUP BY ALL
ORDER BY month, total;

He is experimenting with LLM-based assistants for this, but likes knowing that he does not depend on them: “If LLMs ever become too expensive, I could easily pick right back up with DuckDB and write it myself. It's like riding a bike at this point.”

Why DuckLake

As open table formats took off, Matt worked extensively with Apache Iceberg and Delta Lake. Both are good at managing large datasets and their version history. The trouble starts when the metadata grows. Iceberg writes several metadata files per transaction, including metadata JSON, manifest lists, and manifests. Over time, a query that used to plan in seconds can slow down significantly because thousands of metadata files need to be resolved first.

DuckLake takes a different approach: it stores all metadata in a SQL database. “Organizations have been doing SQL for over four decades and have gotten very, very good at optimizing queries,” Matt says. In his experience, catalog operations that take seconds with Iceberg resolve in under a millisecond with DuckLake, because they are queries against indexed tables.

What Matt likes most is how little setup it takes: standing up an Iceberg lakehouse involves a lot of tooling, while a DuckLake on object storage takes two statements, one to create a secret for the bucket and one to attach the lake:

CREATE SECRET (TYPE s3, PROVIDER credential_chain);

ATTACH 'ducklake:metadata.ducklake' AS my_lake (
    DATA_PATH 's3://bucket/prefix/'
);

If the DuckLake does not exist yet, DuckDB creates it and you can start creating tables right away. Matt has also written about streaming events straight into a DuckLake.

The same simplicity applies to existing Iceberg catalogs. Matt has written on his Substack about the Spark configuration of 20-plus lines he once needed to query an Iceberg table in AWS Glue, just to get the right JARs and the right AWS settings. With DuckDB's Iceberg extension, it takes three statements:

INSTALL iceberg;
LOAD iceberg;

ATTACH 'aws_account_id' AS glue_catalog (
    TYPE iceberg,
    ENDPOINT_TYPE glue
);

Looking Forward to DuckDB 2.0

Matt has been testing the DuckDB 2.0 alpha and has not run into any regressions so far. Several features stand out for him.

Dynamic ATTACH statements. While writing the book, Matt needed to attach AWS Glue catalogs in examples. Up to v1.5, the ATTACH path had to be a literal, so the account number had to be hard-coded in the script, as calling getenv raised an error.

Matt filed a feature request, and a pull request followed within two days. In v2.0, ATTACH accepts expressions, so account IDs, usernames, passwords, and connection strings can come from environment variables at runtime instead of living in the script:

ATTACH getenv('AWS_ACCOUNT_ID') AS glue_catalog (
    TYPE iceberg,
    ENDPOINT_TYPE glue
);

Asynchronous I/O. This is the feature Matt calls a game changer. In his benchmarks, published on his Substack, reading 1,000 and 10,000 files from Amazon S3 consistently took 67% to 70% less time with the v2.0 alpha.

Much of the time used to go into sequential requests to list and fetch files. In v2.0, DuckDB issues these in the background. No code changes are needed: a query like the following simply runs faster on v2.0.

SELECT count(*) AS num_events
FROM read_json('s3://bucket/logs/*.json');

For workloads such as log analytics over large numbers of raw files in S3, Google Cloud Storage, or Azure, where teams would traditionally spin up a wide Spark cluster, a single DuckDB process can now keep up, with less overhead and lower cost.

Quack. The Quack remote protocol turns a DuckDB instance into a server that other DuckDB instances can connect to over HTTP. Matt likes the cleaner connect and disconnect syntax in v2.0 and the official DuckLake support.

He has written about serving a DuckLake on an Amazon EC2 instance via Quack. End users only need the Quack address and a token, with no AWS credentials and no knowledge of the DuckLake behind it. He has also built a DuckDB-Wasm dashboard on top of this setup, which he covers in the DuckLake book he is writing. The Wasm build is still on v1.5 and will be updated for the v2.0 release.

Conclusion

Matt's journey does not suggest that distributed systems are unnecessary. It illustrates a different engineering principle: start with the requirements of the workload, and introduce additional infrastructure only when those requirements justify it. For Matt, DuckDB and DuckLake demonstrate how much analytical capability can be delivered without making the underlying system unnecessarily complicated.

We would like to thank Matt for the conversation. You can follow his writing on DuckDB and DuckLake on his Substack. And if you are reading this, are using DuckDB, and would like to share your story, drop us a line at quack@duckdb.org.

Gems of DuckDB 2.0.0

2026-10-26T00:00:00+00:00

Last week we released DuckDB 2.0.0 LTS, our biggest feature release so far. The announcement covered the headline items: the Quack protocol, asynchronous I/O, the first-class VARIANT type, agent mode, the PEG parser, and triggers. DuckDB 2.0.0 is now available in all primary clients – check the installation page for instructions on installing or upgrading.

A release of this size ships many smaller improvements that never make it into the original announcement. In this post, we highlight a few of these gems. To find the documentation for everything that is new in 2.0, see the “What's New in the Docs for 2.0” page.

Python Appender

Inserting rows one at a time is inefficient, and building up a giant INSERT statement is awkward. Therefore, the preferred solution for inserting large amounts of data has been the appender. However, it was only available in some clients (C/C++, Go, Java, etc.) and notably absent from the Python client. DuckDB 2.0.0's Python client ships an appender that collects rows and writes them together in a single operation:

import duckdb

con = duckdb.connect()
con.execute("CREATE TABLE duckdb_releases (version VARCHAR, codename VARCHAR);")

with con.appender("duckdb_releases") as appender:
    appender.append_row("1.4", "Andium")
    appender.append_rows([("1.5", "Variegata"), ("2.0", "Cyanoptera")])

print(con.execute("FROM duckdb_releases").fetchall())

[('1.4', 'Andium'), ('1.5', 'Variegata'), ('2.0', 'Cyanoptera')]

The with block flushes the buffered rows when it exits. For data that already lives in a DataFrame or Arrow table, INSERT INTO ... SELECT is still faster, but the appender is ideal for rows that your Python code produces on the fly.

Usability Improvements in the CLI

Pretty-Printed `.schema`

As of DuckDB 2.0, the .schema dot command pretty-prints and highlights its output by default, which makes reading the schema of a database much easier. If you prefer the earlier single-line form, for example, to copy a statement verbatim, use --no-indent (or its alias --no-format):

duckdb mydata.duckdb ".schema --no-indent"

`.manual` Dot Command

DuckDB 2.0 introduces the .manual dot command, which shows the signatures, description, and examples of a SQL function. For example, to get the details of the bar function, run:

.manual bar

  ────────────────────────────────────────────────────────────────────────────────────────────────
  system.main                                   bar                                scalar function

  SIGNATURE

        bar(x DOUBLE, min DOUBLE, max DOUBLE) -> VARCHAR

        bar(x DOUBLE, min DOUBLE, max DOUBLE, width DOUBLE) -> VARCHAR

  DESCRIPTION

        Draws a band whose width is proportional to (`x - min`) and equal to `width` characters
        when `x` = `max`. `width` defaults to 80.

  EXAMPLES

        bar(5, 0, 20, 10)

The same page is available from the command line via duckdb -manual bar, and the name may contain LIKE wildcards, e.g., .manual list_%. In the coming releases, we will extend the manuals to cover more features such as SQL statements and clauses.

Calculator

You can use the CLI as a calculator. Simply type the expression followed by a semicolon – no more typing into your browser's address bar!

6*7;

┌─────────┐
│ (6 * 7) │
│  int32  │
├─────────┤
│      42 │
└─────────┘

Command History

The CLI now exposes its command history in two ways. The .history dot command shows previously executed statements with syntax highlighting, optionally limited to the last N entries:

.history 10

The same information is available as a table through the shell_history() table function, so you can query your own history with SQL:

SELECT sql
FROM shell_history()
WHERE sql ILIKE '%copy%';

Offline Documentation

As a reminder, both the DuckDB documentation and the DuckDB blog are available for offline browsing:

Documentation as a Markdown file (approx. 5 MB)
Documentation as a PDF file (approx. 10 MB)
Blog as a Markdown file (approx. 3 MB)
Blog as a PDF file (approx. 35 MB)

You can download them to your e-reader (we are not affiliated with any particular brand!) or read them on your laptop on the go. You can also use them to supply context to LLMs and coding agents. At approximately 5 MB, the documentation exceeds the context window of most LLMs, so instead of ingesting it in full, point them to specific sections or let them find the parts they need.

Unsung SQL Features

`SHOW` for Configuration Settings

The SHOW statement can now print a configuration setting by name, PostgreSQL-style, when the name does not refer to a table:

SET threads = 42;
SHOW threads;

threads
42

This is much more succinct than the previous formulation that used the current_setting function (e.g., SELECT current_setting('threads') AS threads;).

When a table and a setting share a name, DuckDB prioritizes the table by default. You can control this with the show_behavior setting, which takes 'auto' (the default), 'table', or 'setting'.

Deprecated Using SHOW to describe a table, view, or query (e.g., SHOW my_table) is deprecated as of DuckDB 2.0 and prints a warning. Use DESCRIBE instead.

`CREATE SECURE VIEW` Statement

Regular views can leak information about their underlying tables through statistics and query plans, for example via the stats() function. The new SECURE VIEW does not:

CREATE SECURE VIEW users AS
    SELECT username
    FROM all_users
    WHERE access_level = 'standard';

On a secure view, stats() returns only uninformative statistics and EXPLAIN ANALYZE shows a single Secure View node instead of the view's query plan, so neither can be used to reconstruct the rows that the view filters out.

Options in `regexp_extract`

The regexp_extract function gained an overload that takes regular expression options as the third argument, with no capture group required. For example, you can now match case-insensitively in a single call:

SELECT regexp_extract('ABC', 'abc', 'i'); -- ABC

Setting a Sequence Value

Sequences gained a PostgreSQL-style setval function to reset a sequence to a given value, which is useful after a bulk load:

CREATE SEQUENCE serial START 1;
SELECT setval('serial', 42);
SELECT nextval('serial');

nextval
43

The next value after setval is value + increment by default. Pass false as the optional third argument (is_called) to make the next nextval return value itself.

Session Variables as Default Parameter Values

Prepared statements can now fall back to a session variable of the same name when a named parameter is not supplied at execution time:

SET VARIABLE multiplier = 10;
SET VARIABLE x = 42;
PREPARE compute AS SELECT $multiplier * $x AS result;
EXECUTE compute;

result
420

An explicit argument always takes precedence over the variable, and because the fallback is resolved at execution time, changing a variable and re-executing the statement picks up the new value.

Custom Partition Paths

When writing partitioned data, the PARTITION_BY clause writes each partition to a col=value subdirectory following the Hive convention. The new PARTITION_PATH option gives you full control over the subdirectory that each partition is written to, using an expression over the partitioning columns. Let's suppose you have a table named orders that stores customer orders.

Click here to see the code to create the orders table.

Announcing DuckDB 2.0.0 LTS “Cinnamon Teal”

2026-10-21T00:00:00+00:00

Update 2.0.0 packed so many features that we published a followup blog post titled “Gems of DuckDB 2.0.0”.

We are proud to release DuckDB 2.0.0, codenamed “Cinnamon Teal” after the Spatula cyanoptera, a strikingly reddish-brown duck found in the western Americas.

While DuckDB 1.0 was all about stability and backward compatibility, DuckDB 2.0 is above all a feature release. In this blog post, we summarize the highlights and then walk through the breaking changes and how to migrate. We showcased many of the new features in “A Preview of DuckDB v2.0”, which covers each one in more depth. For the complete release notes, see the release page on GitHub.

Tip To install the new version, please visit the installation page.

With this release, there are currently three DuckDB lines available. DuckDB 2.0 is a long-term support (LTS) release with support until at least October 2027. The previous LTS release, version 1.4, is still supported until November 17, 2026. To ensure a smooth transition and allow sufficient time for testing, we extended the support period of DuckDB 1.5 until March 2027, effectively also turning it into an LTS release. For details, see the release calendar.

To find your way around the documentation for all of the new features, we also published a “What's New in the Docs for 2.0” page that links each feature below to its documentation.

DuckDB as a Server

DuckDB has been an in-process database since day one. In DuckDB 1.5.3, released in May, we introduced the Quack protocol that lets you run DuckDB in a traditional client-server setup, along with the quack extension, which implements DuckDB's native protocol, so any DuckDB process can serve its databases.

Quack and `CONNECT`

DuckDB 2.0.0 ships a polished version of the Quack protocol with substantial performance and stability improvements. It also introduces the new CONNECT statement that allows attaching and switching to a remote database in a single command for better ergonomics:

DuckDB Server

CALL quack_serve(
    token = 'my_token'
);

CREATE TABLE events
    (date DATE, name VARCHAR);

INSERT INTO events VALUES
    ('2027-04-07',
     'DuckCon #8');

quack:

DuckDB Client

ATTACH 'quack:localhost'
    AS qk (TOKEN 'my_token');

CONNECT qk;
SELECT * FROM events;

┌────────────┬────────────┐
│    date    │    name    │
│    date    │  varchar   │
├────────────┼────────────┤
│ 2027-04-07 │ DuckCon #8 │
└────────────┴────────────┘

For simplicity, this example runs the server and the client on the same machine. By default, the server only listens on localhost. To accept connections from other machines, keep the server on localhost and put it behind a TLS-terminating reverse proxy. The allow_other_hostname option, which exposes the server directly, is only meant for testing.

The CONNECT statement is not limited to Quack: it points your session at any remote database that supports it, including PostgreSQL and MySQL. For these, the new remote pushdown optimizer ships queries directly to the remote server instead of pulling tables over the wire.

Quack in the CLI

The CLI also has shortcuts for setting up a Quack client-server pair: the new -serve and -connect arguments start a Quack server or connect to one directly from the command line:

CLI Server

duckdb -serve

CREATE TABLE t AS
      SELECT 42 AS x;

quack:

CLI Client

duckdb -connect

FROM t;

┌───────┐
│   x   │
│ int32 │
├───────┤
│    42 │
└───────┘

Performance

Asynchronous I/O

DuckDB 2.0 introduces asynchronous I/O for reading Parquet and CSV files, as well as DuckDB's native storage format. The I/O layer now scales independently from the query processing layer, issuing reads in the background and keeping multiple requests in flight. This means far more parallelism for remote reads and dramatically faster queries on object storage such as s3://.

We described the design in detail in a dedicated blog post. In that post, we showed the network throughput over the course of a query with and without asynchronous I/O:

Network throughput. Higher utilization and shorter runtime are better.

The `VARIANT` Type Becomes a First-Class Citizen

The VARIANT type stores semi-structured, schema-less data such as JSON while keeping the performance of columnar execution. So you can think of it as JSON on steroids: a VARIANT column can store differently shaped data in every row, but unlike JSON it is not a text format.

DuckDB 1.5 already introduced VARIANT, but DuckDB 2.0 fully integrates it. DuckDB 2.0 detects the common structure in your data and “shreds” it, so it compresses well and executes fast. The pipeline works end to end, from shredded execution straight out of storage to shredded VARIANT reading and writing for Parquet, backed by a family of variant_* functions. For example:

CREATE TABLE events (payload VARIANT);
INSERT INTO events
VALUES
    ('{"user": {"id": 42, "tags": ["a", "b"]}}'::JSON::VARIANT),
    ('{"event": "click", "x": 10, "y": 20}'::JSON::VARIANT);

SELECT variant_type(payload), variant_keys(payload)
FROM events;

┌───────────────────────┬───────────────────────┐
│ variant_type(payload) │ variant_keys(payload) │
│        varchar        │       varchar[]       │
├───────────────────────┼───────────────────────┤
│ OBJECT                │ [user]                │
│ OBJECT                │ [event, x, y]         │
└───────────────────────┴───────────────────────┘

Agent Mode in the CLI

Coding agents such as Claude Code, Codex, and Cursor increasingly drive the DuckDB CLI on their users' behalf. The 2.0 CLI detects when it is being run by an agent and switches to agent mode: it prints compact Markdown tables instead of padded boxes, says clearly when a result was truncated, stops runaway queries early, and reports errors as JSON.

Long-running SQL queries report their expected cost prior to running, so the agent can decide whether to wait. In our experiments on TPC-H, agent mode cut the amount of DuckDB output the model had to read by 59%. You can turn it on for manual testing using the -agent flag or instruct your agent to turn it off using the -no-agent flag.

Extensibility

A Brand New SQL Parser

DuckDB has always used a parser derived from PostgreSQL's. In 2.0, our own modern, extensible PEG-based parser, which debuted as an opt-in feature in DuckDB 1.5, becomes the default. Extensions can now hook into the grammar itself (an idea we first explored in our research on runtime-extensible parsers), so expect extensions that expose entirely new SQL syntax. Error messages now come with precise source locations. For details, see the PEG parser documentation. The new parser is designed to be compatible with the old one, apart from a few deliberate changes listed in the migration guide. If you notice any other differences, please file an issue. In the meantime, you can switch back to the old parser using the legacy_parser extension by running SET enable_legacy_parser = true;.

The old parser had served DuckDB since its first commit in 2018. Its grammar was generated with YACC/Bison, which produces an LALR(1) parser, so as DuckSQL grew, even small additions could collide with existing rules and raise shift/reduce or reduce/reduce conflicts. A PEG grammar avoids this by trying alternatives in the order they are written, so adding syntax no longer risks breaking unrelated rules. Python itself moved to a PEG-based parser in version 3.9 for the same flexibility.

Write Extensions Once and Host Them Yourself

DuckDB 2.0 ships a revamped C API v2, which is considerably more powerful than its predecessor and exposes components such as file systems and table functions to extensions. Before marking it as stable, we would like to test it in the field, so please give it a try and send us your feedback!

You can also register and host your own trusted extension repositories, so an organization can host and sign its own extensions and have them install and load just like the built-in ones:

SET allow_extension_repositories = 'allowed';

CREATE EXTENSION REPOSITORY my_repo
    WITH PREFIX 'https://extensions.example.org';

INSTALL AND LOAD my_ext FROM my_repo;

A repository is a name, a URL prefix, and one or more RSA public keys that are trusted to sign the extensions served from it. The prefix can point at anything DuckDB can read: a local path, https, s3, you name it. At CREATE time, DuckDB fetches the repository's public keys and pins them into the repository definition, printing each key's SHA-256 fingerprint so you can compare it against one published out of band.

This feature is still a work in progress and its syntax may change in future releases. For more details, see the Extension Repositories page.

Core Extension Updates

DuckDB's core extensions are also shipping several new features, fixes, and performance optimizations. These are too numerous to all list here, so we only highlight a few of them and will cover them in dedicated blog posts in the coming weeks:

We released DuckLake 1.1, including a new specification and reference implementation.
The Iceberg extension strives to become a feature-complete implementation. To this end, it adds REST catalog ATTACH options that derive a table's location from its namespace or load table metadata from the metadata-location file returned by the catalog. Further options control the URL prefix and the namespace separator.
The Delta extension can create Delta tables with CREATE TABLE (including partitioning and Delta table properties), supports time travel, and improves VARIANT support.

SQL Features

Triggers

DuckDB 2.0 delivers triggers: the CREATE TRIGGER statement with BEFORE and AFTER triggers, FOR EACH ROW and FOR EACH STATEMENT, transition tables via REFERENCING OLD/NEW TABLE, multiple triggers per event, RETURNING on triggered tables, and DROP TRIGGER. The classic use case is audit tables, where a trigger records what changed in response to a table modification. Some combinations, such as BEFORE ... FOR EACH ROW and UPDATE ... FOR EACH ROW triggers, are not yet supported; see the limitations.

CREATE TRIGGER trg_audit
    AFTER UPDATE ON target
    REFERENCING
        OLD TABLE AS o
        NEW TABLE AS n
    FOR EACH STATEMENT
        INSERT INTO audit
        SELECT n.id, o.val, n.val
        FROM o JOIN n ON o.id = n.id;

Triggers can also maintain materialized views incrementally! For example, to keep track of the average age of people per nationality in a table age_statistics, you can use two triggers – one for insert and another for delete operations – and store the sum and the count alongside the average.

To see more details, click here.

From JSON Strings to Typed Columns

2026-10-16T00:00:00+00:00

DuckDB's read_json turns JSON files into tables. It reads through part of the data to work out a schema, then gives you typed columns. However, a lot of JSON does not come as files. It comes as a text column: event payloads in a Postgres table, API responses stored by an ingestion job, a metadata column in a CSV export.

People regularly ask how to get the read_json treatment for such a column, and one workaround that comes up is to COPY the column to a file and read the file back.

You do not need the round trip. In this post, we get the same result directly in SQL.

The Data

Here is a small table of events, with the payload stored as VARCHAR:

CREATE TABLE events (id INTEGER, payload VARCHAR);
INSERT INTO events VALUES
    (1, '{"user": {"id": 17, "country": "NL"}, "action": "click", "ts": "2026-10-01 10:00:00"}'),
    (2, '{"user": {"id": 23, "country": "BE"}, "action": "view", "ts": "2026-10-01 10:00:05", "duration_ms": 1200}'),
    (3, '{"user": {"id": 17}, "action": "purchase", "ts": "2026-10-01 10:02:11", "items": [{"sku": "A-1", "qty": 2}, {"sku": "B-7", "qty": 1}]}');

Like most real payloads, the objects do not all have the same keys. Only the second event has a duration_ms, only the third has items, and the third has no country.

Picking Out a Few Fields

If you only need a handful of fields, the JSON extraction operators are enough. -> returns JSON and ->> returns VARCHAR. Both accept a key name or a JSON path:

SELECT
    id,
    payload ->> '$.user.country' AS country,
    payload ->> 'action' AS action,
    (payload ->> 'duration_ms')::INTEGER AS duration_ms
FROM events;

┌───────┬─────────┬──────────┬─────────────┐
│  id   │ country │  action  │ duration_ms │
│ int32 │ varchar │ varchar  │    int32    │
├───────┼─────────┼──────────┼─────────────┤
│     1 │ NL      │ click    │        NULL │
│     2 │ BE      │ view     │        1200 │
│     3 │ NULL    │ purchase │        NULL │
└───────┴─────────┴──────────┴─────────────┘

Missing keys come back as NULL, which is what you want for optional fields. However, this approach parses the JSON again for every extracted field, and it becomes verbose once you want more than a few columns.

Inferring a Structure from the Column

json_structure returns the structure of a single JSON value. Its aggregate variation, json_group_structure, merges the structures of all values in a column into one. It needs JSON input, so cast the VARCHAR column first:

SELECT json_group_structure(payload::JSON) AS structure
FROM events;

┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                      structure                                                                      │
│                                                                        json                                                                         │
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ {"user":{"id":"UBIGINT","country":"VARCHAR"},"action":"VARCHAR","ts":"VARCHAR","duration_ms":"UBIGINT","items":[{"sku":"VARCHAR","qty":"UBIGINT"}]} │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Every key that appears in any row is in there, including duration_ms and the array of items.

For a large table, you do not have to scan every row for this. Running it over a sample, for example FROM events USING SAMPLE 10_000 ROWS, is much cheaper, at the risk of missing keys that are very rare.

Adjusting the Types

The inferred structure needs a few fixes. ts came out as VARCHAR because JSON has no timestamp type, and the integers came out as UBIGINT, which is wider than needed.

Since the structure is itself a JSON document, you can override parts of it with json_merge_patch. Store the result in a variable so you can use it in the next query:

SET VARIABLE payload_structure = (
    SELECT json_merge_patch(
        json_group_structure(payload::JSON),
        '{
            "user": {"id": "INTEGER"},
            "ts": "TIMESTAMP",
            "duration_ms": "INTEGER",
            "items": [{"sku": "VARCHAR", "qty": "INTEGER"}]
        }'
    )
    FROM events
);

The patch only lists what you want to change. Keys you leave out, such as user.country and action, keep their inferred types.

Building the Table

json_transform (also available as from_json) converts a JSON value into a DuckDB STRUCT following the structure you give it. unnest then expands the top-level fields of that struct into columns:

CREATE TABLE events_typed AS
    SELECT id, unnest(json_transform(payload, getvariable('payload_structure')))
    FROM events;

DESCRIBE events_typed;

┌─────────────┬─────────────────────────────────────┬─────────┬─────────┬─────────┬─────────┐
│ column_name │             column_type             │  null   │   key   │ default │  extra  │
│   varchar   │               varchar               │ varchar │ varchar │ varchar │ varchar │
├─────────────┼─────────────────────────────────────┼─────────┼─────────┼─────────┼─────────┤
│ id          │ INTEGER                             │ YES     │ NULL    │ NULL    │ NULL    │
│ action      │ VARCHAR                             │ YES     │ NULL    │ NULL    │ NULL    │
│ user        │ STRUCT(country VARCHAR, id INTEGER) │ YES     │ NULL    │ NULL    │ NULL    │
│ ts          │ TIMESTAMP                           │ YES     │ NULL    │ NULL    │ NULL    │
│ duration_ms │ INTEGER                             │ YES     │ NULL    │ NULL    │ NULL    │
│ items       │ STRUCT(sku VARCHAR, qty INTEGER)[]  │ YES     │ NULL    │ NULL    │ NULL    │
└─────────────┴─────────────────────────────────────┴─────────┴─────────┴─────────┴─────────┘

The JSON is parsed once, at load time. From here on, the columns are regular DuckDB columns, stored and compressed by type, and queries on them do not touch JSON at all:

SELECT id, "user".id AS user_id, "user".country, action, ts
FROM events_typed;

┌───────┬─────────┬─────────┬──────────┬─────────────────────┐
│  id   │ user_id │ country │  action  │         ts          │
│ int32 │  int32  │ varchar │ varchar  │      timestamp      │
├───────┼─────────┼─────────┼──────────┼─────────────────────┤
│     1 │      17 │ NL      │ click    │ 2026-10-01 10:00:00 │
│     2 │      23 │ BE      │ view     │ 2026-10-01 10:00:05 │
│     3 │      17 │ NULL    │ purchase │ 2026-10-01 10:02:11 │
└───────┴─────────┴─────────┴──────────┴─────────────────────┘

Nested Structs and Name Collisions

Recursive unnesting with unnest(..., recursive := true) flattens nested structs in a single pass. By default it names each column after its innermost key, so the id inside user ends up as a column called id, right next to the event's own id:

SELECT id, unnest("user", recursive := true)
FROM events_typed
LIMIT 1;

┌───────┬─────────┬───────┐
│  id   │ country │  id   │
│ int32 │ varchar │ int32 │
├───────┼─────────┼───────┤
│     1 │ NL      │    17 │
└───────┴─────────┴───────┘

With two columns both named id, a later query cannot tell which one you mean. Pass keep_parent_names := true to prefix each column with the keys it came from, so names from different levels stay distinct. On the full payload, user.id and user.country no longer clash with the top-level columns, and you can select them by their prefixed names:

SELECT id, "user.id", "user.country"
FROM (
    SELECT id, unnest(json_transform(payload, getvariable('payload_structure')), recursive := true, keep_parent_names := true)
    FROM events
    LIMIT 1
);

┌───────┬─────────┬──────────────┐
│  id   │ user.id │ user.country │
│ int32 │  int32  │   varchar    │
├───────┼─────────┼──────────────┤
│     1 │      17 │ NL           │
└───────┴─────────┴──────────────┘

Arrays of Objects

In events_typed, the items array is stored as a STRUCT(sku VARCHAR, qty INTEGER)[], a list of structs, as the DESCRIBE output above shows. To get one row per item, unnest the list in the FROM clause:

SELECT id, u.item.sku, u.item.qty
FROM events_typed, unnest(items) AS u(item);

┌───────┬─────────┬───────┐
│  id   │   sku   │  qty  │
│ int32 │ varchar │ int32 │
├───────┼─────────┼───────┤
│     3 │ A-1     │     2 │
│     3 │ B-7     │     1 │
└───────┴─────────┴───────┘

Events without items do not appear here. If you need to keep them, write the unnest as a lateral join: FROM events_typed LEFT JOIN unnest(items) AS u(item) ON true.

Values That Do Not Fit

By default, json_transform does not raise an error on a value it cannot convert. The value becomes NULL instead:

SELECT json_transform('{"qty": "two"}', '{"qty": "INTEGER"}') AS result;

┌─────────────────────┐
│       result        │
│ struct(qty integer) │
├─────────────────────┤
│ {'qty': NULL}       │
└─────────────────────┘

That is convenient for messy data, but it can hide problems. json_transform_strict raises an error instead:

SELECT json_transform_strict('{"qty": "two"}', '{"qty": "INTEGER"}') AS result;

Invalid Input Error: Failed to cast value to numerical: "two"

Use the strict version while you are working out the structure, and switch to the lenient version once you know which oddities you are willing to accept.

Should the Column Be `VARCHAR` or `JSON`?

This question comes up regularly in the DuckDB discussions. The JSON type is stored exactly like a VARCHAR, so there is no difference in size or scan speed. What it adds is validation on the way in, since an invalid document fails the insert, and the guarantee that the extraction functions do not have to deal with malformed input.

If you will query fields from the column many times, the type of the raw column matters much less than whether you extract the fields at all. Parsing JSON at query time is work that has to be repeated for every query. Turning it into typed columns once, as shown above, does that work a single time.

Benchmarking DuckDB 2.0-dev on NVIDIA Vera

2026-10-16T00:00:00+00:00

We love running benchmarks on all sorts of new hardware. This often includes somewhat exotic hardware – where we sometimes need to compile a new DuckDB binary. This time, in collaboration with the NVIDIA team, we got our hands on their new Vera chip, aimed at agentic workloads.

NVIDIA Vera comes with 88 “Olympus” cores, Spatial Multithreading and up to 1.2 terabytes per second (TB/s) of memory bandwidth. What's more, Vera runs arm64 binaries – so it runs DuckDB's regular build without any modifications.

To see how this new hardware performs, we ran a benchmark derived from TPC-H that included both the power test and the throughput test on scale factor 1,000. As an x86 baseline, we also ran the benchmark on an Intel Xeon 6 machine.

Note This workload is derived from the TPC-H Benchmark and is not comparable to published TPC-H Benchmark results, as this implementation does not comply with all requirements of the TPC-H Benchmark.

Environment

System Setup

We used two bare-metal machines running Ubuntu 26.04:

an NVIDIA Vera server with 1536 GB of memory, and
an x86 baseline, an m8id.metal-48xl EC2 instance with an Intel Xeon 6975P-C CPU and 768 GiB of memory.

On both machines, we stored the database on a single NVMe SSD with 3+ TB of storage, formatted as ext4. At the time of writing, DuckDB 2.0 was not released yet, so we used a preview build, 1.6.0.dev379 (a codename for 2.0-dev). As a software baseline, we also ran the benchmark with DuckDB 1.5.5.

To run the experiments, we used our duckdb-tpch benchmark harness. To re-create our benchmark environment, see our script below.

Click here to reveal the setup script.

What Is inside an Iceberg Table?

2026-10-13T00:00:00+00:00

An Apache Iceberg table looks like a regular table when you query it, but underneath it is a small tree of files: one JSON file at the top, a few layers of Avro files in the middle, and Parquet files at the bottom. When something is slow or looks wrong, it helps to know what is in that tree.

DuckDB can read all three formats, so you can inspect every layer with plain SQL. In this post, we create a small Iceberg table, change it a few times, and then walk through the files from the top down.

Getting Started

You need DuckDB with the iceberg extension and an Iceberg REST catalog to attach to. The catalogs page has instructions for catalogs such as Apache Polaris, Lakekeeper, and Amazon S3 Tables.

A catalog is needed because the files on their own cannot tell you which version of the table is current. An Iceberg table writes a new metadata.json on every change and never overwrites the old one. The catalog holds the one piece of mutable state: a pointer to the current metadata file. When you query the table, DuckDB asks the catalog for that pointer first, then follows the tree of files down from there.

For this post, we run a catalog locally with the Iceberg REST catalog fixture, a small reference catalog that the Iceberg project publishes as a Docker image. It is the quickest catalog to set up: it runs from a single image, needs no Cloud account and no authentication, and keeps everything on local disk where you can open the files yourself.

We start it like this:

docker run -p 8181:8181 \
    -v /data/lake:/data/lake \
    -e CATALOG_WAREHOUSE=/data/lake \
    apache/iceberg-rest-fixture

Each part of that command maps to something DuckDB relies on later:

-p 8181:8181 publishes the catalog's REST API on localhost:8181, the address DuckDB connects to.
-v /data/lake:/data/lake mounts a host directory into the container, so the metadata, manifest, and Parquet files survive after the container stops and you can read them directly.
-e CATALOG_WAREHOUSE=/data/lake tells the catalog where to write those files, which is why every path in the output below begins with /data/lake.

Note Run the examples below in the local DuckDB CLI, not in the browser-based DuckDB Shell, because a page served from the web cannot reach a catalog on your own localhost.

With the fixture running, attach it like this:

ATTACH 'warehouse' AS lake (
    TYPE iceberg,
    ENDPOINT 'http://localhost:8181',
    AUTHORIZATION_TYPE 'none'
);

This opens a catalog connection named lake that you query like any attached database.

TYPE iceberg tells DuckDB to speak the Iceberg REST protocol.
ENDPOINT is the address published by the -p flag above.
AUTHORIZATION_TYPE 'none' matches the fixture, which expects no credentials.
The first argument is the warehouse name. The fixture serves a single warehouse, so any name works.
With object storage, the paths in the output below start with s3:// instead of /data/lake, but everything else is the same.

A Table with History

We create a table of train services, partitioned by date, and change it three times: an insert of three days of data, an insert of a fourth day, and a delete.

CREATE SCHEMA lake.demo;
CREATE TABLE lake.demo.services (
    service_id BIGINT,
    service_date DATE,
    station VARCHAR,
    delay_min INTEGER
)
PARTITIONED BY (service_date);

INSERT INTO lake.demo.services
    SELECT
        i,
        DATE '2026-10-01' + (i % 3)::INTEGER,
        ['Utrecht Centraal', 'Delft', 'Leiden Centraal'][1 + (i % 3)::INTEGER],
        (i % 9)::INTEGER
    FROM range(1, 9_001) t(i);

INSERT INTO lake.demo.services
    SELECT i, DATE '2026-10-04', 'Den Haag HS', (i % 5)::INTEGER
    FROM range(9_001, 12_001) t(i);

DELETE FROM lake.demo.services
WHERE service_date = DATE '2026-10-02' AND delay_min = 4;

Each of the two inserts and the delete commits on its own. A write does not change the table in place: DuckDB reads the current snapshot, writes new data and metadata files alongside the old ones, and then asks the catalog to move the table pointer from that snapshot to the new one. That move is the commit.

The commit names the snapshot it expects to replace, so two writers that start from the same snapshot cannot both succeed. The first commit moves the pointer, and the second is rejected, because the snapshot it meant to replace is no longer current. The rejected write then retries on top of the new snapshot. When run in sequence from a single session, the three statements here never conflict, because each one starts from the snapshot the previous one committed.

The table now has 11,000 rows: the first insert added 9,000, the second added 3,000, and the delete removed the 1,000 rows of October 2 that had a delay of 4 minutes. Each of the three statements created a new snapshot, which iceberg_snapshots lists as follows:

SELECT sequence_number, snapshot_id, operation
FROM iceberg_snapshots(lake.demo.services)
ORDER BY sequence_number;

┌─────────────────┬─────────────────────┬───────────┐
│ sequence_number │     snapshot_id     │ operation │
│     uint64      │       uint64        │  varchar  │
├─────────────────┼─────────────────────┼───────────┤
│               1 │ 7523747767380427861 │ append    │
│               2 │ 3062515218093523051 │ append    │
│               3 │ 6456433563311493009 │ delete    │
└─────────────────┴─────────────────────┴───────────┘

The Tree

Before going through the layers one by one, here is the whole structure. Each layer points to the files in the layer below it:

metadata.json                    one per table version
└── manifest list (Avro)         one per snapshot
    └── manifests (Avro)         one per write (for simplicity)
        ├── data files (Parquet)
        └── delete files (Parquet)

All of these files are immutable. Every change writes new files and a new metadata.json, then commits by moving the catalog pointer to it.

Layer 1: The Metadata File

The catalog stores a single pointer per table: the location of the current metadata file. iceberg_load_table_response returns what the catalog sent back when DuckDB loaded the table, including that location:

SELECT metadata_location
FROM iceberg_load_table_response(lake.demo.services);

┌────────────────────────────────────────────────────────────────────────────────────────────┐
│                                     metadata_location                                      │
│                                          varchar                                           │
├────────────────────────────────────────────────────────────────────────────────────────────┤
│ /data/lake/demo/services/metadata/00003-b79b5ffd-f118-44fd-9d8e-3551c147ef3b.metadata.json │
└────────────────────────────────────────────────────────────────────────────────────────────┘

The file name starts with 00003 because this is the fourth version of the metadata, counting from 00000 when the table was created.

It is a JSON document, so read_json can read it. We store the location in a variable to keep the queries short:

SET VARIABLE metadata_file = (
    SELECT metadata_location FROM iceberg_load_table_response(lake.demo.services)
);

SELECT
    "format-version",
    "current-snapshot-id",
    len(snapshots) AS snapshots,
    len("metadata-log") AS previous_metadata_files
FROM read_json(getvariable('metadata_file'));

┌────────────────┬─────────────────────┬───────────┬─────────────────────────┐
│ format-version │ current-snapshot-id │ snapshots │ previous_metadata_files │
│     int64      │        int64        │   int64   │          int64          │
├────────────────┼─────────────────────┼───────────┼─────────────────────────┤
│              2 │ 6456433563311493009 │         3 │                       3 │
└────────────────┴─────────────────────┴───────────┴─────────────────────────┘

Besides the schema and partition spec, the metadata file holds the list of all snapshots that are still around, and each snapshot has a summary with counters. These counters are a cheap way to see what each change did, without reading any data:

SELECT
    s."sequence-number" AS seq,
    s.summary.operation,
    s.summary."added-records",
    s.summary."added-position-deletes",
    s.summary."total-records",
    s.summary."total-data-files",
    s.summary."total-delete-files"
FROM (SELECT unnest(snapshots) AS s FROM read_json(getvariable('metadata_file')))
ORDER BY seq;

┌───────┬───────────┬───────────────┬────────────────────────┬───────────────┬──────────────────┬────────────────────┐
│  seq  │ operation │ added-records │ added-position-deletes │ total-records │ total-data-files │ total-delete-files │
│ int64 │  varchar  │    varchar    │        varchar         │    varchar    │     varchar      │      varchar       │
├───────┼───────────┼───────────────┼────────────────────────┼───────────────┼──────────────────┼────────────────────┤
│     1 │ append    │ 9000          │ NULL                   │ 9000          │ 3                │ 0                  │
│     2 │ append    │ 3000          │ NULL                   │ 12000         │ 4                │ 0                  │
│     3 │ delete    │ NULL          │ 1000                   │ 12000         │ 4                │ 1                  │
└───────┴───────────┴───────────────┴────────────────────────┴───────────────┴──────────────────┴────────────────────┘

In the last row, after the delete, total-records is still 12,000 and there are still four data files. The delete did not touch the data files. It added one delete file with 1,000 entries. We will get to that file in a moment.

Note The Iceberg spec treats these summary fields as optional, so an engine is free to leave any of them out. The counters above are populated by DuckDB, but a table written by another engine may be missing some of them, in which case the field reads as NULL. Treat them as a convenient hint rather than a guarantee.

Tip The metadata file grows with every commit, because it keeps the snapshot list and a log of previous metadata files. On a table that receives thousands of small commits, it can become large enough to slow down loading the table. Expiring old snapshots, which most catalogs and engines can do, keeps it in check.

Layer 2: The Manifest List

Each snapshot points to a manifest list, an Avro file that lists the manifests making up that snapshot. The avro extension reads it with read_avro. We take the manifest list of the latest snapshot:

SET VARIABLE manifest_list = (
    SELECT manifest_list
    FROM iceberg_snapshots(lake.demo.services)
    ORDER BY sequence_number DESC
    LIMIT 1
);

SELECT
    regexp_replace(manifest_path, '.*/metadata/', '') AS manifest,
    content,
    sequence_number,
    added_files_count,
    added_rows_count
FROM read_avro(getvariable('manifest_list'));

┌──────────────────────────────────────────────┬─────────┬─────────────────┬───────────────────┬──────────────────┐
│                   manifest                   │ content │ sequence_number │ added_files_count │ added_rows_count │
│                   varchar                    │  int32  │      int64      │       int32       │      int64       │
├──────────────────────────────────────────────┼─────────┼─────────────────┼───────────────────┼──────────────────┤
│ ffbf7076-8d36-46f0-aaa2-76e7d95442f0-m0.avro │       0 │               1 │                 3 │             9000 │
│ 74fa5b1f-0908-4e34-964a-504d946671de-m0.avro │       0 │               2 │                 1 │             3000 │
│ 5282235d-42b4-4b5d-a190-147a867994b7-m0.avro │       1 │               3 │                 1 │             1000 │
└──────────────────────────────────────────────┴─────────┴─────────────────┴───────────────────┴──────────────────┘

We created one manifest with every write. content is 0 for manifests that track data files and 1 for manifests that track delete files. The manifest list also stores the range of partition values in each manifest (not shown here), so a reader can skip entire manifests when a query filters on the partition column.

The latest snapshot still includes the manifests written by the two earlier snapshots. To make a commit cheap to write, an engine can add a new manifest for every write, as happened here. This comes at the cost of read performance, because of the rapidly accumulating small manifests. An alternative method merges new additions into an existing manifest, which is more expensive for the writer but more optimized for the reader.

Layer 3: The Manifests

The manifests list the actual files, one entry per file, with the partition values, the row count, and per-column statistics. They are Avro too:

SELECT
    status,
    data_file.content,
    regexp_replace(data_file.file_path, '.*/data/', '') AS file,
    data_file.record_count,
    data_file.partition
FROM read_avro('/data/lake/demo/services/metadata/*-m*.avro')
ORDER BY file;

┌────────┬─────────┬──────────────────────────────────────────────────────────────────────────────┬──────────────┬─────────────────────────────────────────┐
│ status │ content │                                     file                                     │ record_count │                partition                │
│ int32  │  int32  │                                   varchar                                    │    int64     │  struct(identity_service_date_2 date)   │
├────────┼─────────┼──────────────────────────────────────────────────────────────────────────────┼──────────────┼─────────────────────────────────────────┤
│      1 │       0 │ service_date=2026-10-01/01a12bb2-ad80-7cf4-8f63-beee9619f7c0.parquet         │         3000 │ {'identity_service_date_2': 2026-10-01} │
│      1 │       0 │ service_date=2026-10-02/01a12bb2-ad7f-7690-bda8-690b622b119a.parquet         │         3000 │ {'identity_service_date_2': 2026-10-02} │
│      1 │       1 │ service_date=2026-10-02/58a752c7-9925-4add-99ca-785505e17589-deletes.parquet │         1000 │ {'identity_service_date_2': 2026-10-02} │
│      1 │       0 │ service_date=2026-10-03/01a12bb2-ad80-71f7-b4cd-58684f05b129.parquet         │         3000 │ {'identity_service_date_2': 2026-10-03} │
│      1 │       0 │ service_date=2026-10-04/01a12bb2-adbe-790f-bde9-50db3acba4b2.parquet         │         3000 │ {'identity_service_date_2': 2026-10-04} │
└────────┴─────────┴──────────────────────────────────────────────────────────────────────────────┴──────────────┴─────────────────────────────────────────┘

status = 1 means the file was added in the snapshot that wrote the manifest. content is 0 for a data file and 1 for a position delete file. The partition value is stored per file, which is what lets a reader skip files without opening them.

The extension has a function that does this for you. iceberg_metadata follows the current snapshot through the manifest list and the manifests and returns one row per file:

SELECT
    manifest_content,
    regexp_replace(file_path, '.*/data/', '') AS file,
    record_count
FROM iceberg_metadata(lake.demo.services)
ORDER BY file;

┌──────────────────┬──────────────────────────────────────────────────────────────────────────────┬──────────────┐
│ manifest_content │                                     file                                     │ record_count │
│     varchar      │                                   varchar                                    │    int64     │
├──────────────────┼──────────────────────────────────────────────────────────────────────────────┼──────────────┤
│ DATA             │ service_date=2026-10-01/01a12bb2-ad80-7cf4-8f63-beee9619f7c0.parquet         │         3000 │
│ DATA             │ service_date=2026-10-02/01a12bb2-ad7f-7690-bda8-690b622b119a.parquet         │         3000 │
│ DELETE           │ service_date=2026-10-02/58a752c7-9925-4add-99ca-785505e17589-deletes.parquet │         1000 │
│ DATA             │ service_date=2026-10-03/01a12bb2-ad80-71f7-b4cd-58684f05b129.parquet         │         3000 │
│ DATA             │ service_date=2026-10-04/01a12bb2-adbe-790f-bde9-50db3acba4b2.parquet         │         3000 │
└──────────────────┴──────────────────────────────────────────────────────────────────────────────┴──────────────┘

Reading the Avro files directly is useful when you want to see fields that the function does not return, or when you are looking at an older snapshot or a table that is not in a catalog.

Column Statistics

The manifests also store, for every data file, the lower and upper bound of each column. iceberg_column_stats returns them in a readable form:

SELECT
    column_name,
    regexp_replace(file_path, '.*/data/', '') AS file,
    lower_bound,
    upper_bound
FROM iceberg_column_stats(lake.demo.services)
WHERE column_name = 'service_id'
ORDER BY file;

┌─────────────┬──────────────────────────────────────────────────────────────────────────────┬─────────────┬─────────────┐
│ column_name │                                     file                                     │ lower_bound │ upper_bound │
│   varchar   │                                   varchar                                    │   varchar   │   varchar   │
├─────────────┼──────────────────────────────────────────────────────────────────────────────┼─────────────┼─────────────┤
│ service_id  │ service_date=2026-10-01/01a12bb2-ad80-7cf4-8f63-beee9619f7c0.parquet         │ 3           │ 9000        │
│ service_id  │ service_date=2026-10-02/01a12bb2-ad7f-7690-bda8-690b622b119a.parquet         │ 1           │ 8998        │
│ service_id  │ service_date=2026-10-02/58a752c7-9925-4add-99ca-785505e17589-deletes.parquet │ NULL        │ NULL        │
│ service_id  │ service_date=2026-10-03/01a12bb2-ad80-71f7-b4cd-58684f05b129.parquet         │ 2           │ 8999        │
│ service_id  │ service_date=2026-10-04/01a12bb2-adbe-790f-bde9-50db3acba4b2.parquet         │ 9001        │ 12000       │
└─────────────┴──────────────────────────────────────────────────────────────────────────────┴─────────────┴─────────────┘

These bounds are what DuckDB uses to skip files when you filter on a column that is not the partition column. A query with WHERE service_id > 10000 only has to open the file for October 4, because the bounds of the other three files rule them out.

The ranges for the first three days overlap almost completely, because the rows were spread over the days round-robin. Data that is written in the order of a commonly filtered column gets narrower bounds and more skipping.

Partition Statistics

The bounds above are per column, per data file. For the partition column there is a coarser view, one row per manifest, that the manifest list keeps so a reader can skip an entire manifest before opening any of its files. iceberg_partition_stats returns it:

SELECT
    partition_field_name,
    partition_field_transform,
    lower_bound,
    upper_bound
FROM iceberg_partition_stats(lake.demo.services)
ORDER BY lower_bound;

┌─────────────────────────┬───────────────────────────┬─────────────┬─────────────┐
│  partition_field_name   │ partition_field_transform │ lower_bound │ upper_bound │
│         varchar         │          varchar          │   varchar   │   varchar   │
├─────────────────────────┼───────────────────────────┼─────────────┼─────────────┤
│ identity_service_date_2 │ identity                  │ 2026-10-01  │ 2026-10-03  │
│ identity_service_date_2 │ identity                  │ 2026-10-02  │ 2026-10-02  │
│ identity_service_date_2 │ identity                  │ 2026-10-04  │ 2026-10-04  │
└─────────────────────────┴───────────────────────────┴─────────────┴─────────────┘

There is one row per manifest, so the bounds trace the three writes: the first insert spread three days across one manifest, the second insert added October 4, and the delete touched only October 2. The partition_field_transform is identity because we partitioned on the date column directly. A query with WHERE service_date = DATE '2026-10-04' can rule out the first and third manifests on these bounds alone, without reading a single manifest entry.

Layer 4: Data Files and Delete Files

At the bottom of the tree are Parquet files. The data files are regular Parquet that any reader can open. The delete file works differently. It is a position delete file, and it lists rows to ignore by file and row position:

SELECT regexp_replace(file_path, '.*/data/', '') AS data_file, pos
FROM read_parquet(
    '/data/lake/demo/services/data/*/*-deletes.parquet',
    hive_partitioning = false
)
ORDER BY pos
LIMIT 5;

┌──────────────────────────────────────────────────────────────────────┬───────┐
│                              data_file                               │  pos  │
│                               varchar                                │ int64 │
├──────────────────────────────────────────────────────────────────────┼───────┤
│ service_date=2026-10-02/01a12bb2-ad7f-7690-bda8-690b622b119a.parquet │     1 │
│ service_date=2026-10-02/01a12bb2-ad7f-7690-bda8-690b622b119a.parquet │     4 │
│ service_date=2026-10-02/01a12bb2-ad7f-7690-bda8-690b622b119a.parquet │     7 │
│ service_date=2026-10-02/01a12bb2-ad7f-7690-bda8-690b622b119a.parquet │    10 │
│ service_date=2026-10-02/01a12bb2-ad7f-7690-bda8-690b622b119a.parquet │    13 │
└──────────────────────────────────────────────────────────────────────┴───────┘

This is how the extension implements DELETE and UPDATE: it leaves the data file alone and records the positions of the deleted rows, a strategy called merge-on-read.

Every reader then has to combine the two files to get the correct rows. Writes are cheap, and reads pay a small cost, which grows with the number of delete files. We set hive_partitioning = false because the service_date=... directory would otherwise be added as an extra column.

For tables with format-version 3, the same information is stored as a compact binary deletion vector in a Puffin file instead of a Parquet file.

Why Look at This at All?

Most of the time, you do not need to. FROM lake.demo.services returns the right 11,000 rows, and the extension handles the tree. It becomes useful when something does not behave as you expect:

A query reads more files than you expected. Compare the partition values and column bounds in the manifests with your filter.
A table gets slower over time. Count the manifests and delete files. Many small commits and many deletes both add files that every query has to process.
Row counts look off. The snapshot summaries show how many rows each commit added and deleted.
Another engine wrote the table. The manifests show what it wrote, including properties that DuckDB does not use.

Since every layer is a file in a format DuckDB already reads, you can answer these questions with the same tool you use to query the data.

Agent Mode in the DuckDB CLI

2026-10-09T00:00:00+00:00

Let's Ask Claude

We asked Claude to measure the effect of agent mode. In the experiment, Claude Code answered the 22 TPC-H questions on a scale factor 100 dataset. The questions were phrased in plain English, without any SQL. Each question was answered three times with agent mode and three times with -no-agent. All 132 runs produced the correct answer. With agent mode, the DuckDB output the model read went from 123.6k to 50.8k tokens, a 59% reduction. Claude summarized the result as follows:

Quote Agent mode fixes the things about a terminal shell that quietly mislead a model. When I answered the 22 TPC-H questions through the DuckDB CLI, agent mode cut the tokens I read back by 60 percent, and every answer was still right.

– Claude Fable 5.1

The savings did not show up in the total cost or runtime. Each turn re-reads the agent’s system prompt, which is much larger than the query output, so the saved tokens amount to about 0.5% of the total input. The agent mode runs also took slightly more turns (237 vs. 224). Claude wrote up the setup, the full results, and a follow-up experiment in a report.

Background

The DuckDB CLI was built for a person sitting at a terminal. Its default renderer pads columns so they line up, draws borders around the result, and prints the column types under the names:

┌────────────┬───────┬────────┬───────────────┐
│    name    │   n   │ total  │  avg_amount   │
│  varchar   │ int64 │ int128 │ decimal(10,2) │
├────────────┼───────┼────────┼───────────────┤
│ customer_0 │     3 │    777 │        259.00 │
│ customer_1 │     3 │    888 │        296.00 │
└────────────┴───────┴────────┴───────────────┘

For a large result, it shows the first and last 20 rows with a row of dots in the middle:

┌───────────────┬───────┬────────┬───────────────┐
│     name      │   n   │ total  │  avg_amount   │
│    varchar    │ int64 │ int128 │ decimal(10,2) │
├───────────────┼───────┼────────┼───────────────┤
│ customer_0    │     3 │    777 │        259.00 │
│ customer_1    │     3 │    888 │        296.00 │
│       ·       │   ·   │   ·    │       ·       │
│       ·       │   ·   │   ·    │       ·       │
│ customer_4998 │     4 │   1203 │        300.75 │
│ customer_4999 │     2 │    510 │        255.00 │
├───────────────┴───────┴────────┴───────────────┤
│ 5000 rows (40 shown)                4 columns  │
└────────────────────────────────────────────────┘

A person sees the dots and the (40 shown) footer and knows that rows are missing.

Today, many CLI invocations do not come from a person at all. Coding agents such as Claude Code, Codex, Cursor, Gemini CLI, and GitHub Copilot run duckdb -c "..." once per shell call, capture stdout through a pipe, and pass the text to a language model. This has quickly become a common way to use DuckDB.

This reader differs from a person in a few ways:

It has no terminal, so there is no scrolling and no pager.
It cannot ask for more. Whatever was printed is all it sees.
It pays for every token it reads, including padding and border characters.
It handles errors by pattern matching on text.

For this reader, the box renderer is a poor fit. The alignment padding costs tokens and carries no information for a model. A truncated result also looks a lot like a complete one. The footer says (40 shown), but a model can skim past that line and then draw conclusions from data it never saw.

Agent mode changes the CLI’s defaults for this reader. This post describes how the CLI detects an agent, what changes in the output, and how to try it.

New Agent mode shipped with DuckDB 2.0.

Detecting an Agent

Agents mark the commands they run with environment variables. There is no single convention for this yet, so the shell checks a list of them:

Variable	Set by
`AI_AGENT`, `AGENT`	Claude Code, Goose, Amp
`CLAUDECODE`	Claude Code
`CODEX_CI`, `CODEX_SANDBOX`, `CODEX_THREAD_ID`	Codex
`CURSOR_AGENT`	Cursor
`GEMINI_CLI`	Gemini CLI
`COPILOT_AGENT`, `COPILOT_CLI`, `COPILOT_AGENT_SESSION_ID`	GitHub Copilot

Agent mode turns on when three things are true: one of these variables is set, stdout is not a terminal, and no output format was given on the command line. If you pass -csv, -json, or -markdown, you asked for a specific format, and agent mode stays out of the way.

Two flags override detection. -agent forces the mode on and -no-agent forces it off. The two combine with format flags, so duckdb -agent -csv keeps the agent behavior and only changes how the rows are printed. The DUCKDB_AGENT_MODE environment variable does the same from the environment, for when the command line cannot be changed, such as a script run from an agent’s shell. For example, to manually enable agent mode, run:

DUCKDB_AGENT_MODE=1 duckdb

The .show command reports whether the mode is active and which agent was detected:

agent: claude-code

Not every agent sets a variable. For those, a run that fails while writing to a pipe prints one hint line on stderr, as long as no agent was identified and neither flag was given:

hint: -agent renders errors as JSON and results compactly for AI coding agents (duckdb -help lists all options)

An agent that nobody told about the mode can find it this way the first time something goes wrong.

What Changes in Agent Mode

Everything below is a default. Explicit settings such as .mode, .maxrows, EXPLAIN (FORMAT ...), or anything in your .duckdbrc still take precedence.

Compact Tables

Results are printed as Markdown tables without alignment padding, with the column type in the header cell:

| name:VARCHAR | n:BIGINT | total:HUGEINT | avg_amount:DECIMAL(10,2) |
|---|---|---|---|
| customer_0 | 3 | 777 | 259.00 |
| customer_1 | 3 | 888 | 296.00 |

A model reads the | separators and does not need the columns lined up. Since this is still valid Markdown, a person reading the agent’s transcript gets a rendered table. On typical results this format is 25 to 65% smaller than the box output, with the largest savings on wide results.

Large Results

Printing every row would avoid silent truncation, but a million rows is 17 MB on the pipe. Agent mode prints a result of up to 1,000 rows and 10,000 bytes in full. Anything larger becomes a sample of the first and last 20 rows around an explicit marker row, with a footer that says what happened:

| 19 | customer_19 | 13.3 |
| … 4960 rows omitted … |
| 4980 | customer_30 | 86.0 |
...
first 20 and last 20 of 5000 rows (.maxbytes 0 for all), hash 7389c2f5ff3cefb6

The footer also tells the reader how to get the rest. The limits can be changed with .maxrows N and .maxbytes N, where -1 or 0 means no limit. Long cell values are cut at 500 characters with a visible marker such as …(+4500 chars), adjustable with .maxcellwidth N.

Stopping Early

The compact table needs no column widths, so the result can be streamed. Once the cap is reached, the remaining rows are only counted and hashed. After 100,000 more rows, the query is stopped and the count is reported as a lower bound. For example:

FROM range(1_000_000_000);

This returns in 30 milliseconds with the footer first 20 of at least 102400 rows (query stopped early; .maxrows -1 for all).

A Hash of the Result

The footer includes a hash of the entire result, covering the rows past the cap. The hash does not depend on row order and distinguishes NULL from the string 'NULL'. An agent that wants to know whether a result changed after an edit can compare two footers instead of printing two results.

The footer is printed for empty results, for results of 10 rows or more, and whenever the output was capped. It is left out when the query was stopped early, because the hash would not cover the whole result.

Errors as JSON

Errors go to stderr as JSON, using the existing errors_as_json setting. Errors raised by the shell itself, such as an unknown dot command, are wrapped in the same structure:

{"exception_type":"Binder","exception_message":"Referenced column \"foo\" not found in FROM clause!\nCandidate bindings: \"bar\"","error_subtype":"COLUMN_NOT_FOUND","location":"[7,3]","position":"7","name":"foo"}

The shell drops the candidates field, because the message already lists the candidates. Without that change, a single “no matching function” error weighs 2.7 kB.

Compact Query Plans

EXPLAIN defaults to a new compact format with one operator per line, indented by depth. Estimates, actual row counts, and timings go in parentheses, followed by the operator’s properties:

UNGROUPED_AGGREGATE (est=1) Aggregates: count_star()
  HASH_JOIN (est=0) Join Type: INNER; Conditions: range = range
    FILTER (est=2) Expression: (r > 2)
      RANGE (est=10) Function: RANGE
    FILTER (est=1) Expression: (s > 2)
      RANGE (est=5) Function: RANGE

EXPLAIN ANALYZE starts with a summary line such as QUERY (time=0.0012s, read=1.2 MB). This is a regular format, so you can use EXPLAIN (FORMAT compact) outside agent mode too.

Cost Estimates and Progress

When the planner expects a statement to read a million rows or more, its estimate goes to stderr before execution starts:

estimate: ~7501215 rows read (lineitem ~6001215, orders ~1500000), ~4 rows returned

The agent can then decide whether to wait or to interrupt and rewrite the query. Smaller queries print nothing, since the line would only be noise. While a long query runs, the terminal progress bar is replaced by a plain line on stderr every 5 seconds:

progress: 42% (elapsed 12.0s, remaining ~16.5s)
progress: done (elapsed 28.7s)

Listing Tables

.tables prints one line per table, with an approximate row count and the columns:

memory.main.t1 (table, ~15 rows): a INTEGER PK, b VARCHAR

The usual box layout is hard to read through a pipe, and this gives an agent the schema in a single call.

The Startup Line

When agent mode turns on by detection, the shell prints one line on stderr before running anything:

duckdb agent mode on: CLAUDECODE is set and stdout is not a terminal; -no-agent turns it off, .help agent explains the output

The line states why the mode is on and how to turn it off.

The first version printed three lines of explanation (634 bytes) on every run. That was larger than most query results, and a model keeps its context between runs anyway, so it only needs the explanation once.

The longer text is available via .help agent. It describes the output rules and settings, and it points to a few features an agent might not find by itself:

SET max_execution_time = ms to put an upper bound on a query’s runtime
DESCRIBE query to get the result columns without running the query
SUMMARIZE for a quick profile of a table or query
.tables for the schema
duckdb_functions() for function documentation

If you do not want the startup line, add .startup_text none to your ~/.duckdbrc.

Trying It Out

If you use a coding agent, there is nothing to configure. Agents such as Claude Code, Codex, Cursor, Gemini CLI, and GitHub Copilot already set one of the environment variables from the table above, and they capture a command's output through a pipe rather than a terminal. Those are two of the three conditions for agent mode.

So as long as the duckdb binary on the agent’s path is v2.0 or newer and the agent does not request a specific output format with a flag such as -csv, -json, or -markdown, the mode turns on by itself. A format flag is treated as a deliberate choice, so it switches agent mode off: the CLI assumes you want exactly that format and does not override it.

To see what the agent sees, run a query through a pipe with the flag set:

duckdb -agent -c "SUMMARIZE FROM 'my_data.parquet'" | cat

Conclusion

With agent mode, an agent spends fewer tokens per call and does not act on partial results as if they were complete. That means fewer steps to a correct result.

The current behavior is a first attempt, and the conventions around agents are a work in progress. If your agent is not detected, or the output confuses it, please open an issue or comment on the pull request. Transcripts of an agent being confused are especially useful.

A DuckDB Database with No Data in It

2026-10-07T00:00:00+00:00

A DuckDB database file does not have to contain any data. It can store nothing but view definitions that point at files stored somewhere else, such as Parquet on object storage. The file stays a few hundred kilobytes no matter how large the data it describes, and anyone who attaches it sees a set of named relations that are ready to query.

Nikolas Goebel described this idea in DuckDB doesn't need data, which argues that modern databases have separated where data is stored from how it is read, to the point that most databases no longer need to hold any data of their own. From that perspective, DuckDB becomes a browser for the data cloud, where you access a dataset through a URL rather than a copy on your own disk.

In this blog post, we turn that idea into a small, shareable catalog.

The Idea

A DuckDB table stores its rows inside the database file. A view stores only the text of a query. When you read the view, DuckDB runs that query and returns the current result. If the query reads a remote Parquet file, the data lives in object storage and the view holds only the instruction for how to read it.

A database file made entirely of such views contains no rows of its own. It is a catalog: a set of named, ready-to-query relations created over data that stays in its original location and format. You can publish that catalog as a curated semantic layer over a data lake, where the view definitions encode the joins, filters, and column names that consumers should use, without copying or moving the underlying data.

Building a Catalog of Views

The examples use DuckDB's public train services dataset, which records one row for every stop a Dutch railway train makes. The data comes from the open datasets published by the Rijden de Treinen (Are the trains running?) application, and you can read it straight from its URL.

First, install the httpfs extension so that DuckDB can read over the network. This only needs to be run once.

INSTALL httpfs;

Now create a persistent database file and define a view over the remote data. Use a fully qualified location, a remote URL or an absolute path, so that the view resolves regardless of the client's working directory.

ATTACH 'rail_catalog.duckdb' AS rail;
USE rail;

CREATE VIEW services AS
    SELECT *
    FROM 'https://blobs.duckdb.org/train_services.parquet';

The catalog now exposes a services relation that reads from the remote file whenever it is queried.

SELECT departure_time, station_name, type
FROM rail.services
LIMIT 3;

departure_time	station_name	type
2023-05-15 00:00:00	Rotterdam Centraal	Intercity
2023-05-15 00:13:00	Delft	Intercity
2023-05-15 00:29:00	Den Haag HS	Intercity

Because the file stores only the view definition, it stays small no matter how large the referenced dataset grows. The train_services.parquet file holds 380,959 rows, yet the catalog that reads it is a few hundred kilobytes.

Hosting the Catalog

Upload rail_catalog.duckdb to any location that DuckDB can read over the network, for example an s3:// bucket or an https:// endpoint. The same bucket that already holds the Parquet files is a convenient place to put it.

Sharing the catalog then means sharing one URL instead of a list of individual file paths. The recipient does not need to know how many files back each view or where they are stored, only the address of the catalog.

Attaching the Catalog Read-Only

Consumers attach the catalog read-only and query the views as if they were local tables. Each query reads the current data from the referenced files, so the result reflects whatever is in object storage at that moment.

ATTACH 'https://example.com/rail_catalog.duckdb' AS rail (READ_ONLY);

SELECT station_name, count(*) AS calls
FROM rail.services
GROUP BY station_name
ORDER BY calls DESC
LIMIT 3;

station_name	calls
Utrecht Centraal	7663
Amsterdam Centraal	7591
Zwolle	5013

Consumers need the httpfs extension and, for s3:// sources, credentials that grant access to the underlying data. DuckDB reads only the columns and row groups a query needs, so a filtered or aggregated query over a large remote dataset does not download the whole thing.

Surviving Changes to the Data Layout

Because a query reads the view rather than the files directly, the publisher can change the files without changing the query. The consumer runs the same query before and after.

Suppose the single Parquet file becomes too large and the publisher splits it into one file per year. Updating the view to read a directory of files is a one-line change in the catalog, and consumers notice nothing.

CREATE OR REPLACE VIEW services AS
    SELECT *
    FROM read_parquet('s3://my-bucket/rail/services/year=*/*.parquet');

A view can also keep the schema stable when the files change. If a later batch of files renames station_name to station, the view can map the new column back to the name consumers already use.

CREATE OR REPLACE VIEW services AS
    SELECT
        * EXCLUDE (station),
        station AS station_name
    FROM read_parquet('s3://my-bucket/rail/services/*.parquet');

If you repartition the files or move them to a new bucket, you only have to update the catalog. Anyone querying it keeps running the same queries as before.

When to Use It

Use this pattern when you want to share a dataset without sending the recipient a long list of individual file paths, and when the data is already in a format DuckDB reads well, such as Parquet. Readers get a set of named relations to query, and the publisher can change the file layout without breaking those queries.

Note the following limitations:

Consumers must be able to access both the catalog file and every data source that its views reference.
Connections over HTTPS and the S3 API are read-only, so the catalog cannot be modified in place. To change a view, edit a local copy and upload it again.
Query performance depends on the network and the layout of the referenced files. The tips for querying remote files apply.

Conclusion

A DuckDB file that holds only views is a small catalog you can share as a single URL. The data stays in its original location, the publisher can change how it is stored, and the reader gets a stable set of relations to query.

For the full reference, see the Share a DuckDB Database That Only Stores Views guide.

DuckDB

How JetBrains Built Cross-Database Queries into DataGrip with DuckDB

Querying across Data Sources

Choosing DuckDB over Building Its Own

DuckDB in Plain Sight

An Initial Solution, Still Evolving

About JetBrains and DataGrip

Bringing Local Analytics to the Quant's Desktop

Fifteen Years of Heavyweight Infrastructure

Problems That Didn't Need a Heavyweight Answer

A Different Starting Point

QStudio: A Local Analytical Sandbox

Pulse: DuckDB by Default

PeachQ: Back to Q

Learn More

From Spreadsheets to DuckLake: A Data Engineer's Road to DuckDB

From Copy-Paste to Code

Teradata, BigQuery, and the Spark Years

Meeting DuckDB

Why DuckLake

Looking Forward to DuckDB 2.0

Conclusion

Gems of DuckDB 2.0.0

Python Appender

Usability Improvements in the CLI

Pretty-Printed .schema

.manual Dot Command

Calculator

Command History

Offline Documentation

Unsung SQL Features

SHOW for Configuration Settings

CREATE SECURE VIEW Statement

Options in regexp_extract

Setting a Sequence Value

Session Variables as Default Parameter Values

Custom Partition Paths

Announcing DuckDB 2.0.0 LTS “Cinnamon Teal”

DuckDB as a Server

Quack and CONNECT

DuckDB Server

DuckDB Client

Quack in the CLI

CLI Server

CLI Client

Performance

Asynchronous I/O

The VARIANT Type Becomes a First-Class Citizen

Agent Mode in the CLI

Extensibility

A Brand New SQL Parser

Write Extensions Once and Host Them Yourself

Core Extension Updates

SQL Features

Triggers

From JSON Strings to Typed Columns

The Data

Picking Out a Few Fields

Inferring a Structure from the Column

Adjusting the Types

Building the Table

Nested Structs and Name Collisions

Arrays of Objects

Values That Do Not Fit

Should the Column Be VARCHAR or JSON?

Further Reading

Benchmarking DuckDB 2.0-dev on NVIDIA Vera

Environment

System Setup

What Is inside an Iceberg Table?

Getting Started

A Table with History

The Tree

Layer 1: The Metadata File

Layer 2: The Manifest List

Layer 3: The Manifests

Column Statistics

Partition Statistics

Layer 4: Data Files and Delete Files

Why Look at This at All?

Pretty-Printed `.schema`

`.manual` Dot Command

`SHOW` for Configuration Settings

`CREATE SECURE VIEW` Statement

Options in `regexp_extract`

Quack and `CONNECT`

The `VARIANT` Type Becomes a First-Class Citizen

Should the Column Be `VARCHAR` or `JSON`?