Tabular data ingest, explained#

The hardware store#

A hardware store sells three things that all look like they solve the same problem: a funnel, a sieve, and a jig.

A funnel gets liquid into a narrow opening without spilling. It does not care what the liquid is. A sieve separates what should pass from what should not, and you choose the mesh. A jig holds a workpiece in a known position so every cut lands in the same place, which is how you get a hundred identical parts instead of a hundred similar ones.

Reading a data file is the funnel. Deciding which rows are acceptable is the sieve. Having a schema that says what the data must look like, so every load lands the same way, is the jig.

Most people arrive at this category looking for a funnel, buy a very good one, and discover six months later that what they needed was the jig.

The problem#

Somebody sends you a file. A spreadsheet, a CSV export, a tab-delimited dump. It contains information you need inside a system you run — a database, an application, a report.

Getting it in sounds like a single step. It is four, and they fail differently:

1. Read — open the file and find the rows. Encodings, delimiters, the difference between .xls and .xlsx, a file too large for the machine.
2. Type — decide what each column is. Is 00071 a number or an identifier? Is 03/04/2026 the fourth of March or the third of April? Is an empty cell the same as a cell containing the word NULL?
3. Validate — decide whether each row is acceptable. Required fields, plausible ranges, references that must exist, duplicates that must not.
4. Load — put the rows somewhere, in batches, in a transaction, and know what happened to the ones that did not make it.

Comparisons of this category almost always measure step one. Almost every problem people actually have lives in steps two, three and four.

Why step two is where it goes wrong#

A file is text. Text does not say what it means. So a reader either guesses what each column is, or it is told.

Guessing is useful. Handed an unfamiliar export with a hundred columns, a library that works out which are numbers, which are dates and which are text has saved a great deal of tedious work, and that convenience is why the most popular tools in this category guess by default.

Guessing is also how data quietly becomes wrong. A column of zero-padded reference numbers looks like numbers, so 00071 becomes 71 — a different reference number, stored successfully, with no error. That is not a bug. It is a library doing exactly what it was built to do, in a situation where nobody wanted it to.

The distinction that matters is not which library is better. It is who owns the schema:

• The file owns it when you are exploring and do not know what is in there. Guessing is the feature.
• You own it when the data is going into a table that already exists. Guessing is a liability, because now there are two descriptions of the data and nothing makes them agree.

The same code path, the same defaults, opposite outcomes. This is the single most useful idea in the category and it is invisible in any comparison that ranks by speed.

The approaches#

Read it as text and convert it yourself. Every value arrives as a string; your code turns each one into what it should be, against a definition you wrote. Nothing guesses, so nothing disagrees. The cost is writing the conversions.

Read it into a dataframe. A dataframe is a table in memory with typed columns, built for analysis — filtering, grouping, joining, summarizing. Powerful, and the tool most people reach for. Its type inference is the guessing described above.

Query the file directly. Some database engines read a CSV as though it were already a table, so the whole job becomes one statement: select from the file, convert in the select, reject what fails, insert the rest. The rows never enter your program at all, which is why this handles files bigger than the machine’s memory.

Describe the data separately, then check against it. Rather than a schema implied by code, a schema written down as an object — these columns, these types, these permitted ranges — and every incoming file checked against it. Some of these descriptions can be written in a format other programs and other languages can read, which matters when the sender is someone you do not employ.

Most working imports combine two or three of these. That is normal, and it is why “which library should I use” is usually the wrong first question.

The trade-offs, in general terms#

Convenience against control. Every feature that saves you from describing your data is a decision made on your behalf, correctly most of the time, silently always.

Speed against memory. The fastest readers hold everything in memory at once; the ones that use almost no memory read a row at a time and are slower. These rank the field differently, so any single “fastest” ranking has picked one and not told you.

Convenience against dependency weight. Some of these tools are hundreds of megabytes. For a long-running server nobody notices. For a small container, a command-line tool, or anything shipped to people who did not choose its dependencies, it is most of the budget.

Loud failure against quiet success. A tool that refuses a malformed file is annoying at 2am and correct. A tool that accepts it and gives you something is convenient at 2am and may be wrong for a year. Which you want depends on whether anyone will check.

When this domain is relevant to you#

• Data arrives from outside your system, in files, produced by people or by software you do not control.
• It has to end up somewhere structured — a database table, an application’s records — that has its own idea of what the data should look like.
• Somebody will eventually ask which rows did not load, and why.

When it is not#

• You are reading a file to answer a question now and discard the result. Use the convenient tool; nothing here applies.
• Your data arrives through an API in a structured format that already carries its types. That is a different category — the types are not in question, so nothing has to guess.
• The files are small, well-formed, produced by a system you control, and the schema on both sides is the same. This is real, and it is rarer than it looks, because the file format usually sits between the two systems and throws the types away.

The thing to remember#

A spreadsheet knows more about your data than a CSV does. A .xlsx file records that a column was formatted as text, so 00071 survives. A CSV records only the characters, so whether it is a number or an identifier has to be guessed by whoever reads it.

The file format with the messy reputation is carrying information that the tidy-looking one deleted. Asking someone to “just send it as CSV” is asking them to throw away the part that would have made the import correct.

Sources#

• PEP 305 — CSV File API, the specification behind Python’s standard-library reader, and the source for the dialect and quoting behavior described above.
• The Frictionless Data specifications (Table Schema, Data Package), for the idea that a schema can be a portable document rather than code.
• ECMA-376 / ISO/IEC 29500, the Office Open XML standard, for why an .xlsx file can carry cell formatting and a .csv cannot.
• The specific library behaviors described here were measured on 2026-09-09; the fixtures, the harness and the raw results are in harness/1-057-tabular-data-ingest/, and the findings are in S2.

The category: getting rows into a schema#

Tabular ingest is the job of turning a file somebody sends you — a CSV export, a spreadsheet with three sheets and a merged header, a tab-delimited dump from an old system — into rows in a table you already have, and knowing which rows did not make it.

It is not one library’s job. It is a path with four stops, and most comparisons of this category only benchmark the first:

A parser shootout measures the first stop. Every production import problem lives in the other three.

The boundary test: who decides the schema#

The split that organizes this category, and the one that speed comparisons cannot see:

Does the library infer the schema from the file, or accept a schema you already have?

Inference is a feature when the file is the authority — exploratory analysis, an unknown CSV, a one-off. It is a defect when a schema already exists, because the library’s guess and the target table now disagree, and the disagreement surfaces as a type error at insert time rather than as a validation message pointing at a cell.

The two stances are not a quality ranking. They answer different questions, and a tool built for one is awkward at the other. Where a library supports both, the survey says which one it does well.

Two layers, covered together#

Readers and engines — the stdlib csv module, pandas, polars, DuckDB, and the spreadsheet-specific readers. These get bytes to rows.

Validators and loaders — pandera, Great Expectations, frictionless, pydantic, and the pipeline tools that wrap the whole path. These decide which rows are acceptable and what happens to the ones that are not.

Splitting them into two surveys would be tidier and less useful. The choice at one layer constrains the other: a pandera schema is a dataframe schema and presumes a dataframe reader; a pydantic model is row-shaped and presumes an iterator. Picking a reader without knowing what will validate its output is how an import ends up with a fast parse and no answer to “which rows were rejected and why”.

Scope#

In: reading delimited text and spreadsheet formats in Python; type inference and coercion; row- and frame-level validation; error reporting; the loading step insofar as it constrains the choice above it.

Out, and where it lives instead:

• Extracting content from Word, PowerPoint or PDF, and the internals of the xlsx format — 1.102 Document Parsing Libraries, which is surveyed and already covers openpyxl, xlsxwriter and pdfplumber from the file-format side. 1.102 asks how to get bytes out of an office format. This survey asks how to get rows into a schema. They meet at openpyxl and pandas, and the sections on those two say what each survey answers.
• JSON as an interchange format — 1.056 JSON Libraries.
• YAML — 1.059, reserved and unwalked.
• The database write path, connection handling and bulk-insert mechanics — 1.180 Python ORMs, 1.180.1 Python Database Drivers.
• Web form validation — 1.115 Form & Validation Libraries, a different job with a different shape: one record at a time, arriving from a browser, with a user waiting.
• Orchestration and scheduling of recurring loads — 1.204.

What the numbers in these files are#

Registry and repository figures are a snapshot taken 2026-09-09 and recorded in harness/1-057-tabular-data-ingest/registry-2026-09-09.json, so any of them can be re-fetched and compared. Version, release date, license, star count and monthly download count come from PyPI, the GitHub API and pypistats on that date; each is marked where it appears.

S1 states no performance figure it did not read somewhere, and marks those (source: …) or (unverified). Measurement is S2’s job, and the measurement plan is written before S2 runs.

DuckDB#

DuckDB is an embedded analytical database that reads CSV and Parquet files directly as if they were tables. It belongs in this survey because it answers the ingest question in a different shape from every other option here: instead of parsing a file into a Python object and then inserting it, it queries the file and inserts the result, and Python never holds the rows.

MIT. Version 1.5.5, released 2026-07-22, requires Python 3.10 or newer. 41,105 GitHub stars, 868 open issues, repository created 2018-06-26, last push 2026-09-09. 60,787,626 downloads in the last month (source: PyPI, GitHub API and pypistats, accessed 2026-09-09).

Why it changes the shape of the job#

The four stops in approach.md — read, type, validate, load — normally happen in Python, one after another, with the whole dataset materialized somewhere in the middle. DuckDB collapses them into a statement: select from the file, cast in the select list, filter the rows that fail a condition, insert the rest. The rejected rows are a second query against the same file rather than a list accumulated in a loop.

One consequence: memory stops being a function of file size. A file larger than RAM is a normal case rather than a streaming exercise, because the engine was built for out-of-core work.

Type handling and the sniffer#

The CSV reader auto-detects delimiter, quoting, header presence and column types, and reports what it decided — the detection is inspectable before the data is read, which is the property that separates a sniffer you can audit from one that guesses silently.

Types can be stated explicitly instead, per column, and the reader has an option to reject rather than coerce values that do not fit. It also has a mode for capturing malformed rows into a separate table rather than failing the whole read, which is close to the per-row error report this category usually has to hand-build.

The open issue count of 868 against 41,105 stars is the lowest ratio among the engines in this survey (source: GitHub API, accessed 2026-09-09). S4 examines whether that reflects triage practice or defect rate, since the two are not distinguishable from the count alone.

The spreadsheet gap#

CSV and Parquet are native. Spreadsheets are not the same story: .xlsx support arrives through an extension rather than the core reader, which puts it on a different footing from the CSV path — a different maintenance surface and a different set of format edge cases. Anyone whose input is mostly spreadsheets is choosing DuckDB for the loading half and still needs a reader from spreadsheet-readers.md for the first stop.

What it costs#

A second data model. The import now runs in SQL, and the schema knowledge lives in a statement rather than in Python objects. For a team whose validation rules are already expressed as Python — a pydantic model, an ORM definition — that is a translation, and translations drift.

Validation is SQL-shaped. Constraints, casts and filters cover a great deal, and they express set-level checks such as uniqueness and referential lookups better than a row-wise validator does. What they do not produce is a structured, per-row, human-readable explanation of why row 3,847 failed. Getting that back means writing the diagnostic queries.

It is a database. For an application that already has one, adding a second engine to read files is a real decision rather than a dependency detail, and it needs a reason beyond convenience.

Where it is the right choice#

When files are large relative to memory. When the target is a database table and the transformation is expressible in SQL. When the rejected rows need to be queried rather than logged. When the same ingest must handle both CSV and Parquet.

Where it is the wrong choice#

When the input is mostly spreadsheets. When the validation rules are complex, row-wise and already written in Python. When adding a database engine to the deployment is harder than the import it solves.

pandas#

pandas is the default answer to “how do I read this file in Python”, and its read_csv is probably the most-called ingest function in the language. It is also the option whose defaults do the most work without being asked, which is what makes it both the fastest path to a first result and the most common source of the errors this category is about.

BSD-3-Clause. Version 3.0.5, released 2026-07-22, requires Python 3.11 or newer. 49,709 GitHub stars, 2,738 open issues, repository created 2010-08-24, last push 2026-09-09. 637,058,384 downloads in the last month (source: PyPI, GitHub API and pypistats, accessed 2026-09-09).

What it does#

One call reads a delimited file or a spreadsheet into a DataFrame, inferring column types as it goes. It reads CSV, TSV and arbitrary delimiters; fixed-width text; Excel through a pluggable engine; and a long tail of other formats. It handles compression transparently, reads in chunks when asked, and takes an explicit dtype mapping when the caller knows better than the inference.

The inference is the product. Given a column of digits it produces integers; given something date-shaped and a parse_dates instruction it produces timestamps; given mixed content it falls back to object. For a file whose schema nobody has written down, this is an enormous amount of work not done.

Where the inference hurts#

Every property that makes inference convenient makes it hazardous when a schema already exists.

Leading zeros. A column of zero-padded identifiers — postal codes, account numbers, part numbers — infers as integer and the padding is gone. The data is now wrong in a way that does not raise.

Mixed columns and silent widening. A numeric column containing one "n/a" becomes object dtype, and downstream arithmetic either fails far from the cause or coerces in ways nobody chose.

Chunk-dependent inference. Reading a large file in chunks infers types per chunk, so a column can be integer in one chunk and object in another depending on where the first non-numeric value falls. The classic symptom is a DtypeWarning that most code ignores.

Empty versus missing. An empty CSV cell and the literal string NA both become NaN by default, which conflates “not supplied” with “explicitly recorded as unknown” — a distinction the target schema usually cares about.

None of these are bugs. They are the cost of a library whose job is to make a reasonable guess, used in a situation where guessing was not wanted.

The error-reporting gap#

pandas reports parse failures as exceptions against the file, not as a per-row verdict. There is no built-in structure for “3,997 rows loaded, 3 rejected, here is which and why”. Building that means reading with everything as strings, coercing column by column, and collecting failures — at which point the inference has been turned off and the value of using pandas for this step is the API rather than the guessing.

on_bad_lines handles rows with the wrong number of fields. It does not handle rows with the right number of fields and the wrong content, which is the common case.

The 3.0 change#

pandas 3.0 is a major release, and its Python floor of 3.11 (source: PyPI metadata, accessed 2026-09-09) is higher than every other reader in this survey. That matters for a library or an application that has not moved yet, and it is the first time in years that choosing pandas has constrained the interpreter version rather than the other way round. S4 assesses what the 3.x line changes for long-lived code; S2 checks whether the copy semantics that the 3.0 line advertises change the memory profile of a large read.

Ecosystem weight#

The download figure above is roughly two orders of magnitude above the specialist tools in this survey, and it is the practical argument for pandas that has nothing to do with parsing: it is probably already installed, every developer has met it, and the answer to a question about it exists. Against that, it is a large dependency to add to an application that needs one CSV read — the wheel carries a compiled extension and pulls NumPy — and it brings an entire computational model along with the reader.

Where it is the right choice#

When the file is the authority and the shape is unknown. When the work after the read is analysis rather than insertion. When it is already a dependency for other reasons, and the import is small enough that inference errors would be noticed.

Where it is the wrong choice#

When a schema already exists and disagreement between it and the inferred types would be silent. When the import must report per-row outcomes. When the deployment cannot carry the dependency, or is pinned below Python 3.11.

polars#

polars is a dataframe library written in Rust with a Python binding, built on the Apache Arrow memory layout. In this category its distinguishing property is not speed — it is that the schema is a first-class object the caller can state, inspect and enforce, rather than a result of reading.

MIT. Version 1.44.2, released 2026-09-09, requires Python 3.10 or newer. 39,695 GitHub stars, 2,868 open issues, repository created 2020-05-13, last push 2026-09-09. 63,340,166 downloads in the last month (source: PyPI, GitHub API and pypistats, accessed 2026-09-09).

What it does differently at the read#

read_csv takes an explicit schema mapping column names to types, and a separate option to override only some of them. Given a schema, it does not infer — it parses to what was declared and raises where the file disagrees. The disagreement is the point: an ingest that knows the target types wants to hear about a mismatch at the read, not discover it at insert.

Scanning is the other half. scan_csv returns a lazy frame that has read the header and nothing else, so the schema can be checked against the target before any data is loaded. For a large file arriving from outside, confirming the columns are what they claim to be before paying to read it is a different operation from reading it and then looking.

Strict parsing is the default posture rather than an option. A value that does not fit its declared type is an error, not a silent coercion to object.

Type handling#

Arrow’s type system distinguishes cases that matter here and that a looser reader conflates: null is a distinct state rather than a floating-point sentinel, so “empty cell” and “recorded as unknown” can be told apart if the file distinguishes them; integer widths are explicit; and there is a real date type rather than a datetime standing in for one.

The practical consequence for ingest is that a schema written against the target table transfers with less loss. A column declared as a string stays a string, leading zeros included, because nothing is trying to help.

Where it is weaker#

The spreadsheet path is not its own. read_excel delegates to another library — calamine, openpyxl or xlsx2csv depending on the engine selected — so for .xlsx input polars is a wrapper, and the properties that matter come from whichever engine is underneath. This survey treats those engines on their own terms in spreadsheet-readers.md.

The validation story ends at types. polars enforces that a column is an integer. It has no vocabulary for “between 1 and 5”, “must exist in this other table”, or “no duplicates on this key”. That work goes to the validation layer, and pandera is the option that speaks polars natively.

Ecosystem depth. The download figure is about a tenth of pandas’, and the gap is larger than that in tutorials, answered questions and third-party integrations. A team that hits an unusual file will find fewer people who have hit it before.

API churn. A 1.x line with releases as frequent as this one — 1.44.2 on the day of writing (source: PyPI, accessed 2026-09-09) — moves faster than a long-lived import script may want. S4 assesses what the cadence implies for code nobody will touch for two years.

Where it is the right choice#

When the target schema exists and should govern the read. When the file is large enough that streaming or lazy evaluation earns its keep. When null and empty need to stay distinguishable. When strictness at the read is preferred to leniency followed by debugging.

Where it is the wrong choice#

When the input is primarily spreadsheets, where the dependency that decides behavior is the engine rather than polars. When the surrounding code is already pandas-shaped and the conversion cost exceeds the benefit. When the team’s familiarity is the binding constraint, which for a one-off import it often is.

S1 verdict: the field, and what is still open#

The field#

(Source: PyPI, GitHub API and pypistats, accessed 2026-09-09; recorded in harness/1-057-tabular-data-ingest/registry-2026-09-09.json.)

Every option is permissively licensed. Nothing here is a copyleft or commercial decision, which is unusual for a category this size and removes a constraint that often decides these choices elsewhere.

Three findings from the rapid pass#

The most-used library in the category has not shipped in over two years. openpyxl’s 3.1.5 dates from 2024-06-28 and draws 309.9 million downloads a month (source: PyPI and pypistats, accessed 2026-09-09). Its default-branch history shows two commits in October 2025 and nothing since. It is also the default .xlsx engine for pandas, so an enormous amount of spreadsheet reading in Python runs through a library nobody should expect a fix from. A stable format can support a finished library — this is a fact to plan around, not a scandal — but the fact is not widely known, partly because openpyxl lives on Mercurial at Heptapod and therefore appears in no star-count comparison.

The engine choice for spreadsheets is made a layer below where people make it. Neither pandas nor polars reads .xlsx itself; both delegate. Choosing pandas for spreadsheet ingest is choosing openpyxl by default without saying so, and the option that changes the properties — engine="calamine" — is one keyword away and rarely used.

The category’s real split is not speed. It is whether the library infers the schema or accepts one. Inference is what makes pandas the fastest path to a first result and the most common source of silent wrongness when a target schema already exists: leading zeros lost to integer inference, chunk-dependent types, empty and missing conflated. Speed comparisons cannot see this, and it decides more imports than throughput does.

What S1 cannot settle#

Whether calamine is actually faster than openpyxl, and by how much on what. The claim is why people switch and it is repeated everywhere (unverified). It needs a workload, a file size, a sheet shape and a version pair before it means anything.

What the inference failures cost in practice. That leading zeros are lost is a documented behavior; how often a realistic file triggers it, and whether the alternatives avoid it or merely fail differently, is a measurement.

Whether DuckDB’s rejected-row capture is as usable as it sounds. It is the closest thing in the field to a built-in error report, and the gap between “captures malformed rows” and “tells you why row 3,847 failed” is exactly the kind of gap S2 exists to find.

Whether pandera’s lazy validation scales. Collecting every failure rather than raising on the first is the right shape; whether it stays usable when a file is substantially wrong — tens of thousands of failures rather than three — is not something the documentation answers.

These four questions are what the S2 measurement plan is built from. None of them is a throughput benchmark, which is the point: the numbers this category is usually compared on are not the numbers that decide it.

The spreadsheet readers: openpyxl, python-calamine, xlsxwriter#

When a person rather than a system produced the file, it is a spreadsheet. None of the engines in the previous files read .xlsx themselves — pandas and polars both delegate to one of the libraries here, so for spreadsheet input the choice that matters is this one, made one layer below the call that appears to make it.

pandas dispatches read_excel to one of five engines: xlrd, openpyxl, odf, pyxlsb and calamine (source: pandas/io/excel/_base.py on main, accessed 2026-09-09).

openpyxl#

The default for .xlsx and .xlsm almost everywhere, and by download volume the most used library in this entire survey.

MIT. Version 3.1.5, released 2024-06-28, requires Python 3.8 or newer. 309,869,897 downloads in the last month (source: PyPI and pypistats, accessed 2026-09-09).

Its last release was over two years before this was written. The preceding releases run 3.1.2 on 2023-03-11, 3.1.3 on 2024-05-29 and 3.1.4 on 2024-06-12 (source: PyPI release history, accessed 2026-09-09), so the 3.1 line has been in maintenance for a while and then stopped.

The repository is not on GitHub. It is Mercurial, hosted at foss.heptapod.net, which has two consequences. The first is practical: every star-count comparison of Python spreadsheet libraries silently omits the most-used one, because it has no stars to count. The second is about contribution — the barrier to sending a patch is a different forge and a different version control system, and that shows in the commit history.

What that history shows, read directly: the default branch carries two commits dated 2025-10-01, one of them reverting an accidental merge, and before those the 2024-06-28 release commits (source: Heptapod API, projects/openpyxl%2Fopenpyxl/repository/commits, accessed 2026-09-09). The project’s last_activity_at reads 2026-08-31, which reflects tracker traffic rather than code.

This is a description, not a verdict. A format-reading library for a stable format can legitimately be finished, and xlsx has not changed. What it means for a decision is narrower and firmer: nobody should plan on a fix. A bug encountered today in openpyxl is a bug to work around, and S4 examines what that implies for code with a long life.

It reads and writes, preserves formatting, formulas, charts and comments, and has a read-only mode that streams rows instead of building the whole workbook in memory. For writing a formatted spreadsheet it remains the only full-featured option here.

python-calamine#

A Python binding to the Rust calamine crate, and the newest option in the field.

MIT. Version 0.8.2, released 2026-07-13, requires Python 3.10 or newer. 497 GitHub stars, 17 open issues, repository created 2021-08-29, last push 2026-09-07. 6,600,453 downloads in the last month (source: PyPI, GitHub API and pypistats, accessed 2026-09-09).

It reads, and only reads. That is the design: no formatting, no formula evaluation, no writing. In exchange it handles .xlsx, .xls, .xlsb and .ods through one interface, which is unusual — the legacy .xls path elsewhere means xlrd, and .ods means odfpy, whose last release was 2020-01-18 (source: PyPI, accessed 2026-09-09).

Its position is stronger than 497 stars suggests, because it is a registered pandas engine: a great deal of its use arrives through read_excel(engine="calamine") rather than through its own API, so neither stars nor imports capture it. Its 6.6 million monthly downloads are about one forty-seventh of openpyxl’s 309.9 million (source: pypistats, accessed 2026-09-09) — a real gap, and a floor rather than a measure, since the engine path means many users install it without ever naming it in their own code.

Reputation holds it to be substantially faster than openpyxl on large workbooks (unverified). That claim is exactly what S2 is for; it is repeated here because it is the reason people reach for it, not because this pass checked it.

The risks are the ones any young binding carries: a small maintainer base, a compiled dependency that must have a wheel for the target platform, and a 0.x version number.

xlsxwriter#

Write-only, and the counterpart rather than a competitor to the readers above.

BSD-2-Clause. Version 3.2.9, released 2025-09-16, requires Python 3.8 or newer. 3,971 GitHub stars, 30 open issues, repository created 2013-01-04, last push 2026-08-04. 92,651,992 downloads in the last month (source: PyPI, GitHub API and pypistats, accessed 2026-09-09).

It cannot open a file, so it is out of scope for ingest and in scope for the other half of the job that ingest always turns out to need: handing back a spreadsheet of the rows that failed. Thirty open issues against a repository of that age and volume is the healthiest ratio in this survey (source: GitHub API, accessed 2026-09-09), and it is still shipping.

Its constant-memory mode writes row by row without holding the workbook, which matters when the rejection report is large.

Choosing among them#

For ingest specifically, the reading engine is the only one of these that is on the path, and the decision is between openpyxl’s breadth and calamine’s narrowness. The maintenance signals point one way and the capability signals point the other: openpyxl does more and has not shipped in two years; calamine does one thing, ships regularly, and is young.

The case where it does not matter is when the file is a single flat sheet with a header row, which is most files people send. The case where it does is a workbook with merged cells, multiple sheets, formulas whose cached values matter, or .xls input — and those resolve to different libraries, which is why the choice cannot be made once and forgotten.

The stdlib csv module#

Python ships a CSV reader in the standard library. It has been there since Python 2.3, it is specified by PEP 305, and it is the option most comparisons of this category leave out — which is a problem, because for a large share of real imports it is the correct answer and the only one with no dependency cost.

What it does#

It reads and writes delimited text. It handles the parts of the format that are hard and that hand-rolled split(",") code gets wrong: quoted fields containing the delimiter, embedded newlines inside quotes, doubled quote characters, and configurable dialects for the tab- and semicolon-delimited files that European exports produce. The Sniffer class will guess a dialect from a sample.

It reads one row at a time from any iterable of lines, so memory use is a function of the widest row rather than the file. A 2 GB export streams through it on a laptop.

What it does not do#

No type inference. Every field arrives as a string. A date is a string, a number is a string, an empty cell is an empty string. This is the module’s most consequential property, and whether it is a limitation or a feature is the whole argument of this category.

No schema. Nothing describes what the columns should be, so nothing can tell you the file has changed shape. A column reordered upstream is silently accepted.

No validation, no error accumulation. There is no notion of a bad row. Malformed input either parses into something wrong or raises, and either way the module has no opinion.

No spreadsheet support. .xlsx is a ZIP of XML and the csv module cannot open it. A file named .csv that is actually a spreadsheet — common, because people rename them — fails here in a way that a sniffer will not diagnose.

Where it is the right choice#

When the schema is already known and lives somewhere else. If a target table defines the types, then a reader that hands back strings and lets the caller coerce them against that definition is doing exactly the right amount: it is not guessing, so there is nothing to disagree with. Every wrong value has one place it was converted and one place to report it from.

When the file is large relative to memory. Streaming is the default rather than a mode.

When dependencies are the constraint. There is nothing to install, nothing to pin, nothing to audit, and no wheel that fails to build on an unusual platform. For a library, a CLI, or anything that ships to users who did not choose its dependencies, that matters more than convenience.

Where it is the wrong choice#

When the file is the authority and its shape is unknown. Exploratory work over an unfamiliar CSV with a hundred columns is miserable without inference, and writing coercion by hand for a hundred columns is worse than accepting a library’s guess.

When the work after the read is columnar — group, join, aggregate, pivot. Rebuilding that over lists of strings reimplements a dataframe badly.

When the input is a spreadsheet, which it often is when a person rather than a system produced it.

Maintenance and risk#

Part of CPython, covered by the language’s own deprecation policy and release cadence, and governed by PEP 305 rather than by a maintainer’s availability. The abandonment risk that S4 assesses for every other option in this survey does not apply: the module outlives any library that wraps it.

The practical risk runs the other way. Because it has no schema, code built on it accumulates ad-hoc coercion and ad-hoc error handling, and that code is rarely reviewed as carefully as a schema declaration would be. What the module saves in dependencies it can spend in hand-written conversion logic that nothing tests.

Numbers#

Version and support follow CPython itself. Python 3.13 and 3.14 are the versions in support at the time of writing; there is no separate release, star count or download figure for a stdlib module, and its absence from download-based popularity rankings is a measurement artifact rather than a signal about usage.

Sniffer is the one part with a reputation for unreliability on real files — it samples a prefix and guesses, and a file whose first rows are unrepresentative defeats it. That is a widely repeated claim rather than one this pass tested (unverified); S2 measures it against the alternatives’ own detection.

The validation and loading layer#

A reader answers “what is in this file”. Everything that decides whether the contents are acceptable, and what happens to the rows that are not, lives here. This is the half of the category with no default answer — where the reader question has an obvious first choice, this one does not, and the options differ more from each other than the readers do.

pandera#

Schema validation for dataframes, declared as a schema object and applied to a frame.

MIT. Version 0.33.1, released 2026-09-01, requires Python 3.10 or newer. 4,451 GitHub stars, 421 open issues, repository created 2018-11-01, last push 2026-09-09. 8,833,282 downloads in the last month (source: PyPI, GitHub API and pypistats, accessed 2026-09-09).

It sits closest to the boundary test in approach.md, because its whole premise is that the schema exists first and the data is checked against it. A schema names columns, their types, whether nulls are allowed, and arbitrary checks on values — ranges, membership, regular expressions, and relationships between columns.

The property that matters for ingest is lazy validation: rather than raising on the first failure, it collects every failure and returns them as a frame of what went wrong, where. That is the per-row error report that pandas and polars do not produce and that most hand-built importers reimplement badly.

It supports both pandas and polars frames, which makes it the one validation choice that does not lock the reader decision.

The 421 open issues against 4,451 stars is the second-highest ratio in this survey (source: GitHub API, accessed 2026-09-09), and S4 examines it — a validation library’s backlog is worth reading, because unfixed edge cases in a checker are cases that pass when they should not.

Great Expectations#

A data-quality framework rather than a validation library, and the difference is scope.

Apache-2.0. Version 1.22.0, released 2026-08-31, requires Python 3.10 up to but excluding 3.14. 11,778 GitHub stars, 41 open issues, repository created 2017-09-11, last push 2026-09-09. 23,048,049 downloads in the last month (source: PyPI, GitHub API and pypistats, accessed 2026-09-09).

It brings a vocabulary of expectations, a store for them, profiling that proposes expectations from data it has seen, and generated documentation describing what a dataset is supposed to look like. For a pipeline that runs repeatedly against sources that drift, that machinery is the product: the value is in noticing that this month’s file differs from every previous month’s.

For a single import it is a great deal of apparatus. The configuration surface is substantial, and the framework wants to own a directory, a context and a run lifecycle. The 41 open issues against 11,778 stars is the lowest ratio among the validators here (source: GitHub API, accessed 2026-09-09).

The Python ceiling matters: it is the only library in this survey that declares an upper bound, excluding 3.14 (source: PyPI metadata, accessed 2026-09-09). An application already on a newer interpreter cannot adopt it.

frictionless#

An implementation of the Frictionless Data specifications — Table Schema and Data Package — which are open standards for describing tabular data rather than one library’s schema format.

MIT. Version 5.19.0, released 2026-04-13, requires Python 3.8 or newer. 842 GitHub stars, 240 open issues, repository created 2014-12-23, last push 2026-08-27. 665,456 downloads in the last month (source: PyPI, GitHub API and pypistats, accessed 2026-09-09).

Its argument is portability. A Table Schema is JSON, it is not Python, and it can travel with the data to a consumer who uses a different language or none. Where a schema has to be agreed between two parties — a public data publisher and its users, an organization and its suppliers — that is a real advantage over a schema expressed as Python objects.

It validates, describes, extracts and transforms, and reports errors with row and field positions.

The signals are mixed: 240 open issues against 842 stars is the highest ratio in this survey, and the release before this writing was 2026-04-13, roughly five months earlier (source: GitHub API and PyPI, accessed 2026-09-09) — slower than every other maintained option here, though the repository was pushed to 2026-08-27. S4 takes this up properly.

pydantic#

Not a tabular library, and included because it is the validator a large number of Python applications already have.

MIT. Version 2.13.5, released 2026-08-28, requires Python 3.9 or newer (source: PyPI, accessed 2026-09-09).

It validates one record at a time against a model, with coercion rules, custom validators and structured errors naming the field and the reason. Applied to ingest, that means iterating rows and validating each — which fits naturally with the stdlib csv module, because both are row-shaped, and fits badly with a dataframe, because validating a frame row by row through Python discards the reason for having a frame.

Its strength for this job is that the model may already exist. When an application defines its records as pydantic models, validating an import against the same model means one definition rather than two that can disagree. Its weakness is that it has no notion of a dataset: uniqueness across rows, referential checks against another table, and “no more than 5% of rows may be missing this field” are not expressible in a per-record model.

The pipeline tools#

Four libraries that wrap the whole path rather than one stop, sharing an assumption that ingest is a recurring job rather than a one-off.

• dlt — Apache-2.0, 1.30.0 released 2026-08-11, 5,837 stars, 427 open issues, 5,998,650 monthly downloads, repository created 2022-01-26. The youngest and by far the fastest growing. It infers a schema from a source, evolves it as the source changes, and loads to a destination, which makes schema drift a managed event rather than a failure. Aimed at recurring loads from APIs and files.
• petl — MIT, 1.7.25 released 2026-09-09, 1,317 stars, 84 open issues, 780,935 monthly downloads, repository created 2011-08-19. Lazy, row-based table transformations with a small memory footprint and no dataframe. Long-lived and still shipping.
• csvkit — MIT, 2.2.0 released 2025-12-15, 6,411 stars, 41 open issues, 278,889 monthly downloads, repository created 2011-04-01. Command-line tools rather than a library, and the right answer when the question is asked at a shell prompt rather than in code.
• tablib — MIT, 3.10.0 released 2026-07-31, 4,756 stars, 54 open issues, 5,148,468 monthly downloads, repository created 2011-03-28. A format-agnostic dataset object that converts between CSV, xlsx, JSON and others, which makes it a useful export layer more than an ingest one.

(All figures source: PyPI, GitHub API and pypistats, accessed 2026-09-09.)

What separates these options#

The choice follows from two questions rather than from feature counts: does the schema need to leave Python, and does this import run once or every month. A portable schema points at frictionless, a recurring load with drift points at Great Expectations or dlt, a one-time load against an existing application model points at pydantic, and a dataframe already in hand points at pandera.

The workload, and how to read these numbers#

Five levels ran on 2026-09-09, in the order the measurement plan set. Everything here was measured; nothing was recalled.

The machine#

aarch64, Linux 6.6.87.2 under WSL2 on a laptop, 8 cores, 12.2 GB RAM, CPython 3.12.3. Pinned: pandas 3.0.5, polars 1.44.2, duckdb 1.5.5, openpyxl 3.1.5, python-calamine 0.8.2, pandera 0.33.1, pydantic 2.13.5, petl 1.7.25, pyarrow 25.0.1, XlsxWriter 3.2.9.

These figures may be compared with each other and with nothing else. polars, DuckDB, calamine and pyarrow are compiled; the stdlib module, petl and much of pandas are interpreted CPython. A compiled-against-interpreted ratio measured on aarch64 does not share an axis with one measured on x86_64, and this corpus made that mistake once, in 1.253. Anyone setting these against a vendor’s published range needs the x86 arm first, which is L7 and did not run.

The fixtures#

Generated rather than downloaded, so every hazard sits at a known position and “did the reader get this right” is checkable instead of eyeballed. gen_fixtures.py is the answer key: its HAZARDS dict carries the twelve hazards, and HAZARD_ROWS the row each lands on.

• clean.csv — 10,000 rows, 8 columns, well-formed, with 100k and 1M variants for the scale sweep. Twelve is the hazard count, not the column count.
• hostile.csv — the same shape carrying zero-padded identifiers, a numeric column with one n/a, empty cells beside literal NA, three date formats, thousands separators, a quoted field containing the delimiter, an embedded newline, a value that looks like a formula, a leading +, and a final row with too few fields.
• hostile.xlsx — the same content as a workbook, plus a merged banner above the header row, a text-formatted column holding digits, a real date cell, and a prose sheet placed before the data so that taking sheet zero by index returns the wrong thing.
• damaged_{1,10,50}.csv — 20,000 rows with a known fraction carrying an unparseable amount.

Timing method#

Each reader ran in a fresh subprocess, so peak RSS is that reader’s own high-water mark rather than whatever ran before it. Wall time is the minimum of five rounds inside that process — three at 1M rows, recorded per size in the results file — because the floor is the measurement and everything above it is scheduler noise.

The leveling check#

Any parameter set differently per candidate becomes the finding rather than the comparison. So the job is defined at the boundary rather than at each library’s API: every reader is asked for the same observable result — every row seen, and one column’s total string length — and l3_throughput.py asserts all readers returned identical row counts and identical checksums before any timing is compared, recording the verdict under fair in the results file. It passed at all three sizes.

Without it, a lazy reader reports the cost of building a plan and appears a hundred times faster than it is. polars (scan + collect) is in the table for exactly this reason: it is the lazy path forced to materialize, so it can be read against the eager one.

Where a library must be configured to be comparable, both arms run. pandas appears with defaults and with dtype=str; polars with inference and with infer_schema=False; DuckDB with its sniffer and with the flags its own error message recommends. Measuring only the configured arm would delete L2’s finding, because the difference between the arms is the finding.

S2 measurement plan#

Written before S2 runs, per Step 3.5 of ADDING-RESEARCH.md. Levels are ordered by cost. The default cut line is everything that runs in one container, with no external service and no second machine; anything past it is deferred and filed rather than half-done.

What S1 could not settle#

1. Is calamine faster than openpyxl, and by how much on what. The claim is why people switch and S1 marked it (unverified).
2. What the inference failures cost. That pandas loses leading zeros to integer typing is documented behavior. Whether a realistic file triggers it, and whether the alternatives avoid it or fail differently, is a measurement.
3. Whether DuckDB’s rejected-row capture is usable. The gap between “captures malformed rows” and “tells you why row 3,847 failed” is the gap that decides whether an importer has to be hand-built anyway.
4. Whether pandera’s lazy validation scales. Collecting every failure is the right shape; whether it survives a file that is substantially wrong rather than slightly is not something the documentation answers.

Only one of these is a throughput question. That is the point of S1’s central finding, and the plan is weighted accordingly.

The structural constraint#

The readers do not share an output shape, so a harness that hands them all “the same job” has to define the job at the boundary rather than at the API:

The leveling rule. Any parameter set differently per candidate becomes the finding rather than the comparison — this corpus has the 1.194 case on record, where an accidental blocking mismatch moved a score by more than the gap between the candidates. So before any timing is compared, the level asserts that every reader returned an identical row count and an identical checksum over the same column of the same file, and records the verdict alongside the timings. A reader that cannot be made to agree is reported as disagreeing rather than quietly dropped.

Where a library must be configured to be comparable — pandas with dtype=str, polars with an explicit schema — both arms run: defaults and configured. The difference between them is question 2’s answer, so measuring only the configured arm would delete the finding.

The fixtures#

Three files, generated by the harness and committed as a generator rather than as data:

• clean.csv — 10,000 rows, 8 columns, well-formed. The baseline nobody argues about.

• hostile.csv — 10,000 rows of 8 columns carrying, in known positions, the things that actually break imports: zero-padded identifiers, a numeric column with one n/a, empty cells beside literal NA, three date formats, 1,234.00 thousands separators, a quoted field containing the delimiter, an embedded newline, a leading +, a value that looks like a formula, and a trailing row with too few fields. Twelve in all, positions recorded, so “did it get this right” is checkable rather than eyeballed.

  A mixed-encoding hazard was planned and cut — a Windows-1252 byte in an otherwise UTF-8 file. It is a real failure mode and it needs its own level: every reader here would raise rather than return a wrong value, so it belongs with error handling rather than in a correctness matrix scored on returned cells. Filed, not dropped.

• hostile.xlsx — the same content as a workbook, plus the spreadsheet-only hazards: a merged header cell, a prose sheet placed before the data sheet so sheet zero is the wrong one, a column formatted as text holding digits, and a date cell whose displayed value differs from its serial.

Scale sweeps reuse clean at 10k, 100k and 1M rows.

The levels#

L1 through L5 are inside the cut line. L6 needs an external service and L7 needs a second machine, so both are filed rather than attempted — and until L7 runs, any figure comparing a compiled reader (polars, calamine, DuckDB) against an interpreted one states the architecture in the finding. This corpus made the opposite mistake once, in 1.253, and the rule that came out of it is in docs/map/17-the-evidence-ladder.md.

What gets published#

Raw per-run numbers, the pinned versions, the machine, and the generator go in harness/1-057-tabular-data-ingest/, not in the survey directory. The survey carries the findings and a pointer.

L2’s output is a table, not a ratio — for each hazard, what each reader produced. It is the most useful thing this survey can publish and it is not a benchmark, which is why it sits above L3 in the order despite costing more.

What would make L2 a floor model#

L2 is a behavior table over a fixture file, which is the shape that recomputes in a browser: the hazards are small, the fixture is generated rather than downloaded, and the question “what does your reader do with this cell” is one a reader wants answered about their own file. Whether the Python readers here can run under the pinned Pyodide is the gating question and is not assumed — polars, DuckDB and calamine are compiled, and the current lock ships what it ships. Checked before proposing, not after.

What the measurements settle#

The four open questions from S1#

Is calamine faster than openpyxl? Yes, by 15× at 50,000 rows reading directly, and by 7.4× through pandas.read_excel where the difference is one keyword argument. It also uses 29% of openpyxl’s default memory. S1 carried this as (unverified); it is now measured on aarch64.

What do the inference failures cost? pandas returns three of twelve hazard cells wrong with defaults, polars one. The pandas failures split into two separate mechanisms — type inference for the zero-padded identifier, and a nineteen-string default NA list for n/a and NA — which matters because they are disabled by different arguments and a reader who sets dtype=str has turned off only one of them.

Is DuckDB’s rejected-row capture usable? Yes. store_rejects records the physical line, the column name, the error type and the offending text, in a table that can be queried and joined. It is the only option measured where the rejects are data rather than a list built in a loop.

Does pandera’s lazy validation scale? Yes, and so does everything else: cost is flat in the damage rate for all three stacks, from 1% bad rows to 50%. The concern that motivated the question is not supported.

What was not expected#

polars is the heaviest install in the survey, at 176.1 MB against pandas’ 97.1 MB, and two obvious ways of measuring it under-report by a factor of forty because the Rust engine ships as a separate distribution.

A spreadsheet is a safer interchange format than a CSV when types matter. Every xlsx engine returned the text-formatted 00071 correctly, because the format stores the sender’s intent; every CSV reader left to guess lost it. The format with the messy reputation is the one carrying the metadata.

Correctness is nearly free. Configuring pandas for it costs 26% more wall time at a million rows; polars’ two arms land inside the noise of re-running the level. The throughput argument for leaving inference on does not survive contact with the numbers.

Speed and memory rank the field differently. polars is roughly 30× faster than the stdlib module streaming and uses about 34× more memory. DuckDB is roughly 12× faster at a third of polars’ memory. No single ordering answers both, so any recommendation that names one winner has picked an axis without saying so.

Choosing, by what binds#

The combination the measurements point at#

For an import against a known schema that has to report what failed, the stack that came out best on every axis measured is not a product anyone markets: the standard library’s reader in text mode, a per-record model over it, and a spreadsheet writer for the rejects. It scored 12/12 on the hazards, ran fastest of the three validation stacks, used under a fifth of pandera’s memory, produced the most legible failure messages, and costs 7.6 MB installed.

What it cannot do is set-level validation, and it does not stream out of the box — the caller has to not materialize, and L3 shows that choice is worth about 126× in peak memory.

For files large enough that this stops being comfortable, the same job in DuckDB keeps the per-row reporting and drops the memory further, at the cost of expressing the rules in SQL.

polars and pandas remain the right answer when the work after the read is analysis rather than insertion. That is a different job, and the point of L2 is that reaching for the tool built for it, on a file that is going into a table, is what loses the leading zeros.

What each option costs before it reads a byte#

Each library installed into its own empty virtual environment, with the whole site-packages tree weighed afterward. Import cost is the minimum of seven cold subprocess imports in that environment.

(measured 2026-09-09, aarch64 under WSL2, CPython 3.12.3; raw output in harness/1-057-tabular-data-ingest/results/l1-install.json)

polars is the largest install in the survey#

176.1 MB, against pandas’ 97.1 MB. That is close to the opposite of this category’s reputation, in which polars is the lean modern option and pandas the heavy incumbent.

It is also close to invisible to the obvious ways of checking. polars 1.44.2 ships its Rust engine as a separate distribution — polars-runtime-32, carrying a 171.8 MB shared object — so:

• du -sh site-packages/polars reads 9.3 MB;
• the polars distribution’s own RECORD reads 4.3 MB;
• the actual cost of pip install polars is 176.1 MB.

Both cheap measurements were tried here first and both were wrong by a factor of forty. Anyone auditing dependency weight by walking the import-name directory will get this wrong in the same direction, and DuckDB is wrong the same way for the same reason — its 54.5 MB engine ships as _duckdb...so beside the duckdb/ package, so the directory reads 0.5 MB for a 55.1 MB install.

The two extremes are both worth naming#

python-calamine imports in 0.50 ms — about three and a half times faster than the standard library’s own csv module, and roughly 280 times faster than pandas. For 2.3 MB with zero dependencies, it is the cheapest real reader in the survey by every measure taken here, and L4 shows it is also the fastest at the job it does.

pandas costs 141.79 ms of import time and 97.1 MB, before reading anything. For a long-running service that is paid once and never noticed. For a CLI, a serverless function, or anything a user waits on, an import that takes a seventh of a second is a visible part of the budget.

Dependency counts do not track size#

DuckDB and pyarrow have zero dependencies and weigh 55.1 MB and 143.0 MB. pandera has nine and weighs 9.3 MB. Counting dependencies measures supply-chain surface — how many separate projects can break, be yanked, or be compromised — and it says nothing about weight. The two questions are usually conflated and answered with whichever number is handier.

pandera’s 9.3 MB is also incomplete as a total, in the direction that matters: it does not include a dataframe library, and pandera validates dataframes. Its real cost is 9.3 MB plus pandas’ 97.1 MB or polars’ 176.1 MB, whichever the project already carries.

Where the stdlib’s advantage actually is#

Zero bytes and no supply chain, which no other row can claim. Its 1.73 ms import is second only to calamine’s.

L3 shows what that buys and what it costs: the same module is roughly 30 times slower than polars at a million rows, and it is the only reader in the survey whose peak memory can be held flat at 11.2 MB regardless of file size. For a tool that ships to people who did not choose its dependencies, and for files that arrive one at a time rather than by the million, that combination is difficult to beat and easy to overlook.

What each reader does with a hostile file#

Twelve hazards, at known positions in hostile.csv, scored against what a reader that respects an existing text schema must hand back. Rows are located by their row_id value rather than by position, so a reader that loses a row boundary is scored as wrong about that row rather than silently shifting every later answer.

Both counts are published because only the pair settles the question. The lookup is keyed by row_id, which collapses duplicates — so a reader that split one row into two fragments sharing an id, or emitted one twice, would show an unchanged distinct count. Records equal distinct ids for every reader here, which is what says no row was split or duplicated.

(measured 2026-09-09; versions and machine in approach.md; raw output in harness/1-057-tabular-data-ingest/results/l2-correctness.json)

The three cells pandas gets wrong#

The first is type inference: the column reads as integer, and the padding is not part of an integer. polars does this too — both infer zero_padded_id as a 64-bit integer.

The second and third are not type inference, and this is the more useful finding because it is less known. pandas carries a default list of nineteen strings it treats as missing regardless of dtype:

'', '#N/A', '#N/A N/A', '#NA', '-1.#IND', '-1.#QNAN', '-NaN', '-nan', '1.#IND',
'1.#QNAN', '<NA>', 'N/A', 'NA', 'NULL', 'NaN', 'None', 'n/a', 'nan', 'null'

(read from pandas._libs.parsers.STR_NA_VALUES, pandas 3.0.5, 2026-09-09)

The amount column in the fixture infers as string in both pandas and polars, because one value in it is not numeric. polars therefore returns n/a and NA as the text they are. pandas returns null for both, because the NA-string substitution fires after the column is already text. Two libraries, the same inferred type, different values.

NA is the ISO 3166-1 country code for Namibia. A column of country codes read with pandas’ defaults loses every Namibian row to null, and nothing raises.

keep_default_na=False turns the substitution off. It is a separate switch from dtype, so a reader who sets dtype=str believing they have disabled all inference has disabled only half of it.

Empty and missing are the same value to pandas#

The scoring here is charitable and worth being explicit about. The harness renders a null of any flavor as the empty string, which is the most generous reading available — it is what a reader that conflated the two would have to claim. Under that rendering pandas scores the empty_cell hazard correct. It fails literal_na instead, and the failure is the same root cause: an empty cell and a cell containing NA come back indistinguishable, so “not supplied” and “recorded as unknown” cannot be told apart downstream.

DuckDB refuses the file#

The only reader in the survey that declines rather than returning something. Its sniffer raises an InvalidInputException on hostile.csv, defeated by the final row with too few fields:

Invalid Input Error: Error when sniffing file "fixtures/hostile.csv".

The message names the flags that would fix it — null_padding=true, strict_mode=false, ignore_errors=true — and with those it reads the file and scores 12 / 12.

Refusing is a defensible design and arguably the best behavior in the table, because the alternative every other reader chose is to return data that is quietly wrong. Which way it cuts, plainly: DuckDB is the only option here that will not silently hand you a mangled row, and the price is that it will not hand you anything until you have read the error and chosen a flag.

What this settles#

The schema-ownership split from S1 is now measured rather than argued. Every reader that was told the schema — dtype=str, infer_schema=False, explicit DuckDB columns — scored 12 / 12. Every reader left to guess lost cells, and the two guessers lost different cells for different reasons.

The stdlib module and petl score 12 / 12 without being configured at all, because they never guess in the first place. That is the whole of their argument in this category, and it costs them everything L3 measures.

Read cost: time and memory#

clean.csv at three sizes. Every reader returned identical row counts and identical checksums at every size, asserted before any timing was compared.

Wall time, seconds#

Peak resident memory, MB#

(measured 2026-09-09, aarch64 under WSL2, CPython 3.12.3; minimum of five rounds, three at 1M; raw output in harness/1-057-tabular-data-ingest/results/l3-throughput.json, which records the round count per size)

How precise these are#

The three sub-100 ms readers are at the limit of what this method resolves. Re-running the whole level moved polars’ 1M figure between 0.025 s and 0.043 s — a swing wider than the gap between its own two arms — while every reader above 1 s reproduced to within a few percent.

So ratios against the fast readers are order-of-magnitude claims and are quoted that way below. Ratios among them are not supportable at this workload, and the two polars arms in particular should be read as indistinguishable rather than as one beating the other. Separating them needs more rounds or a larger file, which is L7’s territory.

The two tables disagree, and that is the result#

polars reads 1M rows roughly 30 times faster than the stdlib module streaming, and uses about 34 times more memory to do it. DuckDB is roughly 12 times faster than the stdlib module at a third of polars’ memory — 140.4 MB against 380.0 MB.

There is no ranking here that survives being asked about both axes at once, which is why a category comparison that publishes only throughput is answering half a question.

Materializing is the memory decision, not the reader#

The stdlib module appears twice on purpose. Streaming, its peak is 11.2 MB at every size — flat from 10,000 rows to a million, on an 82 MB file. Materialized into a list of dicts, the same module peaks at 1,406.2 MB, the highest figure in the survey and about 126 times its own streaming number.

Nothing about the reader changed between those two rows. What changed is whether a Python dict was built for every row and kept. The cost that gets attributed to “the stdlib is slow and hungry” is mostly the cost of the data structure the caller chose, and it is the one cost in this table that the caller can remove without changing libraries.

petl streams by construction and lands at 23.1 MB flat, which is what the same discipline looks like when the library enforces it rather than leaving it to the caller.

The dataframe readers have no flat mode here#

polars and pandas both build the whole frame. Their memory grows with the file — polars 380.0 MB and pandas 592.5 MB for an 82 MB input, roughly five and seven times the file on disk. scan_csv does not avoid this once collect() is called; it costs slightly more (517.8 MB) than the eager read, because the lazy plan materializes on top of its own scaffolding. Lazy scanning is a way to check a schema and push filters down before reading, which is a real capability, and it is not a way to read a whole file in less memory.

DuckDB is the exception: 140.4 MB for the same file, because it aggregates in the engine and hands Python two numbers instead of a million rows. That advantage disappears the moment the rows are pulled into Python, and it is the strongest practical argument for doing the transformation in SQL rather than around it.

Configuring for correctness is close to free#

The dtype=str arm that scored 12/12 in L2 runs 1.394 s against the default arm’s 1.104 s at 1M rows — 26% slower, and still comfortably the same order. polars pays nothing measurable: its two arms land at 0.025 s and 0.043 s in a range where re-running moves either of them further than that.

Correctness in this category is not bought with throughput, which removes the argument that usually defends leaving inference on.

The spreadsheet engines#

S1 repeated a claim it had not tested: that calamine is substantially faster than openpyxl. It is.

Wall time and peak memory, clean workbook#

(measured 2026-09-09, aarch64 under WSL2, CPython 3.12.3, openpyxl 3.1.5, python-calamine 0.8.2, pandas 3.0.5; minimum of three rounds; raw output in harness/1-057-tabular-data-ingest/results/l4-spreadsheet.json)

At 50,000 rows calamine is 15 times faster than openpyxl’s default path and 10 times faster than its read_only mode, at 29% of the default’s memory.

Through pandas the gap narrows and survives: engine="calamine" against engine="openpyxl" is 7.4 times faster on the same call, same file, same result. That is one keyword argument, and it is the single highest-leverage line in this survey.

read_only=True is worth knowing for anyone staying on openpyxl: it takes 2.583 s to 1.676 s and 262.6 MB to 77.0 MB. It is not close to calamine and it is a substantial improvement over the default for free.

What the engines return for the spreadsheet hazards#

Every engine got the zero-padded column right, which is the opposite of the CSV result and the most useful thing in this file. The cell was formatted as text, and xlsx stores that formatting, so the type does not have to be guessed. A spreadsheet carries type information a CSV cannot, and the hazard that defeats pandas’ CSV reader does not arise here.

The practical consequence runs against intuition: for data whose types matter and whose producer is a person, .xlsx is a safer interchange format than .csv, because the sender’s intent survives the round trip. The format with a reputation for being messy is the one carrying the metadata.

Three ways to read the wrong thing#

Taking the first sheet by position. The fixture’s data sheet is not sheet zero — a prose notes sheet is, which is the ordinary case for a workbook a person has been keeping. read_excel(path, sheet_name=0) returns a 2-row, 1-column frame whose first cell reads “This sheet is not the data. A reader that takes sheet 0 by index…” instead of the 10,000-row table. Nothing raises. Naming the sheet is the fix, and it has to be a decision rather than a default.

The merged banner is row one. Every engine returns it, so read_excel(...) with its default header=0 takes “Quarterly export — do not edit” as the column names and every real column name becomes data. Also silent. The fix is skiprows=1, and knowing to apply it requires having looked at the file.

calamine returns a date, openpyxl returns a datetime. Same cell, same file. Through pandas both normalize to a timestamp, but code reading either library directly gets a different type, and date and datetime do not compare equal in Python. Switching engines for the 7.4× speedup can change a comparison’s result somewhere far from the read.

What pandas will not tell you#

read_excel does not report which engine it used. The probe asked, and there is no attribute to read — the choice is made from the file extension and what is installed. So the maintenance and performance properties of a spreadsheet import depend on a decision that is invisible at the call site and can change when a dependency is added.

Given L1’s numbers, the engine that is chosen by default is also the one that has not shipped a release since 2024-06-28.

What was narrowed from the plan#

The measurement plan said “each of its five engines”. pandas registers five — xlrd, openpyxl, odf, pyxlsb, calamine — and only two of them open .xlsx: xlrd is legacy .xls, odf is OpenDocument, pyxlsb is the binary .xlsb format. Running the other three against an .xlsx fixture would measure a refusal rather than a reader, so the level covers the two that can do the job.

Reporting which rows failed, and why#

Three stacks over 20,000 rows at three damage levels, where a known fraction carry an unparseable amount. Scored on what each can say about a specific bad row, because “an error was reported” and “row 101, column amount, not a number” are different products.

(measured 2026-09-09, aarch64 under WSL2, CPython 3.12.3, pandera 0.33.1, duckdb 1.5.5, pydantic 2.13.5; raw output in harness/1-057-tabular-data-ingest/results/l5-errors.json)

All three scale flatly#

S1 asked whether pandera’s lazy validation survives a file that is substantially wrong rather than slightly. It does, and so do the others: cost is flat in the damage rate. pandera runs 0.362 s at 1% and 0.427 s at 50%; DuckDB 0.155 to 0.123; pydantic 0.089 to 0.110. Half the file failing is no more expensive than one row in a hundred.

That answers the question that motivated the level, and it removes a reason people avoid collecting all failures. The fear that a badly wrong file will make the validator fall over is not supported here.

The reasons are not equally useful#

pandera names the row and the column and then describes the check that fired rather than what was wrong with the value. coerce_dtype('float64') is accurate and it is not something to put in front of whoever sent the file. The other two produce a sentence a person can act on without knowing the library.

pandera reports twice as many failures as there are bad rows#

200 bad rows produced 400 reported failures; 2,000 produced 4,000; 10,000 produced 20,000. Exactly double at every level, because the coercion failure and the column check each raise for the same cell.

Any count taken from failure_cases is therefore not a count of bad rows, and a summary line reading “4,000 rows rejected” built from it would overstate by 100%. Deduplicating on the row index fixes it, and nothing warns that it is needed.

Three libraries, three row numbers for the same row#

The first bad row is row_id 100. It sits on line 101 of the file, counting the header, and at position 99 in a zero-indexed dataframe.

pandera’s number is the frame’s index, which is not the file’s line number and is off by two in the ordinary case of one header row and zero-based indexing. Mapping a reported failure back to the file a person will open in a spreadsheet means adding that offset, and getting it wrong points the sender at the wrong row.

The pydantic figure is only right because the loop was written to track the line — csv gives no row number of its own, so accurate reporting there is the caller’s work.

What this means for the choice#

The lightest option is the one nobody markets: a stdlib reader with a per-record model validating each row. It is the fastest of the three at every damage level, uses under a fifth of pandera’s memory, and gives the most readable reason. What it does not do is anything set-level — a duplicate key, a total that must balance, a foreign key — because a per-record model cannot see other records.

DuckDB’s reject table is real and usable, which S1 could not settle. It records the line, the column, the error type and the offending text, queryable as a table rather than accumulated in a loop. It is the only one of the three where the rejected rows are themselves data you can join against.

pandera’s argument is not error legibility, which is its weakest axis here. It is that the schema is a declarative object covering column types, ranges, membership and cross-column relationships in one place, over a frame that already exists.

Six situations, and what eliminates options in each#

The rest of this pass takes each in turn. The libraries do not sort into these cleanly, and where a situation has two defensible answers both are given with what separates them.

Two situations that look like one#

“The table already exists” and “the file is the authority” are the same job to a library and opposite jobs to the person doing it. Inference is the feature in one and the defect in the other, and it is the same code path. Most bad advice in this category comes from answering one of them with the other’s tool, which is why they lead the list.

What is not a persona here#

Analysis. Someone reading a CSV to plot it, group it, or fit something to it wants a dataframe and should have one; the whole apparatus of schema declaration and per-row error reporting is overhead for a question that will be answered and discarded in ten minutes. That reader is well served by the default advice this category already gives, and nothing in this survey argues otherwise.

The situations below are the ones where the default advice goes wrong.

Picks by situation#

The one decision that changes the most#

Not which library. Whether the reader is allowed to guess.

Five of the six situations above are decided by that question before a library is chosen, and it is settable on every reader measured. Every reader told the schema scored 12 of 12 on the hazard fixture; every reader left to guess lost cells, and the two guessers lost different cells for different reasons. Configuring for it costs 26% more wall time in pandas at a million rows and nothing measurable in polars — not enough to trade correctness for.

Where the same answer keeps appearing#

The standard library’s reader shows up as a first or second choice in three of six situations, which is not what its absence from this category’s comparisons would suggest. It scores full marks on the hazards without configuration, holds 11.2 MB flat at any file size, costs nothing to install, and cannot be abandoned.

What it cannot do is guess, and in the one situation where guessing is the product it is the wrong tool entirely. That is the whole of its profile, and it is more useful stated that way than as a ranking.

Where a persona changes the answer completely#

Somebody outside has to agree the schema is the only situation here with one option, and it is decided by a property no benchmark measures: whether the schema is readable by someone who does not run Python. frictionless has the weakest maintenance signals of any maintained library in the survey and is still the right pick, because the alternative is not a different library but not having a portable schema at all.

That is the clearest demonstration in this survey that library quality and library fit are separate questions, and that a comparison ranking the field on health would recommend the wrong thing here.

What no single library covers#

Ingest is four stops — read, type, validate, load — and nothing measured does all four well. Every working import in the situations above is a combination: a reader that does not guess, a validator that matches the shape of the rules, and something to write the rejects back to whoever sent the file.

The combination that came out best on every axis measured is a stdlib reader, a per-record model, and xlsxwriter for the rejection report: 12/12 on hazards, fastest of the three validation stacks, under a fifth of pandera’s memory, the most legible failure messages, 7.6 MB installed. It is not a product, nobody markets it, and it is assembled rather than adopted — which is why a category comparison that ranks libraries against each other will not find it.

The file is bigger than the machine#

Who#

Someone whose input has outgrown the obvious approach. A log export, a year of transactions, a scientific instrument’s output, a database dump being migrated. The file is not enormous by any absolute standard — it is simply larger than the memory available where the job has to run, which is often a small container rather than a workstation.

The pain#

The read works on the developer’s laptop and dies in the container. Or it works until the file grows 20% and then dies, which is worse, because it worked yesterday.

The instinct is to reach for the fastest reader. L3 says that instinct is measuring the wrong axis: polars is roughly 30 times faster than the standard library streaming and uses about 34 times more memory to do it. Speed and footprint rank the field differently, and this situation is bound by the one that gets published less.

What fits#

DuckDB, which is the strongest answer measured. 140.4 MB peak on an 82 MB file, against polars’ 380.0 MB and pandas’ 592.5 MB, because it aggregates in the engine and hands Python a result rather than a million rows. It was built for out-of-core work, so a file larger than RAM is an ordinary case rather than a technique. The cost is expressing the transformation in SQL.

The stdlib module, streaming. 11.2 MB peak — flat from ten thousand rows to a million, on an 82 MB input. Nothing else in the survey holds a constant footprint regardless of file size. It is roughly 30× slower than polars at a million rows, and for an overnight job that is not a constraint anyone feels.

petl, which is the same discipline enforced by the library rather than left to the caller. 23.1 MB flat, at roughly the stdlib’s speed, with transformations expressed as lazy table operations instead of hand-written loops.

What to avoid#

Any dataframe reader, including the fast one. polars and pandas both build the whole frame, at five to seven times the file’s size on disk. There is no configuration that changes this.

Assuming scan_csv solves it. Lazy scanning is a way to check a schema and push filters down before reading, and it is a real capability. It is not a way to read a whole file in less memory: measured, scan_csv(...).collect() peaked at 517.8 MB against the eager read’s 380.0 MB — more, because the plan materializes on top of its own scaffolding.

Materializing without noticing. The single largest number in this survey is the standard library’s own reader at 1,406.2 MB, and the only difference between it and the 11.2 MB row is a list(). The memory cost that gets blamed on a library is very often the cost of the data structure the caller chose, and it is the one cost removable without changing libraries at all.

The trade-off accepted#

DuckDB means a second engine in the deployment and rules written in SQL rather than Python. For an application that already has a database this is a smaller step than it sounds; for one that does not, it is a real decision.

Streaming means giving up random access. A second pass over the data is a second read of the file, and anything requiring the whole dataset at once — a sort, a join against itself, a median — is either out of reach or has to be pushed to something that can do it, which is the argument that leads back to DuckDB.

How it is decided#

Measure peak RSS, not wall time, and measure it at the size the file will be in a year rather than the size it is now. If the transformation is expressible in SQL, DuckDB wins on the numbers. If it must stay in Python and the job can be done in one pass, the standard library’s 11.2 MB is difficult to argue with.

The table already exists#

Who#

Someone maintaining an application with a database behind it. The schema is written down — in migrations, in an ORM model, in a CREATE TABLE — and it is the authority. A file arrives that has to become rows in that table.

The distinguishing fact is not the size of the file or the speed of the reader. It is that two descriptions of the data now exist, the table’s and whatever the reader decides, and nothing in the usual tooling makes them reconcile.

The pain#

The import runs. It reports success. Three weeks later somebody notices that account 00071 is in the system as 71, and that every customer in Namibia has a null country code because their NA was read as missing.

Neither of those raised an error, and neither is a bug in any library. They are what inference does when it is not told the answer, measured at three of twelve hazard cells for pandas with defaults and one for polars.

The second pain is discovering, months in, that the import has no idea which rows it skipped. It loaded 3,997 of 4,000 and there is no record of the three.

What fits#

Any reader, told the schema. Every reader measured scored 12/12 on the hazards once it was told not to guess — dtype=str with keep_default_na=False for pandas, infer_schema=False for polars, explicit columns for DuckDB, or the stdlib module, which never guessed in the first place. This is a configuration decision more than a library decision, and it is the single highest-value line in this survey.

polars with an explicit schema, where the target types are known and should govern the read. It parses to the declaration and raises where the file disagrees, so a mismatch surfaces at the read rather than at the insert. scan_csv reads the header alone, which means the columns can be checked against the table before any data is paid for.

The stdlib module with a per-record model, where the schema is already expressed as application objects. One definition rather than two that can drift, and L5 measured it as the fastest and lightest of the three validation stacks, with the most legible failure messages.

What to avoid#

pandas with defaults, which is the most likely thing to be reached for. Not because it is a poor library, but because its two independent guessing behaviors — type inference and a nineteen-string default NA list — are both on, are disabled by different arguments, and fail silently. dtype=str alone leaves the second one running.

Assuming dtype=str is the whole fix. It is half of it.

The trade-off accepted#

Configuring for correctness costs 26% more wall time for pandas at a million rows, and for polars the two arms are inside the noise of re-running the level. For an import measured in seconds that is not a cost anyone will feel, and it is the reason the throughput argument for leaving inference on does not hold here.

The cost that lands is having to know the schema and write it down twice — once for the table and once for the reader — unless the reader can be pointed at the definition that already exists. That is the argument for a per-record model over a dataframe schema in this situation, and it is why an ORM-shaped project and a dataframe-shaped project end up in different places.

How it is decided#

Does a definition of these records already exist in the codebase? If yes, validate against that definition and use the lightest reader that will feed it. If no, declare the schema at the reader and let it raise.

Either way the file does not get a vote on what the types are.

The file is the authority#

Who#

Someone handed an export nobody has documented, with a hundred columns and a question that is not yet formed. A data dump from a system being replaced, a public dataset, a regulator’s release, a partner’s monthly extract seen for the first time.

The defining fact is that no schema exists to disagree with. There is nothing to check the file against, so the file is what is true and the first job is finding out what it says.

The pain#

Writing coercion by hand for a hundred unknown columns is worse than accepting a library’s guess. A text-mode reader here hands back a wall of strings and leaves every question — is this column numeric, are these dates, how many distinct values does this have — as work.

The second pain arrives later, when exploratory code becomes the import that runs every month, and the inference that was correct for the sample is wrong for the file that arrives in March.

What fits#

pandas with defaults, which is what this situation is for and where its reputation was earned. Inference is doing an enormous amount of work nobody has to write, the ecosystem means an unfamiliar problem has been met before, and 637 million monthly downloads mean the answer to a question about it exists.

polars with defaults, for the same job on a bigger file. Measured at 0.025-0.043 s for a million rows against pandas’ 1.104 s, and its inference is narrower — it declined the two NA-string substitutions pandas made, returning n/a and NA as the text they are.

DuckDB, when the exploration is naturally a set of questions rather than a set of transformations. Querying the file directly, with the sniffer’s decisions inspectable before the data is read, and 140.4 MB peak on an 82 MB file.

What to avoid#

The stdlib module. It is the right answer to the neighbouring situation and the wrong one here — no inference means the exploration is all hand-written, and there is no schema to justify the cost.

Letting the exploratory read become the production import unchanged. This is the specific failure this survey exists to name. The moment there is a table to load into, the situation has changed to a different one with a different answer, and the code does not announce the transition. Inference that was a feature on Monday is a silent-wrongness generator on Friday.

The trade-off accepted#

Both inference paths lose information and they lose different information, which is worth knowing even here. pandas turns nineteen distinct strings into null regardless of dtype; polars keeps them. On a file being explored, a column that reads as 40% missing may be 40% missing or may be full of the word None, and the two libraries will disagree about which.

Chunk-dependent typing is the other one: reading a large file in chunks infers types per chunk, so a column can be integer in one and object in another depending on where the first non-numeric value falls.

How it is decided#

Ask whether anything downstream will compare this data to a definition. If nothing will — the output is a chart, a count, an answer in a notebook — inference is free and convenient. If something will, this is not the situation; it is the previous one.

A person made the file#

Who#

Someone whose input arrives as a workbook because the person who produced it works in a spreadsheet. A colleague’s export, a supplier’s price list, a client’s inventory, a volunteer’s sign-up sheet. The file was built to be looked at, not to be parsed.

This is the most common shape of real ingest and the least represented in comparisons of this category, which mostly benchmark CSV.

The pain#

The file has a banner across the top saying what it is. The column names are on row two. There are three sheets and the data is on the second one. A column of reference numbers is formatted as text so the leading zeros display, and another column’s dates show as 04/03/2026 while the cell holds a serial number.

read_excel(path) returns something. It has “Quarterly export — do not edit” as a column name, because header=0 took the banner. Nothing raised.

What fits#

python-calamine, when the job is reading. Measured at 15× openpyxl’s default and 10× its read_only mode at 50,000 rows, on 29% of the memory, with one interface covering .xlsx, .xls, .xlsb and .ods — which matters here more than anywhere else, because a person who has been keeping this file since 2009 may well send .xls.

engine="calamine" on pandas.read_excel, for a project already using pandas. Same call, same file, same result, 7.4× faster. One keyword argument, and it also moves the import off a library that has not shipped a release since 2024-06-28.

openpyxl, when something other than cell values matters — formatting, formulas, charts, comments — or when the job includes writing a workbook back. It is the only full-featured option, and read_only=True is free and worth taking for reads.

What to avoid#

Trusting the default engine. read_excel does not report which engine it used, and the choice is made from the file extension and what happens to be installed. The properties of the import — speed, memory, maintenance status — depend on something invisible at the call site that changes when a dependency is added.

Taking sheet zero by index. Measured: on the fixture, whose data sheet is not first, read_excel(path, sheet_name=0) returns a two-row frame of prose instead of ten thousand rows, and nothing raises. Name the sheet.

Assuming the header is row one. It very often is not, and the failure produces column names rather than an exception.

The trade-off accepted#

Switching to calamine gives up formatting, formulas and writing. For an ingest path that is usually nothing, and where a formatted output is needed — the report of rejected rows, which this job always turns out to need — xlsxwriter covers it at 0.9 MB with the healthiest maintenance signals in the survey.

The subtler cost is a type change. calamine returns a date where openpyxl returns a datetime for the same cell, and those do not compare equal in Python. A comparison somewhere downstream can start failing on a change made for speed.

The compensation nobody expects#

A spreadsheet is a safer interchange format than a CSV here. Every xlsx engine measured returned the text-formatted 00071 correctly, because the format stores the sender’s formatting and the type does not have to be guessed. The same value in a CSV was lost by both inferring readers.

So the file that feels messy — merged cells, multiple sheets, a banner — is carrying metadata that the tidy-looking CSV export threw away. Asking a colleague to “just send it as CSV” is asking them to delete the type information, and for zero-padded identifiers and dates it makes the import harder rather than easier.

Somebody outside has to agree the schema#

Who#

Someone whose data has consumers they do not employ. A public agency publishing datasets, a standards body, a research group sharing a corpus, an organization receiving the same return from two hundred suppliers.

The schema is not an internal implementation detail. It is an agreement, and both sides have to be able to read it.

The pain#

The schema exists as Python — a pandera object, a pydantic model, an ORM definition — so the authoritative description of the format is executable by one party and opaque to the other. What the consumers get is prose documentation that describes the schema, drifts from it, and cannot be checked against anything.

Every supplier then builds their own interpretation, and the differences surface as rejected submissions that each side believes the other caused.

What fits#

frictionless, which is the only option in the survey answering this. It implements the Frictionless Data specifications — Table Schema and Data Package — and the schema is JSON. It is not Python, it travels with the data, and a consumer working in R or JavaScript or a spreadsheet can read the same document. Errors come back with row and field positions.

Both parties validating against the same artifact is the thing being bought, and no amount of good Python achieves it.

What to avoid#

A Python-object schema as the published contract. It can be excellent internally and it cannot be the agreement, because half the parties to the agreement cannot execute it.

Prose documentation as the contract. It drifts, silently and immediately, and nothing detects the drift.

Assuming a JSON Schema is equivalent. JSON Schema describes JSON documents. Table Schema describes tabular data — column order, primary keys, foreign keys between resources, missing-value conventions — and those are the things two parties exchanging CSVs actually disagree about.

The trade-off accepted#

frictionless carries the weakest maintenance signals of any maintained option here: 240 open issues against 842 stars, the highest ratio in the survey, and a release cadence slower than every other maintained library measured, though the repository was pushed to 2026-08-27.

That is a real risk and it is the wrong reason to reject the choice, because the alternative is not another implementation — it is not having a portable schema. The mitigating fact is that the specification is separable from the library: Table Schema is an open spec with implementations in other languages, so a project that outlives this library’s maintenance still owns a readable, standard description of its data. Depending on a specification is a different risk profile from depending on a package, and S4 takes that up.

How it is decided#

Ask who has to be able to read the schema. If the answer is only code you deploy, keep it in Python and take the better-maintained library. If the answer includes anyone you do not employ, the schema has to leave Python, and there is one option.

The same file arrives every month#

Who#

Someone maintaining an import that runs on a schedule against a source they do not control. A partner’s monthly extract, a payment processor’s settlement file, a supplier’s catalog, an internal system’s nightly dump.

The file is not the problem. The file changing is the problem, and the change is someone else’s decision, made without telling anyone.

The pain#

For eleven months the import runs. In the twelfth, a column is renamed, a currency field starts arriving with a symbol in it, an optional field becomes mandatory upstream, or a column order shifts. The import either fails loudly — the good case — or succeeds and puts subtly wrong data in the table, which is discovered in a quarterly report.

A one-shot validator answers “is this file acceptable” and has no opinion about “is this file like last month’s”, which is the question that would have caught it.

What fits#

Great Expectations, which is built for exactly this and is heavy for anything else. Its product is the memory: an expectation store, profiling that proposes expectations from data it has already seen, and generated documentation describing what the dataset is supposed to look like. Drift becomes an event with a name rather than an anomaly somebody notices.

dlt, which takes the opposite stance and is the newer idea. Rather than treating a schema change as a failure, it infers a schema from the source and evolves it as the source changes, so drift is a managed migration. It is the fastest-growing option in the survey by repository age against adoption, and the right fit when the source is expected to move and the target can follow.

pandera with a version-controlled schema, for a team that wants the check in the codebase and reviewed like code, without adopting a framework. The schema is a Python object in the repository; a change to what is acceptable is a diff someone approves.

What to avoid#

A validator with no history. A per-record model checks that this row is well-formed. It cannot tell you that 4% of rows had a null in this column last month and 40% do today, because it has never seen last month.

Trusting the reader’s inference to be stable. Inference is a function of the data, so the same code produces different types for different files. A column that has been integer for a year becomes text the first month somebody types n/a into it, and every downstream comparison changes behavior with no code change.

Great Expectations for a single import. The apparatus wants a directory, a context and a run lifecycle. For one file, once, that is most of the work.

The trade-off accepted#

Great Expectations is the only library in this survey declaring an upper Python bound, excluding 3.14. An application already on a newer interpreter cannot adopt it, and that is a constraint that gets discovered late because it looks like a dependency detail rather than a strategic one.

dlt’s schema evolution is the mirror-image risk: a target that follows the source automatically will follow it somewhere wrong. Evolution is right when the source is authoritative and the target is a warehouse; it is wrong when the target is an application whose code expects particular columns.

How it is decided#

Ask what should happen when the file changes. If the answer is “stop and tell someone”, the schema belongs in version control and the check should fail. If the answer is “keep the data flowing and reconcile later”, schema evolution is the right machinery. Those are different businesses, not different preferences, and the tools do not substitute for each other.

openpyxl: what a finished library means for code with a long life#

309,869,897 downloads a month. Last release 2024-06-28. Two commits on the default branch since, both dated 2025-10-01, one of them a revert. It is the default .xlsx engine for pandas.read_excel, so a large fraction of all spreadsheet reading in Python runs through it whether or not anyone chose it.

This is the most consequential viability question in the category, and it deserves a more careful answer than the release gap alone supports.

The case that this is fine#

The format stopped moving. xlsx is an ISO-standardized format that has not changed in any way that affects a reader for years. A library that reads a frozen format can be finished in a way that a web framework never can, and “no releases” and “no unfixed need” look identical from outside.

The measured behavior is correct. L4 put it against the hazard workbook and it returned the merged banner, the text-formatted identifier and the date cell correctly — the same answers as the actively developed alternative, except for a datetime-versus-date difference that is a design choice rather than a defect.

The download figure is not inertia alone. It is the pandas engine default, so much of it is machinery rather than choice, and it is also the only full-featured option: reading and writing, formatting, formulas, charts and comments. Nothing else in the category does that set, so there is no library it is losing to on capability.

The case that it is a real risk#

Nobody should plan on a fix. This is the operative sentence and it does not depend on whether the project is healthy. A bug encountered today in openpyxl is a bug to work around, and code that will be maintained for five years should be written knowing that.

Contribution is throttled by the forge. Mercurial on Heptapod is a different version control system and a different host from the one nearly every Python contributor uses. The barrier to sending a patch is higher than for any other library in this survey, and the 2025-10-01 commits — a merge and a revert of an accidental merge — are what a project with few hands looks like.

Its invisibility compounds the risk. Because it has no stars, it appears in no health-ranking of the category, so a team doing exactly the right kind of dependency review will not see it flagged. The library most likely to be an unmaintained dependency is the one least likely to show up in a scan sorted by repository signals.

The bus factor is unmeasurable from outside. For every other library here the contributor graph is a public API call. Here it is not, and absence of a signal is being read by most people as absence of a problem.

What this means by time horizon#

Writing a spreadsheet, any horizon. Fine. It is the only full-featured writer, the job is well-defined, and xlsxwriter covers the subset that is only writing with better signals — 30 open issues, 8 per thousand stars, still shipping.

Reading, short horizon. Fine, and calamine is 15× faster for the same result, so the argument to move is performance rather than risk.

Reading, long horizon. Move. Not because openpyxl will break — it probably will not — but because the alternative is faster, actively released, covers .xls and .ods as well, and costs one keyword argument through pandas. When the cheaper option is also the lower-risk one, the decision is not finely balanced.

Depending on it implicitly. This is the case to fix first, and most projects are in it without knowing. pandas.read_excel chooses an engine from the file extension and what is installed, reports nothing about which it chose, and defaults here. Naming the engine explicitly converts an invisible dependency into a decision, and it is a one-line change.

The signal to watch#

A release. Not activity on the tracker, which has continued — the project’s last_activity_at reads 2026-08-31 while its code has not moved since October 2025. Those two facts together are the thing to notice: a tracker taking reports that the code is not answering is a different state from a quiet, finished project, and it is the state a re-check should be looking for.

The readers: what each is exposed to#

stdlib csv#

Part of CPython, specified by PEP 305, covered by the language’s own deprecation policy. The abandonment question every other entry in this survey has to answer does not arise: it outlives any library that wraps it, and no dependency review will ever flag it.

The exposure runs the other way and is real. Because it has no schema, code built on it accumulates hand-written coercion and hand-written error handling, and that code is rarely reviewed as carefully as a schema declaration. L5 measured this indirectly — the accurate line numbers in the pydantic stack are accurate only because the loop was written to track them, and csv offers no row number of its own. What the module saves in dependency risk it can spend in bespoke logic that nothing tests.

Ten-year outlook: it will be there and it will behave identically. Whether the code around it still makes sense is a question about the codebase, not the library.

pandas#

Sixteen years old, 637 million monthly downloads, the largest ecosystem in the category by an order of magnitude. Nothing about its survival is in question, and the practical argument for it — that the answer to any question about it already exists, and that every developer and every coding assistant has met it — is a legitimate input rather than a distraction from one.

The strategic fact for a long-lived project is the 3.0 line’s Python floor of 3.11, the highest of any reader here. For the first time in years, choosing pandas constrains the interpreter rather than the other way round, and a project pinned lower is choosing an older pandas as well.

The risk that matters is not abandonment but silent behavior. Its two independent guessing mechanisms are on by default, are disabled by different arguments, and produce wrong data rather than errors. That is a property to design around permanently, not a bug awaiting a release.

polars#

Five years old, releasing on the day this was written, 39,695 stars against 63 million monthly downloads. Growing on every axis measured.

Three things a five-year outlook has to price:

The install is 176.1 MB, the largest in the survey, and its Rust engine ships as a separate distribution that makes the cost invisible to the two obvious ways of checking. A dependency budget audited by directory size will be wrong by a factor of forty.

The release cadence is fast — 1.44.2 within a 1.x line. Frequent minor releases are a sign of life and a maintenance cost for an import script nobody will touch for two years.

The ecosystem gap is wider than the download gap. A tenth of pandas’ installs understates the difference in tutorials, answered questions and third-party integrations. A team meeting an unusual file will find fewer people who have met it.

Against those: it declined the NA-string substitutions pandas made, its schema is a first-class argument rather than an override, and it was the fastest reader measured by a wide margin. The technical position is strong and the ecosystem position is not, which is the same shape 1.212’s agent-tooling cluster found elsewhere.

DuckDB#

The healthiest signals in the reader field: 868 open issues against 41,105 stars, 21 per thousand, the lowest ratio among the engines, with zero Python dependencies and 55.1 MB installed.

Its strategic profile is unusual because it is a database, and adding one to a deployment is a decision with a blast radius beyond this category. For a project that already has one, DuckDB is a library-shaped addition. For one that does not, the choice being made is larger than “which CSV reader”, and it should be made on those terms.

The behavior L2 found — refusing a malformed file rather than returning something wrong — is the most conservative default in the survey and the best fit for long-lived imports. It is also the behavior most likely to be configured away by whoever hits it first at 2am, and the flags that do so are named in the error message.

python-calamine#

The youngest and smallest, 497 stars, 0.8.2, one binding maintainer visible. Ordinarily that profile argues for caution.

Two things change the reading. It is a registered pandas engine, so it has a large installed base that neither its stars nor its 6.6 million downloads capture, and a dependency of pandas’ extension surface is harder to abandon quietly than a standalone package. And it is a thin binding over the Rust calamine crate — the risk concentrates in the binding rather than the engine, and a binding is a smaller thing to replace or fork than an implementation.

The 0.x version number and the compiled-wheel requirement are real. A platform without a wheel means building Rust, and that is a deployment constraint the pure-Python alternative does not have.

petl#

Fifteen years old, released the day this was written, 1,317 stars and 780,935 monthly downloads. Small, and small for a long time, which is a different profile from small and new.

Continuous maintenance over that span with no ambition to grow is a reasonable thing to depend on for a job that also does not change. Its 23.1 MB flat memory profile and zero-dependency install make it the lowest-risk way to get streaming discipline enforced by a library rather than by the caller’s care.

Strategic picks and the risks they carry#

Three paths#

Conservative. stdlib csv in text mode, a per-record model for validation, xlsxwriter for the rejection report, and python-calamine or openpyxl behind a named engine= argument for spreadsheets. Total install 7.6 MB plus the spreadsheet reader. Nothing here can be abandoned in a way that strands the project: the reader is CPython, the writer has the healthiest signals in the survey, and the validator is usually already present. The cost is hand-written coercion and no set-level rules.

Performance-first. polars with an explicit schema for reading, pandera for validation, calamine for spreadsheets. Fastest measured on every read axis. The prices are a 176.1 MB install that two obvious audit methods under-report by forty times, a fast release cadence against a long-lived script, and a validator whose backlog density is the second-highest here.

Out-of-core. DuckDB for reading, typing, filtering and rejecting, with store_rejects as the error report. The best memory profile measured — 140.4 MB on an 82 MB file — the healthiest engine signals, and the most conservative default behavior in the survey, which is refusing a malformed file rather than returning something wrong. The price is a second engine in the deployment and rules expressed in SQL.

What to fix first, whichever path#

Name the spreadsheet engine. pandas.read_excel chooses from the file extension and what is installed, reports nothing about which it chose, and defaults to a library whose last release was 2024-06-28. Passing engine= explicitly converts an invisible dependency into a decision and, with calamine, is 7.4× faster on the same call. One line.

Turn off both of pandas’ guessing mechanisms, not one. dtype=str disables type inference. It leaves the nineteen-string default NA list running, which is what turns NA and n/a into null regardless of dtype. keep_default_na=False is the other half.

Risks ranked by how quietly they arrive#

Every one of those is silent. None produces a stack trace, and the ordering is by how long it takes to notice rather than by severity — which for this category is the more useful axis, because a loud failure in an import is the good case.

What a re-check should look at#

Decay class: fast. Two of the four things this survey measured could change within a year.

• An openpyxl release. It would change the strongest recommendation here. Tracker activity is not the signal — that has continued while the code has not moved. A release is.
• calamine’s version number leaving 0.x, which would remove the main structural objection to the fastest spreadsheet reader.
• pandas’ NA-string behavior, since 3.x is a young major line and this is the kind of default a major version can revisit. If it changes, L2’s central finding changes with it.
• The Frictionless specification’s own activity, which is the dependency actually standing behind the portable-schema recommendation and does not appear on PyPI.

What will not move: the standard library’s behavior, the shape of the four-stop path, and the fact that a spreadsheet carries type information a CSV cannot.

The validation layer: where the risks differ from the readers#

pandera#

421 open issues against 4,451 stars — 95 per thousand, the second-highest density in the survey. For a validation library that number deserves more weight than it would elsewhere: unfixed edge cases in a checker are cases that pass when they should not, so a backlog here is not the same kind of debt as a backlog in a reader, where a defect usually announces itself.

Two structural facts a five-year view has to hold. It supports both pandas and polars, which makes it the one validation choice that does not lock the reader decision — real strategic value, because the reader is the thing most likely to be revisited. And its cost is never its own: 9.3 MB plus whichever dataframe library it validates, so its real footprint is 106 MB with pandas or 185 MB with polars.

L5 found its weakest axis is one nobody lists as a feature: it reports the check that fired rather than what was wrong with the value, and it double-counts failures. Both are fixable by the caller and neither is visible until an import fails in front of someone.

Great Expectations#

The healthiest signals of any library in the survey — 41 open issues against 11,778 stars, 3 per thousand — and the only one carrying an upper Python bound, excluding 3.14.

That ceiling is the strategic fact. Every other library here constrains the floor; this one constrains both ends, which means an interpreter upgrade can strand it. A project on a release cadence that tracks CPython closely should treat the ceiling as a scheduling dependency rather than a packaging detail, and check it before an upgrade rather than during.

Its weight is deliberate and it is the reason the fit is narrow. The store, the profiling and the generated documentation are the product, and they are worth their configuration cost only when the same source arrives repeatedly. Adopted for a single import, most of what is being maintained is never used.

frictionless#

The weakest signals of any maintained option: 240 open issues against 842 stars — 285 per thousand, three times the next highest — and a release cadence slower than everything else here, with 5.19.0 dating from 2026-04-13 against a repository pushed 2026-08-27.

Taken as a library comparison that is a clear argument against. Taken strategically it is not, and the reason is the most important thing in this file:

The dependency is on a specification, not only on a package. Table Schema and Data Package are open specifications with implementations in other languages. A project that outlives this library’s maintenance still owns a readable, standard, portable description of its data, and can move to another implementation or write against the spec directly. That is a materially different risk profile from depending on a package whose format is its own invention, and it is why the weakest-maintained library in this survey can still be the right long-horizon choice for the one job it does.

What would change the assessment is the specification stalling rather than the library. That is the signal to watch, and it is not on PyPI.

pydantic#

Not a tabular library and the lowest-risk entry in this layer, because for most projects adopting it is not a new dependency at all — it is already present, usually via a web framework. Validating an import against the model the application already defines means one definition rather than two that can drift, which is a maintenance property rather than a performance one.

Its ceiling is structural and permanent: a per-record model cannot see other records, so uniqueness, referential checks and dataset-level thresholds are out of reach by design. That is not a gap awaiting a release. A project whose rules grow set-level will outgrow it, and the successor is pandera or SQL, not a newer pydantic.

dlt#

Four years old, 5,837 stars, 427 open issues, growing fastest of anything here. Apache-2.0, commercially backed.

The strategic question is not survival, it is direction. Schema evolution — following the source as it changes — is the right machinery for a warehouse and the wrong machinery for an application whose code expects particular columns. A tool that makes drift painless makes drift invisible, and invisible drift into an application schema is the failure this survey’s first persona is about.

Commercial backing cuts both ways on a five-year horizon, and this corpus has a rule about it: funding answers will this survive, never is this good. It survives. Whether its model fits is a question about the project adopting it.

The viability signals#

Every figure read from PyPI, the GitHub API and pypistats on 2026-09-09, and recorded in harness/1-057-tabular-data-ingest/registry-2026-09-09.json so any of it can be re-fetched.

How to read the issue ratio#

Issues per thousand stars is a crude number and it is here because the raw counts mislead in both directions: 2,738 open issues sounds alarming for pandas and is unremarkable for a project of that size, while 240 sounds mild for frictionless and is the highest density in the table by a factor of three.

It measures attention against backlog, and it cannot distinguish a project that triages aggressively from one that receives few reports. Great Expectations at 3 per thousand and csvkit at 6 are doing different things to arrive at similar numbers — one is actively curated, the other is a finished tool nobody files against.

The one row without repository figures#

openpyxl has no star count because it is not on GitHub. Its source is Mercurial at foss.heptapod.net, and this is not a footnote: every star-based comparison of Python spreadsheet libraries silently omits the most-downloaded one. Its 309.9 million monthly downloads are more than pandas’ nearest specialist competitor by two orders of magnitude and it appears in no ranking that sorts by stars.

Read from the Heptapod API on 2026-09-09: the default branch carries two commits dated 2025-10-01, one reverting an accidental merge, and before those the 2024-06-28 release commits. The project’s last_activity_at reads 2026-08-31, which is tracker traffic rather than code.