Skip to content

Tables: big, external, and lazy

Tables are the most common thing in a folio, and the only kind of item where size forces decisions. This page covers the normal case in four lines and then the two situations that actually need thought: the table is too big to load, or the table is not yours.

Parquet is the canonical format. pandas is the default frame. Everything else is opt-in.

The normal case

folio.add("features", df)                     # pandas or Polars, in it goes
df = folio.get("features")                    # pandas DataFrame back
pl_df = folio.get("features", frame="polars") # Polars DataFrame back
lf = folio.scan_table("features")             # lazy Polars scan

What was recorded alongside it:

folio.item_info("features")
# {'item_type': 'included_table', 'filename': 'features--r2.parquet',
#  'num_rows': 3, 'num_cols': 3, 'columns': ['cell_id', 'width', 'depth'],
#  'dtypes': {'cell_id': 'int64', 'width': 'double', 'depth': 'double'},
#  'size_bytes': 2290, 'checksum': '…', 'description': '…', …}

Column names, dtypes, and row count are in the catalog, so describe() and any reader can answer "what is in this table" without opening it.

Indexes, Series, and LazyFrames

A non-default index is dropped. Parquet stores columns. DataFolio warns when it drops one, and gives you the escape hatch:

folio.add("indexed", df)
# UserWarning: Table 'indexed' has a non-default pandas index that will NOT be
# stored (parquet keeps columns only). Call reset_index() first to keep it as a
# column, or pass preserve_index=True to store it.

folio.add("indexed", df, preserve_index=True)   # index stored as ordinary columns

With preserve_index=True the index becomes real columns (readable by any tool) and is recorded in the catalog as index_columns, so a pandas get() restores it. Polars and direct file readers just see the columns.

A Series is stored as a one-column table. add() accepts a pandas or Polars Series and keeps its index (a default RangeIndex is skipped). The catalog records it as series (with the original name), so a pandas get() returns a Series again. Polars and direct file readers see a one-column table.

folio.add("counts", df["cell_type"].value_counts())
folio.get("counts")    # pandas Series, index and name restored

Writing a LazyFrame is streamed. add() accepts a Polars LazyFrame and sinks it to Parquet with bounded memory — the full result is never held at once:

lf = pl.scan_parquet("raw/*.parquet").filter(pl.col("keep")).select(["a", "b"])
folio.add("clean", lf)

Cloud writes are streamed to a temp file and uploaded, so they are bounded too.

Bringing in a table file

A parquet, CSV, or feather file on disk doesn't need to go through a DataFrame first:

folio.import_table("cells", "~/Downloads/cells.parquet")
folio.import_table(None, "measurements.csv")      # name from the file: 'measurements'

The result is an ordinary included table. import_table never holds the table in memory, so it works for files larger than RAM:

  • parquet is copied byte for byte, with no re-encoding.
  • feather / Arrow IPC is converted one record batch at a time.
  • CSV is parsed and converted one block at a time. Column types are inferred from the first block. A later value that doesn't fit raises ValueError; read that file with pandas and add() it, or keep it unchanged with add_file().

Writing to a cloud folio stages the parquet file locally first, so you need local disk space for the output. datafolio add in the CLI uses the same path.

When the table is too big to load

DataFolio has no query API and will never grow one. For anything that should not be loaded whole, its job is to hand you a lazy frame or a path and get out of the way.

An eager get() above the folio's ceiling (max_eager_bytes, 500 MB by default) refuses rather than quietly filling memory:

folio.get("transactions")
# ValueError: Table 'transactions' is ~1433.6 MB, above the ~500.0 MB eager-load
# limit. Use scan_table('transactions') for a lazy polars scan with
# predicate/projection pushdown (pip install 'datafolio[polars]' if needed), or
# pass allow_full_load=True (or set max_eager_bytes=None) to load it all anyway.

With Polars — the recommended path:

import polars as pl

df = (
    folio.scan_table("transactions")     # lazy: nothing read yet
    .filter(pl.col("amount") > 1000)     # pushdown: filtered at the file
    .select(["user_id", "amount"])       # pushdown: two columns read
    .collect()                           # I/O happens here
)

Everything Polars can do works on that LazyFrame — joins, window functions, collect(engine="streaming") for larger-than-memory results. See the Polars lazy API. This is also the only path for sharded datasets, and it works on external references without downloading them.

scan_table() is honest about laziness. It streams (footer and range reads, no up-front download) for local paths, s3://, gs://, az://, and http(s):// where the server supports range requests. For a location it cannot scan lazily, it raises rather than silently downloading the object and calling it lazy. If you want the download, ask for it with get(name, frame="polars").

Staying in pandas: take the path and use pyarrow directly. Column pruning and predicate pushdown are pandas/pyarrow features, not DataFolio ones.

import pandas as pd

df = pd.read_parquet(
    folio.item_path("transactions"),
    columns=["user_id", "amount"],
    filters=[("amount", ">", 1000)],
    engine="pyarrow",
)

Overriding the guard when you really do want it all:

folio.get("transactions", allow_full_load=True)
DataFolio(path, max_eager_bytes=None)          # disable the guard for this folio

When the table is not yours

reference_table() catalogs external data without copying it. Use it for released datasets, lab archives, anything large or owned by someone else.

folio.reference_table(
    "raw_measurements",
    "gs://lab-data/run-12/measurements.parquet",
    description="Published source table; not owned by this folio",
)

Creating a reference is a cheap, offline catalog write. It makes no network requests — no stat, no schema read, no existence check — so linking a private or currently-unreachable URI never blocks on credentials or the network.

Read it exactly like any other table:

folio.get("raw_measurements")                  # eager (subject to the size guard)
folio.scan_table("raw_measurements")           # lazy, no download
folio.item_path("raw_measurements")            # 'gs://lab-data/run-12/measurements.parquet'

Enriching a reference

When you do want schema, size, row count, and point-in-time source identity recorded, ask for it explicitly. This performs I/O and raises actionable errors if the object is unreachable:

info = folio.inspect_table("raw_measurements")
info["columns"], info["num_rows"], info["size_bytes"], info["source_identity"]

inspect_table() works on included tables too, refreshing their recorded stats.

What a reference does and does not promise

A reference is a link. DataFolio never copies, owns, or freezes the bytes at the other end.

  • The referenced data can change or disappear without the folio noticing.
  • Reading it may need credentials the folio knows nothing about.
  • validate() reports False for a reference that is unreachable from here — that is "could not verify", not necessarily "gone".
  • Snapshots preserve the link, not the content. folio.mutable_references() lists them, and get_snapshot_info() flags them in any snapshot.

If you need the bytes frozen, add() a copy instead of referencing it. That is the tradeoff, stated plainly: cheap and non-duplicating, or owned and guaranteed. Pick per table.

Sharded and partitioned datasets are Polars-only

A reference to a directory of Parquet shards (including hive-partitioned layouts) is marked polars-only: lazy scans and Polars frames work, plain pandas get() raises a clear error because pandas mishandles such layouts.

folio.reference_table("events", "s3://bucket/events/")   # hive-partitioned dir
folio.scan_table("events").collect()                     # ok
folio.get("events")                                      # raises: polars-only

DataFolio reads sharded data. It does not create sharded layouts.

Formats

Parquet is canonical; CSV is accepted for references. Delta and Iceberg are rejected up front with an actionable error — convert to plain Parquet and reference that. Supporting table formats with their own transaction logs would mean owning a second consistency model, which is exactly the kind of weight DataFolio declines.

pip install 'datafolio[polars]'

Polars is an optional extra. Without it, pandas access works normally and the lazy methods raise an ImportError that says so.

Next