Skip to content

Migrating to DataFolio 2.0

DataFolio 2.0 consolidates the item API: one add() and one get() replace the per-type method families, plus a handful of explicit verbs where being explicit matters.

The on-disk format is upgraded to manifest v2: items.json becomes the single authoritative file (metadata and snapshots embedded, one revision, one atomic commit). Folios written by 1.x open in 2.0 with no manual step and migrate on your first write (the old metadata.json/snapshots.json sidecars are then removed). A folio written by 2.0 cannot be written by 1.x — old versions refuse it cleanly rather than corrupting it.

Method mapping

1.x 2.0
add_table(name, df, ...) add(name, df, ...)
add_numpy(name, arr, ...) add(name, arr, ...)
add_json(name, x, ...) add(name, x, ...)
add_timestamp(name, dt, ...) add(name, dt, ...) (timezone-aware datetime)
add_data(name, x, ...) add(name, x, ...)
add_data(name, reference=p) reference_table(name, path=p)
add_sklearn(name, m, ...) add_model(name, m, ...)
add_artifact(name, path, ...) add_file(path, name=name, ...)
get_table(name) get(name)
get_table(name, frame='polars') get(name, frame='polars')
get_numpy / get_json / get_data / get_timestamp get(name)
get_sklearn(name) get_model(name)
get_lazy(name) scan_table(name)
get_artifact_path(name) get(name) or item_path(name)
get_table_path / get_model_path / get_numpy_path / get_json_path / get_timestamp_path / get_data_path / get_item_path item_path(name)
get_table_info / get_model_info / get_artifact_info item_info(name)
cache_status / clear_cache / invalidate_cache / refresh_cache removed (caching subsystem cut)
DataFolio(..., cache_enabled=, cache_dir=, cache_ttl=) removed
code= (on any add method / update_item) removed — git owns code history
add_model(..., hyperparameters=) removed — estimator.get_params() owns this; store it as a JSON item if you want it in the folio
add(..., models=[...]) on tables removed — put model names in inputs (old manifests' models fields still count as lineage on read)
reproduce_instructions() / datafolio snapshot reproduce removed — the captured git/env data is in get_snapshot_info()
folio.data.name.polars get(name, frame='polars') or scan_table(name).collect()

Unchanged: reference_table, inspect_table, scan_table, describe, list_contents, delete, update_item, archive/unarchive, copy, validate/is_valid, batch, refresh, all lineage methods, the whole snapshots API, folio.data accessor syntax, and folio.metadata.

Behavior changes

Overwrite is uniform. Replacing any existing item requires overwrite=True — including items pinned by a snapshot (the pinned version is preserved via copy-on-write). In 1.x, snapshotted tables could be silently replaced; that no longer happens.

Strings are always JSON. add(name, "results.txt") stores the string. In 1.x, a string was stored as a file artifact whenever a file by that name happened to exist in the working directory. Files enter the folio only via add_file(path).

Model auto-detection is narrower. add() recognizes actual estimator instances (scikit-learn BaseEstimator, XGBoost, LightGBM, CatBoost). An arbitrary object that merely has fit/predict methods must be stored explicitly with add_model() — pickling something is an explicit act.

Numbers are JSON, timestamps are datetimes. add(name, 1705318200) stores a JSON number. To store a timestamp, pass a timezone-aware datetime:

from datetime import datetime, timezone
folio.add('start', datetime.fromtimestamp(1705318200, tz=timezone.utc))
folio.get('start', as_unix=True)  # -> 1705318200.0

Unknown options raise. A typo'd keyword (folio.get('cfg', colums=[...])) now raises TypeError instead of being silently ignored.

skops models require explicit trust. Models saved with custom=True (skops format) refuse non-standard types on load unless you opt in: get_model(name, trusted=True). Joblib-format models are pickle — only load folios you trust.

Object-dtype numpy arrays are rejected at add() time. In 1.x the write succeeded and every read failed. Convert to a concrete dtype, or store via add_model().

A non-default pandas index warns. Parquet stores columns; a non-default index is dropped. 2.0 warns when this happens and adds an escape hatch: add(name, df, preserve_index=True) — this stores the index as ordinary columns (readable by any tool) and records them in the manifest as index_columns, so the pandas read path can set_index() them back. polars and direct file readers simply see the columns.

Snapshots no longer embed requirements.txt. capture_environment=True records the Python version, platform, and uv.lock hash; the full dependency text belongs to git.

Item names are validated. Names are segments of letters, digits, ., _, - (starting with a letter or digit), optionally namespaced with / (e.g. 'examples/weights'). Names containing .., leading _, or spaces are rejected.

Snapshot integrity fixes (also in 2.0): delete() of a snapshotted item preserves the pinned version; restore_snapshot(confirm=True) also restores items deleted after the snapshot; create_snapshot() inside batch() raises; update_item() on a snapshotted item no longer edits the snapshot's view; diff_from_snapshot() defaults to the newest snapshot.

Mechanical migration

Most code migrates with search-and-replace:

# in your notebooks/scripts
sed -i '' \
  -e 's/\.add_table(/\.add(/g'  -e 's/\.add_numpy(/\.add(/g' \
  -e 's/\.add_json(/\.add(/g'   -e 's/\.add_data(/\.add(/g' \
  -e 's/\.add_sklearn(/\.add_model(/g' \
  -e 's/\.get_table(/\.get(/g'  -e 's/\.get_numpy(/\.get(/g' \
  -e 's/\.get_json(/\.get(/g'   -e 's/\.get_data(/\.get(/g' \
  -e 's/\.get_timestamp(/\.get(/g' -e 's/\.get_sklearn(/\.get_model(/g' \
  -e 's/\.get_lazy(/\.scan_table(/g' \
  your_script.py

Then handle the argument-order change for add_artifact → add_file, any add_timestamp(unix_number) calls, and overwrite=True where you relied on silently replacing snapshotted items.