Sharing a folio
A folio is a directory of ordinary files with a readable catalog. That is what makes sharing simple: most of this page is about which ordinary mechanism to use, not about DataFolio features.
Put it where the team can see it
The path can be local or cloud storage. Nothing else changes.
folio = DataFolio("gs://team-analysis/experiment-12")
folio.add("features", df, description="…")
folio.get("features")
Credentials come from your environment via cloudfiles — the same configuration your other GCS/S3 tooling uses. DataFolio holds no credentials of its own and never records them.
For a public bucket you only want to read:
folio = DataFolio("gs://public-data/atlas-v3", use_https=True)
http(s):// paths are always opened read-only.
Give collaborators the least dangerous thing that works
A path, for people who do not use DataFolio. Every item has a real, openable location:
folio.item_path("features")
# 'gs://team-analysis/experiment-12/tables/features--r2.parquet'
folio.describe(show_paths=True) # the whole inventory with paths
They run pd.read_parquet(...) and are done. Or point them at
CONTENTS.md in the directory, which lists every item, its location, and its
description in a Markdown table — no tooling required. See
Reading a folio without DataFolio.
A read-only open, for people who do:
folio = DataFolio("gs://team-analysis/experiment-12", read_only=True)
folio.get("features") # fine
folio.add("x", df) # RuntimeError: Cannot modify a read-only DataFolio
Use this in any notebook that is only meant to consume a shared folio. It costs nothing and removes a whole category of accident.
A frozen, self-contained folio, for a submission or a handoff:
folio.create_snapshot("paper-v1", description="Figures 2–4")
folio.export_snapshot("paper-v1", "gs://team-analysis/paper-final")
The export contains only that snapshot's items — no history, no scratch work.
A fork, for someone continuing the work:
folio.copy("gs://team-analysis/experiment-13",
metadata_updates={"parent": folio.path})
Multiple readers, one writer
Several people (or notebooks) can read one folio at the same time. A reader picks up another process's committed writes automatically on its next read, so this works without ceremony:
# notebook A
folio_a.add("results", df)
# notebook B, already open
folio_b.get("results") # sees it
folio_b.refresh() # or force a reload explicitly
Writing is where the guarantees narrow, and they are worth stating exactly.
Locally, writes are serialized by a per-folio lock, and the whole mutation — payload write plus catalog publish — happens under it. A writer whose view is behind the on-disk state is rejected rather than allowed to overwrite newer work:
folio.add("results", df)
# ConcurrentWriteError: items.json was modified by another writer (on-disk
# revision 12 > loaded 11). Call refresh() and re-apply your change (datafolio
# supports many readers but one writer per bundle).
folio.refresh() # reload, then re-apply
New payloads always get fresh versioned filenames, so an interrupted or stale
writer can never damage a file the committed catalog points at. The cost is
that a failed write may leave an unreferenced orphan file, which is harmless
and swept only when you ask (cleanup_orphaned_versions()).
On cloud storage, object stores do not offer the primitive that lock depends on. Stale writers are usually still detected — the catalog revision check runs the same way — but two simultaneous cloud writers can race.
Treat a cloud folio as single-writer
One person (or one job) writes; everyone else opens with read_only=True.
DataFolio does not implement distributed locking and is not going to. If
you need genuine multi-writer coordination, you need a database, not a
directory.
If you want a block of reads to see one consistent state — and to skip the per-read staleness check, which costs round trips on cloud storage — pin it:
with folio.pinned():
for name in folio.tables:
summarize(folio.get(name))
Moving a folio around
A folio is plain files with a relative-path catalog, so ordinary sync tools work and produce a complete, working folio:
gsutil -m rsync -r gs://team-analysis/experiment-12 ~/analysis/experiment-12
aws s3 sync s3://bucket/experiments/exp-12 ~/analysis/exp-12
rclone sync remote:bucket/exp-12 ~/analysis/exp-12
rsync -a colleague-machine:/data/exp-12/ ~/analysis/exp-12/
offline = DataFolio("~/analysis/experiment-12") # works, including snapshots
Use a sync tool when you want a mirror (same history, same snapshots); use
folio.copy() when you want a fork (current versions only, fresh identity).
One thing does not travel: external references still point where they always
pointed. Syncing a folio does not bring the referenced dataset with it, and
the new reader may not have access to it. folio.validate() will tell you:
folio.validate()
# {'features': True, 'raw_measurements': False, …} # reference unreachable from here
Trust
Opening a folio someone else wrote means reading files they wrote.
get_model()on a joblib model executes pickle. Only load models from folios you trust. Models saved withcustom=True(skops) refuse unknown types unless you passtrusted=Trueafter reviewing the type list — see Models.- Tables, arrays, and JSON are inert data formats; reading them is safe.
- Checksums are recorded at write time and verified by
validate(). They detect corruption and truncation, not a deliberate substitution by someone with write access to the bucket.
Next
- Reading a folio without DataFolio — what to tell a collaborator who will never install it.
- Snapshots — naming the state you are handing over.
- What DataFolio is not.