Package API
datafolio exports one class, three exceptions, and four registry
functions. The class has its own page — see
DataFolio Methods for every method and property.
Everything else in the package (handlers, storage backends, readers) is
internal and not part of the public API.
Folio registry and search
datafolio.find(pattern='*', *, regex=False, item_type=None, metadata=False, folios=None, aliases_only=False, local_only=False, include_archived=False, case_sensitive=False, descriptions=False, max_workers=DEFAULT_WORKERS)
Find items (or metadata keys) across the folios in the user registry.
Searches every aliased and recently used folio (see
:func:datafolio.list_folios) unless folios narrows the search. Only
each folio's manifest is read, never item data. Folios are read
concurrently; one that can't be opened (moved, deleted, no credentials)
produces a UserWarning and is skipped.
Parameters:
-
pattern(str, default:'*') –Glob pattern matched against the whole item name (
'*'matches everything). Withmetadata=Trueit matches metadata keys, and'key=value'also matches the value's string form. -
regex(bool, default:False) –Treat the pattern(s) as regular expressions, matched anywhere in the string (
re.search) instead of as globs. -
item_type(Optional[Union[str, list[str]]], default:None) –Restrict to item types:
'table','model','artifact','array','json','timestamp', an exact item type such as'referenced_table', or a list of these. Ignored withmetadata=True. -
metadata(bool, default:False) –Search folio-level metadata keys instead of item names.
-
folios(Optional[Union[str, Path, list[Union[str, Path]]]], default:None) –Alias(es) or path(s) to search instead of the whole registry.
-
aliases_only(bool, default:False) –Search only aliased folios, not recents.
-
local_only(bool, default:False) –Skip cloud folios.
-
include_archived(bool, default:False) –Include archived items.
-
case_sensitive(bool, default:False) –Match case-sensitively (default: case-insensitive).
-
descriptions(bool, default:False) –Also match item descriptions. An item matches if its name matches the pattern or its description contains it (
'synapse'finds "the synapse table I liked"). Globs match anywhere in the description; regexes already do. -
max_workers(int, default:DEFAULT_WORKERS) –Number of folios read in parallel.
Returns:
-
Any–pandas DataFrame, one row per match. Item searches have columns
-
Any–alias,folio_path,name,item_type,description -
Any–and
created_at; metadata searches havealias,folio_path, -
Any–keyandvalue. Empty (with those columns) when nothing matches.
Raises:
-
ValueError–For an unknown
item_typeor an invalid regex.
Examples:
>>> datafolio.find('cells*')
>>> datafolio.find('synapse|soma', regex=True, item_type='table')
>>> datafolio.find('*', folios='demo', item_type='model')
>>> datafolio.find('dataset=minnie*', metadata=True)
>>> datafolio.find('synapse', descriptions=True)
datafolio.list_folios()
List registered aliases and recently used folios as a DataFrame.
Returns:
-
Any–pandas DataFrame with columns
alias,path,last_accessed -
Any–and
exists(Nonefor cloud paths, which are not checked). -
Any–Aliased folios come first.
Examples:
>>> datafolio.list_folios()
alias path last_accessed exists
0 demo /Users/me/analysis/demo None True
datafolio.set_alias(alias, path, overwrite=False)
Register alias for the folio at path in the user registry.
Parameters:
-
alias(str) –Alias name (letters, digits,
.,_,-). -
path(Union[str, Path]) –Folio path (local or cloud).
-
overwrite(bool, default:False) –Rebind the alias if it already points elsewhere.
Returns:
-
str–The normalized path the alias points to.
Examples:
>>> import datafolio
>>> datafolio.set_alias('demo', '~/analysis/demo')
>>> folio = datafolio.DataFolio(alias='demo')
datafolio.remove_alias(alias)
Remove an alias from the user registry (the folio is untouched).
Parameters:
-
alias(str) –Alias name.
Examples:
>>> datafolio.remove_alias('demo')
Exceptions
datafolio.ConcurrentWriteError
Bases: RuntimeError
Raised when a write would clobber a newer manifest from another writer.
datafolio supports many readers but a single writer per bundle. If a second
writer has advanced the manifest revision since this instance last loaded
it, writing would silently lose their changes; this error surfaces that
instead. Call :meth:DataFolio.refresh and re-apply your change.
This is also raised when the per-folio local write lock cannot be acquired within the bounded timeout (another live writer is holding it).
datafolio.ManifestReadError
Bases: ConcurrentWriteError
Raised when items.json cannot be read and verified before a write.
The manifest is the commit record: if it is missing, unreadable, or
malformed at write time, proceeding could silently clobber another
writer's committed work — so datafolio fails CLOSED. Reopen or
:meth:DataFolio.refresh once the manifest is readable again, then
retry. (Subclasses :class:ConcurrentWriteError so existing
catch-and-retry handling keeps working.)
datafolio.UnsupportedManifestVersionError
Bases: RuntimeError
Raised when opening a folio written by a newer, unknown Datafolio.
items.json records a schema_version. If it is newer than any version
this build knows how to read, the manifest is refused rather than silently
reinterpreted — reading it under the wrong assumptions could corrupt data on
the next write. Upgrade the datafolio package to open the folio.