Skip to content

Package API

datafolio exports one class, three exceptions, and four registry functions. The class has its own page — see DataFolio Methods for every method and property. Everything else in the package (handlers, storage backends, readers) is internal and not part of the public API.

datafolio.find(pattern='*', *, regex=False, item_type=None, metadata=False, folios=None, aliases_only=False, local_only=False, include_archived=False, case_sensitive=False, descriptions=False, max_workers=DEFAULT_WORKERS)

Find items (or metadata keys) across the folios in the user registry.

Searches every aliased and recently used folio (see :func:datafolio.list_folios) unless folios narrows the search. Only each folio's manifest is read, never item data. Folios are read concurrently; one that can't be opened (moved, deleted, no credentials) produces a UserWarning and is skipped.

Parameters:

  • pattern (str, default: '*' ) –

    Glob pattern matched against the whole item name ('*' matches everything). With metadata=True it matches metadata keys, and 'key=value' also matches the value's string form.

  • regex (bool, default: False ) –

    Treat the pattern(s) as regular expressions, matched anywhere in the string (re.search) instead of as globs.

  • item_type (Optional[Union[str, list[str]]], default: None ) –

    Restrict to item types: 'table', 'model', 'artifact', 'array', 'json', 'timestamp', an exact item type such as 'referenced_table', or a list of these. Ignored with metadata=True.

  • metadata (bool, default: False ) –

    Search folio-level metadata keys instead of item names.

  • folios (Optional[Union[str, Path, list[Union[str, Path]]]], default: None ) –

    Alias(es) or path(s) to search instead of the whole registry.

  • aliases_only (bool, default: False ) –

    Search only aliased folios, not recents.

  • local_only (bool, default: False ) –

    Skip cloud folios.

  • include_archived (bool, default: False ) –

    Include archived items.

  • case_sensitive (bool, default: False ) –

    Match case-sensitively (default: case-insensitive).

  • descriptions (bool, default: False ) –

    Also match item descriptions. An item matches if its name matches the pattern or its description contains it ('synapse' finds "the synapse table I liked"). Globs match anywhere in the description; regexes already do.

  • max_workers (int, default: DEFAULT_WORKERS ) –

    Number of folios read in parallel.

Returns:

  • Any –

    pandas DataFrame, one row per match. Item searches have columns

  • Any –

    alias, folio_path, name, item_type, description

  • Any –

    and created_at; metadata searches have alias, folio_path,

  • Any –

    key and value. Empty (with those columns) when nothing matches.

Raises:

  • ValueError –

    For an unknown item_type or an invalid regex.

Examples:

>>> datafolio.find('cells*')
>>> datafolio.find('synapse|soma', regex=True, item_type='table')
>>> datafolio.find('*', folios='demo', item_type='model')
>>> datafolio.find('dataset=minnie*', metadata=True)
>>> datafolio.find('synapse', descriptions=True)

datafolio.list_folios()

List registered aliases and recently used folios as a DataFrame.

Returns:

  • Any –

    pandas DataFrame with columns alias, path, last_accessed

  • Any –

    and exists (None for cloud paths, which are not checked).

  • Any –

    Aliased folios come first.

Examples:

>>> datafolio.list_folios()
     alias                    path  last_accessed  exists
0     demo  /Users/me/analysis/demo           None    True

datafolio.set_alias(alias, path, overwrite=False)

Register alias for the folio at path in the user registry.

Parameters:

  • alias (str) –

    Alias name (letters, digits, ., _, -).

  • path (Union[str, Path]) –

    Folio path (local or cloud).

  • overwrite (bool, default: False ) –

    Rebind the alias if it already points elsewhere.

Returns:

  • str –

    The normalized path the alias points to.

Examples:

>>> import datafolio
>>> datafolio.set_alias('demo', '~/analysis/demo')
>>> folio = datafolio.DataFolio(alias='demo')

datafolio.remove_alias(alias)

Remove an alias from the user registry (the folio is untouched).

Parameters:

  • alias (str) –

    Alias name.

Examples:

>>> datafolio.remove_alias('demo')

Exceptions

datafolio.ConcurrentWriteError

Bases: RuntimeError

Raised when a write would clobber a newer manifest from another writer.

datafolio supports many readers but a single writer per bundle. If a second writer has advanced the manifest revision since this instance last loaded it, writing would silently lose their changes; this error surfaces that instead. Call :meth:DataFolio.refresh and re-apply your change.

This is also raised when the per-folio local write lock cannot be acquired within the bounded timeout (another live writer is holding it).

datafolio.ManifestReadError

Bases: ConcurrentWriteError

Raised when items.json cannot be read and verified before a write.

The manifest is the commit record: if it is missing, unreadable, or malformed at write time, proceeding could silently clobber another writer's committed work — so datafolio fails CLOSED. Reopen or :meth:DataFolio.refresh once the manifest is readable again, then retry. (Subclasses :class:ConcurrentWriteError so existing catch-and-retry handling keeps working.)

datafolio.UnsupportedManifestVersionError

Bases: RuntimeError

Raised when opening a folio written by a newer, unknown Datafolio.

items.json records a schema_version. If it is newer than any version this build knows how to read, the manifest is refused rather than silently reinterpreted — reading it under the wrong assumptions could corrupt data on the next write. Upgrade the datafolio package to open the folio.