Skip to content

Extensions

The remote_store.ext package provides higher-level operations built on top of the core Store API.

Available Extensions

Module Extra Description Guide Example
ext.batch (none) Bulk delete, copy, and exists operations Guide Example
ext.cache (none) Store-level caching with TTL and auto-invalidation Guide Example
ext.glob (none) Portable glob pattern matching for file listing Guide Example
ext.integrity (none) Checksum computation and verification helpers — —
ext.observe (none) Callback-based observability hooks for Store operations Guide Example
ext.partition (none) Hive-style partition path helpers — —
ext.streams (none) Composable BinaryIO wrappers for progress and checksums — —
ext.transfer (none) Upload, download, and cross-store transfer Guide Example
ext.write (none) Write helpers with guaranteed client-side content hashing Guide —
aio.ext.write (none) Async write helpers with guaranteed client-side content hashing Guide —
ext.arrow arrow PyArrow FileSystem adapter Guide Example
ext.parquet arrow Managed Parquet datasets with manifests and completion markers Guide Example
ext.otel otel OpenTelemetry tracing and metrics bridge Guide Example
ext.pydantic pydantic Pydantic BaseModel/BaseSettings adapter — Example
ext.yaml yaml YAML config file loader — Example
ext.dagster dagster Dagster IO Manager adapter Guide Example

Using Extensions

Always-available extensions (pure Python)

Extensions with no extra dependencies (marked *(none)* in the table above) are re-exported from the top-level package for convenience:

from remote_store import batch_delete, glob_files, observe, upload, download
from remote_store import cache                   # ext.cache
from remote_store import checksum, verify       # ext.integrity
from remote_store import partition_path, parse_partition  # ext.partition
from remote_store import ProgressReader, ChecksumReader   # ext.streams
from remote_store import write_with_hash, open_atomic_with_hash, HashingAtomicWriter  # ext.write

Or import from the extension module directly:

from remote_store.ext.batch import batch_delete
from remote_store.ext.cache import cache
from remote_store.ext.glob import glob_files
from remote_store.ext.integrity import checksum, verify
from remote_store.ext.observe import observe
from remote_store.ext.partition import partition_path, parse_partition
from remote_store.ext.streams import ProgressReader, ChecksumReader
from remote_store.ext.transfer import upload
from remote_store.ext.write import write_with_hash, open_atomic_with_hash, HashingAtomicWriter

aio.ext.write has no extra dependency but is not re-exported from remote_store — import it from its module directly:

from remote_store.aio.ext.write import write_with_hash  # async counterpart to ext.write

Seekable reads are built into the core API via Store.read_seekable() — no extension import needed.

Optional-dependency extensions

ext.arrow and ext.parquet require PyArrow, ext.otel requires the OpenTelemetry API, ext.pydantic requires Pydantic v2, and ext.dagster requires Dagster. Install the relevant extra first:

pip install "remote-store[arrow]"     # PyArrow filesystem adapter + parquet datasets
pip install "remote-store[otel]"      # OpenTelemetry tracing and metrics
pip install "remote-store[pydantic]"  # Pydantic BaseSettings adapter
pip install "remote-store[yaml]"      # YAML config file loader
pip install "remote-store[dagster]"   # Dagster IO Manager adapter

Then import from the extension module directly:

from remote_store.ext.arrow import pyarrow_fs
from remote_store.ext.parquet import ParquetDatasetStore
from remote_store.ext.otel import otel_hooks
from remote_store.ext.pydantic import from_pydantic
from remote_store.ext.yaml import from_yaml
from remote_store.ext.dagster import dagster_io_manager

If the required dependency is not installed, importing the extension module raises a ModuleNotFoundError with installation instructions.

Extension Guarantees

All extensions follow the same contract — see the extension architecture and optional-extension export rules ADRs for the rationale:

  • Public API only — extensions use only the public Store / Backend API. They never access private internals.
  • No lifecycle ownership — extensions never close the Store. The caller owns the Store's lifecycle.
  • CapabilityNotSupported propagates — if a backend lacks a required capability, the error reaches the caller immediately.

Writing Your Own Extension

See the "Adding an Extension" checklist in CONTRIBUTING.md.

See also