# remote-store

> Write file storage code once. Run it against local files, S3, SFTP, Azure, or OneDrive.

Unified file-storage API for Python. Write storage code once and switch
between Local, S3 (optionally via PyArrow), Azure (Blob & ADLS Gen2), SFTP,
SQL databases, or Microsoft Graph (OneDrive/SharePoint) via configuration.

A `Store` scopes every operation to a root path, so paths are relative and
`store.child(prefix)` narrows the root. Backends implement storage-specific
behavior; extensions (`remote_store.ext`) add optional capabilities; a native
async API (`remote_store.aio`) mirrors the sync interface. Backends declare
capabilities, so code checks support explicitly with `store.supports(...)`.
The pages below carry the full detail and examples.

## Source

- [GitHub repository](https://github.com/haalfi/remote-store): source tree, issues, and discussions.
- [README](https://raw.githubusercontent.com/haalfi/remote-store/master/README.md): project overview and quick start (raw Markdown).
- [FEATURES](https://raw.githubusercontent.com/haalfi/remote-store/master/FEATURES.md): authoritative backend, extension, capability, and install-extras list (raw Markdown).
- [PyPI](https://pypi.org/project/remote-store/): releases and `pip install remote-store[...]` extras.

## API context (for coding agents)

- [llms-api.txt](https://docs.remotestore.dev/stable/llms-api.txt): the
  full public API surface as a code-shaped skeleton — every class and
  function (backends, async, and extensions included) with its exact
  signature, type annotations, and complete docstring, bodies elided.
  The whole public surface in a single file for a coding agent's context.

# Tutorial

# Tutorial

The hands-on intro to `remote-store`. By the end you'll have a working `Store` reading and writing files, and the rest of the docs will make sense.

# Getting Started

## Installation

Install from [PyPI](https://pypi.org/project/remote-store/):

```
pip install remote-store
```

Backends that need extra dependencies use extras:

```
pip install "remote-store[s3]"           # Amazon S3 / MinIO
pip install "remote-store[s3-pyarrow]"   # S3 via PyArrow (analytical workloads)
pip install "remote-store[sftp]"         # SFTP / SSH
pip install "remote-store[azure]"        # Azure Blob / ADLS Gen2
pip install "remote-store[graph]"        # Microsoft Graph (OneDrive / SharePoint / Teams), async-only
pip install "remote-store[sql]"          # SQL Blob (SQLite, PostgreSQL, ...)
pip install "remote-store[sql-query]"    # SQL Query (read-only, SQLAlchemy + PyArrow)
```

Optional extras for integrations:

```
pip install "remote-store[requests]"       # HTTP backend with requests (connection pooling)
pip install "remote-store[httpx]"          # HTTP backend with httpx (HTTP/2)
pip install "remote-store[arrow]"          # PyArrow filesystem adapter
pip install "remote-store[otel]"           # OpenTelemetry instrumentation
pip install "remote-store[yaml]"           # YAML config support
pip install "remote-store[pydantic]"       # Pydantic BaseSettings config
pip install "remote-store[toml]"           # TOML config on Python < 3.11
```

## Quick Start

The simplest way to use `remote-store` ([`examples/getting_started/quickstart.py`](https://github.com/haalfi/remote-store/blob/master/examples/getting_started/quickstart.py)):

```
from remote_store import Store
from remote_store.backends import LocalBackend

store = Store(LocalBackend(root="/tmp/data"))
store.write_text("hello.txt", "Hello, world!")
print(store.read_text("hello.txt"))  # 'Hello, world!'
```

For applications that manage multiple backends or switch between environments, use a Registry with declarative config:

```
from remote_store import Registry, RegistryConfig

config = RegistryConfig.from_dict({
    "backends": {"main": {"type": "local", "options": {"root": "/tmp/data"}}},
    "stores": {"data": {"backend": "main", "root_path": ""}},
})

with Registry(config) as registry:
    store = registry.get_store("data")
    store.write_text("hello.txt", "Hello, world!")
    print(store.read_text("hello.txt"))  # 'Hello, world!'
```

### Same code, different environment

Switch from local to S3 by changing the config file. The application code stays the same:

**Dev (local filesystem):**

```
[backends.main]
type = "local"
options = { root = "/tmp/data" }

[stores.reports]
backend = "main"
root_path = "reports"
```

**Production (S3):**

```
[backends.main]
type = "s3"
options = { bucket = "analytics-data" }

[stores.reports]
backend = "main"
root_path = "reports"
```

```
# Identical in both environments:
config = RegistryConfig.from_toml("remote-store.toml")
with Registry(config) as registry:
    store = registry.get_store("reports")
    store.write_text("monthly/2026-03.csv", report_csv)
```

Configuration supports TOML, YAML, Pydantic BaseSettings, and plain dicts. Credentials are automatically masked in `repr()`/`str()` to prevent leakage in logs.

## Who this is for

- **Platform and internal tooling teams:** provide one stable storage interface across environments
- **Data engineering teams:** pipelines that run against local storage, S3, or SFTP depending on the environment
- **Teams that include citizen developers:** analysts and domain experts who write Python shouldn't need to learn cloud SDKs just to read and write files
- **Anyone tired of writing storage wrappers in every project**

## What you get

- **One interface, many backends:** local filesystem, S3, SFTP, Azure, OneDrive / SharePoint (Microsoft Graph, async), in-memory, and more
- **Folder-scoped stores:** each Store is rooted at a folder; compose layouts with multiple stores or narrow scope with `child()`
- **Swap backends via config:** move between environments without changing code
- **Streaming by default:** large files just work without blowing up memory
- **Atomic writes where supported:** safer updates for file-producing workflows
- **Async support:** `remote_store.aio` provides `AsyncStore` with coroutine methods; wrap any sync backend with `SyncBackendAdapter`
- **Established libraries underneath:** `s3fs`, `paramiko`, etc. do the real work

Zero runtime dependencies, strict mypy, spec-driven test suite. Optional integrations for PyArrow, OpenTelemetry, and more. See [features](https://github.com/haalfi/remote-store/blob/master/FEATURES.md) for the full list.

## What it is not

- Not a query engine (no SQL, no predicate pushdown)
- Not a table format (no Delta Lake log, no Iceberg manifests)
- Not a filesystem reimplementation (delegates to `s3fs`, `paramiko`, `pyarrow`, etc., the libraries you'd pick anyway)
- Not a file-transfer server (no SFTP/FTP/WebDAV service such as [SFTPGo](https://github.com/drakkan/sftpgo))

## Supported Backends

| Backend                                               | Extra                      | Library                                                                                               | Atomic write | Native glob | `move()` atomic       |
| ----------------------------------------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------- | ------------ | ----------- | --------------------- |
| Local filesystem                                      | *(built-in)*               | stdlib                                                                                                | Yes          | Yes         | Yes\*                 |
| Memory (in-process)                                   | *(built-in)*               | —                                                                                                     | Yes          | —           | Yes                   |
| HTTP/HTTPS (read-only)                                | *(built-in)*               | stdlib                                                                                                | —            | —           | —                     |
| Amazon S3 / MinIO                                     | `remote-store[s3]`         | [`s3fs`](https://pypi.org/project/s3fs/)                                                              | Yes          | Yes         | — (copy+delete)       |
| S3 (PyArrow)                                          | `remote-store[s3-pyarrow]` | [`pyarrow`](https://pypi.org/project/pyarrow/) + [`s3fs`](https://pypi.org/project/s3fs/)             | Yes          | Yes         | — (copy+delete)       |
| SFTP / SSH                                            | `remote-store[sftp]`       | [`paramiko`](https://pypi.org/project/paramiko/)                                                      | Yes          | —           | —\*\*                 |
| Azure Blob / ADLS                                     | `remote-store[azure]`      | [`azure-storage-file-datalake`](https://pypi.org/project/azure-storage-file-datalake/)                | Yes          | Yes         | HNS: Yes / non-HNS: — |
| Microsoft Graph (OneDrive / SharePoint / Teams)\*\*\* | `remote-store[graph]`      | [`httpx`](https://pypi.org/project/httpx/) + [`msal`](https://pypi.org/project/msal/)                 | Yes          | —           | —\*\*\*\*             |
| SQL Blob (SQLite, PostgreSQL, ...)                    | `remote-store[sql]`        | [`sqlalchemy`](https://pypi.org/project/SQLAlchemy/)                                                  | Yes          | Yes         | Yes                   |
| SQL Query (read-only)                                 | `remote-store[sql-query]`  | [`sqlalchemy`](https://pypi.org/project/SQLAlchemy/) + [`pyarrow`](https://pypi.org/project/pyarrow/) | —            | —           | —                     |

\* Same-filesystem only; cross-filesystem falls back to copy+delete. * *Attempts `posix_rename` (atomic on POSIX-compliant servers) but falls back to copy+delete; atomicity cannot be guaranteed, so `ATOMIC_MOVE` is not declared. \** *Async-only: construct via `AsyncStore(backend=GraphBackend(...))`; there is no sync `Store` wrapper or config `type=` string. \**\*\* Native server-side move (`PATCH driveItem`, identity-preserving); may complete asynchronously, so `ATOMIC_MOVE` is not declared.

All backends except HTTP and SQL Query support read, write, delete, list, copy, move, and metadata. HTTP is read-only. SQL Query is read-only: it materializes SQL queries to Parquet/CSV/Arrow IPC on read. Glob is natively supported by most backends; for those that lack it, the portable fallback `ext.glob.glob_files()` works with any `LIST`-capable backend. Seekable reads are available via `Store.read_seekable()` on backends that declare `SEEKABLE_READ`. See [features](https://github.com/haalfi/remote-store/blob/master/FEATURES.md), the [capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/), and the [concurrency guide](https://docs.remotestore.dev/stable/explanation/concurrency/) for full details.

## Store API

The Store provides methods across read/write, browsing, management, and utility. Key highlights:

```
store.read_text("path/to/file.txt")             # → str
store.write_text("path/to/file.txt", content)   # write string
store.read_bytes("path/to/file.csv")            # → bytes
store.write("path/to/data.bin", binary_stream)  # streaming write

store.list_files("reports/", pattern="*.csv")   # iterate FileInfo
store.glob("**/*.parquet")                      # native glob (capability-gated)
store.exists("path/to/file.txt")                # → bool
store.head("path/to/file.txt")                  # WriteResult snapshot (size, etag, …)

store.move("old.txt", "new.txt")                # move / rename
store.copy("src.txt", "dst.txt")                # copy
store.delete("path/to/file.txt")                # delete

store.child("subfolder")                        # scoped child store
store.supports(Capability.ATOMIC_WRITE)         # runtime capability check (gates a method)
store.supports(Capability.ATOMIC_MOVE)          # quality flag — move() atomicity guarantee
store.resolve("path/to/file.txt")               # resolution plan (introspection)
store.ping()                                    # health check
```

For the full method list, see the [API reference](https://docs.remotestore.dev/stable/reference/api/store/). All write, move, and copy methods accept `overwrite=True` to replace existing files.

## Performance

Per-operation overhead is small relative to network round-trip time for most workloads. S3 listing is significantly faster via s3fs connection caching. See the [performance guide](https://docs.remotestore.dev/stable/explanation/performance/) for full comparative benchmarks, methodology, and per-operation breakdowns.

## Extensions

The core library handles storage operations. Extensions add optional capabilities on top: PyArrow integration, observability, caching, or bulk operations. All live in `remote_store.ext`; import only what you need.

| Extension            | Extra                   | What it does                                                                                   |
| -------------------- | ----------------------- | ---------------------------------------------------------------------------------------------- |
| PyArrow adapter      | `remote-store[arrow]`   | Use any Store as a `pyarrow.fs.FileSystem`; works with Parquet, Pandas, Polars, DuckDB         |
| Parquet datasets     | `remote-store[arrow]`   | Managed Parquet datasets with manifests, `_SUCCESS` markers, and multi-part layouts            |
| Batch operations     | *(none)*                | Bulk delete, copy, and exists with error aggregation                                           |
| Transfer operations  | *(none)*                | Upload, download, and cross-store transfer with progress                                       |
| Observability hooks  | *(none)*                | Callback-based instrumentation for logging, metrics, and tracing                               |
| OpenTelemetry bridge | `remote-store[otel]`    | Pre-built OTel spans and metrics for Store operations                                          |
| Caching middleware   | *(none)*                | TTL-based read cache with automatic invalidation on mutations                                  |
| Stream wrappers      | *(none)*                | Composable BinaryIO wrappers for progress tracking and checksums                               |
| Integrity helpers    | *(none)*                | Checksum computation and verification over Store's public API                                  |
| Write helpers        | *(none)*                | Client-side content hashing for write operations, compatible with any backend                  |
| Dagster integration  | `remote-store[dagster]` | IOManager adapter, config-driven Store resource, and compute log manager for Dagster pipelines |

Plus glob helpers, partition helpers, YAML and Pydantic config adapters. See the [extensions guide](https://docs.remotestore.dev/stable/guides/extensions/) for details.

## Quality & Testing

Storage behavior must be predictable and correct. We verify this across multiple dimensions:

- **Spec-driven development:** behavior specifications are the source of truth; tests link directly to them. *Prevents feature drift.*
- **Design by Contract:** pre/post conditions and invariants catch incorrect usage early. *Fails fast on misuse.*
- **Examples and snippets:** runnable code in [`examples/`](https://github.com/haalfi/remote-store/tree/master/examples) and [notebooks](https://github.com/haalfi/remote-store/tree/master/examples/notebooks); docs are tested against actual behavior. *Keeps examples real.*
- **Extensive unit tests:** high coverage across all backends, focused on behavior. *Catches integration issues early.*
- **Dependency drift guard:** scheduled CI re-resolves extras against latest versions to catch silent transitive upgrades. *Surfaces upstream breakage early.*
- **Property-based testing:** randomized input generation via [Hypothesis](https://hypothesis.readthedocs.io/) surfaces edge cases no hand-written test would find. *Finds blind spots.*
- **Formal verification:** critical paths are proven correct in [Dafny](https://dafny.org/) before implementation. *Eliminates logic errors.*
- **Mutation testing:** [gremlins](https://pypi.org/project/pytest-gremlins/) modify the code; if they survive the tests, the tests have gaps. *Exposes weak test coverage.*
- **Benchmarks:** performance tracked per operation and backend. *Provides baseline for optimization.*

## Learn more

To explore `remote-store` beyond the Quick Start:

- **Examples:** self-contained scripts in [`examples/`](https://github.com/haalfi/remote-store/tree/master/examples) covering core operations (file I/O, streaming, atomic writes, error handling, etc.) and backend-specific setups for S3, SFTP, and Azure.
- **Notebooks:** interactive [Jupyter notebooks](https://github.com/haalfi/remote-store/tree/master/examples/notebooks) that walk through common workflows step by step.
- **Guides:** topic-focused walkthroughs in the [documentation](https://docs.remotestore.dev/stable/) covering backends, extensions, configuration, and patterns like data lake layouts or health checks.
- **For AI coding agents:** point your agent at [`llms.txt`](https://docs.remotestore.dev/stable/llms.txt) (index) or [`llms-full.txt`](https://docs.remotestore.dev/stable/llms-full.txt) (the full docs). For the public API surface alone — every signature and docstring, backends included — use [`llms-api.txt`](https://docs.remotestore.dev/stable/llms-api.txt).

## How it compares

There are several excellent Python libraries for file I/O across backends. Here is where `remote-store` sits:

|               | [fsspec](https://pypi.org/project/fsspec/) | [smart_open](https://pypi.org/project/smart-open/) | [cloudpathlib](https://pypi.org/project/cloudpathlib/) | [obstore](https://pypi.org/project/obstore/) | **remote-store**            |
| ------------- | ------------------------------------------ | -------------------------------------------------- | ------------------------------------------------------ | -------------------------------------------- | --------------------------- |
| API surface   | many methods                               | `open()` only                                      | pathlib-style                                          | ~10 methods                                  | full Store API              |
| Backends      | many filesystems                           | S3, GCS, Az, SFTP                                  | S3, GCS, Azure                                         | S3, GCS, Azure                               | Local, S3, SFTP, Az, Memory |
| SFTP          | via sshfs                                  | Yes                                                | —                                                      | —                                            | Built-in                    |
| Streaming I/O | Yes                                        | Yes                                                | — (downloads)                                          | Bytes-oriented                               | Yes (BinaryIO)              |
| Atomic writes | —                                          | —                                                  | —                                                      | —                                            | Yes (capability-gated)      |
| Async         | Yes                                        | —                                                  | —                                                      | Yes (first-class)                            | Yes (`remote_store.aio`)    |
| Observability | —                                          | —                                                  | —                                                      | —                                            | `ext.observe` + OTel        |
| Config model  | Per-filesystem                             | URI-based                                          | Per-client                                             | Per-store kwargs                             | Immutable Registry          |
| Runtime deps  | Yes                                        | Minimal                                            | SDK-based                                              | Rust binary                                  | Zero (core)                 |

*Feature sets may change as these libraries evolve. Check each project's documentation for the current state.*

**In short:** `remote-store` is for teams that need more than `open()` (smart_open) but less than a full filesystem abstraction (fsspec), with streaming, SFTP, atomic writes, observability, and immutable config. Under the hood, it delegates to the same libraries you'd pick anyway (`s3fs`/`boto3`, `paramiko`, Azure SDK, PyArrow).
# Guides

# Guides

Recipes for the things you'll actually do: pick a backend, add caching, stream big files, plug into PyArrow or Dagster. One page, one task.

# Async/Sync Bridge Adapters

`remote-store` ships two adapter classes that bridge the gap between synchronous and asynchronous code. Both are in the `remote_store` package; choose the one that matches the direction you need.

## Decision table

| Question                      | `SyncBackendAdapter`                          | `AsyncBackendSyncAdapter`                            |
| ----------------------------- | --------------------------------------------- | ---------------------------------------------------- |
| **Direction**                 | Sync backend → usable from async code         | Async backend → usable from sync code                |
| **You have…**                 | A `Backend` (sync)                            | An `AsyncBackend` (async-native)                     |
| **You need to call it from…** | `async` functions / `AsyncStore`              | Ordinary sync functions / `Store`                    |
| **Typical consumer**          | `AsyncStore` wrapping a local or SFTP backend | Sync code (or a `Store`) driving `AsyncAzureBackend` |

## `SyncBackendAdapter` — sync → async

```
from remote_store.aio import AsyncStore, SyncBackendAdapter  # noqa: F811
from remote_store.backends import MemoryBackend  # noqa: F811

backend = MemoryBackend()
async_backend = SyncBackendAdapter(backend)

async with AsyncStore(async_backend) as store:
    await store.write("report.csv", b"col,val\n1,2")
    content = await store.read_bytes("report.csv")
```

Use this when you have an existing sync backend and want to drive it from async code without rewriting it.

## `AsyncBackendSyncAdapter` — async → sync

```
from remote_store import AsyncBackendSyncAdapter, Store  # noqa: F811
from remote_store.aio import AsyncMemoryBackend  # noqa: F811

async_backend = AsyncMemoryBackend()

with AsyncBackendSyncAdapter(async_backend) as adapter:
    store = Store(adapter)
    store.write("report.csv", b"col,val\n1,2")
    content = store.read_bytes("report.csv")
```

Use this when you have an async-native backend (e.g. `AsyncAzureBackend`) but your calling code is synchronous.

### Constraints to keep in mind

- Cannot be called from a running event loop — use `AsyncStore` instead.
- `read()` returns a forward-only stream (`seekable()` is `False`).
- `close(timeout=…)` drains in-flight work before stopping the private loop; always call it (or use the context manager) to avoid daemon-thread leaks.

## See also

- [API reference](https://docs.remotestore.dev/stable/reference/api/aio/adapters/index.md) — `AsyncBackendSyncAdapter`, `SyncBackendAdapter`, `AsyncStore`
- [Async guide](https://docs.remotestore.dev/stable/guides/async/index.md) — `AsyncStore`, native async backends, and the `SyncBackendAdapter` direction
- [Async-to-sync adapter decision record](https://docs.remotestore.dev/stable/explanation/design/adrs/0025-async-to-sync-backend-adapter/index.md)
- [Async backend API spec](https://docs.remotestore.dev/stable/explanation/design/specs/029-async-store-backend-api/index.md) — `AsyncBackendSyncAdapter` invariants

# Async Store

Use `remote_store.aio` to access any backend with `async`/`await`. The async API mirrors the synchronous `Store` — same methods, same errors, same capability model — so existing knowledge transfers directly.

## Quick start

```
import asyncio

from remote_store.aio import AsyncStore
from remote_store.backends import MemoryBackend

async def main() -> None:
    async with AsyncStore(MemoryBackend(), root_path="reports") as store:
        result = await store.write("summary.txt", b"Q1 results", overwrite=True)
        print(f"wrote {result.size} bytes")
        data = await store.read_bytes("summary.txt")
        print(data.decode())

asyncio.run(main())
```

Any sync `Backend` (Local, S3, SFTP, Azure, Memory) is auto-wrapped via `SyncBackendAdapter`, which delegates each call to the default executor through `asyncio.to_thread()`. No code changes needed on the backend side.

For backends with native async SDK support, use the dedicated async backend class for true non-blocking I/O — see [Native async backends](#native-async-backends).

## Streaming reads

`AsyncStore.read()` returns an `AsyncIterator[bytes]` (not `BinaryIO`), because Python has no standard async file-like protocol. Consume it with `async for`:

```
async for chunk in store.read("large-file.bin"):
    process(chunk)  # 64 KB chunks by default
```

For small files, `read_bytes()` and `read_text()` load the full content into memory in a single call:

```
text = await store.read_text("config.yaml")
```

## Write results and metadata

`write()` and `write_atomic()` return a [`WriteResult`](https://docs.remotestore.dev/stable/reference/api/models/index.md) carrying at minimum the written path and size. Backends that declare `WRITE_RESULT_NATIVE` also populate `digest`, `etag`, and `last_modified` from the upload response. Both methods accept an optional `metadata=` keyword argument (a `Mapping[str, str]`) for backends that declare `USER_METADATA`; others raise `CapabilityNotSupported` if non-empty metadata is passed.

## Writing with async iterators

`write()` and `write_atomic()` accept `bytes` or `AsyncIterator[bytes]`:

```
store = AsyncStore(AsyncMemoryBackend())

async def generate_report() -> AsyncIterator[bytes]:
    yield b"header\n"
    yield b"row1\n"
    yield b"row2\n"

result = await store.write("report.csv", generate_report())
print(f"wrote {result.size} bytes to {result.path}")
```

## Child stores

`child()` is synchronous (no I/O) and returns a new `AsyncStore` scoped to a subfolder. The child shares the parent's backend:

```
reports = store.child("2024/q1")
await reports.write("summary.txt", b"data")
# Visible at <root>/2024/q1/summary.txt
```

## Use with FastAPI

```
from fastapi import FastAPI, UploadFile
from remote_store.aio import AsyncStore
from remote_store.backends import S3Backend

app = FastAPI()
store = AsyncStore(S3Backend(bucket="uploads", anon=False))

@app.post("/upload/{filename}")
async def upload(filename: str, file: UploadFile):
    data = await file.read()
    result = await store.write(filename, data, overwrite=True)
    return {"stored": filename, "size": result.size}

@app.get("/download/{filename}")
async def download(filename: str):
    from starlette.responses import StreamingResponse
    return StreamingResponse(store.read(filename))
```

The module-level `store` is shared across every request handler — safe here because the S3 backend is thread-safe, so the `AsyncStore` thread-pool bridge can drive it concurrently. With a single-connection backend (e.g. SFTP) this sharing would **not** be safe; see [Concurrency](https://docs.remotestore.dev/stable/explanation/concurrency/#concurrent-use-posture).

## Native async backends

`SyncBackendAdapter` runs sync backends in a thread pool — good enough for many workloads, but each call still blocks a thread. Native async backends use the cloud SDK's async clients directly, avoiding thread-pool overhead.

### AsyncAzureBackend

```
from remote_store.aio import AsyncStore, AsyncAzureBackend

backend = AsyncAzureBackend(
    container="my-container",
    hns=True,
    account_name="myaccount",
    account_key="...",
)
async with AsyncStore(backend, root_path="data") as store:
    await store.write("report.csv", b"col1,col2\n1,2", overwrite=True)
```

`AsyncAzureBackend` supports both plain Blob Storage and ADLS Gen2 (HNS-enabled) accounts. Declare which via the required `hns` argument (`True` for ADLS Gen2, `False` for plain Blob Storage); there is no auto-detection. Use `await AzureUtils.adetect_hns(...)` to discover it once if unknown. The constructor otherwise accepts the same credential parameters as the sync `AzureBackend`.

Install the async Azure extras:

```
pip install "remote-store[azure]"
```

No additional dependencies are needed — the Azure SDK's async clients (`azure.storage.blob.aio`, `azure.storage.filedatalake.aio`) are included in the same packages as the sync clients.

### GraphBackend

`GraphBackend` targets OneDrive / SharePoint / Teams files through the Microsoft Graph REST API. It is **async-only** — there is no sync `Store` wrapper or config `type=` string, so it is constructed directly:

```
from remote_store.aio import AsyncStore, GraphAuth, GraphBackend, GraphUtils

# Device-code auth against a personal Microsoft account (consumer OneDrive).
auth = GraphAuth(tenant_id="consumers", client_id="<entra-app-id>")
# Inside async code, use the async resolver — the sync GraphUtils.resolve_drive_id
# runs its own event loop internally and raises RuntimeError from a running one.
drive_id = await GraphUtils.aresolve_drive_id("me", token_provider=auth)

backend = GraphBackend(drive_id, token_provider=auth)
async with AsyncStore(backend, root_path="Documents") as store:
    await store.write("report.csv", b"col1,col2\n1,2", overwrite=True)
```

`GraphAuth` selects device-code (interactive) or client-credentials (app-only) auth depending on whether a `client_secret` / `client_certificate` is supplied. `GraphUtils.resolve_drive_id` (and its async twin `aresolve_drive_id`, used above — call the async form inside a running loop) turns `"me"`, a SharePoint site URL, or a Teams `{"team_id", "channel_id"}` mapping into a `drive_id`.

Install the extra:

```
pip install "remote-store[graph]"
```

See the [Graph backend guide](https://docs.remotestore.dev/stable/guides/backends/graph/index.md) for configuration, capability notes, and the `TMPDIR` / `copy_timeout` operational caveats, and the [Graph setup guide](https://docs.remotestore.dev/stable/guides/backends/graph-setup/index.md) for Entra app registration.

## Health check

`ping()` verifies that the backend is reachable and credentials are valid:

```
await store.ping()  # raises BackendUnavailable on failure
```

Native async backends (like `AsyncAzureBackend`) perform a lightweight async probe. Wrapped sync backends delegate through `asyncio.to_thread()`.

## Context manager

Use `async with` for automatic cleanup:

```
async with AsyncStore(backend) as store:
    await store.write("file.txt", b"data")
# backend resources released here
```

Child stores do not close the parent's backend — only the owning store calls `aclose()` on exit.

## Limitations

- **`read_seekable()` and `open_atomic()` are not available** in the async API. Use `read_bytes()` + `io.BytesIO()` if you need a seekable stream, or `write_atomic()` for single-shot atomic writes.
- **`SyncBackendAdapter` materializes listing iterators** in memory (`list_files`, `list_folders`, `glob`). For very large directories this may use more memory than the sync API. Native async backends stream without materialisation.
- **`SyncBackendAdapter` + `SFTPBackend` is not safe for concurrent use.** Paramiko's `SFTPClient` is not thread-safe; concurrent `asyncio.gather` calls against a single `SFTPBackend` instance race on the shared socket and may hang. Create one `SFTPBackend` per thread, or use a native async SFTP library. See [SFTP backend guide](https://docs.remotestore.dev/stable/guides/backends/sftp/#connection-behaviour).
- **Concurrent use depends on the backend's posture.** Sharing one store across concurrent tasks is safe for thread-safe backends (Local, Memory, S3, Azure) and for async-native backends on a single loop (Azure, Graph); it is unsafe for single-connection backends (SFTP, HTTP on `urllib`). See [Concurrency](https://docs.remotestore.dev/stable/explanation/concurrency/#concurrent-use-posture).
- **`asyncio` only** — trio and anyio are not supported.

## Async write helpers

`remote_store.aio.ext.write` provides `write_with_hash` for async stores: it streams the content through a client-side hash, writes it, and returns a `WriteResult` with `digest` populated regardless of whether the backend declares `WRITE_RESULT_NATIVE`. This is the async counterpart of `remote_store.ext.write.write_with_hash`. See the [Write Integrity](https://docs.remotestore.dev/stable/guides/write-integrity/index.md) guide for usage examples and the full async API.

## See also

- [API reference](https://docs.remotestore.dev/stable/reference/api/aio/index.md) — `AsyncStore`, `AsyncBackend`, `AsyncAzureBackend`, `GraphBackend`, `SyncBackendAdapter`
- [Async-Sync Bridges](https://docs.remotestore.dev/stable/guides/async-sync-bridges/index.md) — `AsyncBackendSyncAdapter` for calling an async backend from sync code
- [Azure Backend](https://docs.remotestore.dev/stable/guides/backends/azure/index.md) — sync Azure backend configuration and usage
- [Graph Backend](https://docs.remotestore.dev/stable/guides/backends/graph/index.md) — async-only OneDrive / SharePoint / Teams backend
- [Health Check](https://docs.remotestore.dev/stable/guides/health-check/index.md) — `ping()` and `check_health()` details
- [Concurrency](https://docs.remotestore.dev/stable/explanation/concurrency/index.md) — thread safety, atomicity, and `overwrite=False` semantics
- [Example: Async Store](https://docs.remotestore.dev/stable/tutorial/examples/async-store/index.md) — runnable async demo

# Batch Operations

The `ext.batch` module provides convenience functions for operating on collections of paths: batch delete, batch copy, and batch existence checks.

By default, all functions call Store methods sequentially and collect errors into a `BatchResult` instead of failing on the first error. Pass `concurrent=True` for parallel execution via `ThreadPoolExecutor` — cloud backends benefit significantly from concurrent I/O. No extra dependencies are required — the module is pure Python (stdlib only) and always available.

## Quick Start

```
from remote_store import Store, batch_delete, batch_copy, batch_exists
from remote_store.backends import MemoryBackend

store = Store(backend=MemoryBackend())
store.write("a.txt", b"hello")
store.write("b.txt", b"world")

# Check which files exist
exists_map = batch_exists(store, ["a.txt", "b.txt", "c.txt"])
# {"a.txt": True, "b.txt": True, "c.txt": False}

# Copy multiple files
result = batch_copy(store, [("a.txt", "a_copy.txt"), ("b.txt", "b_copy.txt")])
assert result.all_succeeded

# Delete multiple files
result = batch_delete(store, ["a.txt", "b.txt"], missing_ok=True)
assert result.all_succeeded
```

## BatchResult

`batch_delete` and `batch_copy` return a `BatchResult` — a frozen dataclass that separates successes from failures:

```
result = batch_delete(store, ["exists.txt", "missing.txt"])

result.succeeded   # ("exists.txt",)
result.failed      # {"missing.txt": NotFound(...)}
result.all_succeeded  # False
result.total          # 2
```

## Error Handling

By default, batch functions continue on error and collect failures:

```
result = batch_delete(store, ["a.txt", "bad.txt", "c.txt"])
# a.txt deleted, bad.txt fails, c.txt still deleted
```

Use `stop_on_error=True` to halt on the first failure:

```
result = batch_delete(store, ["a.txt", "bad.txt", "c.txt"], stop_on_error=True)
# a.txt deleted, bad.txt fails, c.txt never attempted
```

### Capability Errors

`CapabilityNotSupported` errors always propagate immediately, regardless of `stop_on_error`. These indicate a configuration problem (wrong backend for the operation), not a per-path issue.

## batch_delete

```
batch_delete(store, paths, *, missing_ok=False, stop_on_error=False,
             concurrent=False, max_workers=None) -> BatchResult
```

Deletes each path via `store.delete(path, missing_ok=missing_ok)`.

- `missing_ok=True`: silently skip files that don't exist.
- `stop_on_error=True`: stop on first failure (sequential only).
- `concurrent=True`: execute deletes in parallel via `ThreadPoolExecutor`.
- `max_workers=N`: limit thread pool size (default: executor default).

## batch_copy

```
batch_copy(store, pairs, *, overwrite=False, stop_on_error=False,
           concurrent=False, max_workers=None) -> BatchResult
```

Copies each `(src, dst)` pair via `store.copy(src, dst, overwrite=overwrite)`.

- `overwrite=True`: overwrite existing destinations.
- `stop_on_error=True`: stop on first failure (sequential only).
- `concurrent=True`: execute copies in parallel via `ThreadPoolExecutor`.
- `max_workers=N`: limit thread pool size (default: executor default).

The source path is used as the key in both `succeeded` and `failed`.

## batch_exists

```
batch_exists(store, paths, *, concurrent=False, max_workers=None) -> dict[str, bool]
```

Checks each path via `store.exists(path)`. Returns a dict mapping each path to `True` or `False`.

- `concurrent=True`: execute checks in parallel via `ThreadPoolExecutor`.
- `max_workers=N`: limit thread pool size (default: executor default).

Unlike the other batch functions, `batch_exists` does **not** catch errors. If `store.exists()` raises (e.g., due to a backend failure), the exception propagates immediately. This is intentional — `exists()` should never fail under normal conditions.

## Parallel Execution

Cloud backends benefit significantly from concurrent I/O — sequential execution over hundreds of partition files is a bottleneck. Pass `concurrent=True` to use a thread pool:

```
# Delete 500 partition files in parallel
keys = [f"data/year=2025/month={m:02d}/part.parquet" for m in range(1, 13)]
result = batch_delete(store, keys, concurrent=True, max_workers=16)

# Check existence of many files concurrently
exists_map = batch_exists(store, keys, concurrent=True)
```

**Notes:**

- `stop_on_error=True` is incompatible with `concurrent=True` (raises `ValueError`). Concurrent execution has non-deterministic ordering, so "stop on first error" has no well-defined semantics.
- The order of `succeeded` paths is non-deterministic in concurrent mode.
- Error collection and capability gating work identically in both modes.

## Works with Store.child()

All batch functions operate through the public Store API. They work correctly with `Store.child()`, capability gating, and path rebasing:

```
store = Store(backend=MemoryBackend())
store.write("reports/q1.csv", b"data")
store.write("reports/q2.csv", b"data")

reports = store.child("reports")
result = batch_delete(reports, ["q1.csv", "q2.csv"])
assert result.all_succeeded
```

## See also

- [API reference](https://docs.remotestore.dev/stable/reference/api/extensions/batch/index.md)
- [Example script](https://github.com/haalfi/remote-store/blob/master/examples/extensions/batch_operations.py)

# Caching Guide

`ext.cache` wraps a Store in a caching proxy that reduces backend round-trips for read-heavy and metadata-heavy workloads.

## Quick Start

```
from remote_store import Store, cache
from remote_store.backends import MemoryBackend

store = Store(MemoryBackend())
store.write("config.json", b'{"key": "value"}')

# Wrap with a 5-minute cache
cached = cache(store, ttl=300)

# First call hits the backend
data = cached.read_bytes("config.json")

# Second call returns from cache (no backend I/O)
data = cached.read_bytes("config.json")

# Writes automatically invalidate the cache
cached.write("config.json", b'{"key": "new"}', overwrite=True)

# Next read goes to the backend again
data = cached.read_bytes("config.json")  # b'{"key": "new"}'
```

## What Gets Cached

| Operation           | Cached? | Notes                                                     |
| ------------------- | ------- | --------------------------------------------------------- |
| `exists()`          | Yes     | Including `False` results                                 |
| `is_file()`         | Yes     |                                                           |
| `is_folder()`       | Yes     |                                                           |
| `read_bytes()`      | Yes     | Subject to `max_content_size`                             |
| `get_file_info()`   | Yes     |                                                           |
| `get_folder_info()` | Yes     |                                                           |
| `list_files()`      | Yes     | Materialized on first call; subject to `max_listing_size` |
| `list_folders()`    | Yes     | Materialized on first call; subject to `max_listing_size` |
| `iter_children()`   | Yes     | Materialized on first call; subject to `max_listing_size` |
| `glob()`            | Yes     | Materialized on first call; subject to `max_listing_size` |
| `read()`            | —       | Returns `BinaryIO` stream                                 |

`read()` is deliberately not cached because it returns a `BinaryIO` stream that may be lazily consumed. Use `read_bytes()` when you want cached content reads.

## Automatic Invalidation

Mutating operations automatically invalidate affected cache entries:

- **`write`, `write_atomic`, `open_atomic`** — invalidate the written path and all listing/folder caches.
- **`delete`** — invalidate the deleted path and all listings.
- **`delete_folder`** — clear the entire cache (folder deletion can affect any cached path).
- **`move`** — invalidate both source and destination paths plus listings.
- **`copy`** — invalidate destination path plus listings.

## Limiting Memory Usage

By default, `read_bytes()` caches files of any size. For workloads with large files, set `max_content_size` to prevent memory pressure:

```
# Only cache files up to 1 MB
cached = cache(store, ttl=300, max_content_size=1_048_576)
```

Files larger than the limit are still returned correctly — they just bypass the cache.

Similarly, listing operations (`list_files`, `list_folders`, `iter_children`, `glob`) cache their full result set. For stores with very large directories, set `max_listing_size` to skip caching listings that exceed a given item count:

```
# Cache listings up to 500 items; larger ones bypass the cache
cached = cache(store, ttl=300, max_listing_size=500)
```

Both limits can be combined:

```
cached = cache(store, ttl=300, max_content_size=1_048_576, max_listing_size=500)
```

To limit the total number of cache entries regardless of type, use `max_entries`. When exceeded, the least-recently-used entry is evicted:

```
cached = cache(store, ttl=300, max_entries=1000)
```

## Cache Statistics

```
stats = cached.stats
print(f"Hits: {stats.hits}, Misses: {stats.misses}, Size: {stats.size}")
```

## Manual Invalidation

```
# Invalidate a specific path
cached.invalidate("config.json")

# Clear the entire cache
cached.clear_cache()
```

## Stale Data

The cache cannot detect writes made by other processes or other Store instances sharing the same backend. If your workload involves external mutations, either:

1. Set a short TTL (e.g., `ttl=10`).
1. Call `invalidate(path)` or `clear_cache()` when you know external writes occurred.

## Composing with Observability

`CachedStore` composes with `ObservedStore`. The ordering determines what gets observed:

```
from remote_store import observe, cache

# Observe only cache misses (actual backend calls)
cached = cache(observe(store, on_read=my_hook), ttl=300)

# Observe all reads including cache hits
observed = observe(cache(store, ttl=300), on_read=my_hook)
```

## Thread Safety

`CachedStore` and `MemoryCache` are thread-safe. They work correctly with `batch_exists(concurrent=True)` and similar concurrent access patterns.

## See also

- [API reference](https://docs.remotestore.dev/stable/reference/api/extensions/cache/index.md)
- [Example script](https://github.com/haalfi/remote-store/blob/master/examples/extensions/caching.py)

# Choosing a Backend

This guide helps you pick the right `remote-store` backend for your use case.

## Decision tree

1. **Local filesystem?** Use **Local**. Fast, full capabilities, zero config. Best for development and single-machine workflows.

1. **In-process testing or caching?** Use **Memory**. No disk I/O, instant setup/teardown, ideal for unit tests and ephemeral caches. Lacks native glob (use `ext.glob` fallback).

1. **S3-compatible object store (AWS S3, MinIO, Ceph, etc.)?**

   - Need **analytical workloads** (Parquet column pruning, PyArrow datasets)? Use **S3-PyArrow**. Native C++ `FileSystem` for PyArrow, GIL-free reads.
   - Otherwise use **S3** (fsspec-based). Faster for sequential reads/writes, lighter dependency footprint, same API surface.

1. **Azure Blob Storage or ADLS Gen2?** Use **Azure**. Supports both flat and HNS (hierarchical namespace) accounts — declare which via the required `hns` option. Connection string, SAS token, or DefaultAzureCredential auth. Analytical random access (Parquet footer reads, PyArrow column pruning) is efficient despite Azure not declaring `SEEKABLE_READ`: `read_seekable()` uses a native HTTP-Range reader on the sync backend — see the [Azure guide](https://docs.remotestore.dev/stable/guides/backends/azure/#streaming-and-seekable-reads).

1. **OneDrive, SharePoint, or Microsoft Teams files (Microsoft Graph)?** Use **Graph**. **Async-only** — construct via `AsyncStore(backend=GraphBackend(...))`; there is no sync wrapper or config `type=` string. Device-code or client-credential auth via MSAL; onboarding is the main hurdle, so start from the [setup guide](https://docs.remotestore.dev/stable/guides/backends/graph-setup/index.md). Lacks native glob and seekable reads.

1. **SSH/SFTP server?** Use **SFTP**. Legacy systems, on-prem file servers. Supports password and key-based auth. Lacks native glob (use `ext.glob` fallback).

1. **Store blobs in a relational database (SQLite, PostgreSQL, etc.)?** Use **SQLBlob**. Broad capability set — read, write, list, move, copy, glob, and atomic writes. Useful for embedded storage, metadata-heavy workloads, or environments where a database is already available.

1. **Materialize SQL queries as files (read-only)?** Use **SQLQuery**. Executes a SQL query and exposes the result as Parquet, CSV, or Arrow IPC. Read and metadata only. Useful for ETL pipelines and data exports.

1. **Read-only HTTP/HTTPS endpoint?** Use **HTTP**. Public data, static file servers, REST APIs. Read and metadata only — no write, list, or delete. Zero required dependencies (stdlib `urllib`); optional `requests` or `httpx` transports for connection pooling.

## Trade-offs at a glance

| Backend                                                                               | Dependencies             | Glob     | Throughput | Best for                                   |
| ------------------------------------------------------------------------------------- | ------------------------ | -------- | ---------- | ------------------------------------------ |
| [Local](https://docs.remotestore.dev/stable/guides/backends/local/index.md)           | None                     | Native   | Disk-bound | Dev, single machine                        |
| [Memory](https://docs.remotestore.dev/stable/guides/backends/memory/index.md)         | None                     | Fallback | In-process | Tests, caches                              |
| [S3](https://docs.remotestore.dev/stable/guides/backends/s3/index.md)                 | `s3fs`                   | Native   | Network    | General S3 workloads                       |
| [S3-PyArrow](https://docs.remotestore.dev/stable/guides/backends/s3-pyarrow/index.md) | `pyarrow`                | Native   | Network    | Parquet, PyArrow datasets                  |
| [SFTP](https://docs.remotestore.dev/stable/guides/backends/sftp/index.md)             | `paramiko`               | Fallback | Network    | Legacy, on-prem                            |
| [Azure](https://docs.remotestore.dev/stable/guides/backends/azure/index.md)           | `azure-storage-blob`     | Native   | Network    | Azure workloads                            |
| [Graph](https://docs.remotestore.dev/stable/guides/backends/graph/index.md)           | `httpx` + `msal`         | Fallback | Network    | OneDrive / SharePoint / Teams (async-only) |
| [SQLBlob](https://docs.remotestore.dev/stable/guides/backends/sql-blob/index.md)      | `sqlalchemy`             | Native   | DB-bound   | Embedded, metadata-heavy                   |
| [SQLQuery](https://docs.remotestore.dev/stable/guides/backends/sql-query/index.md)    | `sqlalchemy` + `pyarrow` | Native   | DB-bound   | Read-only ETL exports                      |
| [HTTP](https://docs.remotestore.dev/stable/guides/backends/http/index.md)             | None                     | —        | Network    | Read-only public data                      |

## Switching backends at runtime

The whole point of `remote-store` is that your application code stays the same regardless of backend. Switch via configuration:

```
# dev.toml
[backends.storage]
type = "local"
base_path = "./data"

[stores.default]
backend = "storage"

# prod.toml
[backends.storage]
type = "s3"
bucket = "my-bucket"

[stores.default]
backend = "storage"
```

```
from remote_store import RegistryConfig, Registry

config = RegistryConfig.from_toml("dev.toml")  # or "prod.toml"
registry = Registry(config)
store = registry.get_store("default")
# Same API regardless of backend
```

Config-driven switching covers the sync backends. The async-only **Graph** backend has no config `type=` string — construct it directly via `AsyncStore(backend=GraphBackend(...))` (see the [Graph guide](https://docs.remotestore.dev/stable/guides/backends/graph/index.md)).

## See also

- [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) — full backend x capability table
- [Backends guide](https://docs.remotestore.dev/stable/guides/backends/index.md) — per-backend configuration details
- [Performance guide](https://docs.remotestore.dev/stable/explanation/performance/index.md) — benchmark data across backends

# Build Your Own Backend

Write file storage code once. Run it against local files, S3, SFTP, Azure — or your own custom storage system.

This guide walks you through implementing a custom [`Backend`](https://docs.remotestore.dev/stable/reference/api/backend/index.md) for remote-store. By the end, you'll have a working backend that plugs into [`Store`](https://docs.remotestore.dev/stable/reference/api/store/index.md), [`Registry`](https://docs.remotestore.dev/stable/reference/api/registry/index.md), and every extension in the ecosystem.

______________________________________________________________________

## What you'll build

A **Redis backend** that stores files as Redis keys. It's simple enough to fit in one module, yet exercises every part of the Backend contract: reads, writes, listing, metadata, error mapping, and capability declarations.

**Prerequisites:** `pip install remote-store redis`

______________________________________________________________________

## The Backend contract

Every backend is a subclass of [`Backend`](https://docs.remotestore.dev/stable/reference/api/backend/index.md). The contract is straightforward:

1. **Declare capabilities** — which operations does your backend support?
1. **Implement abstract members** — methods and properties covering CRUD, listing, and metadata. See [Abstract methods](#abstract-methods-must-implement) for the full list.
1. **Map all exceptions** — native errors must become `remote_store` errors. No leaks.

The [`Store`](https://docs.remotestore.dev/stable/reference/api/store/index.md) class wraps your backend, adds path validation, capability gating, and scoping. You implement the raw operations; `Store` handles the policy.

______________________________________________________________________

## Step 1: Scaffold the class

```
from __future__ import annotations

import contextlib
import io
from datetime import datetime, timezone
from typing import TYPE_CHECKING, BinaryIO, ClassVar

try:
    import redis
except ImportError:  # graceful fallback when redis is not installed
    redis = None  # type: ignore[assignment]

from remote_store import (
    AlreadyExists,
    Backend,
    BackendUnavailable,
    Capability,
    CapabilitySet,
    CapabilityNotSupported,
    DirectoryNotEmpty,
    FileInfo,
    FolderEntry,
    FolderInfo,
    InvalidPath,
    NotFound,
    PermissionDenied,
    RemotePath,
    WriteResult,
)

if TYPE_CHECKING:
    from collections.abc import Iterator, Mapping
    from contextlib import AbstractContextManager

    from remote_store._types import WritableContent
```

Every backend starts with these imports. The key types:

| Import                                                                                                                                                                                                                                                 | Purpose                                                                                                                 |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- |
| [`Backend`](https://docs.remotestore.dev/stable/reference/api/backend/index.md)                                                                                                                                                                        | Abstract base class you subclass                                                                                        |
| [`Capability`](https://docs.remotestore.dev/stable/reference/api/capabilities/index.md), [`CapabilitySet`](https://docs.remotestore.dev/stable/reference/api/capabilities/index.md)                                                                    | Declare supported operations                                                                                            |
| [`NotFound`](https://docs.remotestore.dev/stable/reference/api/errors/index.md), [`AlreadyExists`](https://docs.remotestore.dev/stable/reference/api/errors/index.md), ...                                                                             | Normalized error types                                                                                                  |
| [`FileInfo`](https://docs.remotestore.dev/stable/reference/api/models/index.md), [`FolderEntry`](https://docs.remotestore.dev/stable/reference/api/models/index.md), [`FolderInfo`](https://docs.remotestore.dev/stable/reference/api/models/index.md) | Return types for listing and metadata                                                                                   |
| [`RemotePath`](https://docs.remotestore.dev/stable/reference/api/models/index.md)                                                                                                                                                                      | Immutable, validated path type                                                                                          |
| [`WriteResult`](https://docs.remotestore.dev/stable/reference/api/models/index.md)                                                                                                                                                                     | Return type for `write()` and `write_atomic()`; carries the written path, size, and optional backend-native digest/etag |
| `WritableContent`                                                                                                                                                                                                                                      | Type alias: `bytes \| BinaryIO`                                                                                         |

______________________________________________________________________

## Step 2: Declare capabilities

```
# Redis doesn't support atomic rename or native glob.
_REDIS_CAPABILITIES = CapabilitySet(
    {
        Capability.READ,
        Capability.WRITE,
        Capability.DELETE,
        Capability.LIST,
        Capability.MOVE,
        Capability.COPY,
        Capability.METADATA,
        Capability.SEEKABLE_READ,  # We return BytesIO, which is always seekable
    }
)
```

**Capabilities gate Store methods.** If you don't declare `ATOMIC_WRITE`, calls to `store.write_atomic()` raise `CapabilityNotSupported` automatically — you don't need to handle it.

`_REDIS_CAPABILITIES` is assigned to the class-level `CAPABILITIES` attribute in Step 3. Tooling and conformance tests read `YourBackend.CAPABILITIES` without instantiating the class, so the constant must be a class attribute — not computed in `__init__`.

Three capabilities added in v0.23.0 are worth declaring when they apply:

- **`USER_METADATA`** — declare this when your backend stores the `metadata=` mapping passed to `write()` and `write_atomic()`. Without it, `Store` raises `CapabilityNotSupported` if the caller passes non-empty metadata.
- **`WRITE_RESULT_NATIVE`** — declare this when your backend populates `WriteResult` fields beyond the two mandatory ones (`path` and `size`). See the [WriteResult reference](https://docs.remotestore.dev/stable/reference/api/models/index.md) for the full field list.
- **`LAZY_READ`** — declare this when `read()` fetches data lazily from the remote source. A `BytesIO` return does not qualify — data is already materialized.

The Redis example declares neither `USER_METADATA` nor `WRITE_RESULT_NATIVE` (it stores raw bytes without a metadata column and returns only `path` and `size` at write time).

Each capability gates specific Store methods. See the [Capability reference](https://docs.remotestore.dev/stable/reference/api/capabilities/index.md) for the full list.

______________________________________________________________________

## Step 3: Constructor and properties

```
CAPABILITIES: ClassVar[CapabilitySet] = _REDIS_CAPABILITIES

def __init__(self, url: str = "redis://localhost:6379/0", prefix: str = "rs:") -> None:
    self._client = redis.Redis.from_url(url, decode_responses=False)
    self._prefix = prefix

@property
def name(self) -> str:
    return "redis"

@property
def capabilities(self) -> CapabilitySet:
    return self.CAPABILITIES
```

**Rules:**

- `name` must be a unique string. Used in error messages and the registry.
- `CAPABILITIES: ClassVar[CapabilitySet]` exposes the capability set at class level — no instantiation required. The `capabilities` property delegates to `self.CAPABILITIES` so both the class view and the instance view always agree.
- Constructor parameters become `options:` in YAML config (more on this later).

______________________________________________________________________

## Step 4: Internal helpers

Before implementing the abstract methods, add helpers for key management and error mapping.

```
# -- Key helpers --

def _key(self, path: str) -> str:
    """Convert a backend-relative path to a Redis key."""
    return f"{self._prefix}file:{path}"

def _folder_marker(self, path: str) -> str:
    """Key for folder existence markers."""
    return f"{self._prefix}dir:{path}"

def _all_file_keys_pattern(self) -> str:
    """Pattern to scan all file keys."""
    return f"{self._prefix}file:*"

def _path_from_key(self, key: bytes) -> str:
    """Extract the backend-relative path from a Redis key."""
    prefix = f"{self._prefix}file:"
    return key.decode().removeprefix(prefix)
```

Redis has no concept of folders, so we use key prefixes to simulate a hierarchical namespace. Files live under `rs:file:<path>`, and folder markers (optional) under `rs:dir:<path>`.

```
# -- Error mapping --

def _map_error(self, exc: redis.RedisError, path: str = "") -> None:
    """Map Redis exceptions to remote-store errors. Always raises."""
    if isinstance(exc, redis.AuthenticationError):
        raise PermissionDenied(
            f"Redis authentication failed: {exc}",
            path=path or None,
            backend=self.name,
        ) from exc
    if isinstance(exc, redis.ConnectionError):
        raise BackendUnavailable(
            f"Redis connection failed: {exc}",
            path=path or None,
            backend=self.name,
        ) from exc
    raise BackendUnavailable(
        f"Redis error: {exc}",
        path=path or None,
        backend=self.name,
    ) from exc
```

**The cardinal rule:** backend-native exceptions must never leak. Every Redis error becomes a `remote_store` error. The `from exc` preserves the original traceback for debugging.

______________________________________________________________________

## Step 5: Existence checks

```
def exists(self, path: str) -> bool:
    if not path or path == ".":
        return True  # Root always exists
    try:
        return bool(self._client.exists(self._key(path)) or self._has_children(path))
    except redis.RedisError as exc:
        self._map_error(exc, path)

def is_file(self, path: str) -> bool:
    if not path or path == ".":
        return False
    try:
        return bool(self._client.exists(self._key(path)))
    except redis.RedisError as exc:
        self._map_error(exc, path)

def is_folder(self, path: str) -> bool:
    if not path or path == ".":
        return True  # Root is always a folder
    try:
        return self._has_children(path)
    except redis.RedisError as exc:
        self._map_error(exc, path)

def _has_children(self, path: str) -> bool:
    """Check if any keys exist under this path prefix."""
    pattern = f"{self._prefix}file:{path}/*"
    cursor, keys = self._client.scan(cursor=0, match=pattern, count=1)
    return bool(keys)
```

**Key invariants:**

- `exists()` **never raises `NotFound`** — always returns `bool`.
- `""` and `"."` are root aliases. Root always exists and is always a folder.
- `is_file("")` is always `False`. `is_folder("")` is always `True`.

______________________________________________________________________

## Step 6: Reading

```
def read(self, path: str) -> BinaryIO:
    try:
        data = self._client.hget(self._key(path), "data")
    except redis.RedisError as exc:
        self._map_error(exc, path)
    if data is None:
        raise NotFound(f"File not found: {path}", path=path, backend=self.name)
    return io.BytesIO(data)

def read_bytes(self, path: str) -> bytes:
    try:
        data = self._client.hget(self._key(path), "data")
    except redis.RedisError as exc:
        self._map_error(exc, path)
    if data is None:
        raise NotFound(f"File not found: {path}", path=path, backend=self.name)
    return bytes(data)
```

**Notes:**

- `read()` returns a `BinaryIO`. Since we return `BytesIO`, streams are seekable — that's why we declared `SEEKABLE_READ`.
- `read_bytes()` can be more efficient than `read().read()` because it avoids wrapping in a stream object.
- Both raise `NotFound` for missing files.

Since our `read()` returns seekable streams, we don't need to override `read_seekable()` — the default implementation detects seekability and returns the stream as-is.

______________________________________________________________________

## Step 7: Writing

```
def write(
    self,
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult:
    if not path or path == ".":
        raise InvalidPath(
            "Path must not be empty for file operations",
            path=path,
            backend=self.name,
        )

    raw = content if isinstance(content, bytes) else content.read()

    try:
        if not overwrite and self._client.exists(self._key(path)):
            raise AlreadyExists(
                f"File already exists: {path}",
                path=path,
                backend=self.name,
            )
        self._client.hset(
            self._key(path),
            mapping={
                "data": raw,
                "size": str(len(raw)),
                "modified_at": datetime.now(timezone.utc).isoformat(),
            },
        )
    except (AlreadyExists, InvalidPath):
        raise  # Don't re-map our own errors
    except redis.RedisError as exc:
        self._map_error(exc, path)

    return WriteResult(path=RemotePath(path), size=len(raw))

def write_atomic(
    self,
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult:
    # Redis HSET is already atomic, but we didn't declare ATOMIC_WRITE.
    # Store will reject this call before it reaches us.
    # If you want to support it, declare the capability and implement here.
    raise CapabilityNotSupported(
        "Redis backend does not support atomic writes",
        capability="atomic_write",
        backend=self.name,
    )

@contextlib.contextmanager
def open_atomic(self, path: str, *, overwrite: bool = False) -> Iterator[BinaryIO]:
    raise CapabilityNotSupported(
        "Redis backend does not support atomic writes",
        capability="atomic_write",
        backend=self.name,
    )
    yield  # Unreachable, but satisfies the generator contract
```

**Key patterns:**

- `content` is `bytes | BinaryIO`. Normalize with `content if isinstance(content, bytes) else content.read()`.
- **Both `write()` and `write_atomic()` accept `metadata: Mapping[str, str] | None = None`.** If your backend declares `USER_METADATA`, persist the mapping alongside the file. If it doesn't, ignore the argument — `Store` rejects non-empty metadata before reaching your implementation.
- **Both methods must return [`WriteResult`](https://docs.remotestore.dev/stable/reference/api/models/index.md).** Construct it with at minimum `path=RemotePath(path)` and `size=len(raw)`. If your backend can populate richer fields, declare `WRITE_RESULT_NATIVE` and include them. The Redis example returns the two-field minimum.
- **Write creates parent folders implicitly** — in Redis, there's nothing to create, but filesystem-based backends must `mkdir -p`.
- Re-raise your own errors (`AlreadyExists`, `InvalidPath`) before the catch-all `RedisError` handler.
- Even though Store gates `write_atomic()` via capabilities, implement the methods anyway (they're abstract). Raise `CapabilityNotSupported` as a safety net.

______________________________________________________________________

## Step 8: Deletion

```
def delete(self, path: str, *, missing_ok: bool = False) -> None:
    if not path or path == ".":
        raise InvalidPath(
            "Path must not be empty for file operations",
            path=path,
            backend=self.name,
        )
    try:
        removed = self._client.delete(self._key(path))
    except redis.RedisError as exc:
        self._map_error(exc, path)
    if not removed and not missing_ok:
        raise NotFound(f"File not found: {path}", path=path, backend=self.name)

def delete_folder(self, path: str, *, recursive: bool = False, missing_ok: bool = False) -> None:
    if not path or path == ".":
        raise InvalidPath(
            "Cannot delete root folder",
            path=path,
            backend=self.name,
        )

    try:
        children = list(self._iter_file_paths_under(path))
    except redis.RedisError as exc:
        self._map_error(exc, path)

    if not children and not self._has_children(path):
        if not missing_ok:
            raise NotFound(f"Folder not found: {path}", path=path, backend=self.name)
        return

    if children and not recursive:
        raise DirectoryNotEmpty(
            f"Folder not empty: {path}",
            path=path,
            backend=self.name,
        )

    if recursive:
        try:
            keys = [self._key(p) for p in children]
            if keys:
                self._client.delete(*keys)
        except redis.RedisError as exc:
            self._map_error(exc, path)
```

**Invariants:**

- `delete()` targets files. `delete_folder()` targets folders.
- `missing_ok=True` suppresses `NotFound`.
- `delete_folder(recursive=False)` raises `DirectoryNotEmpty` if the folder has contents.
- You cannot delete root (`""` or `"."`).

______________________________________________________________________

## Step 9: Listing

```
def list_files(self, path: str, *, recursive: bool = False) -> Iterator[FileInfo]:
    try:
        for file_path in self._iter_file_paths_under(path):
            # If not recursive, only yield immediate children
            if not recursive:
                rel = file_path.removeprefix(f"{path}/" if path else "")
                if "/" in rel:
                    continue  # Skip nested files

            info = self._build_file_info(file_path)
            if info is not None:
                yield info
    except redis.RedisError as exc:
        self._map_error(exc, path)

def list_folders(self, path: str) -> Iterator[FolderEntry]:
    seen: set[str] = set()
    try:
        for file_path in self._iter_file_paths_under(path):
            # Extract the immediate subfolder name
            prefix = f"{path}/" if path else ""
            rel = file_path.removeprefix(prefix)
            if "/" in rel:
                folder_name = rel.split("/", 1)[0]
                if folder_name not in seen:
                    seen.add(folder_name)
                    folder_path = f"{prefix}{folder_name}"
                    yield FolderEntry(
                        path=RemotePath(folder_path),
                        name=folder_name,
                    )
    except redis.RedisError as exc:
        self._map_error(exc, path)

def _iter_file_paths_under(self, path: str) -> Iterator[str]:
    """Scan Redis for all file keys under a path prefix."""
    if path and path != ".":
        pattern = f"{self._prefix}file:{path}/*"
    else:
        pattern = self._all_file_keys_pattern()

    cursor = 0
    while True:
        cursor, keys = self._client.scan(cursor=cursor, match=pattern, count=100)
        for key in keys:
            yield self._path_from_key(key)
        if cursor == 0:
            break

def _build_file_info(self, path: str) -> FileInfo | None:
    """Build a FileInfo from Redis hash fields."""
    fields = self._client.hgetall(self._key(path))
    if not fields:
        return None
    return FileInfo(
        path=RemotePath(path),
        name=path.rsplit("/", 1)[-1],
        size=int(fields.get(b"size", b"0")),
        modified_at=datetime.fromisoformat(fields[b"modified_at"].decode()),
    )
```

**Key rules:**

- `list_files(path="")` lists from root.
- `recursive=False` (default) yields only immediate children.
- `list_folders()` is always non-recursive — only immediate subfolders.
- Non-existent paths yield nothing (no exception).
- [`FileInfo`](https://docs.remotestore.dev/stable/reference/api/models/index.md)`.path` must be a [`RemotePath`](https://docs.remotestore.dev/stable/reference/api/models/index.md).

______________________________________________________________________

## Step 10: Metadata

```
def get_file_info(self, path: str) -> FileInfo:
    if not path or path == ".":
        raise NotFound("File not found: (empty path)", path=path, backend=self.name)
    try:
        info = self._build_file_info(path)
    except redis.RedisError as exc:
        self._map_error(exc, path)
    if info is None:
        raise NotFound(f"File not found: {path}", path=path, backend=self.name)
    return info

def get_folder_info(self, path: str) -> FolderInfo:
    try:
        file_count = 0
        total_size = 0
        latest: datetime | None = None

        for file_path in self._iter_file_paths_under(path):
            info = self._build_file_info(file_path)
            if info is not None:
                file_count += 1
                total_size += info.size
                if latest is None or info.modified_at > latest:
                    latest = info.modified_at
    except redis.RedisError as exc:
        self._map_error(exc, path)

    if file_count == 0 and path and path != ".":
        raise NotFound(f"Folder not found: {path}", path=path, backend=self.name)

    return FolderInfo(
        path=RemotePath.from_backend_path(path),
        file_count=file_count,
        total_size=total_size,
        modified_at=latest,
    )
```

**Contrast with existence checks:**

- `get_file_info()` raises `NotFound` if missing.
- `get_folder_info()` raises `NotFound` if the folder doesn't exist.
- `exists()` never raises — returns `bool`.

______________________________________________________________________

## Step 11: Move and copy

```
def move(self, src: str, dst: str, *, overwrite: bool = False) -> None:
    if not src or src == ".":
        raise InvalidPath("Source path must not be empty", path=src, backend=self.name)
    if not dst or dst == ".":
        raise InvalidPath("Destination path must not be empty", path=dst, backend=self.name)

    try:
        # Read source
        data = self._client.hgetall(self._key(src))
        if not data:
            raise NotFound(f"Source not found: {src}", path=src, backend=self.name)

        # Check destination
        if not overwrite and self._client.exists(self._key(dst)):
            raise AlreadyExists(
                f"Destination already exists: {dst}",
                path=dst,
                backend=self.name,
            )

        # Atomic: write destination then delete source
        pipe = self._client.pipeline()
        pipe.hset(self._key(dst), mapping=data)
        pipe.delete(self._key(src))
        pipe.execute()
    except (NotFound, AlreadyExists, InvalidPath):
        raise
    except redis.RedisError as exc:
        self._map_error(exc, src)

def copy(self, src: str, dst: str, *, overwrite: bool = False) -> None:
    if not src or src == ".":
        raise InvalidPath("Source path must not be empty", path=src, backend=self.name)
    if not dst or dst == ".":
        raise InvalidPath("Destination path must not be empty", path=dst, backend=self.name)

    try:
        data = self._client.hgetall(self._key(src))
        if not data:
            raise NotFound(f"Source not found: {src}", path=src, backend=self.name)

        if not overwrite and self._client.exists(self._key(dst)):
            raise AlreadyExists(
                f"Destination already exists: {dst}",
                path=dst,
                backend=self.name,
            )

        # Update modified_at for the copy
        data[b"modified_at"] = datetime.now(timezone.utc).isoformat().encode()
        self._client.hset(self._key(dst), mapping=data)
    except (NotFound, AlreadyExists, InvalidPath):
        raise
    except redis.RedisError as exc:
        self._map_error(exc, src)
```

______________________________________________________________________

## Step 12: Lifecycle methods

```
def check_health(self) -> None:
    try:
        self._client.ping()
    except redis.AuthenticationError as exc:
        raise PermissionDenied(
            f"Redis authentication failed: {exc}",
            backend=self.name,
        ) from exc
    except redis.RedisError as exc:
        raise BackendUnavailable(
            f"Redis is not reachable: {exc}",
            backend=self.name,
        ) from exc

def close(self) -> None:
    self._client.close()
```

`check_health()` should be the **cheapest possible read-only operation**. Redis `PING` is ideal. For S3 it's a `HEAD` on the bucket. For a database it's `SELECT 1`.

______________________________________________________________________

## Step 13: Register and use

### Direct instantiation

```
from remote_store import Store

backend = RedisBackend(url="redis://localhost:6379/0", prefix="myapp:")
store = Store(backend=backend)

store.write("reports/q1.csv", b"revenue,100\n")
data = store.read_bytes("reports/q1.csv")
print(data)  # b'revenue,100\n'

for info in store.list_files("reports"):
    print(f"{info.name}: {info.size} bytes")
```

### Via Registry (YAML config)

Register your backend type before creating a [`Registry`](https://docs.remotestore.dev/stable/reference/api/registry/index.md):

```
from remote_store import Registry, RegistryConfig, register_backend

register_backend("redis", RedisBackend)

config = RegistryConfig.from_yaml("stores.yaml")
registry = Registry(config)
store = registry.get_store("cache")
```

```
# stores.yaml
backends:
  redis-main:
    type: redis
    options:
      url: "redis://localhost:6379/0"
      prefix: "app:"

stores:
  cache:
    backend: redis-main
    root_path: "cache/v2"
```

The `options` dict is unpacked as `**kwargs` to your constructor. Parameter names in YAML must match your `__init__` signature exactly.

______________________________________________________________________

## Step 14: Extensions work automatically

Because your backend implements the `Backend` contract, every remote-store extension works out of the box:

```
from remote_store.ext.batch import batch_copy
from remote_store.ext.cache import cache
from remote_store.ext.observe import observe

# Observability
observed = observe(store, hooks=[my_logging_hook])

# Caching
fast = cache(store, ttl=300)

# Batch operations
results = batch_copy(store, [("a.txt", "b.txt"), ("c.txt", "d.txt")])
```

Extensions that require specific capabilities will check at runtime. For example, `ext.glob.glob_files()` works with any `LIST`-capable backend — it doesn't need the `GLOB` capability.

______________________________________________________________________

## Partial-capability backends

Not every backend supports every operation. The HTTP backend, for example, is read-only:

```
class _ReadOnlyBackend(Backend):  # type: ignore[abstract]
    CAPABILITIES: ClassVar[CapabilitySet] = CapabilitySet(
        {
            Capability.READ,
            Capability.LIST,
            Capability.METADATA,
        }
    )

    @property
    def capabilities(self) -> CapabilitySet:
        return self.CAPABILITIES
```

When a user calls `store.write()` on an HTTP-backed store, the `Store` layer raises `CapabilityNotSupported` before your backend code runs. You still need to implement the abstract methods (Python requires it), but they can raise `CapabilityNotSupported`:

```
def write(
    self,
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult:
    raise CapabilityNotSupported(
        "HTTP backend is read-only",
        capability="write",
        backend=self.name,
    )
```

______________________________________________________________________

## Error mapping checklist

Every backend-native exception must map to one of these:

| remote-store error                                                                            | When to raise                                              |
| --------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| [`NotFound`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)               | File/folder doesn't exist (for operations that require it) |
| [`AlreadyExists`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)          | Target exists and `overwrite=False`                        |
| [`PermissionDenied`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)       | Auth failure, insufficient permissions                     |
| [`InvalidPath`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)            | Malformed path, null bytes, `..` traversal                 |
| [`DirectoryNotEmpty`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)      | Non-empty folder and `recursive=False`                     |
| [`BackendUnavailable`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)     | Network error, service down                                |
| [`CapabilityNotSupported`](https://docs.remotestore.dev/stable/reference/api/errors/index.md) | Operation not supported by this backend                    |

**Pattern:** catch the SDK's base exception class, classify by error code/type, and raise the appropriate remote-store error with `from exc`.

______________________________________________________________________

## Testing your backend

remote-store ships a per-topic conformance suite under `tests/backends/conformance/` that validates any backend against the formal `BackendContract` specification. Backends contributed to the repo plug into this infrastructure and run through the full suite automatically. Standalone backends can either reuse this suite or write focused tests against the same categories.

______________________________________________________________________

### Conformance suite overview

The suite lives in [`tests/backends/conformance/`](https://github.com/haalfi/remote-store/tree/master/tests/backends/conformance), split into per-topic files that share the same parameterized `backend` fixture — every registered backend runs the full suite automatically.

| Topic file                                                                                                                   | Coverage                                                                         | Run with                                                 |
| ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------- |
| [`test_identity.py`](https://github.com/haalfi/remote-store/blob/master/tests/backends/conformance/test_identity.py)         | Identity, capabilities, lifecycle, `resolve`, native path round-trip             | `pytest tests/backends/conformance/test_identity.py`     |
| [`test_io.py`](https://github.com/haalfi/remote-store/blob/master/tests/backends/conformance/test_io.py)                     | `exists`, `is_file`/`is_folder`, read, write, delete, `to_key` round-trip        | `pytest tests/backends/conformance/test_io.py`           |
| [`test_listing.py`](https://github.com/haalfi/remote-store/blob/master/tests/backends/conformance/test_listing.py)           | `list_files`/`list_folders`, `iter_children`, glob, completeness                 | `pytest tests/backends/conformance/test_listing.py`      |
| [`test_atomic.py`](https://github.com/haalfi/remote-store/blob/master/tests/backends/conformance/test_atomic.py)             | `write_atomic`, `open_atomic` (SAW-*), `WriteResult` (WR-*), move/copy semantics | `pytest tests/backends/conformance/test_atomic.py`       |
| [`test_metadata.py`](https://github.com/haalfi/remote-store/blob/master/tests/backends/conformance/test_metadata.py)         | `get_file_info`/`get_folder_info`, `size`, `modified_at`, aggregates             | `pytest tests/backends/conformance/test_metadata.py`     |
| [`test_streaming.py`](https://github.com/haalfi/remote-store/blob/master/tests/backends/conformance/test_streaming.py)       | Streaming reads, `LAZY_READ` laziness, resource cleanup                          | `pytest tests/backends/conformance/test_streaming.py`    |
| [`test_errors.py`](https://github.com/haalfi/remote-store/blob/master/tests/backends/conformance/test_errors.py)             | Typed-error fidelity across read/write/delete/move/copy paths                    | `pytest tests/backends/conformance/test_errors.py`       |
| [`test_check_health.py`](https://github.com/haalfi/remote-store/blob/master/tests/backends/conformance/test_check_health.py) | `check_health()` contract — error mapping never leaks native SDK exceptions      | `pytest tests/backends/conformance/test_check_health.py` |

Run the whole suite at once with `pytest tests/backends/conformance/`.

**Extended (Dafny-derived) cases** — error fidelity, precondition ordering, depth filtering, move/copy edge semantics, resource cleanup — are not a separate file. They are individual tests marked [`@pytest.mark.extended_conformance`](https://github.com/haalfi/remote-store/tree/master/tests/backends/conformance) spread across the topic files above, so they run with the rest of the suite by default and can be selected on their own:

```
pytest -m extended_conformance
```

Async backends have their own extended sibling, [`test_async_extended.py`](https://github.com/haalfi/remote-store/blob/master/tests/backends/conformance/test_async_extended.py), which exercises the `AsyncBackend` contract (ASYNC- *mirroring BE-*).

The conformance suite itself is validated by running it against a mathematically verified oracle compiled from the formal Dafny specification (`sdd/formal/MemoryBackend.dfy`). If the oracle passes a test, the test is known-correct. This means passing the conformance suite is a strong guarantee of correctness — not just "matches what existing backends happen to do." See [`sdd/formal/README.md`](https://github.com/haalfi/remote-store/blob/master/sdd/formal/README.md) § Compiled Oracle for details.

______________________________________________________________________

### Registering in the conformance fixture (contributing backends)

If you are contributing a backend to remote-store, this is step 3 of [CONTRIBUTING.md § Adding a New Backend](https://docs.remotestore.dev/stable/explanation/contributing/#adding-a-new-backend). Add your backend to [`tests/backends/conftest.py`](https://github.com/haalfi/remote-store/blob/master/tests/backends/conftest.py); the entire conformance suite then runs against it automatically.

**1. Add an availability guard near the top of `conftest.py`:**

```
def _redis_available() -> bool:
    try:
        import redis  # noqa: F401
        return True
    except ImportError:
        return False
```

**2. Add a `pytest.param` constant:**

```
_redis_param = pytest.param(
    "redis",
    marks=pytest.mark.skipif(not _redis_available(), reason="redis-py not installed"),
)
```

If your backend requires an external service (like S3, SFTP, or Azurite), add a reachability check and a session-scoped server fixture following the existing `moto_server` / `sftp_server` / `azurite_server` pattern.

**3. Add it to the `backend` fixture's `params` list and `elif` branch:**

```
@pytest.fixture(
    params=[
        _local_param,
        _memory_param,
        # ... existing params ...
        _redis_param,   # ← add here
    ]
)
def backend(request, moto_server, sftp_server, azurite_server, http_server):
    ...
    elif request.param == "redis":
        from remote_store.backends._redis import RedisBackend
        b = RedisBackend(url="redis://localhost:6379/0", prefix=f"test-{uuid.uuid4().hex}:")
        yield b
        b.close()
```

`conftest.py` already imports `uuid` at the top — ensure yours does too if you are starting from scratch. Use a unique prefix per test so isolation is guaranteed even without a full teardown.

______________________________________________________________________

### Capability gating with `_require()`

Backends may declare a subset of capabilities. The `_require()` helper skips a test when the backend lacks the needed capability — so a read-only backend cleanly skips all write, move, copy, and delete tests without failures:

```
def _require(backend: Backend, *caps: Capability) -> None:
    for cap in caps:
        if not backend.capabilities.supports(cap):
            pytest.skip(f"Backend does not support {cap.name}")
```

Use the same pattern in your own tests:

```
from remote_store import Capability
import pytest

def test_move_preserves_content(backend):
    _require(backend, Capability.MOVE)
    backend.write("src.txt", b"hello")
    backend.move("src.txt", "dst.txt")
    assert backend.read_bytes("dst.txt") == b"hello"
```

A backend declaring only `READ` and `LIST` will skip every `WRITE`, `MOVE`, `COPY`, and `DELETE` test. The suite still passes — skips are not failures.

______________________________________________________________________

### Flat-namespace vs. hierarchical backends

Backends fall into two models that affect a handful of conformance tests.

**Hierarchical** backends (Local, SFTP, Memory) have real directory objects. Writing a file creates its parent directories; a path can be either a file *or* a directory, never both.

**Flat-namespace** backends (S3, Azure, HTTP) have no real directory entries. Folders are virtual — inferred from key prefixes. A path `a/b/c` implies a prefix `a/b/` but no actual directory object exists.

The conformance suite reads this from the per-backend `flat_namespace` flag declared in `tests/backends/fixtures/backends.toml`:

```
# tests/backends/fixtures/backends.toml
[backend.<your-backend>]
transport         = "fs"        # http | ssh | fs | memory | sql
flat_namespace    = true        # set true for flat-namespace backends
self_op_supported = true
```

A per-fixture override in `fixtures.toml` is also possible — Azurite (the flat emulator) and live ADLS Gen2 (HNS) share `backend == "azure"` but disagree on `flat_namespace`, so the fixture-level value takes precedence. Tests that rely on real directory semantics call `_skip_flat_namespace()`, which reads the resolved flag from the per-fixture record attached by the conformance indirect fixture; no identity-set lookup is needed.

Key behavioral differences that the conformance tests check:

| Behavior                                             | Hierarchical (Local, SFTP, Memory) | Flat-namespace (S3, Azure, HTTP)             |
| ---------------------------------------------------- | ---------------------------------- | -------------------------------------------- |
| Write to a path that is an existing directory        | Raises `InvalidPath`               | Typically allowed (no real directory)        |
| `delete_folder(recursive=False)` on non-empty folder | Raises `DirectoryNotEmpty`         | Behaviour varies; some tests are skipped     |
| Explicit directory creation                          | Required (mkdir semantics)         | Not needed; folders emerge from key prefixes |
| `is_folder(path)` for a prefix with no keys          | `False`                            | `False`                                      |

If your backend is hierarchical (the common case), no action is needed — the full extended suite applies.

______________________________________________________________________

### Conformance checklist

Before a backend is considered conformant, verify:

| Level             | What                                                                                    | Command                                                                         |
| ----------------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| **Conformance**   | All `tests/backends/conformance/` tests pass or self-skip (declared capability missing) | `pytest tests/backends/conformance/ -k <backend-name>`                          |
| **Extended**      | All `@pytest.mark.extended_conformance` cases pass or self-skip                         | `pytest -m extended_conformance -k <backend-name>`                              |
| **Error mapping** | Every native exception maps to a `remote_store` error — nothing leaks                   | Error mapping checklist above                                                   |
| **Repr safety**   | `repr(backend)` does not expose secrets                                                 | `pytest tests/backends/conformance/test_identity.py -k test_repr_masks_secrets` |

Skips are expected and acceptable when a backend doesn't declare the relevant capability. Failures (not skips) in either suite are blocking.

______________________________________________________________________

### Standalone backend testing

> **Not contributing to the repo?** Skip the fixture registration above and write focused tests directly. The categories below mirror what the conformance suite verifies.

If you are building a backend outside the remote-store repository, write focused tests covering the same categories the conformance suite verifies:

#### Happy paths

- Read/write round-trip
- Overwrite behavior (`overwrite=True` and `overwrite=False`)
- List files and folders (recursive and non-recursive)
- Move and copy
- Metadata accuracy (`size`, `modified_at`)

#### Error paths

- `read()` on missing file raises `NotFound`
- `write()` on existing file with `overwrite=False` raises `AlreadyExists`
- `delete(missing_ok=False)` on missing file raises `NotFound`
- `delete_folder(recursive=False)` on non-empty folder raises `DirectoryNotEmpty`
- Path naming a wrong type (file path to `get_folder_info`, directory path to `read`) raises `InvalidPath`
- Backend unavailable raises `BackendUnavailable`

#### Edge cases

- Empty path (`""`) and root alias (`"."`) — root always exists and is always a folder
- `is_file("")` always returns `False`; `exists("")` never raises
- Deeply nested paths (`"a/b/c/d/e/file.txt"`)
- Non-existent paths to `list_files` / `list_folders` yield nothing (no exception)
- `repr(backend)` does not expose credentials or secrets
- Concurrent access (if thread-safety matters)

#### Example test structure

```
import pytest
from remote_store import AlreadyExists, NotFound, Store

@pytest.fixture
def store():
    backend = RedisBackend(url="redis://localhost:6379/15", prefix="test:")
    backend._client.flushdb()  # Clean slate
    return Store(backend=backend)

def test_read_write_roundtrip(store):
    store.write("hello.txt", b"world")
    assert store.read_bytes("hello.txt") == b"world"

def test_write_no_overwrite(store):
    store.write("hello.txt", b"first")
    with pytest.raises(AlreadyExists):
        store.write("hello.txt", b"second")

def test_read_missing(store):
    with pytest.raises(NotFound):
        store.read("nope.txt")

def test_list_files(store):
    store.write("a/1.txt", b"one")
    store.write("a/2.txt", b"two")
    store.write("b/3.txt", b"three")
    files = list(store.list_files("a"))
    assert len(files) == 2
    names = {f.name for f in files}
    assert names == {"1.txt", "2.txt"}

def test_list_files_recursive(store):
    store.write("a/b/deep.txt", b"deep")
    store.write("a/top.txt", b"top")
    files = list(store.list_files("a", recursive=True))
    assert len(files) == 2

def test_list_folders(store):
    store.write("docs/readme.md", b"# Hello")
    store.write("src/main.py", b"pass")
    folders = {f.name for f in store.list_folders("")}
    assert "docs" in folders
    assert "src" in folders
```

______________________________________________________________________

## Design decisions

### When to declare `SEEKABLE_READ`

Declare it only if `read()` **always** returns a seekable stream with zero overhead. `BytesIO` qualifies. Streams backed by network iterators don't.

If your `read()` returns a non-seekable stream, don't worry — `Store` handles it. `read_seekable()` will spool to a temp file automatically. You can also override `read_seekable()` for an optimized path (like Azure's HTTP Range reader).

### When to support `ATOMIC_WRITE`

Support it if your backend can guarantee that readers never see partial content. Filesystem backends use temp-file-and-rename. Databases can use transactions. If your backend's writes are inherently atomic (single Redis `HSET`), you could declare it — but be honest about the guarantee. "Atomic at the key level" isn't the same as "atomic rename of a visible path."

### Thread safety

Backends may be called from multiple threads (e.g., `batch_copy` with concurrency). Use locking if your internal state is mutable. Redis clients are generally thread-safe, so our example doesn't need explicit locking.

______________________________________________________________________

## Quick reference

### Abstract methods (must implement)

| Member                                                  | Type                                                                                                 | Raises on error                                                                                                                                                                     |
| ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `CAPABILITIES` (class attribute)                        | [`ClassVar[CapabilitySet]`](https://docs.remotestore.dev/stable/reference/api/capabilities/index.md) | —                                                                                                                                                                                   |
| `name` (property)                                       | `str`                                                                                                | —                                                                                                                                                                                   |
| `capabilities` (property)                               | [`CapabilitySet`](https://docs.remotestore.dev/stable/reference/api/capabilities/index.md)           | —                                                                                                                                                                                   |
| `exists(path)`                                          | `bool`                                                                                               | Never raises [`NotFound`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)                                                                                        |
| `is_file(path)`                                         | `bool`                                                                                               | —                                                                                                                                                                                   |
| `is_folder(path)`                                       | `bool`                                                                                               | —                                                                                                                                                                                   |
| `read(path)`                                            | `BinaryIO`                                                                                           | [`NotFound`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)                                                                                                     |
| `read_bytes(path)`                                      | `bytes`                                                                                              | [`NotFound`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)                                                                                                     |
| `write(path, content, overwrite, metadata=None)`        | [`WriteResult`](https://docs.remotestore.dev/stable/reference/api/models/index.md)                   | [`AlreadyExists`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)                                                                                                |
| `write_atomic(path, content, overwrite, metadata=None)` | [`WriteResult`](https://docs.remotestore.dev/stable/reference/api/models/index.md)                   | [`AlreadyExists`](https://docs.remotestore.dev/stable/reference/api/errors/index.md), [`CapabilityNotSupported`](https://docs.remotestore.dev/stable/reference/api/errors/index.md) |
| `open_atomic(path, overwrite)`                          | `ContextManager[BinaryIO]`                                                                           | [`AlreadyExists`](https://docs.remotestore.dev/stable/reference/api/errors/index.md), [`CapabilityNotSupported`](https://docs.remotestore.dev/stable/reference/api/errors/index.md) |
| `delete(path, missing_ok)`                              | `None`                                                                                               | [`NotFound`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)                                                                                                     |
| `delete_folder(path, recursive, missing_ok)`            | `None`                                                                                               | [`NotFound`](https://docs.remotestore.dev/stable/reference/api/errors/index.md), [`DirectoryNotEmpty`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)           |
| `list_files(path, recursive)`                           | `Iterator[`[`FileInfo`](https://docs.remotestore.dev/stable/reference/api/models/index.md)`]`        | —                                                                                                                                                                                   |
| `list_folders(path)`                                    | `Iterator[`[`FolderEntry`](https://docs.remotestore.dev/stable/reference/api/models/index.md)`]`     | —                                                                                                                                                                                   |
| `get_file_info(path)`                                   | [`FileInfo`](https://docs.remotestore.dev/stable/reference/api/models/index.md)                      | [`NotFound`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)                                                                                                     |
| `get_folder_info(path)`                                 | [`FolderInfo`](https://docs.remotestore.dev/stable/reference/api/models/index.md)                    | [`NotFound`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)                                                                                                     |
| `move(src, dst, overwrite)`                             | `None`                                                                                               | [`NotFound`](https://docs.remotestore.dev/stable/reference/api/errors/index.md), [`AlreadyExists`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)               |
| `copy(src, dst, overwrite)`                             | `None`                                                                                               | [`NotFound`](https://docs.remotestore.dev/stable/reference/api/errors/index.md), [`AlreadyExists`](https://docs.remotestore.dev/stable/reference/api/errors/index.md)               |

### Optional overrides

| Method                | Default behavior                         |
| --------------------- | ---------------------------------------- |
| `read_seekable(path)` | Spools non-seekable streams to temp file |
| `iter_children(path)` | Chains `list_files()` + `list_folders()` |
| `glob(pattern)`       | Raises `CapabilityNotSupported`          |
| `to_key(native_path)` | Identity function                        |
| `native_path(path)`   | Identity function                        |
| `check_health()`      | No-op                                    |
| `close()`             | No-op                                    |
| `unwrap(type_hint)`   | Raises `CapabilityNotSupported`          |

______________________________________________________________________

## See also

- [Backend API reference](https://docs.remotestore.dev/stable/reference/api/backend/index.md) — full method documentation
- [Error types API reference](https://docs.remotestore.dev/stable/reference/api/errors/index.md) — all error classes
- [Backend Adapter Contract](https://docs.remotestore.dev/stable/explanation/design/specs/003-backend-adapter-contract/index.md) — formal spec
- [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) — all backends and their capabilities
- [Choosing a Backend](https://docs.remotestore.dev/stable/guides/choosing-a-backend/index.md) — decision guide for built-in backends
- [Architecture Overview](https://docs.remotestore.dev/stable/explanation/architecture/index.md) — how Store, Backend, and extensions fit together

# Dagster Integration

How to use remote-store as the IO manager and compute log backend for Dagster pipelines.

## The idea

Teams already using remote-store should not duplicate their Store configuration (credentials, retry policy, caching, observability) into `dagster-aws` / `dagster-azure`. The `dagster_io_manager` adapter wraps any existing `Store` as a Dagster `IOManager` with zero config duplication.

This also fills a gap for **SFTP backends** — Dagster has no native SFTP IO manager, but remote-store covers that backend directly.

## Installation

```
pip install "remote-store[dagster]"

# For Parquet serializer support:
pip install "remote-store[dagster,arrow]"
```

## Quick start

```
from dagster import Definitions, asset, IOManager, io_manager
from remote_store import Store
from remote_store.backends import LocalBackend
from remote_store.ext.dagster import dagster_io_manager


@io_manager
def my_io_manager() -> IOManager:
    store = Store(LocalBackend(root="/data/dagster"))
    return dagster_io_manager(store, serializer="pickle")


@asset
def raw_data() -> dict:
    return {"rows": [1, 2, 3]}


defs = Definitions(
    assets=[raw_data],
    resources={"io_manager": my_io_manager},
)
```

## Serializers

The `serializer` parameter controls how Python objects are converted to bytes for storage and back.

| Name    | `serializer=`        | Extension  | Dependency      | Best for                                  |
| ------- | -------------------- | ---------- | --------------- | ----------------------------------------- |
| Pickle  | `"pickle"` (default) | `.pkl`     | stdlib          | Any picklable object                      |
| JSON    | `"json"`             | `.json`    | stdlib          | JSON-serializable data                    |
| Parquet | `"parquet"`          | `.parquet` | `pyarrow>=14.0` | DataFrames (pandas, polars), Arrow Tables |

### Pickle (default)

```
mgr = dagster_io_manager(store, serializer="pickle")
```

Universal — works with any picklable Python object. The default choice when you don't need human-readable storage.

### JSON

```
mgr = dagster_io_manager(store, serializer="json")
```

Human-readable, but limited to JSON-serializable types (dicts, lists, strings, numbers, booleans, None).

### Parquet

```
mgr = dagster_io_manager(store, serializer="parquet")
```

Efficient columnar storage for DataFrames. Requires PyArrow. Accepts pandas DataFrames, polars DataFrames (via `.to_arrow()`), and Arrow Tables. Deserializes to a PyArrow Table.

## Custom serializer

Any object matching the `Serializer` protocol can be used:

```
from remote_store.ext.dagster import Serializer, dagster_io_manager


class MsgpackSerializer:
    extension = ".msgpack"

    def serialize(self, obj):
        import msgpack
        return msgpack.packb(obj)

    def deserialize(self, data):
        import msgpack
        return msgpack.unpackb(data)


mgr = dagster_io_manager(store, serializer=MsgpackSerializer())
```

## Path generation

Storage paths are derived automatically from Dagster asset keys and partition keys:

| Asset key           | Partition   | Path                     |
| ------------------- | ----------- | ------------------------ |
| `["raw", "events"]` | *(none)*    | `raw/events.pkl`         |
| `["raw", "events"]` | `"2026-01"` | `raw/events/2026-01.pkl` |
| `["report"]`        | *(none)*    | `report.pkl`             |

The Store's `root_path` acts as a namespace prefix — it is not embedded in the path.

## Multi-partition loading

When a downstream asset consumes multiple partitions of an upstream asset (e.g. a time-window aggregation), `load_input` automatically returns a `dict[str, Any]` mapping each partition key to its deserialized object.

```
from dagster import (  # noqa: F811
    AssetIn,
    Definitions,
    MonthlyPartitionsDefinition,
    TimeWindowPartitionMapping,
    asset,
    io_manager,
)

from remote_store import Store  # noqa: F811
from remote_store.backends import LocalBackend
from remote_store.ext.dagster import dagster_io_manager  # noqa: F811

monthly = MonthlyPartitionsDefinition(start_date="2026-01-01")

@io_manager
def my_io_manager() -> IOManager:
    store = Store(LocalBackend(root="/data/dagster"))
    return dagster_io_manager(store, serializer="json")

@asset(partitions_def=monthly)
def sales_monthly() -> dict[str, int]:
    """Upstream asset — one partition per month."""
    return {"revenue": 100}

@asset(
    partitions_def=monthly,
    ins={
        "sales_monthly": AssetIn(
            partition_mapping=TimeWindowPartitionMapping(start_offset=-2),
        ),
    },
)
def sales_rolling_3m(sales_monthly: dict[str, Any]) -> dict[str, Any]:
    """Downstream — receives last 3 months as dict[str, Any].

    ``sales_monthly`` is ``{"2026-01": {...}, "2026-02": {...}, "2026-03": {...}}``
    when the current partition is ``"2026-03"``.
    """
    total = sum(v["revenue"] for v in sales_monthly.values())
    return {"rolling_revenue": total}

defs = Definitions(
    assets=[sales_monthly, sales_rolling_3m],
    resources={"io_manager": my_io_manager},
)
```

Single-partition inputs continue to return a single deserialized object (not wrapped in a dict). If any partition is missing, `load_input` raises `NotFound` immediately — no partial results are returned. This applies to both the [bytes-serializer IO manager](https://docs.remotestore.dev/stable/reference/api/extensions/dagster/index.md) and the [dataset IO manager](https://docs.remotestore.dev/stable/reference/api/extensions/dagster/#remote_store.ext.dagster.dagster_dataset_io_manager).

Each partition is loaded individually. For high partition counts over remote backends, consider pre-aggregating upstream or limiting the time-window span.

## Using with Registry

For teams using `Registry` for multi-backend configuration:

```
from remote_store import Registry

@io_manager
def production_io_manager() -> IOManager:
    registry = Registry(config)
    store = registry.get_store("production")
    return dagster_io_manager(store, serializer="pickle")
```

## Lifecycle

The caller owns the Store. The IO manager does not close it. If the Store was created inline, the caller is responsible for cleanup.

## Dagster-config-driven Store (v2)

Use v1 (`dagster_io_manager`) when you already have a Store. Use v2 (`RemoteStoreIOManager`) when Dagster should construct the Store from config — for example in `Definitions` files where no Store exists outside Dagster.

```
from dagster import Definitions, asset
from remote_store.ext.dagster import RemoteStoreIOManager


@asset
def raw_data() -> dict:
    return {"rows": [1, 2, 3]}


defs = Definitions(
    assets=[raw_data],
    resources={
        "io_manager": RemoteStoreIOManager(
            backend_type="local",
            backend_options={"root": "/data/dagster"},
            serializer="pickle",
        )
    },
)
```

`RemoteStoreIOManager` is a Dagster `ConfigurableIOManagerFactory` that constructs and owns the Store lifecycle — setup and teardown happen automatically when Dagster initialises and cleans up resources.

The `backend_type` field accepts `"local"`, `"s3"`, `"azure"`, `"sftp"`, `"memory"`, `"sql-blob"`, and any other backend registered with the remote-store factory. `backend_options` accepts the same keyword arguments as the corresponding backend constructor.

For direct Store access in assets (outside the IO manager), use `DagsterStoreResource` as a standalone resource.

### Dataset mode

For Parquet dataset I/O via `ParquetDatasetStore`, use `dagster_dataset_io_manager(store)` (v1-style) or pass `serializer="parquet-dataset"` on `RemoteStoreIOManager`:

```
from remote_store.ext.dagster import RemoteStoreIOManager

resources = {
    "io_manager": RemoteStoreIOManager(
        backend_type="s3",
        backend_options={"bucket": "my-bucket", "prefix": "dagster/"},
        serializer="parquet-dataset",
    )
}
```

Requires `pip install "remote-store[dagster,arrow]"`.

## Compute logs

`RemoteStoreComputeLogManager` is the second half of the Dagster integration. Where the IO manager persists asset and op *return values*, the compute log manager persists the raw `stdout` / `stderr` a step emits while it runs — the text the Dagster UI shows under a run's stdout/stderr tabs.

A [`ComputeLogManager`](https://docs.dagster.io/api/dagster/internals) is not a resource. It is a Dagster *instance* component, so it is configured in `dagster.yaml` rather than wired into `Definitions`:

```
compute_logs:
  module: remote_store.ext.dagster
  class: RemoteStoreComputeLogManager
  config:
    backend_type: s3
    backend_options:
      bucket: my-logs-bucket
    root_path: dagster/compute-logs
    upload_interval: 30
```

With this in place every run worker, the webserver, and the daemon stream compute logs to the same backend the rest of your Dagster storage already uses — no `dagster-aws` / `dagster-azure` needed. This is the standard fix for ephemeral run workers (Kubernetes, ECS): logs survive the pod or task being reclaimed.

### Config fields

| Field              | Type   | Default         | Purpose                                                                         |
| ------------------ | ------ | --------------- | ------------------------------------------------------------------------------- |
| `backend_type`     | string | *(required)*    | Registered backend type (`local`, `s3`, `sftp`, `azure`, `memory`, ...)         |
| `backend_options`  | dict   | `{}`            | Keyword arguments for the backend constructor                                   |
| `root_path`        | string | `""`            | Store root prefix for all log objects                                           |
| `local_dir`        | string | system temp dir | Local staging directory for capture                                             |
| `prefix`           | string | `"dagster"`     | Path prefix within the Store                                                    |
| `skip_empty_files` | bool   | `false`         | Skip uploading zero-byte log files                                              |
| `upload_interval`  | int    | `None`          | Seconds between partial uploads while a step runs; `None` disables live tailing |

Logs are captured to `local_dir` at the file-descriptor level — a descriptor cannot point at a remote object — then uploaded to the Store on step completion. Each stream is stored at `{prefix}/storage/<log-key>/<step>.out` (or `.err`); uploads are truncated at 50 MB, matching Dagster's own cloud compute log managers. Credential-named `backend_options` (`secret`, `password`, ...) are wrapped in `Secret` so they are masked in `repr()` and tracebacks.

The webserver streams log bytes through itself rather than minting a signed download URL, so the process running the Dagster UI must also be able to reach the backend.

## See also

- [ext.dagster API reference](https://docs.remotestore.dev/stable/reference/api/extensions/dagster/index.md) — full API docs
- [Dagster v2 resource example](https://docs.remotestore.dev/stable/tutorial/examples/dagster-v2-resource/index.md) — config-driven Store construction with `RemoteStoreIOManager`
- [Dagster compute log example](https://docs.remotestore.dev/stable/tutorial/examples/dagster-compute-log-manager/index.md) — capturing op `stdout` / `stderr` to a Store with `RemoteStoreComputeLogManager`
- [Medallion + Dagster Showcase](https://docs.remotestore.dev/stable/tutorial/examples/medallion-dagster/index.md) — end-to-end Bronze/Silver/Gold pipeline demonstrating extensions over live MeteoSwiss data
- [Data Lake Patterns](https://docs.remotestore.dev/stable/guides/data-lake-patterns/index.md) — medallion architecture with `Store.child()` and PyArrow, complementary to Dagster orchestration
- [PyArrow Adapter](https://docs.remotestore.dev/stable/guides/pyarrow-adapter/index.md) — use Store as a PyArrow filesystem for Parquet I/O

# Data Lake Patterns

How to use remote-store as the storage plumbing layer for a multi-layer Parquet data lake — and where to draw the line.

## The idea

remote-store provides a **unified, backend-agnostic I/O layer** that makes data lake patterns portable across storage backends. A team can develop locally against Memory or Local backends, test in CI without cloud credentials, and deploy to S3 or Azure without changing application code.

The value proposition is not competing with Databricks, Spark, or any query/compute engine. It is making **the storage layer underneath them swappable and testable**. remote-store owns the I/O path; table formats (Delta Lake, Iceberg) and query engines (Spark, DuckDB, Polars) sit on top via the PyArrow adapter.

| Layer                       | Components                                                          |
| --------------------------- | ------------------------------------------------------------------- |
| **Query / Compute**         | Polars, DuckDB, Pandas, Spark, Databricks                           |
| **Table Format** (optional) | Delta Lake, PyIceberg, plain Parquet                                |
| **PyArrow FileSystem**      | `pyarrow_fs(store)`                                                 |
| **remote-store**            | `Store`, `child()`, `ext.batch`, `ext.transfer`                     |
| **Backend**                 | Local, Memory, S3, S3-PyArrow, SFTP, Azure, HTTP, SQLBlob, SQLQuery |

## What this gets you

1. **Backend portability.** Develop against `MemoryBackend`, deploy to S3/Azure. Same code, zero cloud dependencies in dev.
1. **Layer isolation.** `store.child("bronze")`, `store.child("silver")`, `store.child("gold")` — each layer gets its own namespace without separate credentials, connections, or config.
1. **Streaming by default.** Reads and writes never fully materialize large files in memory. Critical for multi-GB Parquet partitions.
1. **Atomic writes.** `write_atomic()` uses rename-based semantics, reducing partial-write risk for individual files.
1. **Batch operations.** `batch_delete`, `batch_copy`, `batch_exists` for partition-scale workflows with error aggregation.
1. **Observability.** `ext.observe` hooks and OpenTelemetry bridge give visibility into every I/O operation without instrumenting application code.

## What this does not get you

Being honest about the boundary:

- **No table format protocol.** remote-store does not implement the Delta Lake transaction log, Iceberg manifest management, or any table format's commit protocol. Libraries like `deltalake` (delta-rs) or `pyiceberg` handle that — remote-store is the I/O layer they write through.
- **No query execution.** No SQL, no predicate pushdown, no shuffle. Polars, DuckDB, or Spark handle query planning and execution.
- **No catalog integration.** Unity Catalog, Hive Metastore, and Glue Catalog are metadata services remote-store does not interact with.
- **No Databricks-native credential passthrough.** Databricks clusters use instance profiles, workspace tokens, and Unity Catalog vended credentials. remote-store's `BackendConfig` takes explicit credentials. On Databricks, you would either pass credentials explicitly or use the native DBFS/ABFSS paths directly and skip the abstraction.
- **GIL-free fast path for native backends.** When the backend exposes a native PyArrow filesystem (e.g., `S3PyArrowBackend`), the adapter uses Tier 1 fast-path reads that bypass Python I/O entirely — full C++ range requests with I/O coalescing and zero GIL overhead. Backends without native PyArrow support fall back to Tier 2 (full-file materialization) or Tier 3 (PythonFile streaming).

The bridge metaphor works best with a clear boundary: **remote-store owns portable, testable, observable storage I/O. Everything above the storage layer — formats, queries, catalogs — belongs to purpose-built tools.**

## Bronze / Silver / Gold with `Store.child()`

The medallion architecture maps naturally to child stores:

```
from remote_store import BackendConfig, Registry, RegistryConfig, StoreProfile

config = RegistryConfig(
    backends={
        "lake": BackendConfig(
            type="s3",
            options={"bucket": "my-data-lake"},
        ),
    },
    stores={"lake": StoreProfile(backend="lake", root_path="v1")},
)

with Registry(config) as registry:
    lake = registry.get_store("lake")

    bronze = lake.child("bronze")   # raw ingestion
    silver = lake.child("silver")   # cleaned and typed
    gold = lake.child("gold")       # aggregated / business-ready
```

Each layer is a full `Store` — same API, same backend connection, independent namespace. In dev/test, swap the backend to `MemoryBackend` or `LocalBackend` with zero code changes.

## Writing Parquet via PyArrow

```
import pyarrow as pa
import pyarrow.parquet as pq
from remote_store.ext.arrow import pyarrow_fs

# Bronze: ingest raw data as Parquet
bronze_fs = pyarrow_fs(bronze)
raw = pa.table({
    "timestamp": ["2026-03-01T10:00:00", "2026-03-01T10:01:00"],
    "sensor_id": ["s-001", "s-002"],
    "reading": [23.4, 19.8],
})
pq.write_table(raw, "readings/2026-03-01.parquet", filesystem=bronze_fs)
```

## Reading and transforming with Polars

```
import polars as pl
import pyarrow.parquet as pq
from remote_store.ext.arrow import pyarrow_fs

# Read bronze partition via PyArrow, convert to Polars
bronze_fs = pyarrow_fs(bronze)
raw = pq.read_table("readings/2026-03-01.parquet", filesystem=bronze_fs)
df = pl.from_arrow(raw)

# Clean and type (bronze -> silver)
silver_df = (
    df.with_columns(
        pl.col("timestamp").str.to_datetime(),
        pl.col("reading").cast(pl.Float64),
    )
    .filter(pl.col("reading").is_not_null())
)

# Write to silver layer via PyArrow
silver_fs = pyarrow_fs(silver)
pq.write_table(silver_df.to_arrow(), "readings/2026-03-01.parquet", filesystem=silver_fs)
```

## Partitioned datasets with PyArrow

PyArrow's dataset API handles Hive-style partitioned layouts automatically:

```
import pyarrow.dataset as ds

# Write partitioned by date
part = pa.table({
    "date": ["2026-03-01", "2026-03-01", "2026-03-02"],
    "sensor": ["s-001", "s-002", "s-001"],
    "value": [23.4, 19.8, 21.1],
})
ds.write_dataset(
    part,
    "sensors",
    filesystem=pyarrow_fs(silver),
    format="parquet",
    partitioning=ds.partitioning(pa.schema([("date", pa.string())])),
)

# Read with partition pruning
dataset = ds.dataset(
    "sensors",
    filesystem=pyarrow_fs(silver),
    format="parquet",
    partitioning="hive",
)
march_1 = dataset.to_table(filter=ds.field("date") == "2026-03-01")
```

## DuckDB integration

```
import duckdb
import pyarrow.dataset as ds
from remote_store.ext.arrow import pyarrow_fs

silver_fs = pyarrow_fs(silver)
dataset = ds.dataset("readings", filesystem=silver_fs, format="parquet")

# DuckDB queries PyArrow datasets directly
result = duckdb.sql("""
    SELECT sensor_id, AVG(reading) as avg_reading
    FROM dataset
    GROUP BY sensor_id
""")
print(result.fetchdf())
```

## Delta Lake (via deltalake)

The `deltalake` library accepts PyArrow filesystems for storage:

```
from deltalake import DeltaTable, write_deltalake

gold_fs = pyarrow_fs(gold)

# Write a Delta table
write_deltalake(
    "aggregates",
    silver_df.to_arrow(),
    filesystem=gold_fs,
)

# Read it back
dt = DeltaTable("aggregates", filesystem=gold_fs)
df = pl.from_arrow(dt.to_pyarrow_table())
```

Note: `deltalake` (delta-rs) owns the transaction log and commit protocol. remote-store provides the I/O — it does not replace Delta's ACID guarantees.

## Batch operations across partitions

```
from remote_store import batch_delete, batch_copy

# Archive old bronze partitions within the same store
old_files = [
    f.path for f in bronze.list_files("readings", recursive=True)
    if f.path < "readings/2026-01-01"
]
batch_copy(bronze, [(f, f"archive/{f}") for f in old_files])
batch_delete(bronze, old_files)
```

For partition discovery without scanning the full tree, use `max_depth` to limit traversal:

```
# List only top-level partition folders (e.g. date directories)
for folder in bronze.list_folders("readings", max_depth=0):
    print(folder.name)  # 2026-01-01, 2026-02-01, ...

# Files in readings/ and its immediate partition folders only
for f in bronze.list_files("readings", max_depth=1):
    print(f.path)
```

## Cross-backend transfer

Move data between backends without intermediate files:

```
from remote_store import transfer

# Stream a partition from S3 bronze to Azure silver
transfer(
    s3_bronze, "readings/2026-03-01.parquet",
    azure_silver, "readings/2026-03-01.parquet",
    on_progress=lambda n: print(f"{n} bytes transferred"),
)
```

## Testing without cloud credentials

The entire pipeline can run against `MemoryBackend` in unit tests:

```
import pytest
from remote_store import Store
from remote_store.backends import MemoryBackend
from remote_store.ext.arrow import pyarrow_fs


@pytest.fixture
def lake():
    """In-memory data lake for testing."""
    store = Store(backend=MemoryBackend())
    return {
        "bronze": store.child("bronze"),
        "silver": store.child("silver"),
        "gold": store.child("gold"),
    }


def test_bronze_to_silver(lake):
    import pyarrow as pa
    import pyarrow.parquet as pq

    # Write raw data to bronze
    bronze_fs = pyarrow_fs(lake["bronze"])
    table = pa.table({"id": [1, 2], "value": [10.0, 20.0]})
    pq.write_table(table, "batch.parquet", filesystem=bronze_fs)

    # Transform and write to silver
    silver_fs = pyarrow_fs(lake["silver"])
    raw = pq.read_table("batch.parquet", filesystem=bronze_fs)
    # ... apply transformations ...
    pq.write_table(raw, "batch.parquet", filesystem=silver_fs)

    # Verify silver has the data
    result = pq.read_table("batch.parquet", filesystem=silver_fs)
    assert result.num_rows == 2
```

## Features that complement this pattern

These extensions work well with data lake workflows:

| Feature                                                                                                            | What it does                                                                                |
| ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- |
| [Parallel batch operations](https://docs.remotestore.dev/stable/guides/batch-operations/index.md)                  | Concurrent I/O for partition-scale deletes and copies (`concurrent=True`)                   |
| [Hive partition helpers](https://docs.remotestore.dev/stable/reference/api/extensions/partition/index.md)          | Build and parse `year=2026/month=03/` paths with `partition_path()` and `parse_partition()` |
| [PyArrow Tier 1 fast-path](https://docs.remotestore.dev/stable/guides/pyarrow-adapter/index.md)                    | Zero-GIL C++ range requests for large Parquet workloads (S3-PyArrow)                        |
| [Streaming atomic writes](https://docs.remotestore.dev/stable/reference/api/store/#remote_store.Store.open_atomic) | `open_atomic()` context manager for multi-GB Parquet exports                                |
| [Caching middleware](https://docs.remotestore.dev/stable/guides/cache/index.md)                                    | Reduces round-trips for metadata-heavy listing workflows                                    |
| [Parquet Dataset Store](https://docs.remotestore.dev/stable/guides/parquet-datasets/index.md)                      | Managed Parquet datasets with manifests, `_SUCCESS` markers, and multi-part layouts         |

## See also

- [Medallion + Dagster Showcase](https://docs.remotestore.dev/stable/tutorial/examples/medallion-dagster/index.md) — end-to-end Bronze/Silver/Gold pipeline demonstrating extensions over live MeteoSwiss data
- [Data Lake Medallion notebook](https://github.com/haalfi/remote-store/blob/master/examples/notebooks/04_data_lake_medallion.ipynb) — runnable end-to-end Bronze/Silver/Gold pipeline
- [Dagster Integration](https://docs.remotestore.dev/stable/guides/dagster/index.md) — use any Store as a Dagster IO manager for orchestrated pipelines
- [PyArrow FileSystem Adapter](https://docs.remotestore.dev/stable/guides/pyarrow-adapter/index.md) — adapter configuration and tiered read strategy
- [Transfer Operations](https://docs.remotestore.dev/stable/guides/transfer-operations/index.md) — cross-store and local-path transfers
- [Batch Operations](https://docs.remotestore.dev/stable/guides/batch-operations/index.md) — bulk delete, copy, exists
- [Concurrency](https://docs.remotestore.dev/stable/explanation/concurrency/index.md) — TOCTOU and atomicity considerations

# Extensions

The `remote_store.ext` package provides higher-level operations built on top of the core Store API.

## Available Extensions

| Module                                                                                             | Extra      | Description                                                     | Guide                                                                              | Example                                                                                         |
| -------------------------------------------------------------------------------------------------- | ---------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| [`ext.batch`](https://docs.remotestore.dev/stable/reference/api/extensions/batch/index.md)         | *(none)*   | Bulk delete, copy, and exists operations                        | [Guide](https://docs.remotestore.dev/stable/guides/batch-operations/index.md)      | [Example](https://docs.remotestore.dev/stable/tutorial/examples/batch-operations/index.md)      |
| [`ext.cache`](https://docs.remotestore.dev/stable/reference/api/extensions/cache/index.md)         | *(none)*   | Store-level caching with TTL and auto-invalidation              | [Guide](https://docs.remotestore.dev/stable/guides/cache/index.md)                 | [Example](https://docs.remotestore.dev/stable/tutorial/examples/caching/index.md)               |
| [`ext.glob`](https://docs.remotestore.dev/stable/reference/api/extensions/glob/index.md)           | *(none)*   | Portable glob pattern matching for file listing                 | [Guide](https://docs.remotestore.dev/stable/guides/glob-pattern-matching/index.md) | [Example](https://docs.remotestore.dev/stable/tutorial/examples/glob-pattern-matching/index.md) |
| [`ext.integrity`](https://docs.remotestore.dev/stable/reference/api/extensions/integrity/index.md) | *(none)*   | Checksum computation and verification helpers                   | —                                                                                  | —                                                                                               |
| [`ext.observe`](https://docs.remotestore.dev/stable/reference/api/extensions/observe/index.md)     | *(none)*   | Callback-based observability hooks for Store operations         | [Guide](https://docs.remotestore.dev/stable/guides/observe/index.md)               | [Example](https://docs.remotestore.dev/stable/tutorial/examples/observe-hooks/index.md)         |
| [`ext.partition`](https://docs.remotestore.dev/stable/reference/api/extensions/partition/index.md) | *(none)*   | Hive-style partition path helpers                               | —                                                                                  | —                                                                                               |
| [`ext.streams`](https://docs.remotestore.dev/stable/reference/api/extensions/streams/index.md)     | *(none)*   | Composable BinaryIO wrappers for progress and checksums         | —                                                                                  | —                                                                                               |
| [`ext.transfer`](https://docs.remotestore.dev/stable/reference/api/extensions/transfer/index.md)   | *(none)*   | Upload, download, and cross-store transfer                      | [Guide](https://docs.remotestore.dev/stable/guides/transfer-operations/index.md)   | [Example](https://docs.remotestore.dev/stable/tutorial/examples/transfer-operations/index.md)   |
| [`ext.write`](https://docs.remotestore.dev/stable/reference/api/extensions/write/index.md)         | *(none)*   | Write helpers with guaranteed client-side content hashing       | [Guide](https://docs.remotestore.dev/stable/guides/write-integrity/index.md)       | —                                                                                               |
| [`aio.ext.write`](https://docs.remotestore.dev/stable/reference/api/aio/extensions/write/index.md) | *(none)*   | Async write helpers with guaranteed client-side content hashing | [Guide](https://docs.remotestore.dev/stable/guides/write-integrity/index.md)       | —                                                                                               |
| [`ext.arrow`](https://docs.remotestore.dev/stable/reference/api/extensions/arrow/index.md)         | `arrow`    | PyArrow FileSystem adapter                                      | [Guide](https://docs.remotestore.dev/stable/guides/pyarrow-adapter/index.md)       | [Example](https://docs.remotestore.dev/stable/tutorial/examples/pyarrow-adapter/index.md)       |
| [`ext.parquet`](https://docs.remotestore.dev/stable/reference/api/extensions/parquet/index.md)     | `arrow`    | Managed Parquet datasets with manifests and completion markers  | [Guide](https://docs.remotestore.dev/stable/guides/parquet-datasets/index.md)      | [Example](https://docs.remotestore.dev/stable/tutorial/examples/parquet-dataset/index.md)       |
| [`ext.otel`](https://docs.remotestore.dev/stable/reference/api/extensions/otel/index.md)           | `otel`     | OpenTelemetry tracing and metrics bridge                        | [Guide](https://docs.remotestore.dev/stable/guides/observe/index.md)               | [Example](https://docs.remotestore.dev/stable/tutorial/examples/otel-tracing/index.md)          |
| [`ext.pydantic`](https://docs.remotestore.dev/stable/reference/api/extensions/pydantic/index.md)   | `pydantic` | Pydantic BaseModel/BaseSettings adapter                         | —                                                                                  | [Example](https://docs.remotestore.dev/stable/tutorial/examples/config-loaders/index.md)        |
| [`ext.yaml`](https://docs.remotestore.dev/stable/reference/api/extensions/yaml/index.md)           | `yaml`     | YAML config file loader                                         | —                                                                                  | [Example](https://docs.remotestore.dev/stable/tutorial/examples/config-loaders/index.md)        |
| [`ext.dagster`](https://docs.remotestore.dev/stable/reference/api/extensions/dagster/index.md)     | `dagster`  | Dagster IO Manager adapter                                      | [Guide](https://docs.remotestore.dev/stable/guides/dagster/index.md)               | [Example](https://docs.remotestore.dev/stable/tutorial/examples/dagster-io-manager/index.md)    |

## Using Extensions

### Always-available extensions (pure Python)

Extensions with no extra dependencies (marked `*(none)*` in the table above) are re-exported from the top-level package for convenience:

```
from remote_store import batch_delete, glob_files, observe, upload, download
from remote_store import cache                   # ext.cache
from remote_store import checksum, verify       # ext.integrity
from remote_store import partition_path, parse_partition  # ext.partition
from remote_store import ProgressReader, ChecksumReader   # ext.streams
from remote_store import write_with_hash, open_atomic_with_hash, HashingAtomicWriter  # ext.write
```

Or import from the extension module directly:

```
from remote_store.ext.batch import batch_delete
from remote_store.ext.cache import cache
from remote_store.ext.glob import glob_files
from remote_store.ext.integrity import checksum, verify
from remote_store.ext.observe import observe
from remote_store.ext.partition import partition_path, parse_partition
from remote_store.ext.streams import ProgressReader, ChecksumReader
from remote_store.ext.transfer import upload
from remote_store.ext.write import write_with_hash, open_atomic_with_hash, HashingAtomicWriter
```

`aio.ext.write` has no extra dependency but is **not** re-exported from `remote_store` — import it from its module directly:

```
from remote_store.aio.ext.write import write_with_hash  # async counterpart to ext.write
```

Seekable reads are built into the core API via `Store.read_seekable()` — no extension import needed.

### Optional-dependency extensions

`ext.arrow` and `ext.parquet` require PyArrow, `ext.otel` requires the OpenTelemetry API, `ext.pydantic` requires Pydantic v2, and `ext.dagster` requires Dagster. Install the relevant extra first:

```
pip install "remote-store[arrow]"     # PyArrow filesystem adapter + parquet datasets
pip install "remote-store[otel]"      # OpenTelemetry tracing and metrics
pip install "remote-store[pydantic]"  # Pydantic BaseSettings adapter
pip install "remote-store[yaml]"      # YAML config file loader
pip install "remote-store[dagster]"   # Dagster IO Manager adapter
```

Then import from the extension module directly:

```
from remote_store.ext.arrow import pyarrow_fs
from remote_store.ext.parquet import ParquetDatasetStore
from remote_store.ext.otel import otel_hooks
from remote_store.ext.pydantic import from_pydantic
from remote_store.ext.yaml import from_yaml
from remote_store.ext.dagster import dagster_io_manager
```

If the required dependency is not installed, importing the extension module raises a `ModuleNotFoundError` with installation instructions.

## Extension Guarantees

All extensions follow the same contract — see the [extension architecture](https://docs.remotestore.dev/stable/explanation/design/adrs/0008-extension-architecture/index.md) and [optional-extension export rules](https://docs.remotestore.dev/stable/explanation/design/adrs/0013-drop-optional-extension-reexports/index.md) ADRs for the rationale:

- **Public API only** — extensions use only the public Store / Backend API. They never access private internals.
- **No lifecycle ownership** — extensions never close the Store. The caller owns the Store's lifecycle.
- **CapabilityNotSupported propagates** — if a backend lacks a required capability, the error reaches the caller immediately.

## Writing Your Own Extension

See the "Adding an Extension" checklist in CONTRIBUTING.md.

## See also

- [API reference](https://docs.remotestore.dev/stable/reference/api/store/index.md) — core Store interface that extensions build on
- [Architecture](https://docs.remotestore.dev/stable/explanation/architecture/index.md) — extension design principles

# Glob Pattern Matching

The `ext.glob` module provides portable pattern matching for file listing across all backends. For simple name-based filtering, use `Store.list_files(pattern=...)` directly — it works with every backend.

Three [tiers of pattern matching](https://docs.remotestore.dev/stable/explanation/design/adrs/0009-glob-three-tier-design/index.md) are available:

1. **`list_files(pattern=...)`** — `fnmatch` name filtering at the Store level
1. **`Store.glob(pattern)`** — native backend glob, capability-gated
1. **`ext.glob.glob_files()`** — portable full glob with recursive patterns

No extra dependencies are required — the module is pure Python and always available.

## Quick Start

```
from remote_store import Store, glob_files
from remote_store.backends import MemoryBackend

store = Store(backend=MemoryBackend())
store.write("data/report.csv", b"r1")
store.write("data/summary.csv", b"r2")
store.write("data/readme.txt", b"r3")
store.write("logs/app.log", b"l1")

# Tier 1: simple name filtering (works with every backend)
csvs = list(store.list_files("data", pattern="*.csv"))
# [FileInfo("data/report.csv", ...), FileInfo("data/summary.csv", ...)]

# Tier 3: full recursive glob (works with every backend)
all_csvs = list(glob_files(store, "**/*.csv"))
# [FileInfo("data/report.csv", ...), FileInfo("data/summary.csv", ...)]
```

## Tier 1: list_files(pattern=...)

The simplest and most common option. The `pattern` parameter on `list_files` filters results by **file name** using `fnmatch`:

```
# All CSVs in the data folder
store.list_files("data", pattern="*.csv")

# All files starting with "report"
store.list_files("", pattern="report.*")

# Single-character wildcard
store.list_files("", pattern="file?.txt")  # file1.txt, fileA.txt

# Character class
store.list_files("", pattern="*.[ct]sv")  # matches .csv and .tsv
```

This works with every backend (needs only `LIST` capability). The filtering is applied at the Store level after path rebasing.

**Composing with `max_depth`:** `pattern` and `max_depth` compose naturally --- depth filtering applies first (which files to consider), then pattern filtering (which names to keep):

```
# CSVs in data/ and its immediate subfolders only
store.list_files("data", max_depth=1, pattern="*.csv")
```

**Limitation:** `pattern` matches against the file's **name** (basename), not the full path. For path-based patterns like `data/**/*.csv`, use `glob_files()` (Tier 3).

## Tier 2: Store.glob() — native backend glob

For backends with native pattern matching, `Store.glob()` provides direct access. It is capability-gated on `Capability.GLOB` — like `Store.unwrap()`, it is an opt-in feature for users who know their backend:

```
from remote_store import Capability

if store.supports(Capability.GLOB):
    for info in store.glob("**/*.csv"):
        print(info.path)
```

`LocalBackend`, `S3Backend`, `S3PyArrowBackend`, and `AzureBackend` implement native glob. `LocalBackend` uses `pathlib.Path.glob()`; the cloud backends use prefix-optimized listing with client-side regex filtering.

## Tier 3: glob_files() — portable full glob

`glob_files()` is the recommended API when `list_files(pattern=)` isn't enough and you want code that works across all backends:

```
from remote_store.ext.glob import glob_files

# Recursive: find all logs at any depth
for info in glob_files(store, "**/*.log"):
    print(info.path, info.size)

# Subdirectory wildcard
for info in glob_files(store, "data/2024/*.csv"):
    print(info.path)

# Match everything
for info in glob_files(store, "**/*"):
    print(info.path)
```

When the backend supports `Capability.GLOB`, `glob_files` delegates to the native `store.glob()`. Otherwise it extracts the longest non-wildcard prefix, calls `store.list_files()` with that prefix, and filters client-side using a regex compiled from the pattern.

## Pattern Syntax

| Pattern  | Matches                                |
| -------- | -------------------------------------- |
| `*`      | Any characters except `/`              |
| `**`     | Zero or more path segments (recursive) |
| `?`      | Single non-separator character         |
| `[abc]`  | Character class                        |
| `[!abc]` | Negated character class                |

`**` must be a complete path segment (`**/`, `/**`, or the entire pattern). Patterns like `**error` where `**` is embedded within a segment raise `ValueError`.

## Works with Store.child()

All glob operations work correctly through `Store.child()` path scoping:

```
store = Store(backend=MemoryBackend())
store.write("reports/q1.csv", b"data")
store.write("reports/q2.csv", b"data")
store.write("reports/archive/old.csv", b"data")

reports = store.child("reports")

# Tier 1: name filter within child scope
list(reports.list_files("", pattern="*.csv"))
# [FileInfo("q1.csv"), FileInfo("q2.csv")]

# Tier 3: recursive glob within child scope
list(glob_files(reports, "**/*.csv"))
# [FileInfo("q1.csv"), FileInfo("q2.csv"), FileInfo("archive/old.csv")]
```

## Choosing the Right Tier

| Need                                               | Use                                      |
| -------------------------------------------------- | ---------------------------------------- |
| Filter files in one folder by name                 | `list_files(pattern="*.csv")`            |
| Recursive search across directories                | `glob_files(store, "**/*.csv")`          |
| Native backend glob (Local, S3, S3-PyArrow, Azure) | `store.glob("**/*.csv")`                 |
| Works with every backend                           | `list_files(pattern=)` or `glob_files()` |

## See also

- [API reference](https://docs.remotestore.dev/stable/reference/api/extensions/glob/index.md)
- [Example script](https://github.com/haalfi/remote-store/blob/master/examples/extensions/glob_pattern_matching.py)

# Health Check

Verify backend connectivity and credentials before your application starts processing requests.

## Quick start

```
from remote_store import Store
from remote_store.backends import LocalBackend

store = Store(LocalBackend(root="/data/inbox"))
store.ping()  # raises on failure, silent on success
```

## Use cases

- **Startup gates** — fail fast before accepting traffic if the backend is unreachable or credentials are invalid.
- **Liveness probes** — Kubernetes `livenessProbe` or similar health endpoints.
- **Connection validation** — verify config after loading from TOML/YAML.

## Error handling

`ping()` raises the same exceptions as other Store operations:

| Exception            | Meaning                                             |
| -------------------- | --------------------------------------------------- |
| `PermissionDenied`   | Invalid credentials or insufficient permissions     |
| `NotFound`           | Bucket, container, or root directory does not exist |
| `BackendUnavailable` | Network error, DNS failure, or timeout              |

```
from remote_store import BackendUnavailable, NotFound, PermissionDenied

try:
    store.ping()
except PermissionDenied:
    log.error("Bad credentials for %s", store)
except NotFound:
    log.error("Missing bucket/container for %s", store)
except BackendUnavailable:
    log.error("Backend unreachable for %s", store)
```

## Per-backend strategies

Each backend uses the cheapest possible read-only operation:

| Backend                                                                               | Operation                                  | What it validates                   |
| ------------------------------------------------------------------------------------- | ------------------------------------------ | ----------------------------------- |
| [Local](https://docs.remotestore.dev/stable/guides/backends/local/index.md)           | `root.exists()` + `os.access(R_OK)`        | Directory exists and is readable    |
| [S3](https://docs.remotestore.dev/stable/guides/backends/s3/index.md)                 | `head_bucket`                              | Bucket exists, credentials valid    |
| [S3-PyArrow](https://docs.remotestore.dev/stable/guides/backends/s3-pyarrow/index.md) | `get_file_info(bucket)`                    | Bucket accessible via PyArrow       |
| [SFTP](https://docs.remotestore.dev/stable/guides/backends/sftp/index.md)             | `stat(base_path)`                          | SSH connection, path exists         |
| [Azure](https://docs.remotestore.dev/stable/guides/backends/azure/index.md)           | `get_container_properties()`               | Container exists, credentials valid |
| [Memory](https://docs.remotestore.dev/stable/guides/backends/memory/index.md)         | No-op                                      | Always healthy                      |
| [HTTP](https://docs.remotestore.dev/stable/guides/backends/http/index.md)             | `HEAD` to `base_url` (falls back to `GET`) | Server reachable                    |
| [SQLBlob](https://docs.remotestore.dev/stable/guides/backends/sql-blob/index.md)      | `SELECT 1`                                 | Database connection valid           |
| [SQLQuery](https://docs.remotestore.dev/stable/guides/backends/sql-query/index.md)    | `SELECT 1`                                 | Database connection valid           |

## Observability

`ping()` integrates with `ext.observe`:

```
from remote_store.ext.observe import observe

def on_ping(event):
    print(f"Ping took {event.duration_ms:.1f}ms")

observed = observe(store, on_ping=on_ping)
observed.ping()
```

## Design notes

- **No return value** — success is silent (`None`), failure raises. This matches the Go convention of `Ping() error` and keeps the API minimal.
- **No caching** — every call performs a real connectivity check.
- **No timeout parameter** — use backend-level timeouts (e.g. `RetryPolicy`).
- **Not capability-gated** — all backends support health checks.

## See also

- [API reference](https://docs.remotestore.dev/stable/reference/api/store/index.md) — `Store.ping()` method
- [Backends guide](https://docs.remotestore.dev/stable/guides/backends/index.md) — per-backend configuration and timeouts

# Observability Hooks

The `ext.observe` extension wraps a Store in a proxy that fires user-defined callbacks after each operation. This enables logging, metrics collection, auditing, and tracing without modifying business code.

## Quick Start

```
from remote_store import observe

def on_write(event):
    print(f"Wrote {event.path} in {event.duration_ms:.1f}ms")

observed = observe(store, on_write=on_write)
observed.write("data/report.csv", csv_bytes)
# prints: Wrote data/report.csv in 12.3ms
```

The returned `ObservedStore` is a full `Store` subclass — it passes `isinstance(observed, Store)` and works everywhere a Store is expected.

## Hook Types

### Per-operation hooks (after-only)

Each hook fires **after** the operation completes (success or failure):

| Hook        | Operations                                                                                                                  |
| ----------- | --------------------------------------------------------------------------------------------------------------------------- |
| `on_read`   | `read`, `read_bytes`                                                                                                        |
| `on_write`  | `write`, `write_atomic`, `open_atomic`                                                                                      |
| `on_delete` | `delete`, `delete_folder`                                                                                                   |
| `on_copy`   | `copy`                                                                                                                      |
| `on_move`   | `move`                                                                                                                      |
| `on_list`   | `list_files`, `list_folders`, `iter_children`, `glob`, `get_file_info`, `get_folder_info`, `exists`, `is_file`, `is_folder` |
| `on_ping`   | `ping`                                                                                                                      |
| `on_error`  | Any operation that raises an exception                                                                                      |
| `on_any`    | Every operation (catch-all)                                                                                                 |

### Around hook (context manager)

The `around` parameter accepts a factory that returns a context manager wrapping the entire operation:

```
import contextlib

@contextlib.contextmanager
def trace_span(op, path, backend):
    span = tracer.start_span(f"store.{op}")
    span.set_attribute("path", path)
    try:
        yield
    finally:
        span.end()

observed = observe(store, around=trace_span)
```

## StoreEvent

Every hook receives a `StoreEvent` frozen dataclass:

```
@dataclasses.dataclass(frozen=True)
class StoreEvent:
    operation: str           # "read", "write", "delete", ...
    path: str                # store-relative key
    backend: str             # backend name
    started_at: float        # time.monotonic()
    duration_ms: float       # elapsed milliseconds
    error: Exception | None  # None on success
    metadata: dict[str, Any] # op-specific: overwrite, dst, recursive, ...
    correlation_id: str | None  # from set_correlation_id(), None if not set
```

### Correlation IDs

Use `set_correlation_id()` to tag all events within a scope with a shared identifier (e.g., a request ID):

```
from remote_store import observe, set_correlation_id

observed = observe(store, on_any=lambda e: print(e.correlation_id))

token = set_correlation_id("req-abc-123")
observed.read_bytes("data.csv")   # correlation_id="req-abc-123"
observed.write("out.csv", b"...")  # correlation_id="req-abc-123"
set_correlation_id(None)  # or: from contextvars import Token; _correlation_id.reset(token)
```

The correlation ID is stored in a `contextvars.ContextVar`, so it works correctly with threading and async tasks.

## BufferedObserver

For high-throughput scenarios, `BufferedObserver` collects events in a thread-safe queue and flushes them in batches:

```
from remote_store import BufferedObserver, observe

def send_to_analytics(events):
    for event in events:
        analytics.track("store_op", {
            "op": event.operation,
            "path": event.path,
            "duration_ms": event.duration_ms,
        })

observer = BufferedObserver(send_to_analytics, flush_interval=10.0)
observed = observe(store, on_any=observer.on_event)

# ... use observed store ...

observer.close()  # final flush + stop background thread
```

Parameters:

- `max_queue` (default 1000): events are dropped when the queue is full.
- `flush_interval` (default 5.0): seconds between automatic flushes.

## Intrinsic Logging

The library ships with built-in stdlib logging. Every module uses `logging.getLogger(__name__)`, so all loggers live under the `remote_store` namespace (e.g. `remote_store._store`, `remote_store.backends._local`, `remote_store.ext.observe`). A `NullHandler` is attached to the root `"remote_store"` logger so the library is silent by default.

### Showing only warnings

Set the level on the `"remote_store"` logger — this applies to all child loggers in the hierarchy:

```
import logging

logging.basicConfig()  # ensure at least one handler on root
logging.getLogger("remote_store").setLevel(logging.WARNING)
```

### Verbose debug output

```
import logging

logging.basicConfig(level=logging.WARNING)  # keep other libs quiet
logging.getLogger("remote_store").setLevel(logging.DEBUG)

# DEBUG remote_store._store: read path='data/file.csv'
# INFO  remote_store._store: write complete path='output.csv'
```

### Logging levels used

| Level     | When                                                      |
| --------- | --------------------------------------------------------- |
| `DEBUG`   | Method entry (every Store operation)                      |
| `INFO`    | Mutating-operation completion (write, delete, move, copy) |
| `WARNING` | Suppressed hook exceptions, fallback behaviour            |
| `ERROR`   | Before re-raising backend errors                          |

### Structured `extra` fields

Log records include structured `extra` fields accessible via custom formatters or structlog processors:

| Field     | Description                               |
| --------- | ----------------------------------------- |
| `op`      | Operation name (`"read"`, `"write"`, ...) |
| `path`    | Store-relative key                        |
| `backend` | Backend name (`"local"`, `"s3"`, ...)     |

## Error Handling

- **Operation errors always propagate.** The proxy catches exceptions only to build the `StoreEvent` (with `error` set) and fire hooks, then re-raises.
- **Hook exceptions are suppressed.** A failing hook never breaks the observed operation. Hook errors are logged at WARNING level.

## Composing with Other Extensions

`ext.observe` wraps at the Store level, so it composes naturally with other extensions:

```
from remote_store import observe, batch_delete

observed = observe(store, on_any=print_event)

# batch_delete calls observed.delete() for each path --
# each individual delete fires hooks
batch_delete(observed, ["a.txt", "b.txt", "c.txt"])
```

The `ext.transfer` `on_progress` callback remains separate (different concern: UI progress vs telemetry).

## Layer 3: OpenTelemetry Bridge

The `ext.otel` module provides pre-built hooks that emit [OpenTelemetry](https://opentelemetry.io/) spans and metrics. Install the optional dependency:

```
pip install "remote-store[otel]"
```

This depends only on `opentelemetry-api` (not the SDK). If no SDK is configured at runtime, all OTel calls become zero-cost no-ops.

### Quick start

```
from remote_store.ext.otel import otel_observe

observed = otel_observe(store)
observed.write("data/report.csv", csv_bytes)
# -> OTel span "store.write" + metrics recorded automatically
```

Or compose with other hooks using `otel_hooks()`:

```
from remote_store import observe
from remote_store.ext.otel import otel_hooks

observed = observe(
    store,
    **otel_hooks(),
    on_error=my_error_alerter,
)
```

### Spans

Each operation creates a span with:

- **Name:** `store.{operation}` (e.g. `store.read`, `store.write`)
- **Kind:** `SpanKind.CLIENT` (outbound call to storage)
- **Attributes:** `remote_store.operation`, `remote_store.backend`, `remote_store.path`
- **On error:** Status set to ERROR, exception recorded, `error.type` attribute added

### Metrics

Three instruments are created:

| Type      | Name                              | Unit | Attributes                                         |
| --------- | --------------------------------- | ---- | -------------------------------------------------- |
| Counter   | `remote_store.operations`         | `1`  | `operation`, `backend`, `status`                   |
| Counter   | `remote_store.errors`             | `1`  | `operation`, `backend`, `error.type`               |
| Histogram | `remote_store.operation.duration` | `s`  | `operation`, `backend`; plus `error.type` on error |

Note: `path` is intentionally excluded from metric attributes to avoid high-cardinality issues with metric backends like Prometheus.

### Configuring the SDK

The bridge works with any OpenTelemetry SDK configuration. A typical production setup exports to an OTLP-compatible collector:

```
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter())
)

# Now otel_observe() spans flow to your collector
observed = otel_observe(store)
```

## See also

- [API reference](https://docs.remotestore.dev/stable/reference/api/extensions/observe/index.md)
- [Example script](https://github.com/haalfi/remote-store/blob/master/examples/extensions/observe_hooks.py)

# Parquet Datasets

Write and read managed Parquet datasets with manifests, completion markers, and multi-part file layouts using `ext.parquet.ParquetDatasetStore`.

## When to use this

Use `ParquetDatasetStore` when you need:

- **Reliable write completion** — `_SUCCESS` markers prevent consumers from reading partially-written datasets.
- **Manifest metadata** — row counts, schema hashes, run IDs, and custom metadata travel with the data.
- **Multi-part layouts** — split large tables into part files with `max_rows_per_file` to control per-file memory.
- **Consistent conventions** — every dataset follows the same file layout, eliminating ad-hoc manifest schemas.

For raw PyArrow filesystem access without manifests, use [`ext.arrow`](https://docs.remotestore.dev/stable/guides/pyarrow-adapter/index.md) directly.

## Background

The pattern behind `ParquetDatasetStore` — multi-file Parquet datasets with a manifest and a completion marker — evolved from big-data execution frameworks, not from a single formal specification:

- **Apache Hadoop / MapReduce** pioneered the `_SUCCESS` marker file convention. A job's output directory is considered complete only when the empty `_SUCCESS` file is present, preventing downstream consumers from reading partial results. ([Hadoop FileOutputCommitter](https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.html))
- **Apache Spark** adopted and popularized the `part-NNNNN.parquet` naming convention for partitioned writes. Spark's `DataFrameWriter` splits output into numbered part files and writes a `_SUCCESS` marker on commit. ([Spark SQL Guide — Data Sources](https://spark.apache.org/docs/latest/sql-data-sources.html))
- **Databricks DBIO** (Delta Lake's predecessor) extended the pattern with manifest files that describe dataset contents, enabling incremental reads and schema evolution without listing the directory. ([Delta Lake Protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md))
- **Apache Hive** uses the same `_SUCCESS` / part-file layout for its managed tables, with metastore-level schemas serving a similar role to manifests.

`ParquetDatasetStore` distills the common subset of these conventions into a lightweight library-level abstraction: part files + JSON manifest + `_SUCCESS` marker, without requiring a metastore, transaction log, or full table format. Teams that outgrow this pattern can graduate to [Delta Lake](https://delta.io/) or [Apache Iceberg](https://iceberg.apache.org/) via the existing [`ext.arrow`](https://docs.remotestore.dev/stable/guides/pyarrow-adapter/index.md) PyArrow filesystem adapter.

## Installation

`ParquetDatasetStore` requires PyArrow:

```
pip install "remote-store[arrow]"
```

## Quick start

```
import pyarrow as pa
from remote_store import Store
from remote_store.backends import LocalBackend
from remote_store.ext.parquet import ParquetDatasetStore

store = Store(LocalBackend("/data/warehouse"))
pds = ParquetDatasetStore(store)

# Write
table = pa.table({"id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"]})
manifest = pds.write_dataset(table, "silver/customers")

# Read
customers = pds.read_dataset("silver/customers")

# Read with column projection
ids_only = pds.read_dataset("silver/customers", columns=["id"])
```

## Multi-part writes

For large tables, split into multiple Parquet files:

```
pds = ParquetDatasetStore(store, max_rows_per_file=100_000)
manifest = pds.write_dataset(large_table, "silver/events")
print(f"Written as {len(manifest.parts)} parts")
```

This produces:

```
silver/events/
├── part-00000.parquet
├── part-00001.parquet
├── ...
├── manifest.json
└── _SUCCESS
```

## Overwrite semantics

By default, writing to an existing dataset raises `AlreadyExists`:

```
from remote_store import AlreadyExists

pds.write_dataset(table, "silver/customers")
# Raises AlreadyExists:
pds.write_dataset(table, "silver/customers")
# Replaces the dataset:
pds.write_dataset(updated_table, "silver/customers", overwrite=True)
```

## Manifest inspection

Read the manifest without loading the full dataset:

```
manifest = pds.read_manifest("silver/customers")
print(manifest.row_count)    # 3
print(manifest.schema_hash)  # deterministic hash of the Arrow schema
print(manifest.compression)  # "zstd"
print(manifest.run_id)       # pipeline run ID, if provided
```

## Pipeline integration

Embed pipeline metadata via `run_id` and `metadata`:

```
manifest = pds.write_dataset(
    table,
    "silver/orders",
    run_id="dagster-abc123",
    metadata={"source": "meteoswiss", "version": "3"},
)
```

## Scoping with `Store.child()`

Combine with `Store.child()` for layer-scoped stores:

```
silver = Store(LocalBackend("/data/warehouse")).child("silver")
pds = ParquetDatasetStore(silver)
pds.write_dataset(table, "customers")  # writes to /data/warehouse/silver/customers/
```

## Error handling

```
from remote_store.ext.parquet import DatasetIncomplete, ManifestCorrupted

try:
    table = pds.read_dataset("silver/orders")
except DatasetIncomplete:
    print("Dataset write did not complete -- missing _SUCCESS or parts")
except ManifestCorrupted as e:
    print(f"Bad manifest: {e.reason}")
```

## Capability requirements

| Operation        | Required capabilities                                   |
| ---------------- | ------------------------------------------------------- |
| `write_dataset`  | `ATOMIC_WRITE` (always), `DELETE` (if `overwrite=True`) |
| `read_dataset`   | `READ`                                                  |
| `delete_dataset` | `DELETE`                                                |
| `dataset_exists` | `READ`                                                  |

## See also

- [ext.parquet API reference](https://docs.remotestore.dev/stable/reference/api/extensions/parquet/index.md) — full class and method docs
- [Parquet Dataset example](https://docs.remotestore.dev/stable/tutorial/examples/parquet-dataset/index.md) — runnable example
- [PyArrow Adapter](https://docs.remotestore.dev/stable/guides/pyarrow-adapter/index.md) — lower-level Store-as-FileSystem bridge
- [Data Lake Patterns](https://docs.remotestore.dev/stable/guides/data-lake-patterns/index.md) — medallion architecture patterns

# PyArrow FileSystem Adapter

The `ext.arrow` module wraps any `Store` into a `pyarrow.fs.PyFileSystem`, unlocking seamless interop with the PyArrow ecosystem: datasets, Pandas, Polars, DuckDB, PyIceberg, and Delta Lake all accept `pyarrow.fs.FileSystem` objects for I/O.

## Installation

```
pip install "remote-store[arrow]"
```

This installs `pyarrow >= 12.0.0` as an optional dependency. The adapter works with any backend — no extra configuration needed.

## Quick Start

```
import pyarrow as pa
import pyarrow.parquet as pq

from remote_store import Store
from remote_store.backends import MemoryBackend
from remote_store.ext.arrow import pyarrow_fs

store = Store(backend=MemoryBackend())
fs = pyarrow_fs(store)

# Now use `fs` anywhere PyArrow accepts a filesystem:
table = pa.table({"col": [1, 2, 3]})
pq.write_table(table, "data.parquet", filesystem=fs)
result = pq.read_table("data.parquet", filesystem=fs)
```

## Parquet Round-Trip

```
import pyarrow as pa
import pyarrow.parquet as pq
from remote_store.ext.arrow import pyarrow_fs

fs = pyarrow_fs(store)

# Write
table = pa.table({"id": [1, 2], "value": ["a", "b"]})
pq.write_table(table, "output.parquet", filesystem=fs)

# Read
result = pq.read_table("output.parquet", filesystem=fs)
```

## Pandas Integration

```
import pandas as pd

df = pd.DataFrame({"x": [10, 20], "y": ["foo", "bar"]})
df.to_parquet("pandas.parquet", engine="pyarrow", filesystem=fs)

result = pd.read_parquet("pandas.parquet", engine="pyarrow", filesystem=fs)
```

## Dataset Discovery

PyArrow's dataset API discovers partitioned data automatically:

```
import pyarrow.dataset as ds

dataset = ds.dataset("data/", filesystem=fs, format="parquet")
table = dataset.to_table()
```

## Configuration

The adapter accepts two tuning parameters:

| Parameter                   | Default | Description                                                                                                                                                             |
| --------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `materialization_threshold` | 64 MB   | Max file size for full-file read in `open_input_file`. Files above this threshold stream via `PythonFile` (if seekable) or fall back to materialization with a warning. |
| `write_spill_threshold`     | 64 MB   | Max in-memory buffer size for writes. Exceeding this spills to a temporary file on disk.                                                                                |

```
fs = pyarrow_fs(
    store,
    materialization_threshold=128 * 1024 * 1024,  # 128 MB
    write_spill_threshold=32 * 1024 * 1024,        # 32 MB
)
```

## Tiered Read Strategy

`open_input_file` (used by Parquet readers) selects the best approach:

| Condition                         | Strategy                                     | Memory      | Notes                             |
| --------------------------------- | -------------------------------------------- | ----------- | --------------------------------- |
| Backend has native PyArrow FS     | **Tier 1**: native `open_input_file`         | ~range size | Zero overhead, C++ range requests |
| File \<= threshold                | **Tier 2**: `read_bytes()` -> `BufferReader` | Full file   | Zero GIL overhead                 |
| File > threshold, seekable stream | **Tier 3**: `read()` -> `PythonFile`         | Streaming   | GIL per read call                 |
| File > threshold, non-seekable    | **Tier 2 fallback** (with warning)           | Full file   | S3/Azure HTTP streams             |

**Tier 1** is automatically enabled for backends that expose a native PyArrow filesystem via `unwrap()` (currently `S3PyArrowBackend`). The handler detects this at construction time and bypasses Python I/O entirely for reads — the full C++ `ReadAt` -> HTTP Range request -> I/O coalescing pipeline runs with zero GIL overhead. This matters for analytical workloads (Parquet column pruning, dataset scans) where PyArrow issues many small range reads. For sequential byte streaming, Tier 1 does not provide a speed advantage — the regular S3 backend is faster for that use case (see [Performance](https://docs.remotestore.dev/stable/explanation/performance/#s3-vs-s3-pyarrow)).

## Thread Safety

The handler holds no shared mutable state. PyArrow's C++ layer may call handler methods from background threads (with the GIL acquired). Thread safety depends on the backend — all built-in backends (Memory, Local, S3, SFTP, Azure) are safe under concurrent calls. If using a custom backend, ensure its methods are thread-safe.

## Limitations

- **No append support.** `open_append_stream` raises `NotImplementedError`. Most backends (S3, Azure) lack native append semantics.
- **Root deletion blocked.** `delete_dir("")`, `delete_dir_contents("")`, and `delete_root_dir_contents()` raise `NotImplementedError` as a safety guard.
- **Write buffering.** Writes are buffered until `close()` — data is not visible in the Store during the write. This is inherent to the Store's single-shot `write()` API.
- **Tier 1 limited to S3-PyArrow.** `S3PyArrowBackend` exposes a native PyArrow filesystem for zero-copy Tier 1 reads. Other backends provide `native_path()` but use Tier 2/3 for data transfer.
- **Process exit on Linux.** PyArrow's C++ atexit handlers can deadlock during interpreter shutdown when a `PyFileSystem` is still alive. If your script hangs after completing, explicitly `del` the PyArrow filesystem and dataset objects before exit, or call `os._exit(0)` after cleanup.

## Error Mapping

Store errors are translated to standard Python exceptions:

| Store error              | Python exception      |
| ------------------------ | --------------------- |
| `NotFound`               | `FileNotFoundError`   |
| `AlreadyExists`          | `FileExistsError`     |
| `PermissionDenied`       | `PermissionError`     |
| `InvalidPath`            | `ValueError`          |
| `CapabilityNotSupported` | `NotImplementedError` |
| Other `RemoteStoreError` | `OSError`             |

No `RemoteStoreError` leaks to PyArrow callers — all exceptions are mapped with `from` chaining for debuggability.

## See also

- [API reference](https://docs.remotestore.dev/stable/reference/api/extensions/arrow/index.md)
- [Example script](https://github.com/haalfi/remote-store/blob/master/examples/integrations/pyarrow_adapter.py)

# Retry Policy

`RetryPolicy` provides unified retry configuration across backends. Each backend maps the policy to its native retry mechanism, avoiding retry multiplication.

## Quick Start

```
from remote_store import RetryPolicy, Store
from remote_store.backends import SFTPBackend

# Custom retry: 5 attempts, 2s base backoff, 30s max, 0.5s jitter
policy = RetryPolicy(
    max_attempts=5,
    backoff_base=2.0,
    backoff_max=30.0,
    jitter=0.5,
)

backend = SFTPBackend(host="sftp.example.com", retry=policy)
store = Store(backend)
```

## RetryPolicy Fields

| Field          | Type            | Default | Description                                                |
| -------------- | --------------- | ------- | ---------------------------------------------------------- |
| `max_attempts` | `int`           | `3`     | Total attempts (including the initial one). Must be >= 1.  |
| `backoff_base` | `float`         | `1.0`   | Minimum delay in seconds between retries.                  |
| `backoff_max`  | `float`         | `60.0`  | Maximum delay in seconds between retries.                  |
| `jitter`       | `float`         | `1.0`   | Random jitter added to each delay (0 to `jitter` seconds). |
| `timeout`      | `float \| None` | `None`  | Overall timeout in seconds. `None` means no timeout.       |

All fields are validated at construction time:

- `max_attempts >= 1`
- `backoff_base >= 0`
- `backoff_max >= 0`
- `jitter >= 0`
- `timeout > 0` or `None`

## Disabling Retries

Use the `disabled()` factory to create a single-attempt (no retry) policy:

```
from remote_store import RetryPolicy

no_retry = RetryPolicy.disabled()
# Equivalent to: RetryPolicy(max_attempts=1)
```

## TOML / YAML Configuration

Retry can be configured per-backend in config files:

```
[backends.production]
type = "sftp"

[backends.production.options]
host = "sftp.example.com"

[backends.production.retry]
max_attempts = 5
backoff_base = 2.0
backoff_max = 30.0
jitter = 0.5
```

```
backends:
  production:
    type: sftp
    options:
      host: sftp.example.com
    retry:
      max_attempts: 5
      backoff_base: 2.0
      backoff_max: 30.0
      jitter: 0.5
```

Load with the standard config loaders:

```
from remote_store import RegistryConfig, Registry

config = RegistryConfig.from_toml("config.toml")
registry = Registry(config)
store = registry.get_store("my-store")
```

## Per-Backend Mapping

Each backend translates `RetryPolicy` to its native retry mechanism. Not all fields are mappable to every backend — unmappable fields are logged at debug level.

### SFTP

Full mapping via tenacity:

| RetryPolicy field | Tenacity equivalent       |
| ----------------- | ------------------------- |
| `max_attempts`    | `stop_after_attempt(N)`   |
| `backoff_base`    | `wait_exponential(min=N)` |
| `backoff_max`     | `wait_exponential(max=N)` |
| `jitter`          | `+ wait_random(0, N)`     |
| `timeout`         | `\| stop_after_delay(N)`  |

Retry scope: **connection only** (not per-operation).

### S3

Only `max_attempts` is mappable to botocore:

```
botocore.config.Config(retries={"max_attempts": N, "mode": "standard"})
```

`backoff_base`, `backoff_max`, `jitter`, and `timeout` are managed by botocore internally and cannot be overridden.

### S3-PyArrow

Dual mapping:

- **PyArrow C++ side:** `AwsStandardS3RetryStrategy(max_attempts=N)`
- **s3fs side:** Same botocore config as S3.

Only `max_attempts` is mappable on both sides.

### Azure

Three fields are mappable to `ExponentialRetry`:

| RetryPolicy field | Azure equivalent                                |
| ----------------- | ----------------------------------------------- |
| `max_attempts`    | `retry_total = max_attempts - 1`                |
| `backoff_base`    | `initial_backoff = max(1, round(backoff_base))` |
| `jitter`          | `random_jitter_range = round(jitter)`           |

Azure expects integer seconds, so fractional values are rounded (with a minimum of 1 for `initial_backoff`).

`backoff_max` and `timeout` are not mappable.

### Local and Memory

These backends do not accept a `retry` parameter — passing one raises `TypeError`. Local filesystem and in-memory operations do not have transient failures that benefit from retry.

## See also

- [Retry policy example](https://docs.remotestore.dev/stable/tutorial/examples/retry-policy/index.md) — runnable script
- [Backend guides](https://docs.remotestore.dev/stable/guides/backends/index.md) — per-backend configuration details

# Transfer Operations

The `ext.transfer` module provides three functions for moving data between local files and Stores, or between two Stores: `upload`, `download`, and `transfer`.

All functions stream data — no file is ever fully loaded into memory. An optional `on_progress` callback fires per chunk with the byte count. No extra dependencies are required — the module is pure Python and always available.

## Quick Start

```
from remote_store import Store, upload, download, transfer
from remote_store.backends import MemoryBackend

store = Store(backend=MemoryBackend())

# Upload a local file to the store
upload(store, "local/report.csv", "reports/report.csv")

# Download a remote file to a local path
download(store, "reports/report.csv", "local/copy.csv")

# Transfer between two stores
other = Store(backend=MemoryBackend())
transfer(store, "reports/report.csv", other, "archive/report.csv")
```

## upload

```
upload(store, local_path, remote_path, *, overwrite=False, on_progress=None) -> None
```

Opens the local file in binary read mode and streams it to the Store via `store.write()`. The file is never fully loaded into memory.

- `overwrite=True`: overwrite an existing remote file.
- `on_progress`: callback receiving the byte count per read (not cumulative).

Raises `FileNotFoundError` if the local file does not exist. This check happens before any Store interaction.

```
upload(store, "/data/input.csv", "input.csv", overwrite=True)
```

## download

```
download(store, remote_path, local_path, *, overwrite=False, on_progress=None) -> None
```

Reads the remote file in 1 MiB chunks and writes each chunk to a local file.

- `overwrite=True`: overwrite an existing local file.
- `on_progress`: callback receiving the byte count per chunk written (not cumulative).

Raises `FileExistsError` if the local file already exists and `overwrite` is `False`. This check happens before calling `store.read()`.

The remote stream is always closed, even if an error occurs.

```
download(store, "reports/output.csv", "/tmp/output.csv")
```

## transfer

```
transfer(src_store, src_path, dst_store, dst_path, *, overwrite=False, on_progress=None) -> None
```

Reads the source file and streams it directly to the destination via `dst_store.write()`. The source and destination stores may be the same instance.

- `overwrite=True`: overwrite an existing file in the destination store.
- `on_progress`: callback receiving the byte count per read (not cumulative).

The source stream is always closed, even if an error occurs.

```
transfer(s3_store, "data/file.csv", sftp_store, "inbox/file.csv", overwrite=True)
```

## Progress Tracking

All three functions accept an `on_progress` callback. It receives the number of bytes processed in each chunk (not a cumulative total):

```
total = 0

def track(n: int) -> None:
    global total
    total += n
    print(f"{total} bytes transferred")

upload(store, "big_file.bin", "big_file.bin", on_progress=track)
```

## Error Handling

- **Local file errors** use stdlib exceptions: `FileNotFoundError` (upload) and `FileExistsError` (download).
- **Remote errors** use `RemoteStoreError` subtypes (`NotFound`, `AlreadyExists`, `CapabilityNotSupported`, etc.).
- `CapabilityNotSupported` always propagates immediately.
- **Partial files on failed download:** if `download` fails mid-transfer (e.g., network error), a partial local file may remain. Callers that need atomic semantics should download to a temporary path and rename on success. When retrying, pass `overwrite=True` to replace the partial file.

## Works with Store.child()

All transfer functions operate through the public Store API. They work correctly with `Store.child()`, capability gating, and path rebasing:

```
store = Store(backend=MemoryBackend())
reports = store.child("reports")

upload(reports, "local/q1.csv", "q1.csv")
# File is at "reports/q1.csv" in the root store
```

## See also

- [API reference](https://docs.remotestore.dev/stable/reference/api/extensions/transfer/index.md)
- [Example script](https://github.com/haalfi/remote-store/blob/master/examples/extensions/transfer_operations.py)

# Troubleshooting

Common errors and their solutions when using `remote-store`.

## ImportError for optional dependencies

**Symptom:** `ImportError: No module named 'pyarrow'` (or `paramiko`, `azure.storage.blob`, etc.)

**Cause:** Backend-specific dependencies are optional extras.

**Fix:** Install the extra for your backend:

```
pip install "remote-store[s3]"         # S3 backend (fsspec + s3fs)
pip install "remote-store[s3-pyarrow]" # S3-PyArrow backend
pip install "remote-store[sftp]"       # SFTP backend (paramiko)
pip install "remote-store[azure]"      # Azure backend
pip install "remote-store[all]"        # Everything
```

## Windows file-locking errors (WinError 32)

**Symptom:** `PermissionError: [WinError 32] The process cannot access the file because it is being used by another process`

**Cause:** An unclosed stream from `store.read()` keeps a file handle open. On Windows (unlike Unix), open handles prevent deletion and cleanup.

**Fix:** Always close streams or use a context manager:

```
# Good
stream = store.read("data.csv")
try:
    content = stream.read()
finally:
    stream.close()

# Better
with store.read("data.csv") as stream:
    content = stream.read()
```

## Unicode / cp1252 encoding errors on Windows

**Symptom:** `UnicodeEncodeError: 'charmap' codec can't encode character`

**Cause:** Windows console uses cp1252 by default. Characters like em dashes, arrows, or box-drawing characters crash `print()`.

**Fix:** Use ASCII-only characters in print statements. For Polars DataFrames, use `iter_rows(named=True)` with manual formatting instead of `print(df)`.

## SFTP host-key verification failure

**Symptom:** `SSHException: Server host key not found` or similar.

**Cause:** Paramiko requires host-key verification by default.

**Fix:** Set the host-key policy via `SFTPUtils.HostKeyPolicy` or a config dict. Available policies: `STRICT` (default), `TRUST_ON_FIRST_USE`, `AUTO_ADD` (dev/testing only).

Programmatic:

```
from remote_store.backends import SFTPUtils, SFTPBackend

backend = SFTPBackend(
    host="sftp.example.com",
    username="user",
    password="pass",
    host_key_policy=SFTPUtils.HostKeyPolicy.TRUST_ON_FIRST_USE,
)
```

Dict config (for `RegistryConfig`):

```
config = {
    "backends": {
        "my-sftp": {
            "type": "sftp",
            "host": "sftp.example.com",
            "username": "user",
            "password": "pass",
            "host_key_policy": "tofu",  # or "auto" for dev/testing only
        }
    },
    "stores": {"default": {"backend": "my-sftp"}},
}
```

See the [SFTP backend guide](https://docs.remotestore.dev/stable/guides/backends/sftp/index.md) for full configuration details.

## SFTP `IncompatiblePeer` on connect

**Symptom:** `paramiko.ssh_exception.IncompatiblePeer: Incompatible ssh peer (no acceptable {host key | kex algorithm | cipher | MAC})` during `SFTPBackend` connect. The error wraps four distinct negotiation failures; the actionable next step depends on which one.

**Diagnose first.** [`SFTPUtils.scan_host_algorithms()`](https://docs.remotestore.dev/stable/reference/api/sftp-utils/#remote_store.backends.SFTPUtils.scan_host_algorithms) parses the server's `SSH_MSG_KEXINIT` advertisement over a raw socket (no paramiko, no authentication). Print the relevant name-list to identify which list the server narrowed.

**Fix per failure mode:**

- `no acceptable host key` — typically a legacy server advertising only `ssh-rsa` against a modern paramiko (5+) that removed it from defaults. See the SFTP guide's [Legacy Servers](https://docs.remotestore.dev/stable/guides/backends/sftp/#legacy-ssh-rsa) section; `SFTPUtils.enable_ssh_rsa_compat()` re-enables `ssh-rsa` at process startup.
- `no acceptable kex algorithm` / `cipher` / `MAC` — server narrowed a different list. Widen the matching list via the SFTP constructor's `connect_kwargs={"disabled_algorithms": ...}`; the `enable_ssh_rsa_compat()` helper does not address these.

## Azure: HNS vs flat namespace

**Symptom:** `move()` or `copy()` fails on Azure with unexpected errors.

**Cause:** Azure Blob Storage has two modes: flat namespace (default) and hierarchical namespace (HNS / ADLS Gen2). Some operations behave differently.

**Fix:** Ensure the `hns` you declared matches the actual account type. The Azure backend does not auto-detect HNS — you pass `hns=True` for ADLS Gen2 or `hns=False` for flat Blob Storage. A mismatch (e.g. `hns=False` against a real HNS account) makes the backend use the wrong code path. If you are unsure of an account's type, call `AzureUtils.detect_hns(...)` once and pass the result. HNS accounts support true directory operations; flat namespace accounts simulate them, and HNS is recommended for data lake workloads.

## S3 endpoint configuration for MinIO / local S3

**Symptom:** Connection errors when using MinIO or another S3-compatible service.

**Cause:** The default S3 endpoint points to AWS. Local services need an explicit endpoint URL.

**Fix:**

```
config = {
    "backends": {
        "minio": {
            "type": "s3",
            "bucket": "my-bucket",
            "endpoint_url": "http://localhost:9000",
            "key": "minioadmin",
            "secret": "minioadmin",
        }
    },
    "stores": {"default": {"backend": "minio"}},
}
```

## CapabilityNotSupported error

**Symptom:** `CapabilityNotSupported: Backend 'memory' does not support GLOB`

**Cause:** Not all backends support every operation. Memory and SFTP lack native glob.

**Fix:** Check capabilities before calling, or use the portable fallback:

```
from remote_store import Capability, glob_files

if Capability.GLOB in store.capabilities():
    results = store.glob("**/*.csv")
else:
    results = glob_files(store, "**/*.csv")
```

See the [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) for the full backend x capability table.

## DatasetIncomplete error

**Symptom:** `DatasetIncomplete: Dataset at 'silver/orders' is incomplete`

**Cause:** The `_SUCCESS` marker is missing (partial write) or one or more Parquet part files listed in the manifest cannot be found.

**Fix:**

- Check that the write completed successfully (look for `_SUCCESS` under the dataset key).
- If parts are missing, the dataset was likely interrupted mid-write. Re-run the write with `overwrite=True`.
- Concurrent writers to the same `dataset_key` are not safe — coordinate externally.

## ManifestCorrupted error

**Symptom:** `ManifestCorrupted: Failed to parse manifest JSON`

**Cause:** The `manifest.json` file under a dataset key exists but contains invalid JSON or is missing required fields.

**Fix:**

- Inspect the manifest: `store.read_bytes("silver/orders/manifest.json")`.
- If corrupted, delete and re-write the dataset with `overwrite=True`.
- The `reason` attribute on the exception carries the specific parse failure.

## See also

- [Getting Started](https://docs.remotestore.dev/stable/tutorial/getting-started/index.md) — installation and quick start
- [Choosing a Backend](https://docs.remotestore.dev/stable/guides/choosing-a-backend/index.md) — picking the right backend
- [Error Handling example](https://docs.remotestore.dev/stable/tutorial/examples/error-handling/index.md)

# Write Integrity

Every `Store.write*()` call returns a [`WriteResult`](https://docs.remotestore.dev/stable/reference/api/models/#remote_store.WriteResult) carrying whatever metadata the backend produced during the write — ETag, version ID, last-modified timestamp, and (on Azure) a server-echoed content hash. Backends that fully populate these fields declare `Capability.WRITE_RESULT_NATIVE`; others return a minimal `WriteResult` with `path` and `size` only.

When you need a content hash regardless of backend, use the helpers in [`ext.write`](https://docs.remotestore.dev/stable/reference/api/extensions/write/index.md) (sync) or [`aio.ext.write`](https://docs.remotestore.dev/stable/reference/api/aio/extensions/write/index.md) (async). They compute the digest client-side as bytes flow through the stream, so the hash is always available.

## write_with_hash

Use `write_with_hash` when you have the content as `bytes` or a readable binary stream:

```
from remote_store.ext.write import write_with_hash

store = Store(MemoryBackend())
result = write_with_hash(store, "report.csv", b"col1,col2\n1,2\n")

assert result.digest is not None
print(result.digest.algorithm)  # sha256
print(result.digest.value)  # hex digest
```

`write_with_hash` returns a `WriteResult` with `digest` populated from the client-side hash — the SHA-256 is computed over the bytes as they are written, not after the fact. The default algorithm is `"sha256"`; pass `algorithm="sha512"` (or any `hashlib`-supported name) to override.

## open_atomic_with_hash

Use `open_atomic_with_hash` for streaming writes where you build the content incrementally. The context manager yields a `HashingAtomicWriter`; on clean exit `writer.result` holds the `WriteResult`:

```
from remote_store.ext.write import open_atomic_with_hash

store = Store(MemoryBackend())
with open_atomic_with_hash(store, "data.bin") as writer:
    writer.write(b"chunk one ")
    writer.write(b"chunk two")

result = writer.result
assert result is not None
assert result.digest is not None
print(result.digest.value)  # sha256 of "chunk one chunk two"
```

This requires `Capability.ATOMIC_WRITE`. When called without `metadata=`, `CapabilityNotSupported` is raised before any data is written. When `metadata=` is supplied, capability checks run on exit — see the [`open_atomic_with_hash` docstring](https://docs.remotestore.dev/stable/reference/api/extensions/write/index.md) for the metadata-branch caveat.

## Comparing write-time and read-time digests

Call `store.head()` after a write to retrieve a `WriteResult` from a metadata lookup (`source="sidecar"`). On backends that echo a digest natively, the values agree:

```
from remote_store.ext.write import write_with_hash

store = Store(MemoryBackend())
write_result = write_with_hash(store, "archive.bin", b"payload")

head = store.head("archive.bin")
print(head.size)  # 7
print(head.source)  # "sidecar"

# Compare digests if the backend echoed one back natively:
if write_result.digest and head.digest:
    assert write_result.digest.value == head.digest.value
```

`head()` is gated on `Capability.METADATA` and works on read-only backends that declare it (for example, the HTTP backend). Use it when you want a `WriteResult`-shaped view of a file that was written elsewhere.

## Storing user metadata alongside a file

Backends that declare `Capability.USER_METADATA` accept an optional `metadata=` mapping on `write*()` calls:

```
result = store.write(
    "report.csv",
    content,
    metadata={"owner": "data-team", "run-id": "2026-04-18"},
)
```

`metadata=` is a strict capability gate: passing a non-empty mapping to a backend without `USER_METADATA` raises `CapabilityNotSupported`. Check `store.supports(Capability.USER_METADATA)` before using it in backend-agnostic code. See the [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) for which backends support it.

## Async usage

For `AsyncStore`, use `write_with_hash` from `aio.ext.write` — the interface is identical, accepts `bytes` or `AsyncIterator[bytes]`, and hashes inline without buffering:

```
from remote_store.aio.ext.write import write_with_hash

store = AsyncStore(AsyncMemoryBackend())
result = await write_with_hash(store, "report.csv", b"col1,col2\n1,2\n")

assert result.digest is not None
print(result.digest.algorithm)  # sha256
print(result.digest.value)  # hex digest
```

## See also

- [`ext.write` API reference](https://docs.remotestore.dev/stable/reference/api/extensions/write/index.md)
- [`aio.ext.write` API reference](https://docs.remotestore.dev/stable/reference/api/aio/extensions/write/index.md)
- [`WriteResult` and `ContentDigest`](https://docs.remotestore.dev/stable/reference/api/models/#remote_store.WriteResult)
- [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) — `WRITE_RESULT_NATIVE` and `USER_METADATA` rows
- [Concurrency guide](https://docs.remotestore.dev/stable/explanation/concurrency/index.md) — atomicity semantics for `write_atomic` and `open_atomic`

# Backends

`remote-store` uses a pluggable backend system. Each backend implements the `Backend` abstract class and declares its capabilities. Pick a backend based on where your files live, install the optional extra, and everything else stays the same — the `Store` API is identical across all backends.

## Supported Backends

| Backend                                                                                                               | Status                | Install                                  |
| --------------------------------------------------------------------------------------------------------------------- | --------------------- | ---------------------------------------- |
| [Local filesystem](https://docs.remotestore.dev/stable/guides/backends/local/index.md)                                | Built-in              | `pip install remote-store`               |
| [Memory](https://docs.remotestore.dev/stable/guides/backends/memory/index.md)                                         | Built-in              | `pip install remote-store`               |
| [HTTP/HTTPS (read-only)](https://docs.remotestore.dev/stable/guides/backends/http/index.md)                           | Built-in              | `pip install remote-store`               |
| [Amazon S3 / MinIO](https://docs.remotestore.dev/stable/guides/backends/s3/index.md)                                  | Built-in              | `pip install "remote-store[s3]"`         |
| [S3 (PyArrow)](https://docs.remotestore.dev/stable/guides/backends/s3-pyarrow/index.md)                               | Built-in              | `pip install "remote-store[s3-pyarrow]"` |
| [SFTP / SSH](https://docs.remotestore.dev/stable/guides/backends/sftp/index.md)                                       | Built-in              | `pip install "remote-store[sftp]"`       |
| [Azure Blob / ADLS](https://docs.remotestore.dev/stable/guides/backends/azure/index.md)                               | Built-in              | `pip install "remote-store[azure]"`      |
| [Microsoft Graph (OneDrive / SharePoint / Teams)](https://docs.remotestore.dev/stable/guides/backends/graph/index.md) | Built-in (async-only) | `pip install "remote-store[graph]"`      |
| [SQL Blob (SQLite, PostgreSQL, ...)](https://docs.remotestore.dev/stable/guides/backends/sql-blob/index.md)           | Built-in              | `pip install "remote-store[sql]"`        |
| [SQL Query (read-only)](https://docs.remotestore.dev/stable/guides/backends/sql-query/index.md)                       | Built-in              | `pip install "remote-store[sql-query]"`  |

## Custom Backends

You can register your own backend using `register_backend`:

```
from remote_store import register_backend, Backend

class MyBackend(Backend):
    ...

register_backend("my-backend", MyBackend)
```

See the [Backend API reference](https://docs.remotestore.dev/stable/reference/api/backend/index.md) for the full interface to implement, and the [Build Your Own Backend](https://docs.remotestore.dev/stable/guides/custom-backend-guide/index.md) guide for a step-by-step walkthrough.

## See also

- [Choosing a Backend](https://docs.remotestore.dev/stable/guides/choosing-a-backend/index.md) — decision guide with trade-offs
- [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) — full backend x capability table

# Azure HNS Account Setup

Provision a real Azure Data Lake Storage Gen2 (ADLS Gen2 / Hierarchical Namespace) account using the `az` CLI. This guide is for contributors who need to run the live HNS test suite and for users who want to validate their own ADLS Gen2 provisioning against `remote-store` before production.

If you only need flat blob storage, the [Azure backend guide](https://docs.remotestore.dev/stable/guides/backends/azure/index.md) covers Azurite for local emulation. Azurite does not emulate Hierarchical Namespace, so HNS-specific paths (atomic rename, real directories) require a real account. That is what this guide sets up.

The shell snippets below are not executed in CI because they require authenticated Azure access. Treat them as a recipe to copy line by line, not as exact reproducible output.

## Prerequisites

- An Azure subscription. The free trial credit, the always-free tier, and any paid subscription all work. No specific SKU is required.
- The [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) (`az`) version 2.x.

## Sign in

```
az login
```

If your tenant requires multi-factor authentication on a per-resource basis, the default device flow may report `AADSTS50076 ... must use multi-factor authentication`. Sign in directly to that tenant instead:

```
az login --tenant <TENANT_ID>
```

Verify you have an enabled subscription:

```
az account show --query "{name:name, state:state}" -o table
```

The `state` column should read `Enabled`.

## Register the storage resource provider

A new subscription often has `Microsoft.Storage` unregistered. Registration is idempotent and takes about a minute.

```
az provider register --namespace Microsoft.Storage
az provider show --namespace Microsoft.Storage --query "{state:registrationState}" -o table
```

Re-run the second command until the state is `Registered`.

## Pick a region

Choose a region close to your egress location. `westeurope`, `switzerlandnorth`, and `germanywestcentral` are common picks for European users. List the display names with:

```
az account list-locations --query "[].{name:name, displayName:displayName}" -o table
```

Storage accounts incur cross-region egress fees, so keeping the region close to whoever runs the tests reduces both latency and cost.

## Pick an account name

Storage account names are globally unique, lowercase, alphanumeric, and between 3 and 24 characters. Check availability before creating:

```
az storage account check-name --name <ACCOUNT_NAME> -o table
```

If `NameAvailable` is `False`, append a digit suffix and retry.

## Create the resource group, account, and filesystem

The three commands below provision the resource group, the HNS-enabled storage account, and one ADLS Gen2 filesystem ("container" in flat-blob terminology). Run them in sequence; each completes in a few seconds, except for the account creation which takes about half a minute.

```
az group create \
  --name <RESOURCE_GROUP> \
  --location <REGION>

az storage account create \
  --name <ACCOUNT_NAME> \
  --resource-group <RESOURCE_GROUP> \
  --location <REGION> \
  --sku Standard_LRS \
  --kind StorageV2 \
  --enable-hierarchical-namespace true \
  --access-tier Hot \
  --allow-blob-public-access false \
  --min-tls-version TLS1_2

az storage fs create \
  --name <FILESYSTEM_NAME> \
  --account-name <ACCOUNT_NAME> \
  --auth-mode key
```

Why these flags:

- `--enable-hierarchical-namespace true` enables ADLS Gen2 semantics. This cannot be toggled after the account is created.
- `--sku Standard_LRS` is the cheapest redundancy tier and is sufficient for testing.
- `--access-tier Hot` minimises read costs, which matters for tests that read blob properties repeatedly.
- `--allow-blob-public-access false` and `--min-tls-version TLS1_2` align with current Azure security baselines.

If you copy the multi-line `az storage account create` form into a shell that wraps long lines on display, the line break can be interpreted as a command separator. If `isHnsEnabled` comes back as `null`, delete the account and re-run as a single line.

Confirm HNS is enabled:

```
az storage account show \
  --name <ACCOUNT_NAME> \
  --resource-group <RESOURCE_GROUP> \
  --query "{name:name, hns:isHnsEnabled, tls:minimumTlsVersion}" \
  -o table
```

The `hns` column should read `True`.

Declare `hns` when constructing the backend

The Azure backend does not auto-detect Hierarchical Namespace. Because this account has HNS enabled, construct the backend with `hns=True` (config `"hns": true`). For a flat Blob Storage account you would pass `hns=False`. If you ever need to discover an account's status programmatically, call `AzureUtils.detect_hns(...)` (or `await AzureUtils.adetect_hns(...)`) once and pass the result.

## Set CLI defaults

Setting the resource group and account once removes the need to repeat them on every command:

```
az config set defaults.group=<RESOURCE_GROUP>
az config set storage.account=<ACCOUNT_NAME>
az config set storage.auth_mode=key
```

These settings are stored in `~/.azure/config` on your local machine.

## Wire credentials into the environment

`remote-store` test suites and benchmarks read the connection string from the `AZURE_STORAGE_CONNECTION_STRING` environment variable. Pull the string and place it in your project `.env`:

```
az storage account show-connection-string \
  --name <ACCOUNT_NAME> \
  --resource-group <RESOURCE_GROUP> \
  --query connectionString \
  -o tsv
```

Add the resulting line to `.env`:

```
AZURE_STORAGE_CONNECTION_STRING=<paste here>
```

To opt into live HNS coverage, also set:

```
RS_TEST_LIVE_HNS=1
RS_TEST_LIVE_HNS_CONTAINER=<FILESYSTEM_NAME>
```

`tests/backends/azure/test_live_hns.py` carries the `live` pytest marker and exercises sync HNS semantics that the conformance suite against `azure_live` cannot express: directory-blob `hdi_isfolder` probes, WriteResult etag normalisation cross-check, the `write_atomic` streaming-payload guard, the `get_folder_info("")` HNS root carve-out, and the `exists` DataLake probe-fallback on real HNS directories. `live`-marked tests are excluded by default `addopts` and have to be opted into explicitly:

```
hatch run pytest -m live tests/backends/azure/test_live_hns.py
```

`tests/conftest.py` loads `.env` via `python-dotenv` when a `live` mark expression is in play, so dropping all three variables above into `.env` once is enough — a plain `hatch run pytest -m live …` picks them up. `override=False` keeps any value already set in the shell or by CI authoritative, and a regular `hatch run test` (with the default `-m 'not live'`) never loads `.env`.

If `RS_TEST_LIVE_HNS=1` is set but `AZURE_STORAGE_CONNECTION_STRING` is missing, empty, or points at the Azurite local emulator, the suite fails loud with a `pytest.fail` message rather than silently skipping. Azurite does not emulate Hierarchical Namespace, so an Azurite-backed run cannot validate HNS-specific behaviour.

Async HNS coverage lives in `tests/backends/azure/aio/test_live_hns.py`, which uses the same three-layer gate and a dedicated real-account fixture — it is explicitly **not** co-located with the Azurite-backed async live tests in `tests/backends/azure/aio/test_live.py` to avoid the Azurite reachability guard blocking real-ADLS-Gen2 CI.

`.env` is gitignored. Do not commit the connection string.

## Smoke test

A round-trip via `az storage blob` confirms the account is usable:

```
az storage blob upload \
  --container-name <FILESYSTEM_NAME> \
  --name smoke/hello.txt \
  --file pyproject.toml \
  --overwrite

az storage blob list \
  --container-name <FILESYSTEM_NAME> \
  --prefix smoke/ \
  -o table

az storage blob delete \
  --container-name <FILESYSTEM_NAME> \
  --name smoke/hello.txt
```

If the upload returns an `etag` and the list shows the blob, the account is ready for `remote-store` tests.

## Cost and cleanup

Standard_LRS HNS storage is the cheapest redundancy tier. Idle storage and ad-hoc test traffic at this scale are negligible against a free trial credit; consult the [Azure Storage pricing page](https://azure.microsoft.com/en-us/pricing/details/storage/blobs/) for current rates.

When you are done, delete the resource group to remove the account and all containers in one step:

```
az group delete \
  --name <RESOURCE_GROUP> \
  --yes \
  --no-wait
```

`--no-wait` returns immediately; the deletion completes asynchronously in about a minute.

## Rotating the account key

If the connection string is exposed (committed accidentally, pasted into a shared transcript, or copied to a less-trusted machine), rotate the key and re-pull the connection string:

```
az storage account keys renew \
  --account-name <ACCOUNT_NAME> \
  --resource-group <RESOURCE_GROUP> \
  --key key1

az storage account show-connection-string \
  --name <ACCOUNT_NAME> \
  --resource-group <RESOURCE_GROUP> \
  --query connectionString \
  -o tsv
```

Update `.env` with the new connection string. Any session that already holds the old key continues to work until you rotate `key2` as well.

## Troubleshooting

**`SubscriptionNotFound` after sign-in.** Your default tenant has no subscriptions. Sign in with `az login --tenant <TENANT_ID>` against the tenant that owns the subscription.

**`StorageAccountAlreadyTaken`.** The chosen account name is in use by someone else. Pick a more specific name and retry the `az storage account check-name` step.

**`AuthorizationFailed` on `az storage fs create`.** RBAC for data-plane operations propagates a few minutes after account creation. Use `--auth-mode key` (this guide's default) to bypass RBAC and authenticate with the account key.

**`isHnsEnabled` returns `null` or `false`.** The `--enable-hierarchical-namespace` flag did not reach the create call, usually because the multi-line command was split on a wrap. Delete the account and re-run on a single line.

## See also

- [Azure backend guide](https://docs.remotestore.dev/stable/guides/backends/azure/index.md) — using the account from `remote-store`
- [`AzureBackend` API reference](https://docs.remotestore.dev/stable/reference/api/backends/azure/index.md) — backend constructor options and method index
- [`AsyncStore` API reference](https://docs.remotestore.dev/stable/reference/api/aio/store/index.md) — async surface that the live HNS test suite exercises
- [Azure Storage Account documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview)
- [Hierarchical namespace overview](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-namespace)

# Azure Backend

The Azure backend stores files in Azure Blob Storage and Azure Data Lake Storage (ADLS) Gen2 using [`azure-storage-file-datalake`](https://learn.microsoft.com/en-us/python/api/azure-storage-file-datalake/) directly. You declare whether the account has Hierarchical Namespace (HNS) enabled via the required `hns` option; HNS accounts get atomic rename and real directories on ADLS Gen2, while plain Blob Storage accounts (`hns=False`) remain fully functional.

## Installation

```
pip install "remote-store[azure]"
```

This pulls in `azure-storage-file-datalake` and `azure-identity` (for `DefaultAzureCredential`).

## Usage

```
from remote_store import BackendConfig, RegistryConfig, Registry, StoreProfile

config = RegistryConfig(
    backends={
        "my-azure": BackendConfig(
            type="azure",
            options={
                "container": "my-container",
                "hns": True,  # ADLS Gen2; use False for plain Blob Storage
                "account_name": "mystorageaccount",
            },
        ),
    },
    stores={"data": StoreProfile(backend="my-azure", root_path="datasets")},
)

with Registry(config) as registry:
    store = registry.get_store("data")
    store.write("report.csv", b"col1,col2\n1,2\n")
    data = store.read_bytes("report.csv")
```

### Direct construction

```
from remote_store.backends import AzureBackend

# Account key. `hns` is required: True for ADLS Gen2, False for plain Blob Storage.
backend = AzureBackend(
    container="my-container",
    hns=True,
    account_name="mystorageaccount",
    account_key="...",
)

# SAS token
backend = AzureBackend(
    container="my-container",
    hns=True,
    account_name="mystorageaccount",
    sas_token="sv=2023-11-03&...",
)

# Connection string
backend = AzureBackend(
    container="my-container",
    hns=False,
    connection_string="DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;",
)

# DefaultAzureCredential (auto-resolves env vars, managed identity, CLI login, etc.)
backend = AzureBackend(
    container="my-container",
    hns=True,
    account_name="mystorageaccount",
)
```

If you do not know whether an account is HNS-enabled, discover it once with [`AzureUtils.detect_hns()`](#discovering-hns-status) and pass the result.

## Options

| Option                             | Type          | Default      | Description                                                                                                                                                                    |
| ---------------------------------- | ------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `container`                        | `str`         | *(required)* | Azure Storage container name                                                                                                                                                   |
| `hns`                              | `bool`        | *(required)* | Whether the account has Hierarchical Namespace (ADLS Gen2) enabled. No default and no auto-detection — see [HNS vs Non-HNS](#hns-vs-non-hns)                                   |
| `account_name`                     | `str`         | `None`       | Storage account name (builds URL automatically)                                                                                                                                |
| `account_url`                      | `str`         | `None`       | Full account URL (e.g. `https://myaccount.dfs.core.windows.net`)                                                                                                               |
| `account_key`                      | `str`         | `None`       | Storage account key                                                                                                                                                            |
| `sas_token`                        | `str`         | `None`       | Shared Access Signature token                                                                                                                                                  |
| `connection_string`                | `str`         | `None`       | Azure Storage connection string                                                                                                                                                |
| `credential`                       | `Any`         | `None`       | Any credential object (e.g. `DefaultAzureCredential()`)                                                                                                                        |
| `client_options`                   | `dict`        | `None`       | Extra kwargs passed to service clients (see [Upload tuning](#upload-tuning))                                                                                                   |
| `retry`                            | `RetryPolicy` | `None`       | Retry policy for transient failures                                                                                                                                            |
| `max_concurrency`                  | `int`         | `1`          | Parallel connections for uploads/downloads (>1 benefits large files)                                                                                                           |
| `reject_write_under_file_ancestor` | `bool`        | `False`      | If `True`, reject writes whose path nests under an existing regular file (non-HNS HEADs each ancestor; HNS rejects natively). Adds one HEAD per ancestor per nested-path write |

`hns` must be declared explicitly, and at least one of `account_name`, `account_url`, or `connection_string` must be provided.

## Authentication

The backend resolves credentials in this order:

1. **`account_key`** — if provided, used directly
1. **`sas_token`** — if provided, used directly
1. **`credential`** — any credential object (e.g. `DefaultAzureCredential()`)
1. **`DefaultAzureCredential`** — auto-detected from environment (requires `azure-identity`)

`DefaultAzureCredential` automatically tries environment variables, managed identity, Azure CLI, and other sources. See the [Azure Identity docs](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential) for details.

## HNS vs Non-HNS

You declare whether the account has Hierarchical Namespace (HNS) enabled via the required `hns` option. The backend adapts its behavior to the declared value:

| Feature                         | `hns=True` (ADLS Gen2)    | `hns=False` (Blob Storage)    |
| ------------------------------- | ------------------------- | ----------------------------- |
| Directories                     | Real entities             | Virtual (prefix-based)        |
| `write_atomic`                  | Temp file + atomic rename | Direct upload (PUT is atomic) |
| `move`                          | Atomic `rename_file`      | Copy + delete                 |
| `delete_folder(recursive=True)` | Single recursive delete   | Iterate + delete each blob    |

`hns` has no default and is never auto-detected. Declaring it makes the backend deterministic from construction: no account-level network call decides which semantics apply, so a transient failure or a propagation-delayed authorization response can never silently degrade an HNS account to flat behavior.

### Discovering HNS status

If you do not know an account's HNS status, discover it once with the fail-loud helper and pass the result:

```
from remote_store.backends import AzureUtils, AzureBackend

is_hns = AzureUtils.detect_hns(
    account_name="mystorageaccount",
    account_key="...",
)
backend = AzureBackend(container="my-container", hns=is_hns, account_name="mystorageaccount", account_key="...")
```

`AzureUtils.detect_hns()` issues a single account-info call and returns a `bool`. It raises on a probe error rather than guessing. An async sibling, `AzureUtils.adetect_hns(...)`, is available for async code.

Note that non-HNS `move()` (copy + delete) is not atomic and `overwrite=False` has a TOCTOU race on all account types. See the [Concurrency and Atomicity Guarantees](https://docs.remotestore.dev/stable/explanation/concurrency/index.md) guide for details.

## File Metadata

`get_file_info()` and `list_files()` return `FileInfo` objects with the following fields populated by the Azure backend:

| Field    | Source                                        | Notes                                                                                                |
| -------- | --------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `etag`   | `BlobProperties.etag`                         | Double-quotes stripped; lowercased.                                                                  |
| `digest` | `BlobProperties.content_settings.content_md5` | Populated as `ContentDigest("md5", <hex>)` when the blob has a stored Content-MD5; `None` otherwise. |

## Write Results

The Azure backend declares `WRITE_RESULT_NATIVE` and `USER_METADATA`. Write operations return a [`WriteResult`](https://docs.remotestore.dev/stable/reference/api/models/index.md) with `etag` and `last_modified` populated from the upload response. `digest` is populated as `ContentDigest("md5", <hex>)` when Azure echoes back `Content-MD5` in the upload response, and `None` otherwise. On HNS accounts, `write_atomic` always returns `digest=None`: it commits via a temp-file upload plus an atomic rename, and the backend reads only `etag` and `last_modified` from the post-rename properties — not a `Content-MD5` (the sibling `write` populates `digest` from its upload response on the same account). For a guaranteed digest regardless of method or backend, use the `remote_store.ext.write` helpers (e.g. `write_with_hash`). When blob versioning is enabled on a non-HNS container, `version_id` is also populated from the upload response.

Pass `metadata=` to store custom string key-value pairs as Azure blob metadata.

## Capabilities

Supports all capabilities except `SEEKABLE_READ` and `ATOMIC_MOVE`. See the [capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) for full details.

## Streaming and seekable reads

`read()` returns a forward-only streaming handle (not seekable). Data is fetched on demand, not loaded into memory upfront.

For random access (`seek()` / `tell()`) — for example, reading the footer of a large Parquet object — use **`Store.read_seekable()`** instead of materialising the whole blob. The Azure backend does not declare the `SEEKABLE_READ` capability (its `read()` is forward-only), but `read_seekable()` is backed by a native HTTP-Range reader that issues one ranged `download_blob` per `read()`: it fetches only the bytes the consumer seeks to, with no spill to a temp file and no in-RAM copy of the object.

```
import io

with Registry(config) as registry:
    store = registry.get_store("data")
    with store.read_seekable("large-file.parquet") as stream:
        stream.seek(-8, io.SEEK_END)   # only this 8-byte range is fetched
        magic = stream.read(8)
```

The `ext.arrow` / `ext.parquet` extensions read through `read_seekable()`, so analytical reads over Azure range-seek the footer and prune columns rather than download the whole object. Reaching for `read_bytes()` + `io.BytesIO` instead forces the **entire** blob into memory, which defeats the optimisation on multi-GB objects:

```
# Anti-pattern for large objects: materialises the whole blob in RAM.
data = store.read_bytes("large-file.parquet")
seekable_stream = io.BytesIO(data)
```

`io.BytesIO` is still fine for small blobs, where a single download is cheaper than issuing ranged requests.

### Async caveat: no native seekable read

The async API has no `read_seekable()` (see [Async API limitations](https://docs.remotestore.dev/stable/guides/async/#limitations)), and `remote_store.aio.ext` ships only `write` — there is no async `ext.arrow` / `ext.parquet`. To drive those analytical readers against an `AsyncAzureBackend` you must bridge to sync with [`AsyncBackendSyncAdapter`](https://docs.remotestore.dev/stable/guides/async-sync-bridges/index.md), whose `read()` is forward-only (`seekable()` is `False`). That masks Azure's range reader, so `read_seekable()` falls back to spooling the whole object to a temporary file (sized by `TMPDIR`) before the reader can seek. For large analytical reads, prefer the **sync** Azure store, which keeps the native range path.

## Upload tuning

The library sets conservative upload defaults on the Azure service clients to keep memory usage bounded during streaming transfers:

| Setting                            | Library default | SDK default |
| ---------------------------------- | --------------- | ----------- |
| `max_single_put_size`              | 1 MiB           | 64 MiB      |
| `max_block_size`                   | 1 MiB           | 4 MiB       |
| `min_large_block_upload_threshold` | 1               | 4 MiB + 1   |

These defaults cause uploads to use staged-block requests with small blocks. For large files where upload throughput matters more than memory, override via `client_options`:

```
AzureBackend(
    container="my-container",
    hns=False,  # or True for an ADLS Gen2 (HNS) account
    connection_string="...",
    client_options={
        "max_single_put_size": 8 * 1024 * 1024,   # 8 MiB
        "max_block_size": 4 * 1024 * 1024,          # 4 MiB
    },
)
```

## Concurrency and connection pooling

The async Azure SDK rides aiohttp (`AioHttpTransport`). A single `AsyncStore` (or `AsyncAzureBackend`) shared across many coroutines funnels every request through one transport and its connection pool, so under high fan-out — a large `asyncio.gather`, or one shared store behind a FastAPI app — the pool can become the bottleneck before the storage account's own limits do. (Share one store per event loop; see the [concurrency posture](https://docs.remotestore.dev/stable/explanation/concurrency/#concurrent-use-posture).)

Two independent levers:

- **`max_concurrency`** (constructor option, default `1`) sets the parallel connections used *within* a single upload/download. Raise it for large-file throughput; it does not change how many operations run concurrently.
- **Connector pool size.** `client_options` is forwarded verbatim to the Azure service clients, so for very high concurrent fan-out you can supply a custom `transport=` (an [`AioHttpTransport`](https://learn.microsoft.com/en-us/python/api/azure-core/azure.core.pipeline.transport.aiohttptransport) over an `aiohttp.ClientSession` whose `TCPConnector` has a higher `limit`) to raise the shared-pool ceiling.

Size the pool to your fan-out, not arbitrarily high: each connection is a socket against the storage account, which itself throttles (`429`). The right ceiling is best found by measuring against your workload.

## Escape Hatch

Access the underlying `FileSystemClient` when you need Azure-specific features:

```
from azure.storage.filedatalake import FileSystemClient

fs = backend.unwrap(FileSystemClient)
fs.get_paths(path="my-prefix")
```

## Local Development with Azurite

[Azurite](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite) is the official Azure Storage emulator. Start it with Docker:

```
docker run -p 10000:10000 mcr.microsoft.com/azure-storage/azurite
```

Then connect using the well-known Azurite connection string:

```
backend = AzureBackend(
    container="test",
    hns=False,  # Azurite is flat-namespace only
    connection_string=(
        "DefaultEndpointsProtocol=http;"
        "AccountName=devstoreaccount1;"
        "AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq"
        "/K1SZFPTOtr/KBHBeksoGMGw==;"
        "BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;"
    ),
)
```

Note: Azurite does not support Hierarchical Namespace. HNS-specific features (atomic rename, real directories) are tested with mocked SDK objects. To validate against a live ADLS Gen2 account, see [Azure HNS account setup](https://docs.remotestore.dev/stable/guides/backends/azure-hns-setup/index.md).

## See also

- [Capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md)
- [API reference](https://docs.remotestore.dev/stable/reference/api/store/index.md)
- [Example script](https://docs.remotestore.dev/stable/tutorial/examples/azure-backend/index.md)

## API Reference

## AzureBackend

```
AzureBackend(
    container: str,
    *,
    hns: bool | None = None,
    account_name: str | None = None,
    account_url: str | None = None,
    account_key: str | Secret | None = None,
    sas_token: str | Secret | None = None,
    connection_string: str | Secret | None = None,
    credential: Any | None = None,
    client_options: dict[str, Any] | None = None,
    retry: RetryPolicy | None = None,
    max_concurrency: int = 1,
    reject_write_under_file_ancestor: bool = False,
)
```

Bases: `Backend`

Azure Storage backend.

Uses the Blob SDK for non-HNS accounts (plain Blob Storage, Azurite) and the DataLake SDK for HNS accounts (ADLS Gen2) to get atomic rename and real directory support.

Whether the account is HNS (ADLS Gen2) is declared explicitly via the required `hns` argument -- the backend does not probe for it.

`move()` on non-HNS accounts is implemented as a server-side copy followed by a blob delete. This is non-atomic: a failure between the two steps may leave both source and destination present. HNS accounts use `rename_file` which *is* atomic, so an HNS instance could in principle advertise `ATOMIC_MOVE`. The capability is deliberately not declared per-instance: `CAPABILITIES` is a single class-level set shared by HNS and non-HNS instances alike, so it reports the guarantee common to both rather than varying by the declared `hns` value.

Parameters:

- **`container`** (`str`) – Azure Storage container name (required, non-empty).
- **`hns`** (`bool | None`, default: `None` ) – Whether the storage account has Hierarchical Namespace enabled (ADLS Gen2). Required -- there is no default and no runtime auto-detection. Pass True for ADLS Gen2 accounts (atomic rename, real directories) or False for flat Blob Storage. Use AzureUtils.detect_hns() to discover the value once if you do not already know it.
- **`account_name`** (`str | None`, default: `None` ) – Storage account name.
- **`account_url`** (`str | None`, default: `None` ) – Full account URL (e.g. https://myaccount.dfs.core.windows.net).
- **`account_key`** (`str | Secret | None`, default: `None` ) – Storage account key.
- **`sas_token`** (`str | Secret | None`, default: `None` ) – Shared Access Signature token.
- **`connection_string`** (`str | Secret | None`, default: `None` ) – Azure Storage connection string.
- **`credential`** (`Any | None`, default: `None` ) – Any credential object (e.g. DefaultAzureCredential()).
- **`client_options`** (`dict[str, Any] | None`, default: `None` ) – Additional options passed to service clients. The library sets max_single_put_size, max_block_size, and min_large_block_upload_threshold defaults for streaming memory discipline; user-supplied values take precedence.
- **`retry`** (`RetryPolicy | None`, default: `None` ) – Retry policy for transient failures.
- **`max_concurrency`** (`int`, default: `1` ) – Maximum number of parallel connections for uploads and downloads (default 1 -- sequential).
- **`reject_write_under_file_ancestor`** (`bool`, default: `False` ) – If True, write / write_atomic / open_atomic / move / copy HEAD each slash-aligned ancestor of the target path on non-HNS accounts and raise InvalidPath on the first regular-file hit, matching the cross-backend contract that hierarchical filesystems enforce natively. On HNS accounts the kwarg short-circuits: hdi_isfolder rejects the operation natively, and the backend detects the file ancestor on that rejection and re-raises it as InvalidPath, so HNS delivers the cross-backend contract with or without the kwarg set. Default False: enabling the check adds one HEAD per ancestor per nested-path write; paths without slashes short-circuit.

### check_health

```
check_health() -> None
```

Verify the backend is reachable and credentials are valid.

Raises:

- `PermissionDenied` – If credentials are invalid.
- `NotFound` – If the container does not exist.
- `BackendUnavailable` – If the backend cannot be reached.

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` with Azure-specific details.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – Plan with kind="azure" and details containing
- `ResolutionPlan` – container and account_url.

### exists

```
exists(path: str) -> bool
```

Return `True` if a blob or folder exists at *path*; never `NotFound`.

Probes the blob first (one HEAD); if absent, probes for a folder (an HNS directory, or any blob under the `path/` prefix on flat accounts). The root always exists.

Raises:

- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if *path* is an existing blob (not an HNS directory marker).

One HEAD round-trip; a missing blob or an `hdi_isfolder` directory returns `False`.

Raises:

- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if *path* is an existing folder (HNS directory or non-HNS prefix).

The root is always a folder. Costs one directory HEAD (HNS) or a one-item prefix listing (flat).

Raises:

- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### read

```
read(path: str) -> BinaryIO
```

Open *path* for reading and return a streaming handle.

Streams the blob body in chunks, so memory stays constant regardless of size. On HNS accounts one extra HEAD confirms *path* is a file (not an `hdi_isfolder` directory) before streaming.

Raises:

- `NotFound` – If the blob does not exist.
- `InvalidPath` – If path names a directory.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### read_seekable

```
read_seekable(path: str) -> BinaryIO
```

Open *path* for random-access reading and return a seekable handle.

Overrides the base spool: instead of buffering the whole blob, each `read()` issues one HTTP Range request (`download_blob(offset=, length=)`), so only the byte ranges actually read are fetched — ideal for Parquet column pruning.

Raises:

- `NotFound` – If the blob does not exist.
- `InvalidPath` – If path names a directory.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read and return the full blob content as bytes.

Downloads the whole blob into memory (unlike the lazy `read` stream).

Raises:

- `NotFound` – If the blob does not exist.
- `InvalidPath` – If path names a directory (HNS).
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* as a blob.

The content is uploaded with `upload_blob` on both flat and HNS accounts — a block-blob upload that becomes visible only when its final commit succeeds, never as a partially written blob. On flat (non-HNS) accounts this is an atomic replace. On hierarchical-namespace (HNS) accounts a guaranteed atomic replace is not assured; use `write_atomic` there when readers must never observe an intermediate state.

Raises:

- `AlreadyExists` – If the blob exists and overwrite is False.
- `InvalidPath` – If path names a directory, or (with the reject_write_under_file_ancestor opt-in, or natively on HNS) an ancestor exists as a file.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* atomically.

Readers never observe a partial file. On flat accounts this is the plain `PUT` (already atomic). On HNS it streams to a hidden temp file and promotes it with an atomic `rename_file` (the temp file is cleaned up on failure).

Raises:

- `AlreadyExists` – If the blob exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor exists as a file.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Yield a writable buffer promoted to *path* atomically on clean exit.

Writes spool to a temporary file (up to 8 MB in memory, then on disk). On flat accounts the buffer is committed with one `PUT`; on HNS it is uploaded to a temp blob and promoted with an atomic `rename_file`. An exception before exit leaves *path* untouched.

Raises:

- `AlreadyExists` – If the blob exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor exists as a file.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete the blob at *path*.

Raises:

- `NotFound` – If the blob does not exist (or, on HNS, a path component is itself a file) and missing_ok is False.
- `InvalidPath` – If path names a directory (HNS; use delete_folder).
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete a folder.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`recursive`** (`bool`, default: `False` ) – If True, delete all contents first.
- **`missing_ok`** (`bool`, default: `False` ) – If True, do not raise when absent.

Raises:

- `NotFound` – If the folder is missing and missing_ok is False.
- `InvalidPath` – If path names a file (use delete instead).
- `DirectoryNotEmpty` – If non-empty and recursive is False.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> Iterator[FileInfo]
```

Yield files under *path*, one `FileInfo` at a time.

Lazily pages the service listing (`walk_blobs`/`list_blobs` on flat accounts, `get_paths` on HNS); a missing path or a path under a file ancestor yields nothing. `recursive` lists the whole prefix (`max_depth` prunes client-side).

Raises:

- `PermissionDenied` – If credentials are rejected or lack access (401/403), surfaced during iteration.
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure, surfaced during iteration.

### list_folders

```
list_folders(path: str) -> Iterator[FolderEntry]
```

Yield immediate subfolders of *path* as `FolderEntry` records.

One paged prefix listing (`walk_blobs` common-prefixes on flat accounts, non-recursive `get_paths` on HNS); a missing path yields nothing.

Raises:

- `PermissionDenied` – If credentials are rejected or lack access (401/403), surfaced during iteration.
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure, surfaced during iteration.

### iter_children

```
iter_children(
    path: str,
) -> Iterator[FileInfo | FolderEntry]
```

Yield the immediate files and folders under *path* in one paged listing.

Overrides the base two-pass default with a single `walk_blobs` (flat) or `get_paths` (HNS) pass, yielding `FileInfo` for files and `FolderEntry` for folders. A missing path yields nothing.

Raises:

- `PermissionDenied` – If credentials are rejected or lack access (401/403), surfaced during iteration.
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure, surfaced during iteration.

### glob

```
glob(pattern: str) -> Iterator[FileInfo]
```

Match files against a glob pattern.

Parameters:

- **`pattern`** (`str`) – Glob pattern (e.g., "data/\*.csv", "\*\*/\*.txt").

Returns:

- `Iterator[FileInfo]` – An iterator of matching FileInfo objects.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return file metadata for `path`.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path names a directory (HNS: hdi_isfolder=true).

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Return aggregate metadata for the folder at *path*.

File count, total size, and latest modification time are gathered by paging the whole subtree listing, so cost scales with the number of descendants.

Raises:

- `NotFound` – If the folder does not exist.
- `InvalidPath` – If path names a file, not a folder.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename the file *src* to *dst*.

On HNS accounts this is a single native `rename_file` (atomic). On flat accounts it is a server-side copy followed by a delete — not atomic, so a failure between the two steps can leave both *src* and *dst* present; `ATOMIC_MOVE` is therefore not declared. `src == dst` is a no-op.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory, or an ancestor of dst exists as a file.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy the file *src* to *dst* via a server-side copy.

Issues `start_copy_from_url` so the bytes never pass through the client. The copy is not atomic — a failure can leave a partial destination. `src == dst` is a no-op.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory, or an ancestor of dst exists as a file.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

# Microsoft Graph Access Setup

Provision the credentials and a target drive for the Microsoft Graph backend (OneDrive and SharePoint). This guide is for contributors who need to run the live Graph test suite and for users who want to validate their own app registration and drive resolution against `remote-store` before production.

The audience is contributors and integrators standing up Graph access for the first time. It covers the two authentication flows the backend supports, how to register an Entra application for each, how to resolve an opaque `drive_id`, and how to wire the result into your environment.

Unlike the [Azure HNS setup](https://docs.remotestore.dev/stable/guides/backends/azure-hns-setup/index.md), there is no resource to create in an Azure subscription. OneDrive and SharePoint live in Microsoft 365 (Entra ID), not in an Azure subscription. The app registration is an Entra directory object; the drive already exists in someone's OneDrive or a SharePoint document library.

The shell snippets below are not executed in CI because they require authenticated access and, in the app-only case, an admin-consented tenant. Treat them as a recipe to copy line by line, not as exact reproducible output.

## Which flow do you need?

Microsoft Graph offers two ways to obtain the bearer token the backend attaches to every request. They map to two very different account types, and the choice determines everything that follows.

|                  | Client-credentials (app-only)                      | Device-code (delegated)                    |
| ---------------- | -------------------------------------------------- | ------------------------------------------ |
| Account type     | Work/school tenant (Entra ID)                      | Personal Microsoft account, or work/school |
| Human at sign-in | No, non-interactive                                | Yes, browser login on first use            |
| Secret           | Client secret or certificate                       | None (public client)                       |
| Drive targets    | SharePoint library, a user's OneDrive for Business | The signed-in user's OneDrive (`me`)       |
| Best for         | CI, daemons, services                              | Notebooks, CLIs, local development         |

If you have a **work/school Microsoft 365 tenant** and admin rights, use the [app-only path](#app-only-path-workschool-tenant) — it is non-interactive and the natural fit for CI.

If you only have a **personal Microsoft account** (Outlook.com, a Microsoft 365 Family/Personal subscription, or consumer OneDrive), app-only auth is not available to you at all — personal accounts have no tenant, no admin consent, and no application permissions. Use the [device-code path](#device-code-path-personal-account) instead.

> The device-code path below was walked end to end against a live consumer OneDrive while writing this guide. The app-only path is transcribed from Microsoft's documented flow and has **not** been validated in this repository's environment (no work/school tenant was available). A contributor with such a tenant should confirm the app-only steps and amend this guide.

## Prerequisites

- The [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) (`az`) version 2.x, used here to create the app registration. You can do the same in the [Entra admin center](https://entra.microsoft.com) by hand if you prefer a portal.
- For app-only: a work/school tenant where you are a Global Administrator (or can have admin consent granted).
- For device-code: any personal or work Microsoft account that owns a OneDrive.

You register the application in any Entra tenant you control. For the device-code path the registration tenant and the sign-in account need not match — a multi-tenant app registered in your own tenant can sign in a personal account from the `consumers` authority.

## Register the application

### Device-code path (personal account)

Register a public client that allows personal Microsoft accounts. No secret is created.

```
az ad app create \
  --display-name "remote-store-graph-dev" \
  --sign-in-audience AzureADandPersonalMicrosoftAccount \
  --is-fallback-public-client true \
  --public-client-redirect-uris "https://login.microsoftonline.com/common/oauth2/nativeclient"
```

Read back the client id:

```
az ad app list \
  --display-name "remote-store-graph-dev" \
  --query "[0].{appId:appId, audience:signInAudience, publicClient:isFallbackPublicClient}" \
  -o json
```

`audience` must be `AzureADandPersonalMicrosoftAccount` and `publicClient` must be `true`. The `appId` is your `GRAPH_CLIENT_ID`. There is no secret to record — consent for the delegated `Files.ReadWrite` scope is granted by the user at sign-in time.

### App-only path (work/school tenant)

Register a single-tenant confidential client, add a secret, grant application permissions, and admin-consent them.

```
az ad app create \
  --display-name "remote-store-graph" \
  --sign-in-audience AzureADMyOrg
```

Add a client secret (record the returned `password` — it is shown once):

```
az ad app credential reset \
  --id <APP_ID> \
  --append \
  --display-name "graph-secret" \
  --years 1
```

Grant Microsoft Graph **application** permissions. The Graph resource id is `00000003-0000-0000-c000-000000000000`; use the role id matching your drive target:

- `Sites.ReadWrite.All` (`9492366f-7969-46a4-8d15-ed1a20078fff`) for SharePoint document libraries.
- `Files.ReadWrite.All` (`75359482-378d-4052-8f01-80520e7db3cd`) for a user's OneDrive for Business.

```
az ad app permission add \
  --id <APP_ID> \
  --api 00000003-0000-0000-c000-000000000000 \
  --api-permissions 9492366f-7969-46a4-8d15-ed1a20078fff=Role

az ad app permission admin-consent --id <APP_ID>
```

`admin-consent` requires Global Administrator rights. If you cannot run it, have an admin open the consent URL:

```
https://login.microsoftonline.com/<TENANT_ID>/adminconsent?client_id=<APP_ID>
```

Application permissions carry no user context, so app-only tokens cannot reach `GET /me/drive`. They target a SharePoint library or a *named* user's OneDrive for Business (`/users/{id}/drive`). A certificate may be used instead of a secret; see the [Microsoft client-credentials reference](https://learn.microsoft.com/entra/identity-platform/v2-oauth2-client-creds-grant-flow).

## Resolve the drive id

The Graph backend takes one opaque `drive_id`. Its public helper `GraphUtils.resolve_drive_id(...)` accepts three target shapes and returns that id:

- **`"me"`** — the signed-in user's OneDrive, via `GET /me/drive`. Delegated (device-code) only.
- **A SharePoint site URL** (optionally with a library name) — via `GET /sites/{id}/drive` or `GET /sites/{id}/drives`. Works with app-only.
- **A Teams channel** `{team_id, channel_id}` mapping — via the channel's `filesFolder`. Works with app-only.

With the backend installed, resolving and capturing the id is three lines:

```
# pip install "remote-store[graph]"
from remote_store.aio import GraphAuth, GraphUtils

auth = GraphAuth(tenant_id="consumers", client_id="<GRAPH_CLIENT_ID>")
print("GRAPH_DRIVE_ID=" + GraphUtils.resolve_drive_id("me", token_provider=auth))
```

The first run completes a device-code sign-in in the terminal; `GraphAuth` caches the token, so later runs are non-interactive. If you prefer to validate the app registration without installing anything, the hand-rolled verification snippet below does the same with raw `msal` + `httpx`.

## Verify and capture the drive id (device-code)

An alternative to the `GraphUtils` route above: this snippet completes a device-code sign-in and reads back your drive id using `msal` and `httpx` directly (the libraries the built-in `GraphAuth` helper wraps), so it works without the `graph` extra installed — useful for verifying the app registration in isolation. It is hand-written rather than sourced from `examples/snippets/` because it needs real interactive credentials and cannot run in CI.

```
import json, os, sys
import httpx, msal

CLIENT_ID = "<GRAPH_CLIENT_ID>"
AUTHORITY = "https://login.microsoftonline.com/consumers"  # personal accounts
SCOPES = ["Files.ReadWrite", "User.Read"]
CACHE_PATH = "graph_token_cache.json"

cache = msal.SerializableTokenCache()
if os.path.exists(CACHE_PATH):
    cache.deserialize(open(CACHE_PATH, encoding="utf-8").read())

app = msal.PublicClientApplication(CLIENT_ID, authority=AUTHORITY, token_cache=cache)

result = None
if app.get_accounts():
    result = app.acquire_token_silent(SCOPES, account=app.get_accounts()[0])
if not result:
    flow = app.initiate_device_flow(scopes=SCOPES)
    print(flow["message"], flush=True)        # open the URL, enter the code
    result = app.acquire_token_by_device_flow(flow)

if "access_token" not in result:
    sys.exit(result.get("error_description", "auth failed"))
if cache.has_state_changed:
    open(CACHE_PATH, "w", encoding="utf-8").write(cache.serialize())

headers = {"Authorization": f"Bearer {result['access_token']}"}
drive = httpx.get(
    "https://graph.microsoft.com/v1.0/me/drive",
    params={"$select": "id,driveType,owner,webUrl"},
    headers=headers,
    timeout=30,
).json()
print("driveType:", drive["driveType"])       # 'personal' for consumer OneDrive
print("GRAPH_DRIVE_ID=" + drive["id"])
```

Sign in with the account that owns the target OneDrive. The `consumers` authority rejects work/school accounts, which keeps a personal-account run unambiguous. The token cache it writes makes the second run non-interactive — the same property the live test fixture relies on.

For a SharePoint or Teams target (app-only), swap the authority for your tenant, acquire a token with `acquire_token_for_client`, and call `GET /sites/root` or `GET /sites/{id}/drives` instead of `/me/drive`.

## Wire credentials into the environment

`remote-store` live Graph tests read configuration from environment variables, loaded from your project `.env`. The opt-in flag is `RS_TEST_LIVE_GRAPH=1`; the remaining variables differ by flow.

Device-code (personal account) — no secret:

```
RS_TEST_LIVE_GRAPH=1
GRAPH_CLIENT_ID=<your app client id>
GRAPH_TENANT_ID=consumers
GRAPH_DRIVE_ID=<id from the verification snippet>
```

App-only (work/school tenant) — with secret:

```
RS_TEST_LIVE_GRAPH=1
GRAPH_TENANT_ID=<your tenant GUID>
GRAPH_CLIENT_ID=<your app client id>
GRAPH_CLIENT_SECRET=<the secret from az ad app credential reset>
GRAPH_DRIVE_ID=<id resolved from your SharePoint or OneDrive target>
```

`.env` is gitignored. Never commit `GRAPH_CLIENT_SECRET` or the MSAL token cache (it holds a refresh token).

The built-in `GraphAuth` helper persists its MSAL cache under your user config directory (`platformdirs.user_config_dir("remote-store")`). The hand-rolled verification snippet above uses a local cache file instead, to stay self-contained.

## Troubleshooting

**`AADSTS50076` on app creation or sign-in.** Per-resource MFA. Sign in to the specific tenant: `az login --tenant <TENANT_ID>`.

**`AADSTS65001` (consent required).** The app-only permissions were added but not admin-consented. Run `az ad app permission admin-consent` or open the admin-consent URL above.

**`AADSTS700016` / `AADSTS50020` on device-code sign-in.** The account type does not match the app audience — e.g. signing a personal account into a single-tenant app, or vice versa. Confirm the registration used `AzureADandPersonalMicrosoftAccount` and the authority is `consumers`.

**`AADSTS7000215` (invalid client secret).** The secret expired or was mistyped. Re-run `az ad app credential reset` and update `.env`.

**`Tenant does not have a SPO license`** on `GET /me/drive` or `GET /sites/root`. The tenant has no SharePoint Online plan, so no drives exist. This is expected for a bare Azure subscription with no Microsoft 365 licenses; use a personal account (device-code) or add an M365 license to the tenant.

## Cleanup

Delete the app registration when you are done:

```
az ad app delete --id <APP_ID>
```

There is nothing to delete on the drive side — no resource was provisioned; you used an existing OneDrive or SharePoint library. To revoke a personal account's delegated consent, remove the app from [your account's app permissions](https://account.live.com/consent/Manage).

## See also

- [Azure HNS setup](https://docs.remotestore.dev/stable/guides/backends/azure-hns-setup/index.md) — the analogous runbook for a real ADLS Gen2 account
- [Choosing a backend](https://docs.remotestore.dev/stable/guides/choosing-a-backend/index.md) — decision guide with trade-offs
- [Capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) — full backend × capability table
- [`AsyncStore` API reference](https://docs.remotestore.dev/stable/reference/api/aio/store/index.md) — the async surface the Graph backend implements
- [Microsoft Graph authentication overview](https://learn.microsoft.com/graph/auth/auth-concepts)
- [OAuth 2.0 device-code flow](https://learn.microsoft.com/entra/identity-platform/v2-oauth2-device-code)

# Microsoft Graph Backend

The Graph backend stores files in **OneDrive, SharePoint, and Microsoft Teams** through the [Microsoft Graph](https://learn.microsoft.com/en-us/graph/) REST API, talking to a single drive over [`httpx`](https://www.python-httpx.org/) with bearer tokens from [`msal`](https://learn.microsoft.com/en-us/entra/msal/python/).

It is **async-only**: there is no sync `Store` wrapper and no config `type=` string, so it is constructed directly and used through `AsyncStore`. For the one-time credential and app-registration setup, follow the [Microsoft Graph access setup](https://docs.remotestore.dev/stable/guides/backends/graph-setup/index.md) guide first.

Verification coverage

The backend is live-verified against **consumer OneDrive** (device-code auth). SharePoint and OneDrive for Business drives go through the same Graph API and are covered by mocked tests, but are less exercised in practice — see the [operational caveats](#operational-caveats) for the known SharePoint-specific behaviours.

## Installation

```
pip install "remote-store[graph]"
```

This pulls in `httpx`, `msal`, `msal-extensions`, and `platformdirs` (the last two persist the MSAL token cache multi-process-safely).

## Usage

A backend instance targets one drive, identified by an opaque `drive_id`. Resolve the id once at wiring time, then hand it to the backend:

```
import asyncio

from remote_store.aio import AsyncStore, GraphAuth, GraphBackend, GraphUtils


async def main() -> None:
    # 1. A token provider. Device-code (interactive) auth against a personal
    #    Microsoft account needs only tenant + client id — no secret.
    auth = GraphAuth(tenant_id="consumers", client_id="<entra-app-id>")

    # 2. Resolve the target drive ("me" = the signed-in user's OneDrive).
    #    Inside async code, use the async resolver — the sync
    #    GraphUtils.resolve_drive_id runs its own event loop internally and
    #    raises RuntimeError when called from a running one.
    drive_id = await GraphUtils.aresolve_drive_id("me", token_provider=auth)

    # 3. Construct the backend and use it through AsyncStore. On the event loop,
    #    prefer the async auth.aget_token (off-loop acquisition + single-flight).
    backend = GraphBackend(drive_id, token_provider=auth.aget_token)
    async with AsyncStore(backend, root_path="Documents") as store:
        await store.write("report.csv", b"col1,col2\n1,2\n", overwrite=True)
        data = await store.read_bytes("report.csv")


asyncio.run(main())
```

`token_provider` is any `Callable[[], str]` or `Callable[[], Awaitable[str]]` returning a bearer token; `GraphAuth` is the built-in MSAL implementation but any callable works (e.g. a token minted by your own identity layer).

`GraphAuth` exposes both shapes over one instance. On the event loop, prefer the async `auth.aget_token` over passing the instance directly:

```
backend = GraphBackend(drive_id, token_provider=auth.aget_token)
```

`aget_token` offloads the blocking MSAL acquisition (and any contended token-cache lock wait) to a worker thread, so a cold or expired token never stalls sibling coroutines, and it **single-flights** concurrent acquisitions: a large `asyncio.gather` over a shared backend acquires the token once rather than once per coroutine (the same dedup covers the one-shot refresh after a `401`). Passing the instance directly (`token_provider=auth`) uses the synchronous `get_token`, which runs on the event loop and acquires per request — reserve it for synchronous wiring that has no running loop. A user-supplied async provider that wants this dedup must implement its own.

## Authentication

`GraphAuth` selects the OAuth flow from the credentials you supply:

| Supplied                                | Flow                          | Account type            | Use for                    |
| --------------------------------------- | ----------------------------- | ----------------------- | -------------------------- |
| `client_secret` or `client_certificate` | Client-credentials (app-only) | Work/school tenant      | CI, daemons, services      |
| neither                                 | Device-code (delegated)       | Personal or work/school | Notebooks, CLIs, local dev |

```
# App-only (client-credentials) against a work/school tenant:
auth = GraphAuth(
    tenant_id="<tenant-guid>",
    client_id="<entra-app-id>",
    client_secret="<secret>",  # accepts a Secret; masked in repr
)
```

The token is cached by MSAL on disk (default `<user_config_dir("remote-store")>/graph_token_cache.json`; override with `cache_path`). The cache is persisted multi-process-safely — a sibling `<cache_path>.lockfile` coordinates concurrent writers, so several workers or processes sharing the default cache will not corrupt it. Entra app registration, redirect URIs, admin consent, and the `AADSTS*` error catalogue are covered in the [setup guide](https://docs.remotestore.dev/stable/guides/backends/graph-setup/index.md).

## Resolving a drive_id

`GraphUtils.resolve_drive_id` (and its async twin `aresolve_drive_id`) turn a human-meaningful target into the opaque `drive_id` the backend needs:

| Target                                        | Resolves to                                                 |
| --------------------------------------------- | ----------------------------------------------------------- |
| `"me"`                                        | the authenticated user's default OneDrive (`GET /me/drive`) |
| a SharePoint site URL `str`                   | the site's default document library                         |
| `(site_url, library_name)` tuple              | a named document library on that site                       |
| `{"team_id": ..., "channel_id": ...}` mapping | a Teams channel's backing drive                             |

```
drive_id = GraphUtils.resolve_drive_id(
    "https://contoso.sharepoint.com/sites/Marketing",
    token_provider=auth,
)
```

The sync `resolve_drive_id` is for synchronous wiring code (config loading, CLI setup): it runs its own event loop internally, so calling it from inside a running event loop raises `RuntimeError`. Inside `async def` code, always use `await GraphUtils.aresolve_drive_id(...)`.

A `drive_id` is immutable for the life of a backend instance — point a second `GraphBackend` at a different drive rather than mutating one.

## Options

| Option              | Type                | Default                            | Description                                                                                                                                       |
| ------------------- | ------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `drive_id`          | `str`               | *(required)*                       | Opaque Graph drive id (resolve via `GraphUtils`)                                                                                                  |
| `token_provider`    | `Callable`          | *(required)*                       | Sync or async callable returning a bearer token                                                                                                   |
| `base_url`          | `str`               | `https://graph.microsoft.com/v1.0` | Graph API root                                                                                                                                    |
| `http_client`       | `httpx.AsyncClient` | `None`                             | Reuse an existing client; the caller owns its lifecycle (`aclose()` leaves it open). When omitted, one is created lazily and closed by `aclose()` |
| `retry`             | `RetryPolicy`       | `None`                             | Transient-failure retry policy; `None` uses the default profile                                                                                   |
| `upload_chunk_size` | `int`               | 10 MiB                             | Upload-session chunk size; must be a positive multiple of 320 KiB and `< 60 MiB`                                                                  |
| `copy_timeout`      | `float \| None`     | `None`                             | Wall-clock budget for copy/move monitor polling (see [caveat](#operational-caveats))                                                              |
| `base_path`         | `str`               | `""`                               | Scope every operation to this drive subfolder; keys are addressed relative to it and listings return relative keys. Defaults to the drive root    |
| `client_options`    | `dict`              | `None`                             | Extra kwargs passed through to the internal `httpx.AsyncClient`                                                                                   |

## Operational caveats

### Spooling and `TMPDIR`

When you `write()` an `AsyncIterator[bytes]` of unknown length, Graph's upload-session protocol requires the total size up front, so the backend spools the stream to a `SpooledTemporaryFile` (system temp, no explicit directory) and replays it once the length is known. On platforms with a small or restricted temp volume (Windows, locked-down containers), redirect the spill by setting **`TMPDIR`** (or the platform equivalent) before writing large unknown-length streams. Passing `bytes` instead of an iterator avoids the spool pass entirely. Spill events are logged at DEBUG with the marker `graph.upload.spool_spilled`.

The same concern exists on the **read side**: the backend does not declare `SEEKABLE_READ`, so `read_seekable()` — which the `ext.arrow` / `ext.parquet` extensions use for large files (through the [sync adapter](#extensions-sync-vs-async)) — is synthesised by spooling the entire file to the temp volume before the consumer can seek. Reading a large Parquet file over Graph therefore needs temp space for the whole file; size `TMPDIR` accordingly.

### Connection-pool tuning under high fan-out

The internal `httpx.AsyncClient` uses httpx's default connection pool — **100 connections** (`max_keepalive_connections=20`). Under very high concurrent fan-out (a large `asyncio.gather` over a shared backend, or many bridged threads) that ceiling is reached silently: requests queue and, on pool exhaustion, surface as an opaque `BackendUnavailable` rather than anything that names the pool as the cause.

Raise the ceiling by passing an [`httpx.Limits`](https://www.python-httpx.org/advanced/resource-limits/) through `client_options` (forwarded verbatim to the client):

```
backend = GraphBackend(
    drive_id="b!example-drive-id",
    token_provider=lambda: "token",
    client_options={
        "limits": httpx.Limits(
            max_connections=200,
            max_keepalive_connections=50,
        ),
    },
)
```

Size `max_connections` to your fan-out, not arbitrarily high: each connection is a socket against Graph, which itself throttles (`429`). This is the same passthrough mechanism every `client_options` key uses; supplying your own `http_client` instead lets you set limits on the client you construct.

### SharePoint-backed drives are less exercised

Live verification runs against consumer OneDrive only. SharePoint-backed drives have known divergences the backend already accounts for — some ignore HTTP range requests (see [Streaming](#streaming)) — and others that the live tier cannot reach. Treat SharePoint/business deployments as less-travelled ground and validate your workload before relying on it in production.

### `overwrite=True` can still raise `AlreadyExists`

Overwriting an existing **file** with `overwrite=True` is expected to replace it and succeed. One situation can still surface an `AlreadyExists` error despite the flag:

- **SharePoint-backed drives** may reject the replace of an existing file with a conflict. This does not happen on consumer OneDrive (the live-verified tier). The backend deliberately does **not** paper over it — silently treating the rejection as success could mask a genuine collision and would have to guess at a response shape the project cannot reproduce. If you need overwrite-or-create semantics on such a drive, delete the target first and then write, or serialise the writers.

A second situation — **concurrent creation of the same new key** (reproduced on consumer OneDrive) — is now **handled**: when several writers race to create the *same not-yet-existing* key with `overwrite=True`, a loser whose `replace` lands mid-create is re-attempted a bounded number of times. The winner's create has committed by then, so the re-issued `replace` overwrites it and the write succeeds — last-writer-wins, as `overwrite=True` promises. **Content integrity holds** throughout: one writer's bytes land intact, with no tearing or interleaving. (With `overwrite=False` the same race is a race-free create-if-absent: exactly one writer wins and the losers get `AlreadyExists` by design — see below.)

### `copy_timeout=None` is unbounded by default

Graph performs `copy()` (and sometimes `move()`) asynchronously: the backend polls a monitor URL until completion. With the default `copy_timeout=None` there is **no backend-imposed ceiling** — a copy against an unresponsive endpoint can block indefinitely. The backend does not substitute a fallback. Callers that cannot tolerate an unbounded wait must either set `copy_timeout` to a finite value at construction, or wrap the call in an external ceiling (`asyncio.timeout(...)`). On expiry the poller raises `BackendUnavailable` with the (query-stripped) monitor URL, poll count, and last status.

## Concurrency & consistency

`GraphBackend` is async-only and built for concurrent use on a single event loop:

- **One instance, one loop.** A `GraphBackend` is safe for concurrent coroutines driven by one event loop — `asyncio.gather` over a shared instance is fine. It is *not* safe to share across event loops; give each loop its own instance.
- **Safe from threaded sync code.** Driven through [`AsyncBackendSyncAdapter`](https://docs.remotestore.dev/stable/guides/async-sync-bridges/index.md), a single instance is safe for concurrent threads — the adapter serialises them onto its private loop. This is **unlike** the SFTP backend, where you need one instance per thread.
- **`overwrite=False` is a race-free create-if-absent.** It maps to a server-side atomic create (Graph's create-if-absent conflict behaviour), so two writers racing to create the same new key cannot both win — the loser gets `AlreadyExists`. There is no client-side check-then-write window, unlike the TOCTOU behaviour described in the [concurrency guide](https://docs.remotestore.dev/stable/explanation/concurrency/#overwritefalse-and-toctou).
- **`move` and `copy` are not atomic at the point of use.** `copy` (and sometimes `move`) runs server-side and is monitor-polled to completion; a crash mid-operation can leave both source and destination. Two callers racing a `move` / `copy` of the same item resolve to one winner, but the loser's error may be generic rather than typed.
- **Read-your-writes holds.** After a successful `write`, a subsequent `read` on the same instance returns the new content.

See the [concurrency guide](https://docs.remotestore.dev/stable/explanation/concurrency/#concurrent-use-posture) for the cross-backend posture table and the sync/async bridge rules.

## Write results

The backend declares `WRITE_RESULT_NATIVE`. Writes return a [`WriteResult`](https://docs.remotestore.dev/stable/reference/api/models/index.md) populated directly from the `driveItem` response — `size`, `etag`, and `last_modified` — with no extra `HEAD` round trip. It does **not** declare `USER_METADATA`: passing a non-empty `metadata=` raises `CapabilityNotSupported`; `{}` / `None` are no-ops.

## Capabilities

Supports all capabilities except `GLOB`, `SEEKABLE_READ`, `ATOMIC_MOVE`, and `USER_METADATA`. For glob, the portable `ext.glob.glob_files()` fallback works (Graph is `LIST`-capable) — but only through the sync adapter, since the `ext.*` suite is sync-only (see [below](#extensions-sync-vs-async)). See the [capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) for full details.

## Extensions: sync vs async

The extension ecosystem is split between the two Store surfaces, and the async side is much smaller:

| Extension surface                                                                                                         | Native `AsyncStore` | Sync `Store` (via adapter) |
| ------------------------------------------------------------------------------------------------------------------------- | ------------------- | -------------------------- |
| `aio.ext.write` — `write_with_hash`                                                                                       | Yes                 | —                          |
| [`ext.*`](https://docs.remotestore.dev/stable/reference/api/extensions/index.md) — the sync suite (glob, cache, arrow, …) | —                   | Yes                        |

Because `GraphBackend` is async-only, a native `AsyncStore` consumer has no `ext.*` surface at all — only `aio.ext.write.write_with_hash`. To use a sync extension over Graph, wrap the backend in [`AsyncBackendSyncAdapter`](https://docs.remotestore.dev/stable/guides/async-sync-bridges/index.md) and drive it through a sync `Store`. That works, but every call then hops through a bridged event loop and forfeits the native async streaming this backend exists to provide — reserve it for extension features you actually need (e.g. an occasional `glob_files()` or a Parquet read), not as the default way to use the backend.

## Streaming

`AsyncStore.read()` returns a forward-only `AsyncIterator[bytes]` streamed from the item's pre-authenticated download URL. Some SharePoint-backed drives ignore HTTP range requests; when that happens the backend transparently falls back to a full re-read and flags the returned `FileInfo.extra` (key `graph.read.range_fallback`), so a single read still yields correct bytes.

## Escape hatch

Access the underlying `httpx.AsyncClient` for Graph-specific calls the Store API does not expose:

```
import httpx

client = backend.unwrap(httpx.AsyncClient)
```

## See also

- [Microsoft Graph access setup](https://docs.remotestore.dev/stable/guides/backends/graph-setup/index.md) — app registration, auth flows, drive resolution
- [Async Store guide](https://docs.remotestore.dev/stable/guides/async/index.md) — the async API the backend is used through
- [Capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md)
- [API reference](https://docs.remotestore.dev/stable/reference/api/aio/backends/graph/index.md)
- [Example script](https://docs.remotestore.dev/stable/tutorial/examples/graph-backend/index.md)

## API Reference

## GraphBackend

```
GraphBackend(
    drive_id: str,
    *,
    token_provider: TokenProvider,
    base_url: str = _DEFAULT_BASE_URL,
    http_client: AsyncClient | None = None,
    retry: RetryPolicy | None = None,
    upload_chunk_size: int = _DEFAULT_UPLOAD_CHUNK_SIZE,
    copy_timeout: float | None = None,
    base_path: str = "",
    client_options: dict[str, Any] | None = None,
)
```

Bases: `AsyncBackend`

Async Microsoft Graph backend over OneDrive / SharePoint / Teams files.

A single instance targets one drive, identified by an immutable `drive_id`. Items are addressed by `/`-rooted POSIX path; transport is `httpx`; auth is a token-provider callable (the built-in `GraphAuth` helper, or any user-supplied callable).

Parameters:

- **`drive_id`** (`str`) – Opaque Graph drive id. Resolve one from a URL / "me" / Teams channel with GraphUtils.resolve_drive_id.
- **`token_provider`** (`TokenProvider`) – Callable\[[], str\] or Callable\[[], Awaitable[str]\] returning a bearer token, invoked lazily (never in __init__).
- **`base_url`** (`str`, default: `_DEFAULT_BASE_URL` ) – Graph API root (default https://graph.microsoft.com/v1.0).
- **`http_client`** (`AsyncClient | None`, default: `None` ) – Reuse an existing httpx.AsyncClient; the caller owns its lifecycle and close() does not close it. When omitted, one is created lazily on first use and closed by close().
- **`retry`** (`RetryPolicy | None`, default: `None` ) – Retry policy for transient failures; None uses the default RetryPolicy() profile.
- **`upload_chunk_size`** (`int`, default: `_DEFAULT_UPLOAD_CHUNK_SIZE` ) – Upload-session chunk size; must be a positive multiple of 320 KiB and strictly less than 60 MiB (Graph's per-request ceiling). Default 10 MiB.
- **`copy_timeout`** (`float | None`, default: `None` ) – Wall-clock budget for copy/move monitor polling, or None for no backend-imposed ceiling. When set, must be a positive float.
- **`base_path`** (`str`, default: `''` ) – Optional drive subfolder to scope every operation under. When set, all keys are addressed relative to this folder and keys returned by listing / to_key stay relative to it, so the backend behaves as if base_path were its root. Defaults to the drive root.
- **`client_options`** (`dict[str, Any] | None`, default: `None` ) – Extra options passed through to the internal httpx.AsyncClient. (When a future revision adds an explicit httpx-level constructor parameter, it takes precedence over a client_options key of the same name; the backend has no such parameter today, so this is passthrough only.)

Raises:

- `ValueError` – For an empty drive_id, a non-callable token_provider, a non-string base_path, an upload_chunk_size that is not a positive 320 KiB multiple below 60 MiB, or a non-positive copy_timeout.

### name

```
name: str
```

Unique identifier for this backend type — `"graph"`.

### capabilities

```
capabilities: CapabilitySet
```

Declared capabilities of this backend.

### drive_id

```
drive_id: str
```

The immutable target drive id.

### native_path

```
native_path(path: str) -> str
```

Return the Graph item-by-path metadata endpoint for `path`.

`/drives/{drive_id}/root:{encoded_path}:` with each segment percent-encoded. The empty key returns `/drives/{drive_id}/root:` (Graph's drive-root form, no trailing colon delimiter). A configured `base_path` is prepended so every key is scoped under it.

### to_key

```
to_key(native_path: str) -> str
```

Strip the drive-root prefix/delimiter and decode back to a key.

Inverse of `native_path`: removes `/drives/{drive_id}/root:` and the trailing `:` delimiter, then percent-decodes each segment. Inputs without the prefix are returned unchanged; the drive root maps to `""`.

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` carrying the drive id and base URL.

### unwrap

```
unwrap(type_hint: type[T]) -> T
```

Return the underlying `httpx.AsyncClient` (native-handle escape hatch).

Raises:

- `CapabilityNotSupported` – For any type other than httpx.AsyncClient.

### aclose

```
aclose() -> None
```

Close the owned HTTP client, cancel pollers, and flush the auth cache.

Safe to call multiple times. A caller-supplied `http_client` is left open — the caller owns it. Any in-flight copy/move monitor poller is cancelled cooperatively, and any upload session a `write()` left mid-chunk-loop is aborted via best-effort `DELETE` — every cleanup error is swallowed so `close()` never raises. The server-side copy / move continues (Graph monitor URLs have no cancel endpoint).

After this returns, any new op raises `BackendUnavailable` via the `_client` guard, and any op still in flight surfaces a typed error rather than a bare `RuntimeError` from the closing client.

### read

```
read(path: str) -> AsyncIterator[bytes]
```

Stream file content from the pre-signed download URL.

Fetches item metadata first (so the directory check happens before any byte is yielded), then streams the response body from `@microsoft.graph.downloadUrl` with no `Range` header (the URL is pre-signed, so no `Authorization` header is attached either). If the read is interrupted — the URL expires, or the connection drops mid-body — the stream re-fetches metadata for a fresh URL and resumes from the next unread byte with a `Range` request, provided the `eTag` is unchanged.

Raises:

- `NotFound` – If the path does not exist.
- `InvalidPath` – If the path names a folder.
- `PermissionDenied` – If the token is rejected or lacks access to the item (401/403).
- `BackendUnavailable` – If the download URL is missing, the file changed mid-read (eTag mismatch), the pre-signed host returns a non-success status, or the download fails at the transport level (connect/read timeout, DNS, reset).

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read full file content as bytes.

Delegates to `read` so the directory check and not-found mapping live in one place.

Raises:

- `NotFound` – If the path does not exist.
- `InvalidPath` – If the path names a folder.
- `PermissionDenied` – If the token is rejected or lacks access to the item (401/403).
- `BackendUnavailable` – As for read (missing URL, expiry, eTag change, host or transport failure).

### exists

```
exists(path: str) -> bool
```

Return `True` if a file or folder exists at *path*.

Any 404 — including a drive-identity `resourceNotFound` — is suppressed to `False`; never raises `NotFound`. A misconfigured or deleted drive therefore probes as missing rather than raising; it surfaces on the first error-raising operation.

Raises:

- `PermissionDenied` – If the token is rejected or lacks access to the item (401/403).
- `BackendUnavailable` – On throttling, 5xx, or transport failure.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if *path* exists and carries the `file` facet.

A missing item returns `False` (any 404 is suppressed, including a drive-identity `resourceNotFound`).

Raises:

- `PermissionDenied` – If the token is rejected or lacks access to the item (401/403).
- `BackendUnavailable` – On throttling, 5xx, or transport failure.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if *path* exists and carries the `folder` facet.

A missing item returns `False` (any 404 is suppressed, including a drive-identity `resourceNotFound`). The drive root (`""`) carries the `folder` facet and reports `True`.

Raises:

- `PermissionDenied` – If the token is rejected or lacks access to the item (401/403).
- `BackendUnavailable` – On throttling, 5xx, or transport failure.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return file metadata mapped from the Graph `driveItem` body.

`file.hashes` rides `FileInfo.extra["graph.file.hashes"]`; `metadata` is `None` (user metadata is not declared) and `digest` is left unset.

Raises:

- `NotFound` – If the path does not exist.
- `InvalidPath` – If the path names a folder.
- `PermissionDenied` – If the token is rejected or lacks access to the item (401/403).
- `BackendUnavailable` – On throttling, 5xx, or transport failure.

### iter_children

```
iter_children(
    path: str,
) -> AsyncIterator[FileInfo | FolderEntry]
```

Yield immediate files and folders under *path* in a single pass.

Overrides the default (which would chain `list_files` + `list_folders` and double the round-trips) to consume one `/children` response: `folder`-faceted items become `FolderEntry`, `file`-faceted items become `FileInfo`, in the order Graph returns them. A missing or file path yields nothing.

Raises:

- `PermissionDenied` – If the token is rejected or lacks access (401/403), surfaced during iteration.
- `BackendUnavailable` – On throttling, 5xx, or transport failure, surfaced during iteration.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> AsyncIterator[FileInfo]
```

List files under *path*.

When `max_depth` is set it governs traversal depth and `recursive` is ignored; otherwise `recursive=True` walks the subtree unbounded and the default lists only immediate files. A missing or file path yields nothing.

Raises:

- `PermissionDenied` – If the token is rejected or lacks access (401/403), surfaced during iteration.
- `BackendUnavailable` – On throttling, 5xx, or transport failure, surfaced during iteration.

### list_folders

```
list_folders(path: str) -> AsyncIterator[FolderEntry]
```

List immediate subfolders under *path*.

A missing or file path yields nothing.

Raises:

- `PermissionDenied` – If the token is rejected or lacks access (401/403), surfaced during iteration.
- `BackendUnavailable` – On throttling, 5xx, or transport failure, surfaced during iteration.

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Return aggregated folder metadata: recursive file count + total size.

Validates the path first — a missing item raises `NotFound` and a file item raises `InvalidPath`. `file_count` and `total_size` aggregate every file in the subtree; `modified_at` is the folder item's own timestamp.

Raises:

- `NotFound` – If the path does not exist.
- `InvalidPath` – If the path names a file.
- `PermissionDenied` – If the token is rejected or lacks access to the item (401/403).
- `BackendUnavailable` – On throttling, 5xx, or transport failure.

### write

```
write(
    path: str,
    content: AsyncWritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write a file, creating intermediate folders implicitly.

Content `<= 4 MiB` uses `PUT /content`; larger content uploads via a chunked session. An `AsyncIterator` of unknown length is spooled to size it first (the session `Content-Range` needs a known total). The returned `WriteResult` is `source="native"`, populated from the `driveItem` Graph returns. `overwrite` maps to Graph's `@microsoft.graph.conflictBehavior` (`replace` vs `fail`).

On `overwrite=True` (`conflictBehavior=replace`) a concurrent create of the *same not-yet-existing* key can still draw a `409 nameAlreadyExists` (live-reproduced on consumer OneDrive): the loser's replace lands while the winner's create is in flight. The write re-attempts the replace a bounded number of times — the winner's create has committed by the time the 409 returns, so the re-issue overwrites it. On the large-file upload-session path the same race can surface *mid-session* instead: the session opens cleanly, then a racing replace swaps the item and the next chunk `PUT` draws a `404`; under `overwrite=True` that is treated as the same create-race signal and the write re-opens a fresh session and retries. Large-file retries re-upload the whole body, so this convergence is best-effort: under sustained concurrent same-key overwrite a loser may still exhaust the budget and raise `AlreadyExists` (last-writer-wins still holds — one writer's content lands intact). A terminal conflict (a SharePoint-backed drive that rejects the replace outright) likewise keeps 409-ing and surfaces `AlreadyExists` once the budget is spent; the retry re-issues the replace, it never swallows the conflict.

Raises:

- `AlreadyExists` – If the file exists and overwrite=False, or an overwrite=True replace keeps drawing a 409 past the bounded re-attempt budget (a SharePoint-backed replace-rejection, or a loser in a sustained concurrent large-file same-key overwrite).
- `InvalidPath` – If the path names the drive root, an existing folder, or descends through a file ancestor.
- `CapabilityNotSupported` – If a non-empty metadata= reaches the backend directly (USER_METADATA is not declared).
- `PermissionDenied` – If the token is rejected or lacks access to the item (401/403).
- `BackendUnavailable` – On 5xx / throttling / transport failure, or a Graph contract gap (missing uploadUrl / nextExpectedRanges).
- `ResourceLocked` – If the item is locked mid-session. The session stays valid server-side, but its credentialed URL is not exposed in the exception.

### write_atomic

```
write_atomic(
    path: str,
    content: AsyncWritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Atomic write — delegates to `write`.

Graph's own write paths already provide the no-partial-content guarantee (`PUT /content` is service-atomic; an upload session commits only on the final chunk), so no client-side temp-rename is taken. The `WriteResult` shape, the `metadata=` gate, and the raised exceptions are inherited verbatim from `write`.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete a file (Graph moves it to the recycle bin).

Fetches the item first so a folder is rejected before any `DELETE` is issued (a bare `DELETE` on a folder would remove the folder and its contents). A missing item is `NotFound` unless `missing_ok`.

Raises:

- `NotFound` – If the file does not exist and missing_ok is False.
- `InvalidPath` – If the path names a folder (use delete_folder).
- `PermissionDenied` – If the token is rejected or lacks access to the item (401/403).
- `BackendUnavailable` – On throttling, 5xx, or transport failure.

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete a folder, optionally requiring it to be empty.

`recursive=True` is a single `DELETE` — Graph removes the folder and all contents atomically server-side. `recursive=False` first checks the folder is empty (via the item's `folder.childCount`, falling back to a `/children` probe when the count is absent) and raises `DirectoryNotEmpty` if not.

Raises:

- `NotFound` – If the folder does not exist and missing_ok is False.
- `InvalidPath` – If the path names a file.
- `DirectoryNotEmpty` – If non-empty and recursive is False.
- `PermissionDenied` – If the token is rejected or lacks access to the item (401/403).
- `BackendUnavailable` – On throttling, 5xx, or transport failure.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy a file, awaiting the async monitor to completion.

Graph answers `POST copy` with `202 Accepted` and a `Location` monitor URL; this polls it to a terminal state (bounded by `copy_timeout`). `src == dst` short-circuits after a single existence-confirming `GET`. A destination conflict surfaces as `AlreadyExists` (or `InvalidPath` for a folder / file-ancestor target) when `overwrite` is `False`.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src names a folder, or dst names an existing folder or descends through a file ancestor.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `PermissionDenied` – If the token is rejected or lacks access to the item (401/403).
- `BackendUnavailable` – On a 202 without a Location monitor URL, a copy_timeout expiry, or a transient/5xx failure.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename a file, awaiting the monitor when Graph goes async.

Graph answers `PATCH driveItem` synchronously in most cases (`200`); a large-item move may return `202` with a monitor URL (the async trigger is item size / server-side replication, not crossing folders), which is polled to completion exactly as `copy` does. `src == dst` short-circuits after one `GET`. Item identity (id / eTag / property bag) is preserved by Graph; the backend issues no compensating writes.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src names a folder, or dst names an existing folder or descends through a file ancestor.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `PermissionDenied` – If the token is rejected or lacks access to the item (401/403).
- `BackendUnavailable` – On a 202 without a Location monitor URL, a copy_timeout expiry, or a transient/5xx failure.

# HTTP Backend (Read-Only)

The HTTP backend reads files from HTTP/HTTPS URLs. Capabilities: `{READ, METADATA, LAZY_READ}` only — write, delete, list, move, and copy operations are not supported.

**Primary use cases:** government open data portals, dataset registries, static file servers, CDN-hosted assets, package archives, public APIs serving files.

## Usage

```
from remote_store import Store
from remote_store.backends import ReadOnlyHttpBackend

backend = ReadOnlyHttpBackend(
    base_url="https://data.example.com/datasets/",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
)
store = Store(backend=backend)

content = store.read_bytes("population/2024.csv")
info = store.get_file_info("population/2024.csv")
print(f"Size: {info.size}, Modified: {info.modified_at}")
```

### Via Registry

```
from remote_store import BackendConfig, RegistryConfig, Registry, StoreProfile

config = RegistryConfig(
    backends={
        "opendata": BackendConfig(
            type="http",
            options={
                "base_url": "https://data.example.com/datasets/",
                "timeout": 60,
                "headers": {"X-API-Key": "YOUR_KEY"},
            },
        ),
    },
    stores={"data": StoreProfile(backend="opendata")},
)

with Registry(config) as registry:
    store = registry.get_store("data")
    content = store.read_bytes("population/2024.csv")
```

## Options

| Parameter       | Type             | Default      | Description                                                                       |
| --------------- | ---------------- | ------------ | --------------------------------------------------------------------------------- |
| `base_url`      | `str`            | *(required)* | Root URL. Trailing `/` is appended if missing.                                    |
| `headers`       | `dict[str, str]` | `None`       | Custom headers sent with every request (API keys, auth tokens).                   |
| `timeout`       | `float`          | `30.0`       | Request timeout in seconds.                                                       |
| `retry`         | `RetryPolicy`    | `None`       | Retry policy for transient errors (429, 5xx).                                     |
| `http_client`   | `str`            | `None`       | Force transport: `"urllib"`, `"requests"`, or `"httpx"`. Auto-detected if `None`. |
| `verify_ssl`    | `bool`           | `True`       | Whether to verify TLS certificates.                                               |
| `max_redirects` | `int`            | `5`          | Maximum number of HTTP redirects to follow.                                       |

## Capabilities

This is a read-only backend. Only read-related capabilities are supported. See the [capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) for full details.

## HTTP Library

The backend auto-detects the best available HTTP library:

1. **httpx** (if installed) — connection pooling, HTTP/2. Install: `pip install "remote-store[httpx]"`
1. **requests** (if installed) — connection pooling, sessions. Install: `pip install "remote-store[requests]"`
1. **urllib** (always available) — stdlib, zero dependencies.

Override with `http_client="urllib"` (or `"requests"`, `"httpx"`).

## Folder Semantics

HTTP has no folder concept. `is_folder()` always returns `False`. `get_folder_info()` always raises `NotFound`.

## Metadata Mapping

`get_file_info()` maps HTTP response headers to `FileInfo` fields:

| FileInfo field | HTTP header                                  | Fallback             |
| -------------- | -------------------------------------------- | -------------------- |
| `size`         | `Content-Range` total, then `Content-Length` | `0`                  |
| `modified_at`  | `Last-Modified`                              | `datetime.min` (UTC) |
| `etag`         | `ETag`                                       | `None`               |
| `content_type` | `Content-Type`                               | `None`               |
| `extra`        | All headers                                  | `{"headers": {...}}` |

## CDN Compatibility

Some CDN-fronted servers (e.g. Cloudflare) return 403 on `HEAD` requests while allowing `GET`. The backend handles this transparently: when `HEAD` returns 401 or 403, it retries with `GET` + `Range: bytes=0-0` (downloading at most 1 byte). If the ranged `GET` succeeds, the backend remembers that HEAD is blocked and skips it for subsequent calls. This applies to `exists()`, `get_file_info()`, and `check_health()`.

If both `HEAD` and `GET` fail, the original error is raised.

Note that `check_health()` probes `base_url` (the root), not a specific file. Many HTTP servers and CDNs return 403 or 404 for directory URLs while serving individual files normally. A failing health check does not necessarily mean `read()` or `exists()` will fail on actual file paths.

## Composability

The primary value of making HTTP a backend (vs. standalone code):

- **`ext.cache`** — TTL-based caching of `read()` results. Avoids repeated downloads.
- **`ext.transfer`** — `download(store, "file.csv", local_path)` works out of the box.
- **`ext.observe`** — instrument HTTP reads with callbacks (timing, logging).
- **`ext.batch`** — `batch_exists(store, paths)` to check multiple resources.

## See also

- [Example script](https://docs.remotestore.dev/stable/tutorial/examples/http-backend/index.md)
- [Capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md)
- [API reference](https://docs.remotestore.dev/stable/reference/api/store/index.md)

## API Reference

## ReadOnlyHttpBackend

```
ReadOnlyHttpBackend(
    base_url: str,
    *,
    headers: dict[str, str] | None = None,
    timeout: float = 30.0,
    retry: RetryPolicy | None = None,
    http_client: str | None = None,
    verify_ssl: bool = True,
    max_redirects: int = 5,
)
```

Bases: `Backend`

Read-only backend for HTTP/HTTPS URLs.

Treats an HTTP endpoint as a file store with `{READ, METADATA, LAZY_READ}` capabilities (`read()` streams the response body lazily rather than buffering the whole file). Write, delete, list, move, and copy operations raise `CapabilityNotSupported`.

Parameters:

- **`base_url`** (`str`) – Root URL. A trailing / is appended if missing.
- **`headers`** (`dict[str, str] | None`, default: `None` ) – Custom headers sent with every request (e.g. API keys).
- **`timeout`** (`float`, default: `30.0` ) – Request timeout in seconds.
- **`retry`** (`RetryPolicy | None`, default: `None` ) – Retry policy for transient errors.
- **`http_client`** (`str | None`, default: `None` ) – Force a specific transport ("urllib", "requests", or "httpx"). Auto-detected if None.
- **`verify_ssl`** (`bool`, default: `True` ) – Whether to verify TLS certificates.
- **`max_redirects`** (`int`, default: `5` ) – Maximum number of redirects to follow.

### exists

```
exists(path: str) -> bool
```

Check existence via HEAD request (falls back to ranged GET).

### is_file

```
is_file(path: str) -> bool
```

HTTP resources are always files.

### is_folder

```
is_folder(path: str) -> bool
```

HTTP has no folder concept — always returns False.

### read

```
read(path: str) -> BinaryIO
```

Stream-read a file via GET.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Buffered-read a file via GET.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Get file metadata via HEAD request (falls back to ranged GET).

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

HTTP has no folder concept — always raises NotFound.

### check_health

```
check_health() -> None
```

Verify connectivity by sending HEAD to base_url (or GET if HEAD is blocked).

Note

The health check probes `base_url` (the root), not a specific file. Many HTTP servers and CDNs return 403 or 404 for directory URLs while serving individual files normally. A failing health check therefore does not necessarily mean `read()` or `exists()` will fail on actual file paths.

### close

```
close() -> None
```

Close the underlying transport.

### unwrap

```
unwrap(type_hint: type[T]) -> T
```

Return the transport if it matches the requested type.

### native_path

```
native_path(path: str) -> str
```

Return the full URL for a backend-relative key.

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` with HTTP-specific details.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – Plan with kind="http" and details containing
- `ResolutionPlan` – url and method.

### to_key

```
to_key(native_path: str) -> str
```

Strip base_url prefix to get a backend-relative key.

# Local Backend

The local backend stores files on the local filesystem. It is built-in and requires no extra dependencies.

## Installation

Built-in — no extra dependencies. Available with any `remote-store` install.

## Usage

```
from remote_store import BackendConfig, RegistryConfig, Registry, StoreProfile

config = RegistryConfig(
    backends={"local": BackendConfig(type="local", options={"root": "/data"})},
    stores={"files": StoreProfile(backend="local", root_path="files")},
)

with Registry(config) as registry:
    store = registry.get_store("files")
    store.write_text("readme.txt", "Hello!")
```

## Options

| Option | Type  | Description                                |
| ------ | ----- | ------------------------------------------ |
| `root` | `str` | Root directory for file storage (required) |

## Capabilities

All capabilities are supported except `USER_METADATA` — passing non-empty `metadata=` to a Local-backed store raises `CapabilityNotSupported`. The local backend is otherwise the reference implementation. See the [capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) for full details.

## Caveats

- **`overwrite=False` has a TOCTOU race.** The exists-check and write are separate operations. Concurrent writers can both pass the check and overwrite each other.
- **`move()` uses `shutil.move()`**, which delegates to `os.rename()` on the same filesystem (atomic) but falls back to copy+delete across filesystems. `write_atomic()` uses `os.replace()` and is truly atomic.

See the [Concurrency and Atomicity Guarantees](https://docs.remotestore.dev/stable/explanation/concurrency/index.md) guide for details.

## See also

- [Capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md)
- [API reference](https://docs.remotestore.dev/stable/reference/api/store/index.md)
- [Example script](https://docs.remotestore.dev/stable/tutorial/examples/quickstart/index.md)

## API Reference

## LocalBackend

```
LocalBackend(root: str)
```

Bases: `Backend`

Local filesystem backend using only the Python standard library.

`move()` uses `shutil.move`, which calls `os.rename` for same-filesystem moves (atomic) but falls back to copy-then-delete for cross-filesystem moves (not atomic). `ATOMIC_MOVE` is declared because within-root moves are always same-filesystem.

Parameters:

- **`root`** (`str`) – Absolute path to the root directory on the local filesystem.

### check_health

```
check_health() -> None
```

Confirm the root directory exists and is readable.

A lightweight two-syscall probe (`exists` then `os.access`) that touches no file content and is safe to call repeatedly.

Raises:

- `NotFound` – If the root directory does not exist.
- `PermissionDenied` – If the root directory exists but is not readable.

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` with local filesystem details.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – Plan with kind="local" and details containing
- `ResolutionPlan` – root and absolute_path.

### exists

```
exists(path: str) -> bool
```

Return `True` if a file or folder exists at *path*; never `NotFound`.

Returns `False` when a path component is itself a file, since the directory descent cannot proceed.

Raises:

- `InvalidPath` – If path escapes the backend root.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if *path* is an existing regular file.

`False` if *path* is absent, names a directory, or has a file as an ancestor component.

Raises:

- `InvalidPath` – If path escapes the backend root.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if *path* is an existing directory.

`False` if *path* is absent, names a file, or has a file as an ancestor component.

Raises:

- `InvalidPath` – If path escapes the backend root.

### read

```
read(path: str) -> BinaryIO
```

Open *path* for reading and return a lazy binary stream.

The stream reads on demand from the open file handle, so memory stays constant regardless of file size.

Raises:

- `NotFound` – If the file does not exist, or a path component is itself a file (the directory descent fails with ENOTDIR).
- `InvalidPath` – If path names a directory.
- `PermissionDenied` – If the OS denies read access to the file.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read and return the full file content as bytes.

Materialises the whole file in memory (unlike the lazy `read` stream), so size is bounded by available memory.

Raises:

- `NotFound` – If the file does not exist, or a path component is itself a file (ENOTDIR on descent).
- `InvalidPath` – If path names a directory.
- `PermissionDenied` – If the OS denies read access to the file.

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path*, creating parent directories as needed.

The bytes land directly at the final path (no temp-and-rename), so a crash or error mid-write can leave a partial or truncated file there — use `write_atomic` when readers must never observe a half-written file.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor of path exists as a regular file.
- `PermissionDenied` – If the OS denies write access.

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* atomically via a temp file plus `os.replace`.

Readers never observe a partial file: the body is written to a temporary file in the destination directory and atomically renamed over *path* on success (the temp file is removed on error). The rename is atomic because the temp file shares *path*'s directory, hence its filesystem.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor of path exists as a regular file.
- `PermissionDenied` – If the OS denies write access.

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Yield a writable stream promoted to *path* atomically on clean exit.

Writes go to a temp file in *path*'s directory; on normal exit it is atomically renamed over *path* (`os.replace`), and on any exception the temp file is removed and *path* is left untouched. Readers therefore never see a partially written file.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor of path exists as a regular file.
- `PermissionDenied` – If the OS denies write access.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete the file at *path*.

Raises:

- `NotFound` – If the file does not exist (or a path component is itself a file) and missing_ok is False.
- `InvalidPath` – If path names a directory — a type mismatch that missing_ok does not silence (use delete_folder).
- `PermissionDenied` – If the OS denies the unlink.

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete the folder at *path*.

`recursive=True` removes the folder and its whole subtree (`shutil.rmtree`); this is not atomic — an error partway through can leave the tree partially deleted. `recursive=False` removes only an empty folder (`rmdir`).

Raises:

- `NotFound` – If the folder does not exist and missing_ok is False.
- `InvalidPath` – If path names a file, not a folder.
- `DirectoryNotEmpty` – If the folder is non-empty and recursive is False.
- `PermissionDenied` – If the OS denies removal.

### glob

```
glob(pattern: str) -> Iterator[FileInfo]
```

Yield files matching *pattern* under the root, one at a time.

Lazily walks `Path.glob`; entries that escape the root through a symlink are skipped rather than returned, and directories are omitted — only files are yielded.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> Iterator[FileInfo]
```

Yield files under *path*, one `FileInfo` at a time.

Lazy: entries are produced as the directory is scanned, so listing a large tree holds only the current entry in memory. A missing or non-folder *path* yields nothing (no error). With `recursive` and `max_depth` set, traversal is pruned at the depth bound during the `os.walk` rather than filtered afterwards.

### list_folders

```
list_folders(path: str) -> Iterator[FolderEntry]
```

Yield immediate subfolders of *path* as `FolderEntry` records.

Lazy single-level scan; a missing or non-folder *path* yields nothing.

### iter_children

```
iter_children(
    path: str,
) -> Iterator[FileInfo | FolderEntry]
```

Yield the immediate files and folders under *path* in one scan.

Overrides the base (which chains `list_files` and `list_folders`, two passes) to walk the directory once, yielding `FileInfo` for files and `FolderEntry` for folders. A missing or non-folder *path* yields nothing.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return metadata for the file at *path* from a single `stat`.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path names a directory, not a file.

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Return aggregate metadata for the folder at *path*.

File count, total size, and latest modification time are computed by walking the entire subtree and `stat`-ing every file, so cost is O(files-in-subtree) — not a constant-time lookup.

Raises:

- `NotFound` – If the folder does not exist.
- `InvalidPath` – If path names a file, not a folder.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename the file *src* to *dst*.

Atomic when *src* and *dst* are on the same filesystem (`os.rename` via `shutil.move`) — which within a single root they always are, so `ATOMIC_MOVE` is declared. A cross-filesystem move (e.g. a bind-mounted subtree) falls back to copy-then-delete, which is not atomic. `src == dst` is a no-op; parent directories of *dst* are created as needed.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory, or an ancestor of dst exists as a regular file.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `PermissionDenied` – If the OS denies the operation.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy the file *src* to *dst*, preserving mode and mtime (`copy2`).

Not atomic: content is streamed to *dst*, so a crash mid-copy can leave a partial file at the destination. `src == dst` is a no-op; parent directories of *dst* are created as needed.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory, or an ancestor of dst exists as a regular file.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `PermissionDenied` – If the OS denies the operation.

# Memory Backend

The memory backend stores files in an in-process tree data structure. Zero dependencies, no filesystem access, no network. Always available — no optional extra needed.

**Primary use cases:** unit testing (no temp-dir setup/teardown), interactive exploration, documentation examples, CI speed.

## Installation

Built-in — no extra dependencies. Available with any `remote-store` install.

## Usage

```
from remote_store import Store
from remote_store.backends import MemoryBackend

backend = MemoryBackend()
store = Store(backend=backend, root_path="data")

store.write_text("hello.txt", "Hello, world!")
print(store.read_text("hello.txt"))  # 'Hello, world!'
```

### Via Registry

```
from remote_store import BackendConfig, RegistryConfig, Registry, StoreProfile

config = RegistryConfig(
    backends={"mem": BackendConfig(type="memory")},
    stores={"data": StoreProfile(backend="mem", root_path="data")},
)

with Registry(config) as registry:
    store = registry.get_store("data")
    store.write_text("readme.txt", "Hello!")
```

## Options

`MemoryBackend()` takes no constructor arguments. The backend starts empty.

## Capabilities

Supports all capabilities except `GLOB` (no native pattern matching — use `ext.glob.glob_files()` as a portable fallback) and `LAZY_READ` (all data lives in process memory; streams wrap pre-loaded bytes). See the [capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) for full details.

`write_atomic()` behaves identically to `write()` — in-memory writes are inherently atomic.

## Folder Semantics

Folders are explicit tree nodes, not virtual prefixes:

- `write("a/b/c.txt", data)` creates intermediate directory nodes for `a` and `a/b`.
- Deleting the last file in a directory does **not** auto-prune the parent. The empty folder persists until explicitly removed via `delete_folder()`.
- `delete_folder(path, recursive=False)` on a non-empty folder raises `DirectoryNotEmpty`.

This matches `LocalBackend` semantics exactly.

## Thread Safety

All operations are thread-safe. Mutations are serialized under a lock. The lock is never held while you iterate results — listing operations (`list_files`, `list_folders`, `iter_children`) snapshot state under the lock and build results lazily outside it, reducing lock contention for concurrent workloads.

## Testing with MemoryBackend

Replace `LocalBackend` + `tempfile.TemporaryDirectory` in your tests:

```
import pytest
from remote_store import Store
from remote_store.backends import MemoryBackend

@pytest.fixture
def store():
    return Store(backend=MemoryBackend(), root_path="test")

def test_write_and_read(store):
    store.write("file.txt", b"content")
    assert store.read_bytes("file.txt") == b"content"
```

## See also

- [Capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md)
- [API reference](https://docs.remotestore.dev/stable/reference/api/store/index.md)
- [Example script](https://docs.remotestore.dev/stable/tutorial/examples/memory-backend/index.md)

## API Reference

## MemoryBackend

```
MemoryBackend()
```

Bases: `Backend`

In-memory backend using a tree-indexed data structure.

Zero dependencies, no filesystem access, no network. Designed as a drop-in backend for unit testing, interactive exploration, and documentation examples.

All capabilities except `GLOB` are supported. The full conformance suite passes with zero skips.

Every mutating operation runs under a single process-wide lock, so `write`, `write_atomic`, `move`, and `copy` are atomic with respect to concurrent callers in the same process (`ATOMIC_MOVE` is advertised).

### exists

```
exists(path: str) -> bool
```

Return `True` if a file or folder exists at *path*; never `NotFound`.

The root (`""`) always exists.

Raises:

- `InvalidPath` – If path is absolute or contains a .. segment.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if *path* is an existing file (`False` for the root or a folder).

Raises:

- `InvalidPath` – If path is absolute or contains a .. segment.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if *path* is an existing folder; the root is always a folder.

Raises:

- `InvalidPath` – If path is absolute or contains a .. segment.

### read

```
read(path: str) -> BinaryIO
```

Return a binary stream over the stored bytes for *path*.

The value materialises in memory rather than streaming (`LAZY_READ` is not advertised): the returned stream wraps a copy of the resident bytes, so peak memory scales with the object size. Access is guarded by a single process-wide lock.

Raises:

- `NotFound` – If no file exists at path.
- `InvalidPath` – If path names a folder.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Return the full stored content of *path* as bytes.

Copies the resident value out under the lock; like `read` it holds the whole object in memory.

Raises:

- `NotFound` – If no file exists at path.
- `InvalidPath` – If path names a folder.

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Store *content* at *path*, creating parent folders implicitly.

The whole body is buffered into memory before the store (streams are drained fully first — no lazy write), and the swap-in happens under a single process-wide lock, so the write is atomic with respect to concurrent callers in the same process.

Raises:

- `AlreadyExists` – If a file exists at path and overwrite is False.
- `InvalidPath` – If path is empty or names a folder, or an ancestor of path exists as a file.

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Store *content* at *path* atomically (delegates to `write`).

Memory writes are already atomic under the process-wide lock, so this is exactly `write`; the whole body is buffered first.

Raises:

- `AlreadyExists` – If a file exists at path and overwrite is False.
- `InvalidPath` – If path is empty or names a folder, or an ancestor of path exists as a file.

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Yield an in-memory buffer committed to *path* atomically on clean exit.

Writes accumulate in a `BytesIO`; on exit the buffer is stored via `write` under the process-wide lock, so *path* updates in one atomic step. An exception before exit leaves *path* untouched.

Raises:

- `AlreadyExists` – If a file exists at path and overwrite is False.
- `InvalidPath` – If path is empty, or (on commit) names a folder or has a file ancestor.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete the file at *path*.

Raises:

- `NotFound` – If no file exists at path and missing_ok is False.
- `InvalidPath` – If path is empty or names a folder (a type mismatch missing_ok does not silence).

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete the folder at *path*.

`recursive=True` detaches the whole subtree in one locked step; `recursive=False` removes only an empty folder.

Raises:

- `NotFound` – If no folder exists at path and missing_ok is False.
- `InvalidPath` – If path is empty or names a file.
- `DirectoryNotEmpty` – If the folder is non-empty and recursive is False.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> Iterator[FileInfo]
```

Yield files under *path*.

A missing or non-folder *path* yields nothing. The tree is snapshotted under the lock and iterated outside it, so a long listing does not hold the lock; `recursive` walks the whole subtree (`max_depth` prunes it).

### list_folders

```
list_folders(path: str) -> Iterator[FolderEntry]
```

Yield immediate subfolders of *path* as `FolderEntry` records.

A missing or non-folder *path* yields nothing; children are snapshotted under the lock and yielded outside it.

### iter_children

```
iter_children(
    path: str,
) -> Iterator[FileInfo | FolderEntry]
```

Yield the immediate files and folders under *path* in one snapshot.

Overrides the base two-pass default: takes one locked snapshot of the directory and yields `FileInfo` for files and `FolderEntry` for folders. A missing or non-folder *path* yields nothing.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return metadata for the file at *path* from the in-memory node.

Raises:

- `NotFound` – If no file exists at path (including the empty path).
- `InvalidPath` – If path names a folder.

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Return aggregate metadata for the folder at *path*.

File count, total size, and latest modification time are computed by walking the whole subtree, so cost scales with the number of descendants.

Raises:

- `NotFound` – If no folder exists at path.
- `InvalidPath` – If path names a file.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename the file *src* to *dst* by re-linking the tree node.

Detach-from-source and attach-to-destination happen in one locked step, so the move is atomic with respect to concurrent callers (`ATOMIC_MOVE` is advertised) and the payload is never copied. `src == dst` verifies the source is a file and is otherwise a no-op.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst is empty, src names a folder, or dst names an existing folder.
- `AlreadyExists` – If dst is an existing file and overwrite is False.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy the file *src* to *dst* with an independent byte buffer.

The destination receives a fresh `bytearray` copy of the source bytes, all in one locked step (atomic). `src == dst` is a no-op.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst is empty, src names a folder, or dst names an existing folder.
- `AlreadyExists` – If dst is an existing file and overwrite is False.

# S3-PyArrow Backend

Drop-in alternative to the [S3 backend](https://docs.remotestore.dev/stable/guides/backends/s3/index.md) optimized for analytical workloads. Uses [PyArrow's C++ S3 filesystem](https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html) for data-path operations (reads, writes, copies) and s3fs for control-path operations (listing, metadata, deletion). This enables Tier 1 PyArrow integration — Parquet column pruning, I/O coalescing, and GIL-free reads — while keeping the same API.

## Installation

```
pip install "remote-store[s3-pyarrow]"
```

This pulls in both `s3fs` and `pyarrow`.

## Usage

```
from remote_store import BackendConfig, RegistryConfig, Registry, StoreProfile

config = RegistryConfig(
    backends={
        "s3pa": BackendConfig(
            type="s3-pyarrow",
            options={
                "bucket": "my-bucket",
                "key": "AWS_ACCESS_KEY_ID",
                "secret": "AWS_SECRET_ACCESS_KEY",
                "endpoint_url": "https://s3.amazonaws.com",
            },
        ),
    },
    stores={"data": StoreProfile(backend="s3pa", root_path="data")},
)

with Registry(config) as registry:
    store = registry.get_store("data")
    store.write("report.csv", b"col1,col2\n1,2\n")
```

## Options

Same constructor signature as the S3 backend:

| Option           | Type   | Description                                                      |
| ---------------- | ------ | ---------------------------------------------------------------- |
| `bucket`         | `str`  | S3 bucket name (required)                                        |
| `key`            | `str`  | AWS access key ID                                                |
| `secret`         | `str`  | AWS secret access key                                            |
| `region_name`    | `str`  | AWS region name                                                  |
| `endpoint_url`   | `str`  | Custom endpoint for S3-compatible services                       |
| `tls_ca_bundle`  | `str`  | Path to a PEM CA bundle file for custom/self-signed certificates |
| `client_options` | `dict` | Additional options passed to s3fs (control-path)                 |

## Custom TLS Certificates

Use `tls_ca_bundle` when connecting to S3-compatible services with custom or self-signed certificates. The parameter applies to both the PyArrow data path (`tls_ca_file_path`) and the s3fs control path (`client_kwargs.verify`).

```
backend = S3PyArrowBackend(
    bucket="my-bucket",
    endpoint_url="https://minio.internal:9000",
    tls_ca_bundle="/etc/ssl/certs/internal-ca.pem",
)
```

If not set, falls back to env vars: `AWS_CA_BUNDLE` > `REQUESTS_CA_BUNDLE` > `SSL_CERT_FILE`. See the [S3 backend guide](https://docs.remotestore.dev/stable/guides/backends/s3/#custom-tls-certificates) for the full fallback chain and config examples.

## When to use S3-PyArrow vs S3

| Scenario                                 | Recommended backend                  |
| ---------------------------------------- | ------------------------------------ |
| General-purpose file storage             | `s3`                                 |
| Sequential byte streaming (read/write)   | `s3` (faster at every file size)     |
| Analytical workloads (Parquet, datasets) | `s3-pyarrow` (Tier 1 column pruning) |
| Minimal dependencies                     | `s3` (only needs `s3fs`)             |
| PyArrow already in your stack            | `s3-pyarrow` (zero extra deps)       |

S3-PyArrow's C++ data path adds per-call overhead for sequential reads compared to the regular S3 backend. The advantage is native PyArrow integration: when PyArrow reads Parquet files through the [adapter](https://docs.remotestore.dev/stable/guides/pyarrow-adapter/index.md), it uses C++ range requests and I/O coalescing directly — no Python in the loop.

Both backends lack `ATOMIC_MOVE`, but they differ on `USER_METADATA`: the regular S3 backend declares it (write calls accept `metadata=` and store it as S3 object metadata), while S3-PyArrow does not. Code that passes `metadata=` to write methods will get `CapabilityNotSupported` on the PyArrow backend. For workloads that do not use object metadata, switching is safe — change the `type` in your config.

## Escape Hatch

Access the underlying filesystems when you need protocol-level features:

```
from pyarrow.fs import S3FileSystem as PyArrowS3
import s3fs

# PyArrow filesystem (data path)
pa_fs = backend.unwrap(PyArrowS3)

# s3fs filesystem (control path)
s3_fs = backend.unwrap(s3fs.S3FileSystem)
```

## Caveats

- **`move()` is not atomic.** Like the S3 backend, move is copy + delete. A crash between the two steps leaves duplicates.
- **`overwrite=False` has a TOCTOU race.** The exists-check and write are separate API calls. Concurrent writers can both pass the check and overwrite each other.

See the [Concurrency and Atomicity Guarantees](https://docs.remotestore.dev/stable/explanation/concurrency/index.md) guide for details and workarounds.

## See also

- [Capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md)
- [API reference](https://docs.remotestore.dev/stable/reference/api/store/index.md)
- [Example script](https://docs.remotestore.dev/stable/tutorial/examples/s3-pyarrow-backend/index.md)

## API Reference

## S3PyArrowBackend

```
S3PyArrowBackend(
    bucket: str,
    *,
    endpoint_url: str | None = None,
    key: str | Secret | None = None,
    secret: str | Secret | None = None,
    region_name: str | None = None,
    tls_ca_bundle: str | None = None,
    client_options: dict[str, Any] | None = None,
    retry: RetryPolicy | None = None,
    reject_write_under_file_ancestor: bool = False,
)
```

Bases: `_S3Base`

Hybrid S3 backend: PyArrow for reads/writes/copies, s3fs for listing/metadata.

Drop-in alternative to `S3Backend` with the same constructor signature. Uses PyArrow's C++ S3 filesystem for data-path operations (higher throughput for large files) and s3fs for control-path operations (listing, metadata, deletion).

`move()` is implemented as a PyArrow copy followed by an s3fs delete. This is non-atomic: a crash or network error between the two steps may leave both source and destination present. `ATOMIC_MOVE` is not declared.

Parameters:

- **`bucket`** (`str`) – S3 bucket name (required, non-empty).
- **`endpoint_url`** (`str | None`, default: `None` ) – Custom endpoint URL (e.g. for MinIO).
- **`key`** (`str | Secret | None`, default: `None` ) – AWS access key ID.
- **`secret`** (`str | Secret | None`, default: `None` ) – AWS secret access key.
- **`region_name`** (`str | None`, default: `None` ) – AWS region name.
- **`tls_ca_bundle`** (`str | None`, default: `None` ) – Path to a PEM CA bundle file. Falls back to AWS_CA_BUNDLE / REQUESTS_CA_BUNDLE / SSL_CERT_FILE.
- **`client_options`** (`dict[str, Any] | None`, default: `None` ) – Additional options passed to s3fs.
- **`reject_write_under_file_ancestor`** (`bool`, default: `False` ) – If True, write / write_atomic / open_atomic / move / copy HEAD each slash-aligned ancestor of the target path and raise InvalidPath on the first regular-file hit, matching the cross-backend contract that hierarchical filesystems enforce natively. Default False: each nested-path write otherwise pays one HEAD per ancestor; paths without slashes short-circuit.

### read

```
read(path: str) -> BinaryIO
```

Open *path* for reading and return a streaming handle.

Uses PyArrow's `open_input_file` (higher throughput for large objects) rather than the s3fs reader; still lazy, so memory stays constant.

Raises:

- `NotFound` – If the object does not exist.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read and return the full object content as bytes (via PyArrow).

Raises:

- `NotFound` – If the object does not exist.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path*, streaming straight to a multipart upload.

Unlike `S3Backend.write`, a plain streamed write here is **not** atomic: PyArrow's output stream exposes no abort, so a failure mid-body finalises a *truncated* object at *path*. Use `write_atomic` when readers must never observe a partial object. A `bytes` payload (no streaming) commits in one shot.

Raises:

- `AlreadyExists` – If the object exists and overwrite is False.
- `InvalidPath` – With the reject_write_under_file_ancestor opt-in, if an ancestor of path exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* atomically by buffering before upload.

The whole body is buffered first (a `bytes` payload is already materialised and delegates straight through), so a source failure happens off the wire and leaves no object at *path* — closing the atomicity gap in the plain streaming `write`.

Raises:

- `AlreadyExists` – If the object exists and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of path exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Yield a writable buffer committed to *path* atomically on clean exit.

Writes spool to a temporary file (up to 8 MB in memory, then on disk) and upload only on clean exit, so *path* never holds a partial object. An exception before exit leaves *path* untouched.

Raises:

- `AlreadyExists` – If the object exists and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of path exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename the object *src* to *dst*.

Existence checks and the delete go through s3fs; the copy is a PyArrow `copy_file`. Copy-then-delete is not atomic — a failure between the two steps can leave both *src* and *dst* present — so `ATOMIC_MOVE` is not declared. `src == dst` is a no-op.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of dst exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy the object *src* to *dst* via a PyArrow `copy_file`.

Like `move`, the operation carries no cross-operation atomicity guarantee. `src == dst` is a no-op.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of dst exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

# S3 Backend

The S3 backend stores files on Amazon S3 or any S3-compatible service (MinIO, DigitalOcean Spaces, etc.).

## Installation

```
pip install "remote-store[s3]"
```

## Usage

```
from remote_store import BackendConfig, RegistryConfig, Registry, StoreProfile

config = RegistryConfig(
    backends={
        "s3": BackendConfig(
            type="s3",
            options={
                "bucket": "my-bucket",
                "key": "AWS_ACCESS_KEY_ID",
                "secret": "AWS_SECRET_ACCESS_KEY",
                "endpoint_url": "https://s3.amazonaws.com",
            },
        ),
    },
    stores={"data": StoreProfile(backend="s3", root_path="data")},
)

with Registry(config) as registry:
    store = registry.get_store("data")
    store.write("report.csv", b"col1,col2\n1,2\n")
```

## Options

| Option           | Type   | Description                                                      |
| ---------------- | ------ | ---------------------------------------------------------------- |
| `bucket`         | `str`  | S3 bucket name (required)                                        |
| `key`            | `str`  | AWS access key ID                                                |
| `secret`         | `str`  | AWS secret access key                                            |
| `region_name`    | `str`  | AWS region name                                                  |
| `endpoint_url`   | `str`  | Custom endpoint for S3-compatible services                       |
| `tls_ca_bundle`  | `str`  | Path to a PEM CA bundle file for custom/self-signed certificates |
| `client_options` | `dict` | Additional options passed to s3fs                                |

## Custom TLS Certificates

Use `tls_ca_bundle` when connecting to S3-compatible services with custom or self-signed certificates (e.g., on-premises MinIO):

```
backend = S3Backend(
    bucket="my-bucket",
    endpoint_url="https://minio.internal:9000",
    tls_ca_bundle="/etc/ssl/certs/internal-ca.pem",
)
```

Or via config:

```
backends:
  minio:
    type: s3
    options:
      bucket: my-bucket
      endpoint_url: https://minio.internal:9000
      tls_ca_bundle: /etc/ssl/certs/internal-ca.pem
```

If `tls_ca_bundle` is not set, the following environment variables are checked in order (first non-empty value wins):

| Priority | Env var              | Standard |
| -------- | -------------------- | -------- |
| 1        | `AWS_CA_BUNDLE`      | boto3    |
| 2        | `REQUESTS_CA_BUNDLE` | requests |
| 3        | `SSL_CERT_FILE`      | OpenSSL  |

The path is validated at construction time — a `ValueError` is raised immediately if the file does not exist.

This replaces the previous workaround of passing `client_options={"client_kwargs": {"verify": "/path/to/ca.pem"}}`.

## Botocore Client Tuning

Anything beyond the first-class options above (proxies, timeouts, retry mode, S3 addressing style, pool sizes, custom `User-Agent`, …) is configured by passing a `config_kwargs` dict inside `client_options`. The dict is forwarded to `aiobotocore.config.AioConfig(**config_kwargs)`, so every keyword the underlying `botocore.config.Config` accepts is available.

Why `config_kwargs`, not `client_kwargs['config']`

`s3fs.S3FileSystem.set_session` always passes `config=AioConfig(**self.config_kwargs)` to `aiobotocore.create_client()`. Setting `client_kwargs['config']` in addition would duplicate the `config=` keyword and raise `TypeError: got multiple values for keyword argument 'config'`. The [S3 backend spec](https://docs.remotestore.dev/stable/explanation/design/specs/008-s3-backend/#s3-026) pins the routing: every `Config` option flows through `client_options['config_kwargs']`, and a pre-built Config in `client_kwargs` is rejected with a clear `ValueError`.

### Disabling inherited HTTP proxies

Required when the host has `HTTP_PROXY` / `HTTPS_PROXY` set in the environment but the S3 endpoint is reachable directly (typical for on-premises MinIO):

```
backend = S3Backend(
    bucket="my-bucket",
    endpoint_url="https://s3.internal:9000",
    client_options={
        "config_kwargs": {
            "proxies": {"http": None, "https": None},
        },
    },
)
```

### Pointing at a corporate proxy

```
backend = S3Backend(
    bucket="my-bucket",
    client_options={
        "config_kwargs": {
            "proxies": {
                "http": "http://proxy.corp:3128",
                "https": "http://proxy.corp:3128",
            },
        },
    },
)
```

### Retry policy

Prefer the first-class `retry=RetryPolicy(...)` argument — it is portable across backends and applied via the same merge path:

```
backend = S3Backend(
    bucket="my-bucket",
    retry=RetryPolicy(max_attempts=5),
)
```

When you need `mode="adaptive"` or other botocore-specific retry knobs, pass them through `config_kwargs.retries` and **do not** also pass `retry=RetryPolicy(...)`: `RetryPolicy` replaces the entire `retries` dict (matching `botocore.Config.merge` semantics), so any caller-supplied `mode` / non-`max_attempts` fields are lost. Pick one channel:

```
backend = S3Backend(
    bucket="my-bucket",
    client_options={
        "config_kwargs": {
            "retries": {"max_attempts": 5, "mode": "adaptive"},
        },
    },
)
```

### Connect / read timeouts

```
backend = S3Backend(
    bucket="my-bucket",
    client_options={
        "config_kwargs": {
            "connect_timeout": 3.0,
            "read_timeout": 10.0,
        },
    },
)
```

### MinIO-style path addressing

Required for endpoints that do not support virtual-host-style bucket addressing (most on-premises MinIO deployments):

```
backend = S3Backend(
    bucket="my-bucket",
    endpoint_url="https://minio.internal:9000",
    key="AKIA...",
    secret="...",
    client_options={
        "config_kwargs": {
            "s3": {"addressing_style": "path"},
        },
    },
)
```

### Putting it together

A realistic on-premises MinIO configuration combining the pieces above (custom CA bundle continues to come from `tls_ca_bundle=` / env vars, [Custom TLS Certificates](#custom-tls-certificates)):

```
backend = S3Backend(
    bucket="my-bucket",
    endpoint_url="https://s3.internal:9000",
    key="AKIA...",
    secret="...",
    retry=RetryPolicy(max_attempts=5),
    client_options={
        "config_kwargs": {
            "connect_timeout": 3.0,
            "read_timeout": 10.0,
            "s3": {"addressing_style": "path"},
            "proxies": {"http": None, "https": None},
        },
    },
)
```

## File Metadata

`get_file_info()` and `list_files()` return `FileInfo` objects with the following fields populated by the S3 backend:

| Field    | Source                 | Notes                                                                                   |
| -------- | ---------------------- | --------------------------------------------------------------------------------------- |
| `etag`   | `ETag` response header | Double-quotes stripped; lowercased. Example: `"abc123"` → `abc123`.                     |
| `digest` | —                      | Always `None`; S3 checksums require `ChecksumMode: ENABLED` (not requested by default). |

## Write Results

The S3 backend declares `WRITE_RESULT_NATIVE` and `USER_METADATA`. Write operations return a [`WriteResult`](https://docs.remotestore.dev/stable/reference/api/models/index.md) with `digest`, `etag`, and `last_modified` populated from the upload response — contrast with reads, where `digest` is always `None` (see the File Metadata table above).

Pass `metadata=` to store custom string key-value pairs as S3 object metadata. They round-trip through `get_file_info()` in `FileInfo.metadata`.

## Listing Strategies and Performance

S3 listing behavior differs sharply between shallow and recursive traversals. Understanding these trade-offs is critical for large buckets.

### Shallow Listing (Non-Recursive)

Use `list_files(path, recursive=False)` or `iter_children(path)` to list only direct children:

```
# List direct children only
for entry in store.iter_children("data/"):
    print(entry.name)  # Files and folders one level deep
```

**Characteristics:**

- Single S3 `ListObjectsV2` API call (or paginated requests if >1000 entries)
- O(n) cost where n = direct children count
- **Flat cost per call**, not dependent on bucket size
- Suitable for folder-first navigation (e.g., building a file browser UI)

### Recursive Listing (Flat Stream)

Use `list_files(path, recursive=True)` to fetch all files under a prefix:

```
# Stream all files under a prefix, regardless of depth
for file_info in store.list_files("data/", recursive=True):
    process(file_info)
```

**Why flat streaming wins:**

- Internally uses S3's `ListObjectsV2` pagination with a prefix, not delimiter-based folder traversal
- Single logical stream; S3 SDK handles pagination transparently
- O(n) cost where n = total objects in the prefix tree
- Avoids the O(n_folders) × (API calls + parsing overhead) of delimiter-based iteration

**If you need all objects under a prefix, use `recursive=True`:**

```
# ✓ Single flat stream (optimal)
for file in store.list_files("data/", recursive=True):
    process(file)
```

**Do not implement folder-by-folder traversal:**

```
# ❌ This makes one API call per folder level
def traverse_folders(prefix):
    for folder in list_folders(prefix):  # Calls ListObjectsV2 with delimiter=/
        yield from list_files(folder, recursive=False)
        yield from traverse_folders(folder)  # Recursive calls per subfolder
```

The traversal approach costs O(n_folders) API calls, even with few total files.

### Streaming Over Parallelization

For large buckets, use a **single sequential flat stream**, not parallel folder traversal:

```
# ✓ Single flat stream (optimal for large buckets)
for file in store.list_files("data/", recursive=True):
    process(file)
```

**Why not parallelize folder traversal:**

```
# ❌ Avoid parallel folder enumeration
from concurrent.futures import ThreadPoolExecutor

def parallel_traverse(prefix, executor):
    # Spawning threads per folder creates:
    # - O(n_folders) concurrent requests (thundering herd)
    # - Earlier rate-limiting hits (S3 per-partition limits)
    # - Thread pool overhead
    # - Loss of connection pooling benefits
```

**Flat streams are superior:**

- Single sequential `ListObjectsV2` respects S3's request pipelining
- Reuses pooled connections across paginated responses
- No thread overhead for what is already a streaming operation
- More predictable latency and throughput

On large buckets with thousands of folders, a flat stream is **orders of magnitude faster** than parallel traversal.

### Performance

See the [performance guide](https://docs.remotestore.dev/stable/explanation/performance/#comparative-results) for benchmark results. Listing is dominated by S3 API round-trip latency, not file count. Connection pooling is automatic; successive calls reuse connections.

### Directory-listing cache (off by default)

The S3 backend disables the underlying s3fs directory-listing cache by default, so every listing call reflects the current state of the bucket.

s3fs caches directory listings in a cache that **never expires**: once a prefix has been listed, s3fs serves later listings of that prefix from memory until the process restarts. For a single reader that never writes, that saves a round trip on repeated listings. But for any store shared by more than one writer — two processes, two `Store` instances, or another tool writing to the same bucket — a write made elsewhere is then **permanently invisible** to a reader that has already listed the prefix. Fresh listings cost one bounded round trip; the default favours correctness.

Re-enable the cache when you have a single-writer (or read-only) workload and want to avoid repeated listing round trips:

```
backend = S3Backend(
    bucket="my-bucket",
    client_options={"use_listings_cache": True},
)
```

`client_options["use_listings_cache"]` takes precedence over the default, so passing `True` restores the s3fs caching behaviour (and `False` is a harmless no-op). The [`ext.cache`](https://docs.remotestore.dev/stable/guides/cache/index.md) extension is the portable, backend-agnostic alternative when you want caching with explicit invalidation.

### Data Lake Pattern (Few Root Folders, Deep Nesting)

If you have few root-level folders (e.g., `/bronze`, `/silver`, `/gold`) with deeply nested structures:

**To explore the structure:**

```
# ✓ Shallow listing to see root tiers
for folder in store.list_folders(""):  # bronze/, silver/, gold/
    print(folder.name)
```

**To process a specific tier incrementally:**

```
# ✓ Depth-limited listing to explore one branch without full recursion
for file in store.list_files("bronze", max_depth=3):
    # Gets files up to 3 levels deep under bronze/
    process(file)
```

**Only if you need all files across all depths:**

```
# ✓ Full recursive listing (streaming, memory-efficient despite size)
for file in store.list_files("bronze", recursive=True):
    process(file)
```

**Characteristics:**

- `max_depth=0`: Direct children only (equivalent to non-recursive)
- `max_depth=1`: One level of nesting
- `max_depth=None` (default): Defers to `recursive` parameter (non-recursive by default)
- Cost on S3: Full recursive `ListObjectsV2` listing (O(n_total) API cost); client-side depth filter reduces the yielded result set. Local/SFTP/Memory backends prune natively.
- Streaming, memory-efficient (unlike loading entire tree)

**Use case:** Incremental exploration, tier-by-tier processing, or when you know the data structure depth in advance.

### Recommendations

- **Shallow listing (non-recursive):** Interactive UI, folder browsers, or when you only need direct children.
- **Depth-limited listing (max_depth=N):** Data lake patterns with known structure depth. Explore incrementally without full recursion.
- **Recursive listing (full tree):** Data processing, backups, or scanning entire prefix. Streaming operation, memory-efficient despite size.
- **Pattern matching:** Use `glob(pattern)` (internally uses flat stream with filtering) rather than custom folder traversal.
- **Note:** Do not parallelize any listing — single flat streams are already optimal.

## Capabilities

Supports all capabilities except `ATOMIC_MOVE`. See the [capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) for full details.

## Caveats

- **`move()` is not atomic.** S3 has no native rename operation. `move()` is implemented as copy + delete. If the process crashes between the two steps, both source and destination will exist.
- **`overwrite=False` has a TOCTOU race.** The exists-check and write are separate API calls. Concurrent writers can both pass the check and overwrite each other.

See the [Concurrency and Atomicity Guarantees](https://docs.remotestore.dev/stable/explanation/concurrency/index.md) guide for details and workarounds.

## See also

- [Capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md)
- [API reference](https://docs.remotestore.dev/stable/reference/api/store/index.md)
- [S3 backend example](https://docs.remotestore.dev/stable/tutorial/examples/s3-backend/index.md)
- [S3 listing strategies example](https://docs.remotestore.dev/stable/tutorial/examples/s3-listing-strategies/index.md) — demonstrates shallow, recursive, and filtering techniques
- [Performance guide](https://docs.remotestore.dev/stable/explanation/performance/index.md) — benchmark data and overhead analysis

## API Reference

## S3Backend

```
S3Backend(
    bucket: str,
    *,
    endpoint_url: str | None = None,
    key: str | Secret | None = None,
    secret: str | Secret | None = None,
    region_name: str | None = None,
    tls_ca_bundle: str | None = None,
    client_options: dict[str, Any] | None = None,
    retry: RetryPolicy | None = None,
    reject_write_under_file_ancestor: bool = False,
)
```

Bases: `_S3Base`

S3-compatible object storage backend using s3fs.

`move()` is implemented as a server-side copy followed by a delete. This is non-atomic: a crash or network error between the two steps may leave both source and destination present. `ATOMIC_MOVE` is not declared.

Parameters:

- **`bucket`** (`str`) – S3 bucket name (required, non-empty).
- **`endpoint_url`** (`str | None`, default: `None` ) – Custom endpoint URL (e.g. for MinIO).
- **`key`** (`str | Secret | None`, default: `None` ) – AWS access key ID.
- **`secret`** (`str | Secret | None`, default: `None` ) – AWS secret access key.
- **`region_name`** (`str | None`, default: `None` ) – AWS region name.
- **`tls_ca_bundle`** (`str | None`, default: `None` ) – Path to a PEM CA bundle file. Falls back to AWS_CA_BUNDLE / REQUESTS_CA_BUNDLE / SSL_CERT_FILE.
- **`client_options`** (`dict[str, Any] | None`, default: `None` ) – Additional options passed to s3fs.
- **`reject_write_under_file_ancestor`** (`bool`, default: `False` ) – If True, write / write_atomic / open_atomic / move / copy HEAD each slash-aligned ancestor of the target path and raise InvalidPath on the first regular-file hit, matching the cross-backend contract that hierarchical filesystems enforce natively. Default False: each nested-path write otherwise pays one HEAD per ancestor; paths without slashes short-circuit.

### check_health

```
check_health() -> None
```

Confirm the bucket is reachable and credentials valid via one `HeadBucket`.

Raises:

- `NotFound` – If the bucket does not exist.
- `PermissionDenied` – If the credentials are rejected or lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### exists

```
exists(path: str) -> bool
```

Return `True` if an object or prefix exists at *path*; never `NotFound`.

Raises:

- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### is_file

```
is_file(path: str) -> bool
```

Return `True` if *path* is an existing object (`False` if absent or a prefix).

Raises:

- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if *path* is an existing virtual folder (a common prefix).

Raises:

- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### read

```
read(path: str) -> BinaryIO
```

Open *path* for reading and return a streaming handle.

s3fs reads the object lazily in range-backed chunks, so memory stays constant regardless of size.

Raises:

- `NotFound` – If the object does not exist.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read and return the full object content as bytes.

Downloads the whole object into memory (unlike the lazy `read` stream).

Raises:

- `NotFound` – If the object does not exist.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* as an S3 object.

The upload commits atomically — a reader sees either the old object or the new one, never a partial. A streamed write that fails mid-body aborts the multipart upload rather than finalising a truncated object, so no partial object is ever left at *path*.

Raises:

- `AlreadyExists` – If the object exists and overwrite is False.
- `InvalidPath` – With the reject_write_under_file_ancestor opt-in, if an ancestor of path exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* atomically (delegates to `write`).

An S3 `PUT` is already atomic, so this is exactly `write`.

Raises:

- `AlreadyExists` – If the object exists and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of path exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Yield a writable buffer committed to *path* atomically on clean exit.

Writes spool to a temporary file (up to 8 MB in memory, then on disk); on clean exit the buffer is uploaded in a single atomic `PUT`. An exception before exit leaves *path* untouched.

Raises:

- `AlreadyExists` – If the object exists and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of path exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete the object at *path*.

Raises:

- `NotFound` – If the object does not exist and missing_ok is False.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete the virtual folder at *path*.

`recursive=True` removes every object under the prefix; this is a best-effort multi-object delete, not atomic, so an interruption can leave the prefix partially deleted. `recursive=False` removes the prefix only when it has no contents.

Raises:

- `NotFound` – If no object exists under path and missing_ok is False.
- `DirectoryNotEmpty` – If the prefix is non-empty and recursive is False.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return metadata for the object at *path* from one `HeadObject`.

The HEAD is issued with `ChecksumMode=ENABLED` so a stored checksum surfaces as the `FileInfo` digest.

Raises:

- `NotFound` – If the object does not exist.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename the object *src* to *dst*.

Implemented as a server-side copy followed by a delete of *src*. This is not atomic — a crash or network error between the two steps can leave both *src* and *dst* present — so `ATOMIC_MOVE` is not declared. `src == dst` is a no-op.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of dst exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy the object *src* to *dst* via a server-side `CopyObject`.

The bytes are copied entirely server-side (never through the client). Like `move`, the operation carries no cross-operation atomicity guarantee. `src == dst` is a no-op.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of dst exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

# SFTP Backend

The SFTP backend stores files on any SSH/SFTP server using [paramiko](https://www.paramiko.org/). Unlike fsspec's `SFTPFileSystem`, it gives you explicit control over host key verification and handles Azure Key Vault PEM quirks out of the box.

## Installation

```
pip install "remote-store[sftp]"
```

This pulls in `paramiko` and `tenacity` (for automatic retry on transient SSH errors).

## Usage

```
from remote_store import BackendConfig, RegistryConfig, Registry, StoreProfile

config = RegistryConfig(
    backends={
        "my-sftp": BackendConfig(
            type="sftp",
            options={
                "host": "files.example.com",
                "username": "deploy",
                "password": "secret",
                "base_path": "/srv/data",
            },
        ),
    },
    stores={"uploads": StoreProfile(backend="my-sftp", root_path="uploads")},
)

with Registry(config) as registry:
    store = registry.get_store("uploads")
    store.write("report.csv", b"col1,col2\n1,2\n")
    data = store.read_bytes("report.csv")
```

### Key-based authentication

```
from remote_store.backends import SFTPBackend, SFTPUtils

pkey = SFTPUtils.load_private_key("/path/to/id_rsa", from_file=True)

backend = SFTPBackend(
    host="files.example.com",
    username="deploy",
    pkey=pkey,
)
```

Or load a PEM string directly (useful for secrets managers like Azure Key Vault):

```
pkey = SFTPUtils.load_private_key(pem_string)
```

## Options

| Option            | Type            | Default              | Description                                  |
| ----------------- | --------------- | -------------------- | -------------------------------------------- |
| `host`            | `str`           | *(required)*         | SFTP server hostname                         |
| `port`            | `int`           | `22`                 | SSH port                                     |
| `username`        | `str`           | `None`               | SSH username                                 |
| `password`        | `str`           | `None`               | SSH password                                 |
| `pkey`            | `paramiko.PKey` | `None`               | Private key for key-based auth               |
| `base_path`       | `str`           | `"/"`                | Root path on the remote server               |
| `host_key_policy` | `HostKeyPolicy` | `STRICT`             | Host key verification mode (see below)       |
| `known_host_keys` | `str`           | `None`               | Known-hosts string (code-level override)     |
| `host_keys_path`  | `str`           | `~/.ssh/known_hosts` | Path to known_hosts file                     |
| `config`          | `dict`          | `None`               | Config dict (may contain `known_host_keys`)  |
| `timeout`         | `int`           | `10`                 | SSH connection timeout in seconds            |
| `connect_kwargs`  | `dict`          | `None`               | Extra kwargs passed to `SSHClient.connect()` |

## Preflight host-key discovery

To populate a committed `host.keys` file without going through a TOFU connect first, use [`SFTPUtils.scan_host_keys(host, port=22)`](https://docs.remotestore.dev/stable/reference/api/sftp-utils/index.md). It opens a transport, captures the server's *negotiated* host key (no authentication), and returns a single `known_hosts`-formatted line ready to commit:

```
from pathlib import Path
from remote_store.backends import SFTPUtils

entry = SFTPUtils.scan_host_keys("sftp.example.com")
Path("host.keys").write_text(entry + "\n")
```

For non-default ports the entry uses the OpenSSH `[host]:port` form. Network failures (host unreachable, port refused, DNS error) raise `OSError`; KEX failures (legacy server offering only `ssh-rsa`) raise `paramiko.SSHException` — call `enable_ssh_rsa_compat()` first in that case.

`scan_host_keys()` returns the **negotiated** key for one handshake, not every key type the server offers. If the server publishes multiple key types and paramiko later negotiates a type other than the pinned line, the connection fails with `BadHostKeyException`. Call the helper multiple times under different `disabled_algorithms` settings if you need full-type coverage.

## Host Key Verification

The `HostKeyPolicy` enum controls how unknown host keys are handled:

| Policy               | Behaviour                                         | Use case                |
| -------------------- | ------------------------------------------------- | ----------------------- |
| `STRICT`             | Reject unknown hosts. Key must be in known_hosts. | Production (default)    |
| `TRUST_ON_FIRST_USE` | Accept and save on first connect, verify after.   | First-time server setup |
| `AUTO_ADD`           | Accept any key silently.                          | Dev / testing only      |

Known host keys are resolved in order (first match wins):

1. `known_host_keys` constructor parameter
1. `config["known_host_keys"]` dict value
1. `SFTP_KNOWN_HOST_KEYS` environment variable
1. `host_keys_path` file on disk (default: `~/.ssh/known_hosts`)

```
from remote_store.backends import SFTPBackend, SFTPUtils

# Development / testing
backend = SFTPBackend(
    host="localhost",
    port=2222,
    username="test",
    password="test",
    host_key_policy=SFTPUtils.HostKeyPolicy.AUTO_ADD,
)
```

## Legacy Servers (`ssh-rsa` / SHA-1)

**What changed.** Paramiko 5.0 removed `ssh-rsa` from its host-key defaults — empirically verified, see the [research note](https://github.com/haalfi/remote-store/blob/master/sdd/research/research-bk-198-paramiko-ssh-rsa-empirical.md) for the version matrix.

- **paramiko `< 5`** ships `ssh-rsa` in defaults at all four negotiation sites. A freshly-imported paramiko already negotiates against an `ssh-rsa`-only server out of the box.
- **paramiko `>= 5`** has `ssh-rsa` removed from all four sites. Connecting to an `ssh-rsa`-only server raises `IncompatiblePeer: Incompatible ssh peer (no acceptable host key)` during KEX, before authentication is attempted.

The `[sftp]` extra has no upper bound on paramiko, so current resolvers pick paramiko 5+ by default. New installs hit the failure unless they call the helper described below.

### Diagnose first

Before mutating paramiko's defaults, confirm the failure shape. An `IncompatiblePeer` error from paramiko wraps four distinct negotiation failures — host key, KEX, cipher, or MAC — and only the first is fixed by `enable_ssh_rsa_compat()`. The other three need `connect_kwargs={"disabled_algorithms": ...}` instead. [`SFTPUtils.scan_host_algorithms()`](https://docs.remotestore.dev/stable/reference/api/sftp-utils/#remote_store.backends.SFTPUtils.scan_host_algorithms) parses the server's `SSH_MSG_KEXINIT` advertisement (RFC 4253 § 7.1) over a raw socket — no paramiko, no authentication, so the result reflects exactly what the server advertises:

```
from remote_store.backends import SFTPUtils

info = SFTPUtils.scan_host_algorithms("legacy.example.com")
print("host-key algos:", info["server_host_key_algorithms"])
print("kex algos:     ", info["kex_algorithms"])
```

If `server_host_key_algorithms == ["ssh-rsa"]`, this guide applies and the next subsection is the fix. If it's `kex_algorithms` that's narrow (e.g. only `diffie-hellman-group14-sha1`), `enable_ssh_rsa_compat()` will not help; widen the relevant list via `SFTPBackend(connect_kwargs={"disabled_algorithms": ...})`.

### Fix: re-enable `ssh-rsa` at process startup

[`SFTPUtils.enable_ssh_rsa_compat()`](https://docs.remotestore.dev/stable/reference/api/sftp-utils/index.md) adds `ssh-rsa` to all four paramiko host-key sites in one call. It is a no-op on paramiko `< 5` (all four guards short-circuit) and the required recovery path on paramiko `>= 5`:

```
from remote_store.backends import SFTPUtils

# Call once, before any SFTPBackend connect to a legacy server.
SFTPUtils.enable_ssh_rsa_compat()
```

If you observe `IncompatiblePeer: no acceptable kex algorithm`

KEX / cipher / MAC negotiation failures are a separate problem; `enable_ssh_rsa_compat()` does not help. Widen the relevant algorithm list via the `connect_kwargs={"disabled_algorithms": ...}` SFTP constructor argument instead.

Security tradeoff

This is **process-global**: every paramiko transport in the process will then accept SHA-1 host keys. Only enable this if every server your process connects to is under your operational control, and push server operators to upgrade to `rsa-sha2-256`/`rsa-sha2-512` so the shim can be removed.

### Alternative: pin `paramiko<5`

Pinning `paramiko<5` keeps the consumer on the empirically-verified compatible range (`>= 3.0,< 5`) and avoids the helper entirely. The tradeoff is freezing on paramiko 4.x while upstream moves on:

| Approach                  | Loses                                                                              |
| ------------------------- | ---------------------------------------------------------------------------------- |
| `paramiko<5` pin          | Future paramiko 5+ improvements (perf, protocol features, CVE fixes once 4.x EOLs) |
| `enable_ssh_rsa_compat()` | Process-wide SHA-1 host-key acceptance only                                        |

Either composes cleanly with the library's `[sftp]` floor of `paramiko>=3.0`. To pin the consumer must override at their own dependency layer (e.g. `requirements.txt` line `paramiko>=3.0,<5`).

## Connection Behaviour

- **Lazy connect** — no network call happens during construction. The SSH/SFTP connection is established on the first operation.
- **Auto-reconnect** — if the connection goes stale between operations, the backend reconnects transparently.
- **Retry** — transient SSH errors (`SSHException`, `OSError`, `EOFError`) are retried up to 3 times with exponential backoff (2 s min, 10 s max).
- **Single connection, not thread-safe** — each `SFTPBackend` instance holds one paramiko `SFTPClient`. Calling it from multiple threads simultaneously (e.g. via `SyncBackendAdapter` + `asyncio.gather`) races on the shared socket. Create one `SFTPBackend` per thread for parallel workloads.

## Capabilities

The SFTP backend supports all capabilities except `GLOB` and `ATOMIC_MOVE`. See the [capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) for full details.

Atomic write caveat

Atomic writes use a temp file (`.~tmp.<name>.<uuid>`) and rename. If the connection drops between write and rename, the orphan temp file will remain on the server.

Move fallback

`move()` tries `posix_rename` (atomic), then standard `rename()`, then copy + delete as a last resort. Most servers support at least `rename()`.

TOCTOU on `overwrite=False`

Like most backends, the exists-check and write are separate operations. Concurrent writers can both pass the check.

See the [Concurrency and Atomicity Guarantees](https://docs.remotestore.dev/stable/explanation/concurrency/index.md) guide for details and workarounds.

## Escape Hatch

Access the underlying `paramiko.SFTPClient` when you need protocol-level features:

```
import paramiko

sftp_client = backend.unwrap(paramiko.SFTPClient)
sftp_client.listdir_attr("/custom/path")
```

## See also

- [Capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md)
- [API reference](https://docs.remotestore.dev/stable/reference/api/store/index.md)
- [SFTP utilities reference](https://docs.remotestore.dev/stable/reference/api/sftp-utils/index.md) — `scan_host_keys`, `enable_ssh_rsa_compat`, `HostKeyPolicy`
- [Example script](https://docs.remotestore.dev/stable/tutorial/examples/sftp-backend/index.md)

## API Reference

## SFTPBackend

```
SFTPBackend(
    host: str,
    *,
    port: int = 22,
    username: str | None = None,
    password: str | Secret | None = None,
    pkey: Any = None,
    base_path: str = "/",
    host_key_policy: HostKeyPolicy | str = STRICT,
    known_host_keys: str | None = None,
    host_keys_path: str | None = None,
    config: dict[str, Any] | None = None,
    timeout: int = 10,
    connect_kwargs: dict[str, Any] | None = None,
    retry: RetryPolicy | None = None,
)
```

Bases: `Backend`

SFTP backend using pure paramiko.

`move()` attempts `posix_rename` (atomic on POSIX-compliant servers), then falls back to `rename`, and finally to a stream copy followed by a delete. Because atomicity cannot be guaranteed across all servers, `ATOMIC_MOVE` is not declared.

Warning

**Not thread-safe for concurrent access.** This backend maintains a single SSH/SFTP connection (paramiko `SFTPClient`), which is not safe to call from multiple threads simultaneously. Concurrent calls via `SyncBackendAdapter` and `asyncio.gather` will race on the shared socket and may hang or corrupt responses. Create one `SFTPBackend` instance per thread if you need parallel operations.

Parameters:

- **`host`** (`str`) – SFTP server hostname (required, non-empty).
- **`port`** (`int`, default: `22` ) – SSH port (default: 22).
- **`username`** (`str | None`, default: `None` ) – SSH username.
- **`password`** (`str | Secret | None`, default: `None` ) – SSH password.
- **`pkey`** (`Any`, default: `None` ) – paramiko.PKey instance for key-based auth.
- **`base_path`** (`str`, default: `'/'` ) – Root path on the remote server (default: /).
- **`host_key_policy`** (`HostKeyPolicy | str`, default: `STRICT` ) – Host key verification policy (see SFTPUtils.HostKeyPolicy). Accepts enum value or string.
- **`known_host_keys`** (`str | None`, default: `None` ) – Known hosts string (code-level override).
- **`host_keys_path`** (`str | None`, default: `None` ) – Path to known_hosts file (default: ~/.ssh/known_hosts).
- **`config`** (`dict[str, Any] | None`, default: `None` ) – Optional config dict (may contain known_host_keys).
- **`timeout`** (`int`, default: `10` ) – SSH connection timeout in seconds.
- **`connect_kwargs`** (`dict[str, Any] | None`, default: `None` ) – Extra kwargs passed to SSHClient.connect().

### check_health

```
check_health() -> None
```

Confirm the SFTP connection works by `stat`-ing the base path.

Establishes the SSH/SFTP connection lazily if needed (retried at connection scope) and issues one `stat` round-trip.

Raises:

- `NotFound` – If the configured base path does not exist.
- `PermissionDenied` – If the server denies access to the base path.
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established.

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` with SFTP-specific details.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – Plan with kind="sftp" and details containing
- `ResolutionPlan` – host, port, and base_path.

### exists

```
exists(path: str) -> bool
```

Return `True` if a file or folder exists at *path*; never `NotFound`.

Issues one `stat` round-trip; a missing path returns `False`.

Raises:

- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if *path* is an existing regular file (`False` if absent or a folder).

Raises:

- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if *path* is an existing directory (`False` if absent or a file).

Raises:

- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### read

```
read(path: str) -> BinaryIO
```

Open *path* for reading and return a buffered, streaming handle.

Reads lazily over the SFTP channel (wrapped in a `BufferedReader`), so memory stays constant regardless of file size; one extra `stat` round-trip guards against *path* being a directory before the open.

Raises:

- `NotFound` – If the file does not exist, or a path component is itself a file.
- `InvalidPath` – If path names a directory.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails mid-read.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read and return the full file content as bytes.

Prefetches and materialises the whole file in memory (unlike the lazy `read` stream).

Raises:

- `NotFound` – If the file does not exist, or a path component is itself a file.
- `InvalidPath` – If path names a directory.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails mid-read.

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path*, streaming it over the SFTP channel.

The bytes are streamed straight to the destination file (no temp-and-rename), so a dropped connection mid-write can leave a partial or truncated file there — use `write_atomic` when readers must never see a half-written file. Missing parent directories are created first (one `stat` per ancestor).

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor of path exists as a regular file.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails mid-write.

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* atomically via a temp file plus server rename.

Readers never observe a partial file: the body is streamed to a hidden temp file in the destination directory, then promoted with `posix_rename` (atomic on POSIX-compliant servers). Servers without `posix_rename` fall back to a plain `rename` (non-atomic overwrite: the target is removed first), and the temp file is cleaned up on failure.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor of path exists as a regular file.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Yield a writable handle promoted to *path* atomically on clean exit.

Writes stream to a hidden temp file in the destination directory; on clean exit it is promoted with `posix_rename` (atomic on POSIX servers, falling back to `rename`), and on any exception the temp file is removed and *path* is left untouched.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor of path exists as a regular file.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete the file at *path*.

Raises:

- `NotFound` – If the file does not exist (or a path component is itself a file) and missing_ok is False.
- `InvalidPath` – If path names a directory (use delete_folder).
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete the folder at *path*.

`recursive=True` walks and removes the subtree bottom-up (one round-trip per entry — not atomic; an interruption can leave the tree partially removed). `recursive=False` removes only an empty folder after checking it has no entries.

Raises:

- `NotFound` – If the folder does not exist and missing_ok is False.
- `InvalidPath` – If path names a file, not a folder.
- `DirectoryNotEmpty` – If the folder is non-empty and recursive is False.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> Iterator[FileInfo]
```

Yield files under *path*, one `FileInfo` at a time.

Lazily walks the remote directory (`listdir_attr`); a missing *path* yields nothing. `recursive` descends via one directory-listing round-trip per folder (`max_depth` bounds the descent). Failures other than a missing path surface as `RemoteStoreError` during iteration.

### list_folders

```
list_folders(path: str) -> Iterator[FolderEntry]
```

Yield immediate subfolders of *path* as `FolderEntry` records.

One directory-listing round-trip; a missing *path* yields nothing, and other failures surface as `RemoteStoreError` during iteration.

### iter_children

```
iter_children(
    path: str,
) -> Iterator[FileInfo | FolderEntry]
```

Yield the immediate files and folders under *path* in one listing.

Overrides the base two-pass default with a single `listdir_attr` round-trip, yielding `FileInfo` for files and `FolderEntry` for folders. A missing *path* yields nothing; other failures surface as `RemoteStoreError` during iteration.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return metadata for the file at *path* from a single `stat` round-trip.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path names a directory, not a file.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Return aggregate metadata for the folder at *path*.

File count, total size, and latest modification time are gathered by recursively walking the whole subtree (one listing round-trip per folder), so cost scales with the number of descendants.

Raises:

- `NotFound` – If the folder does not exist.
- `InvalidPath` – If path names a file, not a folder.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename the file *src* to *dst*.

Tries `posix_rename` first (atomic on POSIX-compliant servers), then a plain `rename`, and finally a stream copy-then-delete. Because the outcome depends on server support, atomicity is not guaranteed across all servers and `ATOMIC_MOVE` is not declared. `src == dst` is a no-op; missing parent directories of *dst* are created first.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory, or an ancestor of dst exists as a regular file.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy the file *src* to *dst* by streaming through the client.

SFTP has no server-side copy, so the bytes round-trip through the client (download then upload); this is not atomic — an interruption can leave a partial file at *dst*. `src == dst` is a no-op; missing parent directories of *dst* are created first.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory, or an ancestor of dst exists as a regular file.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

# SQL Blob Backend

The SQL blob backend stores files as key-value rows in any SQLAlchemy-supported database. Each row holds one file with its key, binary data, and metadata.

**Primary use cases:** zero-infrastructure persistent store (SQLite), shared database-backed file storage (PostgreSQL, MySQL), embedded metadata+blob co-location, portable single-file archives.

## Installation

```
pip install remote-store[sql]
```

Requires `sqlalchemy>=2.0`.

## Usage

### SQLite (simplest — zero infrastructure)

```
from remote_store import Store
from remote_store.backends import SQLBlobBackend

backend = SQLBlobBackend(url="sqlite:///store.db")
store = Store(backend=backend)

store.write("models/v3.pkl", model_bytes)
data = store.read_bytes("models/v3.pkl")
```

### PostgreSQL

```
backend = SQLBlobBackend(url="postgresql://user:pass@localhost/mydb")
store = Store(backend=backend)
```

### Shared engine (web apps)

```
from sqlalchemy import create_engine

# App's connection pool — shared across the application
engine = create_engine("postgresql://...", pool_size=10)

# Backend borrows the engine; close() is a no-op
backend = SQLBlobBackend(engine=engine)
store = Store(backend=backend)
```

### Via Registry

```
from remote_store import BackendConfig, RegistryConfig, Registry, StoreProfile

config = RegistryConfig(
    backends={"db": BackendConfig(type="sql-blob", options={"url": "sqlite:///store.db"})},
    stores={"files": StoreProfile(backend="db", root_path="data")},
)

with Registry(config) as registry:
    store = registry.get_store("files")
    store.write("readme.txt", b"Hello!")
```

## Options

| Option          | Type             | Default                  | Description                                                           |
| --------------- | ---------------- | ------------------------ | --------------------------------------------------------------------- |
| `url`           | `str \| None`    | `None`                   | SQLAlchemy database URL. Mutually exclusive with `engine`.            |
| `engine`        | `Engine \| None` | `None`                   | Pre-built SQLAlchemy engine. Mutually exclusive with `url`.           |
| `table_name`    | `str`            | `"remote_store_objects"` | Name of the storage table.                                            |
| `create_table`  | `bool`           | `True`                   | Auto-create the table if it doesn't exist.                            |
| `max_blob_size` | `int \| None`    | `None`                   | Maximum blob size in bytes. Raises `ValueError` on write if exceeded. |

Exactly one of `url` or `engine` must be provided.

## Schema

The default table schema:

```
CREATE TABLE remote_store_objects (
    key           TEXT    PRIMARY KEY,
    data          BLOB    NOT NULL,
    size          INTEGER NOT NULL,
    modified_at   REAL    NOT NULL,
    content_type  TEXT,
    digest        TEXT,
    extra         TEXT,
    user_metadata TEXT
);
```

### User metadata column

When the `user_metadata` column is present, `SQLBlobBackend` declares the `WRITE_RESULT_NATIVE` and `USER_METADATA` capabilities. When absent (legacy tables), neither is declared — a `Store.write*()` call with a non-empty `metadata=` kwarg raises `CapabilityNotSupported` before any I/O runs.

To add the column to an existing table:

```
-- hand-written: schema migrations cannot run in CI
ALTER TABLE remote_store_objects ADD COLUMN user_metadata TEXT;
```

### Using an existing table

Set `create_table=False` to use a pre-existing table. Minimum required columns: `key TEXT` (primary key) and `data BLOB`. Optional columns (`size`, `modified_at`, `content_type`, `digest`, `extra`, `user_metadata`) are detected automatically; missing ones degrade gracefully.

## Capabilities

Supports all capabilities except `LAZY_READ` — the entire blob is loaded into memory before a stream is returned. See the [capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) for full details.

### Implementation notes

- **Non-lazy writes.** `write()` materializes the full stream into memory before issuing the SQL INSERT/UPDATE. This is inherent to SQL BLOB columns, which require complete data in a single statement. For files larger than process memory, use a blob-storage backend (S3, Local, Azure) instead.
- `write_atomic()` delegates to `write()` — single SQL statements are inherently atomic.
- `glob()` uses SQL-side narrowing (SQLite `GLOB` or `LIKE`) then client-side regex to enforce standard glob semantics.

## SQLite Optimizations

When using SQLite, the backend automatically:

- Enables **WAL mode** (`PRAGMA journal_mode=WAL`) for better concurrent read performance.
- Sets **`PRAGMA synchronous=NORMAL`** for improved write throughput.

These are set on every new connection via SQLAlchemy event listeners. If you pass a shared engine that already has `pool_events.connect` listeners, they will coexist — the backend guards against duplicate registration.

## Folder Semantics

Folders are virtual (prefix-based), not explicit nodes:

- `is_folder("data")` returns `True` if any key starts with `data/`.
- `list_folders("data")` extracts unique first-level subfolder names from stored keys.
- `delete_folder("data", recursive=True)` deletes all keys starting with `data/`.

## Performance Guidelines

| Blob size   | Recommendation                                 |
| ----------- | ---------------------------------------------- |
| < 10 MB     | Works well across all databases                |
| 10 - 100 MB | Use with caution; set `max_blob_size`          |
| > 100 MB    | Use a blob storage backend (S3, Local) instead |

## Engine Lifecycle

- **Owned engine** (`url` provided): `close()` calls `engine.dispose()`.
- **Borrowed engine** (`engine` provided): `close()` is a no-op.
- **`unwrap(Engine)`**: Returns the underlying SQLAlchemy `Engine`.
- **`check_health()`**: Executes `SELECT 1` to verify connectivity.

## See also

- [Example script](https://docs.remotestore.dev/stable/tutorial/examples/sql-blob-backend/index.md)
- [Capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md)
- [API reference](https://docs.remotestore.dev/stable/reference/api/store/index.md)

## API Reference

## SQLBlobBackend

```
SQLBlobBackend(
    url: str | None = None,
    *,
    engine: Engine | None = None,
    table_name: str = "remote_store_objects",
    create_table: bool = True,
    max_blob_size: int | None = None,
    reject_write_under_file_ancestor: bool = False,
)
```

Bases: `_SQLAlchemyBaseBackend`

SQL key-value blob store implementing the full Backend contract.

Uses a SQL table as key-value storage. Each row holds one "file" with its key, data, and metadata. SQLite receives WAL mode and PRAGMA tuning automatically.

Supports all capabilities except `LAZY_READ`.

Every mutating operation runs inside a single database transaction, so `write`, `write_atomic`, `move`, and `copy` are atomic — a failure rolls back with no partial row left behind (`ATOMIC_MOVE` is advertised).

Note

**Non-lazy reads and writes.** Both `read()` and `write()` materialize the full content in memory. `read()` loads the entire BLOB before returning a stream (no `LAZY_READ`). `write()` reads the full stream before issuing the SQL INSERT/UPDATE because BLOB columns require complete data in a single statement. For files larger than process memory, use a blob-storage backend (S3, Local, Azure) instead.

Parameters:

- **`reject_write_under_file_ancestor`** (`bool`, default: `False` ) – If True, write / write_atomic / open_atomic / move / copy issue one SELECT 1 per slash-aligned ancestor of the target path and raise InvalidPath on the first regular-file hit, matching the cross-backend contract that hierarchical filesystems enforce natively. Default False; paths without slashes short-circuit.

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` with SQL blob details.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – Plan with kind="sql-blob" and details containing
- `ResolutionPlan` – table_name.

### exists

```
exists(path: str) -> bool
```

Return `True` if a key or key-prefix exists at *path*; never `NotFound`.

Folders are virtual — *path* counts as a folder when any key begins with `path + "/"`. The root (`""`) always exists. Costs one or two `SELECT`s.

Raises:

- `InvalidPath` – If path is absolute or contains a .. segment.
- `BackendUnavailable` – If the database is unreachable.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if an exact key exists at *path* (one `SELECT`).

Raises:

- `InvalidPath` – If path is absolute or contains a .. segment.
- `BackendUnavailable` – If the database is unreachable.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if any key begins with `path + "/"` (a virtual folder).

The root is always a folder. Costs one `SELECT`.

Raises:

- `InvalidPath` – If path is absolute or contains a .. segment.
- `BackendUnavailable` – If the database is unreachable.

### read

```
read(path: str) -> BinaryIO
```

Return a binary stream over the stored BLOB for *path*.

Loads the entire BLOB into memory before returning the stream — the read does not stream (`LAZY_READ` is not advertised), so peak memory scales with the object size. For objects larger than process memory use a blob-storage backend (S3, Local, Azure).

Raises:

- `NotFound` – If no key exists at path.
- `InvalidPath` – If path is empty, absolute, or malformed.
- `BackendUnavailable` – If the database is unreachable.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Return the full stored BLOB for *path* as bytes.

Like `read`, materialises the whole object in memory.

Raises:

- `NotFound` – If no key exists at path.
- `InvalidPath` – If path is empty, absolute, or malformed.
- `BackendUnavailable` – If the database is unreachable.

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Store *content* at *path* in a single, atomic transaction.

The whole body is buffered in memory before the `INSERT`/`UPDATE` (BLOB columns need the complete value in one statement — no streaming write), and the row is written inside one transaction, so a failure rolls back with no partial row left behind.

Raises:

- `AlreadyExists` – If a key exists at path and overwrite is False.
- `InvalidPath` – If path is empty/malformed, or (with the reject_write_under_file_ancestor opt-in) an ancestor key exists as a file.
- `BackendUnavailable` – If the database operation fails.

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Store *content* at *path* atomically (delegates to `write`).

SQL writes are already transactional, so this is exactly `write`; the whole body is buffered first.

Raises:

- `AlreadyExists` – If a key exists at path and overwrite is False.
- `InvalidPath` – If path is empty/malformed, or (opt-in) an ancestor key exists as a file.
- `BackendUnavailable` – If the database operation fails.

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Yield an in-memory buffer committed to *path* atomically on clean exit.

Writes accumulate in a `BytesIO`; on exit the buffer is stored via `write` in one transaction. An exception before exit leaves *path* untouched.

Raises:

- `AlreadyExists` – If a key exists at path and overwrite is False.
- `InvalidPath` – If path is empty/malformed, or (opt-in) an ancestor key exists as a file.
- `BackendUnavailable` – If the database operation fails.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete the row at *path* in one transaction.

Raises:

- `NotFound` – If no key exists at path and missing_ok is False.
- `InvalidPath` – If path is empty, absolute, or malformed.
- `BackendUnavailable` – If the database operation fails.

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete every key under the virtual folder *path*.

Folders are key prefixes, not stored rows, so a folder "exists" only when it has children: a non-recursive call on an existing folder therefore always raises `DirectoryNotEmpty`. `recursive=True` deletes all keys under `path + "/"` in one atomic transaction.

Raises:

- `NotFound` – If no key exists under path and missing_ok is False.
- `DirectoryNotEmpty` – If recursive is False (an existing virtual folder is never empty).
- `InvalidPath` – If path is empty, absolute, or malformed.
- `BackendUnavailable` – If the database operation fails.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> Iterator[FileInfo]
```

Yield files under *path*.

One `SELECT` fetches every key under the prefix; folder structure is derived from `/` in the key suffix, and `recursive` / `max_depth` filter client-side. A missing prefix yields nothing.

Raises:

- `BackendUnavailable` – If the database operation fails, surfaced during iteration.

### list_folders

```
list_folders(path: str) -> Iterator[FolderEntry]
```

Yield immediate virtual subfolders of *path* as `FolderEntry` records.

One `SELECT` over keys under the prefix; folder names are the distinct first segments of the key suffixes. A missing prefix yields nothing.

Raises:

- `BackendUnavailable` – If the database operation fails, surfaced during iteration.

### iter_children

```
iter_children(
    path: str,
) -> Iterator[FileInfo | FolderEntry]
```

Yield the immediate files and virtual folders under *path* in one `SELECT`.

Overrides the base two-pass default: a single query over the prefix yields `FileInfo` for direct-child keys and `FolderEntry` for the distinct first suffix segments. A missing prefix yields nothing.

Raises:

- `BackendUnavailable` – If the database operation fails, surfaced during iteration.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return metadata for the file at *path* from one `SELECT`.

Raises:

- `NotFound` – If no key exists at path.
- `InvalidPath` – If path is empty, absolute, or malformed.
- `BackendUnavailable` – If the database is unreachable.

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Return aggregate metadata for the virtual folder *path*.

File count, total size, and latest modification time come from one aggregate `SELECT` (`COUNT`/`SUM`/`MAX`) over keys under the prefix — no per-file round-trips.

Raises:

- `NotFound` – If no key exists under path.
- `InvalidPath` – If path is absolute or malformed.
- `BackendUnavailable` – If the database is unreachable.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move (rename) the key *src* to *dst* in one atomic transaction.

Implemented as an `UPDATE` of the row's key — the BLOB is never transferred through Python. The whole operation (source check, optional destination replace, rename) runs in one transaction, so a failure rolls back cleanly. `src == dst` verifies the source exists and is otherwise a no-op.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists and overwrite is False.
- `InvalidPath` – If src or dst is malformed, or (opt-in) an ancestor of dst exists as a file.
- `BackendUnavailable` – If the database operation fails.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy the key *src* to *dst* in one atomic transaction.

Implemented as a single `INSERT ... SELECT`, so the BLOB is duplicated entirely inside the database — no bytes pass through Python. `src == dst` verifies the source exists and is otherwise a no-op.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists and overwrite is False.
- `InvalidPath` – If src or dst is malformed, or (opt-in) an ancestor of dst exists as a file.
- `BackendUnavailable` – If the database operation fails.

### glob

```
glob(pattern: str) -> Iterator[FileInfo]
```

Yield files whose key matches the glob *pattern*.

Narrows SQL-side with a prefix `LIKE` where the pattern allows (on every dialect — SQLite's native `GLOB` is deliberately avoided because it mishandles `**`), then applies the full glob regex to each row. Costs one `SELECT`.

Raises:

- `BackendUnavailable` – If the database operation fails, surfaced during iteration.

# SQL Query Backend

The SQL query backend is a read-only materializer that maps path keys to SQL queries. On `read()`, it executes the query and serializes the result set to Parquet, CSV, or Arrow IPC based on the key's file extension.

**Primary use cases:** exposing SQL query results as portable data products, on-demand materialization of views and reports, bridging SQL databases into the Store abstraction for consumers who don't need to know the source is a database.

## Installation

```
pip install remote-store[sql-query]
```

Requires `sqlalchemy>=2.0` and `pyarrow>=12.0.0`.

## Usage

```
from remote_store import Store
from remote_store.backends import SQLQueryBackend

backend = SQLQueryBackend(
    url="sqlite:///analytics.db",
    queries={
        "reports/daily_sales.parquet": "SELECT date, SUM(amount) FROM orders GROUP BY date",
        "reports/user_summary.csv": "SELECT * FROM user_summary_mv",
    },
)
store = Store(backend=backend)

# Read executes the query and serializes to the format implied by the extension
data = store.read_bytes("reports/daily_sales.parquet")  # Parquet bytes
```

## Key-to-query resolution

Keys are resolved by exact lookup in the `queries` dict. The serialization format is determined by the file extension:

| Extension        | Format         |
| ---------------- | -------------- |
| `.parquet`       | Apache Parquet |
| `.csv`           | CSV            |
| `.arrow`, `.ipc` | Arrow IPC      |

## Strict mode

`strict=True` (default) restricts resolution to the explicit `queries` dict only. View-based and convention-based discovery (`strict=False`) are planned for a future release.

## Capabilities

Read-only: supports `READ`, `LIST`, `METADATA`, `GLOB`, and `SEEKABLE_READ`. All write operations raise `CapabilityNotSupported`. See the [capabilities matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) for full details.

## Custom serializers

The `ResultSerializer` protocol allows custom serialization:

```
from remote_store.backends import ArrowSerializer, SQLQueryBackend

# Use the built-in ArrowSerializer (default)
backend = SQLQueryBackend(url="sqlite:///db.sqlite", queries={...})

# Or provide a custom serializer
backend = SQLQueryBackend(url="sqlite:///db.sqlite", queries={...}, serializer=my_serializer)
```

## Engine lifecycle

Same as SQL Blob backend --- accepts either a URL string (creates and owns the engine) or a pre-existing `sqlalchemy.Engine` (borrows it). See [SQL Blob Backend](https://docs.remotestore.dev/stable/guides/backends/sql-blob/#engine-lifecycle) for details.

## API Reference

## SQLQueryBackend

```
SQLQueryBackend(
    url: str | None = None,
    *,
    engine: Engine | None = None,
    queries: dict[str, str] | None = None,
    strict: bool = True,
    serializer: ResultSerializer | None = None,
)
```

Bases: `_SQLAlchemyBaseBackend`

Read-only SQL query materializer implementing a subset of the Backend contract.

Maps path keys to SQL queries. On `read()`, executes the query and serializes the result set to the format implied by the key's file extension (Parquet, CSV, or Arrow IPC).

Capabilities: `READ`, `LIST`, `METADATA`, `GLOB`, `SEEKABLE_READ`.

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` with SQL query details.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – Plan with kind="sql-query" and details containing
- `ResolutionPlan` – source and format.
# Reference

# API Reference

Complete reference for all public exports of `remote-store`.

Symbols below are grouped by role. The import path is `remote_store` unless a group says otherwise: storage backends live in `remote_store.backends`, extensions in `remote_store.ext`, and the entire async surface in `remote_store.aio` (see the [Async section](#async), whose layout mirrors this page).

Feeding this to a coding agent?

[`llms-api.txt`](https://docs.remotestore.dev/stable/llms-api.txt) carries this entire surface as one code-shaped skeleton — every signature, type annotation, and docstring (backends, async, and extensions included), with bodies elided — in a single file for an agent's context.

## Core

| Class                                                                           | Description                                      |
| ------------------------------------------------------------------------------- | ------------------------------------------------ |
| [Store](https://docs.remotestore.dev/stable/reference/api/store/index.md)       | Main entry point for all file operations         |
| [ProxyStore](https://docs.remotestore.dev/stable/reference/api/proxy/index.md)  | Base class for building Store middleware         |
| [Registry](https://docs.remotestore.dev/stable/reference/api/registry/index.md) | Creates and manages backend instances and stores |
| [Backend](https://docs.remotestore.dev/stable/reference/api/backend/index.md)   | Abstract base class for storage backends         |

## Backends

Synchronous storage backends (`remote_store.backends`). Native async backends — including the async-only Microsoft Graph backend — live under [Async › Backends](https://docs.remotestore.dev/stable/reference/api/aio/backends/index.md).

| Class                                                                                              | Description                                                  |
| -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| [LocalBackend](https://docs.remotestore.dev/stable/reference/api/backends/local/index.md)          | Local filesystem storage                                     |
| [MemoryBackend](https://docs.remotestore.dev/stable/reference/api/backends/memory/index.md)        | In-process storage for testing                               |
| [ReadOnlyHttpBackend](https://docs.remotestore.dev/stable/reference/api/backends/http/index.md)    | Read-only access to HTTP/HTTPS URLs                          |
| [S3Backend](https://docs.remotestore.dev/stable/reference/api/backends/s3/index.md)                | Amazon S3 and S3-compatible services                         |
| [S3PyArrowBackend](https://docs.remotestore.dev/stable/reference/api/backends/s3-pyarrow/index.md) | S3 via PyArrow C++ for higher throughput                     |
| [SFTPBackend](https://docs.remotestore.dev/stable/reference/api/backends/sftp/index.md)            | SSH/SFTP server storage via paramiko                         |
| [AzureBackend](https://docs.remotestore.dev/stable/reference/api/backends/azure/index.md)          | Azure Blob Storage and ADLS Gen2                             |
| [SQLBlobBackend](https://docs.remotestore.dev/stable/reference/api/backends/sql-blob/index.md)     | SQL database blob storage via SQLAlchemy                     |
| [SQLQueryBackend](https://docs.remotestore.dev/stable/reference/api/backends/sql-query/index.md)   | Read-only SQL query materialization via SQLAlchemy + PyArrow |

## Async

The `remote_store.aio` namespace — the async counterpart of the core API, laid out to mirror this page. See the [async overview](https://docs.remotestore.dev/stable/reference/api/aio/index.md) for the full sync ↔ async map.

| Class                                                                                                              | Description                                                                  |
| ------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- |
| [AsyncStore](https://docs.remotestore.dev/stable/reference/api/aio/store/index.md)                                 | Async counterpart to `Store` with coroutine methods for all operations       |
| [AsyncBackend](https://docs.remotestore.dev/stable/reference/api/aio/backend/index.md)                             | Abstract base class for native async backends                                |
| [SyncBackendAdapter](https://docs.remotestore.dev/stable/reference/api/aio/adapters/#syncbackendadapter)           | Wraps any synchronous backend for async use via thread-pool executor         |
| [AsyncBackendSyncAdapter](https://docs.remotestore.dev/stable/reference/api/aio/adapters/#asyncbackendsyncadapter) | Wraps any `AsyncBackend` as a synchronous `Backend` via a private event loop |
| [AsyncMemoryBackend](https://docs.remotestore.dev/stable/reference/api/aio/backends/memory/index.md)               | In-memory async backend for testing                                          |
| [AsyncAzureBackend](https://docs.remotestore.dev/stable/reference/api/aio/backends/azure/index.md)                 | Native async Azure Blob Storage and ADLS Gen2                                |
| [GraphBackend](https://docs.remotestore.dev/stable/reference/api/aio/backends/graph/index.md)                      | Microsoft Graph backend (OneDrive, SharePoint, Teams files)                  |
| [AsyncWritableContent](https://docs.remotestore.dev/stable/reference/api/aio/adapters/#asyncwritablecontent)       | Type alias: `bytes` or `AsyncIterator[bytes]`                                |

## Utilities

| Class                                                                                          | Description                                                           |
| ---------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| [SFTPUtils](https://docs.remotestore.dev/stable/reference/api/sftp-utils/index.md)             | Key loading and host-key verification helpers for SFTP                |
| [AzureUtils](https://docs.remotestore.dev/stable/reference/api/azure-utils/index.md)           | One-shot HNS detection helper for Azure accounts                      |
| [GraphAuth](https://docs.remotestore.dev/stable/reference/api/aio/backends/graph/#graphauth)   | MSAL token provider (client-credentials / device-code) for Graph      |
| [GraphUtils](https://docs.remotestore.dev/stable/reference/api/aio/backends/graph/#graphutils) | Resolve a Graph `drive_id` from OneDrive / SharePoint / Teams targets |

## Configuration

| Class                                                                                                                 | Description                                         |
| --------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| [RegistryConfig](https://docs.remotestore.dev/stable/reference/api/config/#remote_store.RegistryConfig)               | Top-level configuration holding backends and stores |
| [BackendConfig](https://docs.remotestore.dev/stable/reference/api/config/#remote_store.BackendConfig)                 | Configuration for a single backend                  |
| [StoreProfile](https://docs.remotestore.dev/stable/reference/api/config/#remote_store.StoreProfile)                   | Configuration for a single store                    |
| [RetryPolicy](https://docs.remotestore.dev/stable/reference/api/config/#remote_store.RetryPolicy)                     | Retry policy for backend operations                 |
| [Secret](https://docs.remotestore.dev/stable/reference/api/config/#remote_store.Secret)                               | Sensitive string wrapper with masked repr           |
| [SecretRedactionFilter](https://docs.remotestore.dev/stable/reference/api/config/#remote_store.SecretRedactionFilter) | Logging filter that redacts secrets                 |
| [resolve_env](https://docs.remotestore.dev/stable/reference/api/config/#remote_store.resolve_env)                     | Resolve `${VAR}` placeholders in config dicts       |

## Path & Models

| Class                                                                                                   | Description                                                                |
| ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| [RemotePath](https://docs.remotestore.dev/stable/reference/api/models/#remote_store.RemotePath)         | Validated, immutable path value object                                     |
| [ResolutionPlan](https://docs.remotestore.dev/stable/reference/api/models/#remote_store.ResolutionPlan) | Frozen introspection result from `resolve()`                               |
| [ContentDigest](https://docs.remotestore.dev/stable/reference/api/models/#remote_store.ContentDigest)   | Verified content digest with known algorithm                               |
| [FileInfo](https://docs.remotestore.dev/stable/reference/api/models/#remote_store.FileInfo)             | Metadata for a file (name, size, modified time)                            |
| [WriteResult](https://docs.remotestore.dev/stable/reference/api/models/#remote_store.WriteResult)       | Immutable snapshot of a completed write operation                          |
| [FolderEntry](https://docs.remotestore.dev/stable/reference/api/models/#remote_store.FolderEntry)       | Folder identity returned by listing operations                             |
| [FolderInfo](https://docs.remotestore.dev/stable/reference/api/models/#remote_store.FolderInfo)         | Aggregated folder metadata (file count, total size); satisfies `PathEntry` |
| [PathEntry](https://docs.remotestore.dev/stable/reference/api/models/#remote_store.PathEntry)           | Protocol for uniform listing (name + path)                                 |

## Capabilities

| Class                                                                                                       | Description                            |
| ----------------------------------------------------------------------------------------------------------- | -------------------------------------- |
| [Capability](https://docs.remotestore.dev/stable/reference/api/capabilities/#remote_store.Capability)       | Enum of backend capabilities           |
| [CapabilitySet](https://docs.remotestore.dev/stable/reference/api/capabilities/#remote_store.CapabilitySet) | Set of capabilities a backend supports |

## Errors

| Class                                                                                                                   | Description                              |
| ----------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- |
| [RemoteStoreError](https://docs.remotestore.dev/stable/reference/api/errors/#remote_store.RemoteStoreError)             | Base exception                           |
| [NotFound](https://docs.remotestore.dev/stable/reference/api/errors/#remote_store.NotFound)                             | File or folder not found                 |
| [AlreadyExists](https://docs.remotestore.dev/stable/reference/api/errors/#remote_store.AlreadyExists)                   | File already exists (no overwrite)       |
| [PermissionDenied](https://docs.remotestore.dev/stable/reference/api/errors/#remote_store.PermissionDenied)             | Insufficient permissions                 |
| [InvalidPath](https://docs.remotestore.dev/stable/reference/api/errors/#remote_store.InvalidPath)                       | Path validation failed                   |
| [CapabilityNotSupported](https://docs.remotestore.dev/stable/reference/api/errors/#remote_store.CapabilityNotSupported) | Backend lacks required capability        |
| [BackendUnavailable](https://docs.remotestore.dev/stable/reference/api/errors/#remote_store.BackendUnavailable)         | Backend could not be reached             |
| [DirectoryNotEmpty](https://docs.remotestore.dev/stable/reference/api/errors/#remote_store.DirectoryNotEmpty)           | Non-recursive delete on non-empty folder |
| [ResourceLocked](https://docs.remotestore.dev/stable/reference/api/errors/#remote_store.ResourceLocked)                 | Resource locked by another session       |

## Introspection

| Symbol                                                                                              | Description                                                |
| --------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| [info](https://docs.remotestore.dev/stable/reference/api/info/#remote_store.info)                   | Runtime introspection of available backends and extensions |
| [InfoResult](https://docs.remotestore.dev/stable/reference/api/info/#remote_store.InfoResult)       | TypedDict for the `info()` return value                    |
| [BackendInfo](https://docs.remotestore.dev/stable/reference/api/info/#remote_store.BackendInfo)     | TypedDict for a single backend entry in `InfoResult`       |
| [ExtensionInfo](https://docs.remotestore.dev/stable/reference/api/info/#remote_store.ExtensionInfo) | TypedDict for a single extension entry in `InfoResult`     |

## Functions

| Function                                                                                                      | Description                    |
| ------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| [register_backend](https://docs.remotestore.dev/stable/reference/api/registry/#remote_store.register_backend) | Register a custom backend type |

## Extensions

| Module                                                                                           | Description                                                     |
| ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- |
| [ext.arrow](https://docs.remotestore.dev/stable/reference/api/extensions/arrow/index.md)         | PyArrow `FileSystemHandler` adapter for Store                   |
| [ext.batch](https://docs.remotestore.dev/stable/reference/api/extensions/batch/index.md)         | Batch delete, copy, and exists operations                       |
| [ext.cache](https://docs.remotestore.dev/stable/reference/api/extensions/cache/index.md)         | Store-level caching middleware with TTL                         |
| [ext.dagster](https://docs.remotestore.dev/stable/reference/api/extensions/dagster/index.md)     | Dagster IO Manager adapter for Store                            |
| [ext.glob](https://docs.remotestore.dev/stable/reference/api/extensions/glob/index.md)           | Portable glob pattern matching fallback                         |
| [ext.integrity](https://docs.remotestore.dev/stable/reference/api/extensions/integrity/index.md) | Checksum computation and verification helpers                   |
| [ext.observe](https://docs.remotestore.dev/stable/reference/api/extensions/observe/index.md)     | Callback hooks for store operations                             |
| [ext.otel](https://docs.remotestore.dev/stable/reference/api/extensions/otel/index.md)           | OpenTelemetry bridge for ext.observe                            |
| [ext.partition](https://docs.remotestore.dev/stable/reference/api/extensions/partition/index.md) | Hive-style partition path helpers                               |
| [ext.pydantic](https://docs.remotestore.dev/stable/reference/api/extensions/pydantic/index.md)   | Pydantic model to RegistryConfig adapter                        |
| [ext.streams](https://docs.remotestore.dev/stable/reference/api/extensions/streams/index.md)     | Composable BinaryIO wrappers for progress and checksums         |
| [ext.transfer](https://docs.remotestore.dev/stable/reference/api/extensions/transfer/index.md)   | Upload, download, and cross-store transfer                      |
| [ext.write](https://docs.remotestore.dev/stable/reference/api/extensions/write/index.md)         | Write helpers with guaranteed client-side content hashing       |
| [aio.ext.write](https://docs.remotestore.dev/stable/reference/api/aio/extensions/write/index.md) | Async write helpers with guaranteed client-side content hashing |
| [ext.yaml](https://docs.remotestore.dev/stable/reference/api/extensions/yaml/index.md)           | YAML config loader (PyYAML / ruamel.yaml)                       |

# AzureUtils

Backend-specific module

The helpers in this module are exclusive to the Azure backend. Using them ties your code to `AzureBackend` / `AsyncAzureBackend`.

`AzureUtils` is a namespace of stateless, one-shot helpers for Azure Storage accounts that do not require constructing a full backend — mirroring [`SFTPUtils`](https://docs.remotestore.dev/stable/reference/api/sftp-utils/index.md) and [`GraphUtils`](https://docs.remotestore.dev/stable/reference/api/aio/backends/graph/#graphutils).

Its primary use is discovering whether an account has Hierarchical Namespace (ADLS Gen2) enabled, so you can pass the result to `AzureBackend(hns=...)`. Unlike a silent runtime probe, these helpers are **fail-loud**: a probe error is raised, never swallowed.

## AzureUtils

Stateless helpers for Azure Storage accounts.

A namespace of one-shot utilities that do not require constructing a full `AzureBackend` -- mirroring `SFTPUtils` and `GraphUtils`.

## Methods

### detect_hns

```
detect_hns(
    *,
    account_name: str | None = None,
    account_url: str | None = None,
    account_key: str | Secret | None = None,
    sas_token: str | Secret | None = None,
    connection_string: str | Secret | None = None,
    credential: Any | None = None,
    client_options: dict[str, Any] | None = None,
) -> bool
```

Probe whether a storage account has Hierarchical Namespace enabled.

Issues a single `GetAccountInfo` call against the account, intended for discovering the value to pass as `AzureBackend(hns=...)`. This is **fail-loud**: a probe error is mapped to a `remote_store` error and raised, never swallowed.

Parameters:

- **`account_name`** (`str | None`, default: `None` ) – Storage account name.
- **`account_url`** (`str | None`, default: `None` ) – Full account URL (blob endpoint).
- **`account_key`** (`str | Secret | None`, default: `None` ) – Storage account key.
- **`sas_token`** (`str | Secret | None`, default: `None` ) – Shared Access Signature token.
- **`connection_string`** (`str | Secret | None`, default: `None` ) – Azure Storage connection string. When set, takes precedence over account_url / account_name.
- **`credential`** (`Any | None`, default: `None` ) – Any credential object (e.g. DefaultAzureCredential()).
- **`client_options`** (`dict[str, Any] | None`, default: `None` ) – Additional options passed to the service client.

Returns:

- `bool` – True if Hierarchical Namespace (ADLS Gen2) is enabled, False
- `bool` – for a flat Blob Storage account.

Raises:

- `RemoteStoreError` – If the probe fails (authentication, network, etc.).

### adetect_hns

```
adetect_hns(
    *,
    account_name: str | None = None,
    account_url: str | None = None,
    account_key: str | Secret | None = None,
    sas_token: str | Secret | None = None,
    connection_string: str | Secret | None = None,
    credential: Any | None = None,
    client_options: dict[str, Any] | None = None,
) -> bool
```

Async sibling of `detect_hns` (uses the async Blob SDK).

Accepts the same parameters as `detect_hns`.

Returns:

- `bool` – True if Hierarchical Namespace (ADLS Gen2) is enabled,
- `bool` – False for a flat Blob Storage account.

Raises:

- `RemoteStoreError` – If the probe fails (authentication, network, etc.).

## See also

- [Azure Backend Guide](https://docs.remotestore.dev/stable/guides/backends/azure/index.md) — usage patterns, configuration, and the `hns` declaration
- [Azure Backend example](https://docs.remotestore.dev/stable/tutorial/examples/azure-backend/index.md) — Azure backend in action

# Backend

## Backend

Bases: `ABC`

Abstract base class for all storage backends.

Every backend must implement all abstract methods. Backend-native exceptions must never leak — they must be mapped to `remote_store` errors.

Implementing a backend

Subclass `Backend` and implement all abstract methods. Map every backend-native exception to a `remote_store` error — native exceptions must never leak to callers.

______________________________________________________________________

## Identity

### name

```
name: str
```

Unique identifier for this backend type (e.g. `'local'`, `'s3'`).

### capabilities

```
capabilities: CapabilitySet
```

Declared capabilities of this backend.

______________________________________________________________________

## Checking Existence

### exists

```
exists(path: str) -> bool
```

Check if a file or folder exists. Never raises `NotFound`.

Returns `False` if any ancestor of *path* is a file (file-as-directory-component), as traversal cannot proceed.

Parameters:

- **`path`** (`str`) – Backend-relative key, or "" for the root.

Returns:

- `bool` – True if a file or folder exists at path. False if the path
- `bool` – does not exist or if any ancestor is a file instead of a directory.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if `path` is an existing file.

Returns `False` if the path does not exist, or if any ancestor of *path* is a file (file-as-directory-component).

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `bool` – True if path exists and is a file. False if the path does not
- `bool` – exist or if any ancestor is a file instead of a directory.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if `path` is an existing folder.

Returns `False` if the path does not exist, or if any ancestor of *path* is a file (file-as-directory-component).

Parameters:

- **`path`** (`str`) – Backend-relative key, or "" for the root.

Returns:

- `bool` – True if path exists and is a folder. False if the path does not
- `bool` – exist or if any ancestor is a file instead of a directory.

______________________________________________________________________

## Reading

Requires `Capability.READ`

All read methods raise `CapabilityNotSupported` on backends that do not declare this capability. Most backends declare it.

### read

```
read(path: str) -> BinaryIO
```

Open a file for reading and return a binary stream.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `BinaryIO` – A readable binary stream.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path names a directory, not a file.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read the full content of a file as bytes.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `bytes` – The file content.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path names a directory, not a file.

### read_seekable

```
read_seekable(path: str) -> BinaryIO
```

Open a file for random-access reading and return a seekable stream.

The default implementation delegates to `read()`. If the returned stream is already seekable, it is returned as-is. Otherwise, the stream is spooled into a `SpooledTemporaryFile` (up to 8 MB in RAM, beyond that on disk) and returned positioned at byte 0.

Backends MAY override to provide an optimized implementation. For example, `AzureBackend` returns a range reader that issues HTTP Range requests on each `read()` call.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `BinaryIO` – A seekable binary stream positioned at byte 0.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path names a directory, not a file.

Default implementation

The default spools the stream into a `SpooledTemporaryFile` (up to 8 MB in RAM, beyond that on disk) when the backend stream is not already seekable. Override for efficiency when the backend supports range reads.

______________________________________________________________________

## Writing

Requires `Capability.WRITE`

`write()` raises `CapabilityNotSupported` on backends that do not declare this capability. Most backends declare it.

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write content to a file.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`content`** (`WritableContent`) – Data to write.
- **`overwrite`** (`bool`, default: `False` ) – If False, raise if file already exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user-supplied key/value pairs to store alongside the file. Only honoured when the backend declares USER_METADATA.

Returns:

- `WriteResult` – A WriteResult with at least path and size populated.
- `WriteResult` – Backends declaring WRITE_RESULT_NATIVE also populate etag,
- `WriteResult` – last_modified, and where available digest and version_id.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or if any slash-aligned ancestor of path exists as a regular file. Flat-namespace backends (S3, Azure non-HNS, SQL) cannot detect a file ancestor in O(1) and skip the check by default; the per-backend reject_write_under_file_ancestor opt-in enables it.

Backend-conditional argument: `metadata=`

Passing `metadata` raises `CapabilityNotSupported` on backends that do not declare `Capability.USER_METADATA`. Passing `None` or `{}` is safe on all backends.

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write content atomically via temp file + rename.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`content`** (`WritableContent`) – Data to write.
- **`overwrite`** (`bool`, default: `False` ) – If False, raise if file already exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user-supplied key/value pairs (see write()).

Returns:

- `WriteResult` – A WriteResult (same contract as write()).

Raises:

- `CapabilityNotSupported` – If backend lacks ATOMIC_WRITE.
- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or if any slash-aligned ancestor of path exists as a regular file (see write).

Requires `Capability.ATOMIC_WRITE`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

Backend-conditional argument: `metadata=`

Passing `metadata` raises `CapabilityNotSupported` on backends that do not declare `Capability.USER_METADATA`. Passing `None` or `{}` is safe on all backends.

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> AbstractContextManager[BinaryIO]
```

Yield a writable file object backed by a temporary location.

On successful exit the temp file is atomically promoted to *path*. On exception the temp file is removed and *path* is untouched.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`overwrite`** (`bool`, default: `False` ) – If False, raise if file already exists.

Raises:

- `AlreadyExists` – If path exists and overwrite is False.
- `InvalidPath` – If path names a directory, or if any slash-aligned ancestor of path exists as a regular file (see write).
- `CapabilityNotSupported` – If the backend lacks ATOMIC_WRITE.

Requires `Capability.ATOMIC_WRITE`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

______________________________________________________________________

## Deleting

Requires `Capability.DELETE`

All delete methods raise `CapabilityNotSupported` on backends that do not declare this capability.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete a file.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`missing_ok`** (`bool`, default: `False` ) – If True, do not raise when the file is absent.

Raises:

- `NotFound` – If the file is missing and missing_ok is False.
- `InvalidPath` – If path names a directory (type mismatch is not silenced by missing_ok).

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete a folder.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`recursive`** (`bool`, default: `False` ) – If True, delete all contents first.
- **`missing_ok`** (`bool`, default: `False` ) – If True, do not raise when absent.

Raises:

- `NotFound` – If the folder is missing and missing_ok is False.
- `InvalidPath` – If path names a file, not a folder.
- `DirectoryNotEmpty` – If non-empty and recursive is False.

______________________________________________________________________

## Listing and Iteration

Requires `Capability.LIST`

All listing methods raise `CapabilityNotSupported` on backends that do not declare this capability.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> Iterator[FileInfo]
```

List files under `path`.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.
- **`recursive`** (`bool`, default: `False` ) – If True, include files in all subdirectories.
- **`max_depth`** (`int | None`, default: `None` ) – Optional maximum folder depth to traverse. When set, backends that support native depth limiting prune traversal early. Backends that ignore this parameter still produce correct results — the Store applies client-side filtering as a safety net. None (default) defers to recursive.

Returns:

- `Iterator[FileInfo]` – An iterator of FileInfo objects.

Backend-conditional argument: `max_depth=`

Backends with native depth limiting prune traversal early. Backends that do not support it still return correct results — the Store applies client-side filtering as a safety net.

### list_folders

```
list_folders(path: str) -> Iterator[FolderEntry]
```

List immediate subfolders under `path`.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.

Returns:

- `Iterator[FolderEntry]` – An iterator of FolderEntry objects with .name and .path.

### iter_children

```
iter_children(
    path: str,
) -> Iterator[FileInfo | FolderEntry]
```

Yield both files and folders under `path` in a single pass.

Files are yielded as `FileInfo` objects, folders as `FolderEntry` objects. The default implementation chains `list_files()` and `list_folders()`. Backends that can fetch both in a single I/O call should override this for efficiency.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.

Returns:

- `Iterator[FileInfo | FolderEntry]` – An iterator of FileInfo (files) and FolderEntry (folders).

Default implementation

Chains `list_files()` then `list_folders()`. Override when the backend can fetch both in a single I/O call.

### glob

```
glob(pattern: str) -> Iterator[FileInfo]
```

Match files against a glob pattern.

Non-abstract — backends with native glob support override this and add `Capability.GLOB` to their capability set.

Parameters:

- **`pattern`** (`str`) – Glob pattern (e.g., "data/\*.csv", "\*\*/\*.txt").

Raises:

- `CapabilityNotSupported` – If the backend lacks GLOB.

Requires `Capability.GLOB`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

Non-abstract

The default raises `CapabilityNotSupported`. Backends that provide native glob support override this and declare `Capability.GLOB`.

______________________________________________________________________

## Metadata

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Get metadata for a file.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `FileInfo` – A FileInfo with size, modification time, etc.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path names a directory, not a file.

Requires `Capability.METADATA`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Get metadata for a folder.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.

Returns:

- `FolderInfo` – A FolderInfo with file count, total size, etc.

Raises:

- `NotFound` – If the folder does not exist.
- `InvalidPath` – If path names a file, not a folder.

Requires `Capability.METADATA`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

______________________________________________________________________

## File Operations

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename a file.

When `src == dst` the call is a no-op (data preserved).

Parameters:

- **`src`** (`str`) – Backend-relative source key.
- **`dst`** (`str`) – Backend-relative destination key.
- **`overwrite`** (`bool`, default: `False` ) – If True, replace any existing file at dst.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory, or if any slash-aligned ancestor of dst exists as a regular file.
- `AlreadyExists` – If dst exists and overwrite is False.

Requires `Capability.MOVE`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy a file.

When `src == dst` the call is a no-op (data preserved).

Parameters:

- **`src`** (`str`) – Backend-relative source key.
- **`dst`** (`str`) – Backend-relative destination key.
- **`overwrite`** (`bool`, default: `False` ) – If True, replace any existing file at dst.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory, or if any slash-aligned ancestor of dst exists as a regular file.
- `AlreadyExists` – If dst exists and overwrite is False.

Requires `Capability.COPY`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

______________________________________________________________________

## Introspection

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` describing how *path* maps to storage.

Pure introspection -- no I/O is performed. Backends override this to populate `details` with backend-specific context.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – A frozen ResolutionPlan with kind, backend,
- `ResolutionPlan` – key, native_path, and details.

______________________________________________________________________

## Lifecycle

### check_health

```
check_health() -> None
```

Verify the backend is reachable and credentials are valid.

The default implementation is a no-op (always succeeds). Backends override this to perform a lightweight, non-destructive connectivity check using the cheapest possible read-only operation.

Raises:

- `PermissionDenied` – If credentials are invalid.
- `NotFound` – If the bucket, container, or root path does not exist.
- `BackendUnavailable` – If the backend cannot be reached.

### close

```
close() -> None
```

Release resources. Default is a no-op.

Whether the backend may be reused afterwards is the `close_is_terminal` posture: the default backend is reusable; terminal backends raise `BackendUnavailable` on use-after-close.

______________________________________________________________________

## Interop (Backend-Specific)

Backend-specific methods

Methods in this section expose backend internals. Using them ties your code to a specific backend. For portable alternatives, use the methods above.

### unwrap

```
unwrap(type_hint: type[T]) -> T
```

Return the native backend handle if it matches the requested type.

Parameters:

- **`type_hint`** (`type[T]`) – The expected type (e.g., fsspec.AbstractFileSystem).

Raises:

- `CapabilityNotSupported` – If backend cannot provide the requested type.

### native_path

```
native_path(path: str) -> str
```

Convert a backend-relative key to the backend-native path.

The inverse of `to_key()`. The default implementation is the identity function — backends with a native root (bucket, base_path) override this to prepend their prefix.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `str` – Backend-native path usable with the native handle from unwrap().

### to_key

```
to_key(native_path: str) -> str
```

Convert a backend-native path to a backend-relative key.

Strips the backend's own root/prefix from the path. The default implementation is the identity function — backends with a native root (filesystem path, bucket prefix, base_path) override this.

Parameters:

- **`native_path`** (`str`) – Absolute or backend-native path string.

Returns:

- `str` – Path relative to the backend's root.

**Related types:** [`CapabilitySet`](https://docs.remotestore.dev/stable/reference/api/capabilities/index.md), [`FileInfo`](https://docs.remotestore.dev/stable/reference/api/models/index.md), [`FolderInfo`](https://docs.remotestore.dev/stable/reference/api/models/index.md), [`FolderEntry`](https://docs.remotestore.dev/stable/reference/api/models/index.md), [`ResolutionPlan`](https://docs.remotestore.dev/stable/reference/api/models/index.md).

## See also

- [Build Your Own Backend](https://docs.remotestore.dev/stable/guides/custom-backend-guide/index.md) — step-by-step guide to implementing a custom backend
- [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) — per-backend capability comparison
- [Errors](https://docs.remotestore.dev/stable/reference/api/errors/index.md) — error types backends must raise

# Capabilities

## Capability

Bases: `Enum`

Operations a backend may support.

Most values gate one or more `Store` methods; some are quality flags that inform callers about backend behaviour without gating a specific method (see `ATOMIC_MOVE`, `SEEKABLE_READ`, `LAZY_READ`). Use `Store.supports()` to query at runtime.

Values:

*Core I/O*

- `READ` -- Stream or bulk-read file content. Gates `Store.read()` and `Store.read_bytes()`.
- `WRITE` -- Create or overwrite files. Gates `Store.write()`.
- `DELETE` -- Remove files and folders. Gates `Store.delete()` and `Store.delete_folder()`.

*Navigation*

- `LIST` -- Enumerate files and subfolders. Gates `Store.list_files()` and `Store.list_folders()`.
- `GLOB` -- Native pattern matching against file paths. Gates `Store.glob()`. Not all backends support this — use `ext.glob.glob_files()` as a portable fallback.

*File operations*

- `MOVE` -- Rename or relocate a file within the same backend. Gates `Store.move()`.
- `COPY` -- Duplicate a file within the same backend. Gates `Store.copy()`.

*Atomic variants*

- `ATOMIC_WRITE` -- Write via temp-file-and-rename so readers never see partial content. Gates `Store.write_atomic()` and `Store.open_atomic()`.
- `ATOMIC_MOVE` -- Quality flag: `move()` is guaranteed atomic under concurrent access (e.g. Local via `os.rename`, Memory under lock, SQL in a transaction). Does **not** gate a method — call `store.supports(Capability.ATOMIC_MOVE)` before relying on atomic rename semantics. Backends that implement move as copy-then-delete (e.g. S3, Azure non-HNS) do not declare this capability.

*Metadata*

- `METADATA` -- Retrieve file or folder metadata. Gates `Store.get_file_info()` and `Store.get_folder_info()`.
- `USER_METADATA` -- Store user-supplied key/value pairs alongside a file. Strict gate on the `metadata=` kwarg in `Store.write()`, `Store.write_text()`, and `Store.write_atomic()`: passing a non-empty mapping to a backend that does not declare this capability raises `CapabilityNotSupported` before any I/O. `metadata=None` and `metadata={}` are always allowed.

*Quality flags*

- `SEEKABLE_READ` -- `Store.read()` always returns a seekable stream (`stream.seekable()` is `True`). Backends that declare this capability return seekable streams from both `read()` and `read_seekable()` with zero overhead. Absence of the flag means only that `read()` is forward-only -- not that seekable reads are unavailable. On the sync `Store`, `read_seekable()` is gated on `READ` alone, so it is served on every backend: a backend without the flag serves it either through a native override as cheap as the passthrough (the sync Azure backend issues one ranged download per `read()`, with no temp-file spill) or through a `SpooledTemporaryFile` that copies the object first (HTTP, and any async backend bridged to sync). The async API has no `read_seekable()` -- an async-native backend that omits the flag (`AsyncAzureBackend`, `GraphBackend`) has no seekable read until bridged to sync. The flag does not distinguish the native and spooled costs; the Azure backend guide's "Streaming and seekable reads" section documents the sync Azure range reader in full.
- `LAZY_READ` -- `read()` fetches data lazily on demand from the native source rather than loading the entire file into memory before returning. Backends that pre-load the full file contents (e.g. in-memory backends, SQL blob stores) do **not** declare this flag. Callers can use `store.supports(Capability.LAZY_READ)` to know whether partial reads avoid loading the entire file.
- `WRITE_RESULT_NATIVE` -- Quality flag: the backend populates the rich fields of the returned `WriteResult` (`etag`, `last_modified`, `version_id`, and where applicable `digest`) directly from its write response. Does **not** gate any method — `Store.write*()` works on every backend. Backends without this flag return a `WriteResult` with only `path` and `size` populated (`source == "basic"`); `metadata` is governed independently by the `USER_METADATA` capability and is not subject to this flag. Use `store.supports(Capability.WRITE_RESULT_NATIVE)` to decide whether to call `store.get_file_info()` after a write if you need the full metadata set.

Quality flags vs. method gates

Two kinds of capabilities exist. **Method gates** (e.g. `READ`, `WRITE`, `DELETE`) guard specific Store or Backend methods — calling a gated method on a backend that does not declare the capability raises `CapabilityNotSupported`. **Quality flags** (e.g. `SEEKABLE_READ`, `WRITE_RESULT_NATIVE`) are informational only — they describe behaviour the backend provides but do not guard any method call. For example, a backend that omits `SEEKABLE_READ` is still served by the sync `Store.read_seekable()` — the flag reports only whether `read()` itself is seekable, not whether `read_seekable()` is native or spooled (the async API has no `read_seekable()`). Check the class docstring for the full categorisation.

## CapabilitySet

```
CapabilitySet(capabilities: set[Capability])
```

Immutable set of capabilities declared by a backend.

Parameters:

- **`capabilities`** (`set[Capability]`) – The set of supported capabilities.

### supports

```
supports(cap: Capability) -> bool
```

Check whether a capability is supported.

### require

```
require(cap: Capability, *, backend: str = '') -> None
```

Raise if a capability is not supported.

Raises:

- `CapabilityNotSupported` – If the capability is missing.

## See also

- [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) — per-backend capability comparison
- [Capabilities & Errors example](https://docs.remotestore.dev/stable/tutorial/examples/capabilities-and-errors/index.md) — checking capabilities at runtime

# Configuration

## RegistryConfig

```
RegistryConfig(
    backends: dict[str, BackendConfig] = dict(),
    stores: dict[str, StoreProfile] = dict(),
)
```

Top-level configuration container.

Parameters:

- **`backends`** (`dict[str, BackendConfig]`, default: `dict()` ) – Mapping of backend names to their configs.
- **`stores`** (`dict[str, StoreProfile]`, default: `dict()` ) – Mapping of store names to their profiles.

### validate

```
validate() -> None
```

Validate that all store profiles reference existing backends.

Raises:

- `ValueError` – If a store references a non-existent backend.

### from_dict

```
from_dict(data: dict[str, object]) -> RegistryConfig
```

Construct from a plain dict (e.g. parsed TOML/JSON).

Parameters:

- **`data`** (`dict[str, object]`) – Dict with backends and stores keys.

### from_toml

```
from_toml(
    path: str | Path,
    *,
    table: tuple[str, ...] = (),
    resolve_env_vars: bool = False,
) -> RegistryConfig
```

Load config from a TOML file.

Parameters:

- **`path`** (`str | Path`) – Path to the TOML file.
- **`table`** (`tuple[str, ...]`, default: `()` ) – Dotted table path to extract config from. For pyproject.toml use table=("tool", "remote-store").
- **`resolve_env_vars`** (`bool`, default: `False` ) – When True, resolve ${VAR} placeholders via resolve_env before constructing the config.

Raises:

- `ModuleNotFoundError` – If tomllib is unavailable and tomli is not installed.
- `KeyError` – If a table key is not found, or if resolve_env_vars is True and a placeholder references an unset variable with no default.

## BackendConfig

```
BackendConfig(
    type: str,
    options: dict[str, object] = dict(),
    retry: RetryPolicy | None = None,
)
```

Describes a backend instance.

Parameters:

- **`type`** (`str`) – Backend type identifier (e.g. "local", "s3").
- **`options`** (`dict[str, object]`, default: `dict()` ) – Backend-specific configuration options.
- **`retry`** (`RetryPolicy | None`, default: `None` ) – Optional retry policy for transient errors.

Backend-conditional field: `options`

The `options` mapping contains backend-specific configuration. Keys and accepted values depend on the backend being configured.

## RetryPolicy

```
RetryPolicy(
    max_attempts: int = 3,
    backoff_base: float = 1.0,
    backoff_max: float = 60.0,
    jitter: float = 1.0,
    timeout: float | None = None,
)
```

Retry configuration for transient backend errors.

Backends map these parameters to their native retry mechanisms. Backends that don't support a parameter silently ignore it.

Parameters:

- **`max_attempts`** (`int`, default: `3` ) – Maximum number of attempts (including the initial). Set to 1 to disable retry.
- **`backoff_base`** (`float`, default: `1.0` ) – Base delay in seconds for exponential backoff.
- **`backoff_max`** (`float`, default: `60.0` ) – Maximum delay between retries in seconds.
- **`jitter`** (`float`, default: `1.0` ) – Maximum random jitter added to each delay in seconds. Set to 0.0 to disable jitter.
- **`timeout`** (`float | None`, default: `None` ) – Total wall-clock timeout in seconds for all attempts combined. None means no total timeout.

### disabled

```
disabled() -> RetryPolicy
```

Return a policy that disables retry (single attempt, no backoff).

## StoreProfile

```
StoreProfile(
    backend: str,
    root_path: str = "",
    options: dict[str, object] = dict(),
)
```

Describes a named store.

Parameters:

- **`backend`** (`str`) – Name of the backend config to use.
- **`root_path`** (`str`, default: `''` ) – Path prefix for all operations.
- **`options`** (`dict[str, object]`, default: `dict()` ) – Store-specific options.

## Secret

```
Secret(value: str)
```

Immutable wrapper that prevents accidental exposure of credential strings.

`__repr__` and `__str__` always return a masked value. Call `.reveal()` to obtain the plain-text credential.

Parameters:

- **`value`** (`str`) – The secret string to protect.

Raises:

- `TypeError` – If value is not a str.

### reveal

```
reveal() -> str
```

Return the plain-text secret.

## SecretRedactionFilter

Bases: `Filter`

Logging filter that replaces `Secret` instances in log record args with `'***'`.

Attach to any handler or logger to prevent credential leakage through %-style formatting:

```
handler.addFilter(SecretRedactionFilter())
```

## resolve_env

```
resolve_env(
    data: dict[str, object],
    *,
    environ: Mapping[str, str] | None = None,
) -> dict[str, object]
```

Resolve `${VAR}` placeholders in a config dict.

Recursively walks *data* and replaces `${VAR}` with the value of the environment variable *VAR*. Use `${VAR:-default}` to provide a fallback when *VAR* is not set. Escape with `$${` to produce a literal `${`.

The original dict is never mutated; a deep copy with resolved values is returned.

Parameters:

- **`data`** (`dict[str, object]`) – Config dict (typically parsed from YAML/TOML).
- **`environ`** (`Mapping[str, str] | None`, default: `None` ) – Variable source. Defaults to os.environ.

Returns:

- `dict[str, object]` – A new dict with all placeholder strings resolved.

Raises:

- `KeyError` – If a placeholder references a variable that is not set and has no default. The message includes the variable name and the config key path where it was found.

## See also

- [Retry](https://docs.remotestore.dev/stable/guides/retry/index.md) — configuring retry policies
- [Security Model](https://docs.remotestore.dev/stable/explanation/security-model/index.md) — credential handling and secret redaction
- [Configuration example](https://docs.remotestore.dev/stable/tutorial/examples/configuration/index.md) — backend and store configuration
- [Retry Policy example](https://docs.remotestore.dev/stable/tutorial/examples/retry-policy/index.md) — retry policy in action
- [Config Loaders example](https://docs.remotestore.dev/stable/tutorial/examples/config-loaders/index.md) — TOML, YAML, Pydantic, and env-var interpolation

# Errors

## RemoteStoreError

```
RemoteStoreError(
    message: str = "",
    *,
    path: str | None = None,
    backend: str | None = None,
)
```

Bases: `Exception`

Base class for all remote_store errors.

Parameters:

- **`message`** (`str`, default: `''` ) – Human-readable error description.
- **`path`** (`str | None`, default: `None` ) – The path involved in the error, if any.
- **`backend`** (`str | None`, default: `None` ) – The backend name involved, if any.

## NotFound

```
NotFound(
    message: str = "",
    *,
    path: str | None = None,
    backend: str | None = None,
)
```

Bases: `RemoteStoreError`

Raised when a file or folder does not exist.

Raised by read, delete, metadata, move, and copy operations when the target path does not exist.

## AlreadyExists

```
AlreadyExists(
    message: str = "",
    *,
    path: str | None = None,
    backend: str | None = None,
)
```

Bases: `RemoteStoreError`

Raised when a target already exists and overwrite is not allowed.

Raised by write and copy operations when `overwrite=False` (the default) and the destination already exists.

## PermissionDenied

```
PermissionDenied(
    message: str = "",
    *,
    path: str | None = None,
    backend: str | None = None,
)
```

Bases: `RemoteStoreError`

Raised when access is denied by the storage backend.

Raised by any Store or Backend method when the underlying storage system denies access (e.g., missing credentials, insufficient permissions on the bucket or container).

## InvalidPath

```
InvalidPath(
    message: str = "",
    *,
    path: str | None = None,
    backend: str | None = None,
)
```

Bases: `RemoteStoreError`

Raised for malformed, unsafe, out-of-scope, or wrong-type paths.

Raised by any method that validates paths: empty strings in file-targeted operations, paths containing `..` or null bytes, paths that fall outside the store's root scope, and paths that name the wrong type (e.g. a file operation on a directory path, or a folder operation on a file path).

## CapabilityNotSupported

```
CapabilityNotSupported(
    message: str = "",
    *,
    path: str | None = None,
    backend: str | None = None,
    capability: str = "",
)
```

Bases: `RemoteStoreError`

Raised when an operation requires an unsupported capability.

Raised by capability-gated Store methods and by `CapabilitySet.require()` when a backend does not declare the needed capability. Check `Store.supports()` before calling capability-gated methods.

Parameters:

- **`capability`** (`str`, default: `''` ) – The name of the unsupported capability.

## DirectoryNotEmpty

```
DirectoryNotEmpty(
    message: str = "",
    *,
    path: str | None = None,
    backend: str | None = None,
)
```

Bases: `RemoteStoreError`

Raised when a non-recursive delete targets a non-empty folder.

Raised by `Store.delete_folder()` when `recursive=False` (the default) and the folder contains files or subfolders.

## BackendUnavailable

```
BackendUnavailable(
    message: str = "",
    *,
    path: str | None = None,
    backend: str | None = None,
)
```

Bases: `RemoteStoreError`

Raised when the backend cannot be reached or initialized.

Raised during backend construction or first operation when the storage service is unreachable (e.g., network error, invalid endpoint, missing container).

## ResourceLocked

```
ResourceLocked(
    message: str = "",
    *,
    path: str | None = None,
    backend: str | None = None,
)
```

Bases: `RemoteStoreError`

Raised when a target resource is locked by another session.

The resource exists and the caller is authorised, but another session or process holds it (e.g. an open Office co-authoring session), so the operation cannot proceed now. Not retried by the default policy: the lock has no `Retry-After` and may last indefinitely. Maps from Microsoft Graph `423 Locked`.

## See also

- [Troubleshooting](https://docs.remotestore.dev/stable/guides/troubleshooting/index.md) — diagnosing and resolving common errors
- [Error Handling example](https://docs.remotestore.dev/stable/tutorial/examples/error-handling/index.md) — catching and handling store errors

# Info

## info

```
info() -> InfoResult
```

Return a structured summary of available backends and extensions.

Populates the backend registry, then probes each backend and optional extension for availability in the current environment.

Returns:

- `InfoResult` – An InfoResult with keys version, backends, and
- `InfoResult` – extensions.

## InfoResult

Bases: `TypedDict`

Structured result of the `info` function.

## BackendInfo

```
BackendInfo = TypedDict(
    "BackendInfo",
    {
        "available": bool,
        "extras": str | None,
        "class": str | None,
    },
)
```

## ExtensionInfo

Bases: `TypedDict`

Information about a single extension.

## See also

- [Health Check](https://docs.remotestore.dev/stable/guides/health-check/index.md) — verifying backend reachability at runtime
- [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) — per-backend capability comparison

# Models

## RemotePath

```
RemotePath(raw: str)
```

An immutable, normalized path within a remote store.

Parameters:

- **`raw`** (`str`) – The raw path string to normalize and validate.

Raises:

- `InvalidPath` – If the path is malformed or unsafe.

### name

```
name: str
```

Final component of the path.

### parent

```
parent: RemotePath | None
```

Parent path, or `None` if the path has only one component.

Example: `RemotePath("a/b").parent` returns `RemotePath("a")`, but `RemotePath("a").parent` returns `None`.

### parts

```
parts: tuple[str, ...]
```

Tuple of path components.

### suffix

```
suffix: str
```

File extension including the dot, or empty string.

### as_posix

```
as_posix() -> str
```

Return the path as a forward-slash string.

Mirrors `pathlib.PurePath.as_posix`. The path is always stored with forward slashes, so the result is identical to `str(self)` on every platform. `RemotePath.ROOT.as_posix()` returns `"."`.

### from_backend_path

```
from_backend_path(path: str) -> RemotePath
```

Create a RemotePath, using ROOT for empty paths.

Backends use this in `get_folder_info` to avoid duplicating the `RemotePath(path) if path else RemotePath.ROOT` pattern.

## PathEntry

Bases: `Protocol`

Shared interface for listing results -- every entry has a name and path.

### name

```
name: str
```

Entry name (final path component).

### path

```
path: RemotePath
```

Normalized remote path.

## ContentDigest

```
ContentDigest(algorithm: str, value: str)
```

Verified content digest with known algorithm.

Both `algorithm` and `value` are normalized to lowercase on construction.

Attributes:

- **`algorithm`** (`str`) – Hash algorithm name, always lowercase (e.g., "sha256").
- **`value`** (`str`) – Lowercase hex-encoded digest, no prefix, no separators.

## FileInfo

```
FileInfo(
    path: RemotePath,
    name: str,
    size: int,
    modified_at: datetime,
    digest: ContentDigest | None = None,
    etag: str | None = None,
    content_type: str | None = None,
    metadata: Mapping[str, str] | None = None,
    extra: dict[str, object] = dict(),
)
```

Immutable snapshot of file metadata.

Satisfies the `PathEntry` protocol.

Attributes:

- **`path`** (`RemotePath`) – Normalized remote path.
- **`name`** (`str`) – File name (final path component).
- **`size`** (`int`) – File size in bytes.
- **`modified_at`** (`datetime`) – Last modification time.
- **`digest`** (`ContentDigest | None`) – Verified content digest with known algorithm.
- **`etag`** (`str | None`) – Opaque backend-provided tag for change detection.
- **`content_type`** (`str | None`) – Optional MIME type.
- **`metadata`** (`Mapping[str, str] | None`) – User-supplied key/value metadata echoed from the backend.
- **`extra`** (`dict[str, object]`) – Backend-specific metadata.

Backend-conditional fields

`etag` varies by backend — whether it is populated and what it means depends on the backend. `metadata` requires `Capability.USER_METADATA` and is only present when metadata was stored with the file. `extra` contains backend-specific key/value pairs whose keys depend on the backend implementation.

## WriteResult

```
WriteResult(
    path: RemotePath,
    size: int,
    source: Literal["native", "basic", "sidecar"] = "basic",
    digest: ContentDigest | None = None,
    etag: str | None = None,
    version_id: str | None = None,
    last_modified: datetime | None = None,
    metadata: Mapping[str, str] | None = None,
)
```

Immutable snapshot of a completed write operation.

Returned by `Store.write()`, `Store.write_text()`, and `Store.write_atomic()`.

Attributes:

- **`path`** (`RemotePath`) – Normalized written path, store-relative.
- **`size`** (`int`) – Bytes written.
- **`source`** (`Literal['native', 'basic', 'sidecar']`) – Provenance of the optional fields. "native" — the backend populated them from its write response; trust digest, etag, and last_modified. "basic" — only path and size are reliable; call Store.head() or use ext.write helpers if you need more. "sidecar" — constructed by Store.head() from a subsequent get_file_info() call.
- **`digest`** (`ContentDigest | None`) – Content digest from the write — either a client-computed hash from ext.write helpers, or a hash echoed by the backend from its write response (e.g., Azure echoes the client-supplied MD5 as ContentDigest("md5", …)). None when neither source applies.
- **`etag`** (`str | None`) – Opaque backend change tag; semantics vary by backend.
- **`version_id`** (`str | None`) – Immutable backend version identifier; None when the backend does not version objects.
- **`last_modified`** (`datetime | None`) – Server timestamp from the write response; None when the backend's write response omits it.
- **`metadata`** (`Mapping[str, str] | None`) – Echo of user metadata stored with the object.

Backend-conditional fields

Only `path` and `size` are guaranteed to be populated on every write. All other fields depend on the `source` discriminator:

- `source="native"` — rich fields populated; requires `Capability.WRITE_RESULT_NATIVE`.
- `source="basic"` — only `path` and `size` are reliable.
- `source="sidecar"` — fields sourced from a `get_file_info()` enrichment call.

Always check `source` before reading any optional field.

## FolderEntry

```
FolderEntry(path: RemotePath, name: str)
```

Immutable folder identity returned by listing operations.

Satisfies the `PathEntry` protocol.

Attributes:

- **`path`** (`RemotePath`) – Normalized remote path.
- **`name`** (`str`) – Folder name (final path component).

## FolderInfo

```
FolderInfo(
    path: RemotePath,
    file_count: int,
    total_size: int,
    modified_at: datetime | None = None,
    extra: dict[str, object] = dict(),
)
```

Aggregated folder metadata.

Satisfies the `PathEntry` protocol.

Attributes:

- **`path`** (`RemotePath`) – Normalized remote path.
- **`file_count`** (`int`) – Number of files in the folder.
- **`total_size`** (`int`) – Total size of all files in bytes.
- **`modified_at`** (`datetime | None`) – Optional last modification time.
- **`extra`** (`dict[str, object]`) – Backend-specific metadata.

### name

```
name: str
```

Folder name (final path component).

Unlike `FileInfo` and `FolderEntry`, which store `name` as a constructor field, this is a derived property (`self.path.name`) to avoid redundancy and keep `name` in sync with `path`.

Backend-conditional fields

`extra` contains backend-specific metadata. `modified_at` is `None` on backends that do not track folder modification times.

## ResolutionPlan

```
ResolutionPlan(
    kind: str,
    backend: str,
    key: str,
    native_path: str,
    details: dict[str, Any],
)
```

Describes how a key maps to its storage location.

Returned by `Backend.resolve()` and `Store.resolve()`. The plan captures the resolution strategy, backend identity, resolved key, native path, and backend-specific context -- all without performing any I/O.

`details` is wrapped in `types.MappingProxyType` via `__post_init__` to prevent accidental mutation at runtime.

Parameters:

- **`kind`** (`str`) – Resolution strategy identifier (e.g. "local", "s3", "azure").
- **`backend`** (`str`) – Human-readable backend identifier (typically Backend.name).
- **`key`** (`str`) – The resolved key (store-relative after Store.resolve(), backend-relative after Backend.resolve()).
- **`native_path`** (`str`) – Backend-native location string (same as Backend.native_path() output).
- **`details`** (`dict[str, Any]`) – Backend-specific resolution context. Immutable at runtime. Values should be JSON-serializable primitives.

Backend-conditional field: `details`

The `details` mapping contains backend-specific context. Keys and values depend on the backend implementation.

## See also

- [Getting Started](https://docs.remotestore.dev/stable/tutorial/getting-started/index.md) — using FileInfo and metadata in Store operations
- [File Operations example](https://docs.remotestore.dev/stable/tutorial/examples/file-operations/index.md) — reading, writing, and inspecting files
- [Store.resolve()](https://docs.remotestore.dev/stable/reference/api/store/#introspection) — returns a ResolutionPlan for key introspection

# ProxyStore

Base class for building Store middleware. Subclass it to intercept specific operations while delegating the rest to the inner Store.

ProxyStore is an [internal delegation base by design](https://docs.remotestore.dev/stable/explanation/design/adrs/0014-middleware-path-1-proxy-store-stream-wrappers/index.md) — it centralises the private-attribute coupling that `ObservedStore` and `CachedStore` share. It is [documented publicly](https://docs.remotestore.dev/stable/explanation/design/adrs/0015-proxystore-publicly-documented/index.md) because it is visible in their inheritance chain and useful for anyone building custom Store extensions.

## ProxyStore

```
ProxyStore(inner: Store)
```

Bases: `Store`

Base class for Store proxies that delegate to an inner Store.

All public `Store` methods delegate to `self._inner` by default. Subclasses override only the methods they intercept and must implement `_wrap_child()` to control how `child()` propagates wrapper behavior.

`ObservedStore` and `CachedStore` are built on this base. Subclass it to build your own Store middleware.

Parameters:

- **`inner`** (`Store`) – The Store instance to wrap.

Example

```
from remote_store import ProxyStore, Store

class LoggingStore(ProxyStore):
    def read_bytes(self, path: str) -> bytes:
        print(f"Reading {path}")
        return self.inner.read_bytes(path)

    def _wrap_child(self, inner_child: Store) -> "LoggingStore":
        return LoggingStore(inner_child)
```

### inner

```
inner: Store
```

The wrapped Store instance.

### child

```
child(subpath: str) -> Store
```

Return a child store wrapped with the same proxy behavior.

Delegates to `_wrap_child()` which subclasses must implement.

## Interop (Backend-Specific)

Backend-specific methods

`unwrap`, `native_path`, and `to_key` delegate directly to the inner Store and expose backend internals. Using them ties your code to a specific backend. `supports()` is portable — it works on all backends. See [Store — Interop](https://docs.remotestore.dev/stable/reference/api/store/#interop-backend-specific) for the full contract.

## See also

- [ext.observe](https://docs.remotestore.dev/stable/reference/api/extensions/observe/index.md) — ObservedStore, built on ProxyStore
- [ext.cache](https://docs.remotestore.dev/stable/reference/api/extensions/cache/index.md) — CachedStore, built on ProxyStore
- [ext.streams](https://docs.remotestore.dev/stable/reference/api/extensions/streams/index.md) — composable stream wrappers (an alternative to Store-level proxying)

# Registry

## Registry

```
Registry(config: RegistryConfig | None = None)
```

Manages backend lifecycle and provides access to named stores.

Parameters:

- **`config`** (`RegistryConfig | None`, default: `None` ) – Optional configuration. Validates immediately.

Raises:

- `ValueError` – If config is invalid.

### get_store

```
get_store(name: str) -> Store
```

Get a store by its profile name.

Parameters:

- **`name`** (`str`) – The store profile name.

Raises:

- `KeyError` – If no store profile with this name exists.

### close

```
close() -> None
```

Close all instantiated backends.

If any backend raises during `close()`, the remaining backends are still closed. The first exception encountered is re-raised after all backends have been processed.

## register_backend

```
register_backend(
    type_name: str, cls: type[Backend]
) -> None
```

Register a backend class for a given type string.

Parameters:

- **`type_name`** (`str`) – The type identifier (e.g. "local").
- **`cls`** (`type[Backend]`) – The backend class to instantiate.

## See also

- [Choosing a Backend](https://docs.remotestore.dev/stable/guides/choosing-a-backend/index.md) — backend selection and registry usage
- [Configuration example](https://docs.remotestore.dev/stable/tutorial/examples/configuration/index.md) — registry-based store creation

# SFTPUtils

Backend-specific module

The helpers and enums in this module are exclusive to the SFTP backend. Using them ties your code to `SFTPBackend`.

## SFTPUtils

SFTP setup utilities for key loading and host verification.

Groups helpers that assist with SFTP backend configuration:

- `SFTPUtils.load_private_key(...)` -- load RSA keys from file or PEM string
- `SFTPUtils.HostKeyPolicy` -- enum controlling unknown host key behavior
- `SFTPUtils.scan_host_keys(host, port=22)` -- preflight host-key discovery; returns a `known_hosts`-formatted line for committing into a `host.keys` file
- `SFTPUtils.scan_host_algorithms(host, port=22)` -- raw-socket SSH KEXINIT probe; returns the server's algorithm advertisement (kex / host-key / cipher / MAC / compression name-lists) for diagnosing `IncompatiblePeer` failures
- `SFTPUtils.enable_ssh_rsa_compat()` -- restore `ssh-rsa` (SHA-1) acceptance for legacy SFTP servers (see method docstring for the security tradeoff)

Example

```
from remote_store.backends import SFTPUtils, SFTPBackend

key = SFTPUtils.load_private_key("~/.ssh/id_rsa", from_file=True)
backend = SFTPBackend(
    host="sftp.example.com",
    pkey=key,
    host_key_policy=SFTPUtils.HostKeyPolicy.AUTO_ADD,
)
```

## Methods

### load_private_key

```
load_private_key(
    source: str, *, from_file: bool = False
) -> Any
```

Load an RSA private key from a file path or a PEM string.

Parameters:

- **`source`** (`str`) – File path (if from_file=True) or PEM-encoded string.
- **`from_file`** (`bool`, default: `False` ) – If True, treat source as a file path.

Returns:

- `Any` – paramiko.RSAKey

### enable_ssh_rsa_compat

```
enable_ssh_rsa_compat() -> None
```

Guarantee `ssh-rsa` (SHA-1) acceptance across paramiko's four host-key sites.

Appends `ssh-rsa` to four paramiko class attributes if it is missing — future-proofing the consumer against eventual removal, and restoring state if downstream code has cleared it:

1. `paramiko.Transport._preferred_keys` -- KEX host-key-algorithm negotiation.
1. `paramiko.Transport._key_info` -- host-key parsing dispatch.
1. `paramiko.rsakey.RSAKey.HASHES` -- signature-verification hash dispatch.
1. `paramiko.Transport._preferred_pubkeys` -- client RSA public-key authentication signatures.

Empirically verified across paramiko 2.12 / 3.0 / 3.5 / 4.0 / 5.0 (see `sdd/research/research-bk-198-paramiko-ssh-rsa-empirical.md`):

- On paramiko **< 5.0** all four sites contain `ssh-rsa` by default, so a freshly-imported paramiko already negotiates against an `ssh-rsa`-only server. The helper is a no-op (all four guards short-circuit) and is safe to call eagerly for forward compatibility.
- On paramiko **>= 5.0** all four sites have `ssh-rsa` removed. Bare connect to an `ssh-rsa`-only server fails immediately in `Transport._parse_kex_init` with `IncompatiblePeer: no acceptable host key`. Calling this helper at process startup is the only way to restore the connection without monkey-patching `connect_kwargs["disabled_algorithms"]` on every connect.

For KEX / cipher / MAC negotiation failures (e.g. `IncompatiblePeer: no acceptable kex algorithm`), this helper is not the right tool — use `connect_kwargs={"disabled_algorithms": ...}` to widen those instead.

`disabled_algorithms` cannot re-add a default-removed algorithm, so class-level patching is the only forward-compatible path. The patches are idempotent; calling this multiple times in the same process is safe.

Process-global side effect

Every paramiko transport in this process will accept SHA-1 host keys for the lifetime of the process. Only call this if the consumer connects exclusively to servers under your operational control, or if you have explicitly evaluated the tradeoff for every server in the process.

`ssh-rsa` is appended (not prepended) to the preferred lists, so modern algorithms are still negotiated first when the server offers them.

Single-threaded startup only

The four read-then-write patches are not atomic; if two threads enter the helper at the same time they race on the rebind. Call once at process startup before any backend connect, not from a request-handling code path.

Example

```
from remote_store.backends import SFTPUtils

# Call once at process startup, before any SFTPBackend connect.
SFTPUtils.enable_ssh_rsa_compat()
```

### scan_host_keys

```
scan_host_keys(
    host: str, port: int = 22, *, timeout: float = 10.0
) -> str
```

Discover an SFTP server's host key without authenticating.

Opens a `paramiko.Transport` to *host*:*port*, performs key exchange, captures the server's offered host key, closes the connection, and returns the key as a single `known_hosts`-formatted line. No authentication is attempted; only the SSH key-exchange handshake runs.

Use this to populate a committed `host.keys` file for production `STRICT` policy use, without going through a TOFU connect first.

Parameters:

- **`host`** (`str`) – Hostname or IP address of the SFTP server.
- **`port`** (`int`, default: `22` ) – SSH port (default: 22).
- **`timeout`** (`float`, default: `10.0` ) – Socket and KEX timeout in seconds (default: 10).

Returns:

- `str` – A single known_hosts-format line:
- `str` – "\<host_label> \<key_type> \<base64_key>". Per OpenSSH convention,
- `str` – host_label is "\[host\]:port" when port is not 22, and the
- `str` – bare hostname otherwise. The trailing newline is not included.
- `str` – Returns only the negotiated key for one handshake (whichever
- `str` – key type paramiko picked: usually one of ed25519, ecdsa, rsa),
- `str` – not every key the server offers. ssh-keyscan returns one line
- `str` – per offered type by default; this helper does not. If the server
- `str` – offers multiple key types and paramiko later negotiates a
- `str` – different one than the pinned line, the connection fails with
- `str` – BadHostKeyException. Callers that need full-type coverage
- `str` – must call this helper multiple times under different
- `str` – disabled_algorithms settings to force each type in turn.

Raises:

- `SSHException` – Negotiation failed (e.g. legacy server offering only ssh-rsa; call enable_ssh_rsa_compat() first if so).
- `OSError` – Socket-level failure (host unreachable, port refused, DNS error, timeout).

Example

```
from pathlib import Path

from remote_store.backends import SFTPUtils

entry = SFTPUtils.scan_host_keys("sftp.example.com")
Path("host.keys").write_text(entry + "\n")
```

### scan_host_algorithms

```
scan_host_algorithms(
    host: str, port: int = 22, *, timeout: float = 10.0
) -> dict[str, list[str] | str]
```

Discover an SFTP server's algorithm advertisement without authenticating.

Opens a raw TCP socket, exchanges SSH banners, reads the server's first `SSH_MSG_KEXINIT` packet (RFC 4253 § 7.1), parses the ten name-lists it carries, and returns them as a dictionary. No paramiko, no key exchange completes, no authentication is attempted.

Use this to identify the failure shape behind an `IncompatiblePeer` error. `IncompatiblePeer` wraps four distinct negotiation failures (host key / KEX / cipher / MAC), and only the first is addressable by `enable_ssh_rsa_compat()`. Inspecting the four corresponding name-lists in the result tells you which list the server narrowed and to what.

Pure-socket parsing (rather than driving `paramiko.Transport`) is deliberate: the result reflects what the *server* advertises, independent of any process-global paramiko state mutated by `enable_ssh_rsa_compat()` or downstream code.

Parameters:

- **`host`** (`str`) – Hostname or IP address of the SFTP server.
- **`port`** (`int`, default: `22` ) – SSH port (default: 22).
- **`timeout`** (`float`, default: `10.0` ) – Socket timeout in seconds (default: 10).

Returns:

- `dict[str, list[str] | str]` – A dictionary with eleven entries:
- `dict[str, list[str] | str]` – "banner" -- the server's identification string (e.g. "SSH-2.0-OpenSSH_8.9p1").
- `dict[str, list[str] | str]` – The ten RFC 4253 § 7.1 name-lists, each as a Python list\[str\]: kex_algorithms, server_host_key_algorithms, encryption_algorithms_ctos, encryption_algorithms_stoc, mac_algorithms_ctos, mac_algorithms_stoc, compression_algorithms_ctos, compression_algorithms_stoc, languages_ctos, languages_stoc.

Raises:

- `OSError` – Socket-level failure (host unreachable, port refused, timeout, connection reset, unexpected EOF, or the server's first packet was not SSH_MSG_KEXINIT).

Diagnose an `ssh-rsa`-only legacy server

```
from remote_store.backends import SFTPUtils

info = SFTPUtils.scan_host_algorithms("legacy.example.com")
print(info["server_host_key_algorithms"])
# ['ssh-rsa']  -> classic legacy server; on paramiko 5+,
# call SFTPUtils.enable_ssh_rsa_compat() at process startup.
```

Diagnose a narrow-KEX server

```
info = SFTPUtils.scan_host_algorithms("legacy.example.com")
print(info["kex_algorithms"])
# ['diffie-hellman-group14-sha1']  -> KEX narrowing;
# widen via SFTPBackend(connect_kwargs={"disabled_algorithms": ...}).
```

## Enums

### HostKeyPolicy

Bases: `Enum`

Controls how unknown remote host keys are handled.

Attributes:

- **`STRICT`** – Reject unknown hosts (production default).
- **`TRUST_ON_FIRST_USE`** – Save on first connect, verify after.
- **`AUTO_ADD`** – Accept any key (dev/testing ONLY).

Accepts the enum-name forms (`"auto_add"`, `"trust_on_first_use"`, `"STRICT"`) in addition to the canonical value strings (`"auto"`, `"tofu"`, `"strict"`).

## See also

- [SFTP Backend Guide](https://docs.remotestore.dev/stable/guides/backends/sftp/index.md) — connection setup, host key verification, and Key Vault integration
- [SFTP Backend example](https://docs.remotestore.dev/stable/tutorial/examples/sftp-backend/index.md) — end-to-end SFTP usage

# Store

## Store

```
Store(backend: Backend, root_path: str = '')
```

A logical remote folder scoped to a root path.

All path arguments are validated and prefixed with `root_path` before being delegated to the backend. Supports the context-manager protocol (`with Store(...) as s:`) which calls `close()` on exit.

Parameters:

- **`backend`** (`Backend`) – Backend instance (Local, S3, SFTP, Azure, Memory).
- **`root_path`** (`str`, default: `''` ) – Prefix prepended to every path. "" means the backend root.

Root path creation

The root path does not need to exist before constructing the store. `write()` creates intermediate folders implicitly on all backends:

```
store = Store(backend, root_path="brand-new-folder")
store.write("hello.txt", b"works")  # folder created automatically
```

Thread safety

`Store` is immutable after construction and can be shared across threads. Backend thread safety depends on the backend implementation.

______________________________________________________________________

## Reading

Requires `Capability.READ`

All read methods raise `CapabilityNotSupported` on backends that do not declare this capability. Most backends declare it.

### read

```
read(path: str) -> BinaryIO
```

Return a readable binary stream positioned at the start of *path*.

The caller is responsible for closing the stream (or using a `with` block).

Parameters:

- **`path`** (`str`) – Store-relative file path.

Returns:

- `BinaryIO` – Readable binary stream positioned at byte 0.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path is empty, or if path names a directory.

Quality flag: `Capability.LAZY_READ`

When declared, data is fetched lazily — partial reads avoid loading the whole file. Without it, the backend may buffer content before returning the stream.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read the entire file into memory and return `bytes`.

Parameters:

- **`path`** (`str`) – Store-relative file path.

Returns:

- `bytes` – The file content as bytes.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path is empty, or if path names a directory.

Equivalent to `read(path).read()`.

### read_seekable

```
read_seekable(path: str) -> BinaryIO
```

Return a seekable binary stream for random-access reading.

Always returns a seekable stream. On backends that natively return seekable streams from `read()`, this is zero-overhead. On backends whose `read()` is forward-only, the stream comes either from an optimized native implementation as cheap as the passthrough -- the sync Azure backend uses HTTP Range requests, one ranged download per `read()`, with no temp-file spill -- or from a temporary-file spool that copies the object first (HTTP, and an async backend bridged to sync).

Use `read()` for sequential streaming. Use `read_seekable()` when you need `seek()` / `tell()` -- for example, when passing the stream to PyArrow or other analytical readers.

Parameters:

- **`path`** (`str`) – Store-relative file path.

Returns:

- `BinaryIO` – A seekable binary stream positioned at byte 0.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path is empty, or if path names a directory.

Quality flag: `Capability.SEEKABLE_READ`

When declared, the stream is natively seekable. Without it, the Store falls back to a `SpooledTemporaryFile` (RAM-first, spilling to disk beyond the threshold). Backends may provide a more efficient implementation — for example, Azure issues HTTP Range requests instead of spooling.

### read_text

```
read_text(
    path: str,
    *,
    encoding: str = "utf-8",
    errors: str = "strict",
) -> str
```

Read the entire file and decode it as text.

Parameters:

- **`path`** (`str`) – Store-relative file path.
- **`encoding`** (`str`, default: `'utf-8'` ) – Text encoding, any name accepted by codecs.
- **`errors`** (`str`, default: `'strict'` ) – Error handler: "strict", "ignore", "replace", "backslashreplace". See codecs.register_error for custom handlers.

Returns:

- `str` – The file content as str.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path is empty, or if path names a directory.
- `UnicodeDecodeError` – If decoding fails with errors="strict".

Equivalent to `read_bytes(path).decode(encoding, errors)`.

______________________________________________________________________

## Writing

Requires `Capability.WRITE`

`write()` and `write_text()` raise `CapabilityNotSupported` on backends that do not declare this capability. Most backends declare it. `write_atomic()` and `open_atomic()` additionally require `Capability.ATOMIC_WRITE`.

Quality flag: `Capability.WRITE_RESULT_NATIVE`

When declared, the returned `WriteResult` fields (`etag`, `version_id`, `last_modified`, `digest`) are populated from the backend's write response. Without it, only locally computable fields are set.

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write binary content to *path*. Creates parent folders implicitly.

Parameters:

- **`path`** (`str`) – Store-relative file path.
- **`content`** (`WritableContent`) – bytes or readable binary stream (BinaryIO).
- **`overwrite`** (`bool`, default: `False` ) – If False, raises AlreadyExists when path exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user-supplied key/value pairs to store alongside the file. Requires Capability.USER_METADATA. Keys must be non-empty ASCII strings with no leading underscore; total payload must not exceed 2048 bytes.

Returns:

- `WriteResult` – WriteResult with at least path and size populated.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path is empty.
- `ValueError` – If metadata fails shape validation (keys and values must be str with bounded total size).
- `CapabilityNotSupported` – If metadata is non-empty and the backend lacks USER_METADATA.

Backend-conditional argument: `metadata=`

Passing `metadata` raises `CapabilityNotSupported` on backends that do not declare `Capability.USER_METADATA`. Passing `None` or `{}` is safe on all backends.

### write_text

```
write_text(
    path: str,
    text: str,
    *,
    encoding: str = "utf-8",
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write a string to *path*, encoded with the given encoding.

Parameters:

- **`path`** (`str`) – Store-relative file path.
- **`text`** (`str`) – The string to write.
- **`encoding`** (`str`, default: `'utf-8'` ) – Text encoding.
- **`overwrite`** (`bool`, default: `False` ) – If False, raises AlreadyExists when path exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user-supplied key/value pairs (see write()).

Returns:

- `WriteResult` – WriteResult.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path is empty.

Equivalent to `write(path, text.encode(encoding), overwrite=overwrite, metadata=metadata)`.

Backend-conditional argument: `metadata=`

Passing `metadata` raises `CapabilityNotSupported` on backends that do not declare `Capability.USER_METADATA`. Passing `None` or `{}` is safe on all backends.

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write binary content to *path* atomically.

If the write fails or is interrupted, *path* is not left in a partial state.

Parameters:

- **`path`** (`str`) – Store-relative file path.
- **`content`** (`WritableContent`) – bytes or readable binary stream (BinaryIO).
- **`overwrite`** (`bool`, default: `False` ) – If False, raises AlreadyExists when path exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user-supplied key/value pairs (see write()).

Returns:

- `WriteResult` – WriteResult.

Raises:

- `CapabilityNotSupported` – If backend lacks ATOMIC_WRITE.
- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path is empty.

Requires `Capability.ATOMIC_WRITE`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

Backend-conditional argument: `metadata=`

Passing `metadata` raises `CapabilityNotSupported` on backends that do not declare `Capability.USER_METADATA`. Passing `None` or `{}` is safe on all backends.

Info

Most backends implement this as temp-file + rename. See the [Backend Behavior Matrix](#backend-behavior-matrix) for details.

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Context manager that yields a writable binary stream.

The file is committed atomically on successful exit; on exception the partial write is discarded.

Parameters:

- **`path`** (`str`) – Store-relative file path.
- **`overwrite`** (`bool`, default: `False` ) – If False, raises AlreadyExists when path exists.

Returns:

- `Iterator[BinaryIO]` – Writable binary stream.

Raises:

- `CapabilityNotSupported` – If the backend lacks ATOMIC_WRITE.
- `AlreadyExists` – If path exists and overwrite is False.
- `InvalidPath` – If path is empty.

```
with store.open_atomic("data/output.bin", overwrite=True) as f:
    f.write(b"chunk 1")
    f.write(b"chunk 2")
# file is now visible at data/output.bin
```

Requires `Capability.ATOMIC_WRITE`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

______________________________________________________________________

## Deleting

Requires `Capability.DELETE`

All delete methods raise `CapabilityNotSupported` on backends that do not declare this capability.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete a single file.

Parameters:

- **`path`** (`str`) – Store-relative file path.
- **`missing_ok`** (`bool`, default: `False` ) – If True, silently succeeds when path does not exist.

Raises:

- `NotFound` – If the file is missing and missing_ok is False.
- `InvalidPath` – If path is empty, or if path names a directory (regardless of missing_ok).

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete a folder.

Parameters:

- **`path`** (`str`) – Store-relative folder path. Must not be "" (root).
- **`recursive`** (`bool`, default: `False` ) – If True, delete all contents first. If False, raises DirectoryNotEmpty when folder is non-empty.
- **`missing_ok`** (`bool`, default: `False` ) – If True, silently succeeds when path does not exist.

Raises:

- `NotFound` – If the folder is missing and missing_ok is False.
- `DirectoryNotEmpty` – If the folder is non-empty and recursive is False.
- `InvalidPath` – If path is empty (cannot delete the store root), or if path names a file (use delete instead).

______________________________________________________________________

## Listing and Iteration

Requires `Capability.LIST`

All listing methods raise `CapabilityNotSupported` on backends that do not declare this capability.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    pattern: str | None = None,
    max_depth: int | None = None,
) -> Iterator[FileInfo]
```

Yield `FileInfo` objects for files under *path*.

Parameters:

- **`path`** (`str`) – Store-relative folder path.
- **`recursive`** (`bool`, default: `False` ) – Descend into subfolders. Ignored when max_depth is set.
- **`pattern`** (`str | None`, default: `None` ) – Glob pattern to filter filenames (e.g. "\*.csv"). Matched against each file's name (basename only). For full path-based patterns, use ext.glob.glob_files().
- **`max_depth`** (`int | None`, default: `None` ) – Maximum folder depth to include. 0 means files directly in path only; 1 adds files in its immediate subfolders, and so on. None (default) defers to recursive. When set, recursive is ignored.

Returns:

- `Iterator[FileInfo]` – Iterator of FileInfo with store-relative paths.

Raises:

- `ValueError` – If max_depth is negative.

Backend-conditional argument: `max_depth=`

Backends with native depth limiting prune traversal early. Backends that do not support it still return correct results — the Store applies client-side filtering as a safety net.

### list_folders

```
list_folders(
    path: str,
    *,
    pattern: str | None = None,
    max_depth: int | None = None,
) -> Iterator[FolderEntry]
```

Yield subfolders of *path* as `FolderEntry` objects.

Parameters:

- **`path`** (`str`) – Store-relative folder path.
- **`pattern`** (`str | None`, default: `None` ) – Glob pattern to filter folder names (e.g. "raw\_\*"). Matched against each folder's name (basename only) via fnmatch.fnmatch. Filters yielded results only — does not prune BFS traversal, so non-matching folders are still descended into.
- **`max_depth`** (`int | None`, default: `None` ) – Maximum folder depth to include. None or 0 returns immediate children only (default). 1 adds grandchildren, and so on. BFS traversal runs first; pattern filters what is yielded.

Returns:

- `Iterator[FolderEntry]` – Iterator of FolderEntry with .name and .path (store-relative).

Raises:

- `ValueError` – If max_depth is negative.

Backend-conditional argument: `max_depth=`

Backends with native depth limiting prune traversal early. Backends that do not support it still return correct results — the Store applies client-side filtering as a safety net.

### iter_children

```
iter_children(
    path: str,
) -> Iterator[FileInfo | FolderEntry]
```

Yield all immediate children (files and folders) of *path* in a single pass.

Files are yielded as `FileInfo`, folders as `FolderEntry`. Both have `.name` and `.path` attributes (satisfying the `PathEntry` protocol) so callers can iterate uniformly.

Parameters:

- **`path`** (`str`) – Store-relative folder path.

Returns:

- `Iterator[FileInfo | FolderEntry]` – Iterator of FileInfo (files) and FolderEntry (folders).

### glob

```
glob(pattern: str) -> Iterator[FileInfo]
```

Yield files matching a glob *pattern*, using the backend's native glob implementation.

Requires `Capability.GLOB`.

Parameters:

- **`pattern`** (`str`) – Glob pattern (e.g. "data/\*\*/\*.parquet").

Returns:

- `Iterator[FileInfo]` – Iterator of FileInfo with store-relative paths.

Raises:

- `CapabilityNotSupported` – If the backend lacks GLOB.

Requires `Capability.GLOB`

`glob()` raises `CapabilityNotSupported` on backends that do not declare this capability. Check `store.supports(Capability.GLOB)` before calling.

Ordering and laziness

**Ordering is backend-defined** and may vary between backends (e.g. lexicographic on S3, OS-dependent on local filesystems). Callers must not depend on any particular order.

**Results are yielded lazily.** Backends may use pagination internally. Memory usage stays bounded for large directories.

______________________________________________________________________

## File Operations

Requires `Capability.MOVE` / `Capability.COPY`

`move()` requires `Capability.MOVE`; `copy()` requires `Capability.COPY`. Each raises `CapabilityNotSupported` on backends that do not declare the respective capability.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move (rename) a file from *src* to *dst*.

File-only -- to move a folder, iterate its contents.

Parameters:

- **`src`** (`str`) – Source file path.
- **`dst`** (`str`) – Destination file path.
- **`overwrite`** (`bool`, default: `False` ) – If False, raises AlreadyExists when dst exists.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists and overwrite is False.
- `InvalidPath` – If src or dst is empty, or if src names a directory.

Requires `Capability.MOVE`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

Atomicity

Atomicity is backend-dependent. Local uses `os.replace` (atomic on same filesystem). S3 and Azure use copy-then-delete (not atomic). SFTP atomicity depends on the server. Check `store.supports(Capability.ATOMIC_MOVE)` to query this at runtime.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy a file from *src* to *dst*.

File-only -- to copy a folder, iterate its contents.

Parameters:

- **`src`** (`str`) – Source file path.
- **`dst`** (`str`) – Destination file path.
- **`overwrite`** (`bool`, default: `False` ) – If False, raises AlreadyExists when dst exists.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists and overwrite is False.
- `InvalidPath` – If src or dst is empty, or if src names a directory.

Requires `Capability.COPY`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

Metadata preservation

Metadata preservation is backend-dependent. S3 copies metadata; local preserves metadata (`copy2`); SFTP does not (stream copy).

______________________________________________________________________

## Metadata

Partially requires `Capability.METADATA`

`head()` and `get_file_info()` require `Capability.METADATA`. `get_folder_info()` requires `Capability.METADATA` without `max_depth`, or `Capability.LIST` when `max_depth` is set. `exists()`, `is_file()`, and `is_folder()` are always available.

### head

```
head(path: str) -> WriteResult
```

Return a `WriteResult` snapshot of *path* via a metadata lookup.

Gated on `Capability.METADATA` only — works on read-only backends that declare `METADATA`. The returned `WriteResult` has `source="sidecar"` (populated from a `get_file_info()` call, not from a write response).

Parameters:

- **`path`** (`str`) – Store-relative file path.

Returns:

- `WriteResult` – WriteResult with source="sidecar".

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path is empty, or if path names a directory.
- `CapabilityNotSupported` – If the backend lacks METADATA.

Requires `Capability.METADATA`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

### exists

```
exists(path: str) -> bool
```

Return `True` if *path* exists (file or folder).

Never raises `NotFound` — returns `False` for missing paths instead. Also returns `False` if any ancestor of *path* is a file (file-as-directory-component), as traversal cannot proceed.

Parameters:

- **`path`** (`str`) – Store-relative path.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if *path* exists and is a file.

Returns `False` if any ancestor of *path* is a file (file-as-directory-component).

Parameters:

- **`path`** (`str`) – Store-relative path.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if *path* exists and is a folder.

Returns `False` if any ancestor of *path* is a file (file-as-directory-component).

Parameters:

- **`path`** (`str`) – Store-relative path.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return a `FileInfo` with size, modification time, and content type for a single file.

Parameters:

- **`path`** (`str`) – Store-relative file path.

Returns:

- `FileInfo` – FileInfo.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path is empty, or if path names a directory.

Requires `Capability.METADATA`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

### get_folder_info

```
get_folder_info(
    path: str, *, max_depth: int | None = None
) -> FolderInfo
```

Return a `FolderInfo` with aggregated size and file count for a folder.

Parameters:

- **`path`** (`str`) – Store-relative folder path.
- **`max_depth`** (`int | None`, default: `None` ) – Maximum folder depth to aggregate. 0 means files directly in path only; 1 adds files in its immediate subfolders, and so on. None (default) performs a full recursive traversal via the backend.

Returns:

- `FolderInfo` – FolderInfo.

Raises:

- `NotFound` – If the folder does not exist.
- `InvalidPath` – If path names a file (use get_file_info instead).
- `ValueError` – If max_depth is negative.

Capability depends on `max_depth`

Without `max_depth`: requires `Capability.METADATA`. With `max_depth` set: requires `Capability.LIST` — works on backends that lack `METADATA`.

Backend-conditional argument: `max_depth=`

Backends with native depth limiting prune traversal early. Backends that do not support it still return correct results — the Store applies client-side filtering as a safety net.

______________________________________________________________________

## Introspection

### resolve

```
resolve(key: str) -> ResolutionPlan
```

Return a `ResolutionPlan` describing how *key* maps to storage.

Delegates to the backend's `resolve()` and rebases the key so that `plan.key` is the store-relative key, not the backend-relative path.

Parameters:

- **`key`** (`str`) – Store-relative path. "" resolves the store root.

Returns:

- `ResolutionPlan` – A frozen ResolutionPlan.

Info

`resolve()` is a pure introspection method — it performs no I/O and is never called implicitly by other Store methods. The returned [`ResolutionPlan`](https://docs.remotestore.dev/stable/reference/api/models/index.md) describes how a key maps to its storage location.

______________________________________________________________________

## Lifecycle

### ping

```
ping() -> None
```

Verify that the backend is reachable.

Raises:

- `PermissionDenied` – If credentials are invalid.
- `NotFound` – If the bucket, container, or root path does not exist.
- `BackendUnavailable` – If the backend cannot be reached.

### close

```
close() -> None
```

Release backend resources.

Called automatically when used as a context manager.

### child

```
child(subpath: str) -> Store
```

Return a new `Store` scoped to *subpath* under the current root.

The child shares the same backend instance.

Parameters:

- **`subpath`** (`str`) – Path segment to append to the current root.

Returns:

- `Store` – Store.

Raises:

- `InvalidPath` – If subpath is empty, contains .. segments, or includes null bytes.

```
data = store.child("data/2024")
data.list_files("")  # lists files under <root>/data/2024/
```

______________________________________________________________________

## Interop (Backend-Specific)

Backend-specific methods

Methods in this section expose backend internals. Using them ties your code to a specific backend. For portable alternatives, use the methods above.

### unwrap

```
unwrap(type_hint: type[T]) -> T
```

Return the backend's native client object, cast to *type_hint*.

Parameters:

- **`type_hint`** (`type[T]`) – The expected type of the native client (e.g. pyarrow.fs.FileSystem).

Returns:

- `T` – The native client.

Raises:

- `CapabilityNotSupported` – If the backend cannot provide the requested type.

```
arrow_fs = store.unwrap(pyarrow.fs.FileSystem)
```

### native_path

```
native_path(key: str) -> str
```

Convert a store-relative *key* to the backend's native path representation.

Inverse of `to_key()`.

Parameters:

- **`key`** (`str`) – Store-relative path.

Returns:

- `str` – Backend-native path (e.g. S3 object key, local filesystem path).

### to_key

```
to_key(path: str) -> str
```

Convert a backend-native *path* to a store-relative key.

Inverse of `native_path()`.

Parameters:

- **`path`** (`str`) – Backend-native path string.

Returns:

- `str` – Store-relative key.

Raises:

- `InvalidPath` – If the path does not belong to this store.

### supports

```
supports(capability: Capability) -> bool
```

Check whether the backend supports a given `Capability`.

Parameters:

- **`capability`** (`Capability`) – A Capability enum member.

Returns:

- `bool` – True if the backend declares this capability.

```
if store.supports(Capability.GLOB):
    results = store.glob("**/*.csv")
```

Info

`supports()` itself is portable — it works on all backends. Only the capability-gated methods it guards are backend-specific.

______________________________________________________________________

## Backend Behavior Matrix

How key operations behave across backends. Verify against actual code before relying on these in production.

| Behavior                    | [Local](https://docs.remotestore.dev/stable/guides/backends/local/index.md) | [S3](https://docs.remotestore.dev/stable/guides/backends/s3/index.md) | [S3-PyArrow](https://docs.remotestore.dev/stable/guides/backends/s3-pyarrow/index.md) | [SFTP](https://docs.remotestore.dev/stable/guides/backends/sftp/index.md) | [Azure](https://docs.remotestore.dev/stable/guides/backends/azure/index.md) | [Memory](https://docs.remotestore.dev/stable/guides/backends/memory/index.md) | [HTTP](https://docs.remotestore.dev/stable/guides/backends/http/index.md) | [SQLBlob](https://docs.remotestore.dev/stable/guides/backends/sql-blob/index.md) | [SQLQuery](https://docs.remotestore.dev/stable/guides/backends/sql-query/index.md) |
| --------------------------- | --------------------------------------------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| `move()` atomicity          | Atomic (same FS)                                                            | Copy+delete                                                           | Copy+delete                                                                           | Server-dependent                                                          | Copy+delete                                                                 | Atomic                                                                        | —                                                                         | Atomic (SQL transaction)                                                         | —                                                                                  |
| `copy()` preserves metadata | Yes (`copy2`)                                                               | Yes                                                                   | Yes                                                                                   | —                                                                         | Yes                                                                         | —                                                                             | —                                                                         | Yes                                                                              | —                                                                                  |
| `write_atomic()` mechanism  | temp+rename                                                                 | Direct PUT (atomic)                                                   | Direct PUT (atomic)                                                                   | temp+rename                                                               | Direct PUT or temp+rename                                                   | Direct (atomic)                                                               | —                                                                         | Direct (atomic)                                                                  | —                                                                                  |
| Native `glob()`             | Yes                                                                         | Yes                                                                   | Yes                                                                                   | —                                                                         | Yes                                                                         | —                                                                             | —                                                                         | Yes (SQL GLOB/LIKE)                                                              | Yes (in-memory)                                                                    |
| `list_files()` ordering     | OS-dependent                                                                | Lexicographic                                                         | Lexicographic                                                                         | OS-dependent                                                              | Lexicographic                                                               | Insertion order                                                               | —                                                                         | DB-dependent                                                                     | Lexicographic                                                                      |

## See also

- [Getting Started](https://docs.remotestore.dev/stable/tutorial/getting-started/index.md) — step-by-step guide to reading and writing files
- [Concurrency](https://docs.remotestore.dev/stable/explanation/concurrency/index.md) — thread safety, atomic writes, and move semantics
- [Quickstart example](https://docs.remotestore.dev/stable/tutorial/examples/quickstart/index.md) — minimal config, write, and read

# Async API

The `remote_store.aio` namespace is the async counterpart of the core API. Its layout mirrors the [synchronous reference](https://docs.remotestore.dev/stable/reference/api/index.md): a Store, a Backend protocol, backend implementations, and extensions — each the async twin of its sync sibling, plus the adapters that bridge the two worlds.

For usage patterns (streaming, FastAPI integration, the thread-pool bridge), see the [Async Store Guide](https://docs.remotestore.dev/stable/guides/async/index.md).

## Sync ↔ async map

| Synchronous (`remote_store`)                                                                  | Asynchronous (`remote_store.aio`)                                                                            |
| --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| [`Store`](https://docs.remotestore.dev/stable/reference/api/store/index.md)                   | [`AsyncStore`](https://docs.remotestore.dev/stable/reference/api/aio/store/index.md)                         |
| [`Backend`](https://docs.remotestore.dev/stable/reference/api/backend/index.md)               | [`AsyncBackend`](https://docs.remotestore.dev/stable/reference/api/aio/backend/index.md)                     |
| [`MemoryBackend`](https://docs.remotestore.dev/stable/reference/api/backends/memory/index.md) | [`AsyncMemoryBackend`](https://docs.remotestore.dev/stable/reference/api/aio/backends/memory/index.md)       |
| [`AzureBackend`](https://docs.remotestore.dev/stable/reference/api/backends/azure/index.md)   | [`AsyncAzureBackend`](https://docs.remotestore.dev/stable/reference/api/aio/backends/azure/index.md)         |
| —                                                                                             | [`GraphBackend`](https://docs.remotestore.dev/stable/reference/api/aio/backends/graph/index.md) (async-only) |
| [`ext.write`](https://docs.remotestore.dev/stable/reference/api/extensions/write/index.md)    | [`aio.ext.write`](https://docs.remotestore.dev/stable/reference/api/aio/extensions/write/index.md)           |

## Core

| Class                                                                                  | Description                                                            |
| -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| [AsyncStore](https://docs.remotestore.dev/stable/reference/api/aio/store/index.md)     | Async counterpart to `Store` with coroutine methods for all operations |
| [AsyncBackend](https://docs.remotestore.dev/stable/reference/api/aio/backend/index.md) | Abstract base class for native async backends                          |

## Backends

| Class                                                                                                | Description                                                 |
| ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| [AsyncMemoryBackend](https://docs.remotestore.dev/stable/reference/api/aio/backends/memory/index.md) | In-memory async backend for testing                         |
| [AsyncAzureBackend](https://docs.remotestore.dev/stable/reference/api/aio/backends/azure/index.md)   | Native async Azure Blob Storage and ADLS Gen2               |
| [GraphBackend](https://docs.remotestore.dev/stable/reference/api/aio/backends/graph/index.md)        | Microsoft Graph backend (OneDrive, SharePoint, Teams files) |

Graph is async-only

`GraphBackend` has no synchronous twin. It appears here, not in the [synchronous Backends](https://docs.remotestore.dev/stable/reference/api/backends/index.md) section. To drive it from sync code, wrap it with [`AsyncBackendSyncAdapter`](https://docs.remotestore.dev/stable/reference/api/aio/adapters/index.md).

## Adapters & types

| Symbol                                                                                                             | Description                                                                  |
| ------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- |
| [SyncBackendAdapter](https://docs.remotestore.dev/stable/reference/api/aio/adapters/#syncbackendadapter)           | Wraps any synchronous backend for async use via thread-pool executor         |
| [AsyncBackendSyncAdapter](https://docs.remotestore.dev/stable/reference/api/aio/adapters/#asyncbackendsyncadapter) | Wraps any `AsyncBackend` as a synchronous `Backend` via a private event loop |
| [AsyncWritableContent](https://docs.remotestore.dev/stable/reference/api/aio/adapters/#asyncwritablecontent)       | Type alias: `bytes` or `AsyncIterator[bytes]`                                |

## Utilities

| Symbol                                                                                         | Description                                                           |
| ---------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| [GraphAuth](https://docs.remotestore.dev/stable/reference/api/aio/backends/graph/#graphauth)   | MSAL token provider (client-credentials / device-code) for Graph      |
| [GraphUtils](https://docs.remotestore.dev/stable/reference/api/aio/backends/graph/#graphutils) | Resolve a Graph `drive_id` from OneDrive / SharePoint / Teams targets |

## Extensions

| Module                                                                                           | Description                                                     |
| ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- |
| [aio.ext.write](https://docs.remotestore.dev/stable/reference/api/aio/extensions/write/index.md) | Async write helpers with guaranteed client-side content hashing |

## See also

- [API Reference](https://docs.remotestore.dev/stable/reference/api/index.md) — the full reference index
- [Async Store Guide](https://docs.remotestore.dev/stable/guides/async/index.md) — usage patterns, streaming, FastAPI integration
- [Async-sync bridges](https://docs.remotestore.dev/stable/guides/async-sync-bridges/index.md) — adapter behaviour contract
- [Concurrency](https://docs.remotestore.dev/stable/explanation/concurrency/index.md) — thread safety and atomicity semantics

# Adapters & types

Bridges between the synchronous and asynchronous backend worlds, plus the async content type alias. The adapters let a sync backend run under [`AsyncStore`](https://docs.remotestore.dev/stable/reference/api/aio/store/index.md) and an async backend run under the synchronous [`Store`](https://docs.remotestore.dev/stable/reference/api/store/index.md). See [Async-sync bridges](https://docs.remotestore.dev/stable/guides/async-sync-bridges/index.md) for when to reach for each.

## SyncBackendAdapter

Wraps any synchronous `Backend` as an `AsyncBackend` by dispatching each blocking call to the default executor via `asyncio.to_thread`. `AsyncStore` auto-wraps sync backends on construction; explicit construction is only required when you want to introspect the adapter.

## SyncBackendAdapter

```
SyncBackendAdapter(backend: Backend)
```

Wraps a synchronous `Backend` as an `AsyncBackend`.

Every blocking call is dispatched to the default executor via `asyncio.to_thread`, keeping the event loop responsive.

Parameters:

- **`backend`** (`Backend`) – The synchronous backend instance to wrap.

### name

```
name: str
```

Backend identifier, forwarded from the wrapped backend.

### capabilities

```
capabilities: CapabilitySet
```

Capability set, forwarded from the wrapped backend.

### to_key

```
to_key(native_path: str) -> str
```

Convert a native path to a backend-relative key.

### native_path

```
native_path(path: str) -> str
```

Convert a backend-relative key to the native path.

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a resolution plan for *path*.

### unwrap

```
unwrap(type_hint: type[T]) -> T
```

Return the native backend handle.

### exists

```
exists(path: str) -> bool
```

Check if a file or folder exists.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if *path* is an existing file.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if *path* is an existing folder.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read the full content of a file as bytes.

Raises:

- `InvalidPath` – If path names an existing directory.
- `NotFound` – If the file does not exist.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Get metadata for a file.

Raises:

- `InvalidPath` – If path names an existing directory.
- `NotFound` – If the file does not exist.

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Get metadata for a folder.

Raises:

- `InvalidPath` – If path names an existing file.
- `NotFound` – If the folder does not exist.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename a file.

Raises:

- `InvalidPath` – If src names a directory or dst names an existing directory.
- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy a file.

Raises:

- `InvalidPath` – If src names a directory or dst names an existing directory.
- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete a file.

Raises:

- `NotFound` – If the file is missing and missing_ok is False.
- `InvalidPath` – If the path is empty, or if path names a directory (regardless of missing_ok).

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete a folder.

Raises:

- `InvalidPath` – If path names an existing file (regardless of missing_ok — a type mismatch is not a missing file).
- `NotFound` – If the folder is missing and missing_ok is False.
- `DirectoryNotEmpty` – If non-empty and recursive is False.

### check_health

```
check_health() -> None
```

Verify the backend is reachable.

### read

```
read(path: str) -> AsyncIterator[bytes]
```

Open a file for reading and yield chunks asynchronously.

Raises:

- `InvalidPath` – If path names an existing directory.
- `NotFound` – If the file does not exist.

### write

```
write(
    path: str,
    content: AsyncWritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write content to a file.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory.

### write_atomic

```
write_atomic(
    path: str,
    content: AsyncWritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write content atomically via temp file + rename.

Raises:

- `CapabilityNotSupported` – If backend lacks ATOMIC_WRITE.
- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> AsyncIterator[FileInfo]
```

List files under *path*.

### list_folders

```
list_folders(path: str) -> AsyncIterator[FolderEntry]
```

List immediate subfolders under *path*.

### glob

```
glob(pattern: str) -> AsyncIterator[FileInfo]
```

Match files against a glob pattern.

### iter_children

```
iter_children(
    path: str,
) -> AsyncIterator[FileInfo | FolderEntry]
```

Yield both files and folders under *path*.

### aclose

```
aclose() -> None
```

Release resources held by the wrapped backend.

## AsyncBackendSyncAdapter

Wraps any `AsyncBackend` as a synchronous `Backend` by running a private event loop on a dedicated daemon thread for the adapter's lifetime. See [Async-sync bridges](https://docs.remotestore.dev/stable/guides/async-sync-bridges/index.md) for the full behaviour contract and the [async-to-sync adapter decision record](https://github.com/haalfi/remote-store/blob/master/sdd/adrs/0025-async-to-sync-backend-adapter.md) for the design rationale.

## AsyncBackendSyncAdapter

```
AsyncBackendSyncAdapter(async_backend: AsyncBackend)
```

Wraps an `AsyncBackend` as a synchronous `Backend`.

One private `asyncio` event loop per adapter instance, running on a dedicated daemon thread for the adapter's lifetime. Sync methods submit coroutines via `asyncio.run_coroutine_threadsafe` and block on the returned `concurrent.futures.Future`.

Construction does not enter the wrapped backend's async context manager -- callers that need `__aenter__` semantics should use `AsyncStore` directly.

Parameters:

- **`async_backend`** (`AsyncBackend`) – The async backend instance to wrap.

### name

```
name: str
```

Backend identifier, forwarded from the wrapped async backend.

### capabilities

```
capabilities: CapabilitySet
```

Capabilities reported by the wrapped backend, with stream-shape translation applied.

`SEEKABLE_READ` is masked off unconditionally — the chunk-pull stream this adapter returns is forward-only. `USER_METADATA` and `WRITE_RESULT_NATIVE` are forwarded from the inner async backend unchanged; the `write*()` methods accept and forward `metadata=`.

### unwrap

```
unwrap(type_hint: type[T]) -> T
```

Return a sync-safe native handle if the wrapped backend provides one.

By default raises `CapabilityNotSupported` because async-SDK handles are bound to the adapter's private event loop and cannot be used safely from the caller's thread.

Wrapped backends that can expose a sync-safe handle should implement `_SyncSafeHandleProvider` and return it from `_SyncSafeHandleProvider.sync_safe_unwrap`.

Parameters:

- **`type_hint`** (`type[T]`) – The type of handle to retrieve; passed through to \_SyncSafeHandleProvider.sync_safe_unwrap for backends that support the exemption.

Returns:

- `T` – A sync-safe handle of the type requested via type_hint.

Raises:

- `CapabilityNotSupported` – If the wrapped backend does not implement \_SyncSafeHandleProvider.

Example

```
class _SafeBackend(AsyncBackend, _SyncSafeHandleProvider):
    def sync_safe_unwrap(self, type_hint):
        return self._sync_client

adapter = AsyncBackendSyncAdapter(_SafeBackend())
client = adapter.unwrap(SyncClient)
```

### check_health

```
check_health() -> None
```

Submit a connectivity probe to the wrapped async backend.

Not a no-op: the probe is forwarded to the wrapped `AsyncBackend`, and any connectivity error it raises reaches the sync caller unchanged.

Returns:

- `None` – None on success.

Raises:

- `BackendUnavailable` – Or another backend-specific error if the wrapped backend's health check fails.
- `RuntimeError` – If the adapter is closed or called from a running event loop.

Example

```
try:
    adapter.check_health()
except BackendUnavailable:
    ...  # handle connectivity failure
```

### read

```
read(path: str) -> BinaryIO
```

Open *path* for reading and return a forward-only chunk-pull stream.

The returned stream pulls one async chunk per `read()` call; the file is never fully materialised in memory. At most one `__anext__` is in-flight at a time.

The stream is forward-only: `seekable()` returns `False` and no `seek` / `tell` / `fileno` methods are exposed. Closing via the context manager or `close()` submits `aclose()` on the underlying async iterator so backend resources are released promptly.

Parameters:

- **`path`** (`str`) – Backend-relative key of the file to read.

Returns:

- `BinaryIO` – A forward-only io.RawIOBase stream over the file
- `BinaryIO` – contents.

Raises:

- `RuntimeError` – If the adapter is closed or called from a running event loop.
- `NotFound` – If path does not exist (propagated verbatim from the wrapped backend).

Example

```
with adapter.read("data/report.csv") as stream:
    header = stream.read(512)
```

### close

```
close(timeout: float | None = 30.0) -> None
```

Drain in-flight work, stop the loop, and join the daemon thread.

Drain order: submit `aclose` on the wrapped backend → loop-drain in-flight tasks (repeating while new tasks appear, see below) → stop the private loop → join the daemon thread.

The drain step repeats `_drain_tasks` until the private loop is quiet, **narrowing** (not eliminating) the window where a caller that passed `_guard` *before* the closed flag was set can still submit a coroutine. Each pass snapshots outstanding tasks and waits for them; new tasks that arrive between snapshot and completion trigger another pass. A residual TOCTOU gap remains: a thread that passes `_guard` after the final empty-snapshot check but before the loop is stopped will have its coroutine silently discarded when the loop stops — eliminating this gap would require serialising all submits against a shutdown lock.

If *timeout* expires before the loop is drained, a single `WARNING` record is emitted (message stem `"AsyncBackendSyncAdapter close timed out"`) and the daemon thread is left for process-exit reaping. Idempotent: subsequent calls are no-ops.

Parameters:

- **`timeout`** (`float | None`, default: `30.0` ) – Maximum seconds to wait for in-flight work to finish and the background thread to join. None means wait indefinitely. Defaults to 30.0.

## AsyncWritableContent

## AsyncWritableContent

```
AsyncWritableContent = bytes | AsyncIterator[bytes]
```

## See also

- [AsyncStore](https://docs.remotestore.dev/stable/reference/api/aio/store/index.md) — the async Store the adapters plug into
- [AsyncBackend](https://docs.remotestore.dev/stable/reference/api/aio/backend/index.md) — the async backend protocol
- [Async-sync bridges](https://docs.remotestore.dev/stable/guides/async-sync-bridges/index.md) — choosing a direction

# AsyncBackend

`AsyncBackend` is the abstract base class for native async backends — the async counterpart of [`Backend`](https://docs.remotestore.dev/stable/reference/api/backend/index.md). Subclass it to implement a backend that talks to its store with `async`/`await`. It lives in `remote_store.aio`.

## AsyncBackend

Bases: `ABC`

Abstract base class for all async storage backends.

Every backend must implement all abstract methods. Backend-native exceptions must never leak -- they must be mapped to `remote_store` errors.

Implementing an async backend

Subclass `AsyncBackend` and implement all abstract methods. Map every backend-native exception to a `remote_store` error — native exceptions must never leak to callers.

## Identity

### name

```
name: str
```

Unique identifier for this backend type (e.g. `'local'`, `'s3'`).

### capabilities

```
capabilities: CapabilitySet
```

Declared capabilities of this backend.

## Existence

### exists

```
exists(path: str) -> bool
```

Check if a file or folder exists. Never raises `NotFound`.

Parameters:

- **`path`** (`str`) – Backend-relative key, or "" for the root.

Returns:

- `bool` – True if a file or folder exists at path.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if `path` is an existing file.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `bool` – True if path exists and is a file.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if `path` is an existing folder.

Parameters:

- **`path`** (`str`) – Backend-relative key, or "" for the root.

Returns:

- `bool` – True if path exists and is a folder.

## Reading

Requires `Capability.READ`

All read methods raise `CapabilityNotSupported` on backends that do not declare this capability. Most backends declare it.

### read

```
read(path: str) -> AsyncIterator[bytes]
```

Open a file for reading and return an async iterator of byte chunks.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `AsyncIterator[bytes]` – An async iterator yielding byte chunks.

Raises:

- `InvalidPath` – If path names an existing directory.
- `NotFound` – If the file does not exist.

Quality flag: `Capability.LAZY_READ`

When declared, data is fetched lazily — partial reads avoid loading the whole file. Without it, the backend may buffer content before returning the stream.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read the full content of a file as bytes.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `bytes` – The file content.

Raises:

- `InvalidPath` – If path names an existing directory.
- `NotFound` – If the file does not exist.

## Writing

Requires `Capability.WRITE`

`write()` raises `CapabilityNotSupported` on backends that do not declare this capability. Most backends declare it. `write_atomic()` additionally requires `Capability.ATOMIC_WRITE`.

Quality flag: `Capability.WRITE_RESULT_NATIVE`

When declared, the returned `WriteResult` fields (`etag`, `version_id`, `last_modified`, `digest`) are populated from the backend's write response. Without it, only locally computable fields are set.

### write

```
write(
    path: str,
    content: AsyncWritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write content to a file.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`content`** (`AsyncWritableContent`) – Data to write.
- **`overwrite`** (`bool`, default: `False` ) – If False, raise if file already exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user-defined string metadata.

Returns:

- `WriteResult` – A WriteResult with size, path, and optional native fields.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or if any slash-aligned ancestor of path exists as a regular file. Flat-namespace backends (S3, Azure non-HNS, SQL) cannot detect a file ancestor in O(1) and skip the check by default; the per-backend reject_write_under_file_ancestor opt-in enables it.

Backend-conditional argument: `metadata=`

Passing `metadata` raises `CapabilityNotSupported` on backends that do not declare `Capability.USER_METADATA`. Passing `None` or `{}` is safe on all backends.

### write_atomic

```
write_atomic(
    path: str,
    content: AsyncWritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write content atomically via temp file + rename.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`content`** (`AsyncWritableContent`) – Data to write.
- **`overwrite`** (`bool`, default: `False` ) – If False, raise if file already exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user-defined string metadata.

Returns:

- `WriteResult` – A WriteResult with size, path, and optional native fields.

Raises:

- `CapabilityNotSupported` – If backend lacks ATOMIC_WRITE.
- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or if any slash-aligned ancestor of path exists as a regular file (see write).

Requires `Capability.ATOMIC_WRITE`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

Backend-conditional argument: `metadata=`

Passing `metadata` raises `CapabilityNotSupported` on backends that do not declare `Capability.USER_METADATA`. Passing `None` or `{}` is safe on all backends.

## Deleting

Requires `Capability.DELETE`

All delete methods raise `CapabilityNotSupported` on backends that do not declare this capability.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete a file.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`missing_ok`** (`bool`, default: `False` ) – If True, do not raise when the file is absent.

Raises:

- `NotFound` – If the file is missing and missing_ok is False.
- `InvalidPath` – If the path is empty, or if path names a directory (regardless of missing_ok).

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete a folder.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`recursive`** (`bool`, default: `False` ) – If True, delete all contents first.
- **`missing_ok`** (`bool`, default: `False` ) – If True, do not raise when absent.

Raises:

- `InvalidPath` – If path names an existing file (regardless of missing_ok — a type mismatch is not a missing file).
- `NotFound` – If the folder is missing and missing_ok is False.
- `DirectoryNotEmpty` – If non-empty and recursive is False.

## Listing and Iteration

Requires `Capability.LIST`

All listing methods raise `CapabilityNotSupported` on backends that do not declare this capability.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> AsyncIterator[FileInfo]
```

List files under `path`.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.
- **`recursive`** (`bool`, default: `False` ) – If True, include files in all subdirectories.
- **`max_depth`** (`int | None`, default: `None` ) – Optional maximum folder depth to traverse. When set, backends that support native depth limiting prune traversal early. Backends that ignore this parameter still produce correct results -- the Store applies client-side filtering as a safety net. None (default) defers to recursive.

Returns:

- `AsyncIterator[FileInfo]` – An async iterator of FileInfo objects.

Backend-conditional argument: `max_depth=`

Backends with native depth limiting prune traversal early. Backends that do not support it still return correct results — the Store applies client-side filtering as a safety net.

### list_folders

```
list_folders(path: str) -> AsyncIterator[FolderEntry]
```

List immediate subfolders under `path`.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.

Returns:

- `AsyncIterator[FolderEntry]` – An async iterator of FolderEntry objects with .name and .path.

### iter_children

```
iter_children(
    path: str,
) -> AsyncIterator[FileInfo | FolderEntry]
```

Yield both files and folders under `path` in a single pass.

Files are yielded as `FileInfo` objects, folders as `FolderEntry` objects. The default implementation chains `list_files()` and `list_folders()`. Backends that can fetch both in a single I/O call should override this for efficiency.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.

Returns:

- `AsyncIterator[FileInfo | FolderEntry]` – An async iterator of FileInfo (files) and FolderEntry (folders).

### glob

```
glob(pattern: str) -> AsyncIterator[FileInfo]
```

Match files against a glob pattern.

Non-abstract -- backends with native glob support override this and add `Capability.GLOB` to their capability set.

Parameters:

- **`pattern`** (`str`) – Glob pattern (e.g., "data/\*.csv", "\*\*/\*.txt").

Raises:

- `CapabilityNotSupported` – If the backend lacks GLOB.

Requires `Capability.GLOB`

`glob()` raises `CapabilityNotSupported` on backends that do not declare this capability.

## Metadata

Requires `Capability.METADATA`

`get_file_info()` requires `Capability.METADATA`. `get_folder_info()` requires `Capability.METADATA` without `max_depth`, or `Capability.LIST` when `max_depth` is set.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Get metadata for a file.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `FileInfo` – A FileInfo with size, modification time, etc.

Raises:

- `InvalidPath` – If path names an existing directory.
- `NotFound` – If the file does not exist.

Requires `Capability.METADATA`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Get metadata for a folder.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.

Returns:

- `FolderInfo` – A FolderInfo with file count, total size, etc.

Raises:

- `InvalidPath` – If path names an existing file.
- `NotFound` – If the folder does not exist.

Capability depends on `max_depth`

Without `max_depth`: requires `Capability.METADATA`. With `max_depth` set: requires `Capability.LIST` — works on backends that lack `METADATA`.

## File Operations

Requires `Capability.MOVE` / `Capability.COPY`

`move()` requires `Capability.MOVE`; `copy()` requires `Capability.COPY`. Each raises `CapabilityNotSupported` on backends that do not declare the respective capability.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename a file.

`src == dst` is a no-op (the file is preserved unchanged).

Parameters:

- **`src`** (`str`) – Backend-relative source key.
- **`dst`** (`str`) – Backend-relative destination key.
- **`overwrite`** (`bool`, default: `False` ) – If True, replace any existing file at dst.

Raises:

- `InvalidPath` – If src names a directory, dst names an existing directory, or any slash-aligned ancestor of dst exists as a regular file.
- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.

Requires `Capability.MOVE`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

Quality flag: `Capability.ATOMIC_MOVE`

When declared, `move()` is guaranteed atomic under concurrent access. Check `store.supports(Capability.ATOMIC_MOVE)` to query at runtime.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy a file.

`src == dst` is a no-op (the file is preserved unchanged).

Parameters:

- **`src`** (`str`) – Backend-relative source key.
- **`dst`** (`str`) – Backend-relative destination key.
- **`overwrite`** (`bool`, default: `False` ) – If True, replace any existing file at dst.

Raises:

- `InvalidPath` – If src names a directory, dst names an existing directory, or any slash-aligned ancestor of dst exists as a regular file.
- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.

Requires `Capability.COPY`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

## Lifecycle

### aclose

```
aclose() -> None
```

Release resources. Default is a no-op.

### check_health

```
check_health() -> None
```

Verify the backend is reachable and credentials are valid.

The default implementation is a no-op (always succeeds). Backends override this to perform a lightweight, non-destructive connectivity check using the cheapest possible read-only operation.

Raises:

- `PermissionDenied` – If credentials are invalid.
- `NotFound` – If the bucket, container, or root path does not exist.
- `BackendUnavailable` – If the backend cannot be reached.

## Introspection

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` describing how *path* maps to storage.

Pure introspection -- no I/O is performed. Backends override this to populate `details` with backend-specific context.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – A frozen ResolutionPlan with kind, backend,
- `ResolutionPlan` – key, native_path, and details.

## Interop (Backend-Specific)

Backend-specific methods

Methods in this section expose backend internals. Using them ties your code to a specific backend. For portable alternatives, use the methods above.

### unwrap

```
unwrap(type_hint: type[T]) -> T
```

Return the native backend handle if it matches the requested type.

Parameters:

- **`type_hint`** (`type[T]`) – The expected type (e.g., fsspec.AbstractFileSystem).

Raises:

- `CapabilityNotSupported` – If backend cannot provide the requested type.

### native_path

```
native_path(path: str) -> str
```

Convert a backend-relative key to the backend-native path.

The inverse of `to_key()`. The default implementation is the identity function -- backends with a native root (bucket, base_path) override this to prepend their prefix.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `str` – Backend-native path usable with the native handle from unwrap().

### to_key

```
to_key(native_path: str) -> str
```

Convert a backend-native path to a backend-relative key.

Strips the backend's own root/prefix from the path. The default implementation is the identity function -- backends with a native root (filesystem path, bucket prefix, base_path) override this.

Parameters:

- **`native_path`** (`str`) – Absolute or backend-native path string.

Returns:

- `str` – Path relative to the backend's root.

## See also

- [Backend](https://docs.remotestore.dev/stable/reference/api/backend/index.md) — synchronous counterpart
- [AsyncStore](https://docs.remotestore.dev/stable/reference/api/aio/store/index.md) — the async Store that drives an `AsyncBackend`
- [Adapters](https://docs.remotestore.dev/stable/reference/api/aio/adapters/index.md) — bridge sync ↔ async backend implementations
- [Async Store Guide](https://docs.remotestore.dev/stable/guides/async/index.md) — usage patterns and FastAPI integration

# AsyncStore

`AsyncStore` is the async counterpart of [`Store`](https://docs.remotestore.dev/stable/reference/api/store/index.md): the same methods, the same errors, the same capability model, exposed as coroutines. It lives in `remote_store.aio`.

## AsyncStore

```
AsyncStore(
    backend: AsyncBackend | Backend, root_path: str = ""
)
```

An async logical remote folder scoped to a root path.

All path arguments are validated and prefixed with `root_path` before being delegated to the backend. Supports the async context-manager protocol (`async with AsyncStore(...) as s:`) which calls `aclose()` on exit.

Parameters:

- **`backend`** (`AsyncBackend | Backend`) – Async or sync backend instance. Sync backends are auto-wrapped via SyncBackendAdapter.
- **`root_path`** (`str`, default: `''` ) – Prefix prepended to every path. "" means the backend root.

Async counterpart to `Store`

Same methods, same errors, same capability model. See the [Async Store Guide](https://docs.remotestore.dev/stable/guides/async/index.md) for usage patterns and [Store](https://docs.remotestore.dev/stable/reference/api/store/index.md) for the synchronous counterpart.

Thread safety

`AsyncStore` is immutable after construction and can be shared across tasks on the same event loop. Backend thread safety depends on the backend implementation.

## Reading

Requires `Capability.READ`

All read methods raise `CapabilityNotSupported` on backends that do not declare this capability. Most backends declare it.

### read

```
read(path: str) -> AsyncIterator[bytes]
```

Return an async iterator of byte chunks for *path*.

The caller is responsible for consuming the iterator. Validation (capability check, path check) happens eagerly on call, not lazily on first iteration.

Parameters:

- **`path`** (`str`) – Store-relative file path.

Returns:

- `AsyncIterator[bytes]` – Async iterator of byte chunks.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path is empty, or if path names a directory.

Quality flag: `Capability.LAZY_READ`

When declared, data is fetched lazily — partial reads avoid loading the whole file. Without it, the backend may buffer content before returning the stream.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read the entire file into memory and return `bytes`.

Parameters:

- **`path`** (`str`) – Store-relative file path.

Returns:

- `bytes` – The file content as bytes.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path is empty, or if path names a directory.

Equivalent to collecting all chunks from `read(path)`.

### read_text

```
read_text(
    path: str,
    *,
    encoding: str = "utf-8",
    errors: str = "strict",
) -> str
```

Read the entire file and decode it as text.

Parameters:

- **`path`** (`str`) – Store-relative file path.
- **`encoding`** (`str`, default: `'utf-8'` ) – Text encoding, any name accepted by codecs.
- **`errors`** (`str`, default: `'strict'` ) – Error handler: "strict", "ignore", "replace", "backslashreplace". See codecs.register_error for custom handlers.

Returns:

- `str` – The file content as str.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path is empty, or if path names a directory.
- `UnicodeDecodeError` – If decoding fails with errors="strict".

Equivalent to `(await read_bytes(path)).decode(encoding, errors)`.

## Writing

Requires `Capability.WRITE`

`write()` and `write_text()` raise `CapabilityNotSupported` on backends that do not declare this capability. Most backends declare it. `write_atomic()` additionally requires `Capability.ATOMIC_WRITE`.

Quality flag: `Capability.WRITE_RESULT_NATIVE`

When declared, the returned `WriteResult` fields (`etag`, `version_id`, `last_modified`, `digest`) are populated from the backend's write response. Without it, only locally computable fields are set.

### write

```
write(
    path: str,
    content: AsyncWritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write binary content to *path*. Creates parent folders implicitly.

Parameters:

- **`path`** (`str`) – Store-relative file path.
- **`content`** (`AsyncWritableContent`) – bytes or async iterator of bytes.
- **`overwrite`** (`bool`, default: `False` ) – If False, raises AlreadyExists when path exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user-defined string metadata.

Returns:

- `WriteResult` – WriteResult with at least path and size populated.

Raises:

- `ValueError` – If metadata contains invalid keys or values (see Store.write() for validation rules).
- `CapabilityNotSupported` – If metadata is non-empty and the backend lacks USER_METADATA.
- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path is empty.

Backend-conditional argument: `metadata=`

Passing `metadata` raises `CapabilityNotSupported` on backends that do not declare `Capability.USER_METADATA`. Passing `None` or `{}` is safe on all backends.

### write_text

```
write_text(
    path: str,
    text: str,
    *,
    encoding: str = "utf-8",
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write a string to *path*, encoded with the given encoding.

Parameters:

- **`path`** (`str`) – Store-relative file path.
- **`text`** (`str`) – The string to write.
- **`encoding`** (`str`, default: `'utf-8'` ) – Text encoding.
- **`overwrite`** (`bool`, default: `False` ) – If False, raises AlreadyExists when path exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user-defined string metadata.

Raises:

- `ValueError` – If metadata contains invalid keys or values (see Store.write() for validation rules).
- `CapabilityNotSupported` – If metadata is non-empty and the backend lacks USER_METADATA.
- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path is empty.

Returns:

- `WriteResult` – WriteResult with at least path and size populated.

Equivalent to `await write(path, text.encode(encoding), overwrite=overwrite, metadata=metadata)`.

Backend-conditional argument: `metadata=`

Passing `metadata` raises `CapabilityNotSupported` on backends that do not declare `Capability.USER_METADATA`. Passing `None` or `{}` is safe on all backends.

### write_atomic

```
write_atomic(
    path: str,
    content: AsyncWritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write binary content to *path* atomically.

If the write fails or is interrupted, *path* is not left in a partial state.

Parameters:

- **`path`** (`str`) – Store-relative file path.
- **`content`** (`AsyncWritableContent`) – bytes or async iterator of bytes.
- **`overwrite`** (`bool`, default: `False` ) – If False, raises AlreadyExists when path exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user-defined string metadata.

Raises:

- `ValueError` – If metadata contains invalid keys or values (see Store.write() for validation rules).
- `CapabilityNotSupported` – If metadata is non-empty and the backend lacks USER_METADATA, or if backend lacks ATOMIC_WRITE.
- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path is empty.

Returns:

- `WriteResult` – WriteResult with at least path and size populated.

Requires `Capability.ATOMIC_WRITE`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

Backend-conditional argument: `metadata=`

Passing `metadata` raises `CapabilityNotSupported` on backends that do not declare `Capability.USER_METADATA`. Passing `None` or `{}` is safe on all backends.

## Deleting

Requires `Capability.DELETE`

All delete methods raise `CapabilityNotSupported` on backends that do not declare this capability.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete a single file.

Parameters:

- **`path`** (`str`) – Store-relative file path.
- **`missing_ok`** (`bool`, default: `False` ) – If True, silently succeeds when path does not exist.

Raises:

- `NotFound` – If the file is missing and missing_ok is False.
- `InvalidPath` – If path is empty, or if path names a directory (regardless of missing_ok).

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete a folder.

Parameters:

- **`path`** (`str`) – Store-relative folder path. Must not be "" (root).
- **`recursive`** (`bool`, default: `False` ) – If True, delete all contents first. If False, raises DirectoryNotEmpty when folder is non-empty.
- **`missing_ok`** (`bool`, default: `False` ) – If True, silently succeeds when path does not exist.

Raises:

- `NotFound` – If the folder is missing and missing_ok is False.
- `DirectoryNotEmpty` – If the folder is non-empty and recursive is False.
- `InvalidPath` – If path is empty (cannot delete the store root), or if path names a file (use delete instead).

## Listing and Iteration

Requires `Capability.LIST`

All listing methods raise `CapabilityNotSupported` on backends that do not declare this capability.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    pattern: str | None = None,
    max_depth: int | None = None,
) -> AsyncIterator[FileInfo]
```

Yield `FileInfo` objects for files under *path*.

Validation (capability check, max_depth) happens eagerly on call, not lazily on first iteration.

Parameters:

- **`path`** (`str`) – Store-relative folder path.
- **`recursive`** (`bool`, default: `False` ) – Descend into subfolders. Ignored when max_depth is set.
- **`pattern`** (`str | None`, default: `None` ) – Glob pattern to filter filenames (e.g. "\*.csv"). Matched against each file's name (basename only). For full path-based patterns, use ext.glob.glob_files().
- **`max_depth`** (`int | None`, default: `None` ) – Maximum folder depth to include. 0 means files directly in path only; 1 adds files in its immediate subfolders, and so on. None (default) defers to recursive. When set, recursive is ignored.

Returns:

- `AsyncIterator[FileInfo]` – Async iterator of FileInfo with store-relative paths.

Raises:

- `ValueError` – If max_depth is negative.

Backend-conditional argument: `max_depth=`

Backends with native depth limiting prune traversal early. Backends that do not support it still return correct results — the Store applies client-side filtering as a safety net.

### list_folders

```
list_folders(
    path: str,
    *,
    pattern: str | None = None,
    max_depth: int | None = None,
) -> AsyncIterator[FolderEntry]
```

Yield subfolders of *path* as `FolderEntry` objects.

Validation (capability check, max_depth) happens eagerly on call, not lazily on first iteration.

Parameters:

- **`path`** (`str`) – Store-relative folder path.
- **`pattern`** (`str | None`, default: `None` ) – Glob pattern to filter folder names (e.g. "raw\_\*"). Matched against each folder's name (basename only) via fnmatch.fnmatch. Filters yielded results only — does not prune BFS traversal, so non-matching folders are still descended into.
- **`max_depth`** (`int | None`, default: `None` ) – Maximum folder depth to include. None or 0 returns immediate children only (default). 1 adds grandchildren, and so on. BFS traversal runs first; pattern filters what is yielded.

Returns:

- `AsyncIterator[FolderEntry]` – Async iterator of FolderEntry with .name and .path (store-relative).

Raises:

- `ValueError` – If max_depth is negative.

Backend-conditional argument: `max_depth=`

Backends with native depth limiting prune traversal early. Backends that do not support it still return correct results — the Store applies client-side filtering as a safety net.

### iter_children

```
iter_children(
    path: str,
) -> AsyncIterator[FileInfo | FolderEntry]
```

Yield all immediate children (files and folders) of *path* in a single pass.

Files are yielded as `FileInfo`, folders as `FolderEntry`. Both have `.name` and `.path` attributes (satisfying the `PathEntry` protocol) so callers can iterate uniformly.

Validation (capability check) happens eagerly on call, not lazily on first iteration.

Parameters:

- **`path`** (`str`) – Store-relative folder path.

Returns:

- `AsyncIterator[FileInfo | FolderEntry]` – Async iterator of FileInfo (files) and FolderEntry (folders).

### glob

```
glob(pattern: str) -> AsyncIterator[FileInfo]
```

Yield files matching a glob *pattern*, using the backend's native glob implementation.

Requires `Capability.GLOB`. Validation (capability check) happens eagerly on call, not lazily on first iteration.

Parameters:

- **`pattern`** (`str`) – Glob pattern (e.g. "data/\*\*/\*.parquet").

Returns:

- `AsyncIterator[FileInfo]` – Async iterator of FileInfo with store-relative paths.

Raises:

- `CapabilityNotSupported` – If the backend lacks GLOB.

Requires `Capability.GLOB`

`glob()` raises `CapabilityNotSupported` on backends that do not declare this capability. Check `store.supports(Capability.GLOB)` before calling.

Ordering and laziness

**Ordering is backend-defined** and may vary between backends (e.g. lexicographic on S3, OS-dependent on local filesystems). Callers must not depend on any particular order.

**Results are yielded lazily.** Backends may use pagination internally. Memory usage stays bounded for large directories.

## File Operations

Requires `Capability.MOVE` / `Capability.COPY`

`move()` requires `Capability.MOVE`; `copy()` requires `Capability.COPY`. Each raises `CapabilityNotSupported` on backends that do not declare the respective capability.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move (rename) a file from *src* to *dst*.

File-only -- to move a folder, iterate its contents.

Parameters:

- **`src`** (`str`) – Source file path.
- **`dst`** (`str`) – Destination file path.
- **`overwrite`** (`bool`, default: `False` ) – If False, raises AlreadyExists when dst exists.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists and overwrite is False.
- `InvalidPath` – If src or dst is empty, or if src names a directory.

Requires `Capability.MOVE`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

Atomicity

Atomicity is backend-dependent. Local uses `os.replace` (atomic on same filesystem). S3 and Azure use copy-then-delete (not atomic). SFTP atomicity depends on the server. Check `store.supports(Capability.ATOMIC_MOVE)` to query this at runtime.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy a file from *src* to *dst*.

File-only -- to copy a folder, iterate its contents.

Parameters:

- **`src`** (`str`) – Source file path.
- **`dst`** (`str`) – Destination file path.
- **`overwrite`** (`bool`, default: `False` ) – If False, raises AlreadyExists when dst exists.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists and overwrite is False.
- `InvalidPath` – If src or dst is empty, or if src names a directory.

Requires `Capability.COPY`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

Metadata preservation

Metadata preservation is backend-dependent. S3 copies metadata; local preserves metadata (`copy2`); SFTP does not (stream copy).

## Metadata

Partially requires `Capability.METADATA`

`head()` and `get_file_info()` require `Capability.METADATA`. `get_folder_info()` requires `Capability.METADATA` without `max_depth`, or `Capability.LIST` when `max_depth` is set. `exists()`, `is_file()`, and `is_folder()` are always available.

### head

```
head(path: str) -> WriteResult
```

Return a `WriteResult` snapshot of *path* via a metadata lookup.

Async equivalent of `Store.head()`. Gated on `Capability.METADATA`. Returns `source="sidecar"`.

Parameters:

- **`path`** (`str`) – Store-relative file path.

Returns:

- `WriteResult` – WriteResult with source="sidecar".

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path is empty, or if path names a directory.
- `CapabilityNotSupported` – If the backend lacks METADATA.

Requires `Capability.METADATA`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

### exists

```
exists(path: str) -> bool
```

Return `True` if *path* exists (file or folder).

Parameters:

- **`path`** (`str`) – Store-relative path.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if *path* exists and is a file.

Parameters:

- **`path`** (`str`) – Store-relative path.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if *path* exists and is a folder.

Parameters:

- **`path`** (`str`) – Store-relative path.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return a `FileInfo` with size, modification time, and content type for a single file.

Parameters:

- **`path`** (`str`) – Store-relative file path.

Returns:

- `FileInfo` – FileInfo.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path is empty, or if path names a directory.

Requires `Capability.METADATA`

Raises `CapabilityNotSupported` on backends that do not declare this capability.

### get_folder_info

```
get_folder_info(
    path: str, *, max_depth: int | None = None
) -> FolderInfo
```

Return a `FolderInfo` with aggregated size and file count for a folder.

Parameters:

- **`path`** (`str`) – Store-relative folder path.
- **`max_depth`** (`int | None`, default: `None` ) – Maximum folder depth to aggregate. 0 means files directly in path only; 1 adds files in its immediate subfolders, and so on. None (default) performs a full recursive traversal via the backend.

Returns:

- `FolderInfo` – FolderInfo.

Raises:

- `NotFound` – If the folder does not exist.
- `InvalidPath` – If path names a file (use get_file_info instead).
- `ValueError` – If max_depth is negative.

Capability depends on `max_depth`

Without `max_depth`: requires `Capability.METADATA`. With `max_depth` set: requires `Capability.LIST` — works on backends that lack `METADATA`.

Backend-conditional argument: `max_depth=`

Backends with native depth limiting prune traversal early. Backends that do not support it still return correct results — the Store applies client-side filtering as a safety net.

## Introspection

### resolve

```
resolve(key: str) -> ResolutionPlan
```

Return a `ResolutionPlan` describing how *key* maps to storage.

Delegates to the backend's `resolve()` and rebases the key so that `plan.key` is the store-relative key, not the backend-relative path.

Parameters:

- **`key`** (`str`) – Store-relative path. "" resolves the store root.

Returns:

- `ResolutionPlan` – A frozen ResolutionPlan.

Info

`resolve()` is a pure introspection method — it performs no I/O and is never called implicitly by other Store methods. The returned [`ResolutionPlan`](https://docs.remotestore.dev/stable/reference/api/models/index.md) describes how a key maps to its storage location.

## Lifecycle

### ping

```
ping() -> None
```

Verify that the backend is reachable.

Raises:

- `PermissionDenied` – If credentials are invalid.
- `NotFound` – If the bucket, container, or root path does not exist.
- `BackendUnavailable` – If the backend cannot be reached.

### aclose

```
aclose() -> None
```

Release backend resources.

Called automatically when used as an async context manager.

### child

```
child(subpath: str) -> AsyncStore
```

Return a new `AsyncStore` scoped to *subpath* under the current root.

The child shares the same backend instance.

Parameters:

- **`subpath`** (`str`) – Path segment to append to the current root.

Returns:

- `AsyncStore` – AsyncStore.

Raises:

- `InvalidPath` – If subpath is empty, contains .. segments, or includes null bytes.

```
data = store.child("data/2024")
async for fi in data.list_files(""):
    ...  # lists files under <root>/data/2024/
```

## Interop (Backend-Specific)

Backend-specific methods

Methods in this section expose backend internals. Using them ties your code to a specific backend. For portable alternatives, see [Store](https://docs.remotestore.dev/stable/reference/api/store/index.md) or the [Async Store Guide](https://docs.remotestore.dev/stable/guides/async/index.md).

### unwrap

```
unwrap(type_hint: type[T]) -> T
```

Return the backend's native client object, cast to *type_hint*.

Parameters:

- **`type_hint`** (`type[T]`) – The expected type of the native client (e.g. pyarrow.fs.FileSystem).

Returns:

- `T` – The native client.

Raises:

- `CapabilityNotSupported` – If the backend cannot provide the requested type.

```
arrow_fs = store.unwrap(pyarrow.fs.FileSystem)
```

### native_path

```
native_path(key: str) -> str
```

Convert a store-relative *key* to the backend's native path representation.

Inverse of `to_key()`.

Parameters:

- **`key`** (`str`) – Store-relative path.

Returns:

- `str` – Backend-native path (e.g. S3 object key, local filesystem path).

### to_key

```
to_key(path: str) -> str
```

Convert a backend-native *path* to a store-relative key.

Inverse of `native_path()`.

Parameters:

- **`path`** (`str`) – Backend-native path string.

Returns:

- `str` – Store-relative key.

Raises:

- `InvalidPath` – If the path does not belong to this store.

### supports

```
supports(capability: Capability) -> bool
```

Check whether the backend supports a given `Capability`.

Parameters:

- **`capability`** (`Capability`) – A Capability enum member.

Returns:

- `bool` – True if the backend declares this capability.

```
if store.supports(Capability.GLOB):
    async for fi in store.glob("**/*.csv"):
        ...
```

Info

`supports()` itself is portable — it works on all backends. Only the capability-gated methods it guards are backend-specific.

## See also

- [Async Store Guide](https://docs.remotestore.dev/stable/guides/async/index.md) — usage patterns, streaming, FastAPI integration
- [Example: Async Store](https://docs.remotestore.dev/stable/tutorial/examples/async-store/index.md) — runnable demo script
- [Store](https://docs.remotestore.dev/stable/reference/api/store/index.md) — synchronous counterpart
- [AsyncBackend](https://docs.remotestore.dev/stable/reference/api/aio/backend/index.md) — the backend protocol `AsyncStore` drives
- [Concurrency](https://docs.remotestore.dev/stable/explanation/concurrency/index.md) — thread safety and atomicity semantics

# Async backends

API reference for the native async storage backend classes in `remote_store.aio.backends`. Each implements the [`AsyncBackend`](https://docs.remotestore.dev/stable/reference/api/aio/backend/index.md) protocol and runs on the event loop without a thread-pool bridge.

| Class                                                                                                | Description                                                 |
| ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| [AsyncMemoryBackend](https://docs.remotestore.dev/stable/reference/api/aio/backends/memory/index.md) | In-memory async backend for testing                         |
| [AsyncAzureBackend](https://docs.remotestore.dev/stable/reference/api/aio/backends/azure/index.md)   | Native async Azure Blob Storage and ADLS Gen2               |
| [GraphBackend](https://docs.remotestore.dev/stable/reference/api/aio/backends/graph/index.md)        | Microsoft Graph backend (OneDrive, SharePoint, Teams files) |

Sync backends run under `AsyncStore` too

A native async backend is only needed when you want true non-blocking I/O. Any synchronous [backend](https://docs.remotestore.dev/stable/reference/api/backends/index.md) works under [`AsyncStore`](https://docs.remotestore.dev/stable/reference/api/aio/store/index.md) via the thread-pool [`SyncBackendAdapter`](https://docs.remotestore.dev/stable/reference/api/aio/adapters/index.md), which `AsyncStore` applies automatically.

## See also

- [Backends](https://docs.remotestore.dev/stable/reference/api/backends/index.md) — the synchronous backend classes
- [Async Store Guide](https://docs.remotestore.dev/stable/guides/async/index.md) — usage patterns
- [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) — per-backend capability comparison

# AsyncAzureBackend

Native async Azure Storage backend. Uses the async Blob SDK for non-HNS accounts (plain Blob Storage, Azurite) and the async DataLake SDK for HNS accounts (ADLS Gen2) to get atomic rename and real directory support. Whether the account is HNS is declared via the required `hns` argument; use `AzureUtils.adetect_hns()` to discover it once if unknown.

## AsyncAzureBackend

```
AsyncAzureBackend(
    container: str,
    *,
    hns: bool | None = None,
    account_name: str | None = None,
    account_url: str | None = None,
    account_key: str | Secret | None = None,
    sas_token: str | Secret | None = None,
    connection_string: str | Secret | None = None,
    credential: Any | None = None,
    client_options: dict[str, Any] | None = None,
    retry: RetryPolicy | None = None,
    max_concurrency: int = 1,
    reject_write_under_file_ancestor: bool = False,
)
```

Async Azure Storage backend.

Uses the async Blob SDK for non-HNS accounts (plain Blob Storage, Azurite) and the async DataLake SDK for HNS accounts (ADLS Gen2) to get atomic rename and real directory support. Whether the account is HNS is declared explicitly via the required `hns` argument -- the backend does not probe for it.

Parameters:

- **`container`** (`str`) – Azure Storage container name (required, non-empty).
- **`hns`** (`bool | None`, default: `None` ) – Whether the storage account has Hierarchical Namespace enabled (ADLS Gen2). Required -- there is no default and no runtime auto-detection. Pass True for ADLS Gen2 accounts (atomic rename, real directories) or False for flat Blob Storage. Use AzureUtils.adetect_hns() to discover the value once if you do not already know it.
- **`account_name`** (`str | None`, default: `None` ) – Storage account name.
- **`account_url`** (`str | None`, default: `None` ) – Full account URL (e.g. https://myaccount.dfs.core.windows.net).
- **`account_key`** (`str | Secret | None`, default: `None` ) – Storage account key.
- **`sas_token`** (`str | Secret | None`, default: `None` ) – Shared Access Signature token.
- **`connection_string`** (`str | Secret | None`, default: `None` ) – Azure Storage connection string.
- **`credential`** (`Any | None`, default: `None` ) – Any credential object (e.g. DefaultAzureCredential()).
- **`client_options`** (`dict[str, Any] | None`, default: `None` ) – Additional options passed to service clients. The library sets max_single_put_size, max_block_size, and min_large_block_upload_threshold defaults for streaming memory discipline; user-supplied values take precedence.
- **`retry`** (`RetryPolicy | None`, default: `None` ) – Retry policy for transient failures.
- **`max_concurrency`** (`int`, default: `1` ) – Maximum number of parallel connections for uploads and downloads (default 1 -- sequential).
- **`reject_write_under_file_ancestor`** (`bool`, default: `False` ) – If True, write / write_atomic / move / copy HEAD each slash-aligned ancestor of the target path on non-HNS accounts and raise InvalidPath on the first regular-file hit, matching the cross-backend contract that hierarchical filesystems enforce natively. On HNS accounts the kwarg short-circuits: hdi_isfolder rejects the operation natively, and the backend detects the file ancestor on that rejection and re-raises it as InvalidPath, so HNS delivers the cross-backend contract with or without the kwarg set. Default False.

### name

```
name: str
```

Unique identifier for this backend type.

### capabilities

```
capabilities: CapabilitySet
```

Declared capabilities of this backend.

### check_health

```
check_health() -> None
```

Verify the backend is reachable and credentials are valid.

Raises:

- `PermissionDenied` – If credentials are invalid.
- `NotFound` – If the container does not exist.
- `BackendUnavailable` – If the backend cannot be reached.

### to_key

```
to_key(native_path: str) -> str
```

Convert a backend-native path to a backend-relative key.

Parameters:

- **`native_path`** (`str`) – Absolute or backend-native path string.

Returns:

- `str` – Path relative to the backend's root.

### native_path

```
native_path(path: str) -> str
```

Convert a backend-relative key to the backend-native path.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `str` – Backend-native path (container/path).

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` with Azure-specific details.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – Plan with kind="async-azure" and details containing
- `ResolutionPlan` – container and account_url.

### exists

```
exists(path: str) -> bool
```

Check if a file or folder exists.

Parameters:

- **`path`** (`str`) – Backend-relative key, or "" for the root.

Returns:

- `bool` – True if a file or folder exists at path.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if `path` is an existing file.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `bool` – True if path exists and is a file.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if `path` is an existing folder.

Parameters:

- **`path`** (`str`) – Backend-relative key, or "" for the root.

Returns:

- `bool` – True if path exists and is a folder.

### read

```
read(path: str) -> AsyncIterator[bytes]
```

Open a file for reading and return an async iterator of byte chunks.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `AsyncIterator[bytes]` – An async iterator yielding byte chunks.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path names a directory (HNS accounts only).

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read the full content of a file as bytes.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `bytes` – The file content.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path names a directory (HNS accounts only).

### write

```
write(
    path: str,
    content: AsyncWritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write content to a file.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`content`** (`AsyncWritableContent`) – Data to write (bytes or async iterator of bytes).
- **`overwrite`** (`bool`, default: `False` ) – If False, raise if file already exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user-defined string metadata.

Returns:

- `WriteResult` – WriteResult with native Azure fields (etag, last_modified,
- `WriteResult` – etc.) populated from the SDK upload response.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory.

### write_atomic

```
write_atomic(
    path: str,
    content: AsyncWritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write content atomically via temp file + rename.

For non-HNS accounts, direct upload is atomic (PUT semantics). For HNS accounts, write to temp file via DFS then atomic rename.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`content`** (`AsyncWritableContent`) – Data to write.
- **`overwrite`** (`bool`, default: `False` ) – If False, raise if file already exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user-defined string metadata.

Returns:

- `WriteResult` – WriteResult with native Azure fields populated from the SDK
- `WriteResult` – response (non-HNS) or from get_file_properties() after rename
- `WriteResult` – (HNS).

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete a file.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`missing_ok`** (`bool`, default: `False` ) – If True, do not raise when the file is absent.

Raises:

- `NotFound` – If the file is missing and missing_ok is False.
- `InvalidPath` – If path names a directory (HNS accounts only).

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete a folder.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`recursive`** (`bool`, default: `False` ) – If True, delete all contents first.
- **`missing_ok`** (`bool`, default: `False` ) – If True, do not raise when absent.

Raises:

- `NotFound` – If the folder is missing and missing_ok is False.
- `InvalidPath` – If path names a file (use delete instead).
- `DirectoryNotEmpty` – If non-empty and recursive is False.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> AsyncIterator[FileInfo]
```

List files under `path`.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.
- **`recursive`** (`bool`, default: `False` ) – If True, include files in all subdirectories.
- **`max_depth`** (`int | None`, default: `None` ) – Optional maximum folder depth to traverse.

Returns:

- `AsyncIterator[FileInfo]` – An async iterator of FileInfo objects.

### list_folders

```
list_folders(path: str) -> AsyncIterator[FolderEntry]
```

List immediate subfolders under `path`.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.

Returns:

- `AsyncIterator[FolderEntry]` – An async iterator of FolderEntry objects.

### iter_children

```
iter_children(
    path: str,
) -> AsyncIterator[FileInfo | FolderEntry]
```

Yield both files and folders under `path` in a single pass.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.

Returns:

- `AsyncIterator[FileInfo | FolderEntry]` – An async iterator of FileInfo (files) and FolderEntry (folders).

### glob

```
glob(pattern: str) -> AsyncIterator[FileInfo]
```

Match files against a glob pattern.

Parameters:

- **`pattern`** (`str`) – Glob pattern (e.g., "data/\*.csv", "\*\*/\*.txt").

Returns:

- `AsyncIterator[FileInfo]` – An async iterator of matching FileInfo objects.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Get metadata for a file.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `FileInfo` – A FileInfo with size, modification time, etc.

Raises:

- `InvalidPath` – If path names a directory (HNS: hdi_isfolder=true).
- `NotFound` – If the file does not exist.

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Get metadata for a folder.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.

Returns:

- `FolderInfo` – A FolderInfo with file count, total size, etc.

Raises:

- `NotFound` – If the folder does not exist.
- `InvalidPath` – If path names a file (use get_file_info instead).

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename a file.

Parameters:

- **`src`** (`str`) – Backend-relative source key.
- **`dst`** (`str`) – Backend-relative destination key.
- **`overwrite`** (`bool`, default: `False` ) – If True, replace any existing file at dst.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory (HNS only).
- `AlreadyExists` – If dst exists and overwrite is False.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy a file.

Parameters:

- **`src`** (`str`) – Backend-relative source key.
- **`dst`** (`str`) – Backend-relative destination key.
- **`overwrite`** (`bool`, default: `False` ) – If True, replace any existing file at dst.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory (HNS only).
- `AlreadyExists` – If dst exists and overwrite is False.

### aclose

```
aclose() -> None
```

Release all Azure SDK client resources.

### unwrap

```
unwrap(type_hint: type[T]) -> T
```

Return the native async FileSystemClient if it matches the requested type.

Parameters:

- **`type_hint`** (`type[T]`) – The expected type.

Returns:

- `T` – The native async client instance matching type_hint.

Raises:

- `CapabilityNotSupported` – If backend cannot provide the requested type.

## See also

- [AzureBackend](https://docs.remotestore.dev/stable/reference/api/backends/azure/index.md) — synchronous counterpart
- [Azure Backend Guide](https://docs.remotestore.dev/stable/guides/backends/azure/index.md) — configuration and usage
- [Azure HNS setup](https://docs.remotestore.dev/stable/guides/backends/azure-hns-setup/index.md) — ADLS Gen2 provisioning

# GraphBackend

Native async Microsoft Graph backend over OneDrive, SharePoint document libraries, and Teams files. A single instance targets one drive (`drive_id`); transport is `httpx` and auth is a token-provider callable (the built-in `GraphAuth` helper, or any user-supplied callable). Requires the `graph` extra. See the [Graph setup guide](https://docs.remotestore.dev/stable/guides/backends/graph-setup/index.md) for provisioning credentials and resolving a `drive_id`.

Graph is async-only — there is no synchronous `GraphBackend`. To use it from synchronous code, wrap it with [`AsyncBackendSyncAdapter`](https://docs.remotestore.dev/stable/reference/api/aio/adapters/index.md).

## GraphBackend

```
GraphBackend(
    drive_id: str,
    *,
    token_provider: TokenProvider,
    base_url: str = _DEFAULT_BASE_URL,
    http_client: AsyncClient | None = None,
    retry: RetryPolicy | None = None,
    upload_chunk_size: int = _DEFAULT_UPLOAD_CHUNK_SIZE,
    copy_timeout: float | None = None,
    base_path: str = "",
    client_options: dict[str, Any] | None = None,
)
```

Async Microsoft Graph backend over OneDrive / SharePoint / Teams files.

A single instance targets one drive, identified by an immutable `drive_id`. Items are addressed by `/`-rooted POSIX path; transport is `httpx`; auth is a token-provider callable (the built-in `GraphAuth` helper, or any user-supplied callable).

Parameters:

- **`drive_id`** (`str`) – Opaque Graph drive id. Resolve one from a URL / "me" / Teams channel with GraphUtils.resolve_drive_id.
- **`token_provider`** (`TokenProvider`) – Callable\[[], str\] or Callable\[[], Awaitable[str]\] returning a bearer token, invoked lazily (never in __init__).
- **`base_url`** (`str`, default: `_DEFAULT_BASE_URL` ) – Graph API root (default https://graph.microsoft.com/v1.0).
- **`http_client`** (`AsyncClient | None`, default: `None` ) – Reuse an existing httpx.AsyncClient; the caller owns its lifecycle and close() does not close it. When omitted, one is created lazily on first use and closed by close().
- **`retry`** (`RetryPolicy | None`, default: `None` ) – Retry policy for transient failures; None uses the default RetryPolicy() profile.
- **`upload_chunk_size`** (`int`, default: `_DEFAULT_UPLOAD_CHUNK_SIZE` ) – Upload-session chunk size; must be a positive multiple of 320 KiB and strictly less than 60 MiB (Graph's per-request ceiling). Default 10 MiB.
- **`copy_timeout`** (`float | None`, default: `None` ) – Wall-clock budget for copy/move monitor polling, or None for no backend-imposed ceiling. When set, must be a positive float.
- **`base_path`** (`str`, default: `''` ) – Optional drive subfolder to scope every operation under. When set, all keys are addressed relative to this folder and keys returned by listing / to_key stay relative to it, so the backend behaves as if base_path were its root. Defaults to the drive root.
- **`client_options`** (`dict[str, Any] | None`, default: `None` ) – Extra options passed through to the internal httpx.AsyncClient. (When a future revision adds an explicit httpx-level constructor parameter, it takes precedence over a client_options key of the same name; the backend has no such parameter today, so this is passthrough only.)

Raises:

- `ValueError` – For an empty drive_id, a non-callable token_provider, a non-string base_path, an upload_chunk_size that is not a positive 320 KiB multiple below 60 MiB, or a non-positive copy_timeout.

## GraphAuth

MSAL-backed token provider for `GraphBackend`. Wraps the client-credentials (app-only) and device-code (interactive) flows and exposes the bearer token through the token-provider protocol — a `GraphAuth` instance is itself a `Callable[[], str]`.

## GraphAuth

```
GraphAuth(
    tenant_id: str,
    client_id: str,
    *,
    client_secret: str | Secret | None = None,
    client_certificate: dict[str, Any] | None = None,
    scopes: Sequence[str] | None = None,
    cache_path: str | None = None,
    prompt_callback: Callable[[dict[str, Any]], None]
    | None = None,
)
```

MSAL-backed token provider for `GraphBackend`.

Selects the OAuth flow from the supplied credentials: a `client_secret` or `client_certificate` selects client-credentials (app-only); their absence selects device-code (interactive). The resulting bearer token is reachable synchronously through `get_token()` (and through calling the instance directly) and asynchronously through `aget_token()`, which offloads the blocking MSAL work off the event loop and single-flights concurrent acquisitions — prefer it on the event loop.

Parameters:

- **`tenant_id`** (`str`) – Entra tenant id, or "consumers" / "common" / "organizations" for the device-code multi-tenant authorities.
- **`client_id`** (`str`) – Application (client) id of the Entra app registration.
- **`client_secret`** (`str | Secret | None`, default: `None` ) – Client secret for client-credentials. Accepts a Secret and is masked in repr.
- **`client_certificate`** (`dict[str, Any] | None`, default: `None` ) – Certificate dict for client-credentials, as MSAL's client_credential mapping. Mutually exclusive with client_secret.
- **`scopes`** (`Sequence[str] | None`, default: `None` ) – Override the default scope set. Client-credentials defaults to ["https://graph.microsoft.com/.default"]; device-code defaults to the delegated ["Files.ReadWrite", "User.Read"] (the signed-in user's OneDrive — consumer-compatible). Add Sites.ReadWrite.All for delegated SharePoint access.
- **`cache_path`** (`str | None`, default: `None` ) – Override the MSAL token-cache file location. Defaults to \<user_config_dir("remote-store")>/graph_token_cache.json. The cache is persisted multi-process-safely: a sibling \<cache_path>.lockfile coordinates concurrent writers so a shared default cache survives the common multi-worker deployment.
- **`prompt_callback`** (`Callable[[dict[str, Any]], None] | None`, default: `None` ) – Invoked with the MSAL device-flow dict on device-code login; defaults to printing flow["message"].

Raises:

- `ValueError` – If tenant_id or client_id is empty, or if both client_secret and client_certificate are supplied.

### authority

```
authority: str
```

The MSAL authority URL derived from `tenant_id`.

### get_token

```
get_token() -> str
```

Acquire (or silently refresh) a bearer token.

The token-provider callable the backend invokes. Re-invoking it after a `401` refreshes through MSAL's cache. The acquisition writes the refreshed cache through to disk itself — the backing `PersistedTokenCache` persists under a cross-process lock on every change, so no explicit flush is needed here.

Raises:

- `PermissionDenied` – If MSAL returns no token (auth failure); the error_description is included, never the secret. A typed RemoteStoreError so a failure surfacing mid-read / write stays catchable via except RemoteStoreError.

### __call__

```
__call__() -> str
```

Return a bearer token — makes the instance a token-provider callable.

### aget_token

```
aget_token() -> str
```

Acquire a bearer token without blocking the event loop.

The async token-provider entry point: pass `token_provider=auth.aget_token` (a bound async method is a `Callable[[], Awaitable[str]]`) so the backend awaits it instead of calling the synchronous instance. It offloads the synchronous MSAL acquisition — including any contended token-cache lock wait — to a worker thread so sibling coroutines keep running, and single-flights concurrent callers: N coroutines acquiring at once share **one** acquisition rather than each hitting the identity provider (which also dedupes the one-shot refresh after a `401`).

Use this on the event loop; reuse the synchronous `get_token` / `__call__` from synchronous wiring code that has no running loop. The single-flight state is bound to the loop the first caller runs on — share one instance per loop, mirroring the backend's own single-loop posture. Cancelling one caller (e.g. a `gather` sibling that times out) neither cancels the shared acquisition nor disturbs the other joiners.

Raises:

- `PermissionDenied` – Propagated from get_token to the owning caller and every joiner sharing the in-flight acquisition; a later call retries afresh.

### flush_cache

```
flush_cache() -> None
```

Best-effort no-op retained for the `GraphBackend.close()` hook.

The token cache is a `PersistedTokenCache` that writes through to disk under a cross-process lock on every acquisition, so there is nothing to flush at close. The method stays because `GraphBackend.close()` invokes it duck-typed and user-supplied providers may implement their own. Never raises — teardown must not fail.

## GraphUtils

Namespace helpers for Graph configuration. `resolve_drive_id` turns "my OneDrive" (`"me"`), a SharePoint site URL, or a Teams channel mapping into the opaque `drive_id` the backend requires.

### resolve_drive_id

```
resolve_drive_id(
    target: str | tuple[str, str] | Mapping[str, str],
    *,
    token_provider: TokenProvider,
    http_client: AsyncClient | None = None,
    base_url: str = _DEFAULT_BASE_URL,
) -> str
```

Resolve a `drive_id` from one of the three target shapes.

Sync entry point for application wiring; runs `aresolve_drive_id` under a private event loop. See `aresolve_drive_id` for the accepted shapes and raised errors.

### aresolve_drive_id

```
aresolve_drive_id(
    target: str | tuple[str, str] | Mapping[str, str],
    *,
    token_provider: TokenProvider,
    http_client: AsyncClient | None = None,
    base_url: str = _DEFAULT_BASE_URL,
) -> str
```

Resolve a `drive_id` from one of three target shapes.

Accepted shapes:

- `"me"` — the authenticated user's default drive (`GET /me/drive`).
- a SharePoint site URL `str` — the site's default drive; or a `(site_url, library_name)` tuple — the named document library.
- a `{"team_id": ..., "channel_id": ...}` mapping — a Teams channel's backing drive.

Parameters:

- **`target`** (`str | tuple[str, str] | Mapping[str, str]`) – One of the shapes above.
- **`token_provider`** (`TokenProvider`) – Bearer-token callable (sync or async).
- **`http_client`** (`AsyncClient | None`, default: `None` ) – Reuse an existing client; one is created and closed per call when omitted.
- **`base_url`** (`str`, default: `_DEFAULT_BASE_URL` ) – Graph API root.

Returns:

- `str` – The opaque Graph drive.id string.

Raises:

- `InvalidPath` – If target matches no accepted shape, the SharePoint site URL has no host, or the named library does not exist.
- `NotFound` – If a site/team/channel id resolves but returns 404.
- `PermissionDenied` – If Graph returns 403 for the lookup.
- `BackendUnavailable` – For a transport error, a retryable 5xx / 429 / 507, or a malformed @odata.nextLink while paging a site's document libraries.

## See also

- [Graph Backend Guide](https://docs.remotestore.dev/stable/guides/backends/graph/index.md) — usage patterns and capabilities
- [Graph setup guide](https://docs.remotestore.dev/stable/guides/backends/graph-setup/index.md) — provisioning credentials and resolving a `drive_id`
- [Adapters](https://docs.remotestore.dev/stable/reference/api/aio/adapters/index.md) — run `GraphBackend` from synchronous code

# AsyncMemoryBackend

In-memory async backend using a tree-indexed data structure. Zero dependencies, no filesystem access, no network. Designed as a drop-in async backend for unit testing, interactive exploration, and documentation examples. Supports all capabilities except `GLOB`.

## AsyncMemoryBackend

```
AsyncMemoryBackend()
```

In-memory async backend using a tree-indexed data structure.

Zero dependencies, no filesystem access, no network. Designed as a drop-in async backend for unit testing, interactive exploration, and documentation examples.

Supports all capabilities except `GLOB`. The full conformance suite passes with zero skips.

Note

`LAZY_READ` is included here but absent from `MemoryBackend` (the sync mirror). The sync backend buffers fully in memory; the async backend can yield chunks incrementally. The `mirrors` graph edge between the two does not imply capability parity.

### exists

```
exists(path: str) -> bool
```

Check if a file or folder exists.

Parameters:

- **`path`** (`str`) – Backend-relative key, or "" for the root.

Returns:

- `bool` – True if a file or folder exists at path.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if `path` is an existing file.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `bool` – True if path exists and is a file.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if `path` is an existing folder.

Parameters:

- **`path`** (`str`) – Backend-relative key, or "" for the root.

Returns:

- `bool` – True if path exists and is a folder.

### read

```
read(path: str) -> AsyncIterator[bytes]
```

Open a file for reading and return an async iterator of byte chunks.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `AsyncIterator[bytes]` – An async iterator yielding a single byte chunk (the full content).

Raises:

- `InvalidPath` – If path names an existing directory.
- `NotFound` – If the file does not exist.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read the full content of a file as bytes.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `bytes` – The file content.

Raises:

- `InvalidPath` – If path names an existing directory.
- `NotFound` – If the file does not exist.

### write

```
write(
    path: str,
    content: AsyncWritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write content to a file.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`content`** (`AsyncWritableContent`) – Data to write (bytes or async iterator of bytes).
- **`overwrite`** (`bool`, default: `False` ) – If False, raise if file already exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user-defined string metadata.

Returns:

- `WriteResult` – WriteResult with path, size, last_modified, and
- `WriteResult` – source="native" populated.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory.

### write_atomic

```
write_atomic(
    path: str,
    content: AsyncWritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write content atomically (same as write for in-memory backend).

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`content`** (`AsyncWritableContent`) – Data to write.
- **`overwrite`** (`bool`, default: `False` ) – If False, raise if file already exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user-defined string metadata.

Returns:

- `WriteResult` – WriteResult with path, size, last_modified, and
- `WriteResult` – source="native" populated.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete a file.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`missing_ok`** (`bool`, default: `False` ) – If True, do not raise when the file is absent.

Raises:

- `NotFound` – If the file is missing and missing_ok is False.
- `InvalidPath` – If the path is empty, or if path names a directory (regardless of missing_ok).

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete a folder.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`recursive`** (`bool`, default: `False` ) – If True, delete all contents first.
- **`missing_ok`** (`bool`, default: `False` ) – If True, do not raise when absent.

Raises:

- `NotFound` – If the folder is missing and missing_ok is False.
- `DirectoryNotEmpty` – If non-empty and recursive is False.
- `InvalidPath` – If the path is empty or names an existing file (regardless of missing_ok).

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> AsyncIterator[FileInfo]
```

List files under `path`.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.
- **`recursive`** (`bool`, default: `False` ) – If True, include files in all subdirectories.
- **`max_depth`** (`int | None`, default: `None` ) – Optional maximum folder depth to traverse.

Returns:

- `AsyncIterator[FileInfo]` – An async iterator of FileInfo objects.

### list_folders

```
list_folders(path: str) -> AsyncIterator[FolderEntry]
```

List immediate subfolders under `path`.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.

Returns:

- `AsyncIterator[FolderEntry]` – An async iterator of FolderEntry objects.

### iter_children

```
iter_children(
    path: str,
) -> AsyncIterator[FileInfo | FolderEntry]
```

Yield both files and folders under `path` in a single pass.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.

Returns:

- `AsyncIterator[FileInfo | FolderEntry]` – An async iterator of FileInfo (files) and FolderEntry (folders).

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Get metadata for a file.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `FileInfo` – A FileInfo with size, modification time, etc.

Raises:

- `InvalidPath` – If path names an existing directory.
- `NotFound` – If the file does not exist.

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Get metadata for a folder.

Parameters:

- **`path`** (`str`) – Backend-relative folder key, or "" for the root.

Returns:

- `FolderInfo` – A FolderInfo with file count, total size, etc.

Raises:

- `InvalidPath` – If path names an existing file.
- `NotFound` – If the folder does not exist.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename a file.

`src == dst` is a no-op (the file is preserved unchanged).

Parameters:

- **`src`** (`str`) – Backend-relative source key.
- **`dst`** (`str`) – Backend-relative destination key.
- **`overwrite`** (`bool`, default: `False` ) – If True, replace any existing file at dst.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `InvalidPath` – If src or dst is empty, src names a directory, or dst names an existing directory.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy a file.

`src == dst` is a no-op (the file is preserved unchanged).

Parameters:

- **`src`** (`str`) – Backend-relative source key.
- **`dst`** (`str`) – Backend-relative destination key.
- **`overwrite`** (`bool`, default: `False` ) – If True, replace any existing file at dst.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `InvalidPath` – If src or dst is empty, src names a directory, or dst names an existing directory.

## See also

- [MemoryBackend](https://docs.remotestore.dev/stable/reference/api/backends/memory/index.md) — synchronous counterpart
- [Async Store Guide](https://docs.remotestore.dev/stable/guides/async/index.md) — usage patterns

# Async extensions

API reference for the native async extensions in `remote_store.aio.ext`. These are the async-native counterparts of selected [synchronous extensions](https://docs.remotestore.dev/stable/reference/api/extensions/index.md).

| Module                                                                                           | Description                                                     |
| ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- |
| [aio.ext.write](https://docs.remotestore.dev/stable/reference/api/aio/extensions/write/index.md) | Async write helpers with guaranteed client-side content hashing |

## See also

- [Extensions](https://docs.remotestore.dev/stable/reference/api/extensions/index.md) — the synchronous extension surface
- [Write Integrity guide](https://docs.remotestore.dev/stable/guides/write-integrity/index.md) — hashing workflows for sync and async

# aio.ext.write

Async write helpers with guaranteed client-side content hashing. Async counterpart of [`ext.write`](https://docs.remotestore.dev/stable/reference/api/extensions/write/index.md) — same guarantees, zero extra round trips, no buffering for async-iterator input.

See the [Write Integrity guide](https://docs.remotestore.dev/stable/guides/write-integrity/index.md) for usage examples and the [ext.write spec](https://docs.remotestore.dev/stable/explanation/design/specs/046-ext-write/index.md) for invariants.

## write

Async write helpers with client-side content hashing.

Guarantees a populated `WriteResult.digest` regardless of whether the backend declares `WRITE_RESULT_NATIVE`. The hash is always computed client-side over the bytes as they flow to the backend.

### write_with_hash

```
write_with_hash(
    store: AsyncStore,
    path: str,
    content: bytes | AsyncIterator[bytes],
    *,
    algorithm: str = "sha256",
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* and return a `WriteResult` with `digest` populated.

Hash is computed client-side as data flows to the backend -- zero extra round trips, no buffering for async-iterator input. `source` is preserved from the underlying `store.write()` result.

Parameters:

- **`store`** (`AsyncStore`) – Target async store.
- **`path`** (`str`) – Destination path.
- **`content`** (`bytes | AsyncIterator[bytes]`) – bytes or AsyncIterator[bytes].
- **`algorithm`** (`str`, default: `'sha256'` ) – hashlib algorithm name. Default "sha256".
- **`overwrite`** (`bool`, default: `False` ) – Same semantics as AsyncStore.write.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user metadata; subject to USER_METADATA gate.

Returns:

- `WriteResult` – WriteResult with digest populated from the client-side hash.

# Backends

API reference for all storage backend classes. Each backend implements the [`Backend`](https://docs.remotestore.dev/stable/reference/api/backend/index.md) protocol. For usage guides, see [Backends](https://docs.remotestore.dev/stable/guides/backends/index.md).

| Class                                                                                              | Description                                                  |
| -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| [LocalBackend](https://docs.remotestore.dev/stable/reference/api/backends/local/index.md)          | Local filesystem storage                                     |
| [MemoryBackend](https://docs.remotestore.dev/stable/reference/api/backends/memory/index.md)        | In-process storage for testing                               |
| [ReadOnlyHttpBackend](https://docs.remotestore.dev/stable/reference/api/backends/http/index.md)    | Read-only access to HTTP/HTTPS URLs                          |
| [S3Backend](https://docs.remotestore.dev/stable/reference/api/backends/s3/index.md)                | Amazon S3 and S3-compatible services                         |
| [S3PyArrowBackend](https://docs.remotestore.dev/stable/reference/api/backends/s3-pyarrow/index.md) | S3 via PyArrow C++ for higher throughput                     |
| [SFTPBackend](https://docs.remotestore.dev/stable/reference/api/backends/sftp/index.md)            | SSH/SFTP server storage via paramiko                         |
| [AzureBackend](https://docs.remotestore.dev/stable/reference/api/backends/azure/index.md)          | Azure Blob Storage and ADLS Gen2                             |
| [SQLBlobBackend](https://docs.remotestore.dev/stable/reference/api/backends/sql-blob/index.md)     | SQL database blob storage via SQLAlchemy                     |
| [SQLQueryBackend](https://docs.remotestore.dev/stable/reference/api/backends/sql-query/index.md)   | Read-only SQL query materialization via SQLAlchemy + PyArrow |

## Async-native backends

These backends run natively on the event loop under [`AsyncStore`](https://docs.remotestore.dev/stable/reference/api/aio/store/index.md); they live in `remote_store.aio.backends`. Any synchronous backend above also works under `AsyncStore` via the thread-pool [`SyncBackendAdapter`](https://docs.remotestore.dev/stable/reference/api/aio/adapters/index.md).

| Class                                                                                                | Description                                                              |
| ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| [AsyncMemoryBackend](https://docs.remotestore.dev/stable/reference/api/aio/backends/memory/index.md) | In-memory async backend for testing                                      |
| [AsyncAzureBackend](https://docs.remotestore.dev/stable/reference/api/aio/backends/azure/index.md)   | Native async Azure Blob Storage and ADLS Gen2                            |
| [GraphBackend](https://docs.remotestore.dev/stable/reference/api/aio/backends/graph/index.md)        | Microsoft Graph backend (OneDrive, SharePoint, Teams files) — async-only |

## See also

- [Backend guides](https://docs.remotestore.dev/stable/guides/backends/index.md) — configuration and usage guides for all backends
- [Choosing a Backend](https://docs.remotestore.dev/stable/guides/choosing-a-backend/index.md) — trade-offs and selection criteria
- [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) — per-backend capability comparison

# AzureBackend

API reference for `AzureBackend` — stores files in Azure Blob Storage and ADLS Gen2. Behavior adapts to the declared `hns` value (Hierarchical Namespace); there is no runtime auto-detection.

## AzureBackend

```
AzureBackend(
    container: str,
    *,
    hns: bool | None = None,
    account_name: str | None = None,
    account_url: str | None = None,
    account_key: str | Secret | None = None,
    sas_token: str | Secret | None = None,
    connection_string: str | Secret | None = None,
    credential: Any | None = None,
    client_options: dict[str, Any] | None = None,
    retry: RetryPolicy | None = None,
    max_concurrency: int = 1,
    reject_write_under_file_ancestor: bool = False,
)
```

Azure Storage backend.

Uses the Blob SDK for non-HNS accounts (plain Blob Storage, Azurite) and the DataLake SDK for HNS accounts (ADLS Gen2) to get atomic rename and real directory support.

Whether the account is HNS (ADLS Gen2) is declared explicitly via the required `hns` argument -- the backend does not probe for it.

`move()` on non-HNS accounts is implemented as a server-side copy followed by a blob delete. This is non-atomic: a failure between the two steps may leave both source and destination present. HNS accounts use `rename_file` which *is* atomic, so an HNS instance could in principle advertise `ATOMIC_MOVE`. The capability is deliberately not declared per-instance: `CAPABILITIES` is a single class-level set shared by HNS and non-HNS instances alike, so it reports the guarantee common to both rather than varying by the declared `hns` value.

Parameters:

- **`container`** (`str`) – Azure Storage container name (required, non-empty).
- **`hns`** (`bool | None`, default: `None` ) – Whether the storage account has Hierarchical Namespace enabled (ADLS Gen2). Required -- there is no default and no runtime auto-detection. Pass True for ADLS Gen2 accounts (atomic rename, real directories) or False for flat Blob Storage. Use AzureUtils.detect_hns() to discover the value once if you do not already know it.
- **`account_name`** (`str | None`, default: `None` ) – Storage account name.
- **`account_url`** (`str | None`, default: `None` ) – Full account URL (e.g. https://myaccount.dfs.core.windows.net).
- **`account_key`** (`str | Secret | None`, default: `None` ) – Storage account key.
- **`sas_token`** (`str | Secret | None`, default: `None` ) – Shared Access Signature token.
- **`connection_string`** (`str | Secret | None`, default: `None` ) – Azure Storage connection string.
- **`credential`** (`Any | None`, default: `None` ) – Any credential object (e.g. DefaultAzureCredential()).
- **`client_options`** (`dict[str, Any] | None`, default: `None` ) – Additional options passed to service clients. The library sets max_single_put_size, max_block_size, and min_large_block_upload_threshold defaults for streaming memory discipline; user-supplied values take precedence.
- **`retry`** (`RetryPolicy | None`, default: `None` ) – Retry policy for transient failures.
- **`max_concurrency`** (`int`, default: `1` ) – Maximum number of parallel connections for uploads and downloads (default 1 -- sequential).
- **`reject_write_under_file_ancestor`** (`bool`, default: `False` ) – If True, write / write_atomic / open_atomic / move / copy HEAD each slash-aligned ancestor of the target path on non-HNS accounts and raise InvalidPath on the first regular-file hit, matching the cross-backend contract that hierarchical filesystems enforce natively. On HNS accounts the kwarg short-circuits: hdi_isfolder rejects the operation natively, and the backend detects the file ancestor on that rejection and re-raises it as InvalidPath, so HNS delivers the cross-backend contract with or without the kwarg set. Default False: enabling the check adds one HEAD per ancestor per nested-path write; paths without slashes short-circuit.

### check_health

```
check_health() -> None
```

Verify the backend is reachable and credentials are valid.

Raises:

- `PermissionDenied` – If credentials are invalid.
- `NotFound` – If the container does not exist.
- `BackendUnavailable` – If the backend cannot be reached.

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` with Azure-specific details.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – Plan with kind="azure" and details containing
- `ResolutionPlan` – container and account_url.

### exists

```
exists(path: str) -> bool
```

Return `True` if a blob or folder exists at *path*; never `NotFound`.

Probes the blob first (one HEAD); if absent, probes for a folder (an HNS directory, or any blob under the `path/` prefix on flat accounts). The root always exists.

Raises:

- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if *path* is an existing blob (not an HNS directory marker).

One HEAD round-trip; a missing blob or an `hdi_isfolder` directory returns `False`.

Raises:

- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if *path* is an existing folder (HNS directory or non-HNS prefix).

The root is always a folder. Costs one directory HEAD (HNS) or a one-item prefix listing (flat).

Raises:

- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### read

```
read(path: str) -> BinaryIO
```

Open *path* for reading and return a streaming handle.

Streams the blob body in chunks, so memory stays constant regardless of size. On HNS accounts one extra HEAD confirms *path* is a file (not an `hdi_isfolder` directory) before streaming.

Raises:

- `NotFound` – If the blob does not exist.
- `InvalidPath` – If path names a directory.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### read_seekable

```
read_seekable(path: str) -> BinaryIO
```

Open *path* for random-access reading and return a seekable handle.

Overrides the base spool: instead of buffering the whole blob, each `read()` issues one HTTP Range request (`download_blob(offset=, length=)`), so only the byte ranges actually read are fetched — ideal for Parquet column pruning.

Raises:

- `NotFound` – If the blob does not exist.
- `InvalidPath` – If path names a directory.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read and return the full blob content as bytes.

Downloads the whole blob into memory (unlike the lazy `read` stream).

Raises:

- `NotFound` – If the blob does not exist.
- `InvalidPath` – If path names a directory (HNS).
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* as a blob.

The content is uploaded with `upload_blob` on both flat and HNS accounts — a block-blob upload that becomes visible only when its final commit succeeds, never as a partially written blob. On flat (non-HNS) accounts this is an atomic replace. On hierarchical-namespace (HNS) accounts a guaranteed atomic replace is not assured; use `write_atomic` there when readers must never observe an intermediate state.

Raises:

- `AlreadyExists` – If the blob exists and overwrite is False.
- `InvalidPath` – If path names a directory, or (with the reject_write_under_file_ancestor opt-in, or natively on HNS) an ancestor exists as a file.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* atomically.

Readers never observe a partial file. On flat accounts this is the plain `PUT` (already atomic). On HNS it streams to a hidden temp file and promotes it with an atomic `rename_file` (the temp file is cleaned up on failure).

Raises:

- `AlreadyExists` – If the blob exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor exists as a file.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Yield a writable buffer promoted to *path* atomically on clean exit.

Writes spool to a temporary file (up to 8 MB in memory, then on disk). On flat accounts the buffer is committed with one `PUT`; on HNS it is uploaded to a temp blob and promoted with an atomic `rename_file`. An exception before exit leaves *path* untouched.

Raises:

- `AlreadyExists` – If the blob exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor exists as a file.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete the blob at *path*.

Raises:

- `NotFound` – If the blob does not exist (or, on HNS, a path component is itself a file) and missing_ok is False.
- `InvalidPath` – If path names a directory (HNS; use delete_folder).
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete a folder.

Parameters:

- **`path`** (`str`) – Backend-relative key.
- **`recursive`** (`bool`, default: `False` ) – If True, delete all contents first.
- **`missing_ok`** (`bool`, default: `False` ) – If True, do not raise when absent.

Raises:

- `NotFound` – If the folder is missing and missing_ok is False.
- `InvalidPath` – If path names a file (use delete instead).
- `DirectoryNotEmpty` – If non-empty and recursive is False.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> Iterator[FileInfo]
```

Yield files under *path*, one `FileInfo` at a time.

Lazily pages the service listing (`walk_blobs`/`list_blobs` on flat accounts, `get_paths` on HNS); a missing path or a path under a file ancestor yields nothing. `recursive` lists the whole prefix (`max_depth` prunes client-side).

Raises:

- `PermissionDenied` – If credentials are rejected or lack access (401/403), surfaced during iteration.
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure, surfaced during iteration.

### list_folders

```
list_folders(path: str) -> Iterator[FolderEntry]
```

Yield immediate subfolders of *path* as `FolderEntry` records.

One paged prefix listing (`walk_blobs` common-prefixes on flat accounts, non-recursive `get_paths` on HNS); a missing path yields nothing.

Raises:

- `PermissionDenied` – If credentials are rejected or lack access (401/403), surfaced during iteration.
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure, surfaced during iteration.

### iter_children

```
iter_children(
    path: str,
) -> Iterator[FileInfo | FolderEntry]
```

Yield the immediate files and folders under *path* in one paged listing.

Overrides the base two-pass default with a single `walk_blobs` (flat) or `get_paths` (HNS) pass, yielding `FileInfo` for files and `FolderEntry` for folders. A missing path yields nothing.

Raises:

- `PermissionDenied` – If credentials are rejected or lack access (401/403), surfaced during iteration.
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure, surfaced during iteration.

### glob

```
glob(pattern: str) -> Iterator[FileInfo]
```

Match files against a glob pattern.

Parameters:

- **`pattern`** (`str`) – Glob pattern (e.g., "data/\*.csv", "\*\*/\*.txt").

Returns:

- `Iterator[FileInfo]` – An iterator of matching FileInfo objects.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return file metadata for `path`.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path names a directory (HNS: hdi_isfolder=true).

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Return aggregate metadata for the folder at *path*.

File count, total size, and latest modification time are gathered by paging the whole subtree listing, so cost scales with the number of descendants.

Raises:

- `NotFound` – If the folder does not exist.
- `InvalidPath` – If path names a file, not a folder.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename the file *src* to *dst*.

On HNS accounts this is a single native `rename_file` (atomic). On flat accounts it is a server-side copy followed by a delete — not atomic, so a failure between the two steps can leave both *src* and *dst* present; `ATOMIC_MOVE` is therefore not declared. `src == dst` is a no-op.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory, or an ancestor of dst exists as a file.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy the file *src* to *dst* via a server-side copy.

Issues `start_copy_from_url` so the bytes never pass through the client. The copy is not atomic — a failure can leave a partial destination. `src == dst` is a no-op.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory, or an ancestor of dst exists as a file.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `PermissionDenied` – If credentials are rejected or lack access (401/403).
- `BackendUnavailable` – On throttling (429), 5xx, or transport failure.

## See also

- [Azure Backend Guide](https://docs.remotestore.dev/stable/guides/backends/azure/index.md) — usage patterns, configuration, and examples
- [AzureUtils](https://docs.remotestore.dev/stable/reference/api/azure-utils/index.md) — discover an account's HNS status with `detect_hns()`
- [Azure Backend example](https://docs.remotestore.dev/stable/tutorial/examples/azure-backend/index.md) — Azure backend in action

# ReadOnlyHttpBackend

API reference for `ReadOnlyHttpBackend` — read-only access to files over HTTP/HTTPS. Supports `READ`, `METADATA`, and `LAZY_READ` capabilities only (`read()` streams lazily; write, delete, list, move, and copy are unsupported).

## ReadOnlyHttpBackend

```
ReadOnlyHttpBackend(
    base_url: str,
    *,
    headers: dict[str, str] | None = None,
    timeout: float = 30.0,
    retry: RetryPolicy | None = None,
    http_client: str | None = None,
    verify_ssl: bool = True,
    max_redirects: int = 5,
)
```

Read-only backend for HTTP/HTTPS URLs.

Treats an HTTP endpoint as a file store with `{READ, METADATA, LAZY_READ}` capabilities (`read()` streams the response body lazily rather than buffering the whole file). Write, delete, list, move, and copy operations raise `CapabilityNotSupported`.

Parameters:

- **`base_url`** (`str`) – Root URL. A trailing / is appended if missing.
- **`headers`** (`dict[str, str] | None`, default: `None` ) – Custom headers sent with every request (e.g. API keys).
- **`timeout`** (`float`, default: `30.0` ) – Request timeout in seconds.
- **`retry`** (`RetryPolicy | None`, default: `None` ) – Retry policy for transient errors.
- **`http_client`** (`str | None`, default: `None` ) – Force a specific transport ("urllib", "requests", or "httpx"). Auto-detected if None.
- **`verify_ssl`** (`bool`, default: `True` ) – Whether to verify TLS certificates.
- **`max_redirects`** (`int`, default: `5` ) – Maximum number of redirects to follow.

### exists

```
exists(path: str) -> bool
```

Check existence via HEAD request (falls back to ranged GET).

### is_file

```
is_file(path: str) -> bool
```

HTTP resources are always files.

### is_folder

```
is_folder(path: str) -> bool
```

HTTP has no folder concept — always returns False.

### read

```
read(path: str) -> BinaryIO
```

Stream-read a file via GET.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Buffered-read a file via GET.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Get file metadata via HEAD request (falls back to ranged GET).

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

HTTP has no folder concept — always raises NotFound.

### check_health

```
check_health() -> None
```

Verify connectivity by sending HEAD to base_url (or GET if HEAD is blocked).

Note

The health check probes `base_url` (the root), not a specific file. Many HTTP servers and CDNs return 403 or 404 for directory URLs while serving individual files normally. A failing health check therefore does not necessarily mean `read()` or `exists()` will fail on actual file paths.

### close

```
close() -> None
```

Close the underlying transport.

### unwrap

```
unwrap(type_hint: type[T]) -> T
```

Return the transport if it matches the requested type.

### native_path

```
native_path(path: str) -> str
```

Return the full URL for a backend-relative key.

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` with HTTP-specific details.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – Plan with kind="http" and details containing
- `ResolutionPlan` – url and method.

### to_key

```
to_key(native_path: str) -> str
```

Strip base_url prefix to get a backend-relative key.

## Interop (Backend-Specific)

Backend-specific methods

`unwrap`, `native_path`, and `to_key` expose backend internals. Using them ties your code to `ReadOnlyHttpBackend`. For portable alternatives, use the methods from [Backend](https://docs.remotestore.dev/stable/reference/api/backend/index.md).

## See also

- [HTTP Backend Guide](https://docs.remotestore.dev/stable/guides/backends/http/index.md) — usage patterns, configuration, and examples
- [HTTP Backend example](https://docs.remotestore.dev/stable/tutorial/examples/http-backend/index.md) — read-only HTTP access in action

# LocalBackend

API reference for `LocalBackend` — stores files on the local filesystem. Built-in, no extra dependencies required.

## LocalBackend

```
LocalBackend(root: str)
```

Local filesystem backend using only the Python standard library.

`move()` uses `shutil.move`, which calls `os.rename` for same-filesystem moves (atomic) but falls back to copy-then-delete for cross-filesystem moves (not atomic). `ATOMIC_MOVE` is declared because within-root moves are always same-filesystem.

Parameters:

- **`root`** (`str`) – Absolute path to the root directory on the local filesystem.

### check_health

```
check_health() -> None
```

Confirm the root directory exists and is readable.

A lightweight two-syscall probe (`exists` then `os.access`) that touches no file content and is safe to call repeatedly.

Raises:

- `NotFound` – If the root directory does not exist.
- `PermissionDenied` – If the root directory exists but is not readable.

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` with local filesystem details.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – Plan with kind="local" and details containing
- `ResolutionPlan` – root and absolute_path.

### exists

```
exists(path: str) -> bool
```

Return `True` if a file or folder exists at *path*; never `NotFound`.

Returns `False` when a path component is itself a file, since the directory descent cannot proceed.

Raises:

- `InvalidPath` – If path escapes the backend root.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if *path* is an existing regular file.

`False` if *path* is absent, names a directory, or has a file as an ancestor component.

Raises:

- `InvalidPath` – If path escapes the backend root.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if *path* is an existing directory.

`False` if *path* is absent, names a file, or has a file as an ancestor component.

Raises:

- `InvalidPath` – If path escapes the backend root.

### read

```
read(path: str) -> BinaryIO
```

Open *path* for reading and return a lazy binary stream.

The stream reads on demand from the open file handle, so memory stays constant regardless of file size.

Raises:

- `NotFound` – If the file does not exist, or a path component is itself a file (the directory descent fails with ENOTDIR).
- `InvalidPath` – If path names a directory.
- `PermissionDenied` – If the OS denies read access to the file.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read and return the full file content as bytes.

Materialises the whole file in memory (unlike the lazy `read` stream), so size is bounded by available memory.

Raises:

- `NotFound` – If the file does not exist, or a path component is itself a file (ENOTDIR on descent).
- `InvalidPath` – If path names a directory.
- `PermissionDenied` – If the OS denies read access to the file.

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path*, creating parent directories as needed.

The bytes land directly at the final path (no temp-and-rename), so a crash or error mid-write can leave a partial or truncated file there — use `write_atomic` when readers must never observe a half-written file.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor of path exists as a regular file.
- `PermissionDenied` – If the OS denies write access.

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* atomically via a temp file plus `os.replace`.

Readers never observe a partial file: the body is written to a temporary file in the destination directory and atomically renamed over *path* on success (the temp file is removed on error). The rename is atomic because the temp file shares *path*'s directory, hence its filesystem.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor of path exists as a regular file.
- `PermissionDenied` – If the OS denies write access.

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Yield a writable stream promoted to *path* atomically on clean exit.

Writes go to a temp file in *path*'s directory; on normal exit it is atomically renamed over *path* (`os.replace`), and on any exception the temp file is removed and *path* is left untouched. Readers therefore never see a partially written file.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor of path exists as a regular file.
- `PermissionDenied` – If the OS denies write access.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete the file at *path*.

Raises:

- `NotFound` – If the file does not exist (or a path component is itself a file) and missing_ok is False.
- `InvalidPath` – If path names a directory — a type mismatch that missing_ok does not silence (use delete_folder).
- `PermissionDenied` – If the OS denies the unlink.

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete the folder at *path*.

`recursive=True` removes the folder and its whole subtree (`shutil.rmtree`); this is not atomic — an error partway through can leave the tree partially deleted. `recursive=False` removes only an empty folder (`rmdir`).

Raises:

- `NotFound` – If the folder does not exist and missing_ok is False.
- `InvalidPath` – If path names a file, not a folder.
- `DirectoryNotEmpty` – If the folder is non-empty and recursive is False.
- `PermissionDenied` – If the OS denies removal.

### glob

```
glob(pattern: str) -> Iterator[FileInfo]
```

Yield files matching *pattern* under the root, one at a time.

Lazily walks `Path.glob`; entries that escape the root through a symlink are skipped rather than returned, and directories are omitted — only files are yielded.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> Iterator[FileInfo]
```

Yield files under *path*, one `FileInfo` at a time.

Lazy: entries are produced as the directory is scanned, so listing a large tree holds only the current entry in memory. A missing or non-folder *path* yields nothing (no error). With `recursive` and `max_depth` set, traversal is pruned at the depth bound during the `os.walk` rather than filtered afterwards.

### list_folders

```
list_folders(path: str) -> Iterator[FolderEntry]
```

Yield immediate subfolders of *path* as `FolderEntry` records.

Lazy single-level scan; a missing or non-folder *path* yields nothing.

### iter_children

```
iter_children(
    path: str,
) -> Iterator[FileInfo | FolderEntry]
```

Yield the immediate files and folders under *path* in one scan.

Overrides the base (which chains `list_files` and `list_folders`, two passes) to walk the directory once, yielding `FileInfo` for files and `FolderEntry` for folders. A missing or non-folder *path* yields nothing.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return metadata for the file at *path* from a single `stat`.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path names a directory, not a file.

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Return aggregate metadata for the folder at *path*.

File count, total size, and latest modification time are computed by walking the entire subtree and `stat`-ing every file, so cost is O(files-in-subtree) — not a constant-time lookup.

Raises:

- `NotFound` – If the folder does not exist.
- `InvalidPath` – If path names a file, not a folder.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename the file *src* to *dst*.

Atomic when *src* and *dst* are on the same filesystem (`os.rename` via `shutil.move`) — which within a single root they always are, so `ATOMIC_MOVE` is declared. A cross-filesystem move (e.g. a bind-mounted subtree) falls back to copy-then-delete, which is not atomic. `src == dst` is a no-op; parent directories of *dst* are created as needed.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory, or an ancestor of dst exists as a regular file.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `PermissionDenied` – If the OS denies the operation.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy the file *src* to *dst*, preserving mode and mtime (`copy2`).

Not atomic: content is streamed to *dst*, so a crash mid-copy can leave a partial file at the destination. `src == dst` is a no-op; parent directories of *dst* are created as needed.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory, or an ancestor of dst exists as a regular file.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `PermissionDenied` – If the OS denies the operation.

## See also

- [Local Backend Guide](https://docs.remotestore.dev/stable/guides/backends/local/index.md) — usage patterns, configuration, and examples
- [File Operations example](https://docs.remotestore.dev/stable/tutorial/examples/file-operations/index.md) — full Store API demo using LocalBackend

# MemoryBackend

API reference for `MemoryBackend` — stores files in an in-process data structure. No filesystem access, no network. Ideal for testing and prototyping.

## MemoryBackend

```
MemoryBackend()
```

In-memory backend using a tree-indexed data structure.

Zero dependencies, no filesystem access, no network. Designed as a drop-in backend for unit testing, interactive exploration, and documentation examples.

All capabilities except `GLOB` are supported. The full conformance suite passes with zero skips.

Every mutating operation runs under a single process-wide lock, so `write`, `write_atomic`, `move`, and `copy` are atomic with respect to concurrent callers in the same process (`ATOMIC_MOVE` is advertised).

### exists

```
exists(path: str) -> bool
```

Return `True` if a file or folder exists at *path*; never `NotFound`.

The root (`""`) always exists.

Raises:

- `InvalidPath` – If path is absolute or contains a .. segment.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if *path* is an existing file (`False` for the root or a folder).

Raises:

- `InvalidPath` – If path is absolute or contains a .. segment.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if *path* is an existing folder; the root is always a folder.

Raises:

- `InvalidPath` – If path is absolute or contains a .. segment.

### read

```
read(path: str) -> BinaryIO
```

Return a binary stream over the stored bytes for *path*.

The value materialises in memory rather than streaming (`LAZY_READ` is not advertised): the returned stream wraps a copy of the resident bytes, so peak memory scales with the object size. Access is guarded by a single process-wide lock.

Raises:

- `NotFound` – If no file exists at path.
- `InvalidPath` – If path names a folder.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Return the full stored content of *path* as bytes.

Copies the resident value out under the lock; like `read` it holds the whole object in memory.

Raises:

- `NotFound` – If no file exists at path.
- `InvalidPath` – If path names a folder.

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Store *content* at *path*, creating parent folders implicitly.

The whole body is buffered into memory before the store (streams are drained fully first — no lazy write), and the swap-in happens under a single process-wide lock, so the write is atomic with respect to concurrent callers in the same process.

Raises:

- `AlreadyExists` – If a file exists at path and overwrite is False.
- `InvalidPath` – If path is empty or names a folder, or an ancestor of path exists as a file.

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Store *content* at *path* atomically (delegates to `write`).

Memory writes are already atomic under the process-wide lock, so this is exactly `write`; the whole body is buffered first.

Raises:

- `AlreadyExists` – If a file exists at path and overwrite is False.
- `InvalidPath` – If path is empty or names a folder, or an ancestor of path exists as a file.

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Yield an in-memory buffer committed to *path* atomically on clean exit.

Writes accumulate in a `BytesIO`; on exit the buffer is stored via `write` under the process-wide lock, so *path* updates in one atomic step. An exception before exit leaves *path* untouched.

Raises:

- `AlreadyExists` – If a file exists at path and overwrite is False.
- `InvalidPath` – If path is empty, or (on commit) names a folder or has a file ancestor.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete the file at *path*.

Raises:

- `NotFound` – If no file exists at path and missing_ok is False.
- `InvalidPath` – If path is empty or names a folder (a type mismatch missing_ok does not silence).

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete the folder at *path*.

`recursive=True` detaches the whole subtree in one locked step; `recursive=False` removes only an empty folder.

Raises:

- `NotFound` – If no folder exists at path and missing_ok is False.
- `InvalidPath` – If path is empty or names a file.
- `DirectoryNotEmpty` – If the folder is non-empty and recursive is False.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> Iterator[FileInfo]
```

Yield files under *path*.

A missing or non-folder *path* yields nothing. The tree is snapshotted under the lock and iterated outside it, so a long listing does not hold the lock; `recursive` walks the whole subtree (`max_depth` prunes it).

### list_folders

```
list_folders(path: str) -> Iterator[FolderEntry]
```

Yield immediate subfolders of *path* as `FolderEntry` records.

A missing or non-folder *path* yields nothing; children are snapshotted under the lock and yielded outside it.

### iter_children

```
iter_children(
    path: str,
) -> Iterator[FileInfo | FolderEntry]
```

Yield the immediate files and folders under *path* in one snapshot.

Overrides the base two-pass default: takes one locked snapshot of the directory and yields `FileInfo` for files and `FolderEntry` for folders. A missing or non-folder *path* yields nothing.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return metadata for the file at *path* from the in-memory node.

Raises:

- `NotFound` – If no file exists at path (including the empty path).
- `InvalidPath` – If path names a folder.

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Return aggregate metadata for the folder at *path*.

File count, total size, and latest modification time are computed by walking the whole subtree, so cost scales with the number of descendants.

Raises:

- `NotFound` – If no folder exists at path.
- `InvalidPath` – If path names a file.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename the file *src* to *dst* by re-linking the tree node.

Detach-from-source and attach-to-destination happen in one locked step, so the move is atomic with respect to concurrent callers (`ATOMIC_MOVE` is advertised) and the payload is never copied. `src == dst` verifies the source is a file and is otherwise a no-op.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst is empty, src names a folder, or dst names an existing folder.
- `AlreadyExists` – If dst is an existing file and overwrite is False.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy the file *src* to *dst* with an independent byte buffer.

The destination receives a fresh `bytearray` copy of the source bytes, all in one locked step (atomic). `src == dst` is a no-op.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst is empty, src names a folder, or dst names an existing folder.
- `AlreadyExists` – If dst is an existing file and overwrite is False.

## See also

- [Memory Backend Guide](https://docs.remotestore.dev/stable/guides/backends/memory/index.md) — usage patterns, configuration, and examples
- [Memory Backend example](https://docs.remotestore.dev/stable/tutorial/examples/memory-backend/index.md) — in-memory backend in action

# S3PyArrowBackend

API reference for `S3PyArrowBackend` — drop-in alternative to `S3Backend` that uses PyArrow's C++ S3 filesystem for higher throughput on large files.

## S3PyArrowBackend

```
S3PyArrowBackend(
    bucket: str,
    *,
    endpoint_url: str | None = None,
    key: str | Secret | None = None,
    secret: str | Secret | None = None,
    region_name: str | None = None,
    tls_ca_bundle: str | None = None,
    client_options: dict[str, Any] | None = None,
    retry: RetryPolicy | None = None,
    reject_write_under_file_ancestor: bool = False,
)
```

Hybrid S3 backend: PyArrow for reads/writes/copies, s3fs for listing/metadata.

Drop-in alternative to `S3Backend` with the same constructor signature. Uses PyArrow's C++ S3 filesystem for data-path operations (higher throughput for large files) and s3fs for control-path operations (listing, metadata, deletion).

`move()` is implemented as a PyArrow copy followed by an s3fs delete. This is non-atomic: a crash or network error between the two steps may leave both source and destination present. `ATOMIC_MOVE` is not declared.

Parameters:

- **`bucket`** (`str`) – S3 bucket name (required, non-empty).
- **`endpoint_url`** (`str | None`, default: `None` ) – Custom endpoint URL (e.g. for MinIO).
- **`key`** (`str | Secret | None`, default: `None` ) – AWS access key ID.
- **`secret`** (`str | Secret | None`, default: `None` ) – AWS secret access key.
- **`region_name`** (`str | None`, default: `None` ) – AWS region name.
- **`tls_ca_bundle`** (`str | None`, default: `None` ) – Path to a PEM CA bundle file. Falls back to AWS_CA_BUNDLE / REQUESTS_CA_BUNDLE / SSL_CERT_FILE.
- **`client_options`** (`dict[str, Any] | None`, default: `None` ) – Additional options passed to s3fs.
- **`reject_write_under_file_ancestor`** (`bool`, default: `False` ) – If True, write / write_atomic / open_atomic / move / copy HEAD each slash-aligned ancestor of the target path and raise InvalidPath on the first regular-file hit, matching the cross-backend contract that hierarchical filesystems enforce natively. Default False: each nested-path write otherwise pays one HEAD per ancestor; paths without slashes short-circuit.

### read

```
read(path: str) -> BinaryIO
```

Open *path* for reading and return a streaming handle.

Uses PyArrow's `open_input_file` (higher throughput for large objects) rather than the s3fs reader; still lazy, so memory stays constant.

Raises:

- `NotFound` – If the object does not exist.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read and return the full object content as bytes (via PyArrow).

Raises:

- `NotFound` – If the object does not exist.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path*, streaming straight to a multipart upload.

Unlike `S3Backend.write`, a plain streamed write here is **not** atomic: PyArrow's output stream exposes no abort, so a failure mid-body finalises a *truncated* object at *path*. Use `write_atomic` when readers must never observe a partial object. A `bytes` payload (no streaming) commits in one shot.

Raises:

- `AlreadyExists` – If the object exists and overwrite is False.
- `InvalidPath` – With the reject_write_under_file_ancestor opt-in, if an ancestor of path exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* atomically by buffering before upload.

The whole body is buffered first (a `bytes` payload is already materialised and delegates straight through), so a source failure happens off the wire and leaves no object at *path* — closing the atomicity gap in the plain streaming `write`.

Raises:

- `AlreadyExists` – If the object exists and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of path exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Yield a writable buffer committed to *path* atomically on clean exit.

Writes spool to a temporary file (up to 8 MB in memory, then on disk) and upload only on clean exit, so *path* never holds a partial object. An exception before exit leaves *path* untouched.

Raises:

- `AlreadyExists` – If the object exists and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of path exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename the object *src* to *dst*.

Existence checks and the delete go through s3fs; the copy is a PyArrow `copy_file`. Copy-then-delete is not atomic — a failure between the two steps can leave both *src* and *dst* present — so `ATOMIC_MOVE` is not declared. `src == dst` is a no-op.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of dst exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy the object *src* to *dst* via a PyArrow `copy_file`.

Like `move`, the operation carries no cross-operation atomicity guarantee. `src == dst` is a no-op.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of dst exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

## See also

- [S3-PyArrow Backend Guide](https://docs.remotestore.dev/stable/guides/backends/s3-pyarrow/index.md) — usage patterns, configuration, and examples
- [S3-PyArrow Backend example](https://docs.remotestore.dev/stable/tutorial/examples/s3-pyarrow-backend/index.md) — S3-PyArrow backend in action

# S3Backend

API reference for `S3Backend` — stores files on Amazon S3 or any S3-compatible service (MinIO, DigitalOcean Spaces, etc.).

## S3Backend

```
S3Backend(
    bucket: str,
    *,
    endpoint_url: str | None = None,
    key: str | Secret | None = None,
    secret: str | Secret | None = None,
    region_name: str | None = None,
    tls_ca_bundle: str | None = None,
    client_options: dict[str, Any] | None = None,
    retry: RetryPolicy | None = None,
    reject_write_under_file_ancestor: bool = False,
)
```

S3-compatible object storage backend using s3fs.

`move()` is implemented as a server-side copy followed by a delete. This is non-atomic: a crash or network error between the two steps may leave both source and destination present. `ATOMIC_MOVE` is not declared.

Parameters:

- **`bucket`** (`str`) – S3 bucket name (required, non-empty).
- **`endpoint_url`** (`str | None`, default: `None` ) – Custom endpoint URL (e.g. for MinIO).
- **`key`** (`str | Secret | None`, default: `None` ) – AWS access key ID.
- **`secret`** (`str | Secret | None`, default: `None` ) – AWS secret access key.
- **`region_name`** (`str | None`, default: `None` ) – AWS region name.
- **`tls_ca_bundle`** (`str | None`, default: `None` ) – Path to a PEM CA bundle file. Falls back to AWS_CA_BUNDLE / REQUESTS_CA_BUNDLE / SSL_CERT_FILE.
- **`client_options`** (`dict[str, Any] | None`, default: `None` ) – Additional options passed to s3fs.
- **`reject_write_under_file_ancestor`** (`bool`, default: `False` ) – If True, write / write_atomic / open_atomic / move / copy HEAD each slash-aligned ancestor of the target path and raise InvalidPath on the first regular-file hit, matching the cross-backend contract that hierarchical filesystems enforce natively. Default False: each nested-path write otherwise pays one HEAD per ancestor; paths without slashes short-circuit.

### check_health

```
check_health() -> None
```

Confirm the bucket is reachable and credentials valid via one `HeadBucket`.

Raises:

- `NotFound` – If the bucket does not exist.
- `PermissionDenied` – If the credentials are rejected or lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### exists

```
exists(path: str) -> bool
```

Return `True` if an object or prefix exists at *path*; never `NotFound`.

Raises:

- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### is_file

```
is_file(path: str) -> bool
```

Return `True` if *path* is an existing object (`False` if absent or a prefix).

Raises:

- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if *path* is an existing virtual folder (a common prefix).

Raises:

- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### read

```
read(path: str) -> BinaryIO
```

Open *path* for reading and return a streaming handle.

s3fs reads the object lazily in range-backed chunks, so memory stays constant regardless of size.

Raises:

- `NotFound` – If the object does not exist.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read and return the full object content as bytes.

Downloads the whole object into memory (unlike the lazy `read` stream).

Raises:

- `NotFound` – If the object does not exist.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* as an S3 object.

The upload commits atomically — a reader sees either the old object or the new one, never a partial. A streamed write that fails mid-body aborts the multipart upload rather than finalising a truncated object, so no partial object is ever left at *path*.

Raises:

- `AlreadyExists` – If the object exists and overwrite is False.
- `InvalidPath` – With the reject_write_under_file_ancestor opt-in, if an ancestor of path exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* atomically (delegates to `write`).

An S3 `PUT` is already atomic, so this is exactly `write`.

Raises:

- `AlreadyExists` – If the object exists and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of path exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Yield a writable buffer committed to *path* atomically on clean exit.

Writes spool to a temporary file (up to 8 MB in memory, then on disk); on clean exit the buffer is uploaded in a single atomic `PUT`. An exception before exit leaves *path* untouched.

Raises:

- `AlreadyExists` – If the object exists and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of path exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete the object at *path*.

Raises:

- `NotFound` – If the object does not exist and missing_ok is False.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete the virtual folder at *path*.

`recursive=True` removes every object under the prefix; this is a best-effort multi-object delete, not atomic, so an interruption can leave the prefix partially deleted. `recursive=False` removes the prefix only when it has no contents.

Raises:

- `NotFound` – If no object exists under path and missing_ok is False.
- `DirectoryNotEmpty` – If the prefix is non-empty and recursive is False.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return metadata for the object at *path* from one `HeadObject`.

The HEAD is issued with `ChecksumMode=ENABLED` so a stored checksum surfaces as the `FileInfo` digest.

Raises:

- `NotFound` – If the object does not exist.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename the object *src* to *dst*.

Implemented as a server-side copy followed by a delete of *src*. This is not atomic — a crash or network error between the two steps can leave both *src* and *dst* present — so `ATOMIC_MOVE` is not declared. `src == dst` is a no-op.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of dst exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy the object *src* to *dst* via a server-side `CopyObject`.

The bytes are copied entirely server-side (never through the client). Like `move`, the operation carries no cross-operation atomicity guarantee. `src == dst` is a no-op.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `InvalidPath` – With the opt-in, if an ancestor of dst exists as an object.
- `PermissionDenied` – If the credentials lack access.
- `BackendUnavailable` – On a transport or service failure, or after close().

## See also

- [S3 Backend Guide](https://docs.remotestore.dev/stable/guides/backends/s3/index.md) — usage patterns, configuration, and examples
- [S3 Backend example](https://docs.remotestore.dev/stable/tutorial/examples/s3-backend/index.md) — S3 backend in action

# SFTPBackend

API reference for `SFTPBackend` — stores files on any SSH/SFTP server using paramiko. Explicit host key verification and Azure Key Vault PEM support.

## SFTPBackend

```
SFTPBackend(
    host: str,
    *,
    port: int = 22,
    username: str | None = None,
    password: str | Secret | None = None,
    pkey: Any = None,
    base_path: str = "/",
    host_key_policy: HostKeyPolicy | str = STRICT,
    known_host_keys: str | None = None,
    host_keys_path: str | None = None,
    config: dict[str, Any] | None = None,
    timeout: int = 10,
    connect_kwargs: dict[str, Any] | None = None,
    retry: RetryPolicy | None = None,
)
```

SFTP backend using pure paramiko.

`move()` attempts `posix_rename` (atomic on POSIX-compliant servers), then falls back to `rename`, and finally to a stream copy followed by a delete. Because atomicity cannot be guaranteed across all servers, `ATOMIC_MOVE` is not declared.

Warning

**Not thread-safe for concurrent access.** This backend maintains a single SSH/SFTP connection (paramiko `SFTPClient`), which is not safe to call from multiple threads simultaneously. Concurrent calls via `SyncBackendAdapter` and `asyncio.gather` will race on the shared socket and may hang or corrupt responses. Create one `SFTPBackend` instance per thread if you need parallel operations.

Parameters:

- **`host`** (`str`) – SFTP server hostname (required, non-empty).
- **`port`** (`int`, default: `22` ) – SSH port (default: 22).
- **`username`** (`str | None`, default: `None` ) – SSH username.
- **`password`** (`str | Secret | None`, default: `None` ) – SSH password.
- **`pkey`** (`Any`, default: `None` ) – paramiko.PKey instance for key-based auth.
- **`base_path`** (`str`, default: `'/'` ) – Root path on the remote server (default: /).
- **`host_key_policy`** (`HostKeyPolicy | str`, default: `STRICT` ) – Host key verification policy (see SFTPUtils.HostKeyPolicy). Accepts enum value or string.
- **`known_host_keys`** (`str | None`, default: `None` ) – Known hosts string (code-level override).
- **`host_keys_path`** (`str | None`, default: `None` ) – Path to known_hosts file (default: ~/.ssh/known_hosts).
- **`config`** (`dict[str, Any] | None`, default: `None` ) – Optional config dict (may contain known_host_keys).
- **`timeout`** (`int`, default: `10` ) – SSH connection timeout in seconds.
- **`connect_kwargs`** (`dict[str, Any] | None`, default: `None` ) – Extra kwargs passed to SSHClient.connect().

### check_health

```
check_health() -> None
```

Confirm the SFTP connection works by `stat`-ing the base path.

Establishes the SSH/SFTP connection lazily if needed (retried at connection scope) and issues one `stat` round-trip.

Raises:

- `NotFound` – If the configured base path does not exist.
- `PermissionDenied` – If the server denies access to the base path.
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established.

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` with SFTP-specific details.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – Plan with kind="sftp" and details containing
- `ResolutionPlan` – host, port, and base_path.

### exists

```
exists(path: str) -> bool
```

Return `True` if a file or folder exists at *path*; never `NotFound`.

Issues one `stat` round-trip; a missing path returns `False`.

Raises:

- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if *path* is an existing regular file (`False` if absent or a folder).

Raises:

- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if *path* is an existing directory (`False` if absent or a file).

Raises:

- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### read

```
read(path: str) -> BinaryIO
```

Open *path* for reading and return a buffered, streaming handle.

Reads lazily over the SFTP channel (wrapped in a `BufferedReader`), so memory stays constant regardless of file size; one extra `stat` round-trip guards against *path* being a directory before the open.

Raises:

- `NotFound` – If the file does not exist, or a path component is itself a file.
- `InvalidPath` – If path names a directory.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails mid-read.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Read and return the full file content as bytes.

Prefetches and materialises the whole file in memory (unlike the lazy `read` stream).

Raises:

- `NotFound` – If the file does not exist, or a path component is itself a file.
- `InvalidPath` – If path names a directory.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails mid-read.

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path*, streaming it over the SFTP channel.

The bytes are streamed straight to the destination file (no temp-and-rename), so a dropped connection mid-write can leave a partial or truncated file there — use `write_atomic` when readers must never see a half-written file. Missing parent directories are created first (one `stat` per ancestor).

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor of path exists as a regular file.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails mid-write.

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* atomically via a temp file plus server rename.

Readers never observe a partial file: the body is streamed to a hidden temp file in the destination directory, then promoted with `posix_rename` (atomic on POSIX-compliant servers). Servers without `posix_rename` fall back to a plain `rename` (non-atomic overwrite: the target is removed first), and the temp file is cleaned up on failure.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor of path exists as a regular file.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Yield a writable handle promoted to *path* atomically on clean exit.

Writes stream to a hidden temp file in the destination directory; on clean exit it is promoted with `posix_rename` (atomic on POSIX servers, falling back to `rename`), and on any exception the temp file is removed and *path* is left untouched.

Raises:

- `AlreadyExists` – If the file exists and overwrite is False.
- `InvalidPath` – If path names a directory, or an ancestor of path exists as a regular file.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete the file at *path*.

Raises:

- `NotFound` – If the file does not exist (or a path component is itself a file) and missing_ok is False.
- `InvalidPath` – If path names a directory (use delete_folder).
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete the folder at *path*.

`recursive=True` walks and removes the subtree bottom-up (one round-trip per entry — not atomic; an interruption can leave the tree partially removed). `recursive=False` removes only an empty folder after checking it has no entries.

Raises:

- `NotFound` – If the folder does not exist and missing_ok is False.
- `InvalidPath` – If path names a file, not a folder.
- `DirectoryNotEmpty` – If the folder is non-empty and recursive is False.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> Iterator[FileInfo]
```

Yield files under *path*, one `FileInfo` at a time.

Lazily walks the remote directory (`listdir_attr`); a missing *path* yields nothing. `recursive` descends via one directory-listing round-trip per folder (`max_depth` bounds the descent). Failures other than a missing path surface as `RemoteStoreError` during iteration.

### list_folders

```
list_folders(path: str) -> Iterator[FolderEntry]
```

Yield immediate subfolders of *path* as `FolderEntry` records.

One directory-listing round-trip; a missing *path* yields nothing, and other failures surface as `RemoteStoreError` during iteration.

### iter_children

```
iter_children(
    path: str,
) -> Iterator[FileInfo | FolderEntry]
```

Yield the immediate files and folders under *path* in one listing.

Overrides the base two-pass default with a single `listdir_attr` round-trip, yielding `FileInfo` for files and `FolderEntry` for folders. A missing *path* yields nothing; other failures surface as `RemoteStoreError` during iteration.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return metadata for the file at *path* from a single `stat` round-trip.

Raises:

- `NotFound` – If the file does not exist.
- `InvalidPath` – If path names a directory, not a file.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Return aggregate metadata for the folder at *path*.

File count, total size, and latest modification time are gathered by recursively walking the whole subtree (one listing round-trip per folder), so cost scales with the number of descendants.

Raises:

- `NotFound` – If the folder does not exist.
- `InvalidPath` – If path names a file, not a folder.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move or rename the file *src* to *dst*.

Tries `posix_rename` first (atomic on POSIX-compliant servers), then a plain `rename`, and finally a stream copy-then-delete. Because the outcome depends on server support, atomicity is not guaranteed across all servers and `ATOMIC_MOVE` is not declared. `src == dst` is a no-op; missing parent directories of *dst* are created first.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory, or an ancestor of dst exists as a regular file.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy the file *src* to *dst* by streaming through the client.

SFTP has no server-side copy, so the bytes round-trip through the client (download then upload); this is not atomic — an interruption can leave a partial file at *dst*. `src == dst` is a no-op; missing parent directories of *dst* are created first.

Raises:

- `NotFound` – If src does not exist.
- `InvalidPath` – If src or dst names a directory, or an ancestor of dst exists as a regular file.
- `AlreadyExists` – If dst exists, src != dst, and overwrite is False.
- `PermissionDenied` – If the server denies access (EACCES).
- `BackendUnavailable` – If the SSH/SFTP connection cannot be established or fails.

## See also

- [SFTP Backend Guide](https://docs.remotestore.dev/stable/guides/backends/sftp/index.md) — usage patterns, configuration, and examples
- [SFTP Backend example](https://docs.remotestore.dev/stable/tutorial/examples/sftp-backend/index.md) — SFTP backend in action

# SQLBlobBackend

API reference for `SQLBlobBackend` — stores files as key-value rows in any SQLAlchemy-supported SQL database.

## SQLBlobBackend

```
SQLBlobBackend(
    url: str | None = None,
    *,
    engine: Engine | None = None,
    table_name: str = "remote_store_objects",
    create_table: bool = True,
    max_blob_size: int | None = None,
    reject_write_under_file_ancestor: bool = False,
)
```

SQL key-value blob store implementing the full Backend contract.

Uses a SQL table as key-value storage. Each row holds one "file" with its key, data, and metadata. SQLite receives WAL mode and PRAGMA tuning automatically.

Supports all capabilities except `LAZY_READ`.

Every mutating operation runs inside a single database transaction, so `write`, `write_atomic`, `move`, and `copy` are atomic — a failure rolls back with no partial row left behind (`ATOMIC_MOVE` is advertised).

Note

**Non-lazy reads and writes.** Both `read()` and `write()` materialize the full content in memory. `read()` loads the entire BLOB before returning a stream (no `LAZY_READ`). `write()` reads the full stream before issuing the SQL INSERT/UPDATE because BLOB columns require complete data in a single statement. For files larger than process memory, use a blob-storage backend (S3, Local, Azure) instead.

Parameters:

- **`reject_write_under_file_ancestor`** (`bool`, default: `False` ) – If True, write / write_atomic / open_atomic / move / copy issue one SELECT 1 per slash-aligned ancestor of the target path and raise InvalidPath on the first regular-file hit, matching the cross-backend contract that hierarchical filesystems enforce natively. Default False; paths without slashes short-circuit.

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` with SQL blob details.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – Plan with kind="sql-blob" and details containing
- `ResolutionPlan` – table_name.

### exists

```
exists(path: str) -> bool
```

Return `True` if a key or key-prefix exists at *path*; never `NotFound`.

Folders are virtual — *path* counts as a folder when any key begins with `path + "/"`. The root (`""`) always exists. Costs one or two `SELECT`s.

Raises:

- `InvalidPath` – If path is absolute or contains a .. segment.
- `BackendUnavailable` – If the database is unreachable.

### is_file

```
is_file(path: str) -> bool
```

Return `True` if an exact key exists at *path* (one `SELECT`).

Raises:

- `InvalidPath` – If path is absolute or contains a .. segment.
- `BackendUnavailable` – If the database is unreachable.

### is_folder

```
is_folder(path: str) -> bool
```

Return `True` if any key begins with `path + "/"` (a virtual folder).

The root is always a folder. Costs one `SELECT`.

Raises:

- `InvalidPath` – If path is absolute or contains a .. segment.
- `BackendUnavailable` – If the database is unreachable.

### read

```
read(path: str) -> BinaryIO
```

Return a binary stream over the stored BLOB for *path*.

Loads the entire BLOB into memory before returning the stream — the read does not stream (`LAZY_READ` is not advertised), so peak memory scales with the object size. For objects larger than process memory use a blob-storage backend (S3, Local, Azure).

Raises:

- `NotFound` – If no key exists at path.
- `InvalidPath` – If path is empty, absolute, or malformed.
- `BackendUnavailable` – If the database is unreachable.

### read_bytes

```
read_bytes(path: str) -> bytes
```

Return the full stored BLOB for *path* as bytes.

Like `read`, materialises the whole object in memory.

Raises:

- `NotFound` – If no key exists at path.
- `InvalidPath` – If path is empty, absolute, or malformed.
- `BackendUnavailable` – If the database is unreachable.

### write

```
write(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Store *content* at *path* in a single, atomic transaction.

The whole body is buffered in memory before the `INSERT`/`UPDATE` (BLOB columns need the complete value in one statement — no streaming write), and the row is written inside one transaction, so a failure rolls back with no partial row left behind.

Raises:

- `AlreadyExists` – If a key exists at path and overwrite is False.
- `InvalidPath` – If path is empty/malformed, or (with the reject_write_under_file_ancestor opt-in) an ancestor key exists as a file.
- `BackendUnavailable` – If the database operation fails.

### write_atomic

```
write_atomic(
    path: str,
    content: WritableContent,
    *,
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Store *content* at *path* atomically (delegates to `write`).

SQL writes are already transactional, so this is exactly `write`; the whole body is buffered first.

Raises:

- `AlreadyExists` – If a key exists at path and overwrite is False.
- `InvalidPath` – If path is empty/malformed, or (opt-in) an ancestor key exists as a file.
- `BackendUnavailable` – If the database operation fails.

### open_atomic

```
open_atomic(
    path: str, *, overwrite: bool = False
) -> Iterator[BinaryIO]
```

Yield an in-memory buffer committed to *path* atomically on clean exit.

Writes accumulate in a `BytesIO`; on exit the buffer is stored via `write` in one transaction. An exception before exit leaves *path* untouched.

Raises:

- `AlreadyExists` – If a key exists at path and overwrite is False.
- `InvalidPath` – If path is empty/malformed, or (opt-in) an ancestor key exists as a file.
- `BackendUnavailable` – If the database operation fails.

### delete

```
delete(path: str, *, missing_ok: bool = False) -> None
```

Delete the row at *path* in one transaction.

Raises:

- `NotFound` – If no key exists at path and missing_ok is False.
- `InvalidPath` – If path is empty, absolute, or malformed.
- `BackendUnavailable` – If the database operation fails.

### delete_folder

```
delete_folder(
    path: str,
    *,
    recursive: bool = False,
    missing_ok: bool = False,
) -> None
```

Delete every key under the virtual folder *path*.

Folders are key prefixes, not stored rows, so a folder "exists" only when it has children: a non-recursive call on an existing folder therefore always raises `DirectoryNotEmpty`. `recursive=True` deletes all keys under `path + "/"` in one atomic transaction.

Raises:

- `NotFound` – If no key exists under path and missing_ok is False.
- `DirectoryNotEmpty` – If recursive is False (an existing virtual folder is never empty).
- `InvalidPath` – If path is empty, absolute, or malformed.
- `BackendUnavailable` – If the database operation fails.

### list_files

```
list_files(
    path: str,
    *,
    recursive: bool = False,
    max_depth: int | None = None,
) -> Iterator[FileInfo]
```

Yield files under *path*.

One `SELECT` fetches every key under the prefix; folder structure is derived from `/` in the key suffix, and `recursive` / `max_depth` filter client-side. A missing prefix yields nothing.

Raises:

- `BackendUnavailable` – If the database operation fails, surfaced during iteration.

### list_folders

```
list_folders(path: str) -> Iterator[FolderEntry]
```

Yield immediate virtual subfolders of *path* as `FolderEntry` records.

One `SELECT` over keys under the prefix; folder names are the distinct first segments of the key suffixes. A missing prefix yields nothing.

Raises:

- `BackendUnavailable` – If the database operation fails, surfaced during iteration.

### iter_children

```
iter_children(
    path: str,
) -> Iterator[FileInfo | FolderEntry]
```

Yield the immediate files and virtual folders under *path* in one `SELECT`.

Overrides the base two-pass default: a single query over the prefix yields `FileInfo` for direct-child keys and `FolderEntry` for the distinct first suffix segments. A missing prefix yields nothing.

Raises:

- `BackendUnavailable` – If the database operation fails, surfaced during iteration.

### get_file_info

```
get_file_info(path: str) -> FileInfo
```

Return metadata for the file at *path* from one `SELECT`.

Raises:

- `NotFound` – If no key exists at path.
- `InvalidPath` – If path is empty, absolute, or malformed.
- `BackendUnavailable` – If the database is unreachable.

### get_folder_info

```
get_folder_info(path: str) -> FolderInfo
```

Return aggregate metadata for the virtual folder *path*.

File count, total size, and latest modification time come from one aggregate `SELECT` (`COUNT`/`SUM`/`MAX`) over keys under the prefix — no per-file round-trips.

Raises:

- `NotFound` – If no key exists under path.
- `InvalidPath` – If path is absolute or malformed.
- `BackendUnavailable` – If the database is unreachable.

### move

```
move(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Move (rename) the key *src* to *dst* in one atomic transaction.

Implemented as an `UPDATE` of the row's key — the BLOB is never transferred through Python. The whole operation (source check, optional destination replace, rename) runs in one transaction, so a failure rolls back cleanly. `src == dst` verifies the source exists and is otherwise a no-op.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists and overwrite is False.
- `InvalidPath` – If src or dst is malformed, or (opt-in) an ancestor of dst exists as a file.
- `BackendUnavailable` – If the database operation fails.

### copy

```
copy(
    src: str, dst: str, *, overwrite: bool = False
) -> None
```

Copy the key *src* to *dst* in one atomic transaction.

Implemented as a single `INSERT ... SELECT`, so the BLOB is duplicated entirely inside the database — no bytes pass through Python. `src == dst` verifies the source exists and is otherwise a no-op.

Raises:

- `NotFound` – If src does not exist.
- `AlreadyExists` – If dst exists and overwrite is False.
- `InvalidPath` – If src or dst is malformed, or (opt-in) an ancestor of dst exists as a file.
- `BackendUnavailable` – If the database operation fails.

### glob

```
glob(pattern: str) -> Iterator[FileInfo]
```

Yield files whose key matches the glob *pattern*.

Narrows SQL-side with a prefix `LIKE` where the pattern allows (on every dialect — SQLite's native `GLOB` is deliberately avoided because it mishandles `**`), then applies the full glob regex to each row. Costs one `SELECT`.

Raises:

- `BackendUnavailable` – If the database operation fails, surfaced during iteration.

## See also

- [SQL Blob Backend Guide](https://docs.remotestore.dev/stable/guides/backends/sql-blob/index.md) — usage patterns, configuration, and examples

# SQLQueryBackend

API reference for `SQLQueryBackend` --- read-only SQL query materializer that maps path keys to SQL queries and serializes results to Parquet, CSV, or Arrow IPC.

## SQLQueryBackend

```
SQLQueryBackend(
    url: str | None = None,
    *,
    engine: Engine | None = None,
    queries: dict[str, str] | None = None,
    strict: bool = True,
    serializer: ResultSerializer | None = None,
)
```

Read-only SQL query materializer implementing a subset of the Backend contract.

Maps path keys to SQL queries. On `read()`, executes the query and serializes the result set to the format implied by the key's file extension (Parquet, CSV, or Arrow IPC).

Capabilities: `READ`, `LIST`, `METADATA`, `GLOB`, `SEEKABLE_READ`.

### resolve

```
resolve(path: str) -> ResolutionPlan
```

Return a `ResolutionPlan` with SQL query details.

Parameters:

- **`path`** (`str`) – Backend-relative key.

Returns:

- `ResolutionPlan` – Plan with kind="sql-query" and details containing
- `ResolutionPlan` – source and format.

## Serialization

## ResultSerializer

Converts SQL result rows to bytes in a specific format.

### serialize

```
serialize(
    rows: Sequence[Any], columns: Sequence[str], format: str
) -> bytes
```

Serialize rows with given column names to the specified format.

Parameters:

- **`rows`** (`Sequence[Any]`) – Sequence of row tuples from SQL execution.
- **`columns`** (`Sequence[str]`) – Column name list.
- **`format`** (`str`) – Target format ("parquet", "csv", "arrow").

Returns:

- `bytes` – Serialized bytes.

## ArrowSerializer

Serializes SQL result sets via PyArrow.

Converts rows + columns to a `pyarrow.Table`, then writes to the requested format. Imports `pyarrow` lazily so that `SQLBlobBackend` remains importable without it.

### serialize

```
serialize(
    rows: Sequence[Any], columns: Sequence[str], format: str
) -> bytes
```

Serialize rows to Parquet, CSV, or Arrow IPC.

# Extensions

API reference for all extension modules. Extensions add optional capabilities to Store — install the relevant extra to enable each one. For usage guides, see [Extensions](https://docs.remotestore.dev/stable/guides/extensions/index.md).

| Module                                                                                           | Description                                                               |
| ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------- |
| [ext.arrow](https://docs.remotestore.dev/stable/reference/api/extensions/arrow/index.md)         | PyArrow `FileSystemHandler` adapter for Store                             |
| [ext.batch](https://docs.remotestore.dev/stable/reference/api/extensions/batch/index.md)         | Batch delete, copy, and exists operations                                 |
| [ext.cache](https://docs.remotestore.dev/stable/reference/api/extensions/cache/index.md)         | Store-level caching middleware with TTL                                   |
| [ext.dagster](https://docs.remotestore.dev/stable/reference/api/extensions/dagster/index.md)     | Dagster IO manager, config-driven Store resource, and compute log manager |
| [ext.glob](https://docs.remotestore.dev/stable/reference/api/extensions/glob/index.md)           | Portable glob pattern matching fallback                                   |
| [ext.integrity](https://docs.remotestore.dev/stable/reference/api/extensions/integrity/index.md) | Checksum computation and verification helpers                             |
| [ext.observe](https://docs.remotestore.dev/stable/reference/api/extensions/observe/index.md)     | Callback hooks for store operations                                       |
| [ext.otel](https://docs.remotestore.dev/stable/reference/api/extensions/otel/index.md)           | OpenTelemetry bridge for ext.observe                                      |
| [ext.parquet](https://docs.remotestore.dev/stable/reference/api/extensions/parquet/index.md)     | Managed Parquet datasets with manifests and completion markers            |
| [ext.partition](https://docs.remotestore.dev/stable/reference/api/extensions/partition/index.md) | Hive-style partition path helpers                                         |
| [ext.pydantic](https://docs.remotestore.dev/stable/reference/api/extensions/pydantic/index.md)   | Pydantic model to RegistryConfig adapter                                  |
| [ext.streams](https://docs.remotestore.dev/stable/reference/api/extensions/streams/index.md)     | Composable BinaryIO wrappers for progress and checksums                   |
| [ext.transfer](https://docs.remotestore.dev/stable/reference/api/extensions/transfer/index.md)   | Upload, download, and cross-store transfer                                |
| [ext.write](https://docs.remotestore.dev/stable/reference/api/extensions/write/index.md)         | Write helpers with guaranteed client-side content hashing                 |
| [ext.yaml](https://docs.remotestore.dev/stable/reference/api/extensions/yaml/index.md)           | YAML config loader (PyYAML / ruamel.yaml)                                 |

The async-native extensions live under [Async › Extensions](https://docs.remotestore.dev/stable/reference/api/aio/extensions/index.md):

| Module                                                                                           | Description                                                     |
| ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- |
| [aio.ext.write](https://docs.remotestore.dev/stable/reference/api/aio/extensions/write/index.md) | Async write helpers with guaranteed client-side content hashing |

## See also

- [Extensions guide](https://docs.remotestore.dev/stable/guides/extensions/index.md) — overview of all extensions with installation instructions
- [Async extensions](https://docs.remotestore.dev/stable/reference/api/aio/extensions/index.md) — native async extension surface
- [Choosing a Backend](https://docs.remotestore.dev/stable/guides/choosing-a-backend/index.md) — backend selection guide

# ext.arrow

## arrow

PyArrow FileSystem adapter — wraps any Store into a [pyarrow.fs.PyFileSystem](https://arrow.apache.org/docs/python/generated/pyarrow.fs.PyFileSystem.html).

Install with `pip install "remote-store[arrow]"`.

Example

```
from remote_store.ext.arrow import pyarrow_fs

fs = pyarrow_fs(store)
pq.write_table(table, "data.parquet", filesystem=fs)
```

### StoreFileSystemHandler

```
StoreFileSystemHandler(
    store: Store,
    materialization_threshold: int = 64 * 1024 * 1024,
    write_spill_threshold: int = 64 * 1024 * 1024,
)
```

Bases: `FileSystemHandler`

`pyarrow.fs.FileSystemHandler` backed by a `Store`.

Parameters:

- **`store`** (`Store`) – The Store to expose as a PyArrow filesystem.
- **`materialization_threshold`** (`int`, default: `64 * 1024 * 1024` ) – Max file size (bytes) for Tier 2 full-file materialization in open_input_file. Default 64 MB.
- **`write_spill_threshold`** (`int`, default: `64 * 1024 * 1024` ) – Max in-memory buffer size (bytes) for \_StoreSink before spilling to disk. Default 64 MB.

**Thread safety:** The handler itself holds no shared mutable state. PyArrow's C++ layer may call handler methods from background threads (with the GIL acquired). Thread safety therefore depends on the backend: `MemoryBackend` uses a lock (safe), `LocalBackend` relies on OS file semantics (safe), cloud backends use thread-safe HTTP clients (safe). If using a custom backend, ensure its methods are safe under concurrent calls.

### pyarrow_fs

```
pyarrow_fs(
    store: Store,
    *,
    materialization_threshold: int = 64 * 1024 * 1024,
    write_spill_threshold: int = 64 * 1024 * 1024,
) -> PyFileSystem
```

Create a `pyarrow.fs.PyFileSystem` backed by *store*.

Parameters:

- **`store`** (`Store`) – The Store to expose.
- **`materialization_threshold`** (`int`, default: `64 * 1024 * 1024` ) – See StoreFileSystemHandler.
- **`write_spill_threshold`** (`int`, default: `64 * 1024 * 1024` ) – See StoreFileSystemHandler.

## See also

- [PyArrow Adapter](https://docs.remotestore.dev/stable/guides/pyarrow-adapter/index.md) — guide to using Store as a PyArrow filesystem
- [PyArrow Adapter example](https://docs.remotestore.dev/stable/tutorial/examples/pyarrow-adapter/index.md) — PyArrow adapter in action

# ext.batch

## batch

Batch operations — convenience wrappers for bulk delete, copy, and exists.

All functions call Store methods one-by-one by default (sequential). Pass `concurrent=True` to use a `ThreadPoolExecutor` for parallel I/O — cloud backends benefit significantly from this.

Example

```
from remote_store.ext.batch import batch_delete, batch_copy, batch_exists

result = batch_delete(store, ["a.txt", "b.txt"], missing_ok=True)
result = batch_copy(store, [("a.txt", "copy.txt")], overwrite=True)
exists_map = batch_exists(store, ["a.txt", "missing.txt"])

# Parallel execution (cloud backends):
result = batch_delete(store, keys, concurrent=True, max_workers=8)
```

### BatchResult

```
BatchResult(
    succeeded: tuple[str, ...],
    failed: dict[str, RemoteStoreError],
)
```

Outcome of a batch operation.

Attributes:

- **`succeeded`** (`tuple[str, ...]`) – Paths that completed without error.
- **`failed`** (`dict[str, RemoteStoreError]`) – Mapping from path to the error that occurred.

#### all_succeeded

```
all_succeeded: bool
```

`True` when every path succeeded.

#### total

```
total: int
```

Total number of paths processed (succeeded + failed).

### batch_delete

```
batch_delete(
    store: Store,
    paths: Iterable[str],
    *,
    missing_ok: bool = False,
    stop_on_error: bool = False,
    concurrent: bool = False,
    max_workers: int | None = None,
) -> BatchResult
```

Delete multiple files, collecting errors.

Parameters:

- **`store`** (`Store`) – The Store to delete from.
- **`paths`** (`Iterable[str]`) – File paths to delete.
- **`missing_ok`** (`bool`, default: `False` ) – Forwarded to each store.delete() call.
- **`stop_on_error`** (`bool`, default: `False` ) – Stop on first RemoteStoreError (sequential only).
- **`concurrent`** (`bool`, default: `False` ) – Use a thread pool for parallel execution.
- **`max_workers`** (`int | None`, default: `None` ) – Max threads (forwarded to ThreadPoolExecutor).

Returns:

- `BatchResult` – A BatchResult with succeeded/failed paths.

Raises:

- `ValueError` – If both concurrent and stop_on_error are True.

### batch_copy

```
batch_copy(
    store: Store,
    pairs: Iterable[tuple[str, str]],
    *,
    overwrite: bool = False,
    stop_on_error: bool = False,
    concurrent: bool = False,
    max_workers: int | None = None,
) -> BatchResult
```

Copy multiple files, collecting errors.

Parameters:

- **`store`** (`Store`) – The Store to copy within.
- **`pairs`** (`Iterable[tuple[str, str]]`) – (src, dst) tuples.
- **`overwrite`** (`bool`, default: `False` ) – Forwarded to each store.copy() call.
- **`stop_on_error`** (`bool`, default: `False` ) – Stop on first RemoteStoreError (sequential only).
- **`concurrent`** (`bool`, default: `False` ) – Use a thread pool for parallel execution.
- **`max_workers`** (`int | None`, default: `None` ) – Max threads (forwarded to ThreadPoolExecutor).

Returns:

- `BatchResult` – A BatchResult with succeeded/failed source paths.

Raises:

- `ValueError` – If both concurrent and stop_on_error are True.

### batch_exists

```
batch_exists(
    store: Store,
    paths: Iterable[str],
    *,
    concurrent: bool = False,
    max_workers: int | None = None,
) -> dict[str, bool]
```

Check existence of multiple paths.

Unlike `batch_delete()` and `batch_copy()`, this function does **not** catch errors — any exception from `store.exists()` propagates immediately.

Parameters:

- **`store`** (`Store`) – The Store to query.
- **`paths`** (`Iterable[str]`) – Paths to check.
- **`concurrent`** (`bool`, default: `False` ) – Use a thread pool for parallel execution.
- **`max_workers`** (`int | None`, default: `None` ) – Max threads (forwarded to ThreadPoolExecutor).

Returns:

- `dict[str, bool]` – Dict mapping each path to True/False.

## See also

- [Batch Operations](https://docs.remotestore.dev/stable/guides/batch-operations/index.md) — guide to bulk delete, copy, and exists
- [Batch Operations example](https://docs.remotestore.dev/stable/tutorial/examples/batch-operations/index.md) — batch operations in action

# ext.cache

## cache

Store-level caching middleware with TTL-based expiration.

Wraps a Store in a proxy that caches read-only operations (existence checks, metadata, listings, content) and automatically invalidates on mutations.

Example

```
from remote_store.ext.cache import cache

cached = cache(store, ttl=300)
data = cached.read_bytes("key.csv")   # backend call
data = cached.read_bytes("key.csv")   # cache hit
cached.write("key.csv", b"new", overwrite=True)  # invalidates
data = cached.read_bytes("key.csv")   # backend call again
```

### CacheStats

```
CacheStats(hits: int, misses: int, size: int)
```

Snapshot of cache hit/miss statistics.

### CacheBackend

Bases: `Protocol`

Protocol for pluggable cache backends.

Implement this protocol to provide a custom cache backend to `cache(store, cache_backend=my_backend)`. The default implementation is `MemoryCache`.

#### get

```
get(key: tuple[str, ...]) -> Any
```

Return the cached value, or raise `KeyError` on a cache miss.

#### set

```
set(key: tuple[str, ...], value: Any, ttl: float) -> None
```

Store *value* under *key* with a time-to-live in seconds.

#### delete

```
delete(key: tuple[str, ...]) -> None
```

Remove *key* from the cache (no-op if absent).

#### clear

```
clear() -> None
```

Remove all entries from the cache.

#### clear_prefix

```
clear_prefix(prefix: str) -> None
```

Remove all entries whose first key component matches *prefix*.

#### size

```
size() -> int
```

Return the number of entries currently in the cache.

### MemoryCache

```
MemoryCache(*, max_entries: int | None = None)
```

Thread-safe in-memory cache backend with lazy TTL eviction.

Entries are stored as `{key: (value, expiry)}` where *expiry* is a `time.monotonic()` deadline.

When *max_entries* is set, the cache evicts the least-recently-used entry when the limit is exceeded (LRU eviction). Without a bound, metadata entries (`exists`, `is_file`, listings) for many distinct paths can grow without limit during the TTL window.

#### get

```
get(key: tuple[str, ...]) -> Any
```

Return cached value or raise `KeyError` on miss/expiry.

#### set

```
set(key: tuple[str, ...], value: Any, ttl: float) -> None
```

Store *value* with a TTL in seconds.

#### delete

```
delete(key: tuple[str, ...]) -> None
```

Remove a single entry (no-op if absent).

#### clear

```
clear() -> None
```

Remove all entries.

#### clear_prefix

```
clear_prefix(prefix: str) -> None
```

Remove all entries whose first key element equals *prefix*.

#### clear_prefixes

```
clear_prefixes(prefixes: frozenset[str]) -> None
```

Remove all entries whose first key element is in *prefixes*.

Single dict rebuild instead of one per prefix — O(n) vs O(k\*n).

#### size

```
size() -> int
```

Return count of non-expired entries.

### CachedStore

```
CachedStore(
    inner: Store,
    *,
    ttl: float,
    max_content_size: int | None,
    max_listing_size: int | None,
    max_entries: int | None,
    cache_backend: CacheBackend | None,
    _prefix: str = "",
)
```

Bases: `ProxyStore`

Proxy Store that caches read operations with TTL-based expiration.

All `Store` methods are delegated to the inner store. Read-only methods use the cache; mutating methods invalidate affected entries. Only methods with additional behavior (`invalidate`, `clear_cache`, `ping`, `close`, `child`) are documented individually below.

Do not construct directly -- use `cache()`.

#### stats

```
stats: CacheStats
```

Snapshot of cache hit/miss statistics.

#### invalidate

```
invalidate(path: str) -> None
```

Remove all cached entries for *path* and its ancestor directories.

#### clear_cache

```
clear_cache() -> None
```

Remove all cached entries.

### cache

```
cache(
    store: Store,
    *,
    ttl: float = 300.0,
    max_content_size: int | None = None,
    max_listing_size: int | None = None,
    max_entries: int | None = None,
    cache_backend: CacheBackend | None = None,
) -> CachedStore
```

Wrap a Store with read-through caching.

Parameters:

- **`store`** (`Store`) – The Store to wrap.
- **`ttl`** (`float`, default: `300.0` ) – Time-to-live in seconds for cache entries (default 300).
- **`max_content_size`** (`int | None`, default: `None` ) – Maximum byte length for read_bytes caching. Files larger than this are returned without caching. None means unlimited.
- **`max_listing_size`** (`int | None`, default: `None` ) – Maximum number of items in a listing result (iter_children, list_files, list_folders, glob) for caching. Listings with more items than this are returned without caching. None means unlimited.
- **`max_entries`** (`int | None`, default: `None` ) – Maximum number of cache entries. When exceeded, the least-recently-used entry is evicted. None means no limit. Ignored when cache_backend is provided.
- **`cache_backend`** (`CacheBackend | None`, default: `None` ) – Optional custom cache. When None, a MemoryCache is created.

Returns:

- `CachedStore` – A CachedStore proxy.

Raises:

- `ValueError` – If ttl, max_content_size, or max_listing_size is not positive when set.

## See also

- [Cache](https://docs.remotestore.dev/stable/guides/cache/index.md) — guide to Store-level caching with TTL
- [Caching example](https://docs.remotestore.dev/stable/tutorial/examples/caching/index.md) — caching middleware in action

# ext.dagster

## dagster

Dagster IO Manager adapter — wraps any Store as a Dagster [IOManager](https://docs.dagster.io/_apidocs/io-managers#dagster.IOManager).

Lets teams already using remote-store reuse their Store configuration (credentials, retry policy, caching, observability) inside Dagster pipelines without duplicating config into dagster-aws / dagster-azure.

Install with `pip install "remote-store[dagster]"`.

Example

```
from remote_store.ext.dagster import dagster_io_manager

io_mgr = dagster_io_manager(store, serializer="pickle")
```

### Serializer

Bases: `Protocol`

Protocol for pluggable serializers.

Implement this to provide a custom serializer to `dagster_io_manager(store, serializer=my_serializer)`.

#### serialize

```
serialize(obj: Any) -> bytes
```

Convert a Python object to bytes.

#### deserialize

```
deserialize(data: bytes) -> Any
```

Convert bytes back to a Python object.

### PickleSerializer

Pickle-based serializer. Universal; opaque format.

#### serialize

```
serialize(obj: Any) -> bytes
```

Serialize using pickle.

#### deserialize

```
deserialize(data: bytes) -> Any
```

Deserialize using pickle.

### JsonSerializer

JSON serializer. JSON-serializable objects only.

#### serialize

```
serialize(obj: Any) -> bytes
```

Serialize to JSON bytes.

#### deserialize

```
deserialize(data: bytes) -> Any
```

Deserialize from JSON bytes.

### ParquetSerializer

```
ParquetSerializer()
```

Parquet serializer via PyArrow. DataFrames and Arrow Tables.

#### serialize

```
serialize(obj: Any) -> bytes
```

Serialize a DataFrame to Parquet bytes.

#### deserialize

```
deserialize(data: bytes) -> Table
```

Deserialize Parquet bytes to a PyArrow Table.

### DagsterStoreResource

Bases: `ConfigurableResource`

Dagster resource that constructs a Store from config fields.

Unlike stateless utility extensions, this class owns Store lifecycle because it is a Dagster Resource with `setup_for_execution` / `teardown_after_execution` hooks.

Attributes:

- **`backend_type`** (`str`) – Backend type string (e.g. "local", "s3", "memory"). Must be registered in the backend factory registry.
- **`backend_options`** (`dict[str, Any]`) – Keyword arguments passed to the backend constructor.
- **`root_path`** (`str`) – Optional root path for the Store.

#### setup_for_execution

```
setup_for_execution(context: InitResourceContext) -> None
```

Instantiate and cache the Store (called by Dagster before execution).

Raises:

- `ValueError` – If backend_type is not registered, or if the backend constructor rejects the supplied backend_options.

#### teardown_after_execution

```
teardown_after_execution(
    context: InitResourceContext,
) -> None
```

Close the Store and release resources (called by Dagster after execution).

#### get_store

```
get_store() -> Store
```

Return the underlying Store instance.

Returns:

- `Store` – The Store constructed during setup_for_execution.

Raises:

- `RuntimeError` – If called before setup_for_execution has run.

### RemoteStoreIOManager

Bases: `ConfigurableIOManagerFactory`

IO manager factory that constructs a Store from config fields.

Embeds backend configuration directly so the IO manager owns the full Store lifecycle (setup and teardown). For direct Store access in assets, use `DagsterStoreResource` as a separate resource.

Attributes:

- **`backend_type`** (`str`) – Backend type string (e.g. "local", "s3", "memory").
- **`backend_options`** (`dict[str, Any]`) – Keyword arguments passed to the backend constructor.
- **`root_path`** (`str`) – Optional root path for the Store.
- **`serializer`** (`str`) – Serializer name. Use "parquet-dataset" for multi-file Parquet dataset output via ParquetDatasetStore.

#### setup_for_execution

```
setup_for_execution(context: InitResourceContext) -> None
```

Build and cache the Store before execution.

#### teardown_after_execution

```
teardown_after_execution(
    context: InitResourceContext,
) -> None
```

Close the Store and release resources.

#### create_io_manager

```
create_io_manager(context: Any) -> IOManager
```

Construct the IOManager for the given execution context.

Returns:

- `IOManager` – A \_DatasetIOManagerImpl when serializer="parquet-dataset",
- `IOManager` – otherwise a \_RemoteStoreIOManagerImpl with the resolved serializer.

Raises:

- `RuntimeError` – If called before setup_for_execution.
- `ValueError` – If serializer is an unrecognized string.

### RemoteStoreComputeLogManager

```
RemoteStoreComputeLogManager(
    backend_type: str,
    backend_options: dict[str, Any] | None = None,
    root_path: str = "",
    local_dir: str | None = None,
    prefix: str = "dagster",
    skip_empty_files: bool = False,
    upload_interval: int | None = None,
    inst_data: ConfigurableClassData | None = None,
)
```

Bases: `TruncatingCloudStorageComputeLogManager`, `ConfigurableClass`

Captures op/step `stdout` / `stderr` to any remote-store backend.

A Dagster `ComputeLogManager` wired into `dagster.yaml` as an instance component. Logs are captured to a local staging directory at the file-descriptor level, then uploaded to a `Store` the manager builds itself from `backend_type` + `backend_options`. When `upload_interval` is set, partial uploads also run periodically while a step executes so the Dagster UI can tail them. Subclasses Dagster's `TruncatingCloudStorageComputeLogManager` (capture-then-upload machinery, 50 MB upload truncation) and `ConfigurableClass` (`dagster.yaml` plumbing).

Configure it in `dagster.yaml`:

```
compute_logs:
  module: remote_store.ext.dagster
  class: RemoteStoreComputeLogManager
  config:
    backend_type: s3
    backend_options:
      bucket: my-logs-bucket
    root_path: dagster/compute-logs
    upload_interval: 30
```

Attributes:

- **`backend_type`** – Registered backend type ("local", "s3", "sftp", "azure", "memory", ...).
- **`backend_options`** – Keyword arguments for the backend constructor.
- **`root_path`** – Store root prefix applied to every log object.
- **`local_dir`** – Local staging directory for capture. Defaults to the system temp directory.
- **`prefix`** – Path prefix within the Store. Defaults to "dagster".
- **`skip_empty_files`** – Skip uploading zero-byte log files.
- **`upload_interval`** (`int | None`) – Seconds between partial uploads while a step runs; None (default) disables live tailing.

Build the Store, validate its capabilities, and wire the local manager.

Raises:

- `ValueError` – If backend_type is not registered, if the backend rejects backend_options, or if the backend is missing a capability the manager requires (READ, WRITE, DELETE, METADATA, LIST).

#### inst_data

```
inst_data: ConfigurableClassData | None
```

The `ConfigurableClassData` this manager was rehydrated from, if any.

#### local_manager

```
local_manager: LocalComputeLogManager
```

The `LocalComputeLogManager` that stages captures before upload.

#### upload_interval

```
upload_interval: int | None
```

Seconds between partial uploads, or `None` when live tailing is off.

#### config_type

```
config_type() -> dict[str, Any]
```

The `dagster.yaml` config schema for this manager.

#### from_config_value

```
from_config_value(
    inst_data: ConfigurableClassData | None,
    config_value: Mapping[str, Any],
) -> RemoteStoreComputeLogManager
```

Construct a manager from a validated `dagster.yaml` config value.

#### download_from_cloud_storage

```
download_from_cloud_storage(
    log_key: Sequence[str],
    io_type: ComputeIOType,
    partial: bool = False,
) -> None
```

Stream a log object from the Store into the local staging file.

#### cloud_storage_has_logs

```
cloud_storage_has_logs(
    log_key: Sequence[str],
    io_type: ComputeIOType,
    partial: bool = False,
) -> bool
```

Return whether the Store holds a log object for this key.

#### display_path_for_type

```
display_path_for_type(
    log_key: Sequence[str], io_type: ComputeIOType
) -> str | None
```

A human-readable Store location for the Dagster UI, once capture is done.

#### download_url_for_type

```
download_url_for_type(
    log_key: Sequence[str], io_type: ComputeIOType
) -> str | None
```

No signed-URL primitive in v1 — the webserver streams logs itself.

#### delete_logs

```
delete_logs(
    log_key: Sequence[str] | None = None,
    prefix: Sequence[str] | None = None,
) -> None
```

Delete captured logs by `log_key` or by `prefix`, local and remote.

Raises:

- `CheckError` – If neither log_key nor prefix is given.

#### get_log_keys_for_log_key_prefix

```
get_log_keys_for_log_key_prefix(
    log_key_prefix: Sequence[str], io_type: ComputeIOType
) -> Sequence[Sequence[str]]
```

Enumerate the stored log keys under a log-key prefix.

#### on_subscribe

```
on_subscribe(subscription: CapturedLogSubscription) -> None
```

Register a UI live-tail subscription with the polling manager.

#### on_unsubscribe

```
on_unsubscribe(
    subscription: CapturedLogSubscription,
) -> None
```

Deregister a UI live-tail subscription from the polling manager.

#### dispose

```
dispose() -> None
```

Dispose the subscription and local managers and close the Store.

### dagster_io_manager

```
dagster_io_manager(
    store: Store, *, serializer: str | Serializer = "pickle"
) -> IOManager
```

Wrap a Store as a Dagster IOManager.

Parameters:

- **`store`** (`Store`) – An existing Store instance. The caller owns its lifecycle — the IO manager does not close the Store.
- **`serializer`** (`str | Serializer`, default: `'pickle'` ) – "pickle" (default), "json", "parquet", or a custom object satisfying the Serializer protocol.

Returns:

- `IOManager` – A Dagster IOManager backed by the given Store.

Raises:

- `ValueError` – If serializer is an unrecognized string.

### dagster_dataset_io_manager

```
dagster_dataset_io_manager(store: Store) -> IOManager
```

Wrap a Store as a Dagster IOManager using ParquetDatasetStore.

Unlike `dagster_io_manager` which serializes objects to single files, this manager writes Parquet datasets (multi-file with manifest) via `ParquetDatasetStore`.

Parameters:

- **`store`** (`Store`) – An existing Store instance. The caller owns its lifecycle.

Returns:

- `IOManager` – A Dagster IOManager backed by ParquetDatasetStore.

## See also

- [Dagster](https://docs.remotestore.dev/stable/guides/dagster/index.md) — guide to the Dagster integration (IO manager and compute log manager)
- [Dagster IO Manager example](https://docs.remotestore.dev/stable/tutorial/examples/dagster-io-manager/index.md) — basic Dagster usage (v1)
- [Dagster v2 resource example](https://docs.remotestore.dev/stable/tutorial/examples/dagster-v2-resource/index.md) — config-driven Store with RemoteStoreIOManager
- [Dagster compute log example](https://docs.remotestore.dev/stable/tutorial/examples/dagster-compute-log-manager/index.md) — capturing op stdout/stderr with RemoteStoreComputeLogManager
- [Medallion Dagster example](https://docs.remotestore.dev/stable/tutorial/examples/medallion-dagster/index.md) — multi-layer data pipeline showcase

# ext.glob

## glob

Glob — portable pattern matching for file listing.

Delegates to native backend glob when `Capability.GLOB` is available, otherwise falls back to `Store.list_files()` + client-side filtering.

For simple name-based filtering, prefer `Store.list_files(pattern=...)`. Use `glob_files` when you need full recursive glob patterns (`**`, wildcards in directory segments) and want portable behavior across all backends.

Example

```
from remote_store.ext.glob import glob_files

# Works with any backend — uses native glob or list+filter fallback
for info in glob_files(store, "data/**/*.csv"):
    print(info.path, info.size)
```

### glob_files

```
glob_files(
    store: Store, pattern: str
) -> Iterator[FileInfo]
```

Match files against a glob pattern.

When the backend supports `Capability.GLOB`, delegates to `store.glob()` for native pattern matching. Otherwise extracts the longest non-wildcard prefix, lists files under that prefix, and filters client-side.

Parameters:

- **`store`** (`Store`) – The Store to search.
- **`pattern`** (`str`) – Glob pattern relative to the store root (e.g., "data/\*.csv", "\*\*/\*.txt").

Returns:

- `Iterator[FileInfo]` – Iterator of matching FileInfo objects.

## See also

- [Glob Pattern Matching](https://docs.remotestore.dev/stable/guides/glob-pattern-matching/index.md) — guide to portable glob patterns
- [Glob Pattern Matching example](https://docs.remotestore.dev/stable/tutorial/examples/glob-pattern-matching/index.md) — glob usage in action

# ext.integrity

Pure functions for computing and verifying file checksums over Store's public API. These compose `store.read()` with `ChecksumReader` internally — no manual stream lifecycle management needed.

See the [integrity spec](https://docs.remotestore.dev/stable/explanation/design/specs/034-ext-integrity/index.md) for invariants.

## integrity

Checksum verification helpers over Store's public API.

Pure functions for computing and verifying file integrity. These compose `store.read()` with ChecksumReader internally — users don't need to manage stream lifecycle.

Example

```
from remote_store.ext.integrity import checksum, verify

algorithm, hex_digest = checksum(store, "data/file.bin")
print(algorithm, hex_digest)

ok = verify(store, "data/file.bin", expected="a3f2b8...", algorithm="sha256")
```

### checksum

```
checksum(
    store: Store, path: str, algorithm: str = "sha256"
) -> tuple[str, str]
```

Compute the checksum of a file in the store.

Reads the file in chunks (never fully materialized in memory) and returns a `(algorithm, hex_digest)` tuple.

Parameters:

- **`store`** (`Store`) – The Store to read from.
- **`path`** (`str`) – Store-relative file path.
- **`algorithm`** (`str`, default: `'sha256'` ) – Hash algorithm name (default "sha256").

Returns:

- `tuple[str, str]` – Tuple of (algorithm, hex_digest) with lowercase values.

Raises:

- `NotFound` – If the file does not exist.
- `ValueError` – If the algorithm is not supported by hashlib.

### content_digest

```
content_digest(
    store: Store, path: str, algorithm: str = "sha256"
) -> ContentDigest
```

Compute the content digest of a file in the store.

Like checksum, but returns a ContentDigest instead of a raw tuple.

Parameters:

- **`store`** (`Store`) – The Store to read from.
- **`path`** (`str`) – Store-relative file path.
- **`algorithm`** (`str`, default: `'sha256'` ) – Hash algorithm name (default "sha256").

Returns:

- `ContentDigest` – A ContentDigest with normalized algorithm and hex value.

Raises:

- `NotFound` – If the file does not exist.
- `ValueError` – If the algorithm is not supported by hashlib.

### verify

```
verify(
    store: Store,
    path: str,
    expected: str,
    algorithm: str = "sha256",
) -> bool
```

Verify a file's checksum against an expected hex value.

Parameters:

- **`store`** (`Store`) – The Store to read from.
- **`path`** (`str`) – Store-relative file path.
- **`expected`** (`str`) – Expected hex digest (case-insensitive).
- **`algorithm`** (`str`, default: `'sha256'` ) – Hash algorithm name (default "sha256").

Returns:

- `bool` – True if the computed digest matches expected.

Raises:

- `NotFound` – If the file does not exist.
- `ValueError` – If the algorithm is not supported by hashlib.

### verify_hex

```
verify_hex(
    store: Store,
    path: str,
    algorithm: str,
    expected_hex: str,
) -> bool
```

Verify a file's checksum given an algorithm and expected hex value.

Parameters:

- **`store`** (`Store`) – The Store to read from.
- **`path`** (`str`) – Store-relative file path.
- **`algorithm`** (`str`) – Hash algorithm name.
- **`expected_hex`** (`str`) – Expected hex digest (case-insensitive).

Returns:

- `bool` – True if the computed digest matches expected_hex.

Raises:

- `NotFound` – If the file does not exist.
- `ValueError` – If the algorithm is not supported by hashlib.

## See also

- [ext.streams](https://docs.remotestore.dev/stable/reference/api/extensions/streams/index.md) — composable BinaryIO wrappers used internally by integrity helpers

# ext.observe

## observe

Observability hooks -- callback-based instrumentation for Store operations.

Wraps a Store in a proxy that fires user-defined callbacks before and after each operation, enabling logging, metrics, auditing, and tracing without modifying business code.

Example

```
from remote_store.ext.observe import observe

def on_write(event):
    print(f"Wrote {event.path} in {event.duration_ms:.1f}ms")

observed = observe(store, on_write=on_write)
observed.write("key.txt", b"hello")
```

### StoreEvent

```
StoreEvent(
    operation: str,
    path: str,
    backend: str,
    started_at: float,
    duration_ms: float,
    error: Exception | None,
    metadata: dict[str, Any],
    correlation_id: str | None,
)
```

Immutable record of a single Store operation.

### ObservedStore

```
ObservedStore(
    inner: Store,
    *,
    hooks: dict[str, Any],
    around: Any | None,
)
```

Bases: `ProxyStore`

Proxy Store that fires observation hooks on every public method.

All `Store` methods are delegated to the inner store. Only methods with additional behavior (`ping`, `close`, `child`) are documented individually below; the remaining overrides add hook dispatch and are otherwise transparent.

Do not construct directly -- use `observe()`.

#### ping

```
ping() -> None
```

Delegate ping to inner store.

#### close

```
close() -> None
```

Delegate close to inner store.

#### child

```
child(subpath: str) -> Store
```

Return an observed child store.

#### resolve

```
resolve(key: str) -> ResolutionPlan
```

Resolve a key to a `ResolutionPlan`, emitting observation events.

Event metadata is intentionally empty to prevent sensitive `details` from flowing through observation callbacks.

### BufferedObserver

```
BufferedObserver(
    handler: Callable[[list[StoreEvent]], None],
    *,
    max_queue: int = 1000,
    flush_interval: float = 5.0,
)
```

Collects events and flushes them in batches to a handler.

Parameters:

- **`handler`** (`Callable[[list[StoreEvent]], None]`) – Called with a list of events on each flush.
- **`max_queue`** (`int`, default: `1000` ) – Maximum queue size. Events are dropped when full.
- **`flush_interval`** (`float`, default: `5.0` ) – Seconds between automatic flushes.

#### on_event

```
on_event(event: StoreEvent) -> None
```

Enqueue an event. Drops and warns if queue is full.

#### flush

```
flush() -> None
```

Drain the queue and call the handler with collected events.

#### close

```
close() -> None
```

Stop the background thread and perform a final flush.

### set_correlation_id

```
set_correlation_id(cid: str | None) -> Token[str | None]
```

Set the correlation ID for the current context.

Returns a token that can be used to reset the value:

```
token = set_correlation_id("req-123")
# ... operations here will have correlation_id="req-123" ...
_correlation_id.reset(token)
```

Parameters:

- **`cid`** (`str | None`) – Correlation ID string, or None to clear.

Returns:

- `Token[str | None]` – A Token for resetting the value.

### observe

```
observe(
    store: Store,
    *,
    on_read: Callable[[StoreEvent], None] | None = None,
    on_write: Callable[[StoreEvent], None] | None = None,
    on_delete: Callable[[StoreEvent], None] | None = None,
    on_copy: Callable[[StoreEvent], None] | None = None,
    on_move: Callable[[StoreEvent], None] | None = None,
    on_list: Callable[[StoreEvent], None] | None = None,
    on_ping: Callable[[StoreEvent], None] | None = None,
    on_error: Callable[[StoreEvent], None] | None = None,
    on_any: Callable[[StoreEvent], None] | None = None,
    around: Callable[
        [str, str, str], AbstractContextManager[None]
    ]
    | None = None,
) -> ObservedStore
```

Wrap a Store with observation hooks.

Parameters:

- **`store`** (`Store`) – The Store to observe.
- **`on_read`** (`Callable[[StoreEvent], None] | None`, default: `None` ) – Fires after read/read_bytes/read_text.
- **`on_write`** (`Callable[[StoreEvent], None] | None`, default: `None` ) – Fires after write/write_text/write_atomic/open_atomic.
- **`on_delete`** (`Callable[[StoreEvent], None] | None`, default: `None` ) – Fires after delete/delete_folder.
- **`on_copy`** (`Callable[[StoreEvent], None] | None`, default: `None` ) – Fires after copy.
- **`on_move`** (`Callable[[StoreEvent], None] | None`, default: `None` ) – Fires after move.
- **`on_list`** (`Callable[[StoreEvent], None] | None`, default: `None` ) – Fires after list_files/list_folders/iter_children/glob/ get_file_info/get_folder_info/exists/is_file/is_folder/ resolve.
- **`on_ping`** (`Callable[[StoreEvent], None] | None`, default: `None` ) – Fires after ping.
- **`on_error`** (`Callable[[StoreEvent], None] | None`, default: `None` ) – Fires on any operation that raises an exception.
- **`on_any`** (`Callable[[StoreEvent], None] | None`, default: `None` ) – Fires after every operation (catch-all).
- **`around`** (`Callable[[str, str, str], AbstractContextManager[None]] | None`, default: `None` ) – Context-manager factory (op, path, backend) -> CM wrapping the entire operation.

Returns:

- `ObservedStore` – An ObservedStore proxy.

## See also

- [Observe](https://docs.remotestore.dev/stable/guides/observe/index.md) — guide to callback hooks for store operations
- [Observe Hooks example](https://docs.remotestore.dev/stable/tutorial/examples/observe-hooks/index.md) — observe hooks in action

# ext.otel

## otel

[OpenTelemetry](https://opentelemetry.io/) bridge -- pre-built hooks emitting OTel spans and metrics.

Provides ready-made `around` and `on_any` hooks for `observe()` that emit OpenTelemetry traces and metrics. Depends only on `opentelemetry-api` (not the SDK); if no SDK is configured at runtime, all OTel calls become zero-cost no-ops.

Example

```
from remote_store import observe
from remote_store.ext.otel import otel_hooks

store = observe(store, **otel_hooks())
```

Or as a one-liner:

```
from remote_store.ext.otel import otel_observe
observed = otel_observe(store)
```

Requires: `pip install "remote-store[otel]"`

### otel_hooks

```
otel_hooks(
    *,
    tracer_name: str = "remote_store",
    meter_name: str = "remote_store",
    tracer: Tracer | None = None,
    meter: Meter | None = None,
) -> dict[str, Any]
```

Return hook kwargs for `observe()`.

The returned dict contains an `around` context-manager hook (tracing) and an `on_any` callback (metrics). Unpack it into `observe()`:

```
observed = observe(store, **otel_hooks())
```

Parameters:

- **`tracer_name`** (`str`, default: `'remote_store'` ) – OTel tracer name (default "remote_store"). Ignored when tracer is provided.
- **`meter_name`** (`str`, default: `'remote_store'` ) – OTel meter name (default "remote_store"). Ignored when meter is provided.
- **`tracer`** (`Tracer | None`, default: `None` ) – Explicit tracer instance. When None (default), obtained from the global TracerProvider via tracer_name.
- **`meter`** (`Meter | None`, default: `None` ) – Explicit meter instance. When None (default), obtained from the global MeterProvider via meter_name.

Returns:

- `dict[str, Any]` – A dict with around and on_any keys.

### otel_observe

```
otel_observe(
    store: Store,
    *,
    tracer_name: str = "remote_store",
    meter_name: str = "remote_store",
    tracer: Tracer | None = None,
    meter: Meter | None = None,
) -> ObservedStore
```

Convenience: wrap a Store with OTel tracing + metrics in one call.

Equivalent to `observe(store, **otel_hooks(...))`.

Parameters:

- **`store`** (`Store`) – The Store to observe.
- **`tracer_name`** (`str`, default: `'remote_store'` ) – OTel tracer name (default "remote_store"). Ignored when tracer is provided.
- **`meter_name`** (`str`, default: `'remote_store'` ) – OTel meter name (default "remote_store"). Ignored when meter is provided.
- **`tracer`** (`Tracer | None`, default: `None` ) – Explicit tracer instance (see otel_hooks()).
- **`meter`** (`Meter | None`, default: `None` ) – Explicit meter instance (see otel_hooks()).

Returns:

- `ObservedStore` – An ObservedStore with OTel instrumentation.

## See also

- [Observe](https://docs.remotestore.dev/stable/guides/observe/index.md) — guide to callback hooks and the OpenTelemetry bridge
- [OTel Tracing example](https://docs.remotestore.dev/stable/tutorial/examples/otel-tracing/index.md) — OpenTelemetry tracing in action

# ext.parquet

## parquet

Parquet dataset read/write with manifest metadata and completion markers.

Install with `pip install "remote-store[arrow]"`.

Example

```
from remote_store.ext.parquet import ParquetDatasetStore

pds = ParquetDatasetStore(store)
manifest = pds.write_dataset(table, "silver/orders")
table = pds.read_dataset("silver/orders")
```

### DatasetIncomplete

```
DatasetIncomplete(
    message: str = "",
    *,
    path: str | None = None,
    backend: str | None = None,
)
```

Bases: `RemoteStoreError`

Raised when a dataset is structurally incomplete.

Raised by `ParquetDatasetStore.read_dataset()` when the `_SUCCESS` marker is missing or when parts listed in the manifest cannot be found. Not `NotFound` — some files may exist; the dataset as a whole is not in a readable state.

### ManifestCorrupted

```
ManifestCorrupted(
    message: str = "",
    *,
    path: str | None = None,
    backend: str | None = None,
    reason: str = "",
)
```

Bases: `RemoteStoreError`

Raised when a manifest file cannot be parsed or is structurally invalid.

Raised by `ParquetDatasetStore.read_dataset()` and `read_manifest()` when `manifest.json` exists but contains invalid JSON or is missing required fields.

Parameters:

- **`reason`** (`str`, default: `''` ) – The specific parse or structural failure.

### DatasetManifest

```
DatasetManifest(
    dataset_key: str,
    parts: list[str],
    row_count: int,
    schema_hash: str,
    compression: str,
    created_at_utc: str,
    run_id: str | None = None,
    metadata: dict[str, str] | None = None,
)
```

Immutable metadata record for a written Parquet dataset.

Parameters:

- **`dataset_key`** (`str`) – The store-relative prefix under which the dataset lives.
- **`parts`** (`list[str]`) – Relative filenames of the Parquet part files.
- **`row_count`** (`int`) – Total number of rows across all parts.
- **`schema_hash`** (`str`) – First 16 hex characters of the SHA-256 of schema.to_string().
- **`compression`** (`str`) – Compression codec used (e.g. "zstd", "snappy").
- **`created_at_utc`** (`str`) – ISO 8601 timestamp of dataset creation.
- **`run_id`** (`str | None`, default: `None` ) – Optional caller-supplied run identifier for lineage tracking.
- **`metadata`** (`dict[str, str] | None`, default: `None` ) – Optional caller-supplied key-value metadata.

#### to_json

```
to_json() -> str
```

Serialize the manifest to a JSON string.

Returns:

- `str` – A JSON string with sorted keys and 2-space indentation.

#### from_json

```
from_json(text: str) -> DatasetManifest
```

Deserialize a manifest from a JSON string.

Parameters:

- **`text`** (`str`) – The JSON string to parse.

Returns:

- `DatasetManifest` – A DatasetManifest instance.

Raises:

- `ManifestCorrupted` – If the JSON is invalid, required fields are missing, or field types are wrong.

### ParquetDatasetStore

```
ParquetDatasetStore(
    store: Store,
    *,
    compression: str = "zstd",
    row_group_size: int | None = None,
    max_rows_per_file: int | None = None,
)
```

High-level Parquet dataset operations backed by a `Store`.

Writes multi-part Parquet datasets with manifest metadata and atomic completion markers, enabling reliable dataset exchange over any remote-store backend.

Parameters:

- **`store`** (`Store`) – The Store instance to use for I/O.
- **`compression`** (`str`, default: `'zstd'` ) – Parquet compression codec. Default "zstd".
- **`row_group_size`** (`int | None`, default: `None` ) – Maximum rows per row group within each Parquet file. None (default) uses PyArrow's default.
- **`max_rows_per_file`** (`int | None`, default: `None` ) – If set, split the table into multiple part files of at most this many rows each. None writes a single file.

#### store

```
store: Store
```

The underlying store.

#### compression

```
compression: str
```

The Parquet compression codec.

#### row_group_size

```
row_group_size: int | None
```

Maximum rows per row group, or `None` for PyArrow default.

#### max_rows_per_file

```
max_rows_per_file: int | None
```

Maximum rows per part file, or `None` for single file.

#### write_dataset

```
write_dataset(
    table: Table,
    key: str,
    *,
    overwrite: bool = False,
    run_id: str | None = None,
    metadata: dict[str, str] | None = None,
) -> DatasetManifest
```

Write a PyArrow table as a Parquet dataset under *key*.

Parameters:

- **`table`** (`Table`) – The PyArrow table to write.
- **`key`** (`str`) – Store-relative prefix for the dataset (e.g. "silver/orders").
- **`overwrite`** (`bool`, default: `False` ) – If True, delete any existing dataset at key first. Requires the DELETE capability.
- **`run_id`** (`str | None`, default: `None` ) – Optional run identifier for lineage tracking.
- **`metadata`** (`dict[str, str] | None`, default: `None` ) – Optional key-value metadata to embed in the manifest.

Returns:

- `DatasetManifest` – A DatasetManifest describing the written dataset.

Raises:

- `CapabilityNotSupported` – If the store lacks ATOMIC_WRITE, or DELETE when overwrite is True.
- `AlreadyExists` – If overwrite is False and the dataset exists.

#### read_dataset

```
read_dataset(
    key: str, *, columns: list[str] | None = None
) -> Table
```

Read a Parquet dataset from *key*.

Parameters:

- **`key`** (`str`) – Store-relative prefix for the dataset.
- **`columns`** (`list[str] | None`, default: `None` ) – Optional subset of columns to read. None reads all.

Returns:

- `Table` – A PyArrow table with the concatenated contents of all parts.

Raises:

- `DatasetIncomplete` – If the \_SUCCESS marker is missing or any part listed in the manifest cannot be found.
- `ManifestCorrupted` – If the manifest cannot be parsed.
- `NotFound` – If the manifest file does not exist.

#### read_manifest

```
read_manifest(key: str) -> DatasetManifest
```

Read and parse the manifest for the dataset at *key*.

Parameters:

- **`key`** (`str`) – Store-relative prefix for the dataset.

Returns:

- `DatasetManifest` – The parsed DatasetManifest.

Raises:

- `NotFound` – If manifest.json does not exist.
- `ManifestCorrupted` – If the manifest cannot be parsed.

#### dataset_exists

```
dataset_exists(key: str) -> bool
```

Check whether a completed dataset exists at *key*.

Parameters:

- **`key`** (`str`) – Store-relative prefix for the dataset.

Returns:

- `bool` – True if the \_SUCCESS marker exists, False otherwise.

#### delete_dataset

```
delete_dataset(key: str) -> None
```

Delete an entire dataset at *key*.

Parameters:

- **`key`** (`str`) – Store-relative prefix for the dataset.

Raises:

- `NotFound` – If the dataset folder does not exist.

## See also

- [Parquet Datasets](https://docs.remotestore.dev/stable/guides/parquet-datasets/index.md) — guide to reading and writing managed Parquet datasets
- [Parquet Dataset example](https://docs.remotestore.dev/stable/tutorial/examples/parquet-dataset/index.md) — ParquetDatasetStore in action
- [PyArrow Adapter](https://docs.remotestore.dev/stable/guides/pyarrow-adapter/index.md) — lower-level Store-as-PyArrow-filesystem adapter

# ext.partition

## partition

Partition helpers -- Hive-style partition path building and parsing.

Utilities for constructing and deconstructing partition paths like `year=2026/month=03/day=01/data.parquet`, commonly used in Parquet data lake workflows.

Example

```
from remote_store.ext.partition import partition_path, parse_partition

path = partition_path("data.parquet", year=2026, month="03")
# -> "year=2026/month=03/data.parquet"

parsed = parse_partition(path)
# -> ParsedPartition(partitions={"year": "2026", "month": "03"},
#                     filename="data.parquet")
```

### ParsedPartition

```
ParsedPartition(partitions: dict[str, str], filename: str)
```

Result of parsing a Hive-style partition path.

Attributes:

- **`partitions`** (`dict[str, str]`) – Ordered mapping of partition column names to values.
- **`filename`** (`str`) – The trailing non-partition portion of the path.

### partition_path

```
partition_path(
    filename: str, /, **partitions: str | int
) -> str
```

Build a Hive-style partition path.

Parameters:

- **`filename`** (`str`) – Leaf file name (e.g., "data.parquet"). Must be non-empty and must not contain /.
- **`partitions`** (`str | int`, default: `{}` ) – Partition key-value pairs. Values are coerced to str. Keys and coerced values must be non-empty and must not contain =.

Returns:

- `str` – Forward-slash-joined path like "year=2026/month=03/data.parquet".

Raises:

- `ValueError` – If filename is empty or contains /, if any partition key or coerced value is empty, or if any key or value contains =.

### parse_partition

```
parse_partition(path: str) -> ParsedPartition
```

Parse a Hive-style partition path into its components.

A segment is treated as a partition if it contains exactly one `=` and the key portion is non-empty. Once a non-partition segment is encountered, all remaining segments (including any later `key=value` segments) become part of the filename.

Parameters:

- **`path`** (`str`) – The partition path to parse (e.g., "year=2026/month=03/data.parquet").

Returns:

- `ParsedPartition` – A ParsedPartition with extracted partitions and filename.

Raises:

- `ValueError` – If path is empty.

## See also

- [Data Lake Patterns](https://docs.remotestore.dev/stable/guides/data-lake-patterns/index.md) — guide to Hive-style partitioning and data lake layouts

# ext.pydantic

## pydantic

Pydantic adapter — convert any Pydantic model to a RegistryConfig.

Install with `pip install "remote-store[pydantic]"`.

Example

```
from pydantic_settings import BaseSettings, SettingsConfigDict
from remote_store.ext.pydantic import from_pydantic

class MySettings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="RS_", env_nested_delimiter="__")
    backends: dict = {}
    stores: dict = {}

config = from_pydantic(MySettings())
```

### from_pydantic

```
from_pydantic(model: BaseModel) -> RegistryConfig
```

Convert a Pydantic model to a `RegistryConfig`.

Calls `model.model_dump()` to produce a plain dict, then delegates to `RegistryConfig.from_dict()`. Secret wrapping, unknown-key warnings, and validation all happen in `from_dict()`.

Pydantic `SecretStr` fields in backend `options` dicts are automatically unwrapped to plain strings before reaching `from_dict()`, so `from_dict()`'s sensitive-key detection works correctly. Users may use either `str` or `SecretStr` for credential values in their models.

Parameters:

- **`model`** (`BaseModel`) – A Pydantic model whose model_dump() output has backends and stores keys matching the RegistryConfig schema.

Returns:

- `RegistryConfig` – An immutable RegistryConfig.

Raises:

- `TypeError` – If the model dump does not conform to the expected schema.

## See also

- [Extensions](https://docs.remotestore.dev/stable/guides/extensions/index.md) — overview of all extension modules
- [Config Loaders example](https://docs.remotestore.dev/stable/tutorial/examples/config-loaders/index.md) — Pydantic config loader in action

# ext.streams

Composable `BinaryIO` wrappers for progress tracking and checksum computation. These operate at the stream level — wrap the stream returned by `store.read()` or passed to `store.write()`, no proxy wrapping needed.

See the [streams spec](https://docs.remotestore.dev/stable/explanation/design/specs/033-ext-streams/index.md) for invariants.

## streams

Stream-level wrappers for progress tracking and checksum computation.

Composable `BinaryIO` wrappers that operate at the stream level, not the Store level. No proxy wrapping is needed — just wrap the stream returned by `store.read()` or passed to `store.write()`.

Example

```
from remote_store.ext.streams import ProgressReader, ChecksumReader

stream = ChecksumReader(
    ProgressReader(store.read("file.bin"), callback=update_bar),
    algorithm="sha256",
)
data = stream.read()
assert stream.hexdigest() == expected_hex
```

### ProgressReader

```
ProgressReader(
    inner: BinaryIO, callback: Callable[[int], None]
)
```

Bases: `_StreamWrapper`

Readable `BinaryIO` wrapper that fires a callback per `read()`.

The callback receives the number of bytes read (not cumulative). Empty reads do not fire the callback.

Parameters:

- **`inner`** (`BinaryIO`) – Readable binary stream to wrap.
- **`callback`** (`Callable[[int], None]`) – Called with len(data) after each non-empty read().

### ProgressWriter

```
ProgressWriter(
    inner: BinaryIO, callback: Callable[[int], None]
)
```

Bases: `_StreamWrapper`

Writable `BinaryIO` wrapper that fires a callback per `write()`.

The callback receives the number of bytes written (not cumulative). Empty writes do not fire the callback.

Note

Assumes buffered I/O semantics — `write()` consumes all data or raises. Wrapping a `RawIOBase` (partial-write) stream would cause the reported byte count to diverge from the bytes actually written.

Parameters:

- **`inner`** (`BinaryIO`) – Writable binary stream to wrap.
- **`callback`** (`Callable[[int], None]`) – Called with len(data) after each non-empty write().

### ChecksumReader

```
ChecksumReader(inner: BinaryIO, algorithm: str = 'sha256')
```

Bases: `_StreamWrapper`

Readable `BinaryIO` wrapper that computes a rolling hash.

Parameters:

- **`inner`** (`BinaryIO`) – Readable binary stream to wrap.
- **`algorithm`** (`str`, default: `'sha256'` ) – Hash algorithm name (default "sha256"). Must be supported by hashlib.

#### algorithm

```
algorithm: str
```

Hash algorithm name (lowercase).

#### hexdigest

```
hexdigest() -> str
```

Return the lowercase hex digest of all bytes read so far.

### ChecksumWriter

```
ChecksumWriter(inner: BinaryIO, algorithm: str = 'sha256')
```

Bases: `_StreamWrapper`

Writable `BinaryIO` wrapper that computes a rolling hash.

Note

Assumes buffered I/O semantics — `write()` consumes all data or raises. Wrapping a `RawIOBase` (partial-write) stream would cause the hash to include bytes that were not actually written.

Parameters:

- **`inner`** (`BinaryIO`) – Writable binary stream to wrap.
- **`algorithm`** (`str`, default: `'sha256'` ) – Hash algorithm name (default "sha256"). Must be supported by hashlib.

#### algorithm

```
algorithm: str
```

Hash algorithm name (lowercase).

#### hexdigest

```
hexdigest() -> str
```

Return the lowercase hex digest of all bytes written so far.

### read_with_progress

```
read_with_progress(
    store: Store, path: str, callback: Callable[[int], None]
) -> ProgressReader
```

Read a file with progress tracking.

Convenience wrapper around `ProgressReader(store.read(path), callback)`. The caller is responsible for closing the returned stream.

Parameters:

- **`store`** (`Store`) – The Store to read from.
- **`path`** (`str`) – Store-relative file path.
- **`callback`** (`Callable[[int], None]`) – Called with byte count after each non-empty read().

Returns:

- `ProgressReader` – A ProgressReader wrapping the store's read stream.

## See also

- [Streaming IO example](https://docs.remotestore.dev/stable/tutorial/examples/streaming-io/index.md) — composable stream wrappers in action

# ext.transfer

## transfer

Transfer operations -- upload, download, and cross-store transfer.

All functions stream data and never load full files into memory. An optional `on_progress` callback fires per chunk with the byte count.

Example

```
from remote_store.ext.transfer import upload, download, transfer

upload(store, "local/file.txt", "remote/key.txt", overwrite=True)
download(store, "remote/key.txt", "local/file.txt", overwrite=True)
transfer(src_store, "src.txt", dst_store, "dst.txt", overwrite=True)
```

### upload

```
upload(
    store: Store,
    local_path: str | PathLike[str],
    remote_path: str,
    *,
    overwrite: bool = False,
    on_progress: Callable[[int], None] | None = None,
) -> None
```

Upload a local file to a Store.

Opens the local file in binary read mode and streams it to the Store via `store.write()`. The full file is never loaded into memory.

Parameters:

- **`store`** (`Store`) – The Store to write to.
- **`local_path`** (`str | PathLike[str]`) – Path to the local file.
- **`remote_path`** (`str`) – Destination key in the Store.
- **`overwrite`** (`bool`, default: `False` ) – Forwarded to store.write().
- **`on_progress`** (`Callable[[int], None] | None`, default: `None` ) – Called per read with the byte count (not cumulative).

Returns:

- `None` – None

Raises:

- `FileNotFoundError` – If local_path does not exist.

### download

```
download(
    store: Store,
    remote_path: str,
    local_path: str | PathLike[str],
    *,
    overwrite: bool = False,
    on_progress: Callable[[int], None] | None = None,
) -> None
```

Download a file from a Store to a local path.

Reads the remote file in 1 MiB chunks and writes each chunk to the local file. The full file is never loaded into memory.

Parameters:

- **`store`** (`Store`) – The Store to read from.
- **`remote_path`** (`str`) – Key to read from the Store.
- **`local_path`** (`str | PathLike[str]`) – Destination path on the local filesystem.
- **`overwrite`** (`bool`, default: `False` ) – If False (default) and local_path exists, raises FileExistsError.
- **`on_progress`** (`Callable[[int], None] | None`, default: `None` ) – Called per read with the byte count (not cumulative).

Returns:

- `None` – None

Raises:

- `FileExistsError` – If local_path exists and overwrite is False.

### transfer

```
transfer(
    src_store: Store,
    src_path: str,
    dst_store: Store,
    dst_path: str,
    *,
    overwrite: bool = False,
    on_progress: Callable[[int], None] | None = None,
) -> None
```

Transfer a file from one Store to another.

Reads the source file and streams it to the destination via `dst_store.write()`. The full file is never loaded into memory.

Parameters:

- **`src_store`** (`Store`) – The Store to read from.
- **`src_path`** (`str`) – Key to read from src_store.
- **`dst_store`** (`Store`) – The Store to write to (may be the same as src_store).
- **`dst_path`** (`str`) – Destination key in dst_store.
- **`overwrite`** (`bool`, default: `False` ) – Forwarded to dst_store.write().
- **`on_progress`** (`Callable[[int], None] | None`, default: `None` ) – Called per read with the byte count (not cumulative).

Returns:

- `None` – None

## See also

- [Transfer Operations](https://docs.remotestore.dev/stable/guides/transfer-operations/index.md) — guide to upload, download, and cross-store transfer
- [Transfer Operations example](https://docs.remotestore.dev/stable/tutorial/examples/transfer-operations/index.md) — transfer operations in action

# ext.write

Client-side hashing helpers for write operations. Guarantees a populated `WriteResult.digest` regardless of whether the backend declares `WRITE_RESULT_NATIVE`.

See the [ext.write spec](https://docs.remotestore.dev/stable/explanation/design/specs/046-ext-write/index.md) for invariants.

Async counterpart: [`aio.ext.write`](https://docs.remotestore.dev/stable/reference/api/aio/extensions/write/index.md).

## write

Write helpers with client-side content hashing.

Guarantees a populated `WriteResult.digest` regardless of whether the backend declares `WRITE_RESULT_NATIVE`. The hash is always computed client-side over the bytes as they are written.

### HashingAtomicWriter

```
HashingAtomicWriter(
    inner: BinaryIO, algorithm: str = "sha256"
)
```

Bases: `ChecksumWriter`

Writable stream wrapper used by `open_atomic_with_hash`.

Subclasses `ChecksumWriter` to add a `.result` attribute that is populated with a `WriteResult` after the context manager exits successfully. `result` is `None` if the block raised.

### write_with_hash

```
write_with_hash(
    store: Store,
    path: str,
    content: BinaryIO | bytes,
    *,
    algorithm: str = "sha256",
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> WriteResult
```

Write *content* to *path* and return a `WriteResult` with a client-computed digest.

Works on every backend declaring `Capability.WRITE`. The hash is always computed client-side regardless of `WRITE_RESULT_NATIVE`.

Parameters:

- **`store`** (`Store`) – The Store to write to.
- **`path`** (`str`) – Store-relative file path.
- **`content`** (`BinaryIO | bytes`) – bytes or readable binary stream.
- **`algorithm`** (`str`, default: `'sha256'` ) – Hash algorithm name (default "sha256").
- **`overwrite`** (`bool`, default: `False` ) – If False, raises AlreadyExists when path exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user metadata (see Store.write()).

Returns:

- `WriteResult` – WriteResult with digest populated from the client-side hash.

### open_atomic_with_hash

```
open_atomic_with_hash(
    store: Store,
    path: str,
    *,
    algorithm: str = "sha256",
    overwrite: bool = False,
    metadata: Mapping[str, str] | None = None,
) -> Iterator[HashingAtomicWriter]
```

Context manager for streaming atomic writes with client-side hashing.

Requires `Capability.ATOMIC_WRITE`. On successful exit, the yielded `HashingAtomicWriter.result` is a `WriteResult` with `digest` populated. On exception `result` remains `None`.

**Metadata branch:** When *metadata* is non-empty, the implementation buffers all written bytes in memory (`io.BytesIO`) and then calls `store.write_atomic()` on exit. This means (a) the full payload is held in RAM, and (b) validation errors (`ValueError`) and capability checks (`CapabilityNotSupported`) are raised after the caller has finished writing — not before. For large payloads, prefer calling `store.write()` directly with `metadata=`.

**`source` field:** Both branches always set `source="basic"` on the returned `WriteResult`. The metadata branch discards the `source` returned by `store.write_atomic()` because the rich- source contract cannot be honored for the no-metadata branch (`open_atomic` returns only a stream, not a `WriteResult`), so both branches use the same value for consistency.

Parameters:

- **`store`** (`Store`) – The Store to write to.
- **`path`** (`str`) – Store-relative file path.
- **`algorithm`** (`str`, default: `'sha256'` ) – Hash algorithm name (default "sha256").
- **`overwrite`** (`bool`, default: `False` ) – If False, raises AlreadyExists when path exists.
- **`metadata`** (`Mapping[str, str] | None`, default: `None` ) – Optional user metadata forwarded to Store.write_atomic() on backends that declare USER_METADATA.

Yields:

- `HashingAtomicWriter` – HashingAtomicWriter — write to it as a binary stream.

Raises:

- `CapabilityNotSupported` – If the backend lacks ATOMIC_WRITE.
- `AlreadyExists` – If path exists and overwrite is False.

# ext.yaml

## yaml

YAML config loader — load RegistryConfig from a YAML file.

Install with `pip install "remote-store[yaml]"`.

Example

```
from remote_store.ext.yaml import from_yaml

config = from_yaml("remote-store.yaml")
```

Accepts either [pyyaml](https://pyyaml.org/) or [ruamel.yaml](https://yaml.readthedocs.io/) as the parser.

### from_yaml

```
from_yaml(
    path: str | Path, *, resolve_env_vars: bool = False
) -> RegistryConfig
```

Load config from a YAML file.

Accepts either `pyyaml` or `ruamel.yaml` as the parser.

Parameters:

- **`path`** (`str | Path`) – Path to the YAML file.
- **`resolve_env_vars`** (`bool`, default: `False` ) – When True, resolve ${VAR} placeholders via resolve_env before constructing the config.

Returns:

- `RegistryConfig` – An immutable RegistryConfig.

Raises:

- `ModuleNotFoundError` – If neither pyyaml nor ruamel.yaml is installed.
- `FileNotFoundError` – If path does not exist.
- `TypeError` – If the top-level YAML value is not a mapping.
- `KeyError` – If resolve_env_vars is True and a placeholder references an unset variable with no default.

## See also

- [Extensions](https://docs.remotestore.dev/stable/guides/extensions/index.md) — overview of all extension modules
- [Config Loaders example](https://docs.remotestore.dev/stable/tutorial/examples/config-loaders/index.md) — YAML config loader in action

# Capabilities Matrix

Every backend declares which operations it supports via the [Capability](https://docs.remotestore.dev/stable/reference/api/capabilities/#remote_store.Capability) enum. Use [CapabilitySet](https://docs.remotestore.dev/stable/reference/api/capabilities/#remote_store.CapabilitySet) to query at runtime before calling an operation.

## Backend x Capability

| Capability          | [Local](https://docs.remotestore.dev/stable/guides/backends/local/index.md) | [Memory](https://docs.remotestore.dev/stable/guides/backends/memory/index.md) | [HTTP](https://docs.remotestore.dev/stable/guides/backends/http/index.md) | [S3](https://docs.remotestore.dev/stable/guides/backends/s3/index.md) | [S3-PyArrow](https://docs.remotestore.dev/stable/guides/backends/s3-pyarrow/index.md) | [SFTP](https://docs.remotestore.dev/stable/guides/backends/sftp/index.md) | [Azure](https://docs.remotestore.dev/stable/guides/backends/azure/index.md) | [Graph](https://docs.remotestore.dev/stable/guides/backends/graph/index.md)¹ | [SQLBlob](https://docs.remotestore.dev/stable/guides/backends/sql-blob/index.md) | [SQLQuery](https://docs.remotestore.dev/stable/guides/backends/sql-query/index.md) |
| ------------------- | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| READ                | Yes                                                                         | Yes                                                                           | Yes                                                                       | Yes                                                                   | Yes                                                                                   | Yes                                                                       | Yes                                                                         | Yes                                                                          | Yes                                                                              | Yes                                                                                |
| WRITE               | Yes                                                                         | Yes                                                                           | —                                                                         | Yes                                                                   | Yes                                                                                   | Yes                                                                       | Yes                                                                         | Yes                                                                          | Yes                                                                              | —                                                                                  |
| DELETE              | Yes                                                                         | Yes                                                                           | —                                                                         | Yes                                                                   | Yes                                                                                   | Yes                                                                       | Yes                                                                         | Yes                                                                          | Yes                                                                              | —                                                                                  |
| LIST                | Yes                                                                         | Yes                                                                           | —                                                                         | Yes                                                                   | Yes                                                                                   | Yes                                                                       | Yes                                                                         | Yes                                                                          | Yes                                                                              | Yes                                                                                |
| GLOB                | Yes                                                                         | —                                                                             | —                                                                         | Yes                                                                   | Yes                                                                                   | —                                                                         | Yes                                                                         | —                                                                            | Yes                                                                              | Yes                                                                                |
| MOVE                | Yes                                                                         | Yes                                                                           | —                                                                         | Yes                                                                   | Yes                                                                                   | Yes                                                                       | Yes                                                                         | Yes                                                                          | Yes                                                                              | —                                                                                  |
| COPY                | Yes                                                                         | Yes                                                                           | —                                                                         | Yes                                                                   | Yes                                                                                   | Yes                                                                       | Yes                                                                         | Yes                                                                          | Yes                                                                              | —                                                                                  |
| ATOMIC_WRITE        | Yes                                                                         | Yes                                                                           | —                                                                         | Yes                                                                   | Yes                                                                                   | Yes                                                                       | Yes                                                                         | Yes                                                                          | Yes                                                                              | —                                                                                  |
| ATOMIC_MOVE         | Yes                                                                         | Yes                                                                           | —                                                                         | —                                                                     | —                                                                                     | —                                                                         | —                                                                           | —                                                                            | Yes                                                                              | —                                                                                  |
| METADATA            | Yes                                                                         | Yes                                                                           | Yes                                                                       | Yes                                                                   | Yes                                                                                   | Yes                                                                       | Yes                                                                         | Yes                                                                          | Yes                                                                              | Yes                                                                                |
| SEEKABLE_READ       | Yes                                                                         | Yes                                                                           | —                                                                         | Yes                                                                   | Yes                                                                                   | Yes                                                                       | —                                                                           | —                                                                            | Yes                                                                              | Yes                                                                                |
| LAZY_READ           | Yes                                                                         | —                                                                             | Yes                                                                       | Yes                                                                   | Yes                                                                                   | Yes                                                                       | Yes                                                                         | Yes                                                                          | —                                                                                | —                                                                                  |
| WRITE_RESULT_NATIVE | Yes                                                                         | Yes                                                                           | —                                                                         | Yes                                                                   | Yes                                                                                   | Yes                                                                       | Yes                                                                         | Yes                                                                          | Yes²                                                                             | —                                                                                  |
| USER_METADATA       | —                                                                           | Yes                                                                           | —                                                                         | Yes                                                                   | —                                                                                     | —                                                                         | Yes                                                                         | —                                                                            | Yes²                                                                             | —                                                                                  |

¹ Graph is an **async-only** backend — construct it via `AsyncStore(backend=GraphBackend(...))`; there is no sync `Store` wrapper or config `type=` string. The column lists its declared capabilities.

² `WRITE_RESULT_NATIVE` and `USER_METADATA` are declared by `SQLBlobBackend` only when the backing table includes a `user_metadata` column (see the SQLBlob guide for schema requirements). Legacy tables without this column do not declare either capability.

**Near-full:** Local lacks `USER_METADATA` — passing non-empty `metadata=` raises `CapabilityNotSupported`. S3 and S3-PyArrow lack `ATOMIC_MOVE` (copy-then-delete semantics). SQLBlob lacks `LAZY_READ` — the entire blob is loaded into memory before a stream is returned. Writes also materialize the full stream before the SQL INSERT/UPDATE because BLOB columns require complete data.

**Partial support:** Memory lacks native `GLOB` and `LAZY_READ` (all data lives in process memory; use the portable fallback `ext.glob.glob_files()` — see the [Glob Pattern Matching](https://docs.remotestore.dev/stable/guides/glob-pattern-matching/index.md) guide). SFTP lacks both `GLOB` and `ATOMIC_MOVE`. Azure lacks `SEEKABLE_READ` and `ATOMIC_MOVE` (forward-only chunk iterator, copy-then-delete move). Missing `SEEKABLE_READ` means only that `read()` is forward-only, not that random access is expensive: on the sync backend `read_seekable()` uses a native HTTP-Range reader (one ranged download per read, no temp-file spill) — see the [Azure guide](https://docs.remotestore.dev/stable/guides/backends/azure/#streaming-and-seekable-reads). Graph lacks `GLOB`, `SEEKABLE_READ`, `ATOMIC_MOVE`, and `USER_METADATA` (Microsoft Graph has no server-side glob, forward-only download streams, a native server-side move that may complete asynchronously so atomicity is not guaranteed, and no OneDrive/SharePoint user-metadata surface).

**Read-only:** SQLQuery supports `READ`, `LIST`, `METADATA`, `GLOB`, and `SEEKABLE_READ` — no write operations.

**Minimal:** HTTP supports `READ`, `METADATA`, and `LAZY_READ` (read-only backend).

## Querying capabilities at runtime

```
from remote_store import Capability

if store.supports(Capability.GLOB):
    results = store.glob("**/*.csv")
else:
    from remote_store import glob_files
    results = glob_files(store, "**/*.csv")

# Seekable read — works on any backend. Native/lazy where the backend
# optimizes it (S3, sync Azure's HTTP-Range reader); spooled to a temp
# file otherwise (HTTP, an async backend bridged to sync). SEEKABLE_READ
# reports only whether read() itself is seekable, not this cost.
from remote_store import seekable_read

with seekable_read(store, "report.csv") as f:
    header = f.read(128)
    f.seek(0)  # guaranteed seekable

# Atomic move — quality flag, not a method gate.
# move() is always callable (when MOVE is declared), but only atomic
# on backends that declare ATOMIC_MOVE.
if store.supports(Capability.ATOMIC_MOVE):
    # Atomic: readers see either the old or the new path, never both.
    store.move("staging/data.parquet", "prod/data.parquet")
else:
    # Non-atomic backend: copy-then-delete. A failure between the two
    # steps may leave both paths present — handle errors explicitly.
    store.copy("staging/data.parquet", "prod/data.parquet")
    store.delete("staging/data.parquet")

# Lazy read — quality flag. read() always works, but on backends without
# LAZY_READ the entire file is loaded into memory before the stream is
# returned. Use this flag to decide whether partial reads are efficient.
if store.supports(Capability.LAZY_READ):
    # Stream is connected to the native source; only the bytes you read
    # are transferred. Safe to read a small prefix of a large file.
    with store.read("large_file.bin") as f:
        header = f.read(256)
else:
    # Data is pre-loaded; reading any amount costs the full file.
    # For large files prefer read_bytes() or avoid partial reads.
    data = store.read_bytes("large_file.bin")
    header = data[:256]
```

## See also

- [Choosing a Backend](https://docs.remotestore.dev/stable/guides/choosing-a-backend/index.md) — decision tree for picking the right backend
- [API Reference: Capability](https://docs.remotestore.dev/stable/reference/api/capabilities/index.md)

# Migration Guide

Breaking changes and upgrade paths between `remote-store` versions.

`remote-store` has been published on PyPI since v0.11.0 (first Beta release). The core Store API is stable, but extensions may evolve. This page documents changes that require action when upgrading.

## v0.28.0 to v0.29.0

**Azure HNS is now an explicit, mandatory declaration:**

`AzureBackend` and `AsyncAzureBackend` no longer auto-detect Hierarchical Namespace (ADLS Gen2) by probing the account on first use. You must now declare the account's nature with the required `hns` argument. A backend constructed without `hns` raises `ValueError`.

```
# Before (v0.28.0): HNS auto-detected on first I/O
backend = AzureBackend(container="data", account_name="acct", account_key="...")

# After (v0.29.0): declare hns explicitly
backend = AzureBackend(container="data", hns=True, account_name="acct", account_key="...")
```

The same applies to config `options` (`"hns": true`) and to `AsyncAzureBackend`.

**If you do not know an account's HNS status**, discover it once with the new fail-loud helper and pass the result:

```
from remote_store.backends import AzureUtils

is_hns = AzureUtils.detect_hns(account_name="acct", account_key="...")
backend = AzureBackend(container="data", hns=is_hns, account_name="acct", account_key="...")
```

`AzureUtils.adetect_hns(...)` is the async sibling. Both raise on a probe error rather than silently falling back to flat behavior.

**Why:** the old `GetAccountInfo` probe could fail, return propagation-delayed authorization state, or be denied by least-privilege credentials — silently degrading an HNS account to flat semantics. A declared value is deterministic from construction and removes that failure class.

## v0.24.1 to v0.25.0

**`[sftp]` extra now requires `paramiko>=3.0`:**

The SFTP backend uses paramiko 3.0's `channel_timeout=` connect kwarg. Environments pinned to `paramiko<3` must upgrade. `pip install "remote-store[sftp]"` resolves the correct version automatically; pinned `paramiko==2.x` will now conflict.

**Azure HNS error types now match the canonical mapping:**

On real ADLS Gen2 (Hierarchical Namespace) accounts, many `AzureBackend` and `AsyncAzureBackend` operations previously raised the wrong error type when the path named a directory blob (or, conversely, a file blob where a directory was expected). Stage 3 live verification in this release surfaced the deviations; all now raise `InvalidPath` per the canonical mapping. **If you catch the old error types**, those clauses will no longer fire on HNS:

| Operation                                    | Old error (HNS)                                     | New error     |
| -------------------------------------------- | --------------------------------------------------- | ------------- |
| `read`, `read_bytes`, `read_seekable` on dir | silently returned `b""`                             | `InvalidPath` |
| `delete` on dir (file API)                   | silently destroyed directory marker (**data loss**) | `InvalidPath` |
| `get_file_info` on dir                       | `NotFound`                                          | `InvalidPath` |
| `is_folder` on file                          | `True`                                              | `False`       |
| `get_folder_info` on file                    | `NotFound`                                          | `InvalidPath` |
| `delete_folder` on file                      | `DirectoryNotEmpty` / `NotFound`                    | `InvalidPath` |
| `move` / `copy` on dir source or dest        | `RemoteStoreError(InvalidInput)` / `AlreadyExists`  | `InvalidPath` |
| `open_atomic` on dir target                  | `AlreadyExists`                                     | `InvalidPath` |
| `write` / `write_atomic` on dir target       | `AlreadyExists`                                     | `InvalidPath` |
| `move(p, p)` / `copy(p, p)` self-op          | `AlreadyExists`                                     | no-op         |

Flat-namespace blob accounts (non-HNS) and Azurite were already correct and are unaffected. Sync and async siblings behave identically.

**`Store.move(p, p)` / `copy(p, p)` self-op error type:**

Across all backends, `Store.move` / `copy` and `AsyncStore.move` / `copy` now raise `InvalidPath` (was `NotFound`) when the source path is a directory and `src == dst`. The file no-op case is unchanged.

**`hatch run test-cov` no longer enforces `--cov-fail-under=95`:**

The coverage floor moved to a new `hatch run test-cov-strict` script. Local `test-cov` is now a coverage *report* only; CI runs the strict variant. If your tooling or CI relied on `test-cov` failing under 95% switch to `test-cov-strict`.

## v0.24.0 to v0.24.1

**S3 botocore Config options route through `config_kwargs`:**

Pre-built `botocore.config.Config` objects are no longer accepted in `client_options["client_kwargs"]`. Pass the same constructor kwargs through `config_kwargs` (a plain dict) instead. The old form raised `TypeError` at first I/O on s3fs ≥ 2024.x already; v0.24.1 fails fast with `ValueError` at backend construction and a message naming the supported channel.

- Old: `S3Backend(..., client_options={"client_kwargs": {"config": Config(connect_timeout=10, retries={"max_attempts": 5})}})`
- New: `S3Backend(..., client_options={"config_kwargs": {"connect_timeout": 10, "retries": {"max_attempts": 5}}})`

The new "Botocore Client Tuning" section in `docs-src/guides/backends/s3.md` documents proxies, retries, timeouts, and MinIO path-style addressing with runnable snippets. Applies to both `S3Backend` and `S3PyArrowBackend`.

**Custom backends must declare `CAPABILITIES: ClassVar[CapabilitySet]`:**

If you maintain a custom `Backend` or `AsyncBackend` subclass, add a class-level `CAPABILITIES` attribute exposing the capability set without requiring instantiation, and delegate the `capabilities` property to it. Conformance and the new graph-IR generator both read from this class attribute. See `docs-src/guides/custom-backend-guide.md` § "Step 3" for the template; existing constructor-set capability logic continues to work, but the ClassVar is required for static extraction.

## v0.20.0 to v0.21.0

**`ParquetSerializer.deserialize()` returns Arrow Table:**

`ParquetSerializer.deserialize()` now returns a `pyarrow.Table` instead of a `pandas.DataFrame`. This removes the hidden hard dependency on pandas for `remote-store[dagster,arrow]` users.

- Old: `result = serializer.deserialize(data) # pandas DataFrame`
- New: `result = serializer.deserialize(data) # pyarrow.Table`
- If you need pandas: `df = serializer.deserialize(data).to_pandas()`
- If you need polars: `df = pl.from_arrow(serializer.deserialize(data))`

Custom subclasses that override `deserialize()` (e.g. `PolarsParquetSerializer` from the medallion example) continue to work but the override is now optional — the base class already returns a framework-neutral Arrow Table.

## v0.19.0 to v0.20.0

**Deprecated aliases removed:**

Three factory functions renamed in v0.18.0 have had their old names removed:

- `pydantic_to_registry_config()` → use `from_pydantic()`
- `remote_store_io_manager()` → use `dagster_io_manager()`
- `cached_store()` → use `cache()`

Pre-v1: removed without a deprecation cycle. Find-and-replace is sufficient.

## v0.18.0 to v0.19.0

**Factory function renames:**

Three ext factory functions were renamed for naming consistency. Old names emitted `DeprecationWarning` in v0.18.x and are removed after v0.19.0 (see [above](#v0190-to-v0200)).

- `pydantic_to_registry_config()` → `from_pydantic()`
- `remote_store_io_manager()` → `dagster_io_manager()`
- `cached_store()` → `cache()`

## v0.17.0 to v0.18.0

**Extension imports moved:**

Optional-dependency extensions are no longer re-exported from `remote_store.__init__`. Import them directly from their extension module:

- Old: `from remote_store import pyarrow_fs, StoreFileSystemHandler`
- New: `from remote_store.ext.arrow import pyarrow_fs, StoreFileSystemHandler`
- Old: `from remote_store import otel_hooks, otel_observe`
- New: `from remote_store.ext.otel import otel_hooks, otel_observe`
- Old: `from remote_store import pydantic_to_registry_config`
- New: `from remote_store.ext.pydantic import from_pydantic`
- Old: `from remote_store import from_yaml`
- New: `from remote_store.ext.yaml import from_yaml`

Pure-Python extensions (`ext.batch`, `ext.transfer`, `ext.glob`, `ext.observe`, `ext.cache`, `ext.partition`) are unchanged — they were already unconditionally exported from `remote_store.__init__`.

## v0.15.0 to v0.16.0

**YAML config loader moved to extension:**

- `RegistryConfig.from_yaml()` has been removed from the core class and replaced by `from_yaml()` in `remote_store.ext.yaml`.
- Old: `config = RegistryConfig.from_yaml("config.yaml")`
- New: `from remote_store.ext.yaml import from_yaml` then `config = from_yaml("config.yaml")`
- Install the optional extra: `pip install "remote-store[yaml]"`

## v0.13.0 to v0.14.0

**Config loaders (new feature, no breaking changes):**

- `RegistryConfig.from_toml()` and `from_yaml()` are new. Existing `from_dict()` usage continues to work unchanged.
- `from_dict()` now warns on unknown keys. If you were passing extra keys silently, you will see warnings. Remove the unknown keys or suppress the warning.

## v0.12.0 to v0.13.0

**Credential hygiene:**

- Backend config values for keys named `key`, `secret`, `password`, `account_key`, `sas_token`, and `connection_string` are now automatically wrapped in `Secret` objects by `from_dict()`.
- If you were accessing these values directly as strings, use `secret.reveal()` to get the plain-text value.
- `repr()` and `str()` of config objects now mask credentials with `***`.

## v0.11.0 to v0.12.0

**Glob capability:**

- `Store.glob()` now requires `Capability.GLOB`. Backends that do not support it (Memory, SFTP) will raise `CapabilityNotSupported`.
- Use `ext.glob.glob_files()` as a portable fallback for all backends.

## General upgrade advice

1. Pin to a specific minor version in production: `remote-store>=0.16,<0.17`.
1. Read the [CHANGELOG](https://github.com/haalfi/remote-store/blob/master/CHANGELOG.md) for each version you skip.
1. Run your test suite after upgrading — the library has 95%+ coverage and you should too.

## See also

- [CHANGELOG](https://github.com/haalfi/remote-store/blob/master/CHANGELOG.md)
- [Contributing](https://docs.remotestore.dev/stable/explanation/contributing/index.md) — stability tiers and versioning policy

# Tested upper-bound versions

Each `[<extra>]` declares a floor in `pyproject.toml` and in most cases deliberately no ceiling (the exceptions: `arrow` / `sql-query` pin `pyarrow<25`, and `graph` / `httpx` pin `httpx<1.0` — see the comment on `[project.optional-dependencies]` in `pyproject.toml`). The drift guard (`.github/workflows/drift-guard.yml`) records the last known-good resolution per extra in `infra/drift-locks/` and re-resolves weekly against the latest available versions (including pre-releases) to surface silent transitive upgrades before they reach users.

The table below is the projection of those lock files onto the top-level packages each extra declares. "Tested up to" is the exact version pinned in the lock at capture time — that is what CI was last green against.

## `[arrow]`

*Captured 2026-05-25 on Python 3.13.*

| Package   | Tested up to |
| --------- | ------------ |
| `pyarrow` | `24.0.0`     |

## `[azure]`

*Captured 2026-07-03 on Python 3.13.*

| Package                       | Tested up to |
| ----------------------------- | ------------ |
| `azure-identity`              | `1.26.0b2`   |
| `azure-storage-file-datalake` | `12.25.0`    |

## `[dagster]`

*Captured 2026-07-03 on Python 3.13.*

| Package   | Tested up to |
| --------- | ------------ |
| `dagster` | `1.13.11`    |

## `[graph]`

*Captured 2026-07-09 on Python 3.13.*

| Package           | Tested up to |
| ----------------- | ------------ |
| `httpx`           | `0.28.1`     |
| `msal`            | `1.38.0rc1`  |
| `msal-extensions` | `1.3.1`      |
| `platformdirs`    | `4.10.0`     |

## `[httpx]`

*Captured 2026-07-09 on Python 3.13.*

| Package | Tested up to |
| ------- | ------------ |
| `httpx` | `0.28.1`     |

## `[otel]`

*Captured 2026-07-03 on Python 3.13.*

| Package             | Tested up to |
| ------------------- | ------------ |
| `opentelemetry-api` | `1.43.0`     |

## `[pydantic]`

*Captured 2026-07-03 on Python 3.13.*

| Package             | Tested up to |
| ------------------- | ------------ |
| `pydantic-settings` | `2.14.2`     |

## `[requests]`

*Captured 2026-06-22 on Python 3.13.*

| Package    | Tested up to |
| ---------- | ------------ |
| `requests` | `2.34.2`     |
| `urllib3`  | `2.7.0`      |

## `[s3]`

*Captured 2026-06-22 on Python 3.13.*

| Package | Tested up to |
| ------- | ------------ |
| `s3fs`  | `2026.6.0`   |

## `[s3-pyarrow]`

*Captured 2026-06-22 on Python 3.13.*

| Package   | Tested up to |
| --------- | ------------ |
| `pyarrow` | `24.0.0`     |
| `s3fs`    | `2026.6.0`   |

## `[sftp]`

*Captured 2026-06-13 on Python 3.13.*

| Package    | Tested up to |
| ---------- | ------------ |
| `paramiko` | `5.0.0`      |
| `tenacity` | `9.1.4`      |

## `[sql]`

*Captured 2026-07-03 on Python 3.13.*

| Package      | Tested up to |
| ------------ | ------------ |
| `sqlalchemy` | `2.1.0b3`    |

## `[sql-query]`

*Captured 2026-07-03 on Python 3.13.*

| Package      | Tested up to |
| ------------ | ------------ |
| `pyarrow`    | `24.0.0`     |
| `sqlalchemy` | `2.1.0b3`    |

## `[yaml]`

*Captured 2026-05-25 on Python 3.13.*

| Package  | Tested up to |
| -------- | ------------ |
| `pyyaml` | `6.0.3`      |

# Features — remote-store v0.29.1

Authoritative snapshot of what remote-store delivers in this version. Updated each release. **This is the single reference for the package's feature surface.**

______________________________________________________________________

## What this package does

remote-store gives application code a single, portable file-storage API. Write against `Store` once; swap the backend by changing one line of config — the same `read()`, `write()`, and `list_files()` calls work across all supported backends (see the [Backends](#backends) section).

Three primitives do all the work:

- **`Store`** wraps a backend and exposes capability-gated methods. An operation is available only if the backend supports it. Capability is queried at runtime with `store.supports(Capability.X)`, so code can adapt gracefully instead of failing at import time.
- **`Backend`** is the storage-specific adapter. Built-in backends cover the common targets; a public ABC makes it straightforward to add new ones.
- **`Registry`** loads named backend and store definitions from config (TOML, YAML, dict, or Pydantic models), resolves `${ENV_VAR}` placeholders, and wraps credentials in `Secret` automatically.

______________________________________________________________________

## Store API

Methods are grouped by the capability that gates them. All methods share one invariant: they raise typed errors from `remote_store.errors` and never leak backend-native exceptions to the caller.

The table below is generated from the documentation graph (`graph.json`): each gating capability maps to the methods it gates. The descriptive per-capability subsections that follow add return types and behaviour. Quality-flag capabilities (`SEEKABLE_READ`, `LAZY_READ`, `ATOMIC_MOVE`, `WRITE_RESULT_NATIVE`, `USER_METADATA`) are not gates and so do not appear here — see [Capabilities](#capabilities).

### Methods by capability gate

| Capability     | Gated methods                                              |
| -------------- | ---------------------------------------------------------- |
| `ATOMIC_WRITE` | `open_atomic()`, `write_atomic()`                          |
| `COPY`         | `copy()`                                                   |
| `DELETE`       | `delete()`, `delete_folder()`                              |
| `GLOB`         | `glob()`                                                   |
| `LIST`         | `iter_children()`, `list_files()`, `list_folders()`        |
| `METADATA`     | `get_file_info()`, `get_folder_info()`\*, `head()`         |
| `MOVE`         | `move()`                                                   |
| `READ`         | `read()`, `read_bytes()`, `read_seekable()`, `read_text()` |
| `WRITE`        | `write()`, `write_text()`                                  |

\* `get_folder_info()` is additionally gated on `LIST` when called with `max_depth` (depth-limited traversal).

### Ungated (always available)

These methods carry no capability gate — they are available on every backend. The set is derived from the graph (Store method nodes with `gated: false`).

| Method                 | Returns          | Description                                               |
| ---------------------- | ---------------- | --------------------------------------------------------- |
| `exists(path)`         | `bool`           | Whether a file exists at the path                         |
| `is_file(path)`        | `bool`           | Whether the path resolves to a file (not a folder)        |
| `is_folder(path)`      | `bool`           | Whether the path resolves to a folder                     |
| `ping()`               | `None`           | Health check — raises `BackendUnavailable` if unreachable |
| `close()`              | `None`           | Release backend resources                                 |
| `child(subpath)`       | `Store`          | Scoped sub-store rooted at `subpath`                      |
| `unwrap(type_hint)`    | `T`              | Extract the underlying backend by type                    |
| `resolve(key)`         | `ResolutionPlan` | Resolution plan for a key (type, resolved path, options)  |
| `native_path(key)`     | `str`            | Backend-native path string for a store key                |
| `to_key(path)`         | `str`            | Convert a native path back to a store key                 |
| `supports(capability)` | `bool`           | Query whether a capability is active                      |

### READ

| Method                         | Returns    | Description                                                                                                                                                            |
| ------------------------------ | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `read(path)`                   | `BinaryIO` | Open a binary stream for reading; lazy where the backend supports `LAZY_READ`                                                                                          |
| `read_bytes(path)`             | `bytes`    | Read the entire file into memory                                                                                                                                       |
| `read_text(path, *, encoding)` | `str`      | Read the entire file as a decoded string                                                                                                                               |
| `read_seekable(path)`          | `BinaryIO` | Seekable binary stream; zero-copy where `read()` is already seekable (`SEEKABLE_READ` backends), a native HTTP-Range reader on sync Azure, else spooled to a temp file |

### WRITE

All write methods accept an optional `metadata=` mapping. If the backend declares `Capability.USER_METADATA`, the mapping is persisted alongside the file; otherwise a non-empty `metadata=` raises `CapabilityNotSupported`. All write methods return a `WriteResult` (see Data Models).

| Method                                                     | Returns       | Description                                              |
| ---------------------------------------------------------- | ------------- | -------------------------------------------------------- |
| `write(path, content, *, overwrite, metadata)`             | `WriteResult` | Create or overwrite a file from bytes or a binary stream |
| `write_text(path, text, *, encoding, overwrite, metadata)` | `WriteResult` | Write a string as a file                                 |

### ATOMIC_WRITE

Atomic writes use a temp-and-rename strategy: no reader ever sees a partial file, and a crash mid-write leaves the previous version intact.

| Method                                                | Returns                         | Description                                                  |
| ----------------------------------------------------- | ------------------------------- | ------------------------------------------------------------ |
| `write_atomic(path, content, *, overwrite, metadata)` | `WriteResult`                   | Atomic write from bytes or a stream                          |
| `open_atomic(path, *, overwrite, metadata)`           | context manager → `WriteResult` | Streaming atomic write; `WriteResult` returned on `__exit__` |

### DELETE

| Method                              | Returns | Description                                             |
| ----------------------------------- | ------- | ------------------------------------------------------- |
| `delete(path, *, missing_ok)`       | `None`  | Remove a file; `missing_ok=True` suppresses `NotFound`  |
| `delete_folder(path, *, recursive)` | `None`  | Remove a folder; requires `recursive=True` if non-empty |

### LIST

| Method                                      | Returns                             | Description                                                                                                                            |
| ------------------------------------------- | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `list_files(path, *, max_depth, pattern)`   | `Iterator[FileInfo]`                | Enumerate files under a prefix as `FileInfo` records (use `.path` for the path); optional `pattern=` filters basenames via `fnmatch`   |
| `list_folders(path, *, max_depth, pattern)` | `Iterator[FolderEntry]`             | Enumerate immediate subfolders as `FolderEntry` records (`.name`, `.path`); optional `pattern=` filters folder basenames via `fnmatch` |
| `iter_children(path)`                       | `Iterator[FileInfo \| FolderEntry]` | Iterate files and folders together (`FileInfo` for files, `FolderEntry` for folders)                                                   |

### GLOB

| Method          | Returns              | Description                                                                                 |
| --------------- | -------------------- | ------------------------------------------------------------------------------------------- |
| `glob(pattern)` | `Iterator[FileInfo]` | Native glob pattern matching (`*`, `**`, `?`), yielding matched files as `FileInfo` records |

Backends without native `GLOB` capability can use the `ext.glob` extension as a portable fallback.

### MOVE / COPY

| Method                         | Returns | Description                                |
| ------------------------------ | ------- | ------------------------------------------ |
| `move(src, dst, *, overwrite)` | `None`  | Rename or relocate within the same backend |
| `copy(src, dst, *, overwrite)` | `None`  | Duplicate a file within the same backend   |

### METADATA

| Method                  | Returns       | Description                                                             |
| ----------------------- | ------------- | ----------------------------------------------------------------------- |
| `head(path)`            | `WriteResult` | File metadata without reading content (size, etag, last_modified, etc.) |
| `get_file_info(path)`   | `FileInfo`    | Full file metadata record                                               |
| `get_folder_info(path)` | `FolderInfo`  | Folder metadata with aggregate file count and total size                |

______________________________________________________________________

## Data Models

All models are importable from `remote_store` and are frozen dataclasses.

### `WriteResult`

Returned by every write method and by `head()`.

| Field           | Type                    | Populated when                                                |
| --------------- | ----------------------- | ------------------------------------------------------------- |
| `path`          | `str`                   | Always                                                        |
| `size`          | `int`                   | Always                                                        |
| `source`        | `WriteSource`           | Always (`NativeSource`, `BasicSource`, or `SidecarSource`)    |
| `digest`        | `ContentDigest \| None` | Backend declares `WRITE_RESULT_NATIVE`                        |
| `etag`          | `str \| None`           | Backend declares `WRITE_RESULT_NATIVE`                        |
| `version_id`    | `str \| None`           | Backend declares `WRITE_RESULT_NATIVE` (S3 versioning, Azure) |
| `last_modified` | `datetime \| None`      | Backend declares `WRITE_RESULT_NATIVE`                        |
| `metadata`      | `dict[str, str]`        | Echo of caller's `metadata=` input                            |

`source` signals where the rich fields came from: `NativeSource` = populated from the backend's write response; `BasicSource` = populated from a post-write stat/head; `SidecarSource` = populated by the `ext.write` hash helper.

### `FileInfo`

Returned by `get_file_info()` and yielded by `iter_children()`.

| Field          | Type               |
| -------------- | ------------------ |
| `path`         | `str`              |
| `size`         | `int`              |
| `modified_at`  | `datetime \| None` |
| `etag`         | `str \| None`      |
| `version_id`   | `str \| None`      |
| `content_type` | `str \| None`      |
| `metadata`     | `dict[str, str]`   |

### Other models

| Model            | Description                                                                      |
| ---------------- | -------------------------------------------------------------------------------- |
| `FolderInfo`     | `path`, `file_count`, `total_size` — returned by `get_folder_info()`             |
| `FolderEntry`    | `path`, `name` — folder identity yielded by `list_folders()` / `iter_children()` |
| `ContentDigest`  | `algorithm` (e.g. `"crc32"`, `"sha256"`) + `value` (hex)                         |
| `ResolutionPlan` | Backend type, resolved path, and options — returned by `resolve()`               |
| `BackendConfig`  | `type` string + `options` dict — one entry in a `RegistryConfig`                 |
| `RegistryConfig` | Named backends and stores; the entry point for config-driven setup               |

______________________________________________________________________

## Capabilities

Capabilities are declared by backends at construction time. Two flavours exist:

- **Method gates**: the capability is a hard prerequisite for calling the method. `store.supports()` returns `False` and the method raises `CapabilityNotSupported` if called without it.
- **Quality flags**: the capability signals that a method delivers a stronger guarantee. The method is available regardless; the flag lets callers decide whether to rely on the native behaviour or fall back to an extension.

| Capability            | Flavour      | Gated / signalled methods                                  | Notes                                                                                                                                                                                                                                    |
| --------------------- | ------------ | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `READ`                | Gate         | `read()`, `read_bytes()`, `read_text()`, `read_seekable()` | Declared by all built-in backends                                                                                                                                                                                                        |
| `WRITE`               | Gate         | `write()`, `write_text()`                                  | Declared by all built-in backends                                                                                                                                                                                                        |
| `DELETE`              | Gate         | `delete()`, `delete_folder()`                              | Declared by all built-in backends                                                                                                                                                                                                        |
| `LIST`                | Gate         | `list_files()`, `list_folders()`, `iter_children()`        | Declared by all built-in backends                                                                                                                                                                                                        |
| `GLOB`                | Gate         | `glob()`                                                   | Not declared by `memory`; use `ext.glob` as fallback                                                                                                                                                                                     |
| `MOVE`                | Gate         | `move()`                                                   | Declared by all built-in backends                                                                                                                                                                                                        |
| `COPY`                | Gate         | `copy()`                                                   | Declared by all built-in backends                                                                                                                                                                                                        |
| `ATOMIC_WRITE`        | Gate         | `write_atomic()`, `open_atomic()`                          | Declared by all built-in backends                                                                                                                                                                                                        |
| `METADATA`            | Gate         | `head()`, `get_file_info()`, `get_folder_info()`           | Declared by all built-in backends                                                                                                                                                                                                        |
| `SEEKABLE_READ`       | Quality flag | `read()`                                                   | `read()` returns a natively seekable stream. Absence means only that `read()` is forward-only — the sync `Store.read_seekable()` still serves it (native range read on sync Azure, else spooled); the async API has no `read_seekable()` |
| `LAZY_READ`           | Quality flag | `read()`                                                   | `read()` fetches data lazily; partial reads avoid loading the full file                                                                                                                                                                  |
| `ATOMIC_MOVE`         | Quality flag | `move()`                                                   | `move()` is crash-safe under concurrent access                                                                                                                                                                                           |
| `WRITE_RESULT_NATIVE` | Quality flag | `write*()`, `head()`                                       | Rich `WriteResult` fields (`etag`, `digest`, `version_id`, `last_modified`) come from the backend's own write response                                                                                                                   |
| `USER_METADATA`       | Strict gate  | `metadata=` kwarg on write methods                         | A non-empty `metadata=` raises `CapabilityNotSupported` when unset                                                                                                                                                                       |

______________________________________________________________________

## Backends

Built-in backends cover the most common targets. All implement the same `Backend` ABC; `SFTPUtils`, `S3PyArrowBackend`, and the SQL backends expose additional backend-specific options via `BackendConfig.options`.

| Type         | Class                 | Extra                                   | Capabilities                                        |
| ------------ | --------------------- | --------------------------------------- | --------------------------------------------------- |
| `azure`      | `AzureBackend`        | `remote-store[azure]`                   | All except `ATOMIC_MOVE`, `SEEKABLE_READ`           |
| `http`       | `ReadOnlyHttpBackend` | — (stdlib; `requests`/`httpx` optional) | `LAZY_READ`, `METADATA`, `READ`                     |
| `local`      | `LocalBackend`        | —                                       | All                                                 |
| `memory`     | `MemoryBackend`       | —                                       | All except `GLOB`, `LAZY_READ`                      |
| `s3`         | `S3Backend`           | `remote-store[s3]`                      | All except `ATOMIC_MOVE`                            |
| `s3-pyarrow` | `S3PyArrowBackend`    | `remote-store[s3-pyarrow]`              | All except `ATOMIC_MOVE`                            |
| `sftp`       | `SFTPBackend`         | `remote-store[sftp]`                    | All except `ATOMIC_MOVE`, `GLOB`                    |
| `sql-blob`   | `SQLBlobBackend`      | `remote-store[sql]`                     | All except `LAZY_READ`                              |
| `sql-query`  | `SQLQueryBackend`     | `remote-store[sql-query]`               | `GLOB`, `LIST`, `METADATA`, `READ`, `SEEKABLE_READ` |

The `SEEKABLE_READ` exclusions above concern `read()` only — `read_seekable()` remains available on the sync `Store`, and on Azure it is a native HTTP-Range reader (no temp-file spill), not a spool. See [Streaming and seekable reads](https://docs.remotestore.dev/stable/guides/backends/azure/#streaming-and-seekable-reads).

**Write-result quality flags by backend:**

| Backend      | `WRITE_RESULT_NATIVE`               | `USER_METADATA`                       |
| ------------ | ----------------------------------- | ------------------------------------- |
| `azure`      | Yes                                 | Yes                                   |
| `http`       | —                                   | —                                     |
| `local`      | Yes                                 | —                                     |
| `memory`     | Yes                                 | Yes                                   |
| `s3`         | Yes                                 | Yes                                   |
| `s3-pyarrow` | Yes                                 | —                                     |
| `sftp`       | Yes                                 | —                                     |
| `sql-blob`   | Yes (requires `modified_at` column) | Yes (requires `user_metadata` column) |
| `sql-query`  | —                                   | —                                     |

**Write and move atomicity by backend** — whether each mutating operation completes atomically or via a non-atomic mechanism that can leave partial state on failure. `read`, `list`, and `metadata` are non-mutating (atomicity N/A); `delete` and folder operations carry no atomicity guarantee and are omitted.

| Backend      | `write`       | `write_atomic` | `move`        | `copy`        |
| ------------ | ------------- | -------------- | ------------- | ------------- |
| `azure`      | Atomic§       | Atomic         | Copy+delete†  | Copy+delete   |
| `http`       | — (read-only) | — (read-only)  | — (read-only) | — (read-only) |
| `local`      | Direct        | Atomic         | Atomic\*      | Copy+delete   |
| `memory`     | Atomic        | Atomic         | Atomic        | Atomic        |
| `s3`         | Atomic        | Atomic         | Copy+delete   | Copy+delete   |
| `s3-pyarrow` | Streamed‡     | Atomic         | Copy+delete   | Copy+delete   |
| `sftp`       | Streamed      | Atomic         | Copy+delete†  | Copy+delete   |
| `sql-blob`   | Atomic        | Atomic         | Atomic        | Atomic        |
| `sql-query`  | — (read-only) | — (read-only)  | — (read-only) | — (read-only) |

\* `local` `move` is atomic within one filesystem (`os.rename`); a cross-filesystem move falls back to copy-then-delete. † Azure and SFTP `move` use a native rename that is atomic (Azure HNS `rename_file`, SFTP `posix_rename`), but `ATOMIC_MOVE` is not advertised because it cannot be guaranteed across all configurations (non-HNS Azure accounts, non-POSIX SFTP servers). ‡ `s3-pyarrow` plain `write` streams straight to a multipart upload; PyArrow's stream exposes no abort, so a mid-stream failure finalises a *truncated* object. `write_atomic` buffers the body first, so a failure leaves no object. § `azure` `write` commits atomically on flat (non-HNS) accounts; on hierarchical-namespace accounts use `write_atomic` for a guaranteed atomic replace.

**Read-after-write consistency by backend** — whether a read or listing issued after a write, overwrite, or delete reflects that change. remote-store normalises to **strong** read-after-write on every read/write backend: the object you just wrote or deleted is immediately visible to a subsequent `read`, `head`, or `list_files` on the same store, with no eventual-consistency window to code around. This is a per-caller *visibility* guarantee, not mutual exclusion between *simultaneous* writers — `overwrite=False` remains a TOCTOU convenience guard, and concurrent writers to one key resolve last-writer-wins. See [Concurrency and atomicity](https://docs.remotestore.dev/stable/explanation/concurrency/) for the ordering and race semantics.

| Backend      | Read-after-write | Listing consistency |
| ------------ | ---------------- | ------------------- |
| `azure`      | Strong           | Strong              |
| `http`       | — (read-only)    | — (read-only)       |
| `local`      | Strong           | Strong              |
| `memory`     | Strong           | Strong              |
| `s3`         | Strong           | Strong\*            |
| `s3-pyarrow` | Strong           | Strong\*            |
| `sftp`       | Strong           | Strong              |
| `sql-blob`   | Strong           | Strong              |
| `sql-query`  | — (read-only)    | — (read-only)       |

The async-native backends inherit their sync peer's consistency; the async-only `GraphBackend` is the one distinct case.

| Async backend        | Read-after-write | Listing consistency |
| -------------------- | ---------------- | ------------------- |
| `AsyncAzureBackend`  | Strong           | Strong              |
| `AsyncMemoryBackend` | Strong           | Strong              |
| `GraphBackend`       | Strong†          | Strong†             |

\* `s3` / `s3-pyarrow` listings are strongly consistent by default: the backend leaves the s3fs directory cache **off** (`use_listings_cache=False`), so a listing taken after a write reflects it. Opting into `client_options['use_listings_cache']` trades this for a cache that never expires — a listing can then stay blind to a cross-writer change until the backend is rebuilt. † `GraphBackend` read-your-writes holds on one instance (a write is committed to two datacentre regions before it is acknowledged). `copy` (always) and a large or cross-folder `move` (sometimes) run server-side and are polled to completion before the call returns, so a read or listing afterwards reflects the result.

**Per-operation cost by backend** — the *structural* cost each backend forces per call: whether `read` streams (constant memory) or materializes the whole object, and what a `metadata` probe and a `list` cost per invocation. This is the cost the API shape dictates, distinct from measured latency — see [Performance](https://docs.remotestore.dev/stable/explanation/performance/) for benchmarked overhead. Write-path cost (streaming vs full-buffer, atomic vs copy+delete) is in the atomicity table above; `list` cost scales with the number of entries returned, and enabling hierarchical-write safety (`reject_write_under_file_ancestor`, or the native HNS / SFTP / Graph parent checks) adds one metadata probe per path ancestor on the guarded write path.

| Backend      | `read`                  | `metadata`          | `list`           |
| ------------ | ----------------------- | ------------------- | ---------------- |
| `azure`      | Streaming               | 1 HEAD              | Paginated `LIST` |
| `http`       | Streaming               | 1 HEAD              | — (no `LIST`)    |
| `local`      | Streaming               | `stat` syscall      | `scandir` walk   |
| `memory`     | Buffered in memory\*    | Dict lookup         | Dict scan        |
| `s3`         | Streaming               | 1 HEAD              | Paginated `LIST` |
| `s3-pyarrow` | Streaming               | 1 HEAD              | Paginated `LIST` |
| `sftp`       | Streaming               | 1 `stat` round-trip | Directory walk   |
| `sql-blob`   | Full BLOB into memory\* | 1 `SELECT`          | 1 `SELECT`       |
| `sql-query`  | Query run, buffered\*   | Registry lookup     | Registry scan    |

The async-native backends inherit their sync peer's per-op cost; the async-only `GraphBackend` streams reads but blocks `copy` (always) and a large or cross-folder `move` on a server-side monitor polled to completion — a per-call latency the columns below do not show.

| Async backend        | `read`    | `metadata`  | `list`           |
| -------------------- | --------- | ----------- | ---------------- |
| `AsyncAzureBackend`  | Streaming | 1 HEAD      | Paginated `LIST` |
| `AsyncMemoryBackend` | Streaming | Dict lookup | Dict scan        |
| `GraphBackend`       | Streaming | 1 GET       | Paginated GETs   |

\* `memory` (sync), `sql-blob`, and `sql-query` do not stream a read (`LAZY_READ` absent): each holds the whole object in memory before `read()` returns — `sql-blob` loads the full BLOB, `sql-query` buffers the serialised query result, and sync `memory` keeps the value resident (its async peer `AsyncMemoryBackend` yields chunks and *does* stream). `sql-blob` likewise buffers the entire body before the write `INSERT`/`UPDATE`. For objects larger than process memory use a streaming backend (Local, S3, Azure, SFTP).

**Native async backends** — constructed directly via `AsyncStore(backend=…)`; no RegistryConfig `type=` string (there is no async config registry).

| Class                | Extra                 | Capabilities                                      |
| -------------------- | --------------------- | ------------------------------------------------- |
| `AsyncAzureBackend`  | `remote-store[azure]` | All except `ATOMIC_MOVE`, `SEEKABLE_READ`         |
| `AsyncMemoryBackend` | —                     | All except `GLOB`                                 |
| `GraphBackend`       | `remote-store[graph]` | All except `ATOMIC_MOVE`, `GLOB`, `SEEKABLE_READ` |

The async API has no `read_seekable()`: an async-native backend that omits `SEEKABLE_READ` (`AsyncAzureBackend`, `GraphBackend`) has no seekable read until bridged to sync via `AsyncBackendSyncAdapter`, which spools.

**Write-result quality flags by native async backend:**

| Class                | `WRITE_RESULT_NATIVE` | `USER_METADATA` |
| -------------------- | --------------------- | --------------- |
| `AsyncAzureBackend`  | Yes                   | Yes             |
| `AsyncMemoryBackend` | Yes                   | Yes             |
| `GraphBackend`       | Yes                   | —               |

______________________________________________________________________

## Configuration

`RegistryConfig` decouples storage topology from application code. Define named backends and stores in a config file; application code calls `registry.store("name")` and never sees connection strings. Credentials in standard-looking keys (`password`, `secret_key`, etc.) are wrapped in `Secret` automatically and masked in logs.

```
[backends.primary]
type = "s3"
options.bucket = "my-bucket"
options.access_key = "${AWS_ACCESS_KEY_ID}"
options.secret_key = "${AWS_SECRET_ACCESS_KEY}"

[stores.data]
backend = "primary"
root = "data/"
```

| API                                                          | Location                    | Description                                                        |
| ------------------------------------------------------------ | --------------------------- | ------------------------------------------------------------------ |
| `RegistryConfig.from_dict(data)`                             | `remote_store`              | Construct from a plain dict                                        |
| `RegistryConfig.from_toml(path, *, table, resolve_env_vars)` | `remote_store`              | Load from a TOML file                                              |
| `from_yaml(path, *, resolve_env_vars)`                       | `remote_store.ext.yaml`     | Load from a YAML file                                              |
| `from_pydantic(model)`                                       | `remote_store.ext.pydantic` | Convert a Pydantic settings model                                  |
| `resolve_env(data, *, environ)`                              | `remote_store`              | Resolve `${VAR}` / `${VAR:-default}` placeholders in a config dict |

______________________________________________________________________

## Extensions

Extensions are composable layers on top of `Store` or utility helpers. They use only the public `Store`/`Backend` API and carry no lifecycle ownership — they never call `store.close()`.

### Always available (base install)

| Extension     | Problem it solves                                                                             | Module                       | Key exports                                                                                  |
| ------------- | --------------------------------------------------------------------------------------------- | ---------------------------- | -------------------------------------------------------------------------------------------- |
| **Batch**     | Apply copy/delete/exists across many paths in one call with partial-failure reporting         | `remote_store.ext.batch`     | `BatchResult`, `batch_copy`, `batch_delete`, `batch_exists`                                  |
| **Cache**     | Transparent read-through caching — subsequent reads of the same key go to a fast local store  | `remote_store.ext.cache`     | `CachedStore`, `CacheBackend`, `CacheStats`, `MemoryCache`, `cache`                          |
| **Glob**      | Portable glob for backends that do not declare `Capability.GLOB`                              | `remote_store.ext.glob`      | `glob_files`                                                                                 |
| **Integrity** | Client-side content checksums and verification independent of backend digest support          | `remote_store.ext.integrity` | `checksum`, `verify`, `verify_hex`, `content_digest`                                         |
| **Observe**   | Structured event hooks for every store operation — logging, metrics, tracing, audit trails    | `remote_store.ext.observe`   | `ObservedStore`, `StoreEvent`, `BufferedObserver`, `observe`, `set_correlation_id`           |
| **Partition** | Parse and construct Hive-style partition paths (`key=value/…`)                                | `remote_store.ext.partition` | `ParsedPartition`, `parse_partition`, `partition_path`                                       |
| **Streams**   | Wrap any stream with progress callbacks or rolling checksums without buffering                | `remote_store.ext.streams`   | `ChecksumReader`, `ChecksumWriter`, `ProgressReader`, `ProgressWriter`, `read_with_progress` |
| **Transfer**  | High-level upload / download / store-to-store copy with streaming and progress                | `remote_store.ext.transfer`  | `upload`, `download`, `transfer`                                                             |
| **Write**     | Guaranteed client-side digest on write regardless of backend capability; atomic write variant | `remote_store.ext.write`     | `write_with_hash`, `open_atomic_with_hash`                                                   |

### Optional (require extras)

| Extension         | Problem it solves                                                                       | Module                      | Key exports                                                                                          | Extra                    |
| ----------------- | --------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------ |
| **Arrow**         | Use any `Store` as a PyArrow filesystem for Arrow / Parquet tooling                     | `remote_store.ext.arrow`    | `StoreFileSystemHandler`, `pyarrow_fs`                                                               | `remote-store[arrow]`    |
| **Parquet**       | Read and write Parquet datasets (single files or partitioned) via PyArrow               | `remote_store.ext.parquet`  | `ParquetDatasetStore`, `DatasetManifest`                                                             | `remote-store[arrow]`    |
| **OpenTelemetry** | Emit distributed tracing spans for every store operation                                | `remote_store.ext.otel`     | `otel_hooks`, `otel_observe`                                                                         | `remote-store[otel]`     |
| **Pydantic**      | Derive `RegistryConfig` from a Pydantic settings model                                  | `remote_store.ext.pydantic` | `from_pydantic`                                                                                      | `remote-store[pydantic]` |
| **YAML**          | Load `RegistryConfig` from a YAML file                                                  | `remote_store.ext.yaml`     | `from_yaml`                                                                                          | `remote-store[yaml]`     |
| **Dagster**       | IO manager, config-driven Store resource, and compute log manager for Dagster pipelines | `remote_store.ext.dagster`  | `RemoteStoreIOManager`, `DagsterStoreResource`, `RemoteStoreComputeLogManager`, `dagster_io_manager` | `remote-store[dagster]`  |

______________________________________________________________________

## Error Model

All errors are subclasses of `RemoteStoreError` (importable from `remote_store`). Backend-native exceptions are mapped at the adapter boundary — callers always receive a typed error, never an `S3ServiceError` or `azure.core.…`.

| Error                    | Raised when                                                                                                  |
| ------------------------ | ------------------------------------------------------------------------------------------------------------ |
| `NotFound`               | File or folder does not exist                                                                                |
| `AlreadyExists`          | Target path already exists and `overwrite=False`                                                             |
| `InvalidPath`            | Path is malformed or points at the wrong node type (e.g. a directory where a file is expected)               |
| `CapabilityNotSupported` | A method is called without the required capability                                                           |
| `BackendUnavailable`     | The backend is unreachable (network, auth, service down)                                                     |
| `PermissionDenied`       | Caller lacks access rights                                                                                   |
| `DirectoryNotEmpty`      | Directory is not empty and the operation requires it to be                                                   |
| `ResourceLocked`         | Target resource is held by another session (e.g. an open co-authoring session); maps from Graph `423 Locked` |
| `RemoteStoreError`       | Base class for all errors above                                                                              |

**Retryable vs. terminal** — the HTTP statuses the HTTP-transport backends (`graph`, `http`) classify, and the typed error each surfaces as. Retried statuses are re-attempted under the backend's `RetryPolicy` (default 3 attempts, 1–60 s exponential backoff) and honour `Retry-After`; the typed error is raised only once the attempt budget is exhausted. The other backends reach the *same typed-error vocabulary* by mapping native SDK/OS exceptions rather than HTTP status codes, so the outcome is shared — but the status classification and `Retry-After` handling shown here are specific to the HTTP transports (`s3` and `azure` honour only `max_attempts`; `local`, `memory`, and `sql-*` make no remote calls and do not retry).

| Status | Disposition                     | Surfaced as          |
| ------ | ------------------------------- | -------------------- |
| `429`  | Retried — honours `Retry-After` | `BackendUnavailable` |
| `500`  | Retried                         | `BackendUnavailable` |
| `502`  | Retried                         | `BackendUnavailable` |
| `503`  | Retried                         | `BackendUnavailable` |
| `504`  | Retried                         | `BackendUnavailable` |
| `403`  | Not retried                     | `PermissionDenied`   |
| `404`  | Not retried                     | `NotFound`           |
| `409`  | Not retried                     | `AlreadyExists`      |
| `423`  | Not retried                     | `ResourceLocked`     |
| `507`  | Not retried                     | `BackendUnavailable` |

| Backend      | Transport retry mechanism                                       |
| ------------ | --------------------------------------------------------------- |
| `azure`      | Azure SDK `ExponentialRetry` (all five `RetryPolicy` fields)    |
| `http`       | Hand-rolled loop over the shared backoff helpers†               |
| `local`      | — (no `retry` parameter)                                        |
| `memory`     | — (no `retry` parameter)                                        |
| `s3`         | botocore `standard` mode — honours `max_attempts` only          |
| `s3-pyarrow` | `AwsStandardS3RetryStrategy` — honours `max_attempts` only      |
| `sftp`       | `tenacity` — connection-scope only (reconnect, not per-request) |
| `sql-blob`   | — (errors mapped, not retried)                                  |
| `sql-query`  | — (errors mapped, not retried)                                  |

† `http` additionally retries `408 Request Timeout` (classified as `BackendUnavailable`) — a transport-local extension of the shared retryable set above.

Native async backends inherit their sync peer's mechanism; the async-only `GraphBackend` runs hand-rolled retry loops over the shared backoff helpers, honouring all five `RetryPolicy` fields. `local`, `memory`, `sftp`, and the SQL backends leave a closed backend reusable; `azure`, `s3`, and `graph` treat use after `close()` as terminal (`close_is_terminal`).

______________________________________________________________________

## Async API

`remote_store.aio` provides native `async`/`await` support and two bridge adapters for mixing sync and async backends.

**Native async classes:**

| Class                  | Description                                                                                                                  |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `AsyncStore`           | Async counterpart to `Store`; coroutine methods for all operations; `write*()` returns `WriteResult` and accepts `metadata=` |
| `AsyncBackend`         | ABC for native async backends                                                                                                |
| `AsyncMemoryBackend`   | In-memory async backend (for testing)                                                                                        |
| `AsyncAzureBackend`    | Native async Azure backend via Azure SDK async clients                                                                       |
| `GraphBackend`         | Native async Microsoft Graph backend (OneDrive / SharePoint / Teams) via httpx + msal; companions `GraphAuth`, `GraphUtils`  |
| `AsyncWritableContent` | Type alias: `bytes \| AsyncIterator[bytes]`                                                                                  |

**Sync ↔ async backend equivalence** — generated from the graph's `mirrors` edges. The capability delta names capabilities one side declares that its peer does not. Async-only backends (`GraphBackend`) have no sync mirror and are listed above only.

| Sync backend    | Async backend        | Capability delta       |
| --------------- | -------------------- | ---------------------- |
| `AzureBackend`  | `AsyncAzureBackend`  | —                      |
| `MemoryBackend` | `AsyncMemoryBackend` | async adds `LAZY_READ` |

**Bridge adapters** — when you need to cross the sync/async boundary:

| Class                     | Direction                                  | Mechanism                                                           |
| ------------------------- | ------------------------------------------ | ------------------------------------------------------------------- |
| `SyncBackendAdapter`      | sync `Backend` → usable in async code      | Dispatches each call via `asyncio.to_thread` (thread-pool executor) |
| `AsyncBackendSyncAdapter` | async `AsyncBackend` → usable in sync code | Runs the async backend on a dedicated daemon-thread event loop      |

Both adapters translate capabilities faithfully and are covered by the same conformance suite as native backends.

**Async extensions:**

| Module          | Description                                                               |
| --------------- | ------------------------------------------------------------------------- |
| `aio.ext.write` | `write_with_hash` — client-side SHA-256 checksumming on async write paths |

______________________________________________________________________

## Install extras

```
pip install remote-store[arrow]       # PyArrow filesystem bridge + Parquet extension
pip install remote-store[azure]       # Azure ADLS Gen2 via Azure SDK
pip install remote-store[dagster]     # Dagster IO manager
pip install remote-store[graph]       # Microsoft Graph (OneDrive / SharePoint / Teams) via httpx + msal
pip install remote-store[httpx]       # httpx HTTP adapter for ReadOnlyHttpBackend
pip install remote-store[otel]        # OpenTelemetry distributed tracing
pip install remote-store[pydantic]    # Pydantic settings integration
pip install remote-store[requests]    # requests HTTP adapter for ReadOnlyHttpBackend
pip install remote-store[s3]          # S3 via s3fs
pip install remote-store[s3-pyarrow]  # S3 via PyArrow C++ filesystem
pip install remote-store[sftp]        # SFTP via paramiko
pip install remote-store[sql]         # SQL blob store via SQLAlchemy
pip install remote-store[sql-query]   # SQL query store via SQLAlchemy + PyArrow
pip install remote-store[toml]        # TOML config (stdlib on Python 3.11+)
pip install remote-store[yaml]        # YAML config loading
```

Each extra declares a floor in `pyproject.toml` and in most cases deliberately no ceiling (the `arrow` and `sql-query` extras carry a `pyarrow<25` ceiling). For the exact upper-bound versions CI was last green against, see [Tested upper-bound versions](https://docs.remotestore.dev/stable/reference/tested-versions/).
# Explanation

# Architecture Overview

This page explains *why* `remote-store` is structured the way it is. For API details, see the [API Reference](https://docs.remotestore.dev/stable/reference/api/index.md). For how-to instructions, see the Guides section in the sidebar.

## Three-layer design

```
+-----------+     +-----------+     +-----------+
|   Store   |---->|  Registry |---->|  Backend  |
| (user API)|     | (lifecycle|     | (storage  |
|           |     |  manager) |     |  adapter) |
+-----------+     +-----------+     +-----------+
      |                                   |
      v                                   v
  Relative paths                   Native paths
  (logical keys)                   (bucket/prefix)
```

### Store

The primary user-facing abstraction. A Store is a **logical remote folder** scoped to a root path. All operations use relative paths within that scope.

Stores are **immutable and thread-safe** — cheap to create, safe to share across threads. A Store does not own its backend; multiple stores can share one backend instance.

`Store.child(subpath)` creates a sub-scoped store without additional backend connections.

### Registry

The lifecycle manager. It:

- Loads and validates configuration (`RegistryConfig`)
- Lazily creates backend instances (one per backend config, shared across stores)
- Manages cleanup via `close()`

The Registry is a **passive resource** — it does not spawn threads or async tasks, making it compatible with any concurrency model.

### Backend

The storage adapter layer. Each backend encapsulates all provider-specific behavior:

- Path normalization and validation
- Native I/O operations (streaming-first)
- Error mapping (native exceptions to `remote-store` error types)
- Capability declaration

Backends are an **implementation detail** — user code never interacts with backends directly (except via `Store.unwrap()` for escape-hatch access).

## Error hierarchy

All backend-specific exceptions are normalized into a small set of errors. Backend exceptions **never leak** to user code.

```
RemoteStoreError
  +-- NotFound
  +-- AlreadyExists
  +-- PermissionDenied
  +-- InvalidPath
  +-- CapabilityNotSupported
  +-- DirectoryNotEmpty
  +-- BackendUnavailable
```

Each error carries structured attributes: `message`, `path`, and `backend`.

## Extension model

Extensions in `ext.*` add functionality without modifying the core — covering observability, caching, glob pattern matching, PyArrow integration, and more. Some extensions are always available (no extra dependencies); others require optional extras such as `arrow`, `otel`, or `dagster`.

See the [Extensions guide](https://docs.remotestore.dev/stable/guides/extensions/index.md) for the full list and installation instructions.

Extensions follow the [extension architecture contract](https://docs.remotestore.dev/stable/explanation/design/adrs/0008-extension-architecture/index.md): no singleton state, no global registries, composable with the core Store API.

## Capability system

Backends declare supported operations via the `Capability` enum. This enables:

- **Compile-time clarity** — users know what to expect from each backend
- **Runtime guards** — operations fail fast with `CapabilityNotSupported` instead of mysterious backend errors
- **Portable fallbacks** — extensions like `ext.glob` provide software implementations for backends that lack native support

See the [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) for the full table.

## Configuration philosophy

Configuration follows [strict resolution rules](https://docs.remotestore.dev/stable/explanation/design/adrs/0002-config-resolution-no-merge/index.md):

1. Config-as-code has absolute priority (TOML, YAML, or dict)
1. No merging, no overrides, no magic environment variable layering
1. Deterministic and test-safe — same config always produces same behavior

## Design decisions

Detailed rationale lives in the ADRs under [`sdd/adrs/`](https://docs.remotestore.dev/stable/explanation/design/adrs/index.md).

## See also

- [Security Model](https://docs.remotestore.dev/stable/explanation/security-model/index.md) — credential handling and trust boundaries
- [Performance](https://docs.remotestore.dev/stable/explanation/performance/index.md) — benchmark data and optimization guidance
- [Concurrency](https://docs.remotestore.dev/stable/explanation/concurrency/index.md) — thread safety and atomicity guarantees

# Concurrency and Atomicity Guarantees

remote-store wraps multiple storage backends behind a single API, but each backend inherits the concurrency and atomicity characteristics of its underlying platform. This guide covers three things: whether one backend instance is safe to **share across concurrent callers** (the concurrent-use posture), and two per-operation limitations — non-atomic `move()` and the `overwrite=False` race window.

## Concurrent-use posture

Before the per-operation guarantees below, there is a more basic question: **is one backend instance safe to share across threads** (or, for async backends, across concurrent coroutines on one event loop)? Most backends are; a few wrap a native client that is not, and must be used one-instance-per-thread instead.

| Backend                                                                               | Posture                       | Why — and the remedy where it applies                                                                                                                                                                                                                                                        |
| ------------------------------------------------------------------------------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Local](https://docs.remotestore.dev/stable/guides/backends/local/index.md)           | Thread-safe                   | Stateless; delegates to the OS filesystem.                                                                                                                                                                                                                                                   |
| [Memory](https://docs.remotestore.dev/stable/guides/backends/memory/index.md)         | Thread-safe                   | Guarded by a single internal lock.                                                                                                                                                                                                                                                           |
| [HTTP](https://docs.remotestore.dev/stable/guides/backends/http/index.md)             | Single-connection on `urllib` | Only the auto-detect *fallback* `urllib` opener (shared redirect counter) is unsafe; the `httpx` / `requests` transports — auto-selected ahead of it when installed — are thread-safe. **Remedy:** install and select `http_client='requests'` or `'httpx'`, or use one instance per thread. |
| [S3](https://docs.remotestore.dev/stable/guides/backends/s3/index.md)                 | Thread-safe                   | The boto3 client / s3fs is safe for concurrent per-instance use.                                                                                                                                                                                                                             |
| [S3-PyArrow](https://docs.remotestore.dev/stable/guides/backends/s3-pyarrow/index.md) | Thread-safe ¹                 | Arrow's C++ `S3FileSystem` is safe per instance.                                                                                                                                                                                                                                             |
| [SFTP](https://docs.remotestore.dev/stable/guides/backends/sftp/index.md)             | Single-connection             | One paramiko channel over one socket. **Remedy:** one instance per thread (or a native async SFTP client).                                                                                                                                                                                   |
| [Azure](https://docs.remotestore.dev/stable/guides/backends/azure/index.md)           | Thread-safe                   | The Azure SDK service clients are immutable once built. The async twin (`AsyncAzureBackend`) is safe for concurrent coroutines on a **single** event loop, and never across loops — use one instance per loop.                                                                               |
| [Graph](https://docs.remotestore.dev/stable/guides/backends/graph/index.md)           | Thread-safe ²                 | Async-only: safe for concurrent coroutines on one event loop.                                                                                                                                                                                                                                |
| [SQLBlob](https://docs.remotestore.dev/stable/guides/backends/sql-blob/index.md)      | Thread-safe ³                 | The SQLAlchemy engine pools connections.                                                                                                                                                                                                                                                     |
| [SQLQuery](https://docs.remotestore.dev/stable/guides/backends/sql-query/index.md)    | Thread-safe ³                 | Same pooled engine (read-only).                                                                                                                                                                                                                                                              |

No backend offers multi-operation transactionality — atomicity is per operation only, and ordering between concurrent callers is never guaranteed.

¹ S3-PyArrow's per-instance thread-safety is expected but not yet pinned by a live concurrency probe; treat heavy concurrent sharing as to-be-confirmed.

² One Graph instance is safe for concurrent coroutines on a **single** event loop, and never across loops — use one instance per loop. Driven from synchronous code through the async→sync bridge it is also safe for concurrent threads (unlike SFTP); see [Bridge asymmetry](#bridge-asymmetry) below.

³ On a pooled RDBMS engine (PostgreSQL, MySQL) SQLBlob and SQLQuery are thread-safe. The `sqlite:///:memory:` configuration is the exception: SQLAlchemy gives each thread its own isolated in-memory database, so a shared instance behaves as single-connection — use one instance per thread.

### Bridge asymmetry

remote-store has two adapters that cross the sync/async boundary, and they do **not** confer the same safety — the direction decides it:

- **Async backend → sync callers** ([`AsyncBackendSyncAdapter`](https://docs.remotestore.dev/stable/guides/async-sync-bridges/index.md)): owns one private event loop in a background thread and serialises every concurrent sync caller onto it. Wrapping a loop-safe async backend (e.g. Graph) this way *manufactures* a thread-safe sync backend — concurrent threads are safe.
- **Sync backend → async callers** (`SyncBackendAdapter`, used by `AsyncStore` when you pass it a sync backend): dispatches each call to a thread pool via `asyncio.to_thread` and adds no serialisation of its own. It is safe **only if the wrapped sync backend is thread-safe**. Wrapping a single-connection backend (SFTP, or HTTP on `urllib`) and driving it with `asyncio.gather` is **unsafe** — the concurrent workers touch the same non-thread-safe client. Use one instance per worker, or an external lock.

In short: async→sync *manufactures* thread-safety by funnelling through one loop; sync→async *borrows* the backend's thread-safety, or its absence.

## `overwrite=False` and TOCTOU

When you call `store.write(path, data, overwrite=False)`, the backend checks whether the file exists and then writes it. These are two separate operations — a classic **Time-Of-Check-to-Time-Of-Use (TOCTOU)** race window:

```
Thread A: exists("report.csv") -> False
Thread B: exists("report.csv") -> False    # concurrent check
Thread A: write("report.csv", data_a)      # succeeds
Thread B: write("report.csv", data_b)      # also succeeds — overwrites A's file
```

This affects **most backends**. The check-then-act pattern cannot be made race-free without an external coordination mechanism. `overwrite=False` is a convenience guard against accidental overwrites in single-writer scenarios — it is **not** a mutual exclusion mechanism.

The exception is **[Graph](https://docs.remotestore.dev/stable/guides/backends/graph/index.md)**: its `overwrite=False` maps to a server-side atomic *create-if-absent*, so two writers racing to create the same new key cannot both win — the loser receives `AlreadyExists`, with no client-side check-then-write window. S3 and Azure expose the same primitive natively (conditional PUT / `If-None-Match`), but remote-store does not wire it into `overwrite=False`; only Graph maps to it today.

### Mitigations

- **External locking.** Use a distributed lock (e.g., DynamoDB lock table for S3, blob lease for Azure, advisory locks for SFTP/local) to serialize writers.
- **Idempotent writes.** Design file names to be unique (e.g., include UUIDs or content hashes) so concurrent writers never target the same path.
- **Backend-native conditional writes.** Some platforms offer conditional put (e.g., S3 `If-None-Match`, Azure `If-None-Match` ETag conditions). These are not exposed through remote-store's API, but you can use `backend.unwrap()` to access the native client.

## Non-atomic `move()`

Several backends implement `move(src, dst)` as a **copy followed by a delete**. If the process crashes between the two steps, both the source and destination will exist (data duplication, not data loss).

| Backend                                                                                    | `move()` implementation                                                | Atomic?                                        |
| ------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------- | ---------------------------------------------- |
| [Local](https://docs.remotestore.dev/stable/guides/backends/local/index.md)                | `shutil.move()` (`os.rename()` on same filesystem, copy+delete across) | Yes\*                                          |
| [S3](https://docs.remotestore.dev/stable/guides/backends/s3/index.md)                      | Copy object + delete object                                            | —                                              |
| [S3-PyArrow](https://docs.remotestore.dev/stable/guides/backends/s3-pyarrow/index.md)      | Copy object + delete object                                            | —                                              |
| [Azure](https://docs.remotestore.dev/stable/guides/backends/azure/index.md) (HNS)          | `rename_file()`                                                        | Yes                                            |
| [Azure](https://docs.remotestore.dev/stable/guides/backends/azure/index.md) (non-HNS)      | Copy blob + delete blob                                                | —                                              |
| [Graph](https://docs.remotestore.dev/stable/guides/backends/graph/index.md)                | Server-side move / copy (monitor-polled)                               | —                                              |
| [SFTP](https://docs.remotestore.dev/stable/guides/backends/sftp/index.md) (`posix_rename`) | `posix_rename`                                                         | Yes                                            |
| [SFTP](https://docs.remotestore.dev/stable/guides/backends/sftp/index.md) (`rename`)       | `rename()`                                                             | Yes (but not guaranteed atomic on all servers) |
| [SFTP](https://docs.remotestore.dev/stable/guides/backends/sftp/index.md) (final fallback) | Read + write + delete                                                  | —                                              |
| [Memory](https://docs.remotestore.dev/stable/guides/backends/memory/index.md)              | Dict key reassignment                                                  | Yes                                            |
| [HTTP](https://docs.remotestore.dev/stable/guides/backends/http/index.md)                  | — (read-only)                                                          | —                                              |
| [SQLBlob](https://docs.remotestore.dev/stable/guides/backends/sql-blob/index.md)           | SQL `UPDATE` in transaction                                            | Yes                                            |
| [SQLQuery](https://docs.remotestore.dev/stable/guides/backends/sql-query/index.md)         | — (read-only)                                                          | —                                              |

SFTP tries three strategies in order: `posix_rename` (atomic), standard `rename()`, and finally copy+delete. Most OpenSSH servers support `posix_rename`. Servers that lack it usually still support `rename()`, which is atomic on most POSIX filesystems.

### Mitigations

- **Verify after move.** After `move()`, check that the source no longer exists. If it does, delete it or alert.
- **Write + delete instead of move.** If atomicity matters, write the data to the destination first, verify, then delete the source. This gives you explicit control over each step.
- **Use `write_atomic()` for the write step.** `write_atomic()` uses temp-file-and-rename on backends that support it, ensuring the destination is written atomically even if `move()` is not.

## Summary table

| Backend                                                                               | `move()` atomic?      | `write_atomic()` truly atomic?   | `overwrite=False` race-free?  |
| ------------------------------------------------------------------------------------- | --------------------- | -------------------------------- | ----------------------------- |
| [Local](https://docs.remotestore.dev/stable/guides/backends/local/index.md)           | Yes\*                 | Yes (temp file + `os.replace()`) | No (TOCTOU)                   |
| [S3](https://docs.remotestore.dev/stable/guides/backends/s3/index.md)                 | No (copy + delete)    | Yes (PUT is inherently atomic)   | No (TOCTOU)                   |
| [S3-PyArrow](https://docs.remotestore.dev/stable/guides/backends/s3-pyarrow/index.md) | No (copy + delete)    | Yes (PUT is inherently atomic)   | No (TOCTOU)                   |
| [Azure](https://docs.remotestore.dev/stable/guides/backends/azure/index.md) (HNS)     | Yes (`rename_file`)   | Yes (temp file + rename)         | No (TOCTOU)                   |
| [Azure](https://docs.remotestore.dev/stable/guides/backends/azure/index.md) (non-HNS) | No (copy + delete)    | Yes (direct PUT is atomic)       | No (TOCTOU)                   |
| [Graph](https://docs.remotestore.dev/stable/guides/backends/graph/index.md)           | No (server move/copy) | Yes (server `PUT`)               | Yes (server create-if-absent) |
| [SFTP](https://docs.remotestore.dev/stable/guides/backends/sftp/index.md)             | Yes\*\*               | Yes\*\* (temp file + rename)     | No (TOCTOU)                   |
| [Memory](https://docs.remotestore.dev/stable/guides/backends/memory/index.md)         | Yes                   | Yes (direct)                     | No (TOCTOU)                   |
| [SQLBlob](https://docs.remotestore.dev/stable/guides/backends/sql-blob/index.md)      | Yes (SQL transaction) | Yes (direct)                     | No (TOCTOU)                   |

\* Local `move()` uses `shutil.move()`, which delegates to `os.rename()` on the same filesystem (atomic) but falls back to copy+delete across filesystems. Only `write_atomic()` uses `os.replace()`.

\*\* SFTP `move()` is atomic when `posix_rename` or `rename()` succeeds; falls back to copy+delete as a last resort. `write_atomic()` has an orphan-file risk if the connection drops between write and rename (see the [SFTP backend guide](https://docs.remotestore.dev/stable/guides/backends/sftp/index.md)).

## See also

- [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) — atomicity and move semantics per backend
- [Architecture](https://docs.remotestore.dev/stable/explanation/architecture/index.md) — threading model and Store immutability

# API Graph Visualization

An interactive force-directed map of the project's API surface. Nodes are the types and concepts that make up the codebase; edges are the typed relationships between them. The full node and edge taxonomy is defined in the [doc graph model RFC](https://docs.remotestore.dev/stable/explanation/design/rfcs/rfc-0012-doc-graph-model/index.md).

Use it to trace capability chains, explore the class hierarchy, and inspect individual nodes and their direct neighbors. It is built to be explored rather than just looked at:

- **Role-aware nodes:** concrete backends, abstract base classes, and facades are drawn distinctly; `Store`/`AsyncStore` stand out from other facades.
- **Collapse by default:** method nodes are folded into their class so the opening view is capability-level. Double-click a class (or use *Expand all*) to reveal its methods, clustered around it.
- **Detail with deep links:** select any node *or edge* to see its metadata plus jump-to-source links — the source `file:line` and governing spec on GitHub, and the API docs page on this site.
- **Faceted filtering:** text search plus filters for node kind, class role, edge kind, runtime (sync/async), capability, and a selected node's dependency neighbourhood. Facets combine.

[Open visualization](https://docs.remotestore.dev/stable/explanation/graph_viz.html)

The visualization runs fully in the browser, with no server or build step.

## See also

- [Architecture Overview](https://docs.remotestore.dev/stable/explanation/architecture/index.md): three-layer design
- [API Reference](https://docs.remotestore.dev/stable/reference/api/index.md): full method and capability reference
- [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md): per-backend capability support table

# Performance

remote-store wraps established Python storage libraries. This page presents measured overhead so you can judge whether the abstraction cost matters for your workloads.

## Overhead at a Glance

The chart below shows remote-store's overhead (%) versus raw SDK calls for each backend. Negative values mean remote-store is *faster* than calling the SDK directly (often due to connection pooling and caching).

Patterns from Docker benchmarks (MinIO, Azurite, OpenSSH):

- **S3**: reads and writes add modest overhead over raw boto3; listing is significantly faster via s3fs connection caching.
- **S3-PyArrow**: reads carry more overhead than the S3 backend (PyArrow C++ data path); writes are comparable. The trade-off is native PyArrow integration — Tier 1 C++ range requests — not raw throughput.
- **Azure** and **SFTP**: per-operation overhead is small relative to network round-trip time for most operations.
- **Local**: all operations are sub-millisecond; overhead versus raw pathlib is measurable but negligible for storage workloads.

Regenerate numbers for your own hardware with `hatch run bench-report` (see [Running Benchmarks](#running-benchmarks)).

## What Happens Under Real Latency

Under realistic network round-trip times (20–100 ms), overhead as a percentage shrinks. For example, a 1 ms overhead on a 100 ms round trip is 1%.

The benchmark suite simulates latency using [Toxiproxy](https://github.com/Shopify/toxiproxy) with four named profiles:

| Profile  | Latency | Jitter | Simulates              |
| -------- | ------- | ------ | ---------------------- |
| `clean`  | 0 ms    | 0 ms   | Baseline (passthrough) |
| `rtt20`  | 20 ms   | 7 ms   | Same-region cloud      |
| `rtt50`  | 50 ms   | 17 ms  | Cross-region           |
| `rtt100` | 100 ms  | 33 ms  | Cross-continent        |

## Throughput by File Size

How throughput scales with file size, comparing remote-store to raw SDK:

At larger file sizes, throughput converges as the fixed per-operation overhead is amortized across more bytes.

## S3 vs S3-PyArrow

Both S3 backends connect to the same service. S3 uses s3fs (Python), S3-PyArrow uses PyArrow's C++ `S3FileSystem` for data-path operations. The chart below compares their absolute latencies:

S3-PyArrow reads are slower for sequential workloads because the C++ data path adds connection management and metadata overhead per call. The S3-PyArrow backend's advantage is native [PyArrow integration](https://docs.remotestore.dev/stable/guides/pyarrow-adapter/index.md) — Tier 1 Parquet column pruning, I/O coalescing, and GIL-free reads. For sequential byte streaming, the regular S3 backend is faster.

## Comparative Results

For every operation, the benchmark suite runs the same workload through three interfaces:

1. **remote-store** — the `Backend` / `Store` API
1. **Raw SDK** — direct boto3/paramiko/azure-storage-blob/pathlib calls
1. **fsspec** — s3fs/sshfs/adlfs/fsspec.local

### Sample Results

Results vary by hardware, network, and service version. Generate numbers for your environment with `hatch run bench-report` (summary) or `hatch run bench-report-user` (condensed with verdicts).

For a full per-backend comparison of remote-store against the raw SDK and fsspec, see the Detailed Comparative Tables section on the [Performance page](https://docs.remotestore.dev/stable/explanation/performance/).

## Caveats

- **Docker emulators are not cloud.** Azurite, MinIO, and the local SFTP container approximate real services but have different performance characteristics. Treat these numbers as relative comparisons, not absolute predictions of cloud performance.
- **Listing anomalies.** Some fsspec implementations (s3fs, adlfs) show sub-100us listing times that reflect client-side caching, not real storage-layer performance. `S3Backend` defaults this directory-listing cache off (fresh listings every call), so those sub-100us numbers appear only when the cache is explicitly re-enabled via `client_options={"use_listings_cache": True}`; with the default, the s3fs path issues a fresh listing like raw boto3.
- **Delete overhead.** 2-3x vs raw SDK across all backends is expected from the error-mapping layer and not an optimization target.
- **Streaming reads keep memory constant** regardless of file size.

## Methodology

Benchmarks use [pytest-benchmark](https://pytest-benchmark.readthedocs.io/) with Docker-hosted services (MinIO for S3, Azurite for Azure, OpenSSH for SFTP). Each test runs in an isolated environment — fresh buckets, containers, and directories are created per test fixture and cleaned up after.

| Metric                | How                         | Where                  |
| --------------------- | --------------------------- | ---------------------- |
| **Throughput** (MB/s) | payload_bytes / mean_time   | Write, read, roundtrip |
| **TTFB** (ms)         | Time to write/read 1KB file | Protocol overhead      |
| **Latency** (ms)      | Mean operation time         | Exists, delete, list   |
| **Memory** (MB)       | tracemalloc peak            | Large-file read/write  |
| **Listing speed**     | Time to list N files        | 50, 200, 1k, 10k files |

## Running Benchmarks

```
# Start Docker services
docker compose -f infra/docker-compose.yml up -d --wait

# Quick tier (~2 min/backend)
hatch run bench

# Standard tier (~5 min/backend)
hatch run bench-standard

# Full tier (~20-30 min/backend)
hatch run bench-full

# With simulated latency (single profile)
hatch run bench -- --backend s3-latency,sftp-latency,azure-latency --network-profile rtt50

# Latency matrix (runs rtt20, rtt50, rtt100 sequentially, ~8 min/profile)
hatch run bench-latency-matrix

# Save results as JSON
hatch run bench-save

# Reports
hatch run bench-report                    # summary table
hatch run bench-report-user               # condensed with verdicts
hatch run bench-report-comparative        # remote-store vs raw SDK vs fsspec
hatch run bench-charts                    # generate SVG charts

# Stop services
docker compose -f infra/docker-compose.yml down -v
```

For cloud benchmarks, set the appropriate environment variables (see `benchmarks/README.md` for the full reference table) and use `--infra cloud`.

## Detailed Comparative Tables

Per-backend tables comparing remote-store, raw SDK, and fsspec for each operation. Generated with `hatch run bench-report-comparative-md`.

### Local

| Operation     | remote-store | pathlib             | fsspec              |
| ------------- | ------------ | ------------------- | ------------------- |
| Write 1MB     | 646us        | 588us (1.1x faster) | 574us (1.1x faster) |
| Read 1MB      | 321us        | 249us (1.3x faster) | 259us (1.2x faster) |
| Exists (hit)  | 55us         | 5us (10.2x faster)  | 4us (14.2x faster)  |
| List 50 files | 661us        | 678us               | 143us (4.6x faster) |
| Delete        | 112us        | 38us (2.9x faster)  | 55us (2.0x faster)  |

### S3 (MinIO)

| Operation     | remote-store | boto3                | s3fs                 |
| ------------- | ------------ | -------------------- | -------------------- |
| Write 1MB     | 19.9ms       | 24.5ms (1.2x slower) | 23.8ms (1.2x slower) |
| Read 1MB      | 9.0ms        | 4.9ms (1.9x faster)  | 6.6ms (1.4x faster)  |
| Exists (hit)  | 1.5ms        | 1.3ms (1.1x faster)  | 1.3ms (1.1x faster)  |
| List 50 files | 168us        | 4.0ms (24.1x slower) | 89us (1.9x faster)   |
| Delete        | 3.1ms        | 1.5ms (2.1x faster)  | 1.7ms (1.9x faster)  |

### S3-PyArrow

| Operation     | remote-store | boto3                |
| ------------- | ------------ | -------------------- |
| Write 1MB     | 31.9ms       | 43.3ms (1.4x slower) |
| Read 1MB      | 11.5ms       | 4.8ms (2.4x faster)  |
| Exists (hit)  | 1.9ms        | 1.3ms (1.5x faster)  |
| List 50 files | 144us        | 4.0ms (27.6x slower) |
| Delete        | 4.5ms        | 1.5ms (3.0x faster)  |

### SFTP

| Operation     | remote-store | paramiko             | sshfs                |
| ------------- | ------------ | -------------------- | -------------------- |
| Write 1MB     | 29.6ms       | 29.5ms               | 14.3ms (2.1x faster) |
| Read 1MB      | 11.8ms       | 10.0ms (1.2x faster) | 7.2ms (1.6x faster)  |
| Exists (hit)  | 779us        | 397us (2.0x faster)  | 652us (1.2x faster)  |
| List 50 files | 2.5ms        | 2.1ms (1.2x faster)  | 3.0ms (1.2x slower)  |
| Delete        | 1.6ms        | 398us (4.1x faster)  | 1.2ms (1.4x faster)  |

### Azure

| Operation     | remote-store | azure-blob           | adlfs                |
| ------------- | ------------ | -------------------- | -------------------- |
| Write 1MB     | 15.6ms       | 14.1ms (1.1x faster) | 17.2ms (1.1x slower) |
| Read 1MB      | 5.7ms        | 5.7ms                | 10.2ms (1.8x slower) |
| Exists (hit)  | 1.7ms        | 1.6ms                | 1.9ms (1.1x slower)  |
| List 50 files | 9.6ms        | 9.1ms (1.1x faster)  | 66us (145.3x faster) |
| Delete        | 1.8ms        | 1.8ms                | 4.0ms (2.2x slower)  |

## See also

- [Capabilities Matrix](https://docs.remotestore.dev/stable/reference/capabilities-matrix/index.md) — feature support per backend
- [Choosing a Backend](https://docs.remotestore.dev/stable/guides/choosing-a-backend/index.md) — decision guide with trade-offs
- [PyArrow Adapter](https://docs.remotestore.dev/stable/guides/pyarrow-adapter/index.md) — tiered read strategy and S3 direct I/O

# Security Model

How `remote-store` handles credentials, trust boundaries, and security policies.

## Credential hygiene

All credentials are wrapped in `Secret` objects at the configuration layer. The `Secret` class provides:

- **Masked output** — `repr()` and `str()` return `***`, never plain text
- **Explicit reveal** — `secret.reveal()` is the only way to access the plain-text value, making credential access auditable
- **Immutability** — `Secret` uses `__slots__` with `__setattr__` and `__delattr__` overrides to prevent modification after creation

### Automatic wrapping

`RegistryConfig.from_dict()`, `from_toml()`, and `ext.yaml.from_yaml()` automatically wrap values for these config keys in `Secret`:

- `key`, `secret`, `password`
- `account_key`, `sas_token`, `connection_string`
- `client_secret`, `client_certificate`

You never need to create `Secret` instances manually in configuration code.

### Log redaction

The `SecretRedactionFilter` logging filter replaces `Secret` instances with `***` in log records. Attach it to your logging handlers to prevent credential leakage:

```
import logging
from remote_store import SecretRedactionFilter

handler = logging.StreamHandler()
handler.addFilter(SecretRedactionFilter())
logging.getLogger("remote_store").addHandler(handler)
```

## Trust boundaries

### Backend isolation

- User code interacts with `Store` only — backend instances are internal
- Backend-specific exceptions are mapped to `remote-store` error types and **never leak** to user code
- `Store.unwrap()` is the explicit escape hatch for direct backend access; using it crosses the trust boundary intentionally

### Path validation

- All paths are validated via `RemotePath` before reaching a backend
- Path traversal (`..`) is rejected at the Store layer
- Each backend applies additional provider-specific validation

### Configuration isolation

- Config-as-code has absolute priority — no environment variable overrides unless explicitly opted in
- No config merging prevents accidental credential exposure from layered sources

## Vulnerability reporting

Report security issues via [GitHub Security Advisories](https://github.com/haalfi/remote-store/security/advisories/new), not public issues.

See [SECURITY.md](https://github.com/haalfi/remote-store/blob/master/SECURITY.md) for response timelines, scope, and the fix policy.

## See also

- [Architecture Overview](https://docs.remotestore.dev/stable/explanation/architecture/index.md) — system design and error model
- [API Reference: Secret](https://docs.remotestore.dev/stable/reference/api/config/#remote_store.Secret)
- [Spec 020: Credential Hygiene](https://docs.remotestore.dev/stable/explanation/design/specs/020-credential-hygiene/index.md)