Skip to content

remote-store

A Store is a logical folder. Where files live is configuration.

remote-store provides a single interface for file storage across local filesystems, S3, SFTP, Azure, and more, using the libraries you would choose yourself.

Architecture

flowchart LR
    Ext["Extensions"]

    subgraph CORE[" "]
        direction TB
        API["Your Code"]
        Store["Store API"]
        Methods["Read/Write - List - Copy/Move - Config - Capabilities"]
        API --> Store
        Store -.-> Methods
    end

    subgraph INFRA[" "]
        direction TB
        Backends["Backends"]
        B_list["Local - Memory - S3 - Azure - Graph - SFTP - Http - SQL - ...yours"]
        Libs["Proven Libraries"]
        L_list["stdlib - s3fs - pyarrow - paramiko - azure SDK - httpx/msal"]
        Backends -.-> B_list
        Backends --> Libs
        Libs -.-> L_list
    end

    Ext -. enhance .-> CORE
    CORE --> INFRA
  • The Store provides a portable API
  • Backends implement storage-specific behavior
  • Libraries do the actual I/O
  • Extensions add optional capabilities alongside the core

The core idea

A Store scopes all operations to a root path. Everything is relative.

from remote_store import Store
from remote_store.backends import LocalBackend

store = Store(LocalBackend(root="./data"))
store.write_text("hello.txt", "Hello, world!")
print(store.read_text("hello.txt"))  # 'Hello, world!'

Switch backend without changing application code:

from remote_store import Store
from remote_store.backends import S3Backend

store = Store(S3Backend(bucket="my-bucket"))
store.write_text("file.txt", "hello")
print(store.read_text("file.txt"))  # same API, different backend

Narrow scope with child(); all paths inside are relative to the new root:

sub = store.child("reports/2024")
sub.write_text("summary.txt", "...")

See Store child scoping for more.

Design principles

Zero dependencies in core

Install only what you use. pip install remote-store pulls in nothing. Extras like [s3] or [sftp] bring in only the backend you need.

Proven libraries underneath

Established libraries like s3fs, paramiko, and the Azure SDK do the real work: remote-store adapts, they execute. Backends delegate to the packages you'd pick yourself. See the dependency and version policy for how those ranges are chosen and what stays stable across releases.

Backend-native when possible

glob() and atomic writes work everywhere. Where the backend supports them natively, remote-store uses that. Where not, a portable fallback steps in.

from remote_store import Capability

store.supports(Capability.GLOB)  # True for most backends; see capabilities matrix
store.supports(Capability.ATOMIC_WRITE)  # True for most backends; see capabilities matrix

Extensions sit beside, not around

Extensions like caching, observability, batch operations, and PyArrow integration: import what you need from remote_store.ext. Your Store code doesn't change.

Bring your own

Implement the Backend protocol for a new storage target. Or write an extension. The hooks are public.

from remote_store import Backend, Store

class MyBackend(Backend):
    """Implement the Backend protocol for your storage."""

    ...


store = Store(MyBackend(...))  # works with all extensions

See the custom backend guide for a step-by-step walkthrough.

Quick start

pip install remote-store[s3]

from remote_store import Store
from remote_store.backends import S3Backend

store = Store(S3Backend(bucket="my-bucket"))
store.write_text("file.txt", "hello")
print(store.read_text("file.txt"))  # 'hello'

See Getting Started for a complete walkthrough.

Where to go next