Skip to content

remote-store

A Store is a logical folder. Where files live is configuration.

remote-store provides a single interface for file storage across local filesystems, S3, SFTP, Azure, and more, using the libraries you would choose yourself.

Architecture

flowchart LR
    Ext["Extensions"]

    subgraph CORE[" "]
        direction TB
        API["Your Code"]
        Store["Store API"]
        Methods["Read/Write - List - Copy/Move - Config - Capabilities"]
        API --> Store
        Store -.-> Methods
    end

    subgraph INFRA[" "]
        direction TB
        Backends["Backends"]
        B_list["Local - S3 - SFTP - Azure - Http - Memory - SQL - ...yours"]
        Libs["Proven Libraries"]
        L_list["stdlib - s3fs - pyarrow - paramiko - azure SDK"]
        Backends -.-> B_list
        Backends --> Libs
        Libs -.-> L_list
    end

    Ext -. enhance .-> CORE
    CORE --> INFRA
  • The Store provides a portable API
  • Backends implement storage-specific behavior
  • Libraries do the actual I/O
  • Extensions add optional capabilities alongside the core

The core idea

A Store scopes all operations to a root path. Everything is relative.

from remote_store import Store
from remote_store.backends import LocalBackend

store = Store(LocalBackend(root="/tmp/data"))
store.write_text("hello.txt", "Hello, world!")
print(store.read_text("hello.txt"))  # 'Hello, world!'

Switch backend without changing application code:

from remote_store import Store
from remote_store.backends import S3Backend

store = Store(S3Backend(bucket="my-bucket"))
store.write_text("file.txt", "hello")
print(store.read_text("file.txt"))  # same API, different backend

Narrow scope with child(); all paths inside are relative to the new root:

sub = store.child("reports/2024")
sub.write_text("summary.txt", "...")

See Store child scoping for more.

Design principles

Zero dependencies in core

Install only what you use. pip install remote-store pulls in nothing. Extras like [s3] or [sftp] bring in only the backend you need.

Proven libraries underneath

Established libraries like s3fs, paramiko, and the Azure SDK do the real work: remote-store adapts, they execute. Backends delegate to the packages you'd pick yourself.

Backend-native when possible

glob() and atomic writes work everywhere. Where the backend supports them natively, remote-store uses that. Where not, a portable fallback steps in.

from remote_store import Capability

store.supports(Capability.GLOB)  # True for most backends; see capabilities matrix
store.supports(Capability.ATOMIC_WRITE)  # True for most backends; see capabilities matrix

Extensions sit beside, not around

Extensions like caching, observability, batch operations, and PyArrow integration: import what you need from remote_store.ext. Your Store code doesn't change.

Bring your own

Implement the Backend protocol for a new storage target. Or write an extension. The hooks are public.

from remote_store import Backend, Store

class MyBackend(Backend):
    """Implement the Backend protocol for your storage."""

    ...


store = Store(MyBackend(...))  # works with all extensions

See the custom backend guide for a step-by-step walkthrough.

Quick start

pip install remote-store[s3]

from remote_store import Store
from remote_store.backends import S3Backend

store = Store(S3Backend(bucket="my-bucket"))
store.write_text("file.txt", "hello")
print(store.read_text("file.txt"))  # 'hello'

See Getting Started for a complete walkthrough.

Where to go next