S3 Backend Specification¶

Overview¶

S3Backend implements the Backend ABC for S3-compatible object storage (AWS S3, MinIO, etc.) using s3fs internally. It maps the Backend contract onto S3's flat key-value model, bridging the gap between S3's prefix-based "folders" and the filesystem-like interface expected by Store.

This is the fsspec-based S3 backend. A future native backend using boto3 + pyarrow.fs directly may follow for advanced data engineering workloads.

Dependencies: s3fs (optional extra: pip install "remote-store[s3]")

Spec 011 (S3-PyArrow Hybrid Backend) inherits most invariants from this document and only specifies PyArrow-specific deltas. Edits to shared invariants here propagate to spec 011 automatically via the paired-ID map at the top of 011.

Construction¶

S3-001: Constructor Parameters¶

Invariant: S3Backend is constructed with a required bucket name and optional connection parameters. Signature:

S3Backend(
    bucket: str,
    *,
    endpoint_url: str | None = None,
    key: str | None = None,
    secret: str | None = None,
    region_name: str | None = None,
    tls_ca_bundle: str | None = None,  # see spec 039
    client_options: dict[str, Any] | None = None,
    retry: RetryPolicy | None = None,  # see spec 025
)

Postconditions: The backend stores configuration but does not connect to S3 during construction (see S3-004). The endpoint_url is normalized per S3-025.

S3-002: Backend Name¶

Invariant: name property returns "s3".

S3-003: Capability Declaration¶

Invariant: S3Backend declares capabilities: READ, WRITE, DELETE, LIST, MOVE, COPY, ATOMIC_WRITE, METADATA, GLOB. Native glob via prefix-optimized listing (see 018-glob.md GLOB-018). Rationale: - ATOMIC_WRITE: S3 PUT is inherently atomic — readers never see partial content (see S3-010). - MOVE: Implemented via server-side copy + delete (see S3-013). - COPY: Implemented via S3 server-side copy (see S3-014). - GLOB: Native prefix-optimized glob since v0.12.0 (BK-002).

S3-004: Lazy Connection¶

Invariant: No network call occurs during __init__. The s3fs filesystem is created lazily on first operation. Rationale: Fail-fast at construction is undesirable — the backend may be created during application wiring before the network is available.

S3-005: Construction Validation¶

Invariant: bucket must be a non-empty string. Passing an empty or whitespace-only bucket raises ValueError at construction time. Postconditions: No network validation of bucket existence at construction time. Invalid bucket names that are syntactically non-empty are caught by S3 on first operation and mapped to the appropriate error.

S3 Object Model¶

S3-006: Virtual Folder Semantics¶

Invariant: S3 has no native directories. "Folders" are logical constructs derived from key prefixes delimited by /. The S3 backend presents a filesystem-like view by interpreting common prefixes as folders. Postconditions: This is a fundamental difference from local filesystems. Several Backend operations have S3-specific behavior documented in this section.

S3-007: Folder Detection¶

Invariant: is_folder(path) returns True if any objects exist with prefix {path}/ in the bucket. Postconditions: Does not require an explicit folder marker object. An empty prefix (no objects) returns False. Example:

backend.write("data/file.txt", b"x")
assert backend.is_folder("data") is True
assert backend.is_folder("data/nonexistent") is False

S3-008: Write Does Not Create Folder Markers¶

Invariant: write("a/b/c.txt", content) creates only the object with key a/b/c.txt. No folder marker objects are created for a/ or a/b/. Rationale: Folder markers add PUT overhead and clutter. S3's prefix-based folder detection (S3-007) makes them unnecessary.

S3-009: Folder Lifecycle Tied to Contents¶

Invariant: A "folder" exists only as long as objects exist under its prefix. Deleting the last object under a prefix causes is_folder() to return False. Postconditions: This differs from local filesystems where empty directories persist after their contents are deleted. Example:

backend.write("dir/file.txt", b"x")
assert backend.is_folder("dir") is True
backend.delete("dir/file.txt")
assert backend.is_folder("dir") is False  # folder vanishes

Impact on conformance: delete_folder on an already-empty prefix raises NotFound (with missing_ok=False), because the folder no longer exists.

Operations¶

S3-010: Atomic Write Via S3 PUT¶

Invariant: write_atomic is implemented identically to write — as a direct S3 PUT. Rationale: S3 PUT is inherently atomic. From a reader's perspective, the object transitions from non-existent (or old content) to new content in a single operation. No partial content is ever visible. The temp-file + rename pattern used by local backends is unnecessary and would add latency (extra PUT + COPY + DELETE). Postconditions: Satisfies AW-001's postcondition: "No partial content is ever visible." Mid-stream content failure: If the content source raises partway through a streaming write, the in-flight upload is aborted rather than committed — s3fs's S3File.discard() aborts any multipart upload and drops the buffer, so no truncated object is left in the bucket and no multipart upload is orphaned. Without this, S3File.__exit__ → close() would flush the buffer / complete the multipart upload and commit a complete-looking but truncated object, breaking the postcondition above. The abort applies to both write and write_atomic; write is non-atomic per AW-007 but on this backend cleans up for free via the shared path.

S3-011: delete_folder Recursive¶

Invariant: delete_folder(path, recursive=True) deletes all objects with prefix {path}/. Postconditions: After completion, no objects exist under that prefix. The "folder" ceases to exist (S3-009). Raises: NotFound if no objects exist under the prefix and missing_ok=False.

S3-012: delete_folder Non-Recursive¶

Invariant: delete_folder(path, recursive=False) succeeds only if no file objects exist under the prefix (the folder is "empty"). Raises: NotFound if the folder does not exist and missing_ok=False. Raises a non-empty error if file objects exist under the prefix. Postconditions: Consistent with local filesystem semantics where rmdir fails on non-empty directories.

S3-013: move Via Copy + Delete¶

Invariant: move(src, dst) is implemented as server-side copy followed by delete of the source. Postconditions: Not atomic — if copy succeeds but delete fails, both objects exist. This is inherent to S3 (no native rename). Raises: NotFound if src does not exist. AlreadyExists if dst exists and overwrite=False.

S3-014: copy Via S3 Server-Side Copy¶

Invariant: copy(src, dst) uses S3 server-side copy (no data passes through the client). Postconditions: Efficient for large files — the S3 service handles the copy internally. Raises: NotFound if src does not exist. AlreadyExists if dst exists and overwrite=False.

Error Mapping¶

S3-015: NotFound Mapping¶

Invariant: S3 responses with HTTP 404 or error code NoSuchKey / NoSuchBucket are mapped to NotFound. Postconditions: path and backend attributes are set on the error.

S3-016: PermissionDenied Mapping¶

Invariant: S3 responses with HTTP 403 or error code AccessDenied are mapped to PermissionDenied.

S3-017: BackendUnavailable Mapping¶

Invariant: Connection errors (DNS resolution failure, connection refused, timeouts) are mapped to BackendUnavailable.

S3-018: No Native Exception Leakage¶

Invariant: No s3fs, botocore, or aiobotocore exceptions propagate to callers. All are mapped to remote_store error types per BE-021. Postconditions: backend attribute is set to "s3" on all mapped errors.

Resource Management¶

S3-019: close()¶

Invariant: close() releases the underlying s3fs filesystem resources. Postconditions: Safe to call multiple times. After close, further operations may fail.

S3-020: unwrap()¶

Invariant: unwrap(S3FileSystem) returns the underlying s3fs.S3FileSystem instance. Raises: CapabilityNotSupported for any other type hint. Rationale: Escape hatch for users who need s3fs-specific features (per ADR-0003).

Configuration¶

S3-021: Client Options Passthrough¶

Invariant: The client_options dict is merged into the s3fs configuration, allowing advanced settings (custom SSL, proxy, timeouts, etc.). Postconditions: Explicit constructor parameters (endpoint_url, key, secret, region_name) take precedence over keys in client_options.

For the listings-cache default the builder injects during this merge (a precedence rule in the opposite direction — client_options overrides our default), see S3-027.

S3-022: Default Credential Chain¶

Invariant: When key and secret are not provided, the backend falls back to the standard AWS credential chain (environment variables AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, ~/.aws/credentials, IAM role, etc.). Rationale: Follows the principle of least surprise for AWS users.

FileInfo Population¶

S3-023: ETag Population¶

Invariant: _info_to_fileinfo populates FileInfo.etag from the ETag field of the S3 response dict (always present for existing objects). The raw S3 ETag is double-quoted (e.g. "\"abc123\""); the backend strips the outer quotes and lowercases the value before storing it. Postconditions: - FileInfo.etag is a non-empty lowercase string when the info dict contains an ETag key. - FileInfo.etag is None only when the info dict has no ETag (not expected for well-formed S3 responses). - FileInfo.digest from listing paths (list_files, iter_children) is always None — checksum data requires a separate HeadObject call (see S3-024). - FileInfo.digest from get_file_info may be a ContentDigest even for objects uploaded without an explicit checksum, because Amazon S3 has automatically computed and stored CRC32 checksums for new objects since late 2022 (see S3-024).

S3-024: Digest Population via `ChecksumMode: ENABLED`¶

Invariant: Both write() and get_file_info populate their respective digest fields via head_object(..., ChecksumMode="ENABLED"), using the shared _digest_from_head_response helper. This guarantees WriteResult.digest and FileInfo.digest are always consistent for the same key (WR-001a). Mechanism: 1. After a successful upload, write() calls _fs.call_s3("head_object", Bucket=..., Key=..., ChecksumMode="ENABLED") to retrieve checksum headers and stores the result in WriteResult.digest. 2. get_file_info issues the same head_object call and stores the result in FileInfo.digest. 3. In both cases, the base64-encoded checksum value is decoded to hex and wrapped in a ContentDigest(algorithm, hex_value). Supported algorithms: sha256, sha1, crc32, crc32c (case-insensitive on input, lowercase-normalized in ContentDigest). Postconditions: - WriteResult.digest and FileInfo.digest agree for the same key — both are populated from the same head_object mechanism. - Both fields are a ContentDigest with the correct algorithm and hex value when the HeadObject response contains any known checksum key. - Both may be a ContentDigest even without an explicit checksum algorithm, because Amazon S3 (and moto) automatically computes and stores CRC32 checksums for new objects since late 2022. - Both are None only when the response contains no known checksum key, or base64 decoding the value fails. - FileInfo.digest from listing paths (list_files, iter_children) is always None — the extra request is only issued by get_file_info and write().

S3-025: Endpoint URL Normalization¶

Invariant: endpoint_url is normalized at construction time so that bare host:port values are usable. Rules: - None → None (unchanged). - Empty or whitespace-only → None. - Bare host:port or hostname → prefixed with https://. - URLs with an existing http:// or https:// scheme (case-insensitive per RFC 3986 § 3.1) → whitespace-stripped, otherwise unchanged.

Postconditions: After construction, self._endpoint_url always contains a scheme prefix or is None.

S3-026: config_kwargs is the only Config channel; client_kwargs['config'] is rejected¶

Invariant: Every botocore Config option supplied to an S3 backend flows to s3fs via opts['config_kwargs'] (a dict). The kwargs dict handed to s3fs.S3FileSystem never contains client_kwargs['config'], and aiobotocore.create_client() therefore only ever receives one config= argument — the AioConfig s3fs builds from self.config_kwargs.

Why: s3fs.S3FileSystem.set_session always calls aiobotocore.create_client("s3", config=AioConfig(**self.config_kwargs), **client_kwargs). A parallel client_kwargs['config'] duplicates the config= keyword and raises TypeError: got multiple values for keyword argument 'config' (BUG-178, BUG-185).

Rules: - A top-level config_kwargs dict in client_options is the supported channel for botocore Config options. It is forwarded as-is to s3fs, which reconstructs AioConfig(**config_kwargs) itself. - If the caller passes client_kwargs['config'] (a pre-built botocore.config.Config), the builder raises ValueError with a message pointing at config_kwargs. Silent rewriting is forbidden because it hid two consecutive bugs: it always produced a duplicate config= on s3fs ≥ 2024.x, and the kwarg-shape unit tests asserted at the wrong boundary. - RetryPolicy (when supplied via retry=) replaces the entire retries entry in config_kwargs with {"max_attempts": rp.max_attempts, "mode": "standard"} (plain dict assignment, not a field-level merge). Caller-supplied retry modes (e.g. adaptive) and any other non-max_attempts keys are dropped when both retry= and config_kwargs.retries are supplied; the builder emits a log.warning enumerating the dropped keys so the loss is observable. Use one channel to keep caller-supplied retry knobs. - Caller-supplied fields outside retries (e.g. connect_timeout, read_timeout, s3.addressing_style, proxies) are preserved.

Scope: Applies to both S3Backend and S3PyArrowBackend (both use the _S3Base._build_s3fs_kwargs() builder).

S3-027: Directory-listing cache defaults off¶

Invariant: _build_s3fs_kwargs() injects use_listings_cache=False as a default. Caller-supplied client_options["use_listings_cache"] takes precedence — the s3fs directory-listing cache is opt-in, not on by default.

Why: s3fs caches directory listings in a DirCache that defaults to listings_expiry_time=None, so a listed directory is never re-fetched until invalidate_cache(). For a multi-writer Store, a write from a second instance is then permanently invisible to the first — breaking the "a listing shows what's there" model. The only benefit the cache buys is repeated-list latency when nothing writes in between; the fresh-list cost is one bounded round trip.

Rules: - Absent any caller value, the kwargs handed to s3fs.S3FileSystem carry use_listings_cache=False. - A caller passing client_options={"use_listings_cache": True} (or False) keeps their value unchanged (setdefault semantics, matching anon). - Callers who want caching re-enable it via client_options or the ext.cache extension.