Skip to content

Medallion + Dagster Showcase

A self-contained Dagster project demonstrating remote-store's value proposition through a real-world medallion architecture (Bronze → Silver → Gold) over live MeteoSwiss weather station data.

What This Demonstrates

Four remote-store extensions composing without conflict:

Extension Role
ReadOnlyHttpBackend Read-only backend fetching live CSV data via HTTP
ext.cache TTL-based caching — avoids redundant HTTP downloads
ext.otel OpenTelemetry spans + metrics on every storage operation
ext.dagster 3-line IO manager wrapping any Store for Dagster

Prerequisites

  • Python 3.10+
  • Network access to data.geo.admin.ch (Swiss open government data, no credentials)

Setup

cd examples/medallion_dagster

# Install remote-store with required extras + showcase dependencies
pip install -e "../../[dagster,arrow,otel,requests]" polars dagster-webserver opentelemetry-sdk

Running

dagster dev -f definitions.py

Open the Dagster UI (typically http://localhost:3000) and materialize all assets.

Architecture

MeteoSwiss HTTP ──→ ext.cache (1h TTL) ──→ ext.otel (traces)
       │
       └──→ read_bytes + write ──→ Bronze (raw CSV)
                                      │
                                      ├──→ Silver (cleaned Parquet)
                                      │
                                      └──→ Gold (aggregated Parquet)

Bronze Layer (raw ingest)

  • meteo_stations — station metadata CSV
  • bronze_bern, bronze_zurich, bronze_lugano — daily weather CSVs
  • Uses read_bytes + write directly (file-level copy, no IO manager)

Silver Layer (clean + unify)

  • silver_measurements — all stations cleaned, unified, stored as Parquet
  • Parses semicolon-delimited CSV, normalizes timestamps, drops null rows
  • Uses Dagster IO manager with ParquetSerializer

Gold Layer (analytics)

  • gold_daily_summary — daily avg/min/max temperature, precipitation per station
  • gold_station_stats — per-station row counts, date ranges, mean temperature
  • gold_alerts — frost (< 0°C) and heat (> 30°C) alert days

What to Observe

Dagster UI

  • Asset graph showing Bronze → Silver → Gold dependencies
  • Materialization metadata (path, size) on Silver/Gold assets

Terminal Output

  • OTel spans (JSON lines) for both stores: read_bytes() against the HTTP source, and write() / read_bytes() / child() against the lake. The lake is OTel-wrapped, so these spans follow the backend you swap to (local, S3, Azure).
  • Cache hit/miss stats after each Bronze ingest
  • Row counts from Silver and Gold transforms

Cache Benefit

Run materialization twice within one hour. The second run hits the cache for all Bronze read_bytes() calls — visible in cache stats (4 hits, 0 misses) and shorter OTel span durations.

Configuration

Two environment variables override the defaults without editing code:

Variable Default Purpose
RS_SHOWCASE_SOURCE_URL MeteoSwiss open-data base URL Point the HTTP source at a mirror, cache, or local server
RS_SHOWCASE_LAKE_ROOT ./data/showcase Relocate the local lake directory

Swapping Backends

The core value proposition: swap the lake backend in stores.py from local filesystem to S3 or Azure. The lake is wrapped with otel_observe(...), so observability follows the swap — every Bronze write and Silver/Gold round-trip emits spans against the new backend, not just the HTTP source.

# Local (default)
lake = otel_observe(Store(LocalBackend(root=_LAKE_ROOT)))

# S3 — S3Backend has no prefix= param; scope to a sub-prefix with .child()
lake = otel_observe(Store(S3Backend(bucket="my-bucket")).child("showcase"))

# Azure ADLS Gen2 (Hierarchical Namespace)
lake = otel_observe(
    Store(
        AzureBackend(
            container="my-filesystem",
            hns=True,  # required for ADLS Gen2 — the backend does not auto-detect it
            connection_string=os.environ["AZURE_STORAGE_CONNECTION_STRING"],
        )
    )
)

Install the cloud backend you swap to (remote-store[s3] or remote-store[azure]). hns=True is mandatory for an ADLS Gen2 account (see the Azure HNS setup guide); omitting it falls back to flat-blob semantics. Cloud backends need credentials — pass them explicitly (as above) or let the Azure SDK pick up DefaultAzureCredential.

Everything else — caching, observability, Dagster integration — works unchanged.

Data Source

MeteoSwiss Automatic Weather Stations (SMN) — Swiss Federal Office of Meteorology and Climatology. Public domain data, no API keys required.

Stations used: Bern-Zollikofen (ber), Zurich-Kloten (klo), Lugano (lug). Granularity: daily measurements.

See also