Skip to content

MiniS3 M1 Design and Implementation History

Historical objective: Build the deterministic M1 teaching implementation of an S3-like object store with versioning, listing, durable atomic publication, runnable labs, tests, and concept-mapping documentation.

Architecture: MiniS3 is the public service facade. Bucket owns the in-memory versioning state machine, listing derives S3-style views from the flat key map, and DiskStorage persists immutable version artifacts before atomically replacing a bucket manifest. A manifest replacement is the visibility linearization point; startup recovery discards temporary and unreferenced artifacts.

Tech Stack: Python 3.12+, standard library only at runtime, uv, hatchling, pytest.

File Map

  • src/minis3/model.py: immutable object/version/delete-marker values and quoted MD5 ETags.
  • src/minis3/bucket.py: versioning states and all per-bucket object transitions.
  • src/minis3/listing.py: current-object and all-version projections, delimiter grouping, and opaque pagination.
  • src/minis3/store.py: public multi-bucket API and persistence coordination.
  • src/minis3/storage/atomic.py: fsync + replace publication primitive and crash hooks.
  • src/minis3/storage/disk.py: disk layout, manifests, reload, and recovery.
  • src/minis3/errors.py: public domain exceptions.
  • src/minis3/multipart.py, conditional.py, lifecycle.py: documented M2 boundaries only.
  • labs/*.py: public-API-only demonstrations.
  • tests/*.py: executable contracts for all M1 behavior and crash boundaries.

Milestone 1: Project skeleton and model

  • Added packaging metadata, .gitignore, public exports, and M2 placeholder modules.
  • Historical test coverage included failing model tests for quoted MD5, immutable version values, and flat keys.
  • Historical verification covered the named focused checks.
  • The recorded implementation included ObjectRecord, Version, DeleteMarker, and ETag calculation.
  • Historical regression coverage included the focused model tests to green.

Milestone 2: Versioning state machine

  • Historical test coverage included failing tests covering unversioned null replacement/deletion, enabled versions and markers, version-addressed reads/deletes, and suspended null replacement with historical preservation.
  • Historical verification covered the named focused checks.
  • The recorded implementation included deterministic counters, public errors, Bucket, and the in-memory MiniS3 facade.
  • Historical regression coverage included the focused tests and refactor only while green.

Milestone 3: Listing projections

  • Historical test coverage included failing tests for prefix/delimiter directory illusion, max-keys pagination with opaque tokens, marker-hidden current keys, and flattened version history with is_latest.
  • Historical verification covered the named focused checks.
  • The recorded implementation included immutable listing result values and deterministic list projections.
  • Historical regression coverage included focused tests and the existing suite.

Milestone 4: Atomic disk persistence and recovery

  • Historical test coverage included failing tests for restart durability, pre-publication crash retaining old state, post-publication crash exposing new state, tmp cleanup, unreferenced artifact cleanup, and bucket deletion rules.
  • Historical verification covered the named focused checks.
  • The recorded implementation included unique internal storage IDs, immutable content/metadata files, atomic manifest publication, directory fsync, restart scanning, and cleanup.
  • The recorded integration included DiskStorage with all public mutating operations without changing the public state-machine semantics.
  • Historical regression coverage included storage tests and the full suite.

Milestone 5: Teaching labs and documentation

  • Historical additions included the three requested labs using only imports from minis3.
  • Historical verification covered the named focused checks.
  • Historical test coverage included README quick start and directory guide.
  • Historical test coverage included the three-tier real-S3 mapping, including the pre/post December 2020 list consistency history.
  • Historical test coverage included the explicit non-goals and semantic differences, including single-part-only MD5 ETags and M2 deferrals.

Milestone 6: Final acceptance

  • Historical verification covered the named focused checks.
  • Historical verification covered the named focused checks.
  • Historical verification covered the named focused checks.
  • Historical verification covered the named focused checks.
  • Historical handoff evidence included repository status, confirmation that no commit was created, and module line counts.