MiniLucene Immutable Storage and Commit Design and Implementation History¶

Historical objective: Persist Phase 1 segments with deterministic codecs, publish commits atomically through one manifest, and reopen committed indexes with search results equivalent to the in-memory oracle.

Architecture: A frozen SegmentImage crosses the RAM/disk boundary. SegmentCodec writes immutable, checksummed segment directories. ManifestStore is the only recoverable root. IndexWriter owns the RAM buffer and generation counters; Index.open_reader() consumes only the committed manifest.

Tech Stack: Python 3.12 standard library (pathlib, json, hashlib, os, struct), pytest, Ruff, uv.

Milestone 1: Freeze `SegmentImage` as the storage boundary¶

Recorded file scope: - Added: src/minilucene/storage/__init__.py - Added: src/minilucene/storage/image.py - Added: tests/unit/storage/test_segment_image.py

Recorded activity 1 — Test intent: the failing invariant tests

import pytest

from minilucene.storage.image import SegmentImage


def test_segment_image_rejects_non_dense_documents():
    with pytest.raises(ValueError, match="dense"):
        SegmentImage(
            generation=1,
            schema_fingerprint="abc",
            stored_documents={1: {"id": "late"}},
            postings={},
            field_lengths={},
        )


def test_segment_image_is_immutable(ram_segment):
    image = SegmentImage.from_ram_segment(7, "schema", ram_segment)
    with pytest.raises(TypeError):
        image.stored_documents[0] = {"id": "changed"}

Recorded activity 2 — Verification intent: RED

Historical verification covered targeted or full test coverage, including tests/unit/storage/test_segment_image.py.

Historical expected evidence: import failure because storage.image does not exist.

Recorded activity 3 — Design outcome: the frozen image

The design used immutable tuples and MappingProxyType. Validate generation is positive, document IDs are exactly range(max_doc), posting doc IDs are increasing and in range, positions are increasing, and every field-length vector has max_doc entries.

Public constructor:

@dataclass(frozen=True)
class SegmentImage:
    generation: int
    schema_fingerprint: str
    stored_documents: Mapping[int, Mapping[str, str]]
    postings: Mapping[str, Mapping[str, tuple[Posting, ...]]]
    field_lengths: Mapping[str, tuple[int, ...]]

    @property
    def max_doc(self) -> int: ...

    @classmethod
    def from_ram_segment(
        cls, generation: int, schema_fingerprint: str, segment: MemorySegment
    ) -> "SegmentImage": ...

Recorded activity 4 — Verification intent: GREEN and regression

Historical verification covered targeted or full test coverage, including tests/unit/storage/test_segment_image.py, tests/unit/index.

Milestone 2: Implement unsigned varints and delta sequences¶

Recorded file scope: - Added: src/minilucene/storage/varint.py - Added: tests/unit/storage/test_varint.py

Recorded activity 1 — Test intent: table and malformed-input tests

import pytest

from minilucene.storage.varint import (
    decode_delta_sequence,
    decode_uvarint,
    encode_delta_sequence,
    encode_uvarint,
)


@pytest.mark.parametrize("value", [0, 1, 127, 128, 16_384, 2**63 - 1])
def test_uvarint_round_trip(value):
    encoded = encode_uvarint(value)
    assert decode_uvarint(encoded, 0) == (value, len(encoded))


def test_delta_sequence_requires_strict_increase():
    with pytest.raises(ValueError, match="increasing"):
        encode_delta_sequence((2, 2))


def test_unterminated_varint_is_rejected():
    with pytest.raises(ValueError, match="unterminated"):
        decode_uvarint(b"\x80", 0)

Recorded activity 2 — Verification intent: RED

Historical verification covered targeted or full test coverage, including tests/unit/storage/test_varint.py.

Recorded activity 3 — Design outcome: bounded decoding

decode_uvarint accepts at most ten bytes and rejects overflow, negative offsets, out-of-bounds offsets, and unterminated input. Delta decoding accepts an explicit element count and rejects zero deltas after the first element.

Recorded activity 4 — Verification intent: GREEN

Historical verification covered targeted or full test coverage, including tests/unit/storage/test_varint.py.

Milestone 3: Encode terms, postings, stored fields, and norms¶

Recorded file scope: - Added: src/minilucene/storage/codec.py - Added: tests/unit/storage/test_segment_codec.py - Added: docs/segment-format.md

Recorded activity 1 — Test intent: byte-stability and round-trip tests

from minilucene.storage.codec import SegmentDataCodec


def test_segment_data_codec_is_deterministic(segment_image):
    first = SegmentDataCodec.encode(segment_image)
    second = SegmentDataCodec.encode(segment_image)
    assert first == second
    assert set(first) == {
        "terms.bin",
        "postings.bin",
        "stored.bin",
        "norms.bin",
    }


def test_segment_data_codec_round_trips(segment_image):
    files = SegmentDataCodec.encode(segment_image)
    decoded = SegmentDataCodec.decode(
        generation=segment_image.generation,
        schema_fingerprint=segment_image.schema_fingerprint,
        files=files,
    )
    assert decoded == segment_image

The recorded scope added corruption cases for invalid UTF-8, offsets outside postings.bin, non-monotonic doc IDs, non-monotonic positions, malformed JSON frames, trailing bytes, and field-length count mismatches.

Recorded activity 2 — Verification intent: RED

Historical verification covered targeted or full test coverage, including tests/unit/storage/test_segment_codec.py.

Recorded activity 3 — Design outcome: the documented codec

Encoding contracts:

terms.bin:
  term_count
  repeated(field_utf8, term_utf8, postings_offset, postings_length)

postings.bin:
  posting_count
  repeated(doc_delta, tf, position_count, position_deltas)

stored.bin:
  doc_count
  repeated(json_byte_length, canonical_json_bytes)

norms.bin:
  field_count
  repeated(field_utf8, length_count, document_lengths)

Every integer and byte-string length uses unsigned varints. Terms sort by (field UTF-8 bytes, term UTF-8 bytes). JSON uses sort_keys=True, ensure_ascii=False, separators=(",", ":").

Recorded activity 4 — Document the exact format

docs/segment-format.md records magic/version ownership, every frame, sorting rules, validation rules, and that compatibility with Apache Lucene is absent.

Recorded activity 5 — Verification intent: GREEN and deterministic check

Historical verification covered targeted or full test coverage, static analysis, including tests/unit/storage/test_segment_codec.py, src/minilucene/storage, tests/unit/storage.

Milestone 4: Publish checksummed immutable segment directories¶

Recorded file scope: - Added: src/minilucene/storage/filesystem.py - Added: src/minilucene/storage/segment_store.py - Added: tests/storage/test_segment_store.py

Recorded activity 1 — Test intent: publication and cleanup tests

from pathlib import Path

import pytest

from minilucene.storage.segment_store import SegmentStore


def test_segment_store_writes_metadata_last(tmp_path, segment_image, recording_fs):
    store = SegmentStore(tmp_path, fs=recording_fs)
    descriptor = store.publish(segment_image)
    assert recording_fs.file_syncs[-1].name == "segment.json"
    assert descriptor.path == Path("segments/seg_000001")
    assert not list((tmp_path / "segments").glob(".tmp-*"))


def test_failed_publish_never_creates_final_directory(
    tmp_path, segment_image, failing_fs
):
    store = SegmentStore(tmp_path, fs=failing_fs.fail_on_write("postings.bin"))
    with pytest.raises(OSError):
        store.publish(segment_image)
    assert not (tmp_path / "segments" / "seg_000001").exists()

Recorded activity 2 — Verification intent: RED

Historical verification covered targeted or full test coverage, including tests/storage/test_segment_store.py.

Recorded activity 3 — Design outcome: injected filesystem operations

FileSystemOps wraps mkdir, write, fsync-file, fsync-directory, replace, read, remove-tree, and list-directory. SegmentStore.publish uses a UUID temp directory, writes four data files, fsyncs them, writes segment.json last with magic/version/generation/schema fingerprint/length/SHA-256 entries, fsyncs, renames atomically, and fsyncs segments/.

Recorded activity 4 — Design outcome: strict open

SegmentStore.open(generation, expected_schema_fingerprint) validates metadata before decoding. Unknown version, wrong schema, missing file, wrong length, wrong checksum, or invalid codec data raises CorruptIndexError without returning a partial image.

Recorded activity 5 — Verification intent: GREEN

Historical verification covered targeted or full test coverage, including tests/storage/test_segment_store.py.

Milestone 5: Make the manifest the only committed root¶

Recorded file scope: - Added: src/minilucene/storage/manifest.py - Added: tests/storage/test_manifest_store.py

Recorded activity 1 — Test intent: atomic-root tests

import pytest

from minilucene.storage.manifest import Manifest, ManifestStore


def test_open_ignores_complete_orphan_segment(tmp_path, published_segment):
    store = ManifestStore(tmp_path)
    store.create(schema_fingerprint="schema")
    manifest = store.read()
    assert manifest.segment_generations == ()


def test_replace_failure_preserves_old_manifest(tmp_path, failing_fs):
    store = ManifestStore(tmp_path)
    store.create(schema_fingerprint="schema")
    old = store.read()
    broken = Manifest.next_from(old, segment_generations=(1,))
    with pytest.raises(OSError):
        store.write_atomic(broken, fs=failing_fs.fail_on_replace())
    assert store.read() == old

Recorded activity 2 — Verification intent: RED

Historical verification covered targeted or full test coverage, including tests/storage/test_manifest_store.py.

Recorded activity 3 — Design outcome: immutable manifest values

Manifest fields:

@dataclass(frozen=True)
class SegmentCommit:
    segment_generation: int
    live_docs_generation: int | None
    live_docs_checksum: str | None


@dataclass(frozen=True)
class Manifest:
    format_version: int
    schema_fingerprint: str
    commit_generation: int
    segments: tuple[SegmentCommit, ...]
    next_segment_generation: int
    next_commit_generation: int

The recorded scope added writes persisted schema plus generation-zero manifest. Update writes canonical JSON to manifest.tmp, fsyncs, replaces manifest.json, and fsyncs the index directory. Startup removes neither orphans nor temp files; it ignores anything not referenced by the manifest.

Recorded activity 4 — Verification intent: GREEN

Historical verification covered targeted or full test coverage, including tests/storage/test_manifest_store.py.

Milestone 6: Add index creation, schema persistence, and writer exclusion¶

Recorded file scope: - Added: src/minilucene/index.py - Added: src/minilucene/writer.py - Added: src/minilucene/errors.py - Changed: src/minilucene/__init__.py - Added: tests/contract/test_index_lifecycle.py

Recorded activity 1 — Test intent: public lifecycle tests

import pytest

from minilucene import Index, KeywordField, Schema, TextField
from minilucene.errors import SchemaMismatchError, WriterAlreadyOpenError


def test_create_open_and_schema_fingerprint(tmp_path):
    schema = Schema(id=KeywordField(stored=True), body=TextField(stored=True))
    index = Index.create(tmp_path, schema)
    assert Index.open(tmp_path).schema == schema


def test_open_rejects_supplied_different_schema(tmp_path, schema):
    Index.create(tmp_path, schema)
    with pytest.raises(SchemaMismatchError):
        Index.open(tmp_path, Schema(body=TextField(stored=True)))


def test_only_one_writer_can_be_open(tmp_path, schema):
    index = Index.create(tmp_path, schema)
    first = index.writer()
    with pytest.raises(WriterAlreadyOpenError):
        index.writer()
    first.close()
    index.writer().close()

Recorded activity 2 — Verification intent: RED

Historical verification covered targeted or full test coverage, including tests/contract/test_index_lifecycle.py.

Recorded activity 3 — Design outcome: lifecycle ownership

Index.create refuses non-empty initialized paths, persists schema, and writes manifest zero. Index.open validates schema fingerprint. Index.writer creates an exclusive .writer.lock with O_CREAT | O_EXCL; close() releases it idempotently. Stale-lock recovery is explicit and outside V1.

Recorded activity 4 — Verification intent: GREEN

Historical verification covered targeted or full test coverage, including tests/contract/test_index_lifecycle.py.

Milestone 7: Implement deterministic flush without publication¶

Recorded file scope: - Changed: src/minilucene/writer.py - Added: tests/storage/test_writer_flush.py

Recorded activity 1 — Test intent: visibility and threshold tests

def test_flush_creates_segment_but_does_not_change_manifest(index):
    with index.writer() as writer:
        writer.add_document(id="1", body="alpha")
        segment = writer.flush()
        assert segment.generation == 1
        assert index.manifest().segments == ()


def test_document_threshold_flushes_before_next_add(index_with_threshold_one):
    with index_with_threshold_one.writer() as writer:
        writer.add_document(id="1", body="alpha")
        writer.add_document(id="2", body="beta")
        assert writer.segment_generations == (1,)
        assert writer.buffered_document_count == 1

Recorded activity 2 — Verification intent: RED

Historical verification covered targeted or full test coverage, including tests/storage/test_writer_flush.py.

Recorded activity 3 — Design outcome: flush

Before each add, validate and analyze the complete document into a temporary prepared document. Mutate the RAM buffer only after preparation succeeds. Flush freezes SegmentImage, publishes it, appends its descriptor only to the writer's current segment set, increments generation, and replaces the RAM buffer only after publication succeeds. Empty flush returns None.

Automatic flush uses FlushPolicy(max_documents, max_postings) and checks the logical counts before admitting the next document.

Recorded activity 4 — Verification intent: GREEN

Historical verification covered targeted or full test coverage, including tests/storage/test_writer_flush.py, tests/contract/test_memory_search.py.

Milestone 8: Commit and reopen equivalent disk search¶

Recorded file scope: - Changed: src/minilucene/writer.py - Changed: src/minilucene/index.py - Added: src/minilucene/reader.py - Added: tests/storage/test_commit_recovery.py - Added: tests/contract/test_disk_search.py

Recorded activity 1 — Test intent: old-or-new crash tests

Inject failures at: segment data write, segment metadata write, segment rename, manifest temp write, and manifest replace. For each failure reopen the index and assert it yields the last successfully published manifest only.

def test_complete_segment_without_manifest_is_ignored_after_reopen(index):
    with index.writer() as writer:
        writer.add_document(id="1", body="orphan")
        writer.flush()
    reopened = type(index).open(index.path)
    assert reopened.open_reader().max_doc == 0

Recorded activity 2 — Test intent: equivalence contract

Build the same corpus in MemoryIndex and disk Index, commit, reopen, search term/phrase/prefix/boolean queries, and compare total hits, stored fields, scores with pytest.approx, and final order.

Recorded activity 3 — Verification intent: RED

Historical verification covered targeted or full test coverage, including tests/storage/test_commit_recovery.py, tests/contract/test_disk_search.py.

Recorded activity 4 — Design outcome: commit

Commit flushes, verifies all referenced segments, creates the next immutable manifest, publishes it atomically, then updates writer committed state. A failed manifest publication leaves the prior committed manifest authoritative and keeps the writer open for explicit retry or close.

Recorded activity 5 — Design outcome: committed reader

Index.open_reader() loads exactly the manifest's segments, constructs one multi-segment reader view, computes global corpus statistics, and searches through the Phase 1 scorer/collector path. It never scans unreferenced directories.

Recorded activity 6 — Verification intent: GREEN and phase verification

Historical verification covered targeted or full test coverage, static analysis, bytecode compilation, diff hygiene, including tests/storage/test_commit_recovery.py, tests/contract/test_disk_search.py.

Milestone 9: Accept immutable storage and commit¶

Recorded file scope: - Added: tests/acceptance/test_phase2_storage_commit.py - Added: docs/phase2-storage-commit.md

Recorded activity 1 — Design outcome: one restart acceptance

The recorded scope added two segments, commit, reopen through a fresh Index, and prove stored fields, phrase hits, BM25 scores, schema fingerprint, generation ordering, and absence of orphan visibility match the in-memory oracle.

Recorded activity 2 — Verification intent: phase acceptance

Historical verification covered targeted or full test coverage, static analysis, bytecode compilation, diff hygiene, including tests/acceptance/test_phase2_storage_commit.py.

Recorded activity 3 — Test intent: the phase report

docs/phase2-storage-commit.md records the segment format, manifest authority, flush/commit distinction, crash matrix, verification commands, and the absence of refresh, deletion, merge, query parsing, network, and vector behavior.

Recorded activity 4 — Recorded acceptance snapshot

Historical expected evidence: clean worktree.

MiniLucene Immutable Storage and Commit Design and Implementation History¶

Milestone 1: Freeze SegmentImage as the storage boundary¶

Milestone 2: Implement unsigned varints and delta sequences¶

Milestone 3: Encode terms, postings, stored fields, and norms¶

Milestone 4: Publish checksummed immutable segment directories¶

Milestone 5: Make the manifest the only committed root¶

Milestone 6: Add index creation, schema persistence, and writer exclusion¶

Milestone 7: Implement deterministic flush without publication¶

Milestone 8: Commit and reopen equivalent disk search¶

Milestone 9: Accept immutable storage and commit¶

Milestone 1: Freeze `SegmentImage` as the storage boundary¶