# `mars::vfs` — the layered virtual filesystem `src/mars/vfs/` reimplements the engine's file layer: the `.gob` archives (plain ZIP files) stacked under a loose-file override directory, searched in order, with case-insensitive, separator-agnostic paths. Library target: `mars_vfs` (static, `src/mars/vfs/CMakeLists.txt`). Include as `mars/vfs/
.h`. C++17, no exceptions cross the API; DEFLATE and CRC-32 come from the vendored [miniz 3.1.2](../third_party/miniz/) (MIT, upstream release copied verbatim; built with the archive/stdio/zlib-name surfaces compiled out). The behaviour spec is the RE notes (`sots-re/findings/subsystems/data-model.md` §1–2 and the VFS paragraph of `strings-and-config.md`): the engine's `gobio` layer is a ZIP filesystem with a native-directory override, `sots.gob` and `sots_local_en.gob` are registered and searched in order, and every entry in both is stored uncompressed. ## API ```cpp namespace mars::vfs { // result.h struct Error { enum class Code { NotFound, Io, BadArchive, Corrupt, Unsupported }; Code code; std::string message; }; template class Result; // value() or error(); explicit operator bool // path.h std::string normalize_key(std::string_view); // "Species\\Human\\X.txt" -> "species/human/x.txt" std::string normalize_name(std::string_view); // same, original case kept // zip_archive.h struct ZipEntry { std::string name, key; uint64_t local_header_offset; uint32_t compressed_size, uncompressed_size, crc32; uint16_t method, flags, dos_time, dos_date; bool is_directory; }; class ZipArchive { static Result open(const std::string& path); const std::vector& entries() const; size_t file_count() const; std::string comment() const; const ZipEntry* find(std::string_view rel) const; // files only, any spelling Result> read(const ZipEntry&) const; Result> read(std::string_view rel) const; }; // vfs.h enum class MountKind { Native, Zip }; enum class Order { NativeFirst, Registration }; struct MountInfo { MountId id; MountKind kind; std::string path; size_t file_count; }; struct Stat { std::string name; uint64_t size; MountId mount; MountKind kind; bool compressed; }; struct ListEntry { std::string name; uint64_t size; MountId mount; }; class Vfs { explicit Vfs(Order order = Order::NativeFirst); Result mount_zip(const std::string& archive); Result mount_native(const std::string& directory); Result rescan_native(MountId); const std::vector& mounts() const; std::vector search_order() const; const ZipArchive* archive(MountId) const; bool exists(std::string_view rel) const; std::optional stat(std::string_view rel) const; Result> read(std::string_view rel) const; Result read_text(std::string_view rel) const; std::vector list(std::string_view prefix = {}) const; }; } ``` ## Container format as implemented A `.gob` is a ZIP file in the classic 32-bit layout (PKWARE APPNOTE 4.x): ``` [local file header][name][extra][data] one per entry, in file order ... [central directory header][name][extra][comment] one per entry ... [end of central directory record][comment] last thing in the file ``` `ZipArchive::open`: 1. Reads the file's tail (at most 22 + 65535 bytes) and scans backwards for the EOCD signature `50 4B 05 06`. A candidate is accepted when its comment length fits inside the tail; trailing junk after a comment-less record is therefore tolerated, a record whose declared comment overruns the file is not. 2. From the EOCD: disk numbers (must be 0 / single disk), entry count, central-directory size and offset, comment. `0xFFFF` / `0xFFFFFFFF` sentinel values mean ZIP64 → `Unsupported`. The directory must lie before the EOCD → otherwise `BadArchive`. 3. Reads the whole central directory (≈600 KB for the 8,352-entry `sots.gob`) and walks the 46-byte headers: flags, method, DOS time/date, CRC-32, sizes, name, local-header offset. Any truncated or mis-signed header → `BadArchive`; a per-entry ZIP64 sentinel → `Unsupported`. 4. Names ending in `/` (or `\`) are directory placeholders: kept in `entries()`, excluded from `file_count()`, `find()` and `list()`. Every entry gets a normalised key; if two entries fold to the same key the first in the directory wins. `ZipArchive::read(entry)` seeks to the entry's local header, checks its signature `50 4B 03 04`, and locates the data using the *local* header's name/extra lengths (they may differ from the central copy's — one unit test covers that). Sizes and CRC come from the central directory, which is authoritative even when bit 3 (data descriptor) is set. Method 0 is a single positioned read of `uncompressed_size` bytes; method 8 reads `compressed_size` bytes and inflates with `tinfl_decompress_mem_to_mem` (raw DEFLATE, no zlib header) into an exactly-sized buffer. Every read verifies the CRC-32 (`Corrupt` on mismatch), refuses encrypted entries and other methods (`Unsupported`), and rejects a stored entry whose two sizes disagree (`Corrupt`). Not implemented, by design: ZIP64, multi-disk, encryption, methods other than 0/8, the extra-field time stamps (raw DOS time/date are exposed on the entry untranslated). ## Override rules `Vfs` holds a list of mounts. Lookups (`exists`, `stat`, `read`, `list`) walk `search_order()` and take the first mount that has the key: | `Order` | search order | |----------------|----------------------------------------------------------------| | `NativeFirst` | every native mount in registration order, then every zip mount in registration order — the rule the RE notes describe (a loose file on disk beats the archived copy; archives are consulted in the order registered) | | `Registration` | strictly the order `mount_*` was called | Exact load order in the original is still an open question in the notes, so the policy is a constructor parameter rather than baked in. Under either policy `stat().mount` and `ListEntry::mount` say which mount won. A native mount is indexed once at mount time (recursive walk, regular files only) so lookups are case-insensitive even on a case-sensitive host filesystem. Two on-disk names that fold to one key resolve to the lexicographically first spelling. `rescan_native(id)` re-walks the directory. `..` segments are never resolved, so a lookup cannot escape the mount. `list(prefix)` is a plain prefix match on normalised keys (`""` = everything, `"Weapons/"` = a directory, `"Weapons/_"` = a name stem). Each key appears once, attributed to the winning mount, with that mount's spelling of the name, sorted by key. ### Path normalisation (`path.h`) `/` and `\` both separate; empty and `.` segments and leading/trailing separators are dropped; `A`–`Z` fold to lower case; every other byte (including cp1252 high bytes) is untouched. So `.\Species\HUMAN//sections/CRAIC.SHIPSECTION` finds `Species/Human/sections/CRAIC.shipsection`. ## Performance notes - Opening `sots.gob` (1.45 GB) + `sots_local_en.gob` (599 MB) takes ~5 ms warm: two small tail reads plus one central-directory read each, then an `unordered_map` of ~10k keys. The archives are never loaded whole; each `read` is one positioned `fread` of exactly that entry (a `FILE*` guarded by a mutex, so a shared archive is safe to read from several threads). - Reading and CRC-verifying every entry of both archives (2.15 GB) takes ~4 s from page cache (`SOTS_GOB_FULL=1` in the real-data test), i.e. the reader is I/O-bound; CRC-32 is miniz's table implementation. - `list()` is a linear scan over every mount's entries (fine at 10k entries; a sorted key index would make prefix queries logarithmic if it ever matters). - Memory: the central-directory index (name + key + fixed fields per entry) is ≈2 MB for both archives; the native index is proportional to the number of loose files. ## Tests `tests/mars_vfs/build_and_run.sh` (plain g++, `-Wall -Wextra -Werror`) or the optional `tests/mars_vfs/CMakeLists.txt` (`mars_vfs_unit`, `mars_vfs_realdata`). Unit tests (`unit_tests.cpp`, 13 cases) build ZIPs in-process with an independent minimal writer (`zip_builder.h`: stored entries, directory placeholders, and raw pre-packed records for fabricating deflate / encrypted / odd-method entries). Covered: index + case/separator-insensitive lookup, local-vs-central extra lengths, trailing comment and appended junk, duplicate names, corrupt EOCD (missing / truncated / bad signature / bad directory offset / bad directory signature / ZIP64 sentinel), corrupt entries (flipped data byte → CRC, bad local signature, encrypted, method 12, stored size mismatch, garbage deflate stream), override precedence under both `Order`s, `list` union/dedup/prefix/spelling, native mount case folding and rescan, `..` containment, and the empty VFS. The one deflated fixture is produced by Python's `zipfile` (`make_fixture.py`, an independent compressor) at build time; that case SKIPs when Python is absent. Real-data test (`realdata_test.cpp`) runs when `SOTS_GOB_DIR` holds the two archives (`SOTS_GOB_ORACLE_DIR` the `unzip -l` listings, default the same dir; `SOTS_GOB_FULL=1` adds the read-everything pass) and SKIPs otherwise. ### Oracle results (2026-09-07, owner's archives) | check | ours | oracle | |---|---|---| | `sots.gob` central-directory entries | 8,352 (100 dirs, 8,252 files) | `unzip -l`: 8,352 | | `sots_local_en.gob` entries | 2,035 (19 dirs, 2,016 files) | `unzip -l`: 2,035 | | every oracle name present with equal size, no duplicates | yes | — | | entries using a compression method | 0 | notes: all stored | | `TechTree/MasterTechList.tech` → `mars_parse` | 293 `tech` blocks, 62,234 bytes | 293 | | `Locale/EN/Strings.csv` → `mars_text` | 5,722 raw records / 5,200 data rows | 5,722 | | `*.weapon` via `list()` | 207 (123 under `Weapons/`, 84 under `Species/_NPC/`) | 207 | | loose-file override of `MasterTechList.tech` | native wins (`NativeFirst`); zip wins with native registered last under `Registration` | notes' rule | | read + CRC-verify all 10,268 files (2.15 GB) | 0 errors, ~4 s | — | | byte-equality vs `unzip -p` (asked with upper-case + backslash spelling) | `MasterTechList.tech`, `_weapons.txt`, `globals.txt`, `AI_AV.tga`, `CRAIC.shipsection`, `Strings.csv` all identical | — | The notes' "8,354 / 2,037 entries" are the `unzip -l` line counts including the footer; the archives hold 8,352 / 2,035 directory records. ## Open questions - **Load order in the original.** The notes only establish "native override, archives searched in order"; whether `sots_local_en.gob` is consulted before or after `sots.gob`, and whether a `Mods/` directory is one native mount or several, is unknown. `Order` and mount registration order keep both answers expressible. - **Directory placeholders.** Whether the engine ever asks "does directory X exist" is unknown; `exists()` currently answers files only. - **Native-mount freshness.** The index is a snapshot; if the engine expects files dropped into the game directory mid-session to appear, callers must `rescan_native()`. - **Time stamps.** DOS time/date are exposed raw; nothing in the notes says the engine reads them.