sots-engine/docs/mars-vfs.md

mars::vfs — the layered virtual filesystem

src/mars/vfs/ reimplements the engine's file layer: the .gob archives (plain ZIP files) stacked under a loose-file override directory, searched in order, with case-insensitive, separator-agnostic paths. Library target: mars_vfs (static, src/mars/vfs/CMakeLists.txt). Include as mars/vfs/<header>.h. C++17, no exceptions cross the API; DEFLATE and CRC-32 come from the vendored miniz 3.1.2 (MIT, upstream release copied verbatim; built with the archive/stdio/zlib-name surfaces compiled out).

The behaviour spec is the RE notes (sots-re/findings/subsystems/data-model.md §1–2 and the VFS paragraph of strings-and-config.md): the engine's gobio layer is a ZIP filesystem with a native-directory override, sots.gob and sots_local_en.gob are registered and searched in order, and every entry in both is stored uncompressed.

API

namespace mars::vfs {

// result.h
struct Error { enum class Code { NotFound, Io, BadArchive, Corrupt, Unsupported }; Code code; std::string message; };
template <class T> class Result;             // value() or error(); explicit operator bool

// path.h
std::string normalize_key(std::string_view);   // "Species\\Human\\X.txt" -> "species/human/x.txt"
std::string normalize_name(std::string_view);  // same, original case kept

// zip_archive.h
struct ZipEntry { std::string name, key; uint64_t local_header_offset; uint32_t compressed_size,
                  uncompressed_size, crc32; uint16_t method, flags, dos_time, dos_date; bool is_directory; };
class ZipArchive {
    static Result<ZipArchive> open(const std::string& path);
    const std::vector<ZipEntry>& entries() const;  size_t file_count() const;  std::string comment() const;
    const ZipEntry* find(std::string_view rel) const;               // files only, any spelling
    Result<std::vector<uint8_t>> read(const ZipEntry&) const;
    Result<std::vector<uint8_t>> read(std::string_view rel) const;
};

// vfs.h
enum class MountKind { Native, Zip };
enum class Order { NativeFirst, Registration };
struct MountInfo { MountId id; MountKind kind; std::string path; size_t file_count; };
struct Stat      { std::string name; uint64_t size; MountId mount; MountKind kind; bool compressed; };
struct ListEntry { std::string name; uint64_t size; MountId mount; };
class Vfs {
    explicit Vfs(Order order = Order::NativeFirst);
    Result<MountId> mount_zip(const std::string& archive);
    Result<MountId> mount_native(const std::string& directory);
    Result<size_t>  rescan_native(MountId);
    const std::vector<MountInfo>& mounts() const;  std::vector<MountId> search_order() const;
    const ZipArchive* archive(MountId) const;
    bool exists(std::string_view rel) const;
    std::optional<Stat> stat(std::string_view rel) const;
    Result<std::vector<uint8_t>> read(std::string_view rel) const;
    Result<std::string> read_text(std::string_view rel) const;
    std::vector<ListEntry> list(std::string_view prefix = {}) const;
};
}

Container format as implemented

A .gob is a ZIP file in the classic 32-bit layout (PKWARE APPNOTE 4.x):

[local file header][name][extra][data]      one per entry, in file order
...
[central directory header][name][extra][comment]   one per entry
...
[end of central directory record][comment]  last thing in the file

ZipArchive::open:

1. Reads the file's tail (at most 22 + 65535 bytes) and scans backwards for the EOCD signature 50 4B 05 06. A candidate is accepted when its comment length fits inside the tail; trailing junk after a comment-less record is therefore tolerated, a record whose declared comment overruns the file is not.
2. From the EOCD: disk numbers (must be 0 / single disk), entry count, central-directory size and offset, comment. 0xFFFF / 0xFFFFFFFF sentinel values mean ZIP64 → Unsupported. The directory must lie before the EOCD → otherwise BadArchive.
3. Reads the whole central directory (≈600 KB for the 8,352-entry sots.gob) and walks the 46-byte headers: flags, method, DOS time/date, CRC-32, sizes, name, local-header offset. Any truncated or mis-signed header → BadArchive; a per-entry ZIP64 sentinel → Unsupported.
4. Names ending in / (or \) are directory placeholders: kept in entries(), excluded from file_count(), find() and list(). Every entry gets a normalised key; if two entries fold to the same key the first in the directory wins.

ZipArchive::read(entry) seeks to the entry's local header, checks its signature 50 4B 03 04, and locates the data using the local header's name/extra lengths (they may differ from the central copy's — one unit test covers that). Sizes and CRC come from the central directory, which is authoritative even when bit 3 (data descriptor) is set. Method 0 is a single positioned read of uncompressed_size bytes; method 8 reads compressed_size bytes and inflates with tinfl_decompress_mem_to_mem (raw DEFLATE, no zlib header) into an exactly-sized buffer. Every read verifies the CRC-32 (Corrupt on mismatch), refuses encrypted entries and other methods (Unsupported), and rejects a stored entry whose two sizes disagree (Corrupt).

Not implemented, by design: ZIP64, multi-disk, encryption, methods other than 0/8, the extra-field time stamps (raw DOS time/date are exposed on the entry untranslated).

Override rules

Vfs holds a list of mounts. Lookups (exists, stat, read, list) walk search_order() and take the first mount that has the key:

Order	search order
NativeFirst	every native mount in registration order, then every zip mount in registration order — the rule the RE notes describe (a loose file on disk beats the archived copy; archives are consulted in the order registered)
Registration	strictly the order mount_* was called

Exact load order in the original is still an open question in the notes, so the policy is a constructor parameter rather than baked in. Under either policy stat().mount and ListEntry::mount say which mount won.

A native mount is indexed once at mount time (recursive walk, regular files only) so lookups are case-insensitive even on a case-sensitive host filesystem. Two on-disk names that fold to one key resolve to the lexicographically first spelling. rescan_native(id) re-walks the directory. .. segments are never resolved, so a lookup cannot escape the mount.

list(prefix) is a plain prefix match on normalised keys ("" = everything, "Weapons/" = a directory, "Weapons/_" = a name stem). Each key appears once, attributed to the winning mount, with that mount's spelling of the name, sorted by key.

Path normalisation (path.h)

/ and \ both separate; empty and . segments and leading/trailing separators are dropped; A–Z fold to lower case; every other byte (including cp1252 high bytes) is untouched. So .\Species\HUMAN//sections/CRAIC.SHIPSECTION finds Species/Human/sections/CRAIC.shipsection.

Performance notes

• Opening sots.gob (1.45 GB) + sots_local_en.gob (599 MB) takes ~5 ms warm: two small tail reads plus one central-directory read each, then an unordered_map of ~10k keys. The archives are never loaded whole; each read is one positioned fread of exactly that entry (a FILE* guarded by a mutex, so a shared archive is safe to read from several threads).
• Reading and CRC-verifying every entry of both archives (2.15 GB) takes ~4 s from page cache (SOTS_GOB_FULL=1 in the real-data test), i.e. the reader is I/O-bound; CRC-32 is miniz's table implementation.
• list() is a linear scan over every mount's entries (fine at 10k entries; a sorted key index would make prefix queries logarithmic if it ever matters).
• Memory: the central-directory index (name + key + fixed fields per entry) is ≈2 MB for both archives; the native index is proportional to the number of loose files.

Tests

tests/mars_vfs/build_and_run.sh (plain g++, -Wall -Wextra -Werror) or the optional tests/mars_vfs/CMakeLists.txt (mars_vfs_unit, mars_vfs_realdata).

Unit tests (unit_tests.cpp, 13 cases) build ZIPs in-process with an independent minimal writer (zip_builder.h: stored entries, directory placeholders, and raw pre-packed records for fabricating deflate / encrypted / odd-method entries). Covered: index + case/separator-insensitive lookup, local-vs-central extra lengths, trailing comment and appended junk, duplicate names, corrupt EOCD (missing / truncated / bad signature / bad directory offset / bad directory signature / ZIP64 sentinel), corrupt entries (flipped data byte → CRC, bad local signature, encrypted, method 12, stored size mismatch, garbage deflate stream), override precedence under both Orders, list union/dedup/prefix/spelling, native mount case folding and rescan, .. containment, and the empty VFS. The one deflated fixture is produced by Python's zipfile (make_fixture.py, an independent compressor) at build time; that case SKIPs when Python is absent.

Real-data test (realdata_test.cpp) runs when SOTS_GOB_DIR holds the two archives (SOTS_GOB_ORACLE_DIR the unzip -l listings, default the same dir; SOTS_GOB_FULL=1 adds the read-everything pass) and SKIPs otherwise.

Oracle results (2026-09-07, owner's archives)

check	ours	oracle
sots.gob central-directory entries	8,352 (100 dirs, 8,252 files)	unzip -l: 8,352
sots_local_en.gob entries	2,035 (19 dirs, 2,016 files)	unzip -l: 2,035
every oracle name present with equal size, no duplicates	yes	—
entries using a compression method	0	notes: all stored
TechTree/MasterTechList.tech → mars_parse	293 tech blocks, 62,234 bytes	293
Locale/EN/Strings.csv → mars_text	5,722 raw records / 5,200 data rows	5,722
*.weapon via list()	207 (123 under Weapons/, 84 under Species/_NPC/)	207
loose-file override of MasterTechList.tech	native wins (NativeFirst); zip wins with native registered last under Registration	notes' rule
read + CRC-verify all 10,268 files (2.15 GB)	0 errors, ~4 s	—
byte-equality vs unzip -p (asked with upper-case + backslash spelling)	MasterTechList.tech, _weapons.txt, globals.txt, AI_AV.tga, CRAIC.shipsection, Strings.csv all identical	—

The notes' "8,354 / 2,037 entries" are the unzip -l line counts including the footer; the archives hold 8,352 / 2,035 directory records.

Open questions

• Load order in the original. The notes only establish "native override, archives searched in order"; whether sots_local_en.gob is consulted before or after sots.gob, and whether a Mods/ directory is one native mount or several, is unknown. Order and mount registration order keep both answers expressible.
• Directory placeholders. Whether the engine ever asks "does directory X exist" is unknown; exists() currently answers files only.
• Native-mount freshness. The index is a snapshot; if the engine expects files dropped into the game directory mid-session to appear, callers must rescan_native().
• Time stamps. DOS time/date are exposed raw; nothing in the notes says the engine reads them.