216 lines
11 KiB
Markdown
216 lines
11 KiB
Markdown
# `mars::vfs` — the layered virtual filesystem
|
||
|
||
`src/mars/vfs/` reimplements the engine's file layer: the `.gob` archives
|
||
(plain ZIP files) stacked under a loose-file override directory, searched in
|
||
order, with case-insensitive, separator-agnostic paths. Library target:
|
||
`mars_vfs` (static, `src/mars/vfs/CMakeLists.txt`). Include as
|
||
`mars/vfs/<header>.h`. C++17, no exceptions cross the API; DEFLATE and
|
||
CRC-32 come from the vendored [miniz 3.1.2](../third_party/miniz/) (MIT,
|
||
upstream release copied verbatim; built with the archive/stdio/zlib-name
|
||
surfaces compiled out).
|
||
|
||
The behaviour spec is the RE notes (`sots-re/findings/subsystems/data-model.md`
|
||
§1–2 and the VFS paragraph of `strings-and-config.md`): the engine's `gobio`
|
||
layer is a ZIP filesystem with a native-directory override, `sots.gob` and
|
||
`sots_local_en.gob` are registered and searched in order, and every entry
|
||
in both is stored uncompressed.
|
||
|
||
## API
|
||
|
||
```cpp
|
||
namespace mars::vfs {
|
||
|
||
// result.h
|
||
struct Error { enum class Code { NotFound, Io, BadArchive, Corrupt, Unsupported }; Code code; std::string message; };
|
||
template <class T> class Result; // value() or error(); explicit operator bool
|
||
|
||
// path.h
|
||
std::string normalize_key(std::string_view); // "Species\\Human\\X.txt" -> "species/human/x.txt"
|
||
std::string normalize_name(std::string_view); // same, original case kept
|
||
|
||
// zip_archive.h
|
||
struct ZipEntry { std::string name, key; uint64_t local_header_offset; uint32_t compressed_size,
|
||
uncompressed_size, crc32; uint16_t method, flags, dos_time, dos_date; bool is_directory; };
|
||
class ZipArchive {
|
||
static Result<ZipArchive> open(const std::string& path);
|
||
const std::vector<ZipEntry>& entries() const; size_t file_count() const; std::string comment() const;
|
||
const ZipEntry* find(std::string_view rel) const; // files only, any spelling
|
||
Result<std::vector<uint8_t>> read(const ZipEntry&) const;
|
||
Result<std::vector<uint8_t>> read(std::string_view rel) const;
|
||
};
|
||
|
||
// vfs.h
|
||
enum class MountKind { Native, Zip };
|
||
enum class Order { NativeFirst, Registration };
|
||
struct MountInfo { MountId id; MountKind kind; std::string path; size_t file_count; };
|
||
struct Stat { std::string name; uint64_t size; MountId mount; MountKind kind; bool compressed; };
|
||
struct ListEntry { std::string name; uint64_t size; MountId mount; };
|
||
class Vfs {
|
||
explicit Vfs(Order order = Order::NativeFirst);
|
||
Result<MountId> mount_zip(const std::string& archive);
|
||
Result<MountId> mount_native(const std::string& directory);
|
||
Result<size_t> rescan_native(MountId);
|
||
const std::vector<MountInfo>& mounts() const; std::vector<MountId> search_order() const;
|
||
const ZipArchive* archive(MountId) const;
|
||
bool exists(std::string_view rel) const;
|
||
std::optional<Stat> stat(std::string_view rel) const;
|
||
Result<std::vector<uint8_t>> read(std::string_view rel) const;
|
||
Result<std::string> read_text(std::string_view rel) const;
|
||
std::vector<ListEntry> list(std::string_view prefix = {}) const;
|
||
};
|
||
}
|
||
```
|
||
|
||
## Container format as implemented
|
||
|
||
A `.gob` is a ZIP file in the classic 32-bit layout (PKWARE APPNOTE 4.x):
|
||
|
||
```
|
||
[local file header][name][extra][data] one per entry, in file order
|
||
...
|
||
[central directory header][name][extra][comment] one per entry
|
||
...
|
||
[end of central directory record][comment] last thing in the file
|
||
```
|
||
|
||
`ZipArchive::open`:
|
||
|
||
1. Reads the file's tail (at most 22 + 65535 bytes) and scans backwards for
|
||
the EOCD signature `50 4B 05 06`. A candidate is accepted when its comment
|
||
length fits inside the tail; trailing junk after a comment-less record is
|
||
therefore tolerated, a record whose declared comment overruns the file is
|
||
not.
|
||
2. From the EOCD: disk numbers (must be 0 / single disk), entry count,
|
||
central-directory size and offset, comment. `0xFFFF` / `0xFFFFFFFF`
|
||
sentinel values mean ZIP64 → `Unsupported`. The directory must lie
|
||
before the EOCD → otherwise `BadArchive`.
|
||
3. Reads the whole central directory (≈600 KB for the 8,352-entry
|
||
`sots.gob`) and walks the 46-byte headers: flags, method, DOS time/date,
|
||
CRC-32, sizes, name, local-header offset. Any truncated or mis-signed
|
||
header → `BadArchive`; a per-entry ZIP64 sentinel → `Unsupported`.
|
||
4. Names ending in `/` (or `\`) are directory placeholders: kept in
|
||
`entries()`, excluded from `file_count()`, `find()` and `list()`.
|
||
Every entry gets a normalised key; if two entries fold to the same key the
|
||
first in the directory wins.
|
||
|
||
`ZipArchive::read(entry)` seeks to the entry's local header, checks its
|
||
signature `50 4B 03 04`, and locates the data using the *local* header's
|
||
name/extra lengths (they may differ from the central copy's — one unit test
|
||
covers that). Sizes and CRC come from the central directory, which is
|
||
authoritative even when bit 3 (data descriptor) is set. Method 0 is a single
|
||
positioned read of `uncompressed_size` bytes; method 8 reads
|
||
`compressed_size` bytes and inflates with `tinfl_decompress_mem_to_mem` (raw
|
||
DEFLATE, no zlib header) into an exactly-sized buffer. Every read verifies
|
||
the CRC-32 (`Corrupt` on mismatch), refuses encrypted entries and other
|
||
methods (`Unsupported`), and rejects a stored entry whose two sizes disagree
|
||
(`Corrupt`).
|
||
|
||
Not implemented, by design: ZIP64, multi-disk, encryption, methods other than
|
||
0/8, the extra-field time stamps (raw DOS time/date are exposed on the entry
|
||
untranslated).
|
||
|
||
## Override rules
|
||
|
||
`Vfs` holds a list of mounts. Lookups (`exists`, `stat`, `read`, `list`) walk
|
||
`search_order()` and take the first mount that has the key:
|
||
|
||
| `Order` | search order |
|
||
|----------------|----------------------------------------------------------------|
|
||
| `NativeFirst` | every native mount in registration order, then every zip mount in registration order — the rule the RE notes describe (a loose file on disk beats the archived copy; archives are consulted in the order registered) |
|
||
| `Registration` | strictly the order `mount_*` was called |
|
||
|
||
Exact load order in the original is still an open question in the notes, so
|
||
the policy is a constructor parameter rather than baked in. Under either
|
||
policy `stat().mount` and `ListEntry::mount` say which mount won.
|
||
|
||
A native mount is indexed once at mount time (recursive walk, regular files
|
||
only) so lookups are case-insensitive even on a case-sensitive host
|
||
filesystem. Two on-disk names that fold to one key resolve to the
|
||
lexicographically first spelling. `rescan_native(id)` re-walks the directory.
|
||
`..` segments are never resolved, so a lookup cannot escape the mount.
|
||
|
||
`list(prefix)` is a plain prefix match on normalised keys (`""` = everything,
|
||
`"Weapons/"` = a directory, `"Weapons/_"` = a name stem). Each key appears
|
||
once, attributed to the winning mount, with that mount's spelling of the
|
||
name, sorted by key.
|
||
|
||
### Path normalisation (`path.h`)
|
||
|
||
`/` and `\` both separate; empty and `.` segments and leading/trailing
|
||
separators are dropped; `A`–`Z` fold to lower case; every other byte
|
||
(including cp1252 high bytes) is untouched. So
|
||
`.\Species\HUMAN//sections/CRAIC.SHIPSECTION` finds
|
||
`Species/Human/sections/CRAIC.shipsection`.
|
||
|
||
## Performance notes
|
||
|
||
- Opening `sots.gob` (1.45 GB) + `sots_local_en.gob` (599 MB) takes ~5 ms
|
||
warm: two small tail reads plus one central-directory read each, then an
|
||
`unordered_map` of ~10k keys. The archives are never loaded whole; each
|
||
`read` is one positioned `fread` of exactly that entry (a `FILE*` guarded
|
||
by a mutex, so a shared archive is safe to read from several threads).
|
||
- Reading and CRC-verifying every entry of both archives (2.15 GB) takes
|
||
~4 s from page cache (`SOTS_GOB_FULL=1` in the real-data test), i.e. the
|
||
reader is I/O-bound; CRC-32 is miniz's table implementation.
|
||
- `list()` is a linear scan over every mount's entries (fine at 10k entries;
|
||
a sorted key index would make prefix queries logarithmic if it ever
|
||
matters).
|
||
- Memory: the central-directory index (name + key + fixed fields per entry)
|
||
is ≈2 MB for both archives; the native index is proportional to the number
|
||
of loose files.
|
||
|
||
## Tests
|
||
|
||
`tests/mars_vfs/build_and_run.sh` (plain g++, `-Wall -Wextra -Werror`) or the
|
||
optional `tests/mars_vfs/CMakeLists.txt` (`mars_vfs_unit`, `mars_vfs_realdata`).
|
||
|
||
Unit tests (`unit_tests.cpp`, 13 cases) build ZIPs in-process with an
|
||
independent minimal writer (`zip_builder.h`: stored entries, directory
|
||
placeholders, and raw pre-packed records for fabricating deflate / encrypted
|
||
/ odd-method entries). Covered: index + case/separator-insensitive lookup,
|
||
local-vs-central extra lengths, trailing comment and appended junk,
|
||
duplicate names, corrupt EOCD (missing / truncated / bad signature / bad
|
||
directory offset / bad directory signature / ZIP64 sentinel), corrupt
|
||
entries (flipped data byte → CRC, bad local signature, encrypted, method 12,
|
||
stored size mismatch, garbage deflate stream), override precedence under both
|
||
`Order`s, `list` union/dedup/prefix/spelling, native mount case folding and
|
||
rescan, `..` containment, and the empty VFS. The one deflated fixture is
|
||
produced by Python's `zipfile` (`make_fixture.py`, an independent
|
||
compressor) at build time; that case SKIPs when Python is absent.
|
||
|
||
Real-data test (`realdata_test.cpp`) runs when `SOTS_GOB_DIR` holds the two
|
||
archives (`SOTS_GOB_ORACLE_DIR` the `unzip -l` listings, default the same
|
||
dir; `SOTS_GOB_FULL=1` adds the read-everything pass) and SKIPs otherwise.
|
||
|
||
### Oracle results (2026-09-07, owner's archives)
|
||
|
||
| check | ours | oracle |
|
||
|---|---|---|
|
||
| `sots.gob` central-directory entries | 8,352 (100 dirs, 8,252 files) | `unzip -l`: 8,352 |
|
||
| `sots_local_en.gob` entries | 2,035 (19 dirs, 2,016 files) | `unzip -l`: 2,035 |
|
||
| every oracle name present with equal size, no duplicates | yes | — |
|
||
| entries using a compression method | 0 | notes: all stored |
|
||
| `TechTree/MasterTechList.tech` → `mars_parse` | 293 `tech` blocks, 62,234 bytes | 293 |
|
||
| `Locale/EN/Strings.csv` → `mars_text` | 5,722 raw records / 5,200 data rows | 5,722 |
|
||
| `*.weapon` via `list()` | 207 (123 under `Weapons/`, 84 under `Species/_NPC/`) | 207 |
|
||
| loose-file override of `MasterTechList.tech` | native wins (`NativeFirst`); zip wins with native registered last under `Registration` | notes' rule |
|
||
| read + CRC-verify all 10,268 files (2.15 GB) | 0 errors, ~4 s | — |
|
||
| byte-equality vs `unzip -p` (asked with upper-case + backslash spelling) | `MasterTechList.tech`, `_weapons.txt`, `globals.txt`, `AI_AV.tga`, `CRAIC.shipsection`, `Strings.csv` all identical | — |
|
||
|
||
The notes' "8,354 / 2,037 entries" are the `unzip -l` line counts including
|
||
the footer; the archives hold 8,352 / 2,035 directory records.
|
||
|
||
## Open questions
|
||
|
||
- **Load order in the original.** The notes only establish "native override,
|
||
archives searched in order"; whether `sots_local_en.gob` is consulted
|
||
before or after `sots.gob`, and whether a `Mods/` directory is one native
|
||
mount or several, is unknown. `Order` and mount registration order keep
|
||
both answers expressible.
|
||
- **Directory placeholders.** Whether the engine ever asks "does directory X
|
||
exist" is unknown; `exists()` currently answers files only.
|
||
- **Native-mount freshness.** The index is a snapshot; if the engine expects
|
||
files dropped into the game directory mid-session to appear, callers must
|
||
`rescan_native()`.
|
||
- **Time stamps.** DOS time/date are exposed raw; nothing in the notes says
|
||
the engine reads them.
|