merge mars/vfs
This commit is contained in:
alex
commit
08a129e453
23 changed files with 11875 additions and 0 deletions
216
docs/mars-vfs.md
Normal file
216
docs/mars-vfs.md
Normal file
|
|
@ -0,0 +1,216 @@
|
|||
# `mars::vfs` — the layered virtual filesystem
|
||||
|
||||
`src/mars/vfs/` reimplements the engine's file layer: the `.gob` archives
|
||||
(plain ZIP files) stacked under a loose-file override directory, searched in
|
||||
order, with case-insensitive, separator-agnostic paths. Library target:
|
||||
`mars_vfs` (static, `src/mars/vfs/CMakeLists.txt`). Include as
|
||||
`mars/vfs/<header>.h`. C++17, no exceptions cross the API; DEFLATE and
|
||||
CRC-32 come from the vendored [miniz 3.1.2](../third_party/miniz/) (MIT,
|
||||
upstream release copied verbatim; built with the archive/stdio/zlib-name
|
||||
surfaces compiled out).
|
||||
|
||||
The behaviour spec is the RE notes (`sots-re/findings/subsystems/data-model.md`
|
||||
§1–2 and the VFS paragraph of `strings-and-config.md`): the engine's `gobio`
|
||||
layer is a ZIP filesystem with a native-directory override, `sots.gob` and
|
||||
`sots_local_en.gob` are registered and searched in order, and every entry
|
||||
in both is stored uncompressed.
|
||||
|
||||
## API
|
||||
|
||||
```cpp
|
||||
namespace mars::vfs {
|
||||
|
||||
// result.h
|
||||
struct Error { enum class Code { NotFound, Io, BadArchive, Corrupt, Unsupported }; Code code; std::string message; };
|
||||
template <class T> class Result; // value() or error(); explicit operator bool
|
||||
|
||||
// path.h
|
||||
std::string normalize_key(std::string_view); // "Species\\Human\\X.txt" -> "species/human/x.txt"
|
||||
std::string normalize_name(std::string_view); // same, original case kept
|
||||
|
||||
// zip_archive.h
|
||||
struct ZipEntry { std::string name, key; uint64_t local_header_offset; uint32_t compressed_size,
|
||||
uncompressed_size, crc32; uint16_t method, flags, dos_time, dos_date; bool is_directory; };
|
||||
class ZipArchive {
|
||||
static Result<ZipArchive> open(const std::string& path);
|
||||
const std::vector<ZipEntry>& entries() const; size_t file_count() const; std::string comment() const;
|
||||
const ZipEntry* find(std::string_view rel) const; // files only, any spelling
|
||||
Result<std::vector<uint8_t>> read(const ZipEntry&) const;
|
||||
Result<std::vector<uint8_t>> read(std::string_view rel) const;
|
||||
};
|
||||
|
||||
// vfs.h
|
||||
enum class MountKind { Native, Zip };
|
||||
enum class Order { NativeFirst, Registration };
|
||||
struct MountInfo { MountId id; MountKind kind; std::string path; size_t file_count; };
|
||||
struct Stat { std::string name; uint64_t size; MountId mount; MountKind kind; bool compressed; };
|
||||
struct ListEntry { std::string name; uint64_t size; MountId mount; };
|
||||
class Vfs {
|
||||
explicit Vfs(Order order = Order::NativeFirst);
|
||||
Result<MountId> mount_zip(const std::string& archive);
|
||||
Result<MountId> mount_native(const std::string& directory);
|
||||
Result<size_t> rescan_native(MountId);
|
||||
const std::vector<MountInfo>& mounts() const; std::vector<MountId> search_order() const;
|
||||
const ZipArchive* archive(MountId) const;
|
||||
bool exists(std::string_view rel) const;
|
||||
std::optional<Stat> stat(std::string_view rel) const;
|
||||
Result<std::vector<uint8_t>> read(std::string_view rel) const;
|
||||
Result<std::string> read_text(std::string_view rel) const;
|
||||
std::vector<ListEntry> list(std::string_view prefix = {}) const;
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
## Container format as implemented
|
||||
|
||||
A `.gob` is a ZIP file in the classic 32-bit layout (PKWARE APPNOTE 4.x):
|
||||
|
||||
```
|
||||
[local file header][name][extra][data] one per entry, in file order
|
||||
...
|
||||
[central directory header][name][extra][comment] one per entry
|
||||
...
|
||||
[end of central directory record][comment] last thing in the file
|
||||
```
|
||||
|
||||
`ZipArchive::open`:
|
||||
|
||||
1. Reads the file's tail (at most 22 + 65535 bytes) and scans backwards for
|
||||
the EOCD signature `50 4B 05 06`. A candidate is accepted when its comment
|
||||
length fits inside the tail; trailing junk after a comment-less record is
|
||||
therefore tolerated, a record whose declared comment overruns the file is
|
||||
not.
|
||||
2. From the EOCD: disk numbers (must be 0 / single disk), entry count,
|
||||
central-directory size and offset, comment. `0xFFFF` / `0xFFFFFFFF`
|
||||
sentinel values mean ZIP64 → `Unsupported`. The directory must lie
|
||||
before the EOCD → otherwise `BadArchive`.
|
||||
3. Reads the whole central directory (≈600 KB for the 8,352-entry
|
||||
`sots.gob`) and walks the 46-byte headers: flags, method, DOS time/date,
|
||||
CRC-32, sizes, name, local-header offset. Any truncated or mis-signed
|
||||
header → `BadArchive`; a per-entry ZIP64 sentinel → `Unsupported`.
|
||||
4. Names ending in `/` (or `\`) are directory placeholders: kept in
|
||||
`entries()`, excluded from `file_count()`, `find()` and `list()`.
|
||||
Every entry gets a normalised key; if two entries fold to the same key the
|
||||
first in the directory wins.
|
||||
|
||||
`ZipArchive::read(entry)` seeks to the entry's local header, checks its
|
||||
signature `50 4B 03 04`, and locates the data using the *local* header's
|
||||
name/extra lengths (they may differ from the central copy's — one unit test
|
||||
covers that). Sizes and CRC come from the central directory, which is
|
||||
authoritative even when bit 3 (data descriptor) is set. Method 0 is a single
|
||||
positioned read of `uncompressed_size` bytes; method 8 reads
|
||||
`compressed_size` bytes and inflates with `tinfl_decompress_mem_to_mem` (raw
|
||||
DEFLATE, no zlib header) into an exactly-sized buffer. Every read verifies
|
||||
the CRC-32 (`Corrupt` on mismatch), refuses encrypted entries and other
|
||||
methods (`Unsupported`), and rejects a stored entry whose two sizes disagree
|
||||
(`Corrupt`).
|
||||
|
||||
Not implemented, by design: ZIP64, multi-disk, encryption, methods other than
|
||||
0/8, the extra-field time stamps (raw DOS time/date are exposed on the entry
|
||||
untranslated).
|
||||
|
||||
## Override rules
|
||||
|
||||
`Vfs` holds a list of mounts. Lookups (`exists`, `stat`, `read`, `list`) walk
|
||||
`search_order()` and take the first mount that has the key:
|
||||
|
||||
| `Order` | search order |
|
||||
|----------------|----------------------------------------------------------------|
|
||||
| `NativeFirst` | every native mount in registration order, then every zip mount in registration order — the rule the RE notes describe (a loose file on disk beats the archived copy; archives are consulted in the order registered) |
|
||||
| `Registration` | strictly the order `mount_*` was called |
|
||||
|
||||
Exact load order in the original is still an open question in the notes, so
|
||||
the policy is a constructor parameter rather than baked in. Under either
|
||||
policy `stat().mount` and `ListEntry::mount` say which mount won.
|
||||
|
||||
A native mount is indexed once at mount time (recursive walk, regular files
|
||||
only) so lookups are case-insensitive even on a case-sensitive host
|
||||
filesystem. Two on-disk names that fold to one key resolve to the
|
||||
lexicographically first spelling. `rescan_native(id)` re-walks the directory.
|
||||
`..` segments are never resolved, so a lookup cannot escape the mount.
|
||||
|
||||
`list(prefix)` is a plain prefix match on normalised keys (`""` = everything,
|
||||
`"Weapons/"` = a directory, `"Weapons/_"` = a name stem). Each key appears
|
||||
once, attributed to the winning mount, with that mount's spelling of the
|
||||
name, sorted by key.
|
||||
|
||||
### Path normalisation (`path.h`)
|
||||
|
||||
`/` and `\` both separate; empty and `.` segments and leading/trailing
|
||||
separators are dropped; `A`–`Z` fold to lower case; every other byte
|
||||
(including cp1252 high bytes) is untouched. So
|
||||
`.\Species\HUMAN//sections/CRAIC.SHIPSECTION` finds
|
||||
`Species/Human/sections/CRAIC.shipsection`.
|
||||
|
||||
## Performance notes
|
||||
|
||||
- Opening `sots.gob` (1.45 GB) + `sots_local_en.gob` (599 MB) takes ~5 ms
|
||||
warm: two small tail reads plus one central-directory read each, then an
|
||||
`unordered_map` of ~10k keys. The archives are never loaded whole; each
|
||||
`read` is one positioned `fread` of exactly that entry (a `FILE*` guarded
|
||||
by a mutex, so a shared archive is safe to read from several threads).
|
||||
- Reading and CRC-verifying every entry of both archives (2.15 GB) takes
|
||||
~4 s from page cache (`SOTS_GOB_FULL=1` in the real-data test), i.e. the
|
||||
reader is I/O-bound; CRC-32 is miniz's table implementation.
|
||||
- `list()` is a linear scan over every mount's entries (fine at 10k entries;
|
||||
a sorted key index would make prefix queries logarithmic if it ever
|
||||
matters).
|
||||
- Memory: the central-directory index (name + key + fixed fields per entry)
|
||||
is ≈2 MB for both archives; the native index is proportional to the number
|
||||
of loose files.
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/mars_vfs/build_and_run.sh` (plain g++, `-Wall -Wextra -Werror`) or the
|
||||
optional `tests/mars_vfs/CMakeLists.txt` (`mars_vfs_unit`, `mars_vfs_realdata`).
|
||||
|
||||
Unit tests (`unit_tests.cpp`, 13 cases) build ZIPs in-process with an
|
||||
independent minimal writer (`zip_builder.h`: stored entries, directory
|
||||
placeholders, and raw pre-packed records for fabricating deflate / encrypted
|
||||
/ odd-method entries). Covered: index + case/separator-insensitive lookup,
|
||||
local-vs-central extra lengths, trailing comment and appended junk,
|
||||
duplicate names, corrupt EOCD (missing / truncated / bad signature / bad
|
||||
directory offset / bad directory signature / ZIP64 sentinel), corrupt
|
||||
entries (flipped data byte → CRC, bad local signature, encrypted, method 12,
|
||||
stored size mismatch, garbage deflate stream), override precedence under both
|
||||
`Order`s, `list` union/dedup/prefix/spelling, native mount case folding and
|
||||
rescan, `..` containment, and the empty VFS. The one deflated fixture is
|
||||
produced by Python's `zipfile` (`make_fixture.py`, an independent
|
||||
compressor) at build time; that case SKIPs when Python is absent.
|
||||
|
||||
Real-data test (`realdata_test.cpp`) runs when `SOTS_GOB_DIR` holds the two
|
||||
archives (`SOTS_GOB_ORACLE_DIR` the `unzip -l` listings, default the same
|
||||
dir; `SOTS_GOB_FULL=1` adds the read-everything pass) and SKIPs otherwise.
|
||||
|
||||
### Oracle results (2026-09-07, owner's archives)
|
||||
|
||||
| check | ours | oracle |
|
||||
|---|---|---|
|
||||
| `sots.gob` central-directory entries | 8,352 (100 dirs, 8,252 files) | `unzip -l`: 8,352 |
|
||||
| `sots_local_en.gob` entries | 2,035 (19 dirs, 2,016 files) | `unzip -l`: 2,035 |
|
||||
| every oracle name present with equal size, no duplicates | yes | — |
|
||||
| entries using a compression method | 0 | notes: all stored |
|
||||
| `TechTree/MasterTechList.tech` → `mars_parse` | 293 `tech` blocks, 62,234 bytes | 293 |
|
||||
| `Locale/EN/Strings.csv` → `mars_text` | 5,722 raw records / 5,200 data rows | 5,722 |
|
||||
| `*.weapon` via `list()` | 207 (123 under `Weapons/`, 84 under `Species/_NPC/`) | 207 |
|
||||
| loose-file override of `MasterTechList.tech` | native wins (`NativeFirst`); zip wins with native registered last under `Registration` | notes' rule |
|
||||
| read + CRC-verify all 10,268 files (2.15 GB) | 0 errors, ~4 s | — |
|
||||
| byte-equality vs `unzip -p` (asked with upper-case + backslash spelling) | `MasterTechList.tech`, `_weapons.txt`, `globals.txt`, `AI_AV.tga`, `CRAIC.shipsection`, `Strings.csv` all identical | — |
|
||||
|
||||
The notes' "8,354 / 2,037 entries" are the `unzip -l` line counts including
|
||||
the footer; the archives hold 8,352 / 2,035 directory records.
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Load order in the original.** The notes only establish "native override,
|
||||
archives searched in order"; whether `sots_local_en.gob` is consulted
|
||||
before or after `sots.gob`, and whether a `Mods/` directory is one native
|
||||
mount or several, is unknown. `Order` and mount registration order keep
|
||||
both answers expressible.
|
||||
- **Directory placeholders.** Whether the engine ever asks "does directory X
|
||||
exist" is unknown; `exists()` currently answers files only.
|
||||
- **Native-mount freshness.** The index is a snapshot; if the engine expects
|
||||
files dropped into the game directory mid-session to appear, callers must
|
||||
`rescan_native()`.
|
||||
- **Time stamps.** DOS time/date are exposed raw; nothing in the notes says
|
||||
the engine reads them.
|
||||
26
src/mars/vfs/CMakeLists.txt
Normal file
26
src/mars/vfs/CMakeLists.txt
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
# mars_vfs -- the layered virtual filesystem: .gob (ZIP) archives under a
|
||||
# loose-file override directory. Include from the root with
|
||||
# add_subdirectory(src/mars/vfs); link mars_vfs. Headers are mars/vfs/*.h.
|
||||
#
|
||||
# DEFLATE + CRC-32 come from the vendored miniz (third_party/miniz, MIT).
|
||||
# The target is created here only if nobody else has already (another
|
||||
# module may vendor and build the same file).
|
||||
if(NOT TARGET miniz)
|
||||
add_library(miniz STATIC ${CMAKE_CURRENT_SOURCE_DIR}/../../../third_party/miniz/miniz.c)
|
||||
target_include_directories(miniz PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/../../../third_party/miniz)
|
||||
# inflate + crc only: no archive/zlib-name surface, no stdio, no time.
|
||||
target_compile_definitions(miniz PUBLIC
|
||||
MINIZ_NO_ARCHIVE_APIS MINIZ_NO_ARCHIVE_WRITING_APIS MINIZ_NO_STDIO MINIZ_NO_TIME
|
||||
MINIZ_NO_ZLIB_COMPATIBLE_NAMES)
|
||||
set_target_properties(miniz PROPERTIES POSITION_INDEPENDENT_CODE ON)
|
||||
endif()
|
||||
|
||||
add_library(mars_vfs STATIC
|
||||
path.cpp
|
||||
zip_archive.cpp
|
||||
vfs.cpp
|
||||
)
|
||||
target_include_directories(mars_vfs PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/../..)
|
||||
target_link_libraries(mars_vfs PRIVATE miniz)
|
||||
target_compile_features(mars_vfs PUBLIC cxx_std_17)
|
||||
target_compile_options(mars_vfs PRIVATE -Wall -Wextra -Werror)
|
||||
47
src/mars/vfs/path.cpp
Normal file
47
src/mars/vfs/path.cpp
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
#include "mars/vfs/path.h"
|
||||
|
||||
#include "mars/vfs/result.h"
|
||||
|
||||
namespace mars::vfs {
|
||||
|
||||
namespace {
|
||||
|
||||
std::string normalize(std::string_view path, bool fold) {
|
||||
std::string out;
|
||||
out.reserve(path.size());
|
||||
std::size_t i = 0;
|
||||
while (i < path.size()) {
|
||||
// Take one segment.
|
||||
std::size_t j = i;
|
||||
while (j < path.size() && path[j] != '/' && path[j] != '\\') ++j;
|
||||
std::string_view seg = path.substr(i, j - i);
|
||||
if (!seg.empty() && seg != ".") {
|
||||
if (!out.empty()) out.push_back('/');
|
||||
for (char c : seg) out.push_back(fold ? fold_char(c) : c);
|
||||
}
|
||||
i = j + 1;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
std::string normalize_key(std::string_view path) { return normalize(path, true); }
|
||||
std::string normalize_name(std::string_view path) { return normalize(path, false); }
|
||||
|
||||
bool key_has_prefix(std::string_view key, std::string_view prefix) {
|
||||
return key.size() >= prefix.size() && key.compare(0, prefix.size(), prefix) == 0;
|
||||
}
|
||||
|
||||
const char* to_string(Error::Code code) {
|
||||
switch (code) {
|
||||
case Error::Code::NotFound: return "not found";
|
||||
case Error::Code::Io: return "i/o error";
|
||||
case Error::Code::BadArchive: return "bad archive";
|
||||
case Error::Code::Corrupt: return "corrupt entry";
|
||||
case Error::Code::Unsupported: return "unsupported";
|
||||
}
|
||||
return "?";
|
||||
}
|
||||
|
||||
} // namespace mars::vfs
|
||||
34
src/mars/vfs/path.h
Normal file
34
src/mars/vfs/path.h
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
// mars::vfs -- path normalisation.
|
||||
//
|
||||
// Every path the VFS sees (a lookup, an archive entry name, a loose file's
|
||||
// path relative to its mount) is reduced to one canonical *key* so that the
|
||||
// game's habits all resolve to the same thing:
|
||||
//
|
||||
// * '\' and '/' are both separators Species\Human\sections == Species/Human/sections
|
||||
// * ASCII case is ignored weapons/GAUSS.weapon == Weapons/Gauss.weapon
|
||||
// * empty and '.' segments are dropped ./Data//globals.txt == Data/globals.txt
|
||||
// * leading and trailing separators are dropped /Data/ == Data
|
||||
//
|
||||
// '..' segments are kept literally (they never match an indexed entry, so a
|
||||
// lookup cannot escape a native mount). Bytes >= 0x80 pass through untouched:
|
||||
// only 'A'..'Z' fold, which matches how the shipped file names behave.
|
||||
#pragma once
|
||||
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
|
||||
namespace mars::vfs {
|
||||
|
||||
// Canonical, case-folded lookup key.
|
||||
std::string normalize_key(std::string_view path);
|
||||
|
||||
// Same normalisation but keeps the original spelling (for display / listing).
|
||||
std::string normalize_name(std::string_view path);
|
||||
|
||||
// ASCII-only lowercase of one byte.
|
||||
inline char fold_char(char c) { return (c >= 'A' && c <= 'Z') ? static_cast<char>(c - 'A' + 'a') : c; }
|
||||
|
||||
// True when `key` starts with `prefix` (both already normalised).
|
||||
bool key_has_prefix(std::string_view key, std::string_view prefix);
|
||||
|
||||
} // namespace mars::vfs
|
||||
49
src/mars/vfs/result.h
Normal file
49
src/mars/vfs/result.h
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
// mars::vfs -- error/result types shared by the virtual filesystem.
|
||||
// No exceptions cross this API: every fallible call returns Result<T>.
|
||||
#pragma once
|
||||
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
|
||||
namespace mars::vfs {
|
||||
|
||||
struct Error {
|
||||
enum class Code {
|
||||
NotFound, // no mount has the path
|
||||
Io, // open/seek/read failed on the host filesystem
|
||||
BadArchive, // not a ZIP we can index: EOCD missing, central directory malformed
|
||||
Corrupt, // an entry's bytes disagree with its header (size / CRC / inflate failure)
|
||||
Unsupported, // ZIP feature we do not implement (ZIP64, encryption, other methods)
|
||||
};
|
||||
|
||||
Code code = Code::Io;
|
||||
std::string message;
|
||||
};
|
||||
|
||||
template <class T>
|
||||
class Result {
|
||||
public:
|
||||
Result(T v) : value_(std::move(v)) {} // NOLINT(google-explicit-constructor)
|
||||
Result(Error e) : error_(std::move(e)) {} // NOLINT(google-explicit-constructor)
|
||||
|
||||
bool ok() const { return value_.has_value(); }
|
||||
explicit operator bool() const { return ok(); }
|
||||
|
||||
T& value() & { return *value_; }
|
||||
const T& value() const& { return *value_; }
|
||||
T&& value() && { return std::move(*value_); }
|
||||
|
||||
T* operator->() { return &*value_; }
|
||||
const T* operator->() const { return &*value_; }
|
||||
|
||||
const Error& error() const { return error_; }
|
||||
|
||||
private:
|
||||
std::optional<T> value_;
|
||||
Error error_;
|
||||
};
|
||||
|
||||
const char* to_string(Error::Code code);
|
||||
|
||||
} // namespace mars::vfs
|
||||
246
src/mars/vfs/vfs.cpp
Normal file
246
src/mars/vfs/vfs.cpp
Normal file
|
|
@ -0,0 +1,246 @@
|
|||
#include "mars/vfs/vfs.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cstdio>
|
||||
#include <filesystem>
|
||||
#include <map>
|
||||
#include <system_error>
|
||||
#include <unordered_map>
|
||||
|
||||
#include "mars/vfs/path.h"
|
||||
|
||||
namespace fs = std::filesystem;
|
||||
|
||||
namespace mars::vfs {
|
||||
|
||||
namespace {
|
||||
|
||||
struct NativeFile {
|
||||
std::string name; // relative path, '/'-separated, as spelled on disk
|
||||
fs::path full; // absolute/mount-joined path used to open it
|
||||
std::uint64_t size = 0;
|
||||
};
|
||||
|
||||
struct NativeMount {
|
||||
fs::path root;
|
||||
std::vector<NativeFile> files;
|
||||
std::unordered_map<std::string, std::size_t> index; // key -> files[]
|
||||
|
||||
Result<std::size_t> scan() {
|
||||
files.clear();
|
||||
index.clear();
|
||||
std::error_code ec;
|
||||
if (!fs::is_directory(root, ec)) return Error{Error::Code::Io, root.string() + " is not a directory"};
|
||||
|
||||
std::vector<NativeFile> found;
|
||||
fs::recursive_directory_iterator it(root, fs::directory_options::skip_permission_denied, ec);
|
||||
if (ec) return Error{Error::Code::Io, root.string() + ": " + ec.message()};
|
||||
for (const fs::recursive_directory_iterator end; it != end; it.increment(ec)) {
|
||||
if (ec) return Error{Error::Code::Io, root.string() + ": " + ec.message()};
|
||||
const fs::directory_entry& de = *it;
|
||||
std::error_code ec2;
|
||||
if (!de.is_regular_file(ec2)) continue;
|
||||
NativeFile f;
|
||||
f.full = de.path();
|
||||
f.name = normalize_name(fs::relative(de.path(), root, ec2).generic_string());
|
||||
if (ec2) continue;
|
||||
f.size = de.file_size(ec2);
|
||||
if (ec2) continue;
|
||||
found.push_back(std::move(f));
|
||||
}
|
||||
// Deterministic precedence when two on-disk names fold to one key:
|
||||
// the lexicographically first spelling wins.
|
||||
std::sort(found.begin(), found.end(), [](const NativeFile& a, const NativeFile& b) { return a.name < b.name; });
|
||||
for (auto& f : found) {
|
||||
const std::string key = normalize_key(f.name);
|
||||
if (index.count(key)) continue;
|
||||
index.emplace(key, files.size());
|
||||
files.push_back(std::move(f));
|
||||
}
|
||||
return files.size();
|
||||
}
|
||||
};
|
||||
|
||||
struct Mount {
|
||||
MountInfo info;
|
||||
std::unique_ptr<ZipArchive> zip; // kind == Zip
|
||||
std::unique_ptr<NativeMount> native; // kind == Native
|
||||
};
|
||||
|
||||
Result<std::vector<std::uint8_t>> read_native(const NativeFile& f) {
|
||||
std::FILE* fp = std::fopen(f.full.string().c_str(), "rb");
|
||||
if (!fp) return Error{Error::Code::Io, "cannot open " + f.full.string()};
|
||||
std::vector<std::uint8_t> out;
|
||||
std::uint8_t buf[1 << 16];
|
||||
for (;;) {
|
||||
const std::size_t n = std::fread(buf, 1, sizeof buf, fp);
|
||||
out.insert(out.end(), buf, buf + n);
|
||||
if (n < sizeof buf) break;
|
||||
}
|
||||
const bool bad = std::ferror(fp) != 0;
|
||||
std::fclose(fp);
|
||||
if (bad) return Error{Error::Code::Io, "read error on " + f.full.string()};
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
struct Vfs::Impl {
|
||||
Order order;
|
||||
std::vector<Mount> mounts;
|
||||
std::vector<MountInfo> infos; // mirrors mounts[].info for the public accessor
|
||||
|
||||
void refresh_infos() {
|
||||
infos.clear();
|
||||
for (const Mount& m : mounts) infos.push_back(m.info);
|
||||
}
|
||||
|
||||
std::vector<MountId> search_order() const {
|
||||
std::vector<MountId> ids;
|
||||
if (order == Order::NativeFirst) {
|
||||
for (const Mount& m : mounts)
|
||||
if (m.info.kind == MountKind::Native) ids.push_back(m.info.id);
|
||||
for (const Mount& m : mounts)
|
||||
if (m.info.kind == MountKind::Zip) ids.push_back(m.info.id);
|
||||
} else {
|
||||
for (const Mount& m : mounts) ids.push_back(m.info.id);
|
||||
}
|
||||
return ids;
|
||||
}
|
||||
|
||||
// The winning mount for a key, plus what it found there.
|
||||
struct Hit {
|
||||
const Mount* mount = nullptr;
|
||||
const ZipEntry* zip = nullptr;
|
||||
const NativeFile* native = nullptr;
|
||||
};
|
||||
|
||||
Hit resolve(std::string_view rel) const {
|
||||
const std::string key = normalize_key(rel);
|
||||
for (MountId id : search_order()) {
|
||||
const Mount& m = mounts[static_cast<std::size_t>(id)];
|
||||
if (m.zip) {
|
||||
if (const ZipEntry* e = m.zip->find(key)) return Hit{&m, e, nullptr};
|
||||
} else if (m.native) {
|
||||
auto it = m.native->index.find(key);
|
||||
if (it != m.native->index.end()) return Hit{&m, nullptr, &m.native->files[it->second]};
|
||||
}
|
||||
}
|
||||
return {};
|
||||
}
|
||||
};
|
||||
|
||||
Vfs::Vfs(Order order) : impl_(std::make_unique<Impl>()) { impl_->order = order; }
|
||||
Vfs::~Vfs() = default;
|
||||
Vfs::Vfs(Vfs&&) noexcept = default;
|
||||
Vfs& Vfs::operator=(Vfs&&) noexcept = default;
|
||||
|
||||
Order Vfs::order() const { return impl_->order; }
|
||||
|
||||
Result<MountId> Vfs::mount_zip(const std::string& archive_path) {
|
||||
Result<ZipArchive> z = ZipArchive::open(archive_path);
|
||||
if (!z) return z.error();
|
||||
Mount m;
|
||||
m.info.id = static_cast<MountId>(impl_->mounts.size());
|
||||
m.info.kind = MountKind::Zip;
|
||||
m.info.path = archive_path;
|
||||
m.info.file_count = z->file_count();
|
||||
m.zip = std::make_unique<ZipArchive>(std::move(z).value());
|
||||
impl_->mounts.push_back(std::move(m));
|
||||
impl_->refresh_infos();
|
||||
return impl_->mounts.back().info.id;
|
||||
}
|
||||
|
||||
Result<MountId> Vfs::mount_native(const std::string& directory) {
|
||||
auto nm = std::make_unique<NativeMount>();
|
||||
nm->root = fs::path(directory);
|
||||
Result<std::size_t> n = nm->scan();
|
||||
if (!n) return n.error();
|
||||
Mount m;
|
||||
m.info.id = static_cast<MountId>(impl_->mounts.size());
|
||||
m.info.kind = MountKind::Native;
|
||||
m.info.path = directory;
|
||||
m.info.file_count = n.value();
|
||||
m.native = std::move(nm);
|
||||
impl_->mounts.push_back(std::move(m));
|
||||
impl_->refresh_infos();
|
||||
return impl_->mounts.back().info.id;
|
||||
}
|
||||
|
||||
Result<std::size_t> Vfs::rescan_native(MountId id) {
|
||||
if (id < 0 || static_cast<std::size_t>(id) >= impl_->mounts.size() || !impl_->mounts[static_cast<std::size_t>(id)].native)
|
||||
return Error{Error::Code::NotFound, "mount " + std::to_string(id) + " is not a native mount"};
|
||||
Mount& m = impl_->mounts[static_cast<std::size_t>(id)];
|
||||
Result<std::size_t> n = m.native->scan();
|
||||
if (n) {
|
||||
m.info.file_count = n.value();
|
||||
impl_->refresh_infos();
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
const std::vector<MountInfo>& Vfs::mounts() const { return impl_->infos; }
|
||||
std::vector<MountId> Vfs::search_order() const { return impl_->search_order(); }
|
||||
|
||||
const ZipArchive* Vfs::archive(MountId id) const {
|
||||
if (id < 0 || static_cast<std::size_t>(id) >= impl_->mounts.size()) return nullptr;
|
||||
return impl_->mounts[static_cast<std::size_t>(id)].zip.get();
|
||||
}
|
||||
|
||||
bool Vfs::exists(std::string_view rel) const { return impl_->resolve(rel).mount != nullptr; }
|
||||
|
||||
std::optional<Stat> Vfs::stat(std::string_view rel) const {
|
||||
const Impl::Hit hit = impl_->resolve(rel);
|
||||
if (!hit.mount) return std::nullopt;
|
||||
Stat s;
|
||||
s.mount = hit.mount->info.id;
|
||||
s.kind = hit.mount->info.kind;
|
||||
if (hit.zip) {
|
||||
s.name = normalize_name(hit.zip->name);
|
||||
s.size = hit.zip->uncompressed_size;
|
||||
s.compressed = hit.zip->method != 0;
|
||||
} else {
|
||||
s.name = hit.native->name;
|
||||
s.size = hit.native->size;
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
Result<std::vector<std::uint8_t>> Vfs::read(std::string_view rel) const {
|
||||
const Impl::Hit hit = impl_->resolve(rel);
|
||||
if (!hit.mount) return Error{Error::Code::NotFound, std::string(rel) + " not found in any mount"};
|
||||
if (hit.zip) return hit.mount->zip->read(*hit.zip);
|
||||
return read_native(*hit.native);
|
||||
}
|
||||
|
||||
Result<std::string> Vfs::read_text(std::string_view rel) const {
|
||||
Result<std::vector<std::uint8_t>> r = read(rel);
|
||||
if (!r) return r.error();
|
||||
return std::string(r->begin(), r->end());
|
||||
}
|
||||
|
||||
std::vector<ListEntry> Vfs::list(std::string_view prefix) const {
|
||||
const std::string pfx = normalize_key(prefix);
|
||||
std::map<std::string, ListEntry> seen; // key -> winner; map keeps the output sorted
|
||||
for (MountId id : impl_->search_order()) {
|
||||
const Mount& m = impl_->mounts[static_cast<std::size_t>(id)];
|
||||
if (m.zip) {
|
||||
for (const ZipEntry& e : m.zip->entries()) {
|
||||
if (e.is_directory || !key_has_prefix(e.key, pfx)) continue;
|
||||
seen.emplace(e.key, ListEntry{normalize_name(e.name), e.uncompressed_size, id});
|
||||
}
|
||||
} else if (m.native) {
|
||||
for (const auto& [key, idx] : m.native->index) {
|
||||
if (!key_has_prefix(key, pfx)) continue;
|
||||
const NativeFile& f = m.native->files[idx];
|
||||
seen.emplace(key, ListEntry{f.name, f.size, id});
|
||||
}
|
||||
}
|
||||
}
|
||||
std::vector<ListEntry> out;
|
||||
out.reserve(seen.size());
|
||||
for (auto& kv : seen) out.push_back(std::move(kv.second));
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace mars::vfs
|
||||
95
src/mars/vfs/vfs.h
Normal file
95
src/mars/vfs/vfs.h
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
// mars::vfs -- the layered virtual filesystem the engine reads assets through.
|
||||
//
|
||||
// The engine's file layer is a ZIP filesystem (the .gob archives) with a
|
||||
// native-directory override: a loose file on disk wins over the archived
|
||||
// copy, so extracted assets and mods replace shipped ones without repacking.
|
||||
// Several archives are registered (sots.gob, sots_local_en.gob) and searched
|
||||
// in order.
|
||||
//
|
||||
// Mounts are searched in the order given by `Order`:
|
||||
// NativeFirst every native mount (registration order), then every zip
|
||||
// mount (registration order) -- the documented rule
|
||||
// Registration strictly the order the mounts were added
|
||||
// The first mount that has the path wins for exists/read/stat/list.
|
||||
//
|
||||
// Paths are case-insensitive and accept '/' or '\' (see path.h). A native
|
||||
// mount is indexed once at mount time (recursively) so that lookups are
|
||||
// case-insensitive even on a case-sensitive host filesystem; call
|
||||
// rescan_native() after adding files to a mounted directory.
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
#include <vector>
|
||||
|
||||
#include "mars/vfs/result.h"
|
||||
#include "mars/vfs/zip_archive.h"
|
||||
|
||||
namespace mars::vfs {
|
||||
|
||||
enum class MountKind { Native, Zip };
|
||||
enum class Order { NativeFirst, Registration };
|
||||
|
||||
using MountId = int;
|
||||
|
||||
struct MountInfo {
|
||||
MountId id = -1;
|
||||
MountKind kind = MountKind::Zip;
|
||||
std::string path; // archive file or directory as given to mount_*
|
||||
std::size_t file_count = 0;
|
||||
};
|
||||
|
||||
struct Stat {
|
||||
std::string name; // the entry's spelling in the winning mount, '/'-separated
|
||||
std::uint64_t size = 0; // uncompressed size
|
||||
MountId mount = -1;
|
||||
MountKind kind = MountKind::Zip;
|
||||
bool compressed = false; // zip: deflate; native: always false
|
||||
};
|
||||
|
||||
struct ListEntry {
|
||||
std::string name; // relative path, original spelling of the winning mount
|
||||
std::uint64_t size = 0;
|
||||
MountId mount = -1;
|
||||
};
|
||||
|
||||
class Vfs {
|
||||
public:
|
||||
explicit Vfs(Order order = Order::NativeFirst);
|
||||
~Vfs();
|
||||
Vfs(Vfs&&) noexcept;
|
||||
Vfs& operator=(Vfs&&) noexcept;
|
||||
Vfs(const Vfs&) = delete;
|
||||
Vfs& operator=(const Vfs&) = delete;
|
||||
|
||||
Order order() const;
|
||||
|
||||
// Register an archive / a loose-file directory. Returns the mount id.
|
||||
Result<MountId> mount_zip(const std::string& archive_path);
|
||||
Result<MountId> mount_native(const std::string& directory);
|
||||
// Re-index a native mount (files added/removed on disk). Returns the file count.
|
||||
Result<std::size_t> rescan_native(MountId id);
|
||||
|
||||
const std::vector<MountInfo>& mounts() const;
|
||||
std::vector<MountId> search_order() const;
|
||||
const ZipArchive* archive(MountId id) const; // nullptr unless a zip mount
|
||||
|
||||
bool exists(std::string_view rel) const;
|
||||
std::optional<Stat> stat(std::string_view rel) const;
|
||||
Result<std::vector<std::uint8_t>> read(std::string_view rel) const;
|
||||
Result<std::string> read_text(std::string_view rel) const; // same bytes as a std::string
|
||||
|
||||
// Every file whose normalised path starts with `prefix` (normalised the same
|
||||
// way; "" lists everything, "Weapons/" a directory, "Weapons/_" a name stem).
|
||||
// Each path appears once, attributed to the mount that wins for it. Sorted by key.
|
||||
std::vector<ListEntry> list(std::string_view prefix = {}) const;
|
||||
|
||||
private:
|
||||
struct Impl;
|
||||
std::unique_ptr<Impl> impl_;
|
||||
};
|
||||
|
||||
} // namespace mars::vfs
|
||||
243
src/mars/vfs/zip_archive.cpp
Normal file
243
src/mars/vfs/zip_archive.cpp
Normal file
|
|
@ -0,0 +1,243 @@
|
|||
#include "mars/vfs/zip_archive.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cstdio>
|
||||
#include <cstring>
|
||||
#include <mutex>
|
||||
#include <unordered_map>
|
||||
|
||||
#include "mars/vfs/path.h"
|
||||
#include "miniz.h"
|
||||
|
||||
namespace mars::vfs {
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr std::uint32_t kLocalHeaderSig = 0x04034b50;
|
||||
constexpr std::uint32_t kCentralHeaderSig = 0x02014b50;
|
||||
constexpr std::uint32_t kEocdSig = 0x06054b50;
|
||||
constexpr std::size_t kEocdFixedSize = 22;
|
||||
constexpr std::size_t kEocdMaxSize = kEocdFixedSize + 0xFFFF; // fixed part + max comment
|
||||
constexpr std::size_t kCentralHeaderFixedSize = 46;
|
||||
constexpr std::size_t kLocalHeaderFixedSize = 30;
|
||||
|
||||
std::uint16_t rd16(const std::uint8_t* p) { return static_cast<std::uint16_t>(p[0] | (p[1] << 8)); }
|
||||
std::uint32_t rd32(const std::uint8_t* p) {
|
||||
return static_cast<std::uint32_t>(p[0]) | (static_cast<std::uint32_t>(p[1]) << 8) |
|
||||
(static_cast<std::uint32_t>(p[2]) << 16) | (static_cast<std::uint32_t>(p[3]) << 24);
|
||||
}
|
||||
|
||||
Error err(Error::Code code, std::string message) { return Error{code, std::move(message)}; }
|
||||
|
||||
// A read-only file with 64-bit positioned reads. Serialised by a mutex so a
|
||||
// shared archive can be read from several threads.
|
||||
class File {
|
||||
public:
|
||||
~File() {
|
||||
if (fp_) std::fclose(fp_);
|
||||
}
|
||||
|
||||
bool open(const std::string& path) {
|
||||
fp_ = std::fopen(path.c_str(), "rb");
|
||||
if (!fp_) return false;
|
||||
if (!seek_end() ) return false;
|
||||
size_ = tell();
|
||||
return size_ >= 0;
|
||||
}
|
||||
|
||||
std::int64_t size() const { return size_; }
|
||||
|
||||
bool read_at(std::uint64_t offset, void* dst, std::size_t n) {
|
||||
std::lock_guard<std::mutex> lock(mu_);
|
||||
if (n == 0) return true;
|
||||
if (offset > static_cast<std::uint64_t>(size_) || n > static_cast<std::uint64_t>(size_) - offset) return false;
|
||||
if (!seek_to(offset)) return false;
|
||||
return std::fread(dst, 1, n, fp_) == n;
|
||||
}
|
||||
|
||||
private:
|
||||
bool seek_to(std::uint64_t offset) {
|
||||
#if defined(_WIN32)
|
||||
return _fseeki64(fp_, static_cast<long long>(offset), SEEK_SET) == 0;
|
||||
#else
|
||||
return fseeko(fp_, static_cast<off_t>(offset), SEEK_SET) == 0;
|
||||
#endif
|
||||
}
|
||||
bool seek_end() {
|
||||
#if defined(_WIN32)
|
||||
return _fseeki64(fp_, 0, SEEK_END) == 0;
|
||||
#else
|
||||
return fseeko(fp_, 0, SEEK_END) == 0;
|
||||
#endif
|
||||
}
|
||||
std::int64_t tell() {
|
||||
#if defined(_WIN32)
|
||||
return _ftelli64(fp_);
|
||||
#else
|
||||
return static_cast<std::int64_t>(ftello(fp_));
|
||||
#endif
|
||||
}
|
||||
|
||||
std::FILE* fp_ = nullptr;
|
||||
std::int64_t size_ = -1;
|
||||
std::mutex mu_;
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
struct ZipArchive::Impl {
|
||||
std::string path;
|
||||
File file;
|
||||
std::vector<ZipEntry> entries;
|
||||
std::unordered_map<std::string, std::size_t> index; // key -> entries[] (files only, first wins)
|
||||
std::size_t file_count = 0;
|
||||
std::string comment;
|
||||
};
|
||||
|
||||
ZipArchive::ZipArchive(std::unique_ptr<Impl> impl) : impl_(std::move(impl)) {}
|
||||
ZipArchive::ZipArchive(ZipArchive&&) noexcept = default;
|
||||
ZipArchive& ZipArchive::operator=(ZipArchive&&) noexcept = default;
|
||||
ZipArchive::~ZipArchive() = default;
|
||||
|
||||
Result<ZipArchive> ZipArchive::open(const std::string& path) {
|
||||
auto impl = std::make_unique<Impl>();
|
||||
impl->path = path;
|
||||
if (!impl->file.open(path)) return err(Error::Code::Io, "cannot open " + path);
|
||||
const std::uint64_t file_size = static_cast<std::uint64_t>(impl->file.size());
|
||||
if (file_size < kEocdFixedSize) return err(Error::Code::BadArchive, path + ": too small to be a ZIP");
|
||||
|
||||
// --- end of central directory: scan backwards over the tail for the signature ---
|
||||
const std::size_t tail_len = static_cast<std::size_t>(std::min<std::uint64_t>(file_size, kEocdMaxSize));
|
||||
std::vector<std::uint8_t> tail(tail_len);
|
||||
const std::uint64_t tail_off = file_size - tail_len;
|
||||
if (!impl->file.read_at(tail_off, tail.data(), tail_len)) return err(Error::Code::Io, path + ": read failed");
|
||||
|
||||
std::size_t eocd_pos = tail_len; // sentinel: not found
|
||||
for (std::size_t i = tail_len - kEocdFixedSize + 1; i-- > 0;) {
|
||||
if (rd32(&tail[i]) != kEocdSig) continue;
|
||||
// The comment length must account for whatever follows; a shorter comment
|
||||
// (junk appended) is tolerated, a longer one is not this record.
|
||||
const std::uint16_t comment_len = rd16(&tail[i + 20]);
|
||||
if (i + kEocdFixedSize + comment_len <= tail_len) {
|
||||
eocd_pos = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (eocd_pos == tail_len) return err(Error::Code::BadArchive, path + ": end-of-central-directory record not found");
|
||||
|
||||
const std::uint8_t* e = &tail[eocd_pos];
|
||||
const std::uint16_t disk_no = rd16(e + 4);
|
||||
const std::uint16_t cd_disk = rd16(e + 6);
|
||||
const std::uint16_t entries_this_disk = rd16(e + 8);
|
||||
const std::uint16_t entries_total = rd16(e + 10);
|
||||
const std::uint32_t cd_size = rd32(e + 12);
|
||||
const std::uint32_t cd_offset = rd32(e + 16);
|
||||
const std::uint16_t comment_len = rd16(e + 20);
|
||||
impl->comment.assign(reinterpret_cast<const char*>(e + kEocdFixedSize), comment_len);
|
||||
|
||||
if (disk_no != 0 || cd_disk != 0 || entries_this_disk != entries_total)
|
||||
return err(Error::Code::Unsupported, path + ": multi-disk archives are not supported");
|
||||
if (entries_total == 0xFFFF || cd_size == 0xFFFFFFFFu || cd_offset == 0xFFFFFFFFu)
|
||||
return err(Error::Code::Unsupported, path + ": ZIP64 archives are not supported");
|
||||
const std::uint64_t eocd_abs = tail_off + eocd_pos;
|
||||
if (static_cast<std::uint64_t>(cd_offset) + cd_size > eocd_abs)
|
||||
return err(Error::Code::BadArchive, path + ": central directory lies outside the file");
|
||||
|
||||
// --- central directory ---
|
||||
std::vector<std::uint8_t> cd(cd_size);
|
||||
if (!impl->file.read_at(cd_offset, cd.data(), cd_size)) return err(Error::Code::Io, path + ": read failed");
|
||||
|
||||
impl->entries.reserve(entries_total);
|
||||
std::size_t pos = 0;
|
||||
for (std::uint32_t n = 0; n < entries_total; ++n) {
|
||||
if (pos + kCentralHeaderFixedSize > cd.size())
|
||||
return err(Error::Code::BadArchive, path + ": central directory truncated at entry " + std::to_string(n));
|
||||
const std::uint8_t* h = &cd[pos];
|
||||
if (rd32(h) != kCentralHeaderSig)
|
||||
return err(Error::Code::BadArchive, path + ": bad central header signature at entry " + std::to_string(n));
|
||||
ZipEntry ent;
|
||||
ent.flags = rd16(h + 8);
|
||||
ent.method = rd16(h + 10);
|
||||
ent.dos_time = rd16(h + 12);
|
||||
ent.dos_date = rd16(h + 14);
|
||||
ent.crc32 = rd32(h + 16);
|
||||
ent.compressed_size = rd32(h + 20);
|
||||
ent.uncompressed_size = rd32(h + 24);
|
||||
const std::uint16_t name_len = rd16(h + 28);
|
||||
const std::uint16_t extra_len = rd16(h + 30);
|
||||
const std::uint16_t comment_len2 = rd16(h + 32);
|
||||
ent.local_header_offset = rd32(h + 42);
|
||||
const std::size_t total = kCentralHeaderFixedSize + name_len + extra_len + comment_len2;
|
||||
if (pos + total > cd.size())
|
||||
return err(Error::Code::BadArchive, path + ": central directory truncated at entry " + std::to_string(n));
|
||||
ent.name.assign(reinterpret_cast<const char*>(h + kCentralHeaderFixedSize), name_len);
|
||||
if (ent.compressed_size == 0xFFFFFFFFu || ent.uncompressed_size == 0xFFFFFFFFu ||
|
||||
ent.local_header_offset == 0xFFFFFFFFu)
|
||||
return err(Error::Code::Unsupported, path + ": ZIP64 entry " + ent.name);
|
||||
if (ent.local_header_offset >= eocd_abs)
|
||||
return err(Error::Code::BadArchive, path + ": entry " + ent.name + " points outside the file");
|
||||
ent.is_directory = !ent.name.empty() && (ent.name.back() == '/' || ent.name.back() == '\\');
|
||||
ent.key = normalize_key(ent.name);
|
||||
pos += total;
|
||||
|
||||
const std::size_t idx = impl->entries.size();
|
||||
if (!ent.is_directory) {
|
||||
++impl->file_count;
|
||||
impl->index.emplace(ent.key, idx); // duplicate keys: first in the directory wins
|
||||
}
|
||||
impl->entries.push_back(std::move(ent));
|
||||
}
|
||||
return ZipArchive(std::move(impl));
|
||||
}
|
||||
|
||||
const std::string& ZipArchive::path() const { return impl_->path; }
|
||||
const std::vector<ZipEntry>& ZipArchive::entries() const { return impl_->entries; }
|
||||
std::size_t ZipArchive::file_count() const { return impl_->file_count; }
|
||||
std::string ZipArchive::comment() const { return impl_->comment; }
|
||||
|
||||
const ZipEntry* ZipArchive::find(std::string_view rel) const {
|
||||
auto it = impl_->index.find(normalize_key(rel));
|
||||
return it == impl_->index.end() ? nullptr : &impl_->entries[it->second];
|
||||
}
|
||||
|
||||
Result<std::vector<std::uint8_t>> ZipArchive::read(std::string_view rel) const {
|
||||
const ZipEntry* e = find(rel);
|
||||
if (!e) return err(Error::Code::NotFound, std::string(rel) + " not in " + impl_->path);
|
||||
return read(*e);
|
||||
}
|
||||
|
||||
Result<std::vector<std::uint8_t>> ZipArchive::read(const ZipEntry& entry) const {
|
||||
const std::string where = impl_->path + ": " + entry.name;
|
||||
if (entry.is_directory) return std::vector<std::uint8_t>{};
|
||||
if (entry.encrypted()) return err(Error::Code::Unsupported, where + " is encrypted");
|
||||
if (entry.method != 0 && entry.method != 8)
|
||||
return err(Error::Code::Unsupported, where + " uses compression method " + std::to_string(entry.method));
|
||||
if (entry.method == 0 && entry.compressed_size != entry.uncompressed_size)
|
||||
return err(Error::Code::Corrupt, where + ": stored entry with mismatched sizes");
|
||||
|
||||
// Local header: its own name/extra lengths locate the data (they may differ
|
||||
// from the central directory's), everything else comes from the directory.
|
||||
std::uint8_t lh[kLocalHeaderFixedSize];
|
||||
if (!impl_->file.read_at(entry.local_header_offset, lh, sizeof lh))
|
||||
return err(Error::Code::Io, where + ": cannot read local header");
|
||||
if (rd32(lh) != kLocalHeaderSig) return err(Error::Code::Corrupt, where + ": bad local header signature");
|
||||
const std::uint64_t data_off = entry.local_header_offset + kLocalHeaderFixedSize + rd16(lh + 26) + rd16(lh + 28);
|
||||
|
||||
std::vector<std::uint8_t> out(entry.uncompressed_size);
|
||||
if (entry.method == 0) {
|
||||
if (!impl_->file.read_at(data_off, out.data(), out.size()))
|
||||
return err(Error::Code::Io, where + ": cannot read entry data");
|
||||
} else {
|
||||
std::vector<std::uint8_t> packed(entry.compressed_size);
|
||||
if (!impl_->file.read_at(data_off, packed.data(), packed.size()))
|
||||
return err(Error::Code::Io, where + ": cannot read entry data");
|
||||
const std::size_t n = tinfl_decompress_mem_to_mem(out.data(), out.size(), packed.data(), packed.size(), 0);
|
||||
if (n == TINFL_DECOMPRESS_MEM_TO_MEM_FAILED || n != out.size())
|
||||
return err(Error::Code::Corrupt, where + ": inflate failed");
|
||||
}
|
||||
const std::uint32_t crc = static_cast<std::uint32_t>(mz_crc32(MZ_CRC32_INIT, out.data(), out.size()));
|
||||
if (crc != entry.crc32) return err(Error::Code::Corrupt, where + ": CRC mismatch");
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace mars::vfs
|
||||
74
src/mars/vfs/zip_archive.h
Normal file
74
src/mars/vfs/zip_archive.h
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
// mars::vfs -- one .gob archive, which is a plain ZIP file.
|
||||
//
|
||||
// Container as implemented (PKWARE APPNOTE, the classic 32-bit subset):
|
||||
//
|
||||
// [local header + name + extra + data]* one per entry, in file order
|
||||
// [central directory header]* the index; sizes/CRC/offsets here are authoritative
|
||||
// [end of central directory record] last thing in the file, optional trailing comment
|
||||
//
|
||||
// open() reads only the EOCD and the central directory (about 600 KB for the
|
||||
// 1.45 GB sots.gob) and builds a case-folded name index. Entry bytes are read
|
||||
// on demand with a seek + read of exactly that entry, so the archive is never
|
||||
// loaded whole. Stored (method 0) and Deflate (method 8) entries are
|
||||
// supported -- the shipped archives are entirely stored, but a modder's
|
||||
// re-zipped .gob is typically deflated. ZIP64 and encryption are refused
|
||||
// with Error::Code::Unsupported. Every read verifies the CRC-32.
|
||||
//
|
||||
// The archive is safe to read from several threads at once.
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
#include <vector>
|
||||
|
||||
#include "mars/vfs/result.h"
|
||||
|
||||
namespace mars::vfs {
|
||||
|
||||
struct ZipEntry {
|
||||
std::string name; // as stored in the central directory (original bytes)
|
||||
std::string key; // normalised, case-folded lookup key (see path.h)
|
||||
std::uint64_t local_header_offset = 0;
|
||||
std::uint32_t compressed_size = 0;
|
||||
std::uint32_t uncompressed_size = 0;
|
||||
std::uint32_t crc32 = 0;
|
||||
std::uint16_t method = 0; // 0 = stored, 8 = deflate
|
||||
std::uint16_t flags = 0; // general-purpose bit flags (bit 0 = encrypted)
|
||||
std::uint16_t dos_time = 0; // MS-DOS packed time/date, as stored
|
||||
std::uint16_t dos_date = 0;
|
||||
bool is_directory = false; // name ends in a separator (a directory placeholder)
|
||||
|
||||
bool encrypted() const { return (flags & 0x0001) != 0; }
|
||||
};
|
||||
|
||||
class ZipArchive {
|
||||
public:
|
||||
static Result<ZipArchive> open(const std::string& path);
|
||||
|
||||
ZipArchive(ZipArchive&&) noexcept;
|
||||
ZipArchive& operator=(ZipArchive&&) noexcept;
|
||||
~ZipArchive();
|
||||
ZipArchive(const ZipArchive&) = delete;
|
||||
ZipArchive& operator=(const ZipArchive&) = delete;
|
||||
|
||||
const std::string& path() const;
|
||||
const std::vector<ZipEntry>& entries() const; // central-directory order
|
||||
std::size_t file_count() const; // entries that are not directories
|
||||
std::string comment() const; // the EOCD comment (usually empty)
|
||||
|
||||
// Case-insensitive, separator-agnostic lookup of a *file* entry.
|
||||
// Directory placeholders are never returned.
|
||||
const ZipEntry* find(std::string_view rel) const;
|
||||
|
||||
Result<std::vector<std::uint8_t>> read(const ZipEntry& entry) const;
|
||||
Result<std::vector<std::uint8_t>> read(std::string_view rel) const;
|
||||
|
||||
private:
|
||||
struct Impl;
|
||||
explicit ZipArchive(std::unique_ptr<Impl> impl);
|
||||
std::unique_ptr<Impl> impl_;
|
||||
};
|
||||
|
||||
} // namespace mars::vfs
|
||||
31
tests/mars_vfs/CMakeLists.txt
Normal file
31
tests/mars_vfs/CMakeLists.txt
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
# Optional CMake wiring for the mars_vfs tests; the canonical runner is
|
||||
# build_and_run.sh (plain g++). Include from the root with
|
||||
# add_subdirectory(tests/mars_vfs) after add_subdirectory(src/mars/vfs)
|
||||
# (and src/mars/parse + src/mars/text for the real-data test).
|
||||
add_executable(mars_vfs_unit_tests test_main.cpp unit_tests.cpp)
|
||||
target_link_libraries(mars_vfs_unit_tests PRIVATE mars_vfs)
|
||||
target_include_directories(mars_vfs_unit_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
|
||||
add_test(NAME mars_vfs_unit COMMAND mars_vfs_unit_tests)
|
||||
|
||||
# The deflated fixture comes from Python's zipfile so the inflate path is
|
||||
# checked against an independent producer. Without Python that one test SKIPs.
|
||||
find_package(Python3 COMPONENTS Interpreter QUIET)
|
||||
if(Python3_Interpreter_FOUND)
|
||||
set(_fixture ${CMAKE_CURRENT_BINARY_DIR}/deflated.zip)
|
||||
add_custom_command(OUTPUT ${_fixture}
|
||||
COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/make_fixture.py ${_fixture}
|
||||
DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/make_fixture.py
|
||||
COMMENT "Generating mars_vfs deflated fixture")
|
||||
add_custom_target(mars_vfs_fixture DEPENDS ${_fixture})
|
||||
add_dependencies(mars_vfs_unit_tests mars_vfs_fixture)
|
||||
set_tests_properties(mars_vfs_unit PROPERTIES ENVIRONMENT "MARS_VFS_FIXTURE=${_fixture}")
|
||||
endif()
|
||||
|
||||
add_executable(mars_vfs_cat vfs_cat.cpp)
|
||||
target_link_libraries(mars_vfs_cat PRIVATE mars_vfs)
|
||||
|
||||
if(TARGET mars_parse AND TARGET mars_text)
|
||||
add_executable(mars_vfs_realdata_test realdata_test.cpp)
|
||||
target_link_libraries(mars_vfs_realdata_test PRIVATE mars_vfs mars_parse mars_text)
|
||||
add_test(NAME mars_vfs_realdata COMMAND mars_vfs_realdata_test) # SKIPs without SOTS_GOB_DIR
|
||||
endif()
|
||||
88
tests/mars_vfs/build_and_run.sh
Executable file
88
tests/mars_vfs/build_and_run.sh
Executable file
|
|
@ -0,0 +1,88 @@
|
|||
#!/usr/bin/env bash
|
||||
# Self-contained build + test for src/mars/vfs (plain g++, no CMake needed).
|
||||
#
|
||||
# tests/mars_vfs/build_and_run.sh
|
||||
#
|
||||
# Always runs the unit tests (the deflated fixture is generated with Python's
|
||||
# zipfile if python3 is available). When SOTS_GOB_DIR points at a directory
|
||||
# holding sots.gob + sots_local_en.gob it also runs the oracle-backed
|
||||
# real-data test and byte-compares a few entries against `unzip -p`.
|
||||
#
|
||||
# Env: CXX (g++), CC (gcc), PYTHON (python3), BUILD_DIR (tests/mars_vfs/build),
|
||||
# SOTS_GOB_DIR, SOTS_GOB_ORACLE_DIR (defaults to SOTS_GOB_DIR), SOTS_GOB_FULL=1.
|
||||
set -euo pipefail
|
||||
|
||||
HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
ROOT=$(cd "$HERE/../.." && pwd)
|
||||
BUILD=${BUILD_DIR:-$HERE/build}
|
||||
CXX=${CXX:-g++}
|
||||
CC=${CC:-gcc}
|
||||
PYTHON=${PYTHON:-python3}
|
||||
MINIZ="$ROOT/third_party/miniz"
|
||||
MINIZ_DEFS=(-DMINIZ_NO_ARCHIVE_APIS -DMINIZ_NO_ARCHIVE_WRITING_APIS -DMINIZ_NO_STDIO -DMINIZ_NO_TIME -DMINIZ_NO_ZLIB_COMPATIBLE_NAMES)
|
||||
FLAGS=(-std=c++17 -O2 -Wall -Wextra -Werror -I"$ROOT/src" -I"$HERE" -I"$MINIZ" "${MINIZ_DEFS[@]}")
|
||||
VFS=("$ROOT/src/mars/vfs/path.cpp" "$ROOT/src/mars/vfs/zip_archive.cpp" "$ROOT/src/mars/vfs/vfs.cpp")
|
||||
PARSE=("$ROOT/src/mars/parse/blocks.cpp" "$ROOT/src/mars/parse/effect.cpp" "$ROOT/src/mars/parse/value.cpp")
|
||||
TEXT=("$ROOT/src/mars/text/value.cpp" "$ROOT/src/mars/text/flat_kv.cpp" "$ROOT/src/mars/text/manifest.cpp" "$ROOT/src/mars/text/csv.cpp")
|
||||
|
||||
mkdir -p "$BUILD"
|
||||
|
||||
echo "== building (miniz as C, module + tests as C++17 -Werror)"
|
||||
"$CC" -std=c11 -O2 -w "${MINIZ_DEFS[@]}" -c "$MINIZ/miniz.c" -o "$BUILD/miniz.o"
|
||||
"$CXX" "${FLAGS[@]}" "${VFS[@]}" "$BUILD/miniz.o" "$HERE/test_main.cpp" "$HERE/unit_tests.cpp" -o "$BUILD/unit_tests"
|
||||
"$CXX" "${FLAGS[@]}" "${VFS[@]}" "$BUILD/miniz.o" "$HERE/vfs_cat.cpp" -o "$BUILD/vfs_cat"
|
||||
"$CXX" "${FLAGS[@]}" "${VFS[@]}" "${PARSE[@]}" "${TEXT[@]}" "$BUILD/miniz.o" "$HERE/realdata_test.cpp" -o "$BUILD/realdata_test"
|
||||
|
||||
echo "== deflated fixture"
|
||||
FIXTURE="$BUILD/deflated.zip"
|
||||
if command -v "$PYTHON" >/dev/null 2>&1 && "$PYTHON" "$HERE/make_fixture.py" "$FIXTURE"; then
|
||||
export MARS_VFS_FIXTURE="$FIXTURE"
|
||||
echo "generated $FIXTURE"
|
||||
else
|
||||
echo "python3 unavailable: deflate test will SKIP"
|
||||
unset MARS_VFS_FIXTURE
|
||||
fi
|
||||
|
||||
echo "== unit tests"
|
||||
"$BUILD/unit_tests"
|
||||
|
||||
if [ -z "${SOTS_GOB_DIR:-}" ]; then
|
||||
echo "== SOTS_GOB_DIR not set: skipping real-data tests"
|
||||
exit 0
|
||||
fi
|
||||
if [ ! -f "$SOTS_GOB_DIR/sots.gob" ] || [ ! -f "$SOTS_GOB_DIR/sots_local_en.gob" ]; then
|
||||
echo "== SOTS_GOB_DIR=$SOTS_GOB_DIR does not hold sots.gob + sots_local_en.gob" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "== real-data test"
|
||||
"$BUILD/realdata_test"
|
||||
|
||||
echo "== byte-equality spot check against unzip -p"
|
||||
if ! command -v unzip >/dev/null 2>&1; then
|
||||
echo "unzip not installed: SKIP"
|
||||
exit 0
|
||||
fi
|
||||
check() { # archive rel -- unzip is the oracle; the VFS is asked with a different spelling
|
||||
local gob="$1" rel="$2" exp="$BUILD/spot.expected" got="$BUILD/spot.got"
|
||||
local spelled
|
||||
spelled=$(echo "$rel" | tr '/a-z' '\\A-Z') # backslashes + upper case
|
||||
unzip -p "$gob" "$rel" > "$exp" || { echo "unzip cannot extract $rel" >&2; return 1; }
|
||||
[ -s "$exp" ] || { echo "empty oracle output for $rel" >&2; return 1; }
|
||||
"$BUILD/vfs_cat" --zip "$SOTS_GOB_DIR/sots.gob" --zip "$SOTS_GOB_DIR/sots_local_en.gob" "$spelled" > "$got" \
|
||||
|| { echo "vfs_cat cannot read $spelled" >&2; return 1; }
|
||||
if cmp -s "$exp" "$got"; then
|
||||
echo "same: $rel ($(wc -c < "$exp") bytes)"
|
||||
else
|
||||
echo "DIFFER: $rel" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
check "$SOTS_GOB_DIR/sots.gob" "TechTree/MasterTechList.tech"
|
||||
check "$SOTS_GOB_DIR/sots.gob" "Weapons/_weapons.txt"
|
||||
check "$SOTS_GOB_DIR/sots.gob" "Data/globals.txt"
|
||||
check "$SOTS_GOB_DIR/sots.gob" "Avatars/AI_AV.tga"
|
||||
check "$SOTS_GOB_DIR/sots.gob" "Species/Human/sections/CRAIC.shipsection"
|
||||
check "$SOTS_GOB_DIR/sots_local_en.gob" "Locale/EN/Strings.csv"
|
||||
rm -f "$BUILD/spot.expected" "$BUILD/spot.got"
|
||||
echo "== all good"
|
||||
15
tests/mars_vfs/make_fixture.py
Executable file
15
tests/mars_vfs/make_fixture.py
Executable file
|
|
@ -0,0 +1,15 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Write the deflated test fixture the unit tests read via $MARS_VFS_FIXTURE.
|
||||
|
||||
Contents are fixed so the test can regenerate the expected bytes itself:
|
||||
deflated.txt 2000 lines "line N of the deflated fixture\\n" (ZIP_DEFLATED)
|
||||
stored.txt "stored alongside\\n" (ZIP_STORED)
|
||||
"""
|
||||
import sys
|
||||
import zipfile
|
||||
|
||||
out = sys.argv[1]
|
||||
with zipfile.ZipFile(out, "w") as z:
|
||||
z.writestr("deflated.txt", "".join(f"line {i} of the deflated fixture\n" for i in range(2000)),
|
||||
compress_type=zipfile.ZIP_DEFLATED)
|
||||
z.writestr("stored.txt", "stored alongside\n", compress_type=zipfile.ZIP_STORED)
|
||||
243
tests/mars_vfs/realdata_test.cpp
Normal file
243
tests/mars_vfs/realdata_test.cpp
Normal file
|
|
@ -0,0 +1,243 @@
|
|||
// Oracle-backed test against the owner's real archives. Skips cleanly unless
|
||||
// SOTS_GOB_DIR points at a directory holding sots.gob and sots_local_en.gob.
|
||||
//
|
||||
// Env:
|
||||
// SOTS_GOB_DIR directory with the two archives
|
||||
// SOTS_GOB_ORACLE_DIR directory with list_sots.gob.txt / list_sots_local_en.gob.txt
|
||||
// (`unzip -l` output); defaults to SOTS_GOB_DIR; listing
|
||||
// comparison SKIPs when a file is absent
|
||||
// SOTS_GOB_FULL=1 also read + CRC-verify every entry of both archives
|
||||
#include <algorithm>
|
||||
#include <chrono>
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <filesystem>
|
||||
#include <fstream>
|
||||
#include <map>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "mars/parse/blocks.h"
|
||||
#include "mars/text/csv.h"
|
||||
#include "mars/vfs/path.h"
|
||||
#include "mars/vfs/vfs.h"
|
||||
#include "mars/vfs/zip_archive.h"
|
||||
|
||||
namespace fs = std::filesystem;
|
||||
using namespace mars::vfs;
|
||||
|
||||
static int g_failures = 0;
|
||||
static int g_checks = 0;
|
||||
|
||||
#define CHECK(cond) \
|
||||
do { \
|
||||
++g_checks; \
|
||||
if (!(cond)) { \
|
||||
++g_failures; \
|
||||
std::fprintf(stderr, "FAIL %s:%d: %s\n", __FILE__, __LINE__, #cond); \
|
||||
} \
|
||||
} while (0)
|
||||
|
||||
namespace {
|
||||
|
||||
struct OracleEntry {
|
||||
std::string name;
|
||||
std::uint64_t size = 0;
|
||||
};
|
||||
|
||||
// Parse `unzip -l` output: three header lines, entries, a "---------" footer.
|
||||
// Names start at a fixed column (they may contain spaces).
|
||||
// " 65580 2007-08-07 13:25 Avatars/AI_AV.tga"
|
||||
bool load_listing(const fs::path& path, std::vector<OracleEntry>& out) {
|
||||
std::ifstream in(path);
|
||||
if (!in) return false;
|
||||
std::string line;
|
||||
int lineno = 0;
|
||||
while (std::getline(in, line)) {
|
||||
++lineno;
|
||||
if (!line.empty() && line.back() == '\r') line.pop_back();
|
||||
if (lineno <= 3) continue;
|
||||
if (line.rfind("---------", 0) == 0) break;
|
||||
if (line.size() < 31) continue;
|
||||
OracleEntry e;
|
||||
e.size = std::strtoull(line.c_str(), nullptr, 10);
|
||||
e.name = line.substr(30);
|
||||
out.push_back(std::move(e));
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
void compare_with_listing(const ZipArchive& z, const fs::path& listing) {
|
||||
std::vector<OracleEntry> oracle;
|
||||
if (!load_listing(listing, oracle)) {
|
||||
std::printf(" listing %s: SKIP (not found)\n", listing.string().c_str());
|
||||
return;
|
||||
}
|
||||
std::size_t dirs = 0;
|
||||
for (const auto& e : oracle) if (!e.name.empty() && e.name.back() == '/') ++dirs;
|
||||
std::printf(" oracle %s: %zu entries (%zu directories, %zu files)\n", listing.filename().string().c_str(),
|
||||
oracle.size(), dirs, oracle.size() - dirs);
|
||||
std::printf(" ours %s: %zu entries (%zu files)\n", fs::path(z.path()).filename().string().c_str(),
|
||||
z.entries().size(), z.file_count());
|
||||
CHECK(z.entries().size() == oracle.size());
|
||||
CHECK(z.file_count() == oracle.size() - dirs);
|
||||
|
||||
std::map<std::string, std::uint64_t> ours;
|
||||
for (const ZipEntry& e : z.entries()) ours.emplace(e.name, e.uncompressed_size);
|
||||
std::size_t missing = 0, size_mismatch = 0;
|
||||
for (const auto& o : oracle) {
|
||||
auto it = ours.find(o.name);
|
||||
if (it == ours.end()) {
|
||||
if (missing < 5) std::fprintf(stderr, " not in ours: %s\n", o.name.c_str());
|
||||
++missing;
|
||||
} else if (it->second != o.size) {
|
||||
if (size_mismatch < 5) std::fprintf(stderr, " size differs: %s\n", o.name.c_str());
|
||||
++size_mismatch;
|
||||
}
|
||||
}
|
||||
CHECK(missing == 0);
|
||||
CHECK(size_mismatch == 0);
|
||||
CHECK(ours.size() == oracle.size()); // no duplicate names either side
|
||||
}
|
||||
|
||||
void full_read(const ZipArchive& z) {
|
||||
const auto t0 = std::chrono::steady_clock::now();
|
||||
std::uint64_t bytes = 0;
|
||||
std::size_t errors = 0, files = 0;
|
||||
for (const ZipEntry& e : z.entries()) {
|
||||
if (e.is_directory) continue;
|
||||
++files;
|
||||
auto r = z.read(e);
|
||||
if (!r) {
|
||||
++errors;
|
||||
if (errors < 5) std::fprintf(stderr, " read error: %s\n", r.error().message.c_str());
|
||||
continue;
|
||||
}
|
||||
bytes += r->size();
|
||||
}
|
||||
const double secs = std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count();
|
||||
std::printf(" full read %s: %zu files, %.2f GB, %zu errors, %.1fs\n", fs::path(z.path()).filename().string().c_str(),
|
||||
files, static_cast<double>(bytes) / 1e9, errors, secs);
|
||||
CHECK(errors == 0);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
int main() {
|
||||
const char* dir = std::getenv("SOTS_GOB_DIR");
|
||||
if (!dir || !*dir) {
|
||||
std::printf("realdata_test: SKIP (SOTS_GOB_DIR not set)\n");
|
||||
return 0;
|
||||
}
|
||||
const fs::path gobs(dir);
|
||||
const char* odir = std::getenv("SOTS_GOB_ORACLE_DIR");
|
||||
const fs::path oracle = (odir && *odir) ? fs::path(odir) : gobs;
|
||||
const bool full = std::getenv("SOTS_GOB_FULL") && std::string(std::getenv("SOTS_GOB_FULL")) == "1";
|
||||
|
||||
// --- archives against the unzip -l listings ---
|
||||
const auto t0 = std::chrono::steady_clock::now();
|
||||
Result<ZipArchive> main_gob = ZipArchive::open((gobs / "sots.gob").string());
|
||||
Result<ZipArchive> local_gob = ZipArchive::open((gobs / "sots_local_en.gob").string());
|
||||
const double open_secs = std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count();
|
||||
CHECK(main_gob.ok());
|
||||
CHECK(local_gob.ok());
|
||||
if (!main_gob || !local_gob) {
|
||||
std::fprintf(stderr, "cannot open archives: %s / %s\n", main_gob ? "ok" : main_gob.error().message.c_str(),
|
||||
local_gob ? "ok" : local_gob.error().message.c_str());
|
||||
return 1;
|
||||
}
|
||||
std::printf(" opened both archives in %.3fs\n", open_secs);
|
||||
compare_with_listing(main_gob.value(), oracle / "list_sots.gob.txt");
|
||||
compare_with_listing(local_gob.value(), oracle / "list_sots_local_en.gob.txt");
|
||||
std::size_t deflated = 0;
|
||||
for (const ZipEntry& e : main_gob->entries()) if (e.method != 0) ++deflated;
|
||||
for (const ZipEntry& e : local_gob->entries()) if (e.method != 0) ++deflated;
|
||||
std::printf(" entries using a compression method: %zu\n", deflated);
|
||||
CHECK(deflated == 0); // the notes: every entry is stored
|
||||
|
||||
// --- the layered VFS ---
|
||||
Vfs v;
|
||||
auto id_main = v.mount_zip((gobs / "sots.gob").string());
|
||||
auto id_local = v.mount_zip((gobs / "sots_local_en.gob").string());
|
||||
CHECK(id_main.ok() && id_local.ok());
|
||||
|
||||
// MasterTechList.tech parses into 293 tech blocks, from any spelling.
|
||||
{
|
||||
CHECK(v.exists("TechTree/MasterTechList.tech"));
|
||||
CHECK(v.exists("techtree\\MASTERTECHLIST.TECH"));
|
||||
auto st = v.stat("TechTree/MasterTechList.tech");
|
||||
CHECK(st && st->mount == id_main.value() && st->size == 62234 && !st->compressed);
|
||||
auto text = v.read_text("TechTree/MasterTechList.tech");
|
||||
CHECK(text.ok());
|
||||
if (text) {
|
||||
auto doc = mars::parse::parse_blocks(text.value());
|
||||
CHECK(doc.ok());
|
||||
if (doc) {
|
||||
const std::size_t techs = doc->root.blocks("tech").size();
|
||||
std::printf(" MasterTechList.tech: %zu tech blocks\n", techs);
|
||||
CHECK(techs == 293);
|
||||
}
|
||||
}
|
||||
}
|
||||
// Strings.csv comes from the second archive and yields 5,722 records.
|
||||
{
|
||||
auto st = v.stat("Locale/EN/Strings.csv");
|
||||
CHECK(st && st->mount == id_local.value() && st->size == 397352);
|
||||
auto text = v.read_text("locale/en/strings.csv");
|
||||
CHECK(text.ok());
|
||||
if (text) {
|
||||
auto raw = mars::text::split_csv_records(text.value());
|
||||
auto csv = mars::text::parse_csv(text.value());
|
||||
std::printf(" Strings.csv: %zu raw records, %zu data rows (%zu problems)\n", raw.value.size(),
|
||||
csv.value.rows.size(), csv.problems.size());
|
||||
CHECK(raw.value.size() == 5722); // the notes' "5,722 rows" (same as mars_text's own check)
|
||||
CHECK(csv.value.rows.size() == 5200);
|
||||
}
|
||||
}
|
||||
// Listing: 207 .weapon files in total (123 under Weapons/, 84 under Species/_NPC/).
|
||||
{
|
||||
auto ends_with_weapon = [](const std::string& n) {
|
||||
return n.size() > 7 && normalize_key(n).compare(n.size() - 7, 7, ".weapon") == 0;
|
||||
};
|
||||
std::size_t total = 0, under_weapons = 0;
|
||||
for (const auto& e : v.list()) if (ends_with_weapon(e.name)) ++total;
|
||||
for (const auto& e : v.list("Weapons/")) if (ends_with_weapon(e.name)) ++under_weapons;
|
||||
std::printf(" *.weapon: %zu total, %zu under Weapons/\n", total, under_weapons);
|
||||
CHECK(total == 207);
|
||||
CHECK(under_weapons == 123);
|
||||
CHECK(v.list().size() == main_gob->file_count() + local_gob->file_count());
|
||||
}
|
||||
// A loose-file override wins over the archive; registration order can flip that.
|
||||
{
|
||||
const auto tag = std::chrono::steady_clock::now().time_since_epoch().count();
|
||||
fs::path over = fs::temp_directory_path() / ("mars_vfs_real_" + std::to_string(tag));
|
||||
fs::create_directories(over / "techtree");
|
||||
std::ofstream(over / "techtree" / "mastertechlist.tech") << "tech { name \"OVERRIDE\" }\n";
|
||||
Vfs w;
|
||||
w.mount_zip((gobs / "sots.gob").string());
|
||||
auto nid = w.mount_native(over.string());
|
||||
CHECK(nid.ok());
|
||||
auto st = w.stat("TechTree/MasterTechList.tech");
|
||||
CHECK(st && st->mount == nid.value() && st->kind == MountKind::Native);
|
||||
auto text = w.read_text("TechTree/MasterTechList.tech");
|
||||
CHECK(text && text->rfind("tech { name \"OVERRIDE\"", 0) == 0);
|
||||
|
||||
Vfs r(Order::Registration);
|
||||
r.mount_zip((gobs / "sots.gob").string());
|
||||
r.mount_native(over.string());
|
||||
auto st2 = r.stat("TechTree/MasterTechList.tech");
|
||||
CHECK(st2 && st2->kind == MountKind::Zip);
|
||||
std::error_code ec;
|
||||
fs::remove_all(over, ec);
|
||||
}
|
||||
|
||||
if (full) {
|
||||
full_read(main_gob.value());
|
||||
full_read(local_gob.value());
|
||||
} else {
|
||||
std::printf(" full read: SKIP (set SOTS_GOB_FULL=1)\n");
|
||||
}
|
||||
|
||||
std::printf("realdata_test: %d checks, %d failures\n", g_checks, g_failures);
|
||||
return g_failures == 0 ? 0 : 1;
|
||||
}
|
||||
32
tests/mars_vfs/test_main.cpp
Normal file
32
tests/mars_vfs/test_main.cpp
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
#include "test_main.h"
|
||||
|
||||
namespace testing {
|
||||
|
||||
std::vector<Case>& registry() {
|
||||
static std::vector<Case> r;
|
||||
return r;
|
||||
}
|
||||
|
||||
int& failures() {
|
||||
static int n = 0;
|
||||
return n;
|
||||
}
|
||||
|
||||
void report_failure(const char* file, int line, const std::string& expr) {
|
||||
++failures();
|
||||
std::fprintf(stderr, " FAIL %s:%d: %s\n", file, line, expr.c_str());
|
||||
}
|
||||
|
||||
} // namespace testing
|
||||
|
||||
int main() {
|
||||
int ran = 0;
|
||||
for (const auto& c : testing::registry()) {
|
||||
const int before = testing::failures();
|
||||
c.fn();
|
||||
++ran;
|
||||
if (testing::failures() != before) std::fprintf(stderr, " in test %s\n", c.name);
|
||||
}
|
||||
std::printf("%d tests, %d failures\n", ran, testing::failures());
|
||||
return testing::failures() == 0 ? 0 : 1;
|
||||
}
|
||||
49
tests/mars_vfs/test_main.h
Normal file
49
tests/mars_vfs/test_main.h
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
// Tiny self-contained test harness (no third-party deps).
|
||||
#pragma once
|
||||
|
||||
#include <cstdio>
|
||||
#include <functional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
namespace testing {
|
||||
|
||||
struct Case {
|
||||
const char* name;
|
||||
std::function<void()> fn;
|
||||
};
|
||||
|
||||
std::vector<Case>& registry();
|
||||
int& failures();
|
||||
void report_failure(const char* file, int line, const std::string& expr);
|
||||
|
||||
struct Register {
|
||||
Register(const char* name, std::function<void()> fn) { registry().push_back({name, std::move(fn)}); }
|
||||
};
|
||||
|
||||
} // namespace testing
|
||||
|
||||
#define TEST(name) \
|
||||
static void test_##name(); \
|
||||
static testing::Register reg_##name(#name, test_##name); \
|
||||
static void test_##name()
|
||||
|
||||
#define CHECK(expr) \
|
||||
do { \
|
||||
if (!(expr)) testing::report_failure(__FILE__, __LINE__, #expr); \
|
||||
} while (0)
|
||||
|
||||
#define CHECK_EQ(a, b) \
|
||||
do { \
|
||||
if (!((a) == (b))) \
|
||||
testing::report_failure(__FILE__, __LINE__, \
|
||||
std::string(#a " == " #b " [got: ") + \
|
||||
testing_to_string(a) + " vs " + \
|
||||
testing_to_string(b) + "]"); \
|
||||
} while (0)
|
||||
|
||||
inline std::string testing_to_string(const std::string& s) { return "\"" + s + "\""; }
|
||||
inline std::string testing_to_string(const char* s) { return std::string("\"") + s + "\""; }
|
||||
inline std::string testing_to_string(bool b) { return b ? "true" : "false"; }
|
||||
template <class T>
|
||||
std::string testing_to_string(const T& v) { return std::to_string(v); }
|
||||
421
tests/mars_vfs/unit_tests.cpp
Normal file
421
tests/mars_vfs/unit_tests.cpp
Normal file
|
|
@ -0,0 +1,421 @@
|
|||
// Unit tests for mars::vfs: ZIP container parsing, override precedence,
|
||||
// path folding, error paths. Fixtures are built in-process (zip_builder.h);
|
||||
// the one deflated fixture is produced by Python at build time and read
|
||||
// from $MARS_VFS_FIXTURE (test SKIPs without it).
|
||||
#include <chrono>
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <filesystem>
|
||||
#include <fstream>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "mars/vfs/path.h"
|
||||
#include "mars/vfs/vfs.h"
|
||||
#include "mars/vfs/zip_archive.h"
|
||||
#include "test_main.h"
|
||||
#include "zip_builder.h"
|
||||
|
||||
namespace fs = std::filesystem;
|
||||
using namespace mars::vfs;
|
||||
|
||||
namespace {
|
||||
|
||||
// A scratch directory per test run, removed on exit.
|
||||
struct Scratch {
|
||||
fs::path dir;
|
||||
Scratch() {
|
||||
static int counter = 0;
|
||||
const auto tag = std::chrono::steady_clock::now().time_since_epoch().count();
|
||||
dir = fs::temp_directory_path() / ("mars_vfs_test_" + std::to_string(tag) + "_" + std::to_string(counter++));
|
||||
fs::create_directories(dir);
|
||||
}
|
||||
~Scratch() {
|
||||
std::error_code ec;
|
||||
fs::remove_all(dir, ec);
|
||||
}
|
||||
fs::path write(const std::string& rel, const std::vector<std::uint8_t>& bytes) {
|
||||
fs::path p = dir / rel;
|
||||
fs::create_directories(p.parent_path());
|
||||
std::ofstream(p, std::ios::binary).write(reinterpret_cast<const char*>(bytes.data()), static_cast<std::streamsize>(bytes.size()));
|
||||
return p;
|
||||
}
|
||||
fs::path write(const std::string& rel, std::string_view text) {
|
||||
return write(rel, std::vector<std::uint8_t>(text.begin(), text.end()));
|
||||
}
|
||||
};
|
||||
|
||||
std::string as_text(const std::vector<std::uint8_t>& v) { return std::string(v.begin(), v.end()); }
|
||||
|
||||
std::vector<std::uint8_t> sample_zip() {
|
||||
ziptest::ZipBuilder b;
|
||||
b.add_dir("Data");
|
||||
b.add("Data/globals.txt", "MARS_DEFAULT_COLOR \"255 177 39\"\n");
|
||||
b.add("Species/Human/sections/CruiserAIC.shipsection", "shipsection { health 2800 }");
|
||||
b.add("Locale/EN/Strings.csv", "# id,text\nSOTS_HELLO,Hello\n");
|
||||
b.add("empty.bin", "");
|
||||
return b.build();
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// ---------------------------------------------------------------- paths
|
||||
|
||||
TEST(path_normalisation) {
|
||||
CHECK_EQ(normalize_key("Species\\Human\\sections\\CruiserAIC.shipsection"), std::string("species/human/sections/cruiseraic.shipsection"));
|
||||
CHECK_EQ(normalize_key("./Data//globals.txt"), std::string("data/globals.txt"));
|
||||
CHECK_EQ(normalize_key("/Data/"), std::string("data"));
|
||||
CHECK_EQ(normalize_key(""), std::string(""));
|
||||
CHECK_EQ(normalize_key("a/../b"), std::string("a/../b")); // kept literally
|
||||
CHECK_EQ(normalize_name("Weapons\\Gauss.weapon"), std::string("Weapons/Gauss.weapon"));
|
||||
CHECK_EQ(normalize_key("caf\xE9.txt"), std::string("caf\xE9.txt")); // high byte untouched
|
||||
CHECK(key_has_prefix("weapons/x", "weapons/"));
|
||||
CHECK(key_has_prefix("weapons/x", ""));
|
||||
CHECK(!key_has_prefix("weapon", "weapons/"));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- ZipArchive
|
||||
|
||||
TEST(zip_index_and_lookup) {
|
||||
Scratch s;
|
||||
fs::path p = s.write("a.zip", sample_zip());
|
||||
Result<ZipArchive> r = ZipArchive::open(p.string());
|
||||
CHECK(r.ok());
|
||||
if (!r) { std::fprintf(stderr, " %s\n", r.error().message.c_str()); return; }
|
||||
const ZipArchive& z = r.value();
|
||||
CHECK_EQ(z.entries().size(), std::size_t{5});
|
||||
CHECK_EQ(z.file_count(), std::size_t{4});
|
||||
CHECK(z.entries()[0].is_directory);
|
||||
CHECK_EQ(z.entries()[0].key, std::string("data"));
|
||||
CHECK(z.find("Data") == nullptr); // directories are not files
|
||||
CHECK(z.find("Data/globals.txt") != nullptr);
|
||||
CHECK(z.find("DATA\\GLOBALS.TXT") != nullptr);
|
||||
CHECK(z.find("./data//Globals.txt") != nullptr);
|
||||
CHECK(z.find("Data/globals.txt.bak") == nullptr);
|
||||
CHECK(z.find("globals.txt") == nullptr);
|
||||
|
||||
auto bytes = z.read("species/HUMAN/Sections/cruiseraic.SHIPSECTION");
|
||||
CHECK(bytes.ok());
|
||||
if (bytes) CHECK_EQ(as_text(bytes.value()), std::string("shipsection { health 2800 }"));
|
||||
const ZipEntry* e = z.find("Data/globals.txt");
|
||||
CHECK(e && e->method == 0 && e->uncompressed_size == 32 && e->compressed_size == 32);
|
||||
CHECK(e && e->name == "Data/globals.txt");
|
||||
auto empty = z.read("empty.bin");
|
||||
CHECK(empty.ok() && empty->empty());
|
||||
auto missing = z.read("nope.txt");
|
||||
CHECK(!missing.ok() && missing.error().code == Error::Code::NotFound);
|
||||
CHECK_EQ(z.comment(), std::string(""));
|
||||
}
|
||||
|
||||
TEST(zip_local_header_lengths_are_used) {
|
||||
// The local header carries a longer extra field than the central one;
|
||||
// the data offset must come from the local header.
|
||||
ziptest::ZipBuilder b;
|
||||
ziptest::Item& it = b.add("x.txt", "payload");
|
||||
it.local_extra = {1, 2, 3, 4, 5, 6, 7};
|
||||
it.central_extra = {9};
|
||||
Scratch s;
|
||||
auto z = ZipArchive::open(s.write("a.zip", b.build()).string());
|
||||
CHECK(z.ok());
|
||||
auto bytes = z->read("x.txt");
|
||||
CHECK(bytes.ok());
|
||||
if (bytes) CHECK_EQ(as_text(bytes.value()), std::string("payload"));
|
||||
}
|
||||
|
||||
TEST(zip_trailing_comment_and_junk) {
|
||||
Scratch s;
|
||||
ziptest::ZipBuilder b;
|
||||
b.add("a.txt", "A");
|
||||
auto with_comment = b.build("hello archive");
|
||||
auto z1 = ZipArchive::open(s.write("c.zip", with_comment).string());
|
||||
CHECK(z1.ok());
|
||||
if (z1) CHECK_EQ(z1->comment(), std::string("hello archive"));
|
||||
|
||||
// Junk appended after a comment-less EOCD is tolerated (the scan finds the record).
|
||||
auto junk = b.build();
|
||||
for (int i = 0; i < 100; ++i) junk.push_back(0xAA);
|
||||
auto z2 = ZipArchive::open(s.write("j.zip", junk).string());
|
||||
CHECK(z2.ok());
|
||||
if (z2) CHECK(z2->read("a.txt").ok());
|
||||
}
|
||||
|
||||
TEST(zip_duplicate_names_first_wins) {
|
||||
ziptest::ZipBuilder b;
|
||||
b.add("Same.txt", "first");
|
||||
b.add("same.TXT", "second");
|
||||
Scratch s;
|
||||
auto z = ZipArchive::open(s.write("d.zip", b.build()).string());
|
||||
CHECK(z.ok());
|
||||
CHECK_EQ(z->entries().size(), std::size_t{2});
|
||||
auto bytes = z->read("SAME.txt");
|
||||
CHECK(bytes.ok());
|
||||
if (bytes) CHECK_EQ(as_text(bytes.value()), std::string("first"));
|
||||
}
|
||||
|
||||
TEST(zip_bad_archives) {
|
||||
Scratch s;
|
||||
auto good = sample_zip();
|
||||
|
||||
auto missing = ZipArchive::open((s.dir / "absent.zip").string());
|
||||
CHECK(!missing.ok() && missing.error().code == Error::Code::Io);
|
||||
|
||||
auto empty = ZipArchive::open(s.write("empty.zip", "").string());
|
||||
CHECK(!empty.ok() && empty.error().code == Error::Code::BadArchive);
|
||||
|
||||
auto text = ZipArchive::open(s.write("text.zip", "this is not a zip file at all, just some text").string());
|
||||
CHECK(!text.ok() && text.error().code == Error::Code::BadArchive);
|
||||
|
||||
auto truncated = good;
|
||||
truncated.resize(truncated.size() - 10); // cuts into the EOCD
|
||||
auto t = ZipArchive::open(s.write("trunc.zip", truncated).string());
|
||||
CHECK(!t.ok() && t.error().code == Error::Code::BadArchive);
|
||||
|
||||
auto badsig = good;
|
||||
badsig[badsig.size() - 22] ^= 0xFF; // EOCD signature
|
||||
auto bs = ZipArchive::open(s.write("badsig.zip", badsig).string());
|
||||
CHECK(!bs.ok() && bs.error().code == Error::Code::BadArchive);
|
||||
|
||||
auto badoff = good;
|
||||
badoff[badoff.size() - 22 + 16] = 0xFF; // low byte of the CD offset -> past EOF
|
||||
badoff[badoff.size() - 22 + 17] = 0xFF;
|
||||
auto bo = ZipArchive::open(s.write("badoff.zip", badoff).string());
|
||||
CHECK(!bo.ok() && bo.error().code == Error::Code::BadArchive);
|
||||
|
||||
auto badcd = good;
|
||||
const std::uint32_t cd_off = badcd[badcd.size() - 6] | (badcd[badcd.size() - 5] << 8);
|
||||
badcd[cd_off] ^= 0xFF; // first central header signature
|
||||
auto bc = ZipArchive::open(s.write("badcd.zip", badcd).string());
|
||||
CHECK(!bc.ok() && bc.error().code == Error::Code::BadArchive);
|
||||
|
||||
auto zip64 = good;
|
||||
zip64[zip64.size() - 22 + 10] = 0xFF; // total entries = 0xFFFF
|
||||
zip64[zip64.size() - 22 + 11] = 0xFF;
|
||||
auto z64 = ZipArchive::open(s.write("z64.zip", zip64).string());
|
||||
CHECK(!z64.ok() && z64.error().code == Error::Code::Unsupported);
|
||||
}
|
||||
|
||||
TEST(zip_bad_entries) {
|
||||
Scratch s;
|
||||
{
|
||||
ziptest::ZipBuilder b;
|
||||
b.add("a.txt", "hello world");
|
||||
auto bytes = b.build();
|
||||
bytes[30 + 5 + 2] ^= 0x01; // flip a data byte: local header 30 + name 5, then payload
|
||||
auto z = ZipArchive::open(s.write("crc.zip", bytes).string());
|
||||
CHECK(z.ok());
|
||||
auto r = z->read("a.txt");
|
||||
CHECK(!r.ok() && r.error().code == Error::Code::Corrupt);
|
||||
}
|
||||
{
|
||||
ziptest::ZipBuilder b;
|
||||
b.add("a.txt", "hello world");
|
||||
auto bytes = b.build();
|
||||
bytes[0] ^= 0xFF; // local header signature
|
||||
auto z = ZipArchive::open(s.write("lh.zip", bytes).string());
|
||||
CHECK(z.ok());
|
||||
auto r = z->read("a.txt");
|
||||
CHECK(!r.ok() && r.error().code == Error::Code::Corrupt);
|
||||
}
|
||||
{
|
||||
ziptest::ZipBuilder b;
|
||||
b.add("enc.txt", "secret").flags = 0x0001;
|
||||
b.add_raw("bz.txt", {1, 2, 3}, 12, 3, 0);
|
||||
b.add_raw("short.txt", {1, 2, 3}, 0, 10, 0); // stored but sizes disagree
|
||||
std::vector<std::uint8_t> garbage = {0xFF, 0xFF, 0xFF, 0xFF, 0x00};
|
||||
b.add_raw("bad.deflate", garbage, 8, 100, 0);
|
||||
auto z = ZipArchive::open(s.write("odd.zip", b.build()).string());
|
||||
CHECK(z.ok());
|
||||
auto enc = z->read("enc.txt");
|
||||
CHECK(!enc.ok() && enc.error().code == Error::Code::Unsupported);
|
||||
auto bz = z->read("bz.txt");
|
||||
CHECK(!bz.ok() && bz.error().code == Error::Code::Unsupported);
|
||||
auto sh = z->read("short.txt");
|
||||
CHECK(!sh.ok() && sh.error().code == Error::Code::Corrupt);
|
||||
auto bd = z->read("bad.deflate");
|
||||
CHECK(!bd.ok() && bd.error().code == Error::Code::Corrupt);
|
||||
}
|
||||
}
|
||||
|
||||
TEST(zip_deflated_fixture) {
|
||||
// Produced by build_and_run.sh / CMake with Python's zipfile (ZIP_DEFLATED):
|
||||
// deflated.txt : 2000 lines "line N of the deflated fixture\n"
|
||||
// stored.txt : "stored alongside\n" (ZIP_STORED)
|
||||
const char* fixture = std::getenv("MARS_VFS_FIXTURE");
|
||||
if (!fixture || !*fixture) {
|
||||
std::printf(" zip_deflated_fixture: SKIP (MARS_VFS_FIXTURE not set)\n");
|
||||
return;
|
||||
}
|
||||
auto z = ZipArchive::open(fixture);
|
||||
CHECK(z.ok());
|
||||
if (!z) { std::fprintf(stderr, " %s\n", z.error().message.c_str()); return; }
|
||||
const ZipEntry* e = z->find("deflated.txt");
|
||||
CHECK(e != nullptr);
|
||||
if (!e) return;
|
||||
CHECK_EQ(e->method, std::uint16_t{8});
|
||||
CHECK(e->compressed_size < e->uncompressed_size);
|
||||
std::string expect;
|
||||
for (int i = 0; i < 2000; ++i) expect += "line " + std::to_string(i) + " of the deflated fixture\n";
|
||||
auto got = z->read(*e);
|
||||
CHECK(got.ok());
|
||||
if (got) CHECK_EQ(got->size(), expect.size());
|
||||
if (got) CHECK(as_text(got.value()) == expect);
|
||||
auto st = z->read("STORED.TXT");
|
||||
CHECK(st.ok());
|
||||
if (st) CHECK_EQ(as_text(st.value()), std::string("stored alongside\n"));
|
||||
|
||||
Vfs v;
|
||||
CHECK(v.mount_zip(fixture).ok());
|
||||
auto s = v.stat("deflated.txt");
|
||||
CHECK(s && s->compressed && s->size == expect.size());
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- Vfs
|
||||
|
||||
namespace {
|
||||
|
||||
// Two archives and a loose directory that overlap on "shared.txt":
|
||||
// a.zip : shared.txt="A" only_a.txt Sub/In_A.txt
|
||||
// b.zip : shared.txt="B" only_b.txt sub/in_b.txt
|
||||
// loose : shared.txt="N" only_n.txt SUB/in_n.txt
|
||||
struct Layered {
|
||||
Scratch s;
|
||||
fs::path a, b, loose;
|
||||
Layered() {
|
||||
ziptest::ZipBuilder ba;
|
||||
ba.add("shared.txt", "A");
|
||||
ba.add("only_a.txt", "a");
|
||||
ba.add("Sub/In_A.txt", "a-sub");
|
||||
a = s.write("a.zip", ba.build());
|
||||
ziptest::ZipBuilder bb;
|
||||
bb.add("shared.txt", "B");
|
||||
bb.add("only_b.txt", "b");
|
||||
bb.add("sub/in_b.txt", "b-sub");
|
||||
b = s.write("b.zip", bb.build());
|
||||
loose = s.dir / "loose";
|
||||
s.write("loose/shared.txt", "N");
|
||||
s.write("loose/only_n.txt", "n");
|
||||
s.write("loose/SUB/in_n.txt", "n-sub");
|
||||
}
|
||||
};
|
||||
|
||||
std::string read_str(const Vfs& v, std::string_view rel) {
|
||||
auto r = v.read_text(rel);
|
||||
return r ? r.value() : ("<" + std::string(to_string(r.error().code)) + ">");
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
TEST(vfs_native_first_override) {
|
||||
Layered L;
|
||||
Vfs v; // Order::NativeFirst
|
||||
auto ia = v.mount_zip(L.a.string());
|
||||
auto ib = v.mount_zip(L.b.string());
|
||||
auto in = v.mount_native(L.loose.string()); // registered last, still wins
|
||||
CHECK(ia.ok() && ib.ok() && in.ok());
|
||||
CHECK_EQ(v.mounts().size(), std::size_t{3});
|
||||
CHECK_EQ(v.mounts()[2].file_count, std::size_t{3});
|
||||
std::vector<MountId> so = v.search_order();
|
||||
CHECK(so.size() == 3 && so[0] == in.value() && so[1] == ia.value() && so[2] == ib.value());
|
||||
|
||||
CHECK_EQ(read_str(v, "shared.txt"), std::string("N"));
|
||||
CHECK_EQ(read_str(v, "only_a.txt"), std::string("a"));
|
||||
CHECK_EQ(read_str(v, "only_b.txt"), std::string("b"));
|
||||
CHECK_EQ(read_str(v, "only_n.txt"), std::string("n"));
|
||||
CHECK_EQ(read_str(v, "nothing.txt"), std::string("<not found>"));
|
||||
|
||||
auto st = v.stat("SHARED.TXT");
|
||||
CHECK(st && st->mount == in.value() && st->kind == MountKind::Native && st->size == 1 && st->name == "shared.txt");
|
||||
auto sb = v.stat("only_b.txt");
|
||||
CHECK(sb && sb->mount == ib.value() && sb->kind == MountKind::Zip && !sb->compressed);
|
||||
CHECK(!v.stat("nothing.txt"));
|
||||
CHECK(v.exists("sub\\IN_A.txt") && v.exists("Sub/in_n.txt") && !v.exists("sub"));
|
||||
}
|
||||
|
||||
TEST(vfs_registration_order) {
|
||||
Layered L;
|
||||
{
|
||||
Vfs v(Order::Registration);
|
||||
auto ia = v.mount_zip(L.a.string());
|
||||
v.mount_zip(L.b.string());
|
||||
v.mount_native(L.loose.string());
|
||||
CHECK_EQ(read_str(v, "shared.txt"), std::string("A"));
|
||||
auto st = v.stat("shared.txt");
|
||||
CHECK(st && st->mount == ia.value());
|
||||
}
|
||||
{
|
||||
Vfs v(Order::Registration);
|
||||
v.mount_zip(L.b.string());
|
||||
v.mount_zip(L.a.string());
|
||||
CHECK_EQ(read_str(v, "shared.txt"), std::string("B"));
|
||||
}
|
||||
{
|
||||
Vfs v(Order::Registration);
|
||||
v.mount_native(L.loose.string());
|
||||
v.mount_zip(L.a.string());
|
||||
CHECK_EQ(read_str(v, "shared.txt"), std::string("N"));
|
||||
}
|
||||
}
|
||||
|
||||
TEST(vfs_list) {
|
||||
Layered L;
|
||||
Vfs v;
|
||||
v.mount_zip(L.a.string());
|
||||
v.mount_zip(L.b.string());
|
||||
MountId in = v.mount_native(L.loose.string()).value();
|
||||
|
||||
auto all = v.list();
|
||||
CHECK_EQ(all.size(), std::size_t{7}); // shared once
|
||||
auto sub = v.list("sub/");
|
||||
CHECK_EQ(sub.size(), std::size_t{3});
|
||||
if (sub.size() == 3) {
|
||||
CHECK_EQ(sub[0].name, std::string("Sub/In_A.txt")); // original spelling from a.zip
|
||||
CHECK_EQ(sub[1].name, std::string("sub/in_b.txt"));
|
||||
CHECK_EQ(sub[2].name, std::string("SUB/in_n.txt"));
|
||||
CHECK_EQ(sub[2].mount, in);
|
||||
}
|
||||
auto stem = v.list("ONLY_");
|
||||
CHECK_EQ(stem.size(), std::size_t{3});
|
||||
auto backslash = v.list("SUB\\in_");
|
||||
CHECK_EQ(backslash.size(), std::size_t{3});
|
||||
for (const auto& e : all)
|
||||
if (normalize_key(e.name) == "shared.txt") CHECK(e.mount == in);
|
||||
CHECK(v.list("zzz").empty());
|
||||
}
|
||||
|
||||
TEST(vfs_native_mount_details) {
|
||||
Scratch s;
|
||||
s.write("root/Data/Globals.txt", "x");
|
||||
Vfs v;
|
||||
auto id = v.mount_native((s.dir / "root").string());
|
||||
CHECK(id.ok());
|
||||
CHECK(v.exists("data/globals.TXT"));
|
||||
CHECK(v.exists("DATA\\Globals.txt"));
|
||||
auto st = v.stat("data/globals.txt");
|
||||
CHECK(st && st->name == "Data/Globals.txt");
|
||||
CHECK(!v.exists("../root/Data/Globals.txt")); // '..' never resolves
|
||||
CHECK(!v.exists("Data")); // directories are not files
|
||||
|
||||
// Files added after mounting are seen after a rescan.
|
||||
s.write("root/Data/new.txt", "y");
|
||||
CHECK(!v.exists("Data/new.txt"));
|
||||
auto n = v.rescan_native(id.value());
|
||||
CHECK(n.ok() && n.value() == 2);
|
||||
CHECK(v.exists("Data/new.txt"));
|
||||
CHECK_EQ(v.mounts()[0].file_count, std::size_t{2});
|
||||
|
||||
auto bad = v.mount_native((s.dir / "absent").string());
|
||||
CHECK(!bad.ok() && bad.error().code == Error::Code::Io);
|
||||
auto notzip = v.mount_zip((s.dir / "root/Data/Globals.txt").string());
|
||||
CHECK(!notzip.ok() && notzip.error().code == Error::Code::BadArchive);
|
||||
CHECK(!v.rescan_native(99).ok());
|
||||
CHECK(v.archive(id.value()) == nullptr);
|
||||
}
|
||||
|
||||
TEST(vfs_empty) {
|
||||
Vfs v;
|
||||
CHECK(!v.exists("anything"));
|
||||
CHECK(!v.stat("anything"));
|
||||
CHECK(v.list().empty());
|
||||
auto r = v.read("anything");
|
||||
CHECK(!r.ok() && r.error().code == Error::Code::NotFound);
|
||||
}
|
||||
50
tests/mars_vfs/vfs_cat.cpp
Normal file
50
tests/mars_vfs/vfs_cat.cpp
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
// vfs_cat -- print one file from a layered VFS to stdout (for byte-equality
|
||||
// checks against `unzip -p`, and as a tiny inspection tool).
|
||||
//
|
||||
// vfs_cat [--registration] (--zip ARCHIVE | --native DIR)... REL
|
||||
// vfs_cat --list (--zip ARCHIVE | --native DIR)... [PREFIX]
|
||||
#include <cstdio>
|
||||
#include <cstring>
|
||||
#include <string>
|
||||
|
||||
#include "mars/vfs/vfs.h"
|
||||
|
||||
using namespace mars::vfs;
|
||||
|
||||
int main(int argc, char** argv) {
|
||||
Order order = Order::NativeFirst;
|
||||
bool list = false;
|
||||
std::string rel;
|
||||
std::vector<std::pair<bool, std::string>> mounts; // (is_zip, path)
|
||||
for (int i = 1; i < argc; ++i) {
|
||||
std::string a = argv[i];
|
||||
if (a == "--registration") order = Order::Registration;
|
||||
else if (a == "--list") list = true;
|
||||
else if ((a == "--zip" || a == "--native") && i + 1 < argc) mounts.emplace_back(a == "--zip", argv[++i]);
|
||||
else rel = a;
|
||||
}
|
||||
if (mounts.empty() || (!list && rel.empty())) {
|
||||
std::fprintf(stderr, "usage: vfs_cat [--registration] (--zip A | --native D)... REL\n"
|
||||
" vfs_cat --list (--zip A | --native D)... [PREFIX]\n");
|
||||
return 2;
|
||||
}
|
||||
Vfs v(order);
|
||||
for (const auto& [is_zip, path] : mounts) {
|
||||
auto r = is_zip ? v.mount_zip(path) : v.mount_native(path);
|
||||
if (!r) {
|
||||
std::fprintf(stderr, "mount %s: %s\n", path.c_str(), r.error().message.c_str());
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
if (list) {
|
||||
for (const auto& e : v.list(rel)) std::printf("%10llu m%d %s\n", static_cast<unsigned long long>(e.size), e.mount, e.name.c_str());
|
||||
return 0;
|
||||
}
|
||||
auto bytes = v.read(rel);
|
||||
if (!bytes) {
|
||||
std::fprintf(stderr, "%s: %s\n", rel.c_str(), bytes.error().message.c_str());
|
||||
return 1;
|
||||
}
|
||||
std::fwrite(bytes->data(), 1, bytes->size(), stdout);
|
||||
return 0;
|
||||
}
|
||||
144
tests/mars_vfs/zip_builder.h
Normal file
144
tests/mars_vfs/zip_builder.h
Normal file
|
|
@ -0,0 +1,144 @@
|
|||
// Minimal in-memory ZIP writer for tests: stored entries, directory
|
||||
// placeholders, and "raw" entries whose bytes are supplied pre-packed so a
|
||||
// test can fabricate deflate / encrypted / odd-method records without a
|
||||
// compressor. Written independently of the reader under test.
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
#include <vector>
|
||||
|
||||
namespace ziptest {
|
||||
|
||||
inline std::uint32_t crc32(const std::uint8_t* p, std::size_t n, std::uint32_t crc = 0) {
|
||||
static std::uint32_t table[256];
|
||||
static bool init = false;
|
||||
if (!init) {
|
||||
for (std::uint32_t i = 0; i < 256; ++i) {
|
||||
std::uint32_t c = i;
|
||||
for (int k = 0; k < 8; ++k) c = (c & 1) ? 0xEDB88320u ^ (c >> 1) : c >> 1;
|
||||
table[i] = c;
|
||||
}
|
||||
init = true;
|
||||
}
|
||||
crc = ~crc;
|
||||
for (std::size_t i = 0; i < n; ++i) crc = table[(crc ^ p[i]) & 0xFF] ^ (crc >> 8);
|
||||
return ~crc;
|
||||
}
|
||||
|
||||
inline std::uint32_t crc32(std::string_view s) {
|
||||
return crc32(reinterpret_cast<const std::uint8_t*>(s.data()), s.size());
|
||||
}
|
||||
|
||||
struct Item {
|
||||
std::string name;
|
||||
std::vector<std::uint8_t> packed; // bytes as they sit in the file
|
||||
std::uint32_t uncompressed_size = 0;
|
||||
std::uint32_t crc = 0;
|
||||
std::uint16_t method = 0;
|
||||
std::uint16_t flags = 0;
|
||||
std::vector<std::uint8_t> local_extra; // extra field on the local header only
|
||||
std::vector<std::uint8_t> central_extra;
|
||||
std::uint32_t local_header_offset = 0; // filled by build()
|
||||
};
|
||||
|
||||
class ZipBuilder {
|
||||
public:
|
||||
Item& add(std::string name, std::string_view data) {
|
||||
Item it;
|
||||
it.name = std::move(name);
|
||||
it.packed.assign(data.begin(), data.end());
|
||||
it.uncompressed_size = static_cast<std::uint32_t>(data.size());
|
||||
it.crc = crc32(data);
|
||||
items_.push_back(std::move(it));
|
||||
return items_.back();
|
||||
}
|
||||
|
||||
Item& add_dir(std::string name) {
|
||||
if (name.empty() || name.back() != '/') name.push_back('/');
|
||||
return add(std::move(name), "");
|
||||
}
|
||||
|
||||
// Pre-packed bytes with explicit method / sizes / crc.
|
||||
Item& add_raw(std::string name, std::vector<std::uint8_t> packed, std::uint16_t method,
|
||||
std::uint32_t uncompressed_size, std::uint32_t crc) {
|
||||
Item it;
|
||||
it.name = std::move(name);
|
||||
it.packed = std::move(packed);
|
||||
it.method = method;
|
||||
it.uncompressed_size = uncompressed_size;
|
||||
it.crc = crc;
|
||||
items_.push_back(std::move(it));
|
||||
return items_.back();
|
||||
}
|
||||
|
||||
std::vector<Item>& items() { return items_; }
|
||||
|
||||
std::vector<std::uint8_t> build(std::string_view comment = {}) {
|
||||
std::vector<std::uint8_t> out;
|
||||
for (Item& it : items_) {
|
||||
it.local_header_offset = static_cast<std::uint32_t>(out.size());
|
||||
put32(out, 0x04034b50);
|
||||
put16(out, 20);
|
||||
put16(out, it.flags);
|
||||
put16(out, it.method);
|
||||
put16(out, 0); // time
|
||||
put16(out, 0); // date
|
||||
put32(out, it.crc);
|
||||
put32(out, static_cast<std::uint32_t>(it.packed.size()));
|
||||
put32(out, it.uncompressed_size);
|
||||
put16(out, static_cast<std::uint16_t>(it.name.size()));
|
||||
put16(out, static_cast<std::uint16_t>(it.local_extra.size()));
|
||||
out.insert(out.end(), it.name.begin(), it.name.end());
|
||||
out.insert(out.end(), it.local_extra.begin(), it.local_extra.end());
|
||||
out.insert(out.end(), it.packed.begin(), it.packed.end());
|
||||
}
|
||||
const std::uint32_t cd_offset = static_cast<std::uint32_t>(out.size());
|
||||
for (const Item& it : items_) {
|
||||
put32(out, 0x02014b50);
|
||||
put16(out, 20); // made by
|
||||
put16(out, 20); // needed
|
||||
put16(out, it.flags);
|
||||
put16(out, it.method);
|
||||
put16(out, 0);
|
||||
put16(out, 0);
|
||||
put32(out, it.crc);
|
||||
put32(out, static_cast<std::uint32_t>(it.packed.size()));
|
||||
put32(out, it.uncompressed_size);
|
||||
put16(out, static_cast<std::uint16_t>(it.name.size()));
|
||||
put16(out, static_cast<std::uint16_t>(it.central_extra.size()));
|
||||
put16(out, 0); // comment len
|
||||
put16(out, 0); // disk
|
||||
put16(out, 0); // internal attrs
|
||||
put32(out, 0); // external attrs
|
||||
put32(out, it.local_header_offset);
|
||||
out.insert(out.end(), it.name.begin(), it.name.end());
|
||||
out.insert(out.end(), it.central_extra.begin(), it.central_extra.end());
|
||||
}
|
||||
const std::uint32_t cd_size = static_cast<std::uint32_t>(out.size()) - cd_offset;
|
||||
put32(out, 0x06054b50);
|
||||
put16(out, 0);
|
||||
put16(out, 0);
|
||||
put16(out, static_cast<std::uint16_t>(items_.size()));
|
||||
put16(out, static_cast<std::uint16_t>(items_.size()));
|
||||
put32(out, cd_size);
|
||||
put32(out, cd_offset);
|
||||
put16(out, static_cast<std::uint16_t>(comment.size()));
|
||||
out.insert(out.end(), comment.begin(), comment.end());
|
||||
return out;
|
||||
}
|
||||
|
||||
private:
|
||||
static void put16(std::vector<std::uint8_t>& v, std::uint16_t x) {
|
||||
v.push_back(static_cast<std::uint8_t>(x & 0xFF));
|
||||
v.push_back(static_cast<std::uint8_t>(x >> 8));
|
||||
}
|
||||
static void put32(std::vector<std::uint8_t>& v, std::uint32_t x) {
|
||||
for (int i = 0; i < 4; ++i) v.push_back(static_cast<std::uint8_t>((x >> (8 * i)) & 0xFF));
|
||||
}
|
||||
|
||||
std::vector<Item> items_;
|
||||
};
|
||||
|
||||
} // namespace ziptest
|
||||
272
third_party/miniz/ChangeLog.md
vendored
Normal file
272
third_party/miniz/ChangeLog.md
vendored
Normal file
|
|
@ -0,0 +1,272 @@
|
|||
## Changelog
|
||||
|
||||
### 3.1.2
|
||||
|
||||
- Fix central directory offset overflow when reading zip file header
|
||||
- Add tdefl_write_image_to_png_file_in_memory_ex parameter validation
|
||||
- Add fuzz targets for ZIP validation, writing, and compression APIs
|
||||
- Support cmake 4
|
||||
- Guard against code_len==0 infinite loop in tinfl_decompress
|
||||
|
||||
### 3.1.1
|
||||
|
||||
- Declare function wrappers as inline static to fix warnings
|
||||
|
||||
### 3.1.0
|
||||
|
||||
- Fix warnings: Ensure correct integer promotion when adding
|
||||
- Fix Unicode paths on MinGW32
|
||||
- Prevent min/max conflicts between windows.h and std namespace
|
||||
- Update miniz_tdef.c to enable compiling in forced-C++ mode
|
||||
- Fix missing large file support warning on 64-bit Linux
|
||||
- Bump cmake minimum version
|
||||
- Add some catch2 tests including CI
|
||||
- Remove parameter check in `tinfl_decompress` that breaks `tinfl_decompress_mem_to_heap`
|
||||
- Don't redefine `WIN32_LEAN_AND_MEAN` if already defined
|
||||
- Fix OSS-Fuzz build
|
||||
- Do not redefine `TDEFL_LESS_MEMORY` if already defined
|
||||
- Fix unused arg warnings when building with `MINIZ_NO_TIME`
|
||||
- Support Zip archives not starting at zero offset
|
||||
- Fix offset detection for MZ_ZIP_TYPE_USER
|
||||
- Avoid fdreopen if possible
|
||||
- cmake: new option BUILD_NO_STDIO to enable MINIZ_NO_STDIO
|
||||
- Add fuzzer for mz_zip_add_mem_to_archive_file_in_place function
|
||||
- Replace defines with function wrappers etc. as much as possible
|
||||
|
||||
### 3.0.2
|
||||
|
||||
- Fix buffer overrun in mz_utf8z_to_widechar on Windows
|
||||
|
||||
### 3.0.1
|
||||
|
||||
- Fix compilation error with MINIZ_USE_UNALIGNED_LOADS_AND_STORES=1
|
||||
|
||||
### 3.0.0
|
||||
|
||||
- Reduce memory usage for inflate. This changes `struct tinfl_decompressor_tag` and therefore requires a major version bump (breaks ABI compatibility)
|
||||
- Add padding to structures so it continues to work if features differ. This also changes some structures
|
||||
- Use _ftelli64, _fseeki64 and stat with MinGW32 and OpenWatcom
|
||||
- Fix varios warnings with OpenWatcom compiler
|
||||
- Avoid using unaligned memory access in UBSan builds
|
||||
- Set MINIZ_LITTLE_ENDIAN only if not set
|
||||
- Add MINIZ_NO_DEFLATE_APIS and MINIZ_NO_INFLATE_APIS
|
||||
- Fix use of uninitialized memory in tinfl_decompress_mem_to_callback()
|
||||
- Use wfopen on windows
|
||||
- Use _wstat64 instead _stat64 on windows
|
||||
- Use level_and_flags after MZ_DEFAULT_COMPRESSION has been handled
|
||||
- Improve endianess detection
|
||||
- Don't use unaligned stores and loads per default
|
||||
- Fix function declaration if MINIZ_NO_STDIO is used
|
||||
- Fix MZ_ZIP_GENERAL_PURPOSE_BIT_FLAG_UTF8 not being set
|
||||
- Remove total files check (its 32-bit uint)
|
||||
- tinfl_decompress: avoid NULL ptr arithmetic UB
|
||||
- miniz_zip: fix mz_zip_reader_extract_to_heap to read correct sizes
|
||||
- Eliminate 64-bit operations on 32-bit machines
|
||||
- Disable treating warnings as error with MSVC
|
||||
- Disable building shared lib via CMake by default
|
||||
- Fixed alignment problems on MacOS
|
||||
- Fixed get error string for MZ_ZIP_TOTAL_ERRORS
|
||||
- Write correct FLEVEL 2-bit value in zlib header
|
||||
- miniz.pc.in: fix include path not containing the "miniz" suffix
|
||||
- Fix compatibility with FreeBSD
|
||||
- pkg-config tweaks
|
||||
- Fix integer overflow in header corruption check
|
||||
- Fix some warnings
|
||||
- tdefl_compress_normal: Avoid NULL ptr arithmetic UB
|
||||
- replace use of stdint.h types with mz_ variants
|
||||
|
||||
|
||||
### 2.2.0
|
||||
|
||||
- Fix examples with amalgamation
|
||||
- Modified cmake script to support shared library mode and find_package
|
||||
- Fix for misleading doc comment on `mz_zip_reader_init_cfile` function
|
||||
- Add include location tolerance and stop forcing `_GNU_SOURCE`
|
||||
- Fix: mz_zip_reader_locate_file_v2 returns an mz_bool
|
||||
- Fix large file system checks
|
||||
- Add #elif to enable an external mz_crc32() to be linked in
|
||||
- Write with dynamic size (size of file/data to be added not known before adding)
|
||||
- Added uncompress2 for zlib compatibility
|
||||
- Add support for building as a Meson subproject
|
||||
- Added OSSFuzz support; Integrate with CIFuzz
|
||||
- Add pkg-config file
|
||||
- Fixed use-of-uninitialized value msan error when copying dist bytes with no output bytes written.
|
||||
- mz_zip_validate_file(): fix memory leak on errors
|
||||
- Fixed MSAN use-of-uninitialized in tinfl_decompress when invalid dist is decoded. In this instance dist was 31 which s_dist_base translates as 0
|
||||
- Add flag to set (compressed) size in local file header
|
||||
- avoid use of uninitialized value in tdefl_record_literal
|
||||
|
||||
### 2.1.0
|
||||
|
||||
- More instances of memcpy instead of cast and use memcpy per default
|
||||
- Remove inline for c90 support
|
||||
- New function to read files via callback functions when adding them
|
||||
- Fix out of bounds read while reading Zip64 extended information
|
||||
- guard memcpy when n == 0 because buffer may be NULL
|
||||
- Implement inflateReset() function
|
||||
- Move comp/decomp alloc/free prototypes under guarding #ifndef MZ_NO_MALLOC
|
||||
- Fix large file support under Windows
|
||||
- Don't warn if _LARGEFILE64_SOURCE is not defined to 1
|
||||
- Fixes for MSVC warnings
|
||||
- Remove check that path of file added to archive contains ':' or '\'
|
||||
- Add !defined check on MINIZ_USE_ALIGNED_LOADS_AND_STORES
|
||||
|
||||
### 2.0.8
|
||||
|
||||
- Remove unimplemented functions (mz_zip_locate_file and mz_zip_locate_file_v2)
|
||||
- Add license, changelog, readme and example files to release zip
|
||||
- Fix heap overflow to user buffer in tinfl_status tinfl_decompress
|
||||
- Fix corrupt archive if uncompressed file smaller than 4 byte and the file is added by mz_zip_writer_add_mem*
|
||||
|
||||
### 2.0.7
|
||||
|
||||
- Removed need in C++ compiler in cmake build
|
||||
- Fixed a lot of uninitialized value errors found with Valgrind by memsetting m_dict to 0 in tdefl_init
|
||||
- Fix resource leak in mz_zip_reader_init_file_v2
|
||||
- Fix assert with mz_zip_writer_add_mem* w/MZ_DEFAULT_COMPRESSION
|
||||
- cmake build: install library and headers
|
||||
- Remove _LARGEFILE64_SOURCE requirement from apple defines for large files
|
||||
|
||||
### 2.0.6
|
||||
|
||||
- Improve MZ_ZIP_FLAG_WRITE_ZIP64 documentation
|
||||
- Remove check for cur_archive_file_ofs > UINT_MAX because cur_archive_file_ofs is not used after this point
|
||||
- Add cmake debug configuration
|
||||
- Fix PNG height when creating png files
|
||||
- Add "iterative" file extraction method based on mz_zip_reader_extract_to_callback.
|
||||
- Option to use memcpy for unaligned data access
|
||||
- Define processor/arch macros as zero if not set to one
|
||||
|
||||
### 2.0.4/2.0.5
|
||||
|
||||
- Fix compilation with the various omission compile definitions
|
||||
|
||||
### 2.0.3
|
||||
|
||||
- Fix GCC/clang compile warnings
|
||||
- Added callback for periodic flushes (for ZIP file streaming)
|
||||
- Use UTF-8 for file names in ZIP files per default
|
||||
|
||||
### 2.0.2
|
||||
|
||||
- Fix source backwards compatibility with 1.x
|
||||
- Fix a ZIP bit not being set correctly
|
||||
|
||||
### 2.0.1
|
||||
|
||||
- Added some tests
|
||||
- Added CI
|
||||
- Make source code ANSI C compatible
|
||||
|
||||
### 2.0.0 beta
|
||||
|
||||
- Matthew Sitton merged miniz 1.x to Rich Geldreich's vogl ZIP64 changes. Miniz is now licensed as MIT since the vogl code base is MIT licensed
|
||||
- Miniz is now split into several files
|
||||
- Miniz does now not seek backwards when creating ZIP files. That is the ZIP files can be streamed
|
||||
- Miniz automatically switches to the ZIP64 format when the created ZIP files goes over ZIP file limits
|
||||
- Similar to [SQLite](https://www.sqlite.org/amalgamation.html) the Miniz source code is amalgamated into one miniz.c/miniz.h pair in a build step (amalgamate.sh). Please use miniz.c/miniz.h in your projects
|
||||
- Miniz 2 is only source back-compatible with miniz 1.x. It breaks binary compatibility because structures changed
|
||||
|
||||
### v1.16 BETA Oct 19, 2013
|
||||
|
||||
Still testing, this release is downloadable from [here](http://www.tenacioussoftware.com/miniz_v116_beta_r1.7z). Two key inflator-only robustness and streaming related changes. Also merged in tdefl_compressor_alloc(), tdefl_compressor_free() helpers to make script bindings easier for rustyzip. I would greatly appreciate any help with testing or any feedback.
|
||||
|
||||
The inflator in raw (non-zlib) mode is now usable on gzip or similar streams that have a bunch of bytes following the raw deflate data (problem discovered by rustyzip author williamw520). This version should never read beyond the last byte of the raw deflate data independent of how many bytes you pass into the input buffer.
|
||||
|
||||
The inflator now has a new failure status TINFL_STATUS_FAILED_CANNOT_MAKE_PROGRESS (-4). Previously, if the inflator was starved of bytes and could not make progress (because the input buffer was empty and the caller did not set the TINFL_FLAG_HAS_MORE_INPUT flag - say on truncated or corrupted compressed data stream) it would append all 0's to the input and try to soldier on. This is scary behavior if the caller didn't know when to stop accepting output (because it didn't know how much uncompressed data was expected, or didn't enforce a sane maximum). v1.16 will instead return TINFL_STATUS_FAILED_CANNOT_MAKE_PROGRESS immediately if it needs 1 or more bytes to make progress, the input buf is empty, and the caller has indicated that no more input is available. This is a "soft" failure, so you can call the inflator again with more input and it will try to continue, or you can give up and fail. This could be very useful in network streaming scenarios.
|
||||
|
||||
- The inflator coroutine func. is subtle and complex so I'm being cautious about this release. I would greatly appreciate any help with testing or any feedback.
|
||||
I feel good about these changes, and they've been through several hours of automated testing, but they will probably not fix anything for the majority of prev. users so I'm
|
||||
going to mark this release as beta for a few weeks and continue testing it at work/home on various things.
|
||||
- The inflator in raw (non-zlib) mode is now usable on gzip or similar data streams that have a bunch of bytes following the raw deflate data (problem discovered by rustyzip author williamw520).
|
||||
This version should *never* read beyond the last byte of the raw deflate data independent of how many bytes you pass into the input buffer. This issue was caused by the various Huffman bitbuffer lookahead optimizations, and
|
||||
would not be an issue if the caller knew and enforced the precise size of the raw compressed data *or* if the compressed data was in zlib format (i.e. always followed by the byte aligned zlib adler32).
|
||||
So in other words, you can now call the inflator on deflate streams that are followed by arbitrary amounts of data and it's guaranteed that decompression will stop exactly on the last byte.
|
||||
- The inflator now has a new failure status: TINFL_STATUS_FAILED_CANNOT_MAKE_PROGRESS (-4). Previously, if the inflator was starved of bytes and could not make progress (because the input buffer was empty and the
|
||||
caller did not set the TINFL_FLAG_HAS_MORE_INPUT flag - say on truncated or corrupted compressed data stream) it would append all 0's to the input and try to soldier on.
|
||||
This is scary, because in the worst case, I believe it was possible for the prev. inflator to start outputting large amounts of literal data. If the caller didn't know when to stop accepting output
|
||||
(because it didn't know how much uncompressed data was expected, or didn't enforce a sane maximum) it could continue forever. v1.16 cannot fall into this failure mode, instead it'll return
|
||||
TINFL_STATUS_FAILED_CANNOT_MAKE_PROGRESS immediately if it needs 1 or more bytes to make progress, the input buf is empty, and the caller has indicated that no more input is available. This is a "soft"
|
||||
failure, so you can call the inflator again with more input and it will try to continue, or you can give up and fail. This could be very useful in network streaming scenarios.
|
||||
- Added documentation to all the tinfl return status codes, fixed miniz_tester so it accepts double minus params for Linux, tweaked example1.c, added a simple "follower bytes" test to miniz_tester.cpp.
|
||||
### v1.15 r4 STABLE - Oct 13, 2013
|
||||
|
||||
Merged over a few very minor bug fixes that I fixed in the zip64 branch. This is downloadable from [here](http://code.google.com/p/miniz/downloads/list) and also in SVN head (as of 10/19/13).
|
||||
|
||||
|
||||
### v1.15 - Oct. 13, 2013
|
||||
|
||||
Interim bugfix release while I work on the next major release with zip64 and streaming compression/decompression support. Fixed the MZ_ZIP_FLAG_DO_NOT_SORT_CENTRAL_DIRECTORY bug (thanks kahmyong.moon@hp.com), which could cause the locate files func to not find files when this flag was specified. Also fixed a bug in mz_zip_reader_extract_to_mem_no_alloc() with user provided read buffers (thanks kymoon). I also merged lots of compiler fixes from various github repo branches and Google Code issue reports. I finally added cmake support (only tested under for Linux so far), compiled and tested with clang v3.3 and gcc 4.6 (under Linux), added defl_write_image_to_png_file_in_memory_ex() (supports Y flipping for OpenGL use, real-time compression), added a new PNG example (example6.c - Mandelbrot), and I added 64-bit file I/O support (stat64(), etc.) for glibc.
|
||||
|
||||
- Critical fix for the MZ_ZIP_FLAG_DO_NOT_SORT_CENTRAL_DIRECTORY bug (thanks kahmyong.moon@hp.com) which could cause locate files to not find files. This bug
|
||||
would only have occurred in earlier versions if you explicitly used this flag, OR if you used mz_zip_extract_archive_file_to_heap() or mz_zip_add_mem_to_archive_file_in_place()
|
||||
(which used this flag). If you can't switch to v1.15 but want to fix this bug, just remove the uses of this flag from both helper funcs (and of course don't use the flag).
|
||||
- Bugfix in mz_zip_reader_extract_to_mem_no_alloc() from kymoon when pUser_read_buf is not NULL and compressed size is > uncompressed size
|
||||
- Fixing mz_zip_reader_extract_*() funcs so they don't try to extract compressed data from directory entries, to account for weird zipfiles which contain zero-size compressed data on dir entries.
|
||||
Hopefully this fix won't cause any issues on weird zip archives, because it assumes the low 16-bits of zip external attributes are DOS attributes (which I believe they always are in practice).
|
||||
- Fixing mz_zip_reader_is_file_a_directory() so it doesn't check the internal attributes, just the filename and external attributes
|
||||
- mz_zip_reader_init_file() - missing MZ_FCLOSE() call if the seek failed
|
||||
- Added cmake support for Linux builds which builds all the examples, tested with clang v3.3 and gcc v4.6.
|
||||
- Clang fix for tdefl_write_image_to_png_file_in_memory() from toffaletti
|
||||
- Merged MZ_FORCEINLINE fix from hdeanclark
|
||||
- Fix <time.h> include before config #ifdef, thanks emil.brink
|
||||
- Added tdefl_write_image_to_png_file_in_memory_ex(): supports Y flipping (super useful for OpenGL apps), and explicit control over the compression level (so you can
|
||||
set it to 1 for real-time compression).
|
||||
- Merged in some compiler fixes from paulharris's github repro.
|
||||
- Retested this build under Windows (VS 2010, including static analysis), tcc 0.9.26, gcc v4.6 and clang v3.3.
|
||||
- Added example6.c, which dumps an image of the mandelbrot set to a PNG file.
|
||||
- Modified example2 to help test the MZ_ZIP_FLAG_DO_NOT_SORT_CENTRAL_DIRECTORY flag more.
|
||||
- In r3: Bugfix to mz_zip_writer_add_file() found during merge: Fix possible src file fclose() leak if alignment bytes+local header file write faiiled
|
||||
- In r4: Minor bugfix to mz_zip_writer_add_from_zip_reader(): Was pushing the wrong central dir header offset, appears harmless in this release, but it became a problem in the zip64 branch
|
||||
|
||||
### v1.14 - May 20, 2012
|
||||
|
||||
(SVN Only) Minor tweaks to get miniz.c compiling with the Tiny C Compiler, added #ifndef MINIZ_NO_TIME guards around utime.h includes. Adding mz_free() function, so the caller can free heap blocks returned by miniz using whatever heap functions it has been configured to use, MSVC specific fixes to use "safe" variants of several functions (localtime_s, fopen_s, freopen_s).
|
||||
|
||||
MinGW32/64 GCC 4.6.1 compiler fixes: added MZ_FORCEINLINE, #include <time.h> (thanks fermtect).
|
||||
|
||||
Compiler specific fixes, some from fermtect. I upgraded to TDM GCC 4.6.1 and now static __forceinline is giving it fits, so I'm changing all usage of __forceinline to MZ_FORCEINLINE and forcing gcc to use __attribute__((__always_inline__)) (and MSVC to use __forceinline). Also various fixes from fermtect for MinGW32: added #include , 64-bit ftell/fseek fixes.
|
||||
|
||||
### v1.13 - May 19, 2012
|
||||
|
||||
From jason@cornsyrup.org and kelwert@mtu.edu - Most importantly, fixed mz_crc32() so it doesn't compute the wrong CRC-32's when mz_ulong is 64-bits. Temporarily/locally slammed in "typedef unsigned long mz_ulong" and re-ran a randomized regression test on ~500k files. Other stuff:
|
||||
|
||||
Eliminated a bunch of warnings when compiling with GCC 32-bit/64. Ran all examples, miniz.c, and tinfl.c through MSVC 2008's /analyze (static analysis) option and fixed all warnings (except for the silly "Use of the comma-operator in a tested expression.." analysis warning, which I purposely use to work around a MSVC compiler warning).
|
||||
|
||||
Created 32-bit and 64-bit Codeblocks projects/workspace. Built and tested Linux executables. The codeblocks workspace is compatible with Linux+Win32/x64. Added miniz_tester solution/project, which is a useful little app derived from LZHAM's tester app that I use as part of the regression test. Ran miniz.c and tinfl.c through another series of regression testing on ~500,000 files and archives. Modified example5.c so it purposely disables a bunch of high-level functionality (MINIZ_NO_STDIO, etc.). (Thanks to corysama for the MINIZ_NO_STDIO bug report.)
|
||||
|
||||
Fix ftell() usage in a few of the examples so they exit with an error on files which are too large (a limitation of the examples, not miniz itself). Fix fail logic handling in mz_zip_add_mem_to_archive_file_in_place() so it always calls mz_zip_writer_finalize_archive() and mz_zip_writer_end(), even if the file add fails.
|
||||
|
||||
- From jason@cornsyrup.org and kelwert@mtu.edu - Fix mz_crc32() so it doesn't compute the wrong CRC-32's when mz_ulong is 64-bit.
|
||||
- Temporarily/locally slammed in "typedef unsigned long mz_ulong" and re-ran a randomized regression test on ~500k files.
|
||||
- Eliminated a bunch of warnings when compiling with GCC 32-bit/64.
|
||||
- Ran all examples, miniz.c, and tinfl.c through MSVC 2008's /analyze (static analysis) option and fixed all warnings (except for the silly
|
||||
"Use of the comma-operator in a tested expression.." analysis warning, which I purposely use to work around a MSVC compiler warning).
|
||||
- Created 32-bit and 64-bit Codeblocks projects/workspace. Built and tested Linux executables. The codeblocks workspace is compatible with Linux+Win32/x64.
|
||||
- Added miniz_tester solution/project, which is a useful little app derived from LZHAM's tester app that I use as part of the regression test.
|
||||
- Ran miniz.c and tinfl.c through another series of regression testing on ~500,000 files and archives.
|
||||
- Modified example5.c so it purposely disables a bunch of high-level functionality (MINIZ_NO_STDIO, etc.). (Thanks to corysama for the MINIZ_NO_STDIO bug report.)
|
||||
- Fix ftell() usage in examples so they exit with an error on files which are too large (a limitation of the examples, not miniz itself).
|
||||
|
||||
### v1.12 - 4/12/12
|
||||
|
||||
More comments, added low-level example5.c, fixed a couple minor level_and_flags issues in the archive API's.
|
||||
level_and_flags can now be set to MZ_DEFAULT_COMPRESSION. Thanks to Bruce Dawson <bruced@valvesoftware.com> for the feedback/bug report.
|
||||
|
||||
### v1.11 - 5/28/11
|
||||
|
||||
Added statement from unlicense.org
|
||||
|
||||
### v1.10 - 5/27/11
|
||||
|
||||
- Substantial compressor optimizations:
|
||||
- Level 1 is now ~4x faster than before. The L1 compressor's throughput now varies between 70-110MB/sec. on a Core i7 (actual throughput varies depending on the type of data, and x64 vs. x86).
|
||||
- Improved baseline L2-L9 compression perf. Also, greatly improved compression perf. issues on some file types.
|
||||
- Refactored the compression code for better readability and maintainability.
|
||||
- Added level 10 compression level (L10 has slightly better ratio than level 9, but could have a potentially large drop in throughput on some files).
|
||||
|
||||
### v1.09 - 5/15/11
|
||||
|
||||
Initial stable release.
|
||||
|
||||
|
||||
22
third_party/miniz/LICENSE
vendored
Normal file
22
third_party/miniz/LICENSE
vendored
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
Copyright 2013-2014 RAD Game Tools and Valve Software
|
||||
Copyright 2010-2014 Rich Geldreich and Tenacious Software LLC
|
||||
|
||||
All Rights Reserved.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in
|
||||
all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||
THE SOFTWARE.
|
||||
7922
third_party/miniz/miniz.c
vendored
Normal file
7922
third_party/miniz/miniz.c
vendored
Normal file
File diff suppressed because it is too large
Load diff
1510
third_party/miniz/miniz.h
vendored
Normal file
1510
third_party/miniz/miniz.h
vendored
Normal file
File diff suppressed because it is too large
Load diff
46
third_party/miniz/readme.md
vendored
Normal file
46
third_party/miniz/readme.md
vendored
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
## Miniz
|
||||
|
||||
Miniz is a lossless, high performance data compression library in a single source file that implements the zlib (RFC 1950) and Deflate (RFC 1951) compressed data format specification standards. It supports the most commonly used functions exported by the zlib library, but is a completely independent implementation so zlib's licensing requirements do not apply. Miniz also contains simple to use functions for writing .PNG format image files and reading/writing/appending .ZIP format archives. Miniz's compression speed has been tuned to be comparable to zlib's, and it also has a specialized real-time compressor function designed to compare well against fastlz/minilzo.
|
||||
|
||||
## Usage
|
||||
|
||||
Releases are available at the [releases page](https://github.com/richgel999/miniz/releases) as a pair of `miniz.c`/`miniz.h` files which can be simply added to a project. To create this file pair the different source and header files are [amalgamated](https://www.sqlite.org/amalgamation.html) during build. Alternatively use as cmake or meson module (or build system of your choice).
|
||||
|
||||
## Features
|
||||
|
||||
* MIT licensed
|
||||
* A portable, single source and header file library written in plain C. Tested with GCC, clang and Visual Studio.
|
||||
* Easily tuned and trimmed down by defines
|
||||
* A drop-in replacement for zlib's most used API's (tested in several open source projects that use zlib, such as libpng and libzip).
|
||||
* Fills a single threaded performance vs. compression ratio gap between several popular real-time compressors and zlib. For example, at level 1, miniz.c compresses around 5-9% better than minilzo, but is approx. 35% slower. At levels 2-9, miniz.c is designed to compare favorably against zlib's ratio and speed. See the miniz performance comparison page for example timings.
|
||||
* Not a block based compressor: miniz.c fully supports stream based processing using a coroutine-style implementation. The zlib-style API functions can be called a single byte at a time if that's all you've got.
|
||||
* Easy to use. The low-level compressor (tdefl) and decompressor (tinfl) have simple state structs which can be saved/restored as needed with simple memcpy's. The low-level codec API's don't use the heap in any way.
|
||||
* Entire inflater (including optional zlib header parsing and Adler-32 checking) is implemented in a single function as a coroutine, which is separately available in a small (~550 line) source file: miniz_tinfl.c
|
||||
* A fairly complete (but totally optional) set of .ZIP archive manipulation and extraction API's. The archive functionality is intended to solve common problems encountered in embedded, mobile, or game development situations. (The archive API's are purposely just powerful enough to write an entire archiver given a bit of additional higher-level logic.)
|
||||
|
||||
## Building miniz - Using vcpkg
|
||||
|
||||
You can download and install miniz using the [vcpkg](https://github.com/Microsoft/vcpkg) dependency manager:
|
||||
|
||||
git clone https://github.com/Microsoft/vcpkg.git
|
||||
cd vcpkg
|
||||
./bootstrap-vcpkg.sh
|
||||
./vcpkg integrate install
|
||||
./vcpkg install miniz
|
||||
|
||||
The miniz port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please [create an issue or pull request](https://github.com/Microsoft/vcpkg) on the vcpkg repository.
|
||||
|
||||
## Known Problems
|
||||
|
||||
* No support for encrypted archives. Not sure how useful this stuff is in practice.
|
||||
* Minimal documentation. The assumption is that the user is already familiar with the basic zlib API. I need to write an API wiki - for now I've tried to place key comments before each enum/API, and I've included 6 examples that demonstrate how to use the module's major features.
|
||||
|
||||
## Special Thanks
|
||||
|
||||
Thanks to Alex Evans for the PNG writer function. Also, thanks to Paul Holden and Thorsten Scheuermann for feedback and testing, Matt Pritchard for all his encouragement, and Sean Barrett's various public domain libraries for inspiration (and encouraging me to write miniz.c in C, which was much more enjoyable and less painful than I thought it would be considering I've been programming in C++ for so long).
|
||||
|
||||
Thanks to Bruce Dawson for reporting a problem with the level_and_flags archive API parameter (which is fixed in v1.12) and general feedback, and Janez Zemva for indirectly encouraging me into writing more examples.
|
||||
|
||||
## Patents
|
||||
|
||||
I was recently asked if miniz avoids patent issues. miniz purposely uses the same core algorithms as the ones used by zlib. The compressor uses vanilla hash chaining as described [here](https://datatracker.ietf.org/doc/html/rfc1951#section-4). Also see the [gzip FAQ](https://web.archive.org/web/20160308045258/http://www.gzip.org/#faq11). In my opinion, if miniz falls prey to a patent attack then zlib/gzip are likely to be at serious risk too.
|
||||
Loading…
Add table
Reference in a new issue