From 15ecd22aa69c4df2d8cccb95eeaae92511d8b49e Mon Sep 17 00:00:00 2001 From: alex Date: Mon, 7 Sep 2026 15:55:27 -0400 Subject: [PATCH] verify: Streamable save reader + writer stub + tests + SAVE_FORMAT --- campaign/board.md | 2 +- campaign/open-questions.md | 1 + .../__pycache__/effect_txt.cpython-310.pyc | Bin 3381 -> 0 bytes .../__pycache__/flat_kv.cpython-310.pyc | Bin 5218 -> 0 bytes .../__pycache__/manifest.cpython-310.pyc | Bin 5082 -> 0 bytes .../__pycache__/mars_data.cpython-310.pyc | Bin 6793 -> 0 bytes save-reader/ref/parsers/effect_txt.py | 98 --- save-reader/ref/parsers/flat_kv.py | 127 ---- save-reader/ref/parsers/manifest.py | 121 ---- save-reader/ref/parsers/mars_data.py | 211 ------ save-reader/ref/parsers/verify.py | 634 ------------------ save-reader/ref/save-editor-structs.md | 550 --------------- save-reader/ref/struct-recovery.md | 461 ------------- .../save-reader}/SAVE_FORMAT.md | 0 .../__pycache__/save_reader.cpython-310.pyc | Bin .../save_writer_stub.cpython-310.pyc | Bin .../test_save_reader.cpython-310.pyc | Bin .../save-reader}/fixture_joint.sav | Bin .../save-reader}/fixture_split.sav | Bin .../save-reader}/save_reader.py | 0 .../save-reader}/save_writer_stub.py | 0 .../save-reader}/test_save_reader.py | 0 22 files changed, 2 insertions(+), 2203 deletions(-) delete mode 100644 save-reader/ref/parsers/__pycache__/effect_txt.cpython-310.pyc delete mode 100644 save-reader/ref/parsers/__pycache__/flat_kv.cpython-310.pyc delete mode 100644 save-reader/ref/parsers/__pycache__/manifest.cpython-310.pyc delete mode 100644 save-reader/ref/parsers/__pycache__/mars_data.cpython-310.pyc delete mode 100644 save-reader/ref/parsers/effect_txt.py delete mode 100644 save-reader/ref/parsers/flat_kv.py delete mode 100644 save-reader/ref/parsers/manifest.py delete mode 100644 save-reader/ref/parsers/mars_data.py delete mode 100644 save-reader/ref/parsers/verify.py delete mode 100644 save-reader/ref/save-editor-structs.md delete mode 100644 save-reader/ref/struct-recovery.md rename {save-reader => verify/save-reader}/SAVE_FORMAT.md (100%) rename {save-reader => verify/save-reader}/__pycache__/save_reader.cpython-310.pyc (100%) rename {save-reader => verify/save-reader}/__pycache__/save_writer_stub.cpython-310.pyc (100%) rename {save-reader => verify/save-reader}/__pycache__/test_save_reader.cpython-310.pyc (100%) rename {save-reader => verify/save-reader}/fixture_joint.sav (100%) rename {save-reader => verify/save-reader}/fixture_split.sav (100%) rename {save-reader => verify/save-reader}/save_reader.py (100%) rename {save-reader => verify/save-reader}/save_writer_stub.py (100%) rename {save-reader => verify/save-reader}/test_save_reader.py (100%) diff --git a/campaign/board.md b/campaign/board.md index 9464ea2..d106913 100644 --- a/campaign/board.md +++ b/campaign/board.md @@ -30,5 +30,5 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`). | `Game::StrategyServer` (sim block) | object | mapped | high | 80% | 2026-09-07 | top-level sim state; read FUN_007d27a0 / write FUN_0079fa70; IStreamable +0 | | stream primitive API | subsystem | mapped | high | 100% | 2026-09-07 | IStreamable vft: +0x18 string, +0x1c bool, +0x20 float, +0x24 int, +0x28 nested, +0x30 raw; FUN_00816490 = NetworkObject handle id | | real save for verification | verify | in-progress | — | 0% | 2026-09-07 | need the game running (DXVK+lavapipe on VM140) -> autosave -> parse with recovered layouts | -| save_reader.py | verify | in-progress | — | 0% | 2026-09-07 | Streamable reader (gzip, name-tagged, BEEFBEEF framing) from reference + recovered layouts | +| save_reader.py | verify | mapped | high | 90% | 2026-09-07 | verify/save-reader/: walker+schema, 26 tests pass on synthetic fixtures (both padding modes); SAVE_FORMAT.md. 'verified' only after a real save parses --strict | | Ghidra type write-back | meta | verified | high | 100% | 2026-09-07 | structs saved in project (ServerSystem 87f, ServerPlayer 110f, StarFleet, StarShip, StrategyServer partial, 22 nested); 52 serializers + primitives + ~60 spine fns renamed; decompile shows field names | diff --git a/campaign/open-questions.md b/campaign/open-questions.md index d4032e8..c1de6bb 100644 --- a/campaign/open-questions.md +++ b/campaign/open-questions.md @@ -22,3 +22,4 @@ Each links to the finding that raised it. Promoted to backlog or closed by **re- - **Resolved (R1/R2 contradictions)** — `Bats2`/`rcex` are int64 (R2 wrong); `Abdn`/`Dstyd` bools, `ltis` int; `TRM`/`CstR/E/T`/`shrm`/`RefCap`/`RepCap`/PlayerView `Infra` are floats; `pswd` string; `TShn`/`ETS`/diplomacy counters int16 in memory, int32 on disk; `Nexp` carries `xid/xmin/xmax/xper`; `FtOrig` is Vector3. (from [[struct-recovery]]) - **Battle-load, narrowed** — sim/combat load run on the main thread; the only other threads are net watchdog, TIME_CRITICAL audio streaming (`g_musicCS`), and a star-map mesh builder. Hypothesis: audio-thread critical-section contention or D3D9 runtime/driver threads on many cores. Needs a dynamic profile (x32dbg / ETW) under the software-GPU stack. (from [[turn-spine]]) - **Spine leftovers** — `StrategyServer` struct partial (41 fields); static-initialiser region 0x009be000–0x009c1400 undisassembled; `Mars::Stream` vftable not located; several small ProcessTurn phase fns unnamed. (from [[turn-spine]]) +- **Save framing ambiguities (settle on first real save)** — padding joint `[len][name][value][pad]` vs split; bool vs int for names with len%4==0 (no type byte); on-disk tags for Summary/CreateParameters and count/element tags inside framed arrays unknown (positional for now). A real 3-char bool tag (`NPC`,`Dep`,`hsp`) settles padding. (from [[SAVE_FORMAT]]) diff --git a/save-reader/ref/parsers/__pycache__/effect_txt.cpython-310.pyc b/save-reader/ref/parsers/__pycache__/effect_txt.cpython-310.pyc deleted file mode 100644 index dc1e5ed0acc95a36c550f28cbbf12ef5918ba69c..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 3381 zcmZuz-EJI76|U;;`D>3I+u3BDz%HdEU=JI6cEgX5leJ>vNw79M4z>wo*l1{X)r=eW zbdRfQ?AYqL!9lPRTEq=EOGJ#|VkO>y2jCs*60Y(J+$`r*k0*|l9(7Hfs#B-V?{_+# zob(Ai7cTtu*Z-*!@?Rv*KOPWw38Cr_FyW*{IOR@Edrr%tICo>W=e0aL_u@jY*edo) zt&*h+ak*D%RVe8aUi=$v`C_6~eL#4Lm!A_}7LzA#Yl>I6{};-?$!GZN^J41)e~(|{ zm!EsBY2bX9&jIJ0m;lZh{yx9LufonO|Hc-neJkKDA)?a}p>FGEdZ~ZJ78jWm0T+^W zQpp;2Pu0J>lyk9#ejs%e#$s`_z(prYA|0iPsxA3`b7Pz7UBPx_5Q@c}I1TquP%qF+ zj6DpF(t%#~eFo3=FShaT?ay`A3%XIrI+0YG#Ze;cnNJ!|*g+5v#M@3?XDSThK(d8o z(AyF6CfnZpS)+b`_2E|ICcCyjNVVYCYGXlflVR7cv#v}B{bh#Nr3s6)V(DRG)wqe) zB~}LV{=EL>-4HgdM zfl$C2XtpEbH0d&(<|GevppW|ECJPeI=0Dw7Tfe{F*qryJ=!*aiO+vx?pcQgK%(hY{ zf^e6`;y}cxj1+nx6Hq2lAcLi70#I=Tfl536FZaaJ%{)j?SwDdIv4z|VAD_hs@@nKF zJXC{AblT=os26Ii8=>Zbk0d*Y#35Ta(`<3YUZQI35JHe&j0z>G26=)D4*B=fK_vGS z+Xd@l!I!}t0Z9}KSW)cW=EM89*?e>L(+0cN1lf&Vgs#6fpDV0bm?k=i5?}QpebsFi zC0xJ?rZ-*{rm80({n>&Jc2w=QjYgi{xQ%(bxDzO=$(H7Ev;zvnM@#-I1cVwj6F~y% zT6qdccQrYxIi{-maRe2Z?ukU1Nu_1fZ-;5G2X}u(O)*S`3`NZ|1*kqHjBo$iXqPFC zd}n6zTHQJVeV;X?Ol63do_{D1cM*vIridUd3TWhT3P|9{30P{@TvKkhlb|QsZR5Aw zy_64PoLAfJ{Xr1tJ6{ob0l4k(zj}Q)?TLC2i)VF}YPBdu{Xod5b5yrlN>$I1VgxYu zqU&&6Aug7QNA1cs*jd1oX@lbok_p z0@|U=S?PohD_I5Y#CPa}+{#Gq=b4W;{*H7gpW+vumw|6&>l%S`6Ju@h)Wqo2Ke7pw zl+bJ*xv~xzy5wLywg@c-m^!zvIIb$`PJ+lO5L6Ti;N(`5A0s4<;dXZ8Q zAQSch3-+&Y^Ob`f1&gc41uQ6bd}%qyS#}udU1%UIU;r|&zJ|xRfTPD3ml^zxxAVir zW#nSVmn}e?hfupe&LFrCGlyV0Uc?N#0h2D|(9W}Cu&}F-G z8UCvyzN>TrX1Id8Rw@D`pn~zRZM+d#vL*8G#z#fap3y7ozHFdg90r zGFN_p6lQC_qMe+As&Jr2$Geup#WR2qP{Bh$kc=J>xq&&70gl-7E4q+V+~wYhs~$kr z*xfGV5zg+6xL=ks7tvuKq~+@{Ls%~46(g^L;>K%3C%j)P8$W`=Q96K5jqZmye+TyE zJ2>E{s3~;?^cXGS$foGaZ^K&6lech-a`_?tyAYt)C@yV$@N5%ag_1n*@e262Oy7ry zOjF+hq(+Vq-ZvgB+q>vq6!N;Ov|J8FQgh{xV0pU<3Ls9)NgSqd_&yHfM)JC3?19f< zbJQUq#wg;AuwubtI(>roVmg820sKy!KVBL-8eIF*IVCSagHzg}5xDv9^b7iP;s5}9 z@H=rS0MbW9y>|$1$KMlpIf9hiG=Rp?bNe6N`oS&exn2snpJ$neU&f}$MVN8{Mep?k zy=$DkLqj{~EV0h4fKtDLS0O-D=^VZBwZye%t(aGt4Hq^zc?pNhIJ^tPC{lKwuYM0# zopT%)O{v}HX_zCa8{{w*nLudd1 diff --git a/save-reader/ref/parsers/__pycache__/flat_kv.cpython-310.pyc b/save-reader/ref/parsers/__pycache__/flat_kv.cpython-310.pyc deleted file mode 100644 index bbe3d5c82174cfed459b7b16e58bff6a5ae2e375..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 5218 zcma)A+jARN8Q-%PNs(`9>bhydgj1)jwW%y6>5V#RptnFNm{O-Hb!s$gog-U#wJV=p z*|AwMvh-&x7B9T$*hkG?&ZZ_oMtzS}CE zm?&HLOilgyXAe$W)<5Xx@W;i?cPxt~e}f>bWlJz&hpb~S+l*uR0(gzil^(6(=X%iD2hHAw51Xd16{;w1H} zxi%R!<6e})HjD~htAVH?D=OYjhPHE`5p-I24Y~tZhuqY=VXz}rZZ%)8Yv42H=N7qOPR7uK^54uYv+{>*@dEicAGYG%hMA>Ee$%G zo*Jn#9lKMrl{-J3UAg4XW|dEuUi7s~AxJyWsSB{Ro{E}`wmkTF_@$)tHYJ^P2!i#I za{6{^W!62`cQQM3HY{~DvoiY`tM6v6a5ijpHe=&fpG9YI@8bSa#*#~!wa0{)+52>l zWR%&z!kQ(l6vfx9zMbnz#xwN!rQL6TC_C{Mv5$b##y;10y*}i;%G+yEOd*gu2X-Rj z9!xVgmZWx5(McY$*+;CtYuD?GU(&sAYbTVEE+Y!7G}&&a>%ONZF{0ge)Y#Olty!d5 z^P#2A(fVqAJ?_YQ5X!stBu@(%E?^3^yFS5}VT??6idm%;x72x~w!1&*1u&)%_CE)Z&awqW~U zrN`9yjO~GwL66h2_ZhP`iwC_f=6R!D;9S_PE|CQRW-hASCIUrVL(vFQ;_>Jr@^S#Xm-yvatAqyl7c=Zqh3c+0iiC_5(*RQ z3N_>aY8s7y>JTXfVh{1{B^ssRq+Uj&9b_c!BFQD%L3@lPUC8(FLrhI((AdA6IUs5r zRB?Eh`!BUXs@yO@wzG2^DCFr&T{k15-AoUVAzF#wgD6BtZd;%Zgu<+eO#Z3Ucl%zy zka?N2WvPpZ=tRYZ6h4%>Us%G)yf3Ujf!NO(#wa6PL{utd=)X?!iCzo5-M^12>tPYb zQNhuEP%*tZD3m*RQ{Zt6B^kmns{LLxs1EWRXlf7S>S0!=xR%iR{EVn9=;U*}f;Wso z3C_2XKu~bbwGiUcIS5jzSk2Xpu8&Y6AJha%#YdHi$^kDRc{zb{fQpNn_ROMP&xFz4 z(+i_TG=Xc+Xb0p5|MG2muX&fA;}=KF=M*rZUIomGBBOep8cIcZWBifW_ykvS294`Z znh36luzhr%Rg7#}4+HRc^Q$P_tLCkaK*Bp4>Mu$Rf#RJQ?q=#>OeMb}%U+BcDZR9npL!v>r_L-2rupOb11{38&ga-s>bNBPgnBH<2PdTiU|DP%f(*3Zn^4M>Mi_&zd6k5W2%@Q1(71Ay-lI0*T*$T&tqI(#2&6 zo3YLPV`m^x4M-HthQ|E+sXo##B_PyBFyN0Z8-Jj<_}+6E&UD2}p5BI8)AtixCRI3# z+E%M5(z{o#zH-%fv>T(O(JuX2p$h`9i$bEFbh|;iu376i23~_E_GrK4kCZf@`?tCc z@v%Jpv_*HS;g05Bp&M#V#0Q6$#(#`dx2Dk`ORahZx? znwfhl^ifQZ5e+}Y_wc#uW84g%LE+v!L;ZxLqs*!a^!fo=AhM>Qd zD6G=bAPljzNM#(}$kYXN=h=UZBm?yVNrqT5U1QWa{6wTd_z(w%4axz!biEUcUMO!6 ksU}9c*7&)S4Lh?-cGhxH7?96z1s$432WpC2`4-jD4O#lD@ diff --git a/save-reader/ref/parsers/__pycache__/manifest.cpython-310.pyc b/save-reader/ref/parsers/__pycache__/manifest.cpython-310.pyc deleted file mode 100644 index 643da22874a9ab614aa410011084a44f3465b758..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 5082 zcmZ`-TW=f372erhUUjv6H{x_^D-jusq#Yzck+4DwUlO&cl8R2;Fjj~aXGE>N+@)q# zl0{HJ3K{5AS`>Xuf=WeSjQ)W>_IK=SpYj*bIzPOm+zcWG&NPU z@SBO`B2Q37n>5-fq)1?@Tj;w;N=; zxn{1LZ|0fRu|(z{tXYtm!=lK_%(ICjyE$pTa!0H=C37EGA}KCa!8fMy**y3@OV@5aJ#!Ml@#0;d^D}O7YZ}GaC@QBj=EhL z##nN{@tLj`i#<yCL@?$M1>sdQqrrt=+U!jbFruZKEgKzSOn) zs@IlFH7(n*Zx-rJzc>1#)$bWT#H;Ryam5!)yuRTFGW5D~iH6f(6iVKciU)ouPbg|N z{@~$94DW)n*dLy_q+>q_xRP3`9VyC&!Lon?lr#zeXYK14=jWk}cO&(j z*SPj}WCv0}XS&MyqoGhKRo9E#n^G7~ZBNUJA8HwDKlXQIu*XZJbUJDe3LWWC^G&ei z3lAPX@l>_(;qoVqhxZ>Zf4s7=X!tDLU;Xv+{c7V>LknES?JD;=9@xh`h<2r_&~}_u zq{+HZ{7^)@nm=lkPmI>qJE!&vaIh~-qhqzUW_DU7id{e6zAWc6^VgfdJku(pYQL$WtP1H)0X^>o`uZS1o=|a8hQ;gcpWU>=}1oDceawc(KK|#tE#ccmtt@c#E zHH;JVDtJmy#hr5h@|UCTQ+=oOv~j15@A)>JWBdn1Nulu%=t!%{0FU;kmUJY5@M{Jb zv}IN}_0y4@a50h@8_5F>vm#fw#H=Wa2>@3CAeTa4QOp3k=2Qmkp}!d+H#|5;Yr|{D zk=pBDHqIMw_X7C^AYu{bgdZn4M5_YC^b2MlGZ+&xkZ_;lbc_}z`JRf_1KHI{HnsBc zwD8;UDXTUcJ1zBXJohOVP|Ap{f%S|Xu!i-DkvUx=R#TXH#S&M`VB?jguHy5!QrnEW zvgQTyMNLPst|(dCk;>oLs~ICHT^k_*kU7b$@3nkU0;Q%6i>yCA@`|dN8p?@7EW(N7 zwf`^YEa;Jo?3W~~^}r%&%TE8sDP9Nz7=)<=;l4P-iwsQpuR&+NfsPM8YlWxwKHIkk z_MvrT4cOo8ExVDP6h_Jz)@3KLt9KLTCoD8NNSkY5kreBsD1>ouse*5F#v4|De*BCi zPqiLIZ7ryVnsxD=s@gKdnsXRJcbN00HuOYqCsIm=n&x`Nk6NAoGp=otx5tOK!x)^gv_V`}zICAV08o zEY&$fn;qoAAvdrG`Rf*X`R`B*F>Bz^NFAQ_Z2B+<%EAWZFfu__oN9JIKjinPAwTd3 zeepZTtlJ%smrtyGPa%73*J}NY$tL}o_xN_GWIO7Fz6PX!PvJ$Q0ubW20q#@<^=El0 z&2~sBhVYn``wq?AZ*!iW-b#VgihZG5uaL z(HhdWRv#u=Ej`uVG+vWrJ><(!Bo1Oa$#hh--2(&4X`1p&QfNI~dA#~L(Pd1Pk@^`S zPPwSw#4BN4LLM0=S=uv<5=VKvY4jcx!xo7FCyRbR^eU3+9tn92LwpvfJ?cTk&-+dbp<3`X|;!;LSvI2G$l5n`^ zL0N3${P`xZ|3`QniB&N`eEsCiFgY)Vl1^^nwz>ort9#H=@ zz)fuDF!p;q2HGZ>qHkCt+EutO=a6r9m(?XSY=u?0#}HO`!f7X8#D%wSw6I8wu;V{TWog zWfODg(^`4OR2A!9*@~7YtbOjlNul16p?uL(OJ87tCXlqh0;a#-@^x!iAdbQLXfaHf z4NO!h^0~zEf&l7$M1535C^*k=G%?2TQ`(A5n(Oa1NNd2yC>0piI3k9N|Lm z4(UB)gA{!*3$)IBgOy9Q5;3K3wa zu#HvDiJ&O9zoflWGh71{*z9!}!pt0(($efi`Fy}vP~k5ERnOc!PK^RyXN}UsfMEM< z%qX^ShDLdViV%H)3?f{FiHk5nS~|=KBp!7q&JJ*u7TM>nx+8KT4^!D>s(k=Un2V8k z4w$}&J_84a3+t&^wz>lvf8QQX2t7tFdUVrxZ zLVan03S4t@eBUUN?{3PFB*A4Bi7AY@-fnM_4As>A6#UCFNTA5ORSC_H{E(_^aL#<#W`@M%Bh1#6I8qIHF&EH z__)HR(K89}he=z}svl{JA{K`P0;4lZ3>zrC2JD~g&p^vjf*^s?1Yw&5k=W07&&yH3 zniqJ|xMhmIM`5=>^cScxSD#_YN~2Kt>A0Jol9Ea~`T?!TrlD(i7~{}rI=@Qu!_YSK zh8JOW`AU*&wM5iTtKW@n+E0c+nYpc|XvQ}CYgBTa%l**5t*!ork?fb?@r#8O{tpmZDZ`OI|7EkpObIesJdSG{`Q^%#whS1kOlT=~Ppv7cDhf6~MGUjYxxmL=kU2cWD?O9^EMqGfN|f?y|b zHXYpEz-<*a3$5a2QCNGHD*RDwj(ljTq8j}-~`b5c?#$P!Vn-dI80=l9W)br{p-cG95)l2FpfSyw~)ywJ?KvS6cZ5Tx`j(>0CXPNWwqTDQV$%+t&uNuDHf*AH|oXX2&~ zJ>y65oJ=}NWPC4}lTD+Qp8yqi<3zVYuchhXpo_WVxk{53eTl^oP^4hhuYsC3Y3gOEn##mFYHRvpz0HovhL})6C}`z*VcTp>%8vDP7` zXlhK*CdP?f$r5-MEDuO3tq77!_PtBQo(%jrku#lG%RQYm=xL@V5BD)^H|oek4`S*Q zdBLIAjltIKb6Oy+#gY)sCO$2Fz&SR7>W-)OH1N0iUX$}nXY&Tpp*HhP%ubtu|HXKx+YQ3JzhOF>+31@b z1mBfWo7KRp$50}~3bn$;JlQt72{(nQl3noPGu5PHj80Gbs+1f)f&fw z!Q^BSyaILzJzrdaC&MD7at=}0avp(b9%gdKYaYab7w;p8^j+nG|Kx%Updtg=&wSyB zDmsk8YO){K-lgTBT0iEM!bs}42~$M82@~%LFTf5obixaBlkw02+>l1%oF+zlcR4#NB_ z%4+?s>YS|3*WaPr+xo^k)k@`lq5^*h!V9|f%D0r$Y9br;La^phc#hXv-K_Kx zi2>2IR>%r$eOZNX4T{29u^DO8)LCiZDcKcD6Zd)XRbwL4q}k8^1Rj>vP~JNwVIQa8V%BNqmfk_jaH;O0l{O9#-olG6_@T|VDAGP#?7w9Jo zag>mMEFgIMyWNGp$1I#rZoJMszK&fCCCe3zIr=T@bmrEl?;XZVTk#E&ZYt@yudStp zg$KXf3b$v^-jJCuZ9P3hG@%{6wADMqx_~r#35&!} zd(~~`K+4Qb^phlW5Z|*RoTOvIXRizt^C0Q81N{K+5F%*s!(@BANn#%SMgWv>K^G$a zF983H@}!4irDrErY8{K7liI0sV3|jWII&YJwNLGyo4U&BSY|qLl$(mLQMk~X)K`P>q?pPwx(63y@(?;#$il7`EgiKFFT z_!vcCi}L1tBTVKP4T)*GO@5{aiZ7iM3b%mgHz?SbJ7r-#nUHM9v~t zL2NzHAqA{&uYw@6593T!vXQ>AWaXiPk6u_r)PyIs{Lnkg)LAh%jc>ohuU5*OK!;hm zv9bR1)wRaG)vSP8s=3d$nUxW-l06f3+F2pLIfxQjp-C~aHe#+22QG1#Iqe9Hgqdh% zF6}XM!04H`mZ-mT~Wf{7%nM_441@&umRCEiR&uHF?;gAO%0zM zl^@)g-@dh_ZoGC>V{iuWR((s&rZnD2an)uX=+*53VeQsyW(J=c2g3z3wSgey9Wv9u z2as4NBDGH8oe7Nc7%?Jo`k1N2%`vIPW+zgi1dQgovhTx4lmob`T)-A0OW|mlGW20& z6lFDdh+OE$$pzFfQIMy1AtNlzJZbBF=R$sB3sLqGbQvS5ntu-#>*he#)gG#{n~m ziL*k|#knPO$-cAluv*KCFlREP(xGQUDl?vqa>biUwx%M!9sCF;ZtoDcw*e$1ZgGsi z)M`Kp7%6pF4Hx!fHIM`_A}A}XBVcn#7>yCKu01F(hidj3htpb#rOHtRz6Gz#gm0Fy zhM|QFmb45B2%hHBX(Qj+GNCULT+##;4}Rz;jm9x(+1Mr!%JRSK;%KrjsrrB{b6WA9 zd5$LjTMQR$&<0{C0P^$HdT5!ETnNZ~)6w=H; z&~V8$3Jp@lzu-+w5~sL3DUQa6jMCc+k&^P!kS&P^Vhc=xonwJCK0y%SUWDvP{yAfg z`4F_`It_O*)I`qH_td+n;b5$EJJ~l>P?|R}T;OFC2nzAc%j-~_ zkJ_jVj!rC8Nlxm(RV?%4RG4ePp(f)>%lXD3o8pM(Y&kvDS5#Rk7isNH(AlS43&l72 z8`MCk&Wiarj}RTPPnWB&Td1w_x21ks4_ z`l7|FHZM`;Z>Yjc2mBBP;6hqBh2~FCm#MKIA$~fP}aA?b0E;tlH`0|;kp%$VF&QKmb`tc%VZ5%_< zWsx`z4cix)OV!@dON-oyIrNkLs6*$md^R#A^AaY=iuoCxV-;I00x3#}TG@Pz*XHLm ztkJ-3&Kc}e`VQe%b@MnzYOjsvF7_Fba+dsrplKp?ki^Za427`cLdeCM8v>n)#w8;( zUx4syTpWj8F2^XzOgV19VgPL0z~p5bia0z`xkh&?=&s|N3H+7Bl(7Hf%!Y~KwOZvY zbL8ApS(v*AL$F`VFhoJwtYSc*<7hTXh++H$mVM|9OAxRu0(KInB~O7gsl{r@NerC_ z%||q_a9PYO;ZF%i+dn4~4p~&lnp+rtg)63~l2s9vQmIgKDuoG$-)-E*+RJ!%Ic+fR zY?r*j;ogQZ&c&AnToivmBI0WR=+uP53g?Z~Jubj&kpU>SdA|{s#fkVrf0RydRCGo% z)aXhgd2_=m?;>)RMzaGy^jw=!h=Xgwrw=Yy`)B(F%Hn7NT<9jmuyL3;44n>}IM56_ zGCUPMQ3D6371Xd)x}l#-PBiT0Ao=rmc|kxkaNF1OL662tC*1J@D)-QTqi!#^O9qpm zg+nb~s-UP(g+3Th#ghyNdeiSbsfaoltLu03a~3+SaKnCIj}NZc$gp4d15=S7`+FgJ zb_z|Hwuc7PpniG(!8WzI`rS5WXTY7#+cbH_VY5B3CDjW5-*Uo`^8bR1WB;`ZKM$<~ zc%-=ebPcN=99vSHuaE729ivK>@=eeA0%_RAAH}q=bQ9J7p%94S!uka+9B-TxN1+(s z6+SHMPvR@GCuc3Q%?_F@+y#OeZ(ym+rM^6~krnffP|eL;YENbM!C@wLGsk$}J7$pm z{uLM3r(*~i*T1`F8^U5j70xA{o8^YRrxWS4)1O1Y>j-rdhho7{Z;ca@j+)@Ge<+628!(YsfDWj z+^on>Se`!=rLxj{c^hj90m6#uG~-%hua#_^sO?F|Eh2E2-T$dfRIVd{Ik=Bbl-w0p JOnf&n{(nEDe%t^6 diff --git a/save-reader/ref/parsers/effect_txt.py b/save-reader/ref/parsers/effect_txt.py deleted file mode 100644 index 1ccffab..0000000 --- a/save-reader/ref/parsers/effect_txt.py +++ /dev/null @@ -1,98 +0,0 @@ -"""effect_txt.py -- reader for Effects/*.effect (particle-effect definitions). - -NOT the brace-block format. Layout: - - TXT # magic first line - KEY value # scalar (number, TRUE/FALSE, "quoted") - KEY # group: KEY on its own line, then - BEGIN - ...nested KEY value / groups... - END - -Order matters: 'PARTICLEDATATYPE n' is followed by the CREATION / -VARIATION / OVERLIFE curves that belong to that datatype, and 'MODIFIER' -repeats once per type. So each level is returned as an ordered list of -[key, value] pairs (value = scalar or nested list). to_dict() gives a -dict view (repeats -> lists) when order is not needed. - -Quirks handled: one file has CRLF; 'NAME "New Emitter"' values contain -spaces; indentation is cosmetic (tabs); the format is line-based. - -Stdlib only. -""" -from __future__ import annotations - -from typing import Any - -from flat_kv import split_tokens, strip_comment -from mars_data import coerce - -__all__ = ["parse", "parse_file", "to_dict", "EffectSyntaxError"] - -Pairs = list # list[[key, value]] - - -class EffectSyntaxError(ValueError): - pass - - -def parse(text: str, *, typed: bool = True) -> Pairs: - lines = text.splitlines() - if not lines or lines[0].strip() != "TXT": - raise EffectSyntaxError("missing TXT magic") - stack: list[Pairs] = [[]] - pending_key: str | None = None - for lineno, raw in enumerate(lines[1:], 2): - line = strip_comment(raw).strip() - if not line: - continue - if line == "BEGIN": - if pending_key is None: - raise EffectSyntaxError(f"line {lineno}: BEGIN without a key") - grp: Pairs = [] - stack[-1].append([pending_key, grp]) - stack.append(grp) - pending_key = None - continue - if line == "END": - if len(stack) == 1: - raise EffectSyntaxError(f"line {lineno}: END without BEGIN") - stack.pop() - continue - if pending_key is not None: - raise EffectSyntaxError(f"line {lineno}: key {pending_key!r} not followed by BEGIN") - toks = split_tokens(line) - key = toks[0][0] - if len(toks) == 1: - pending_key = key - continue - vals = [coerce(t) if (typed and not q) else t for t, q in toks[1:]] - stack[-1].append([key, vals[0] if len(vals) == 1 else vals]) - if len(stack) != 1: - raise EffectSyntaxError(f"{len(stack) - 1} unclosed BEGIN group(s)") - if pending_key is not None: - raise EffectSyntaxError(f"trailing key {pending_key!r} without BEGIN") - return stack[0] - - -def to_dict(pairs: Pairs) -> dict: - d: dict = {} - for key, val in pairs: - if isinstance(val, list) and val and isinstance(val[0], list) and len(val[0]) == 2 and isinstance(val[0][0], str): - val = to_dict(val) - if key in d: - if not isinstance(d[key], list) or not getattr(d[key], "_rep", False): - d[key] = _Rep([d[key]]) - d[key].append(val) - else: - d[key] = val - return d - - -class _Rep(list): - _rep = True - - -def parse_file(path, **kw) -> Pairs: - with open(path, "rb") as f: - return parse(f.read().decode("cp1252"), **kw) diff --git a/save-reader/ref/parsers/flat_kv.py b/save-reader/ref/parsers/flat_kv.py deleted file mode 100644 index 1ce67cd..0000000 --- a/save-reader/ref/parsers/flat_kv.py +++ /dev/null @@ -1,127 +0,0 @@ -"""flat_kv.py -- readers for the flat 'KEY value' tuning tables and the -whitespace-positional tables under Data/, Weapons/, Badges/, Avatars/, GUI/. - -Two shapes exist: - - parse_kv(text) KEY value -> {KEY: value} - one constant per line; value is a bareword or a - "quoted string"; '//' comments; colors are quoted - "r g b" (use color()). Files: Data/globals.txt, - Data/species.txt, Data/Strategy/StrategyVars.txt, - Data/Combat/*.txt (most), Data/encounters.txt, ... - - parse_rows(text) tok tok tok ... -> [[tok, ...], ...] - one record per line, whitespace separated, quoted - tokens may contain spaces; '//' comments. Files: - Weapons/_turrets.txt, Weapons/_defaultweapons.txt, - Data/Combat/damfx*.txt, Data/Strategy/playercolors.txt, - Badges/BadgeTable.txt, Avatars/AvatarTable.txt, - GUI/WeaponIconPlacements.txt - -Quirks handled: '//' inside a quoted value is not a comment; a quoted -value may be empty (""); keys repeat in a few files (kept as list); -duplicate-key detection is exposed via parse_kv(..., on_dup=). - -Stdlib only. -""" -from __future__ import annotations - -import re -from typing import Any - -from mars_data import coerce - -__all__ = ["parse_kv", "parse_rows", "duplicates", "color", "strip_comment", - "split_tokens", "parse_kv_file", "parse_rows_file"] - -_TOK_RE = re.compile(r'"([^"]*)"|(\S+)') - - -def strip_comment(line: str) -> str: - """Remove a trailing // comment, ignoring // inside double quotes.""" - in_q = False - i = 0 - n = len(line) - while i < n: - c = line[i] - if c == '"': - in_q = not in_q - elif c == "/" and not in_q and line.startswith("//", i): - return line[:i] - i += 1 - return line - - -def split_tokens(line: str) -> list[tuple[str, bool]]: - """Split a line into (token, was_quoted) pairs.""" - out = [] - for m in _TOK_RE.finditer(line): - if m.group(1) is not None: - out.append((m.group(1), True)) - else: - out.append((m.group(2), False)) - return out - - -def parse_rows(text: str, *, typed: bool = True) -> list[list[Any]]: - rows = [] - for raw in text.splitlines(): - line = strip_comment(raw).strip() - if not line: - continue - toks = split_tokens(line) - rows.append([coerce(t) if (typed and not q) else t for t, q in toks]) - return rows - - -def _pairs(text: str, typed: bool): - for lineno, raw in enumerate(text.splitlines(), 1): - line = strip_comment(raw).strip() - if not line: - continue - toks = split_tokens(line) - key = toks[0][0] - vals = [coerce(t) if (typed and not q) else t for t, q in toks[1:]] - val: Any = None if not vals else (vals[0] if len(vals) == 1 else vals) - yield lineno, key, val - - -def parse_kv(text: str, *, typed: bool = True, on_dup: str = "last") -> dict: - """KEY value per line -> dict. A value made of several unquoted tokens - is kept as a list. on_dup: 'last' (later line wins), 'first', 'error'. - Use duplicates() to find repeated keys.""" - d: dict = {} - for lineno, key, val in _pairs(text, typed): - if key in d: - if on_dup == "error": - raise ValueError(f"line {lineno}: duplicate key {key}") - if on_dup == "first": - continue - d[key] = val - return d - - -def duplicates(text: str) -> dict[str, list[int]]: - """key -> line numbers, for keys that appear more than once.""" - seen: dict[str, list[int]] = {} - for lineno, key, _ in _pairs(text, False): - seen.setdefault(key, []).append(lineno) - return {k: v for k, v in seen.items() if len(v) > 1} - - -def color(value: str) -> tuple: - """'r g b' or 'r g b a' -> tuple of numbers.""" - return tuple(coerce(t) for t in value.split()) - - -def _read(path) -> str: - with open(path, "rb") as f: - return f.read().decode("cp1252") - - -def parse_kv_file(path, **kw) -> dict: - return parse_kv(_read(path), **kw) - - -def parse_rows_file(path, **kw) -> list: - return parse_rows(_read(path), **kw) diff --git a/save-reader/ref/parsers/manifest.py b/save-reader/ref/parsers/manifest.py deleted file mode 100644 index ab4f2eb..0000000 --- a/save-reader/ref/parsers/manifest.py +++ /dev/null @@ -1,121 +0,0 @@ -"""manifest.py -- the numbered id manifests and the '#'-commented CSVs. - -parse_manifest(text) -> Manifest - Weapons/_weapons.txt and Species//sections/_shipsections.txt: - one per line - // DELETED - retired id (still reserved) - Ids are the persistent network / savegame ids. Filenames are matched - case-insensitively (the shipped manifests have 'DEWar.SHIPSECTION', - 'CRAIC.Shipsection' etc. against lower-case files -- Windows FS). - -parse_csv(text) -> list[list[str]] - Rows with '#' or '//' as first non-blank char are comments; blank rows - dropped; RFC-4180 quoting honoured (Strings.csv has one multi-line cell - and quoted commas). Header rows that start with '#' (aitechpri.csv, - "# species" in stock_diplomacy_messages.csv) are returned separately - via parse_csv_with_header(). - -Stdlib only. -""" -from __future__ import annotations - -import csv -import io -import re -from dataclasses import dataclass, field - -__all__ = ["Manifest", "parse_manifest", "parse_manifest_file", - "parse_csv", "parse_csv_file", "parse_csv_with_header", "read_text"] - -_DELETED_RE = re.compile(r"//\s*DELETED\s*-\s*(\d+)", re.I) -_ENTRY_RE = re.compile(r"^\s*(\d+)\s+(\S+)\s*$") - - -@dataclass -class Manifest: - entries: list[tuple[int, str]] = field(default_factory=list) # (id, filename) - deleted: list[int] = field(default_factory=list) - problems: list[str] = field(default_factory=list) - - def by_id(self) -> dict[int, str]: - return dict(self.entries) - - def by_name(self) -> dict[str, int]: - """lower-cased filename -> id""" - return {n.lower(): i for i, n in self.entries} - - -def parse_manifest(text: str) -> Manifest: - m = Manifest() - seen: dict[int, int] = {} - for lineno, raw in enumerate(text.splitlines(), 1): - line = raw.strip() - if not line: - continue - d = _DELETED_RE.search(line) - if d: - m.deleted.append(int(d.group(1))) - continue - if line.startswith("//"): - continue - e = _ENTRY_RE.match(line) - if not e: - m.problems.append(f"line {lineno}: unrecognised {line!r}") - continue - i, name = int(e.group(1)), e.group(2) - if i in seen: - m.problems.append(f"line {lineno}: duplicate id {i} (first at line {seen[i]})") - seen[i] = lineno - m.entries.append((i, name)) - for i in m.deleted: - if i in seen: - m.problems.append(f"id {i} is both DELETED and assigned") - return m - - -def read_text(path) -> str: - with open(path, "rb") as f: - return f.read().decode("cp1252") - - -def parse_manifest_file(path) -> Manifest: - return parse_manifest(read_text(path)) - - -def _is_comment(row: list[str]) -> bool: - if not row: - return True - first = row[0].lstrip() - if first.startswith("#") or first.startswith("//"): - return True - return all(c.strip() == "" for c in row) - - -def parse_csv(text: str, *, strip: bool = True) -> list[list[str]]: - rows = [] - for row in csv.reader(io.StringIO(text, newline="")): - if _is_comment(row): - continue - rows.append([c.strip() for c in row] if strip else row) - return rows - - -def parse_csv_with_header(text: str) -> tuple[list[str] | None, list[list[str]]]: - """Return (header, rows). Header = the first '#'-prefixed row that - contains a comma (e.g. '# ,,...'), with the '#' and - any '<>' stripped; None when there is no such row.""" - header = None - for row in csv.reader(io.StringIO(text, newline="")): - if not row: - continue - first = row[0].lstrip() - if first.startswith("#") and len(row) > 1: - header = [c.strip().lstrip("#").strip().strip("<>") for c in row] - break - if not _is_comment(row): - break - return header, parse_csv(text) - - -def parse_csv_file(path, **kw): - return parse_csv(read_text(path), **kw) diff --git a/save-reader/ref/parsers/mars_data.py b/save-reader/ref/parsers/mars_data.py deleted file mode 100644 index fe5dcd9..0000000 --- a/save-reader/ref/parsers/mars_data.py +++ /dev/null @@ -1,211 +0,0 @@ -"""mars_data.py -- reader for the Mars engine's brace-block key/value format. - -Covers: *.weapon, *.shipsection, *.tech, *.combat, *.def, *.script and the -block-form *.txt files (scenarios, tutorial, credits, systemnames, skydefs, -ctechvars, shipai). - -Grammar (as observed in every shipped SOTS1 file): - - body := (block | pair | item)* - block := NAME '{' body '}' - pair := NAME value - item := QUOTED # bare quoted string inside a block - value := QUOTED | BAREWORD - comment := '//' .* EOL - -A file's top level is itself a body (scenario .txt files mix top-level pairs -and player{} blocks; catalog files hold one or many named blocks). - -Result shape: plain dicts. A key seen once maps to its value; a key seen -more than once maps to a list (use get_list() when you want a list always). -Bare quoted items are collected under the key "_items". - -Quirks handled (all seen in the real data, see parsers-report.md): - * keys are case-insensitive to the engine ("Requires"/"requires", - "badge"/"Badge") -> keys are lower-cased unless keep_case=True - * a block may open on the same line as a preceding pair - ("turretsize small mount {") and a block name may sit on the same line - as its brace ("weapon {") - * backslashes inside quoted strings are literal (Windows paths); there is - no escape syntax - * '//' inside a quoted string is not a comment - * CRLF and LF line endings, cp1252 bytes (decoded losslessly) - * numbers use C float syntax: ".5", "-.8", "7e+8" - -Stdlib only. -""" -from __future__ import annotations - -import re -from typing import Any, Iterator - -__all__ = ["parse", "parse_file", "coerce", "get_list", "MarsSyntaxError"] - - -class MarsSyntaxError(ValueError): - pass - - -# --- tokenizer ------------------------------------------------------------- - -_TOKEN_RE = re.compile( - r""" - (?P\s+) - | (?P//[^\n]*) - | (?P\{) - | (?P\}) - | (?P"[^"]*") - | (?P") - | (?P[^\s{}"]+) - """, - re.VERBOSE, -) - - -def _tokenize(text: str) -> Iterator[tuple[str, str, int]]: - """Yield (kind, value, line). kind in {open, close, quoted, bare}.""" - line = 1 - pos = 0 - n = len(text) - while pos < n: - m = _TOKEN_RE.match(text, pos) - if m is None: # pragma: no cover - regex is exhaustive - raise MarsSyntaxError(f"line {line}: cannot tokenize {text[pos:pos+20]!r}") - kind = m.lastgroup - tok = m.group() - pos = m.end() - if kind == "ws": - line += tok.count("\n") - continue - if kind == "comment": - continue - if kind == "bad_quote": - raise MarsSyntaxError(f"line {line}: unterminated string") - if kind == "quoted": - yield kind, tok[1:-1], line - line += tok.count("\n") - else: - yield kind, tok, line - - -# --- parser ---------------------------------------------------------------- - -_INT_RE = re.compile(r"[+-]?\d+$") -_FLOAT_RE = re.compile(r"[+-]?(\d+\.\d*|\.\d+|\d+)([eE][+-]?\d+)?$") - - -def coerce(tok: str) -> Any: - """Bareword -> int / float / bool when it looks like one, else str.""" - if _INT_RE.match(tok): - return int(tok) - if _FLOAT_RE.match(tok): - return float(tok) - low = tok.lower() - if low == "true": - return True - if low == "false": - return False - return tok - - -def _add(d: dict, key: str, value: Any) -> None: - if key in d: - cur = d[key] - if isinstance(cur, list): - cur.append(value) - else: - d[key] = [cur, value] - else: - d[key] = value - - -def get_list(d: dict, key: str) -> list: - """Always return a list for a key (missing -> [], single -> [x]).""" - v = d.get(key) - if v is None: - return [] - return v if isinstance(v, list) else [v] - - -class _Parser: - def __init__(self, text: str, typed: bool, keep_case: bool, strict: bool, warnings: list | None): - self.toks = list(_tokenize(text)) - self.i = 0 - self.typed = typed - self.keep_case = keep_case - self.strict = strict - self.warnings = warnings if warnings is not None else [] - - def _warn(self, msg: str) -> None: - if self.strict: - raise MarsSyntaxError(msg) - self.warnings.append(msg) - - def _peek(self): - return self.toks[self.i] if self.i < len(self.toks) else None - - def _next(self): - t = self.toks[self.i] - self.i += 1 - return t - - def _key(self, name: str) -> str: - return name if self.keep_case else name.lower() - - def body(self, depth: int) -> dict: - d: dict = {} - while True: - t = self._peek() - if t is None: - if depth: - # 11 shipped shipsections never close their outer block; - # the engine treats EOF as closing every open block. - self._warn(f"end of file inside block (depth {depth})") - return d - kind, val, line = t - if kind == "close": - self._next() - if not depth: - # CrPropaganda.shipsection has one '}' too many. - self._warn(f"line {line}: stray '}}' at top level") - continue - return d - if kind == "open": - raise MarsSyntaxError(f"line {line}: '{{' without a block name") - self._next() - if kind == "quoted": - # bare string item (systemnames.txt lists) -- never a key - _add(d, "_items", val) - continue - nxt = self._peek() - if nxt is None or nxt[0] == "close": - # lone bareword at end of block: treat as flag item - _add(d, "_items", val) - continue - if nxt[0] == "open": - self._next() - _add(d, self._key(val), self.body(depth + 1)) - continue - nkind, nval, _ = self._next() - if nkind == "bare" and self.typed: - nval = coerce(nval) - _add(d, self._key(val), nval) - - -def parse(text: str, *, typed: bool = True, keep_case: bool = False, - strict: bool = False, warnings: list | None = None) -> dict: - """Parse brace-block text into nested dicts. - - typed -- convert bareword numbers/bools (quoted strings stay str) - keep_case -- keep key case instead of lower-casing - strict -- raise on unbalanced braces instead of recovering the way - the engine does (EOF closes open blocks, stray top-level - '}' ignored); pass warnings=[] to collect the recoveries - """ - return _Parser(text, typed, keep_case, strict, warnings).body(0) - - -def parse_file(path, **kw) -> dict: - with open(path, "rb") as f: - raw = f.read() - return parse(raw.decode("cp1252"), **kw) diff --git a/save-reader/ref/parsers/verify.py b/save-reader/ref/parsers/verify.py deleted file mode 100644 index cb4b6d3..0000000 --- a/save-reader/ref/parsers/verify.py +++ /dev/null @@ -1,634 +0,0 @@ -"""verify.py -- parse every shipped SOTS1 data file, cross-link the catalogs, -and emit normalized JSON artifacts. - -usage: python3 verify.py - -Prints a markdown report to stdout; writes to : - tech_tree.json weapons.json shipsections.json strings.json - schema_stats.json crosslink.json tech_tree.dot -""" -from __future__ import annotations - -import collections -import json -import os -import re -import sys - -sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -import effect_txt -import flat_kv -import manifest -import mars_data -from mars_data import get_list - -RACES = ["Human", "Zuul", "Hiver", "Tarkas", "Liir", "Morrigi"] -PLAYABLE = RACES -ALL_RACE_DIRS = RACES + ["_NPC"] - -BRACE_TXT = {"Data/tutorial.txt", "Data/credits.txt", "Data/Strategy/systemnames.txt", - "Data/Combat/ctechvars.txt", "Data/Combat/shipai.txt", - "Models/Skysphere/skydefs.txt", "Models/Skysphere/NodeSpace-skydefs.txt"} -ROWS_TXT = {"Weapons/_turrets.txt", "Weapons/_defaultweapons.txt", "Data/Combat/damfx.txt", - "Data/Combat/damfx_levels.txt", "Data/Strategy/playercolors.txt", - "Badges/BadgeTable.txt", "Avatars/AvatarTable.txt", "GUI/WeaponIconPlacements.txt"} -PROSE = {"Locale/EN/ChatTrans.txt"} - - -def kind_of(rel: str) -> str: - ext = rel.rsplit(".", 1)[-1].lower() - base = os.path.basename(rel) - if ext in ("weapon", "shipsection", "tech", "combat", "def", "script"): - return "brace:" + ext - if ext == "effect": - return "effect" - if ext == "csv": - return "csv" - if ext in ("fx", "fxh"): - return "hlsl" - if ext == "txt": - if base in ("_weapons.txt", "_shipsections.txt"): - return "manifest" - if rel.startswith("Scenarios/") or rel in BRACE_TXT: - return "brace:txt" - if rel in ROWS_TXT: - return "rows" - if rel.startswith("Locale/EN/Desc") or rel in PROSE: - return "prose" - return "kv" - return "other" - - -def walk(root): - for dp, _, fn in os.walk(root): - for f in sorted(fn): - p = os.path.join(dp, f) - yield p, os.path.relpath(p, root).replace(os.sep, "/") - - -# --- schema stats ------------------------------------------------------------ - -def schema_walk(node, path, stats): - """Count key occurrences per block path, and which keys are blocks.""" - st = stats.setdefault(path, {"blocks": 0, "keys": collections.Counter(), "sub": collections.Counter()}) - st["blocks"] += 1 - for k, v in node.items(): - vals = v if isinstance(v, list) else [v] - for x in vals: - if isinstance(x, dict): - st["sub"][k] += 1 - schema_walk(x, path + "." + k, stats) - else: - st["keys"][k] += 1 - - -# --- tech tree --------------------------------------------------------------- - -_RP_RE = re.compile(r"^RP:(\d+)$", re.I) -_PCT_RE = re.compile(r"^(\w+):(\d+)$") - - -def parse_allows(s: str): - toks = s.split() - child = toks[0] - rp = None - pct = {} - extra = [] - for t in toks[1:]: - m = _RP_RE.match(t) - if m: - rp = int(m.group(1)) - continue - m = _PCT_RE.match(t) - if m and m.group(1) in RACES: - pct[m.group(1)] = int(m.group(2)) - continue - extra.append(t) - return child, rp, pct, extra - - -def main(root: str, out: str) -> int: - os.makedirs(out, exist_ok=True) - rep = [] - P = rep.append - - # ---- 1. parse everything --------------------------------------------- - ok = collections.Counter() - fail = collections.Counter() - fails = [] - warns = [] - parsed = {} # rel -> object - for p, rel in walk(root): - k = kind_of(rel) - try: - if k.startswith("brace"): - w = [] - obj = mars_data.parse_file(p, warnings=w) - # strict re-parse to record the recovery - if w: - warns.append((rel, w)) - elif k == "effect": - obj = effect_txt.parse_file(p) - elif k == "csv": - obj = manifest.parse_csv_file(p) - elif k == "manifest": - obj = manifest.parse_manifest_file(p) - if obj.problems: - raise ValueError("; ".join(obj.problems)) - elif k == "rows": - obj = flat_kv.parse_rows_file(p) - elif k == "kv": - txt = manifest.read_text(p) - obj = flat_kv.parse_kv(txt) - d = flat_kv.duplicates(txt) - if d: - warns.append((rel, [f"duplicate keys {d}"])) - else: - continue - parsed[rel] = obj - ok[k] += 1 - except Exception as e: # noqa: BLE001 - fail[k] += 1 - fails.append((rel, repr(e))) - - P("## Parse results") - P("") - P("| reader | file kind | files | parsed | failed |") - P("|---|---|---|---|---|") - reader_of = {"brace": "mars_data", "effect": "effect_txt", "csv": "manifest.parse_csv", - "manifest": "manifest.parse_manifest", "rows": "flat_kv.parse_rows", "kv": "flat_kv.parse_kv"} - for k in sorted(set(ok) | set(fail)): - P(f"| {reader_of[k.split(':')[0]]} | {k} | {ok[k] + fail[k]} | {ok[k]} | {fail[k]} |") - P("") - P(f"Total: {sum(ok.values())} parsed, {sum(fail.values())} failed " - f"(skipped: {sum(1 for _, r in walk(root) if kind_of(r) in ('hlsl', 'prose', 'other'))} " - f"HLSL/prose files that are not data).") - if fails: - P("") - P("Failures:") - for rel, e in fails: - P(f"- `{rel}`: {e}") - if warns: - P("") - P("Lenient recoveries (engine-compatible; strict=True would reject these):") - for rel, w in warns: - P(f"- `{rel}`: {'; '.join(w)}") - P("") - - # ---- 2. schema stats --------------------------------------------------- - stats = {} - for rel, obj in parsed.items(): - k = kind_of(rel) - if k in ("brace:weapon", "brace:shipsection", "brace:tech", "brace:combat", "brace:def", "brace:script"): - schema_walk(obj, k.split(":")[1], stats) - schema_json = {path: {"blocks": st["blocks"], - "keys": dict(st["keys"].most_common()), - "subblocks": dict(st["sub"].most_common())} - for path, st in sorted(stats.items())} - json.dump(schema_json, open(os.path.join(out, "schema_stats.json"), "w"), indent=1) - - P("## Schema stats (key frequency per block type)") - P("") - P("Full table in `schema_stats.json`. Block paths with instance counts and the") - P("keys seen in them (count = number of block instances carrying the key):") - P("") - for path in ["weapon.weapon", "shipsection.shipsection", "tech.tech"]: - st = stats[path] - P(f"### `{path}` ({st['blocks']} instances)") - P("") - P("keys: " + ", ".join(f"{k}:{n}" for k, n in st["keys"].most_common())) - P("") - P("sub-blocks: " + ", ".join(f"{k}:{n}" for k, n in st["sub"].most_common())) - P("") - P("All block paths: " + ", ".join(f"`{p}`({st['blocks']})" for p, st in sorted(stats.items()))) - P("") - - # ---- 3. build catalogs ------------------------------------------------- - techs = parsed["TechTree/MasterTechList.tech"]["tech"] - tech_by = {t["name"].lower(): t for t in techs} - groups = collections.defaultdict(list) - for t in techs: - if "group" in t: - groups[str(t["group"]).upper()].append(t["name"]) - - strings_rows = parsed["Locale/EN/Strings.csv"] - strings = {} - string_dups = [] - for r in strings_rows: - k, v = r[0], (r[1] if len(r) > 1 else "") - if k in strings: - string_dups.append((k, strings[k], v)) - strings[k] = v - strings_lc = {k.lower(): v for k, v in strings.items()} - - def s(key): - return strings_lc.get(key.lower()) - - weapons = {} # stem -> record - for rel, obj in parsed.items(): - if kind_of(rel) != "brace:weapon": - continue - stem = os.path.basename(rel)[:-7] - w = dict(obj["weapon"]) - weapons[stem.lower()] = {"stem": stem, "file": rel, - "scope": "NPC" if rel.startswith("Species/_NPC") else "player", - "id": None, **w} - wman = parsed["Weapons/_weapons.txt"] - for i, name in wman.entries: - key = name.lower()[:-7] - if key in weapons and weapons[key]["scope"] == "player": - weapons[key]["id"] = i - - sections = {} # (race, stem) -> record - for rel, obj in parsed.items(): - if kind_of(rel) != "brace:shipsection": - continue - race = rel.split("/")[1] - stem = os.path.basename(rel)[:-12] - sections[(race, stem.lower())] = {"race": race, "stem": stem, "file": rel, "id": None, **obj["shipsection"]} - for race in ALL_RACE_DIRS: - m = parsed[f"Species/{race}/sections/_shipsections.txt"] - for i, name in m.entries: - key = (race, name.lower()[:-12]) - if key in sections: - sections[key]["id"] = i - section_stems = collections.defaultdict(list) # stem.lower -> [races] - for (race, st), rec in sections.items(): - section_stems[st].append(race) - - # ---- 4. cross-links ---------------------------------------------------- - X = {} - P("## Cross-link results") - P("") - - def tech_exists(name): - n = name.lower() - if n in tech_by: - return True - if n.startswith("grp_") and n[4:].upper() in groups: - return True - return False - - # weapon.requires -> tech (repeated `requires` lines = AND of techs) - dang = [] - case_mismatch = [] - nref = 0 - multi = 0 - for w in weapons.values(): - reqs = get_list(w, "requires") - multi += len(reqs) > 1 - for r in reqs: - nref += 1 - r = str(r) - if not tech_exists(r): - dang.append((w["file"], r)) - elif r.lower() in tech_by and tech_by[r.lower()]["name"] != r: - case_mismatch.append((w["file"], r)) - no_req = [w["file"] for w in weapons.values() if "requires" not in w] - X["weapon_requires_dangling"] = dang - X["weapon_requires_case_mismatch"] = case_mismatch - X["weapon_without_requires"] = no_req - P(f"- weapon `requires` -> tech: {nref} refs in {len(weapons) - len(no_req)} weapons ({multi} weapons list 2+ techs), " - f"{len(dang)} dangling, {len(case_mismatch)} case-mismatched; {len(no_req)} weapons have no `requires` " - f"(NPC: {sum(1 for f in no_req if f.startswith('Species/_NPC'))}, player: " - f"{[os.path.basename(f) for f in no_req if not f.startswith('Species/_NPC')]}).") - for f, r in dang: - P(f" - DANGLING `{f}` requires `{r}`") - for f, r in case_mismatch: - P(f" - case: `{f}` requires `{r}` (tech is `{tech_by[r.lower()]['name']}`)") - - # shipsection.requires / option -> tech - dang = [] - case_mm = [] - opt_dang = [] - scalar_opts = [] - nreq = 0 - nopt = 0 - for rec in sections.values(): - for r in get_list(rec, "requires"): - nreq += 1 - if not tech_exists(str(r)): - dang.append((rec["file"], r)) - elif str(r).lower() in tech_by and tech_by[str(r).lower()]["name"] != r: - case_mm.append((rec["file"], r)) - for blk_key in ("option", "optiondef"): - for blk in get_list(rec, blk_key): - # a few files write a bare `option TECH` at section level - # instead of wrapping it in option { } - opts = get_list(blk, "option") if isinstance(blk, dict) else [blk] - if not isinstance(blk, dict): - scalar_opts.append((rec["file"], blk)) - for o in opts: - nopt += 1 - if not tech_exists(str(o)): - opt_dang.append((rec["file"], o)) - X["shipsection_requires_dangling"] = dang - X["shipsection_requires_case_mismatch"] = case_mm - X["shipsection_option_dangling"] = opt_dang - P(f"- shipsection `requires` -> tech: {nreq} refs, {len(dang)} dangling, {len(case_mm)} case-mismatched.") - for f, r in dang: - P(f" - DANGLING `{f}` requires `{r}`") - for f, r in case_mm: - P(f" - case: `{f}` requires `{r}`") - X["shipsection_scalar_option"] = scalar_opts - P(f"- shipsection `option{{option T}}`/`optiondef` -> tech: {nopt} refs, {len(opt_dang)} dangling. " - f"Two forms coexist: `option {{ option A option B }}` (a mutually-exclusive choice group) and a bare " - f"section-level `option T` ({len(scalar_opts)} occurrences in {len(set(f for f, _ in scalar_opts))} files, " - f"e.g. `option DRV_PlsmFoc` on engine sections) -- both merge under the key `option`, so consumers must " - f"accept str-or-dict list members.") - for f, r in sorted(set(opt_dang)): - P(f" - DANGLING `{f}` option `{r}`") - - # tech.ship.section -> shipsection - dang = [] - nsec = 0 - for t in techs: - for blk in get_list(t, "ship"): - for sname in get_list(blk, "section"): - nsec += 1 - if str(sname).lower() not in section_stems: - dang.append((t["name"], sname)) - X["tech_ship_section_dangling"] = dang - P(f"- tech `ship{{section}}` -> shipsection: {nsec} refs, {len(dang)} dangling " - f"(matched against the union of all race catalogs, case-insensitive).") - for t, sname in dang: - P(f" - DANGLING tech `{t}` unlocks section `{sname}`") - - # tech.weapon.filename -> file - disk = {rel.lower() for _, rel in walk(root)} - dang = [(t["name"], w["filename"]) for t in techs for w in get_list(t, "weapon") if w["filename"].lower() not in disk] - X["tech_weapon_filename_dangling"] = dang - nw = sum(len(get_list(t, "weapon")) for t in techs) - P(f"- tech `weapon{{filename}}` -> file: {nw} refs, {len(dang)} dangling.") - - # tech.requires / allows -> tech - dang_req = [(t["name"], r) for t in techs for r in get_list(t, "requires") if not tech_exists(str(r))] - edges = [] - dang_allow = [] - bad_allow = [] - for t in techs: - for a in get_list(t, "allows"): - child, rp, pct, extra = parse_allows(a) - if extra or rp is None: - bad_allow.append((t["name"], a)) - if child.lower() not in tech_by: - dang_allow.append((t["name"], child)) - edges.append({"from": t["name"], "to": child, "rp": rp, "pct": pct}) - X["tech_requires_dangling"] = dang_req - X["tech_allows_dangling"] = dang_allow - X["tech_allows_unparsed"] = bad_allow - P(f"- tech `requires` -> tech/GRP_: {sum(len(get_list(t, 'requires')) for t in techs)} refs, {len(dang_req)} dangling. " - f"Groups: {dict((g, len(v)) for g, v in groups.items())}.") - for t, r in dang_req: - P(f" - DANGLING tech `{t}` requires `{r}`") - P(f"- tech `allows` edges: {len(edges)}, {len(dang_allow)} point at unknown techs, {len(bad_allow)} unparsable.") - for t, c in dang_allow: - P(f" - DANGLING tech `{t}` allows `{c}`") - roots = [t["name"] for t in techs if not any(e["to"].lower() == t["name"].lower() for e in edges)] - P(f"- techs never allowed by anything (roots/orphans): {len(roots)}: {', '.join(roots)}") - dup_names = [n for n, c in collections.Counter(t["name"].lower() for t in techs).items() if c > 1] - P(f"- duplicate tech names: {dup_names or 'none'}") - - # manifests <-> files - P("- id manifests <-> files:") - man_rep = {} - wfiles = {os.path.basename(rel).lower() for rel in parsed if rel.startswith("Weapons/") and rel.endswith(".weapon")} - listed = {n.lower() for _, n in wman.entries} - man_rep["Weapons"] = {"ids": len(wman.entries), "deleted": wman.deleted, - "listed_but_no_file": sorted(listed - wfiles), "file_but_unlisted": sorted(wfiles - listed)} - for race in ALL_RACE_DIRS: - m = parsed[f"Species/{race}/sections/_shipsections.txt"] - files = {os.path.basename(rel).lower() for rel in parsed if rel.startswith(f"Species/{race}/sections/") and rel.endswith(".shipsection")} - listed = {n.lower() for _, n in m.entries} - man_rep[race] = {"ids": len(m.entries), "deleted": m.deleted, - "listed_but_no_file": sorted(listed - files), "file_but_unlisted": sorted(files - listed)} - X["manifests"] = man_rep - for k, v in man_rep.items(): - P(f" - `{k}`: {v['ids']} ids (deleted {v['deleted'] or 'none'}); " - f"listed-but-no-file {len(v['listed_but_no_file'])}; file-but-unlisted {len(v['file_but_unlisted'])}") - for n in v["listed_but_no_file"]: - P(f" - MISSING FILE for id {[i for i, nn in (wman if k == 'Weapons' else parsed[f'Species/{k}/sections/_shipsections.txt']).entries if nn.lower() == n][0]}: `{n}`") - for n in v["file_but_unlisted"]: - P(f" - UNLISTED file `{n}` (no network/save id)") - npc_weapons_unlisted = sorted(w["file"] for w in weapons.values() if w["scope"] == "NPC") - P(f" - `Species/_NPC/weapons/*.weapon` ({len(npc_weapons_unlisted)} files) have no manifest at all; " - f"they are referenced by filename from `_NPC` shipsection `bank{{weapon}}` lines.") - - # strings - P("- localization:") - P(f" - `Strings.csv`: {len(strings_rows)} data rows -> {len(strings)} keys. {len(string_dups)} keys occur twice " - f"because one copy carries a trailing space (parse_csv strips cells; the later row wins):") - for k, a, b in string_dups: - P(f" - `{k}`: {a!r} then {b!r}") - miss_tn = [t["name"] for t in techs if s("TECHNAME_" + t["name"]) is None] - miss_td = [t["name"] for t in techs if s("TECHDESC_" + t["name"]) is None] - P(f" - TECHNAME_/TECHDESC_ for {len(techs)} techs: {len(miss_tn)} / {len(miss_td)} missing. {miss_tn} {miss_td}") - stems = sorted(section_stems) - miss_sn = [st for st in stems if s("SECTIONNAME_" + st) is None] - miss_sd = [st for st in stems if s("SECTIONDESC_" + st) is None] - P(f" - SECTIONNAME_/SECTIONDESC_ for {len(stems)} distinct section stems: {len(miss_sn)} / {len(miss_sd)} missing.") - for label, miss in (("SECTIONNAME_", miss_sn), ("SECTIONDESC_", miss_sd)): - npc_only = [st for st in miss if section_stems[st] == ["_NPC"]] - other = [st for st in miss if st not in npc_only] - P(f" - missing {label}: {len(npc_only)} are `_NPC`-only stems (never shown in the design UI); " - f"player-race stems: {len(other)} {other}") - miss_wn = [(w["file"], w.get("name")) for w in weapons.values() - if isinstance(w.get("name"), str) and w["name"].startswith("@") and s(w["name"][1:]) is None] - unnamed = [w["file"] for w in weapons.values() if "name" not in w] - P(f" - weapon `name @TOKEN`: {len(miss_wn)} unresolved of {sum(1 for w in weapons.values() if 'name' in w)}; " - f"{len(unnamed)} weapons carry no `name`.") - for f, n in miss_wn: - P(f" - UNRESOLVED `{f}` name `{n}`") - # every @token anywhere in brace files - at_missing = collections.Counter() - at_total = 0 - for rel, obj in parsed.items(): - if not kind_of(rel).startswith("brace"): - continue - for tok in re.findall(r"@([A-Za-z0-9_]+)", manifest.read_text(os.path.join(root, rel))): - at_total += 1 - if s(tok) is None: - at_missing[(rel, tok)] += 1 - P(f" - all `@TOKEN` refs in brace-block files: {at_total} refs, {len(at_missing)} unresolved.") - for (rel, tok), n in sorted(at_missing.items()): - P(f" - UNRESOLVED `{rel}` `@{tok}`") - X["strings"] = {"missing_techname": miss_tn, "missing_techdesc": miss_td, - "missing_sectionname": miss_sn, "missing_sectiondesc": miss_sd, - "unresolved_weapon_name": miss_wn, "unresolved_at_tokens": sorted(f"{r}:@{t}" for r, t in at_missing)} - - # turrets - turrets = parsed["Weapons/_turrets.txt"] - - def last_lc(d, key): - v = get_list(d, key) - return str(v[-1]).lower() if v else None - - tpairs = {(str(r[1]).lower(), str(r[2]).lower()) for r in turrets} # (weapon-size, class) - tslots = {(str(r[0]).lower(), str(r[2]).lower()) for r in turrets} # (mount size, class) - wpairs = collections.Counter((last_lc(w, "turretsize"), last_lc(w, "turretclass")) for w in weapons.values()) - w_unfit = sorted((p, n) for p, n in wpairs.items() if p not in tpairs) - bpairs = collections.Counter() - nobank = 0 - dupkeys = 0 - for rec in sections.values(): - for b in get_list(rec, "bank"): - if "turretsize" not in b: - nobank += 1 - continue - if isinstance(b.get("turretsize"), list) or isinstance(b.get("turretclass"), list): - dupkeys += 1 - bpairs[(last_lc(b, "turretsize"), last_lc(b, "turretclass"))] += 1 - b_unfit = sorted((p, n) for p, n in bpairs.items() if p not in tslots) - X["turrets"] = {"turret_rows": len(turrets), "weapon_size_class_pairs_without_turret": w_unfit, - "bank_size_class_pairs_without_turret": b_unfit, - "banks_without_turretsize": nobank, "banks_with_repeated_size_or_class": dupkeys} - P(f"- `_turrets.txt` ({len(turrets)} rows; size/class values compared case-insensitively -- the data mixes " - f"`Large`/`large`, `Missile`/`missile`, `Standard`/`standard`):") - P(f" - weapon (turretsize,turretclass) pairs with no turret row: {w_unfit or 'none'}") - P(f" - section bank (turretsize,turretclass) pairs with no turret row: {b_unfit or 'none'}") - P(f" - banks with no turretsize at all (NPC fixed-weapon banks): {nobank}; banks that repeat " - f"turretsize/turretclass inside one bank{{}} (last value taken): {dupkeys}") - - # NPC bank{weapon} refs - dang = [] - n = 0 - for rec in sections.values(): - for b in get_list(rec, "bank"): - for wf in get_list(b, "weapon"): - n += 1 - if str(wf).lower() not in disk: - dang.append((rec["file"], wf)) - X["bank_weapon_dangling"] = dang - P(f"- shipsection `bank{{weapon }}` -> file: {n} refs, {len(dang)} dangling.") - for f, w in dang: - P(f" - DANGLING `{f}` -> `{w}`") - - # default weapons - dw = parsed["Weapons/_defaultweapons.txt"] - dang = [r for r in dw if ("weapons/" + str(r[2])).lower() not in disk] - P(f"- `_defaultweapons.txt`: {len(dw)} rows, {len(dang)} name a missing weapon file. {dang or ''}") - - # AI tables - def csv_col(rel, col): - return [r[col] for r in parsed[rel] if len(r) > col and r[col]] - ai = {} - for rel in ("Data/Strategy/AI/aitechpri.csv", "Data/Strategy/AI/aitechgrp.csv", "Data/Strategy/AI/aitechmode.csv"): - rows = parsed[rel] - bad = [t for t in csv_col(rel, 0) if t.lower() not in tech_by] - ai[rel] = bad - if not rows: - P(f"- `{rel}`: 0 data rows -- the shipped file is a comment-only template (schema documented in its " - f"header, no entries); the AI's tech priorities must therefore come from code.") - else: - P(f"- `{rel}`: {len(rows)} rows; col0 not a tech: {bad or 'none'}") - bad = [x for x in csv_col("Data/Strategy/AI/affinity_section.csv", 0) if x.lower() not in section_stems] - ai["affinity_section_unknown"] = bad - P(f"- `AI/affinity_section.csv`: {len(parsed['Data/Strategy/AI/affinity_section.csv'])} rows; unknown sections: {bad or 'none'}") - bad = [x for x in csv_col("Data/Strategy/AI/raider_sections.csv", 0) if x.lower() not in section_stems] - P(f"- `AI/raider_sections.csv`: unknown sections: {bad or 'none'}") - wr = parsed["Data/Strategy/AI/weapon_replacements.csv"] - bad = [x for r in wr for x in r if x and x.lower() not in weapons] - ai["weapon_replacements_unknown"] = bad - P(f"- `AI/weapon_replacements.csv`: {len(wr)} rows; unknown weapon stems: {bad or 'none'}") - fams = collections.Counter(str(w.get("weaponfamily")) for w in weapons.values() if "weaponfamily" in w) - aw = csv_col("Data/Strategy/AI/affinity_weapon.csv", 0) - bad = [x for x in aw if x not in fams] - P(f"- `AI/affinity_weapon.csv`: families {sorted(set(aw))}; not a weaponfamily in any .weapon: {bad or 'none'}. " - f"weaponfamily values in data: {dict(fams)}") - # scenarios - for rel in sorted(parsed): - if rel.startswith("Scenarios/") and rel.endswith("Templates.csv"): - bad = [(r[0], x) for r in parsed[rel] for x in r[1:4] if x.lower() not in section_stems] - P(f"- `{rel}`: {len(parsed[rel])} rows; unknown sections: {bad or 'none'}") - if rel.startswith("Scenarios/") and rel.endswith("Techs.csv"): - bad = [x for x in csv_col(rel, 0) if x.lower() not in tech_by] - P(f"- `{rel}`: {len(parsed[rel])} rows; unknown techs: {bad or 'none'}") - X["ai"] = ai - json.dump(X, open(os.path.join(out, "crosslink.json"), "w"), indent=1) - P("") - - # ---- 5. artifacts ------------------------------------------------------ - nodes = [] - for t in techs: - strat = get_list(t, "strategy") - inc = [x for b in strat for x in get_list(b, "inc")] - dec = [x for b in strat for x in get_list(b, "dec")] - nodes.append({ - "name": t["name"], - "display_name": s("TECHNAME_" + t["name"]), - "description": s("TECHDESC_" + t["name"]), - "family": t.get("family"), - "family_inferred": t["name"].split("_", 1)[0].upper(), - "type": t.get("type"), - "threat": t.get("threat"), - "group": t.get("group"), - "option_cost": t.get("option_cost"), - "requires": [str(r) for r in get_list(t, "requires")], - "benefits_inc": inc, - "benefits_dec": dec, - "sections": [str(x) for b in get_list(t, "ship") for x in get_list(b, "section")], - "weapons": [w["filename"] for w in get_list(t, "weapon")], - "allows": [e["to"] for e in edges if e["from"] == t["name"]], - }) - tech_tree = { - "_about": "SOTS1 MasterTechList.tech normalized. family is only written on ~half the nodes; " - "family_inferred is the name prefix (IND/WEP/DRV/...). edges[].pct: per-race availability % as written; " - "a race absent from pct has no override in the file (the engine default -- believed to be 100 -- " - "is code-owned, not asserted here). rp = research-point cost of the edge. " - "requires may name GRP_, satisfied by any tech with group .", - "races": RACES, - "groups": dict(groups), - "nodes": nodes, - "edges": edges, - } - json.dump(tech_tree, open(os.path.join(out, "tech_tree.json"), "w"), indent=1) - - def norm_weapon(w): - d = dict(w) - d["display_name"] = s(w["name"][1:]) if isinstance(w.get("name"), str) and w["name"].startswith("@") else w.get("name") - return d - json.dump({"_about": "All *.weapon files (Weapons/ = player catalog with ids from _weapons.txt; Species/_NPC/weapons = NPC, no ids). " - "Keys lower-cased; repeated keys -> lists; bareword numbers typed.", - "weapons": [norm_weapon(w) for w in sorted(weapons.values(), key=lambda w: (w["scope"], w["stem"].lower()))]}, - open(os.path.join(out, "weapons.json"), "w"), indent=1) - - def norm_section(r): - d = dict(r) - d["display_name"] = s("SECTIONNAME_" + r["stem"]) - d["description"] = s("SECTIONDESC_" + r["stem"]) - d["unlocked_by"] = [t["name"] for t in techs for b in get_list(t, "ship") if r["stem"].lower() in [str(x).lower() for x in get_list(b, "section")]] - return d - json.dump({"_about": "All Species//sections/*.shipsection; id from the race's _shipsections.txt (null = unlisted). " - "Keys lower-cased; repeated keys (bank, option, thruster, requires) -> lists.", - "sections": [norm_section(r) for r in sorted(sections.values(), key=lambda r: (r["race"], r["stem"].lower()))]}, - open(os.path.join(out, "shipsections.json"), "w"), indent=1) - - json.dump(strings, open(os.path.join(out, "strings.json"), "w"), indent=1, ensure_ascii=False) - - with open(os.path.join(out, "tech_tree.dot"), "w") as f: - f.write("digraph sots_tech {\n rankdir=LR; node [shape=box, fontsize=9];\n") - fam_color = {"IND": "#f4d03f", "NRG": "#e74c3c", "SLD": "#3498db", "DRV": "#9b59b6", "TRP": "#e67e22", - "WAR": "#c0392b", "BAL": "#7f8c8d", "BIO": "#2ecc71", "CCC": "#1abc9c", "DRN": "#95a5a6", "XNC": "#d35400"} - for n in nodes: - col = fam_color.get(str(n["family"]), "#ffffff") - label = n["display_name"] or n["name"] - f.write(f' "{n["name"]}" [label="{label}\\n{n["name"]}", style=filled, fillcolor="{col}"];\n') - for e in edges: - lab = f"{e['rp']}" if e["rp"] is not None else "" - if e["pct"]: - lab += "\\n" + " ".join(f"{r[:2]}{v}" for r, v in e["pct"].items()) - f.write(f' "{e["from"]}" -> "{e["to"]}" [label="{lab}", fontsize=7];\n') - f.write("}\n") - - P("## Artifacts") - P("") - for fn in ("tech_tree.json", "weapons.json", "shipsections.json", "strings.json", "schema_stats.json", "crosslink.json", "tech_tree.dot"): - P(f"- `{fn}` ({os.path.getsize(os.path.join(out, fn)) // 1024} KB)") - P(f"- tech_tree.json: {len(nodes)} nodes, {len(edges)} edges; weapons.json: {len(weapons)}; " - f"shipsections.json: {len(sections)}; strings.json: {len(strings)} keys") - print("\n".join(rep)) - return 0 if not fails else 1 - - -if __name__ == "__main__": - sys.exit(main(sys.argv[1], sys.argv[2])) diff --git a/save-reader/ref/save-editor-structs.md b/save-reader/ref/save-editor-structs.md deleted file mode 100644 index 40abf50..0000000 --- a/save-reader/ref/save-editor-structs.md +++ /dev/null @@ -1,550 +0,0 @@ -# Sword of the Stars 1 (SOTS1) — Save-File Struct Reference (Community RE) - -Harvested from two community save-game editors for cross-checking against binary-recovered -serializable types. SOTS1 uses a self-describing "Streamable" serialization system: **field -order + type matter**, and most values are preceded by their own name string. - -Target game version: **SOTS1 v1.8+** (both editors target the 1.8.x line; repo2 says "1.80 +19"). - -## Sources -- **[R1] BardezAnAvatar/Sots.Sots1.SavedGameEditor** — C#. A *complete, ordered, typed* - Streamable parser. Migrated from SourceForge `sots-sge`. This is the authoritative layout - evidence (read/write methods reproduce exact on-disk order). Key files under - `Bardez.Project.SwordOfTheStars.DataStructures/`: - - `BaseSaveStructures.cs` (primitives + framing), `SharedSaveStructures.cs` (coords/colors), - `SaveGameDataStructure.cs` (top level), `SummarySaveStructures.cs`, - `CreateParametersSaveStructures.cs`, `SimulationSaveStructures.cs` (11,974 lines — the bulk), - `CdTableSaveStructures.cs` (combat/AI table). IO: `Bardez.Project.SwordOfTheStars.IO/{Gzip,SaveFileIO}.cs`. -- **[R2] ghbplayer/SOTSedit** — C#/WPF. A *name-tag scanner* (does NOT model layout; it searches - the decompressed blob for length-prefixed field-name strings and reads the value that follows). - Value comes from `Parse.cs` + the field catalog `SOTSEdit.cfg` (friendly-name ⇄ serialized-name - ⇄ type mapping) and race/semantic hacks. Written 2011 "to learn C#"; author calls the parser weak. - -Both editors independently confirm the same primitives, framing markers, race IDs, and color IDs. - ---- - -## 1. File format & serialization mechanics - -### 1.1 Container -- **Whole `.sav` file = gzip stream.** Decompress first (`GZipStream`, standard gzip). [R1 Gzip.cs, R2 gzip.cs] -- Editors work on the **decompressed** byte stream (R1 names its test artifacts `*.sav.inflate.dat`). -- No separate magic/version header is decoded by either editor beyond the top-level Summary block; - version is implied by scenario/field presence, not a numeric version field. Endianness: **little-endian** - throughout (`BitConverter` on x86). - -### 1.2 Decompressed top-level order [R1 SaveGameData.ReadFromStream] -``` -SaveGameData: - 1. SummarySaveStruct summary - 2. CreateParametersSaveStruct createParams - 3. SimSaveStruct sim <-- the giant one (players/systems/fleets/etc.) - 4. CdTable cdTable <-- combat / AI state table -``` - -### 1.3 Primitive encodings [R1 BaseSaveStructures.cs] -Text encoding is **windows-1252** (R1) / ASCII (R2). - -- **StringStruct** (raw string): `Int32 length` + `length` bytes (no NUL terminator counted). - R2 calls this a "BStr": 4-byte little-endian length prefix + ASCII bytes. -- **Padding rule (critical):** every *basic* value struct is **NUL-padded to a 4-byte boundary**. - `PaddingSize = 4`. Padding is computed over `(sizeof(Int32 desc-length) + description.Length + valueBytes)`. -- **Named value fields** (`BasicSaveStruct` subclasses) are each laid out as: - `StringStruct description` (a field-name tag, often the non-descriptive `"."`) → then the value → - then NUL padding to 4 bytes. Concrete leaf types: - | Struct | Payload after description tag | - |---|---| - | `Int32SaveStruct` | 4-byte Int32 | - | `Int64SaveStruct` | 8-byte Int64 | - | `FloatSaveStruct` | 4-byte IEEE Single | - | `BooleanSaveStruct` | 1 byte (0/1), padded to 4 | - | `StringSaveStruct` | nested StringStruct (len+bytes) | - | `ByteArraySaveStruct` | raw bytes (length externally known) | - - So the on-disk shape of a named scalar is: `[len][name-ascii][pad] [value] [pad]`. R2 exploits exactly - this: it locates a field by searching for `[len][name]` and reads the value immediately after. - -### 1.4 Complex-struct framing (the "BEEFBEEF" envelope) [R1 ComplexSaveStruct] -Every **complex** structure is framed: -``` -StringStruct description (NUL-padded to 4) -UInt32 0xBEEFBEEF (begin marker) -... body (ordered child fields) ... -UInt32 0x41104110 (end marker = bitwise NOT of 0xBEEFBEEF) -``` -`0xBEEFBEEF` / `~0xBEEFBEEF (0x41104110)` bracket every complex object — a reliable resync/validation -signature when scanning the binary. (`ISotsStructure` leaf types are NOT framed; only `ComplexSaveStruct`.) - -### 1.5 Array conventions [R1 BaseSaveStructures.cs] -- **ComplexArraySaveStruct**: framed (has description + BEEFBEEF), body = `Int32SaveStruct count` - then `count` × T. -- **NonComplexArraySaveStruct**: NOT framed; body = `Int32SaveStruct count` then `count` × T. - (Distinguishing which arrays are framed vs. not is itself layout evidence — see per-struct notes.) - -### 1.6 Conditional & polymorphic reads (watch for these in the binary) -- **Optional-by-flag:** a boolean/int gate precedes an optional sub-object. - - `SimPlayerColorSaveStruct`: `Int32 colorIndex`; **iff `colorIndex == -1`**, an `RgbColorInt32` - (custom RGB) follows. Otherwise palette index only. - - `SimPlayerDesignDw2SaveStruct` (weapon slot): `Boolean bId`; if true → `Int32 wId`, else → - `StringSaveStruct wfn` (weapon full resource path); then `Int32 dId`. - - `SimFleetShipDetails`: `Boolean hbq` gates `bq`; `Boolean hsp` gates `sp`. Fleet flight-plan/lay - gated by `hfPlan` / `hLay` booleans. - - `SimSystemDetailNvo.isInd` gates independent-colony sub-block; `SimSystemDetailsIndi.hindi`, - `SimSystemDetailsVonNeumann.vnh` similar boolean gates. -- **Polymorphism by string tag:** `SimSvSctObXscn` reads `StringSaveStruct xcsn`, then switches: - `"crowdefs"`, `"gmtrigger"`, `"traps"`, `"indsys"`/default → different body subclass. -- **Polymorphism by fixed position:** `SimScSctObEncObjArray` (grand-menace/encounter objects) reads a - count then dispatches subclass **by index 0..8** in fixed order: - `0 Infest, 1 Dsn, 2 AsteroidMonitor, 3 TD, 4 WD, 5 Hives, 6 Rsuc, 7 Dfts, 8 Ini2`. - -### 1.7 Write-time quirks worth knowing (R2) -- R2 edits in place and cannot safely change string length (it truncates/space-pads to the original - length). R1 rewrites the whole stream and re-pads. If the binary stores string lengths, the game - reads them dynamically (R1 proves round-trip works when re-padded). -- R2 planet OID/PID hack: on-disk **`OID = PID * 16`** (R2 divides by 16 to show a "PlayerID"). - i.e. the raw owner id field is the player index shifted left 4 bits. - ---- - -## 2. Enums / ID tables (agreed by both editors) - -### 2.1 Species / race ID [R2 Parse.cs addRace(); R1 PlayerSlot.FxSp comment] -| ID | Species | -|---|---| -| 0 | Human | -| 1 | Hiver | -| 2 | Tarka(s) | -| 3 | Liir | -| 4 | `_NPC` / AI-rebellion / grand-menace player (R1: "??? AI Rebellion") | -| 5 | Zuul | -| 6 | Morrigi | - -### 2.2 Player color ID (palette index) [R1 PlayerSlot.FxCrId & SimPlayerColor] -`01 Red, 02 Yellow, 03 Blue, 04 Pink/Magenta, 05 Orange, 06 Green, 07 Aqua, 08 Gray, -09 Dark Green, 10 Purple`. Value **-1 ⇒ custom RGB triplet follows** (see §1.6). - -### 2.3 Difficulty [R1 PlayerSettings.Difficulty] -`0 Easy, 1 Normal, 2 Difficult`. - -### 2.4 Sentinel values seen in fields -`0x7FFFFFFF (Int32.MaxValue)` used as "tag"/unset (PlayerSlot.tag); -`team = -1 (0xFFFFFFFF)` = no team; R2: value `-1` = field absent in this save. -No named C# enums exist for tech IDs / weapon families — techs and weapons are **string resource -names** (e.g. tech `tNm`, weapon `wfn`), not numeric enums. Weapon *family* enumeration lives only in -the CdAi combat block as `aiSitWepFams` (opaque int set). - ---- - -## 3. Summary block [R1 SummarySaveStructures.cs] (complex) - -### SummarySaveStruct (fields in on-disk order) -1. `StringSaveStruct gameName` -2. `Int32 turn` -3. `Int32 numSys` (system count) -4. `Int32 checkSum` -5. `ComplexArray players` -6. `SessionSaveStruct session` -7. `Int32 mapShape` -8. `Int32 incMod` (income modifier) -9. `Int32 resMod` (research modifier) -10. `Boolean alliances` -11. `Boolean teams` -12. `Boolean encounters` -13. `StringSaveStruct scenario` - -### PlayerSlotWrapper (complex): `{ PlayerSlotSaveStruct slot; Int32 rank; }` - -### PlayerSlotSaveStruct (complex) — new-game slot definition, ordered: -`Boolean isPlay, isDead, isReq, isRec, isFxNm` → `String fxNm` (fixed name) → -`Boolean isFxSp` → `Int32 fxSp` (species, §2.1) → `Boolean isFxCr` → -`NestedInt32 fxCrId` (color, §2.2) → `Boolean isFxBd` → `String fxBd` (badge) → -`Boolean isFxAv` → `String fxAv` (avatar) → `Int32 tag` (often 0x7FFFFFFF) → -`Int32 pwd` (password) → `Int32 team` (-1=none) → `PlayerSettingsSaveStruct settings`. - -### PlayerSettingsSaveStruct (complex): -`Int32 initialTreasury, initialColonies, initialTechnologies, difficulty (§2.3)`. - -### SessionSaveStruct (complex) → `TmrsSaveStruct tmrs`: -`Int32 tstl (0x7F7FFFFF), tctl (0x42700000), tqtl (0x7F7FFFFF), tqtle (0)` — timer limits (float bit-patterns stored as int). - ---- - -## 4. CreateParameters block [R1 CreateParametersSaveStructures.cs] (complex) - -### CreateParametersSaveStruct (ordered): -`String name; Int32 id; Int32 rSeed (random seed); Int32 aid; String key;` -`MapPSaveStruct mapP;` `Int32 mapS; Int32 mapF; Int32 nSys;` `Float rEnc (random-encounter rate);` -`Int32 sDist;` `Float sSize; Float sRes;` `Int32 sSuit; Int32 maxP; Int32 aSpec;` -`Boolean bAlly; Int32 nTeam; Boolean tmgrp;` -`Int32 pSav (start savings); Int32 pCol (start colonies); Int32 pTech (start techs);` -`Float incM; Float resM; ScrpSaveStruct scrp;` - -### MapPSaveStruct (complex) — initial map/galaxy generation: -1. `Int32 unknown1` -2. `ComplexArray planetArray` -3. `NonComplexArray> players` (per-player int arrays; "non-complex array of players") -4. `ComplexArray npcArray` (≈ players − 1; independents/NPCs) - -### PlanetSaveStruct (complex) — initial star node geometry: -`SpatialCoordinate coordinates (x,y,z floats)`, `Int32 unknown1..4` -(values seen: `0x7FFFFFFF`, `0x7F7FFFFF`). NOTE: this is the *map-generation* planet record; the -*live* planet/colony state lives in `SimSystemDetailsSaveStruct` (§8). - -### MapPNpc (complex): `Int32 unknown1, unknown2`. ### ScrpSaveStruct (complex): `Int32 spc`. - -### Shared value types [R1 SharedSaveStructures.cs] -- `SpatialCoordinateSaveStruct` (complex): `Float x, y, z`. -- `RgbColorFloat` (leaf): `Float r,g,b`. `RgbaColorFloat` (leaf): `RgbColorFloat rgb; Float a`. -- `RgbColorInt32` (leaf): `Int32 r,g,b`. - ---- - -## 5. Simulation block — top level [R1 SimulationSaveStructures.cs] - -### SimSaveStruct (complex) — the master world state, ordered: -``` -String keyPath -Int32 nMsz, nMlc, nMnx (next-id / size counters) -NonComplexArray playerIds, designIds, systemIds, fleetIds, shipIds, tradeIds -Int32 modCount, frame, gameId -AttributeSaveStruct attribute -RngSaveStruct rng (ByteArray unknownData ~2500 bytes: RNG state) -String gameName -Int32 map, incMod, resMod, enAl, enTm, gOTurn -NestedInt32 gOWinPly -Int32 npcm, npco, npci, npcv, npca, szad, rsad, suad -ComplexArray sprjs (shared/special research projects) -Float randEncAdjustment -Int32 cmbtid (next combat id) -ComplexArray turnstats (per-turn per-player history) -NonComplexArray creps (combat reports) -Int32 ninv, allExc1, allExc2, allExcCF -NonComplexArray players <-- EMPIRES (§6) -SimSpeciesArraySaveStruct species (galaxy species list) -NonComplexArray systems <-- STAR SYSTEMS (§8) -SimNodeGrid2 ndgr2 (node-line / warp graph) -SimTradeManager trdmgr (§9) -SimSpyManager spymgr (Int32 xsid, nspy) -NonComplexArray flt <-- FLEETS (§10) -NonComplexArray acts -SimSvSctOb svSctOb (scenario/encounter objects, §11) -Int32 zdsc, zdsi, zdst (Zuul/system-destroyer counters) -``` - -### Small shared sim types -- `SimPopGSaveStruct` (complex): `Int32 popT; Int32 popS; Int64 popC` (pop type / species / civ count). -- `ResearchSaveStruct` (leaf): `NonComplexArray us; String nm; String ntg`. - `ResearchOptionalSaveStruct`: `Int32 usc, usp`. -- `TurnPly` (leaf): `Int32 ply; PlyHistSaveStruct hist`. -- `PlyHistSaveStruct` (complex): `Int32 ply; PlyHistStatsSaveStruct[] stats`. -- `PlyHistStatsSaveStruct` (complex): `Int64 pop; ComplexArray sacq, slost;` - `Int32 trn, almem, inc, tdinc, sav, col, bat, tch; NonComplexArray cls`. -- `PlyHistStatsSacq` (complex): `Int32 set, ses, seop, senp; NonComplexArray seo`. -- `PlyHistClsSaveStruct` (leaf): `Int32 cls, shpt, shpl, shpk, satt, satl, satk` (ship/sat built/lost/killed by class). - ---- - -## 6. Player / Empire [R1 SimulationSaveStructures.cs] - -### SimPlayerSaveStruct (leaf wrapper): `Int32 playerId; SimPlayerDetailsSaveStruct details`. - -### SimPlayerDetailsSaveStruct (complex) — the empire record, on-disk order: -``` -SimPlayerTechTree techTree <-- TECH TREE (§7) -Int32 homeSystem, playerIndex -String playerName -Int32 species (§2.1) -SimPlayerColorSaveStruct colorId (palette idx or -1 + RGB, §1.6/§2.2) -String badge, avatar -Int32 team, sav (savings) -Float idealSuit, suitTolerance, maxOH -Float resRate, resModifier, resScl (research) -Int32 trm, trp, tra -Float outMod, rebOutMod, scOutMod, incMod, popMod, terraMod (economy multipliers) -Boolean aMine; Float minPure, minRate; Int32 ngts, prGtTrf, gTraf -Int32 cstR, cstE, cstT, maint, shrm, status, elim -Boolean npc, rebAi, reqCL -SimPlayerTeamSaveStruct teamStruct (Int32 alid, al, na, cf) -Int32 hasVac, hasImm, npTrak, hasDisc, hasDiscSp, hasDiscCl, hasEnc, hasEng -SimPlayerEventsSaveStruct events (Int32 evNxId; ComplexArray) -NestedInt32 fngNum -Int32 pvSav, pvMA, aibN -Boolean cnTrd, cnRad, hgs, hadvs, harcc, cnVItl -Float pddm -Int32 bankWrn, bankTrn, bankPr, bankEl -SimPlayerShipRecsEventsSaveStruct shipRecs -Int32 nextPrjId, plcy, pswd, lret, nmeid -Boolean cdp -SimPlayerSpySaveStruct spy2 (Int32 defc2, rtc, evc, ttc) -SimPlayerCivrSaveStruct civR (Float smx; ComplexArray) -Int32 aidf -Boolean srn; Int32 srcTo, lboid, lcid2 -String resTnm -Boolean resErrRoll -SimPlayerModsSaveStruct conMods (3× SimPlayerConModSaveStruct {Float conMod, savMod}) -NonComplexArray ownerIds -NonComplexArray designs <-- SHIP DESIGNS (§7.2) -NonComplexArray droneDesigns -NonComplexArray notes (Int32 ntSys; String ntTxt; Int32 ntTrn) -NonComplexArray pr (Float prm; Int32 prbt) -Boolean hasAiRebellion, cta -NestedInt32 aienf -NonComplexArray nSprj -Int32 nexp, nWeapXcl -ComplexArray ovjs (objectives: Int32 id; Bool cmp; Int32 spr; String dsc; Int32 nid) -ComplexArray dipStats (diplomacy; see below) -ComplexArray comms (Int32 msgt; SimPlayerCommMsg msg) -ComplexArray preps -ComplexArray odes -ComplexArray owep -ComplexArray otch -NestedInt32 aid -Int32 ndeflay, rdtc, tnc -``` -- `SimPlayerDipStat` (complex): `Int32 other; SimPlayerDipStatDetail nap, ally, cf; Int32 deadhome`. - `SimPlayerDipStatDetail` (leaf): `Int32 last_, last_bty, bkn_, bty_`. -- `SimPlayerCommMsg` (complex): `Int32 cid2, snd, rcp, exp, sent, rcpt, sys`. -- `SimPlayerPrepSaveStruct` (complex): `Int32 oid, pid, flds, sav, home, ncol, mpwr, mcls, mmsl, nshp, nsat`. -- `SimPlayerOdes/Owep/Otch` (complex, "old design/weapon/tech" build history): - `Odes {Int32 ontF, otnL, odid, opid}`, `Owep {Int32 ontF, otnL, odet; String owep; Int32 owith}`, - `Otch {Int32 ontF, otnL, odet; String otch; Int32 owith}`. -- `SimPlayerDetailsSpecialProjectT` (leaf): `Int32 sprjT; SimPlayerDetailsSpecialProjectDetails sprj`. - `...Details` (complex): `Int32 stp; SpecialProjectSpi spi; tail (polymorphic AsMon | Tech)`. - `...Spi` (complex): `Int32 sPid, sts; Float cst; Int32 mxC; String name; Int32 trns`. - tail `AsMon`: `Int32 rDn, sys, rMn, rMx; Float rMd`; tail `Tech`: `Int32 rDn; Float aOdd, aInc; String tch; Int32 rCst`. - ---- - -## 7. Tech tree & ship designs - -### 7.1 Tech tree [R1] -- `SimPlayerTechTree` (complex): `NonComplexArray tree; NonComplexArray techs`. -- `SimPlayerTechTreeBranch` (leaf): `String tNm` (tech name); `NonComplexArray branches` (child tech names). -- `SimPlayerTechTreeTech` (leaf), ordered: - `String tNm` (tech name) → `Int32 st` → `Int32 tResCost` (research cost) → - `Int32 tResDone` (progress) → `Int32 tAcq` (turn acquired) → `Int32 tiAcq` (turns-to-acquire) → - `Int32 tbd` → `Boolean tfc` → `Int32 tUnlck` (unlocked flag). - **Techs are identified by string name, not a numeric enum.** - -### 7.2 Ship designs [R1] -- `SimPlayerDesignEntrySaveStruct` (leaf): `Int32 designId; SimPlayerDesignSaveStruct design`. -- `SimPlayerDesignSaveStruct` (complex), ordered: - `Boolean faiDes; Boolean dHide; Int32 dWep; String dName;` `SimPlayerDesignSectionArray sections;` - `Int32 dtc; NonComplexArray dwgv`. -- `SimPlayerDesignSectionArray` (leaf): `Int32 count` + `count` × `SimPlayerDesignSectionEntrySaveStruct`. -- `SimPlayerDesignSectionEntrySaveStruct` (complex): `SimPlayerDesignSectionSaveStruct dsec` (Int32 unknown1, unknown2); - `ComplexArray dgbnk2;` `ComplexArray dOpts`. -- `SimPlayerDesignUnknown1SaveStruct` (complex) → `SimPlayerDesignDw2SaveStruct dw2` (weapon slot, - conditional — see §1.6): `Boolean bId; (bId? Int32 wId : String wfn); Int32 dId`. -- `SimPlayerDesignDwg` (complex): `NonComplexArray wgng`; - `SimPlayerDesignGng` (leaf): `Int32 wgid; SimPlayerDesignDwgWgb wgb` (complex: `Int32 unknown1..3`). - ---- - -## 8. Star systems & planets [R1] - -- `SimSystemSaveStruct` (leaf): `Int32 sysId; SimSystemDetailsSaveStruct details`. -- **`SimSystemDetailsSaveStruct` (complex)** — the live system+colony record, on-disk order: -``` -SpatialCoordinate pos -RgbaColorFloat starColor -Int32 idx -Int32 size (1-10) -Float suit (climate hazard) -Int32 res, aRes, mRes (resources / asteroid / extra) -Boolean noRebAi -Int32 tRes, pop -ComplexArray popG -Float infra -Int32 pvPop; ComplexArray pvPopG; Float pvInfra, pvSuit; Int32 pvRes, pvARes2, pvMRes; Bool pvNoRebAi (previous-turn snapshot) -SimSystemDetailRtsSaveStruct rts (Float sRs, sRt, sRsc, sRtf, sRi, sRoh, sRnr — IO allocations) -Int32 abdn; Boolean dstyd; Int32 tnsOh -Float outMod, repCur, repMax -Int32 ntdev, pbon -ComplexArray pbon2 -Float ibon -Int32 ltis, rbfl, rbtn, rbfr, rbwn, hsrg -NonComplexArray halt (Int32 haltt; Bool haltv) -SimSystemDetailsVonNeumann vnm (Bool vnh gate → details: Bool vnd, vnex3, vnpex3) -String name -SimSystemDetailFlags1 flags1 (Int32 vFlags, eFlags, aFlags, fFlags, gFlags) -Int64 bats2 (recent battles; larger=more recent) -Int64 rcex -SimSystemDetailFlags2 flags2 (Int32 mnRFlags, rfRFlags, clkFlags) -Int32 eggScio, terrFl, tAcq, tfAcq, tDst -ComplexArray dcs; Int32 dsu -ComplexArray cm, pvcm (SimSystemDetailCm: Int32 msp, mv) -ComplexArray cme2 (Int32 mid, mtrT, mn, mtp; ComplexArray mfx; String mdsc) -ComplexArray spies -Int32 pid (owner player id), defF, defSf -SimSystemDetailBq bq (ComplexArray ords; Ord: Int32 desId, con, conleft, sav, ordId) -NonComplexArray adct (Int32 ads, adt) -Int32 numPlgs2 -NonComplexArray flts, gfs, snF, mnF (fleets / gates / stations / monitors present) -NonComplexArray nvos (colonies; see below) -NonComplexArray nve (Int32 ePid, ets, eid) -NonComplexArray nvs (Int32 pid; SimSystemDetailVsPView pview) -SimSystemDetailsIndi indi (Bool hindi gate → indsp, SimPlayerColor indcl, String indnm/indav/indba) -``` -- `SimSystemDetailNvo` (leaf): `Int32 pid, tShn, oId; Boolean isInd; SimSystemDetailNvoIndi indi` - (independent-colony sub-block gated by `isInd`). -- `SimSystemDetailVsPView` (complex — per-player *seen* snapshot of a colony): `Int32 vTrn, pop;` - `ComplexArray pop2; Int32 infra; Float suit; Int32 res, aRes2, mRes; Bool noRebAi;` - `Int32 pbon; ComplexArray pbon2; Float ibon; Int32 terrFl; Bool footer`. - -### 8.1 R2 ⇄ R1 cross-map for planet/colony fields (verifier gold) -R2 finds these serialized tags anywhere in the "Planets" region (between markers `NumSys`…`NdGr2`). -They correspond to fields inside R1's `SimSystemDetailsSaveStruct` / `...VsPView`: - -| R2 serialized tag | type | meaning | R1 field | -|---|---|---|---| -| `Idx` | int | planet/system id | `idx` | -| `Name` | string | name | `name` | -| `Size` | int | 1-10 | `size` | -| `Suit` | float | climate hazard | `suit` | -| `Res` / `ARes2` / `MRes` | int | resources | `res` / `aRes` / `mRes` | -| `Infra` | float | infrastructure | `infra` | -| `ibon` | float | infra bonus | `ibon` | -| `Pop` | int | imperial pop | `pop` | -| `pbon` | int | imperial pop bonus | `pbon` | -| `PopC` | long | civilian pop | (SimPopG `popC` Int64) | -| `OID` | int | owner id (**= PID×16**) | `pid` (owner) | -| `PID` | int | derived player id | (OID/16) | -| `SRt/SRsc/SRtf/SRi/SRoh` | int | IO trade/ship/terraform/infra/overharvest | `rts.sRt/sRsc/sRtf/sRi/sRoh` | -| `Abdn` | short | abandon order | `abdn` | -| `Dstyd` | short | star annihilated | `dstyd` | -| `ltis` | short | last-time-seen | `ltis` | -| `VFlags/EFlags/AFlags/FFlags/GFlags` | int | state flags | `flags1.*` | -| `Bats2` | int | recent combat | `bats2` (Int64 in R1) | -| `nadct` | int | addicted (1=yes) | (in `adct` array) | -| `NumFlts/NumGFs/NumSnF/NumMnF` | int | fleets/gates/stations/monitors | `flts/gfs/snF/mnF` counts | - -Note the **type disagreements** (verifier flags): R2 reads `Abdn`, `Dstyd`, `ltis` as **short (Int16)** -while R1 models them as framed `Int32SaveStruct`; R2 reads `Bats2` as int while R1 uses Int64. R2's -name-scan reads the value bytes directly after the tag+pad, so R2's width is the more literal -on-value-bytes claim for those specific fields; treat as "value is small, low bytes are the datum." - ---- - -## 9. Trade & node grid [R1] -- `SimNodeGrid2` (complex): `ComplexArray paths; Int32 nextId`. - `SimNodeGridPath` (complex): `Int32 npt, npid, npfr(from), npto(to), npctm, npcby, npdtn, npdtf, npenp, npuse, nptf`. -- `SimTradeManager` (complex): `NonComplexArray tradeSectors; Float sctSize; List rt`. - `SimTradeSector` (leaf): `Int32 tradeId; SimTradeSectorTradeSaveStruct trade`. - `SimTradeSectorTradeSaveStruct` (complex): `SpatialCoordinate pos; Int32 tradeSectorGridId;` - `SimTradeSectorTradeCtrSaveStruct tsctr(3 floats); Int32 tssec, tsct, tscr, ptssec, ptsct, ptscr;` - `ComplexArray<...Fwarn {Int32 pId, ntrns}> fwarn; NonComplexArray systems, tsflt`. - `SimTradeSectorRt` (complex): `Int32 tro, trfow, trfr, trfrs, trtow, trto, trtos, trtc`. - ---- - -## 10. Fleets & ships [R1] -- `SimFleet` (leaf): `Int32 fltId; SimFleetDetails flt`. -- **`SimFleetDetails` (complex)**, ordered: - `SpatialCoordinate pos; Int32 pId, locId; FlightPlanContainer fplan (Bool hfPlan gate);` - `String ftName; Int32 ftTrans; SimFleetOrigin ftOrig (Int32 ×3);` - `Int32 ftFlag, ftae, ftpae, ftEnc, ftMs, perm; SpatialCoordinate prvPos;` - `LayContainer lay (Bool hLay gate); NonComplexArray ships`. -- `SimFleetDetailsFlightPlan` (complex): `ComplexArray wpts; Float fPsp2; Int32 fPeta2;` - `Fpogn2 (3 floats); SpatialCoordinate fPdpos; Int32 pnd`. `Wpt`: `Int32 wpt, tp; Nrt {Int32 nrp,nrf,nrt}`. -- `SimFleetShip` (leaf): `Int32 shipId; SimFleetShipDetails ship`. -- **`SimFleetShipDetails` (complex)**, ordered: - `Int32 desId (design id), fltId, plrId; Float range; SimFleetShipHealth health (3 floats: command/mission/drive);` - `Int32 conCap, refCap, repCap, mineCap, plg, act; Boolean dep, atq; Int32 encId;` - `SimFleetShipPrish prish (Int32 prMax; NonComplexArray prSp); Int32 lct, tsd, atsp, tblt;` - `Boolean hbq; SimFleetShipDetailsBq bq (gated by hbq);` - `Boolean hsp; SimFleetShipDetailsSp sp (gated by hsp; two SpPop pop/pPop);` - `NonComplexArray th (Float th, thm)`. - `SimFleetShipDetailsBqOrd`: `Int32 desId, con, conleft, sav, ordId` (same shape as system BqOrd). - ---- - -## 11. Combat reports & scenario/encounter objects [R1] -- `SimCrepSaveStruct` (combat report, complex): `Int32 cid, trn; SpatialCoordinate pos; Int32 sid, auto, dur, cow, cdst, cpk, cpt, cdt, cdi;` - `ComplexArray prep; ComplexArray wrep`. - `SimCrepPrepSaveStruct`: `Int32 plr; Bool ai; Int32 ally, status, mxeng, mxcls, mxmsl;` - `NonComplexArray cls; NonComplexArray sec; Int32 ndam; ComplexArray srep`. - `SimCrepPrepSrepSaveStruct`: `String name; Int32 did, cls; Int64 caps2; Int32 nshp, nfld, nlst, dtak; SimDamsSaveStruct dams`. - `SimDamsSaveStruct`: `Int32 dams, damp, dami, damt`. `SimCrepWrepSaveStruct`: `String wep; SimDams dams`. -- `SimSvSctOb` (complex): `ScnObjStruct scn; NonComplexArray xscn; SimScSctObEncObjArray encObjs`. - `SimSvSctObXscn` = polymorphic-by-string (§1.6): `traps` (ComplexArray), - `gmtrigger` (Int32 gmch), `crowdefs` (Int32 sys; NonComplexArray dsys,des; Float drad), `indsys`/default (empty). -- **`SimScSctObEncObjArray`** = grand-menace/encounter table, dispatched **by fixed index 0-8** (§1.6). - Each subclass carries that menace's state, e.g.: - - `EncInfest` (Hiver infestation): `NonComplexArray asg; ComplexArray infests; Int32 deshive, deslarva`. - - `EncHives`: `Int32 qDesignId; ComplexArray hives; ComplexArray queens; ComplexArray sysMem`. - - `EncAsteroidMonitor`, `EncTD`, `EncWD`, `EncRsuc`, `EncDfts` (Von Neumann; large), `EncIni2`. - ---- - -## 12. CdTable — combat / AI persistence block [R1 CdTableSaveStructures.cs] - -### CdTable (leaf, top of the 4th file-section): `ComplexArray cdt; CdPlayer cdplayer; CdAi[] cdai`. - -### CdPlayer (complex) — **entirely reverse-unlabeled** (fields named `unknown1..35`); shape is known: -`Int32 unknown1(=16); Bool unknown2; Float unknown3; Bool unknown4; Int32 unknown4p5; Bool unknown5..8;` -`Int32 unknown9,10; NonComplexArray unknown11;` -`Int32 unknown12..14; NonComplexArray unknown15,16; NonComplexArray unknown17;` -`NonComplexArray unknown18; Int32 unknown19..21;` -`NonComplexArray unknown22; Int32 unknown23..35`. - -### CdAi (complex) — per-AI-player planner state, ordered: -`AttributeSaveStruct aiAttr; NestedInt32 aiTurnPris; CdAiSit aiSit; NestedInt32 aiPlyHat;` -`ComplexArray prs2; Int32 dsh, nbStab, nmBlst; CdAiAidng aidng;` -`Int32 aiHivJ, sdFlT; NestedInt32 nalat; Int32 lnat, lat;` -`NonComplexArray aiSys; NonComplexArray cmbR; NonComplexArray cl;` -`NonComplexArray prv; NonComplexArray tecs; Int32 fct;` -`ComplexArray apr`. -- `CdAiSit` (complex): `NestedInt32 aiSitSecs; NestedInt32 aiSitWepFams` (**weapon-family set** lives here — opaque ints). -- `CdAiCmbr` (complex): `Int32 crTrnK; Bool crPce; NestedInt32 crSys; CdAiCmbrCrplSv2 crplSv2`. - `...CrplSv2`: `Int32 rpBon, rpBonT, savBonus; Bool maintHf; ComplexArray tacReports`. - `TacReport`: `TrStruct trBy, trTo; TacReportDamage damageStruct; NonComplexArray ships`. - `TrStruct`: `Int32 treHd, treHi, treD, treDp, treDi, treDt, treB`. - `TacReportDamage` (leaf): `Int32 tRid, tRal, tRbal, tRlas, tRmis, tRmin, tRnrg, tRbio, tRbrd; Bool tRsld, tRsldd, tRsldc, tRsldi, tRsldr` - — **damage-by-weapon-family breakdown**: bal(listic)/las(er)/mis(sile)/min(e)/nrg(=energy)/bio/brd(=boarding); sld=shields. - `TacReportShips` (leaf): `Int32 tRships, tRsldr, tRshipL`. - ---- - -## 13. Coverage gaps & contradictions - -**Coverage (what's decoded):** -- R1 decodes essentially the *entire* file top-to-bottom: summary, create-params/map-gen, full sim - (players, tech, designs, systems/colonies incl. per-player fog-of-war snapshots, fleets, ships, - trade, node grid, combat reports, grand-menace/encounter objects) and the CdTable AI block. This is - the most complete community model and the primary Rosetta source. -- R2 decodes only Summary, Player Settings, Players, Species, and Planets (colony) fields, by tag - search — but adds *friendly semantics* and confirms the value-bytes width of several planet fields. - -**Known-unknown fields (labelled `unknown*` in R1 — do NOT treat names as authoritative):** -- All of `CdPlayer` (`unknown1..35`) and `CdAiAidngDnId`, `SimFleetOrigin`, `SimFleetDetailsFlightPlanFpogn2`, - `SimTradeSectorTradeCtr`, `SimPlayerDesignDwgWgb`, `SimSystemDetailNvoIndcl*` bodies. -- `SimFleetShipHealth` three floats guessed as command/mission/drive. -- `PlanetSaveStruct.unknown1..4` (map-gen) unexplained. - -**Not decoded / thin:** -- Tactical/real-time combat geometry: only *reports/summaries* are stored (SimCrep*, CdAiCmbr TacReport). - Per-ship in-battle positions/velocities are not in these editors (likely not in the sim save at all). -- `SimSystemDetailSpy` body is empty in R1 (marked "needs to be populated"); spy detail unresolved. -- RNG state (`RngSaveStruct.unknownData`) is an opaque ~2500-byte blob. -- R1 source comments flag `SimSystemDetailSpiesArray`, `SimSvSctObXscnXsc` and - `SimScSctObEncObjDetails` as incomplete/"wrong" in places — verify encounter bodies against binary. - -**Contradictions between R1 and R2 (reconcile against binary):** -1. **Field widths on planet flags:** R2 reads `Abdn`, `Dstyd`, `ltis` as Int16 and `Bats2` as Int32; - R1 models `abdn/ltis` as Int32 and `bats2` as Int64. → The datum is small; check the true stored - width in the binary struct. -2. **PID vs OID:** R2 asserts `OID = PID*16` (owner id is player index << 4); R1 stores a single `pid` - owner field and does not model the ×16 relationship. → Confirm whether the binary owner field is a - raw index or a shifted/tagged handle. -3. **`_NPC` species id 4:** R2 names it `_NPC`; R1 comment guesses "AI Rebellion". Same numeric id 4, - different label — likely a shared "non-player/rogue" species slot. -4. R2 treats Players and PlayerSettings as separate flat tab regions bounded by marker strings - (`HomeSys`…`ISsp`, `Slot`…`Session`); R1 shows these are actually nested (settings inside the - Summary PlayerSlot, live player data inside SimPlayerDetails). R2's region boundaries - (`Summary`,`Slot`,`Session`,`HomeSys`,`ISsp`,`NumSys`,`NdGr2`,`PlayerIDs`,`DesignIDs`) are useful - **section-marker strings** to locate blocks in the raw binary. - -**High-value binary-scan signatures:** -`0xBEEFBEEF` / `0x41104110` complex-struct brackets; length-prefixed ASCII field-name tags -(`[int32 len][name]`) preceding every named scalar; section marker strings above. diff --git a/save-reader/ref/struct-recovery.md b/save-reader/ref/struct-recovery.md deleted file mode 100644 index f7d5cb9..0000000 --- a/save-reader/ref/struct-recovery.md +++ /dev/null @@ -1,461 +0,0 @@ -# Struct recovery via save-field-name xrefs — Sword of the Stars (2006) - -Program `sots` / "Sword of the Stars.exe", ImageBase 0x00400000, 32-bit MSVC. All addresses are VAs. -Method: every on-disk field is tagged with its name string; the name strings live in `.rdata`; the -functions that reference dozens of a struct's names in sequence are its `IStreamable::Read`/`Write`. -Decompiled those, mapped `this+offset` → name → type. Scripts (on CT111 `/root/`): `SerFind.java` -(string→xref→function ranking), `SerDump.java`/`SerDump2.java` (decompile with `DAT_` → literal -substitution), `VtOwner.java`/`VtOwner2.java` (find the owning vftable + RTTI Complete-Object-Locator -offset), `SubWrite.java` (sub-struct Read/Write by class name). Raw decompiles: CT111 `/tmp/serdump/*.c`. - -Reference cross-checked: `save-editor-structs.md` (R1 = Bardez editor, R2 = SOTSedit). - ---- - -## 0. Serialization runtime facts (needed to read the tables) - -### IStreamable vftable shape -Every streamable class has a 3-slot vftable `{ [0] scalar-deleting dtor, [1] Read(Stream&), [2] Write(Stream&) }` -(`Mars::IStreamable::vftable` @ 0x009e22bc = `{0x4f7230, 0x924fb0, 0x924fb0}`). `Mars::StreamableHelper` / -`Mars::VectorHelper` are thin adaptors: slot [1]/[2] call the object's virtual Read/Write (or, for -POD types like Vector3, a free function). - -### this-adjustment (IMPORTANT for offsets) -The serializer is called through the class's **IStreamable sub-vftable**, whose RTTI COL `offset` field -gives the sub-object offset. The vftable slots point straight at the functions (no adjustor thunks; -prologues verified in the exe: `55 8b ec 6a ff 68 …`), so inside each function `this` = object + COL -offset. **Absolute member offset = decompiled offset + COL offset.** Tables below give both. - -| Class | IStreamable vftable (COL offset) | Read | Write | Primary vftable | -|---|---|---|---|---| -| `Game::ServerSystem` (: StarSystem : StarMapNode) | 0x00a2043c (**+8**) | `FUN_0075d4b0` | `FUN_00749630` | (StarSystem primary 0x00a200e4) | -| `Game::StarSystem` / `ClientSystem` (StarMapNode part) | 0x00a200d4 / 0x00a20144 (+8) | `FUN_00727790` | `FUN_00727820` | 0x00a200e4 / 0x00a20154 | -| `Game::StarMapNode` | 0x00a1e620 (+8) | `FUN_00727790` | `FUN_00727820` | 0x00a1e630 | -| `Game::ServerPlayer` (: StrategyPlayer) | 0x00a32794 (**+0x3a0 = 928**) | `FUN_008804d0` | `FUN_008563e0` | 0x00a327a4 (COL 0, 8 slots) | -| `Game::StarShip` | 0x00a31408 (**+8**) | `FUN_00853fa0` | `FUN_008291f0` | 0x00a31418 (2 slots) | -| `Game::StarFleet` (: StarMapNode) | 0x00a1d5f8 (**+8**) | `FUN_00702470` | `FUN_00701070` | 0x00a1d608 | -| `Game::StrategyServer` (whole sim block) | 0x00a26084 (+0) | `FUN_007d27a0` | `FUN_0079fa70` | 0x00a26034 (COL 4) | -| `Game::StarSystem::PlayerView` | 0x00a201ac (+0) | `FUN_00752af0` | `FUN_007492d0` | — | -| `Game::StarSystem::OutputRates` (POD, via helper) | helper 0x00a1f884 | `FUN_007472a0` | `FUN_00745190` | — | -| `Game::Population` | 0x009f90f0 (+0) | `FUN_005390c0` | `FUN_00537ef0` | — | -| `Game::PopulationGroup` | 0x009f8d50 (+0) | `FUN_00536a80` | `FUN_00536af0` | — | -| `Game::IndependenceInfo` | 0x00a2005c (+0) | `FUN_00748df0` | `FUN_00748ee0` | — | -| `Game::Morale` / `MoraleEvent` | 0x00a1f7c8 / 0x00a2003c | `FUN_00744dd0` / `FUN_007490b0` | `FUN_00744ea0` / `FUN_007491b0` | — | -| `Game::ShipBuildOrder(Def)` | 0x00a0c160 / 0x00a0ad08 | `FUN_00813770` | `FUN_00813800` | — | -| `Game::PlayerNotes` | 0x00a21948 | `FUN_00813250` | `FUN_008132b0` | — | -| `Game::SpyReport` | 0x00a32b2c | `FUN_008843d0` | `FUN_00828ec0` | — | -| `Game::PlayerReport` (preps) | 0x00a21440 | `FUN_008200a0` | `FUN_00817480` | — | -| `Game::DiplomacyStats` | 0x00a21430 | — | `FUN_00818cb0` | — | -| `Game::FlightPlan` / `::Waypoint` / `NodeRoute` | 0x00a1d50c / 0x00a1d39c / 0x00a1cbdc | `FUN_00704c70` / `FUN_00701860` / `FUN_006e2260` | `FUN_00700f60` / `FUN_00700ed0` / `FUN_006e22e0` | — | -| `Game::PrisonerHold` | 0x009fe130 | `FUN_0056eb00` | `FUN_0056ec00` | — | -| `Game::EventStorage` / `PlayerAlliances` / `ShipHealth` / `PlayerColorID` / `Mars::Vector3` | — | — | `FUN_00825cc0` / `FUN_006d2e10` / `FUN_00813e50` / `FUN_0053c080` / `FUN_008a60d0` | — | - -### Stream primitive API (writer side; `Stream` object vftable, `this` = stream) -| call | meaning | wrapper used by serializers | -|---|---|---| -| vft+0x18 `(name, std::string*)` | write string | `FUN_008b9d70(stream,name,std::string*)` | -| vft+0x1c `(name, byte)` | write bool | `FUN_008b9c20(stream,name,bool*)` | -| vft+0x20 `(name, float)` | write float | `FUN_008b9be0(stream,name,float*)` | -| vft+0x24 `(name, int, default=-1)` | write int32 | `FUN_008b9d50(stream,name,int*)`; `FUN_008b9d00(stream,name,int16*)` (widens short→int) | -| vft+0x28 `(name, IStreamable-helper*)` | write nested object (BEEFBEEF frame) | inline `StreamableHelper{vft, 0, T*}` | -| vft+0x30 `(name, ptr, nbytes)` | write raw bytes | used for 8-byte Int64s | -| `FUN_00816490(stream,name,obj*)` | write **handle id** = `obj ? obj->id(+4) : 0` | NetworkObject id at +4 | -| `FUN_008b9c60(stream,name,int64*)` | write int64 (PopC) | | -Reader side mirrors: `FUN_008b9bc0` float, `FUN_008b9d20` int, `FUN_008b9c00` bool, `FUN_008b9d90` string, -`FUN_008b9c40` int64, `FUN_008b9cd0` short, `FUN_008164d0(stream,name)` handle→object* lookup, -stream vft+0x10 int-by-ref (returns found flag), vft+0x14 nested object (NULL helper = skip/legacy). -Readers accept legacy tags (`ISuit`, `Income`, `HPop`, `Bats`, `Builds`, `Clr`, `SensMod`, `ExPopSys`, -`NShps`, `SysID`, `TrdID`, `Caps`, `GtTrf`, `FtSens`, `FtInc`, `Pris`, `NumPlgs`, `lcid`, `morev`, `cme`) -by reading them into scratch/NULL — these are pre-1.8 fields, NOT members. - -Common Mars/MSVC layouts seen: `std::string` = 0x1c bytes (MSVC10 `_Bx` union@0, size@0x14, res@0x18; -`FUN_008b9d70` does the `res>=16 ? heap : sso` check); `std::vector` = {begin@0, end@4, cap@8}; -`std::map/set` node = {left@0, parent@4, right@8, key@0xc, value@0x10, …, color/isnil bytes at tail}; -`std::list` = {head*@0, size@4}. `Mars::NetworkObject` = {vptr@0, int id@4}. - ---- - -## 1. `Game::ServerSystem` (= live star system + colony record; R1 `SimSystemDetailsSaveStruct`) - -Serializers: Write `FUN_00749630` @ 0x00749630 (3453 B), Read `FUN_0075d4b0` @ 0x0075d4b0 (8320 B, has -legacy branches). Both begin with `StarMapNode::Write/Read` (`FUN_00727820`/`FUN_00727790`) which -emits `Pos`. `this` = obj+8. Base layout: `+0` primary vptr (StarSystem 0x00a200e4), `+4` NetworkObject id, -`+8` IStreamable vptr, `+0xc` HandleObject vptr, `+0x10` owner pointer (`*(+0x10)->+0x50` = player-object -table indexed by map key), `+0x18` Pos. - -| abs off | rel(this+8) | type | save name | notes | -|---|---|---|---|---| -| 0x18 | 0x10 | `Mars::Vector3` (3 floats) | `Pos` | via StarMapNode; on disk 3 unnamed floats | -| 0x4c | 0x44 | float | `R` | starColor.r | -| 0x50 | 0x48 | float | `G` | | -| 0x54 | 0x4c | float | `B` | | -| 0x58 | 0x50 | float | `A` | | -| 0x5c | 0x54 | int | `Idx` | system index | -| 0x60 | 0x58 | int | `Size` | 1–10 | -| 0x64 | 0x5c | float | `Suit` | climate hazard (legacy `ISuit` discarded) | -| 0x68 | 0x60 | int | `Res` | | -| 0x6c | 0x64 | int | `ARes2` | (legacy `ARes` read then overwritten) | -| 0x70 | 0x68 | int | `MRes` | | -| 0x74 | 0x6c | int | `TRes` | | -| 0x78..0x7a | 0x70..0x72 | bool[3] | `haltv` | written as `haltc`=3, then 3×(`haltt`=i, `haltv`=v[i]) | -| 0x7c | 0x74 | float | `OutMod` | | -| 0x80 | 0x78 | int | `TAcq` | | -| 0x84 | 0x7c | int | `TFAcq` | | -| 0x88..0xa3 | 0x80 | `StarSystem::OutputRates` (0x1c) | `Rts` | see §1.1; nested object | -| 0xa4 | 0x9c | `BuildQueue*` | `BQ` | written only if owner (`PID`) non-null | -| 0xa8..0xc3 | 0xa0 | `std::string` | `Name` | | -| 0xc4 | 0xbc | bool | `Abdn` | **bool in memory** (R2 "short" is just the value byte; R1 Int32 is the framing) | -| 0xc5 | 0xbd | bool | `Dstyd` | | -| 0xc6 | 0xbe | bool | `vnh` | gate: if true → `vnd`,`vnex3`,`vnpex3` | -| 0xc7 | 0xbf | bool | `vnd` | | -| 0xc8 | 0xc0 | bool | `vnex3` | | -| 0xc9 | 0xc1 | bool | `vnpex3` | | -| 0xcc | 0xc4 | int | `VFlags` | written by value (int) | -| 0xd0 | 0xc8 | int | `EFlags` | | -| 0xd4 | 0xcc | int | `AFlags` | | -| 0xd8 | 0xd0 | int | `FFlags` | | -| 0xdc | 0xd4 | int | `GFlags` | | -| 0xe0 | 0xd8 | int | `MnRFlags` | | -| 0xe4 | 0xdc | int | `RfRFlags` | | -| 0xe8 | 0xe0 | int | `ClkFlags` | | -| 0xf0 | 0xe8 | **int64** | `Bats2` | raw 8 bytes (R1 correct; R2 Int32 reads low half). Legacy `Bats` int | -| 0xf8 | 0xf0 | **int64** | `rcex` | raw 8 bytes | -| 0x100 | 0xf8 | `ServerPlayer*` | `PID` | owner; written as handle id (`->+4`) | -| 0x104..0x117 | 0xfc | `Population` (0x14) | `dcs` | nested | -| 0x118 | 0x110 | float | `dsu` | | -| 0x11c..0x13b | 0x114 | `Morale` (0x20: vptr + int[7]) | `cm` | nested | -| 0x13c..0x147 | 0x134 | `vector` | `cme2` | VectorHelper | -| 0x14c..0x16b | 0x144 | `Morale` | `PvCM` | | -| 0x16c..0x177 | 0x164 | `vector` | `NumFlts` + n×`Flt` | ids via handle | -| 0x17c | 0x174 | float | `RepCur` | | -| 0x180 | 0x178 | float | `RepMax` | | -| 0x184 | 0x17c | int | `EggScio` | | -| 0x188 | 0x180 | bool | `NoRebAI` | | -| 0x189 | 0x181 | bool | `PvNoRebAI` | | -| 0x18c | 0x184 | int | `Pop` | imperial pop | -| 0x190 | 0x188 | float | `Infra` | | -| 0x194 | 0x18c | int | `pbon` | | -| 0x198 | 0x190 | float | `ibon` | | -| 0x19c | 0x194 | int | `TerrFl` | | -| 0x1a0..0x1b3 | 0x198 | `Population` | `Pop2` | civilian pop groups (R1 `popG`) | -| 0x1b4..0x1c7 | 0x1ac | `Population` | `pbon2` | | -| 0x1c8 | 0x1c0 | `IndependenceInfo*` | `hindi` + `indi` | `hindi` = ptr!=NULL | -| 0x1cc..0x1d7 | 0x1c4 | `vector` | `spies2` | VectorHelper | -| 0x1dc | 0x1d4 | int | `rbfl` | written by value | -| 0x1e0 | 0x1d8 | bool | `hsrg` | | -| 0x1e4..0x1ff | 0x1dc | int[7] | `nadct`,(`ads`=i,`adt`=v) | addiction table: count of non-zero entries then sparse (index,value) pairs | -| 0x200 | 0x1f8 | int | `PvPop` | previous-turn snapshot block | -| 0x204 | 0x1fc | float | `PvInfra` | | -| 0x208 | 0x200 | float | `PvSuit` | | -| 0x20c | 0x204 | int | `PvRes` | | -| 0x210 | 0x208 | int | `PvARes2` | | -| 0x214 | 0x20c | int | `PvMRes` | | -| 0x218..0x22b | 0x210 | `Population` | `PvPop2` | | -| 0x238 | 0x230 | `StarFleet*` | `DefF` | handle id | -| 0x23c | 0x234 | `StarFleet*` | `DefSF` | handle id | -| 0x240..0x24b | 0x238 | `vector` | `NumGFs` + n×`GF` | gates | -| 0x250..0x25b | 0x248 | `vector` | `NumSnF` + n×`SnF` | stations | -| 0x260..0x26b | 0x258 | `vector` | `NumMnF` + n×`MnF` | monitors | -| 0x274 / 0x278 | 0x26c / 0x270 | `std::map` head / size | `NVO` + entries | colonies, see §1.2 | -| 0x284 / 0x288 | 0x27c / 0x280 | `std::map` head / size | `NVE` + entries | §1.2 | -| 0x294 / 0x298 | 0x28c / 0x290 | `std::map` head / size | `NVs` + entries | per-player `pview`, §1.3 | -| 0x2a8..0x2b3 | 0x2a0 | `vector` | `NumPlgs2` + n×(`PlgT`=plg->+4, `Plg` obj) | | -| 0x2b8 | 0x2b0 | int | `TnsOH` | | -| 0x2bc | 0x2b4 | int | `TDst` | | -| 0x2c4 | 0x2bc | int | `ntdev` | | -| 0x2c8 | 0x2c0 | int | `ltis` | **int** (R2 "short" wrong width) | -| 0x2cc | 0x2c4 | int | `rbtn` | | -| 0x2d0 | 0x2c8 | int | `rbfr` | by value | -| 0x2d4 | 0x2cc | int | `rbwn` | | - -Object size ≥ 0x2d8. On-disk order = R1 §8 exactly (Pos, RGBA, Idx, Size, Suit, Res, ARes2, MRes, NoRebAI, -TRes, Pop, Pop2, Infra, PvPop, PvPop2, PvInfra, PvSuit, PvRes, PvARes2, PvMRes, PvNoRebAI, Rts, Abdn, Dstyd, -TnsOH, OutMod, RepCur, RepMax, ntdev, pbon, pbon2, ibon, ltis, rbfl, rbtn, rbfr, rbwn, hsrg, halt*, vn*, Name, -*Flags, Bats2, rcex, Mn/Rf/ClkFlags, EggScio, TerrFl, TAcq, TFAcq, TDst, dcs, dsu, cm, PvCM, cme2, spies2, PID, -DefF, DefSF, BQ, nadct/ads/adt, NumPlgs2…, NumFlts/GFs/SnF/MnF, NVO, NVE, NVs, hindi/indi). - -Members that are only ever read with a NULL/scratch target (not stored): `ISuit`, `Income`, `HPop`, `Builds` -(+`Con`,`Sav`,`ConLeft`,`OrID`,`DesID` — old inline build queue), `Slvs`, `dct`, `cme`, `Bats`, `NumPlgs`. - -### 1.1 `Game::StarSystem::OutputRates` (POD, 0x1c) — Write `FUN_00745190` -Memory order ≠ disk order: `+0x00 float SRt`, `+0x04 SRsc`, `+0x08 SRtf`, `+0x0c SRi`, `+0x10 SRoh`, -`+0x14 SRs`, `+0x18 int SRnr`. Disk order: SRs, SRt, SRsc, SRtf, SRi, SRoh, SRnr. Reader: if `SRs` tag is -absent, reads 5 unnamed floats (legacy). - -### 1.2 Colony maps `NVO` / `NVE` (std::map keyed by player-table index) -Write emits `NVO`=size, then per node: `PID` = handle id of `owner->+0x50[key]` (player object table), -`TShn` = **int16** at node+0x12 (value+2), `OID` = int at node+0x14 (value+4) by value, `isind` bool at -node+0x18 (value+8), `indi` = inline `IndependenceInfo` at node+0x1c (value+0xc, 0x70 bytes). Node isnil -byte at +0x8d ⇒ value size 0x7c. `NVE` nodes: `EPid` handle (key→player), `ETS` int16 @ node+0x12, `Eid` -int @ node+0x14 (isnil @ +0x19 ⇒ value 8 bytes). -`OID` is a stored int, distinct from `PID`; the R2 claim "OID = PID×16" is an id-allocation pattern, not -a derivation in this code (open question — check the HandleObject id allocator). - -### 1.3 `Game::StarSystem::PlayerView` (per-player seen snapshot) — Write `FUN_007492d0`, Read `FUN_00752af0` -Stored inline as map value at node+0x10 (`NVs`; node isnil @ +0xad ⇒ value ≈ 0x9c). -`+0 vptr (0x00a201ac)`, `+8 int VTrn`, `+0xc int Pop`, `+0x10 Population Pop2 (0x14)`, `+0x24 float Infra`, -`+0x28 float Suit`, `+0x2c int Res`, `+0x30 int ARes2`, `+0x34 int MRes`, `+0x38 bool NoRebAI`, `+0x3c int pbon`, -`+0x40 Population pbon2`, `+0x54 float ibon`, `+0x58 int TerrFl`; trailer bool `footer`=1. Reader also accepts -legacy `ARes`, `PvPop/PvInfra/PvSuit/PvRes/PvARes/PvARes2/PvMRes/PvNoRebAI` into the same slots. -(R1 lists `Int32 infra` — it is a **float**.) - -### 1.4 `Game::Population` (0x14) / `Game::PopulationGroup` (0x18 stride) -Population: `+0 vptr`, `+4/+8/+0xc vector`; Write emits `PopNG` = count of groups with -PopC>0 (or ≥0 with low word ≠0), then each as nested `PopG`. PopulationGroup: `+4 int PopT`, `+8 int PopS`, -`+0x10 int64 PopC` (R1 `popT,popS,popC` ✓). - -### 1.5 `Game::IndependenceInfo` (0x70) — Write `FUN_00748ee0` -`+4 int indsp`, `+8 PlayerColorID indcl` (nested), `+0x1c string indnm`, `+0x38 string indav`, `+0x54 string indba`. - -### 1.6 `Game::Morale` / `Game::MoraleEvent` -Morale (0x20): `+0 vptr`, `+4 int[7]`; disk: `mnsp`=n then n×(`msp`=index, `mv`=value) (reader tolerates -missing `mnsp` → 7 fixed entries, skipping index 4). MoraleEvent: `+4 mid`, `+8 mtr`, `+0xc mn`, `+0x10 mtp` -(ints), `+0x14 Morale mfx`, `+0x34 string mdsc`. - -### 1.7 `Game::ShipBuildOrder` (build-queue entry) — Write `FUN_00813800` -`+4 int desID`, `+8 int con`, `+0xc int sav`, `+0x10 int conleft`, `+0x14 int ordID`; disk order desID, con, -conleft, sav, ordID (R1 ✓). - -### Where is "Planet"? -`Game::Planet : Actor` (vft 0x009ef144) is a **render/scene actor** and is not streamed. The colony/planet -state the save calls "planet" (R2's Idx/Name/Size/Suit/Res/Infra/Pop/OID…) is entirely in `ServerSystem` -above plus `PlayerView`. The CreateParameters `PlanetSaveStruct` (x,y,z + 4 ints) is map-gen input -(`StarMapParams`), not touched here. - ---- - -## 2. `Game::ServerPlayer` (empire; R1 `SimPlayerDetailsSaveStruct`) - -Serializers: Write `FUN_008563e0` @ 0x008563e0 (4040 B), Read `FUN_008804d0` @ 0x008804d0 (7647 B). -IStreamable sub-object at **+0x3a0** (COL offset 928); `this` = obj+0x3a0, so decompiled offsets are -negative for most members. Primary vptr @+0 (0x00a327a4, StrategyPlayer shape), NetworkObject id @+4. - -| abs off | rel(this+0x3a0) | type | save name | notes | -|---|---|---|---|---| -| 0x28 | -0x378 | int | `PlyrIdx` | | -| 0x2c | -0x374 | `ServerSystem*` | `HomeSys` | handle id | -| 0x30/0x34 | -0x370/-0x36c | `vector` | `NumOwn` + n×`OwnId` | handle ids (R1 `ownerIds`) | -| 0x40..0x5b | -0x360 | `std::string` | `PlryName` | | -| 0x5c | -0x344 | int | `Species` | 0 Human … 6 Morrigi | -| 0x60 | -0x340 | `PlayerColorID` (4 B) | `ClrID` | nested; §2.1 (legacy `Clr` int skipped) | -| 0x74..0x8f | -0x32c | `std::string` | `Bdg` | badge | -| 0x90..0xab | -0x310 | `std::string` | `Avt` | avatar | -| 0xac | -0x2f4 | int | `Team` | | -| 0xb0 | -0x2f0 | float | `IdealSuit` | | -| 0xb4 | -0x2ec | float | `SuitTol` | | -| 0xb8 | -0x2e8 | float | `MaxOH` | | -| 0xbc | -0x2e4 | float | `ResRate` | | -| 0xc0 | -0x2e0 | float | `ResMod` | | -| 0xc4 | -0x2dc | float | `ResScl` | | -| 0xd0 | -0x2d0 | **float** | `TRM` | R1 says Int32 — it is float | -| 0xd4 | -0x2cc | int | `TRA` | note memory order TRA before TRP | -| 0xd8 | -0x2c8 | int | `TRP` | | -| 0xe4/0xe8 | -0x2bc/-0x2b8 | `vector` | `NumDes` + n×(`DesID`=d->+0xa4, `Des` obj) | current designs | -| 0xf4 | -0x2ac | `TechTree*` | `TechTree` | first field on disk | -| 0xf8 | -0x2a8 | bool | `Elim` | | -| 0xfb | -0x2a5 | bool | `NPC` | | -| 0xfc | -0x2a4 | bool | `RebAI` | | -| 0xfd | -0x2a3 | bool | `ReqCL` | | -| 0xfe | -0x2a2 | bool | `AIBn` | | -| 0xff | -0x2a1 | bool | `CnTrd` | | -| 0x100 | -0x2a0 | bool | `CnRad` | | -| 0x101 | -0x29f | bool | `CnVItl` | | -| 0x102 | -0x29e | bool | `hgs` | | -| 0x103 | -0x29d | bool | `hadvs` | | -| 0x104 | -0x29c | bool | `harcc` | | -| 0x108 | -0x298 | float | `pddm` | | -| 0x10c..0x117 | -0x294 | float[3] | `ConMod` ×3 | interleaved on disk as 3×(ConMod[i], SavMod[i]) | -| 0x118..0x123 | -0x288 | float[3] | `SavMod` ×3 | | -| 0x124 | -0x27c | float | `OutMod` | | -| 0x128 | -0x278 | float | `RebOutMod` | | -| 0x12c | -0x274 | float | `ScOutMod` | | -| 0x130 | -0x270 | float | `PopMod` | (legacy `SensMod` float, `ExPopSys` int skipped between IncMod/PopMod/TerraMod) | -| 0x134 | -0x26c | float | `TerraMod` | | -| 0x138 | -0x268 | bool | `AMine` | | -| 0x13c | -0x264 | float | `MinPure` | | -| 0x140 | -0x260 | float | `MinRate` | | -| 0x144 | -0x25c | int | `NGts` | | -| 0x148 | -0x258 | int | `PrGtTrf` | | -| 0x14c | -0x254 | int | `GTraf` | | -| 0x150 | -0x250 | **float** | `CstR` | R1 Int32 → float | -| 0x154 | -0x24c | **float** | `CstE` | | -| 0x158 | -0x248 | **float** | `CstT` | | -| 0x15c | -0x244 | int | `Maint` | (legacy `NShps` skipped) | -| 0x160 | -0x240 | **float** | `shrm` | | -| 0x164 | -0x23c | int | `Status` | by value | -| 0x168..0x177 | -0x238 | `PlayerAlliances` {int ALid, AL, NA, CF} | `Team` (2nd) | nested (R1 `teamStruct`) | -| 0x178/0x17c | -0x228/-0x224 | `vector` | `NumLeg` + n×(`DesID`,`Des`) | legacy/drone designs (R1 `droneDesigns`) | -| 0x188 | -0x218 | int | `PvSav` | | -| 0x18c | -0x214 | bool | `PvMA` | | -| 0x19c | -0x204 | int | `HasDisc` | by value | -| 0x1a0 | -0x200 | int | `HasDiscSp` | | -| 0x1a4 | -0x1fc | int | `HasDiscCl` | | -| 0x1a8 | -0x1f8 | int | `HasEnc` | | -| 0x1ac | -0x1f4 | int | `HasEng` | | -| 0x1b0 | -0x1f0 | `ShipRecords` (inline) | `ShipRecs` | Write `FUN_008176a0` | -| 0x1f4 | -0x1ac | `vector` | `Ojvs` | VectorHelper | -| 0x204/0x208 | -0x19c/-0x198 | `vector<{int xid,xmin,xmax; float xper}>` (16 B) | `Nexp` + n×(`xid`,`xmin`,`xmax`,`xper`) | R1 misses the per-entry body | -| 0x214/0x218 | -0x18c/-0x188 | `vector` | `NWeapXcl` + n×`WeapXcl` | | -| 0x230 | -0x170 | `vector` | `dipstats` | §2.2 | -| 0x240 | -0x160 | `CommMessageContainer*` | `comms` | | -| 0x244 | -0x15c | `vector` | `preps` | §2.3 | -| 0x254 | -0x14c | `vector` | `odes` | | -| 0x264 | -0x13c | `vector` | `owep` | | -| 0x274 | -0x12c | `vector` | `otch` | | -| 0x284 | -0x11c | int | `Sav` | savings | -| 0x288 | -0x118 | int | `HasImm` | by value | -| 0x28c | -0x114 | int | `HasVac` | | -| 0x290 | -0x110 | int | `NPTrk` | | -| 0x294 | -0x10c | `Tech*` (current research) | `ResTNm` | writes `tech ? tech->name(+4) : ""` | -| 0x298 | -0x108 | `FleetNameGenerator*` | `FNG` | | -| 0x29c | -0x104 | `EventStorage` (inline) | `Events` | {`EvNxID`@+0x14, `Events` vector@+4} | -| 0x2b8/0x2bc | -0xe8/-0xe4 | `std::list` head/size | `NumNotes` + n×`Nts` | node value at +0x10: `NtSys`@+4,`NtTxt` str@+8,`NtTrn`@+0x24 | -| 0x2c4 | -0xdc | int | `BnkWrn` | by value | -| 0x2c8 | -0xd8 | int | `BnkTrn` | | -| 0x2cc | -0xd4 | int | `BnkEl` | | -| 0x2d0 | -0xd0 | int | `BnkPr` | | -| 0x2d8 | -0xc8 | int | `plcy` | by value | -| 0x2dc..0x2f7 | -0xc4 | **`std::string`** | `pswd` | R1 says Int32 — it is a string | -| 0x2f8 | -0xa8 | bool | `Srn` | | -| 0x2fc | -0xa4 | `obj*` | `SrnTo` | handle id (R1 `srcTo`) | -| 0x300 | -0xa0 | int | `lboid` | | -| 0x304 | -0x9c | int | `lcid2` | by value (legacy `lcid`) | -| 0x30c | -0x94 | float | `IncMod` | | -| 0x310 | -0x90 | `vector` | `aid` | | -| 0x320/0x324 | -0x80/-0x7c | `vector` | `ndeflay` + n×`deflay` | | -| 0x330 | -0x70 | bool | `cdp` | | -| 0x334 | -0x6c | `SpyReport*` | `spy2` | §2.4 | -| 0x338/0x33c | -0x68/-0x64 | `vector` (0x20 stride) | `rdtc` + n×`rdt` | | -| 0x368 | -0x38 | int | `aidf` | by value | -| 0x370 | -0x30 | `CivilianRatios` (inline) | `civr` | Write `FUN_0082c740` | -| 0x39c | -4 | int | `tnc` | written as max(v,1) | -| **0x3a0** | 0 | vptr | — | IStreamable sub-vftable 0x00a32794 | -| 0x3a4/0x3a8 | +4/+8 | `vector<{float PRm; int PRBt}>` | `NumPR` + n×(`PRm`,`PRBt`) | | -| 0x3b4 | +0x14 | bool | `ResErrRoll` | | -| 0x3b5 | +0x15 | bool | `cta` | | -| 0x3b8 | +0x18 | `AIRebellion*` | `HasAIR` + `AIR` | gate = ptr!=NULL | -| 0x3bc | +0x1c | `AIEncounterFlags*` | `AIEnf` | | -| 0x3c0/0x3c4 | +0x20/+0x24 | `vector` | `NSprj` + n×(`SprjT`=p->+0x3c, `Sprj`) | | -| 0x3d0 | +0x30 | int | `NextPrjID` | | -| 0x3d4 | +0x34 | int | `lret` | | -| 0x3dc | +0x3c | int | `nmeid` | | - -Object size ≥ 0x3e0. Disk order = R1 §6 (TechTree, HomeSys, PlyrIdx, PlryName, Species, ClrID, Bdg, Avt, -Team, Sav, IdealSuit, SuitTol, MaxOH, ResRate, ResMod, ResScl, TRM, TRP, TRA, OutMod, RebOutMod, ScOutMod, -IncMod, PopMod, TerraMod, AMine, MinPure, MinRate, NGts, PrGtTrf, GTraf, CstR/E/T, Maint, shrm, Status, Elim, -NPC, RebAI, ReqCL, Team{ALid,AL,NA,CF}, HasVac, HasImm, NPTrk, HasDisc, HasDiscSp, HasDiscCl, HasEnc, HasEng, -Events, FNG, PvSav, PvMA, AIBn, CnTrd, CnRad, hgs, hadvs, harcc, CnVItl, pddm, BnkWrn/Trn/Pr/El, ShipRecs, -NextPrjID, plcy, pswd, lret, nmeid, cdp, spy2, civr, aidf, Srn, SrnTo, lboid, lcid2, ResTNm, ResErrRoll, -3×(ConMod,SavMod), NumOwn/OwnId, NumDes/DesID/Des, NumLeg/DesID/Des, NumNotes/Nts, NumPR/PRm/PRBt, HasAIR/AIR, -cta, AIEnf, NSprj/SprjT/Sprj, Nexp/xid/xmin/xmax/xper, NWeapXcl/WeapXcl, Ojvs, dipstats, comms, preps, odes, -owep, otch, aid, ndeflay/deflay, rdtc/rdt, tnc). Note: on-disk `Sav` comes right after `Team` though it -lives at 0x284 in memory. - -### 2.1 `Game::PlayerColorID` (4 bytes) — Write `FUN_0053c080` -`+0 int8 index; +1,+2,+3 uint8 r,g,b`; writer emits index via `FUN_008b9cb0` (char→int on disk), and -**iff index == -1** the three r,g,b bytes (matches R1 §1.6). Same struct used by `IndependenceInfo.indcl`. - -### 2.2 `Game::DiplomacyStats` (0x24) — Write `FUN_00818cb0` -`+4 int other`; then **int16** fields at +8 `lastnap`, +0xa `lastnapbty`, +0xc `bknnap`, +0xe `btynap`, -+0x10 `lastally`, +0x12 `lastallybty`, +0x14 `bknally`, +0x16 `btyally`, +0x18 `lastcf`, +0x1a `lastcfbty`, -+0x1c `bkncf`, +0x1e `btycf`, +0x20 `deadhome` (all widened to int32 on disk; R1 `nap/ally/cf{last_,last_bty,bkn_,bty_}` ✓). - -### 2.3 `Game::PlayerReport` (preps, 0x30) — Write `FUN_00817480` -ints `+4 oid, +8 pid, +0xc flds, +0x10 sav, +0x14 home, +0x18 ncol, +0x1c mpwr, +0x20 mcls, +0x24 mmsl, +0x28 nshp, +0x2c nsat` (R1 ✓). - -### 2.4 `Game::SpyReport` — Write `FUN_00828ec0` -Four `std::list`s: `+4 list` (count `defc2`@+8, items `def`), `+0x10 list` -(`rtc`@+0x14, `strd`), `+0x1c list` (`evc`@+0x20, `evs`), `+0x28 list` -(`ttc`@+0x2c, `tt`). R1 only kept the four counts. - -### 2.5 `Game::TechTree` — Write `FUN_005890a0` (tags `NumTechs`, `TNm`, `NumBrs`; per-tech body in a -sub-writer not decompiled here; logs "TechTree: Tech %d not found saving tech tree"). - ---- - -## 3. `Game::StarFleet` (R1 `SimFleetDetails`) — Write `FUN_00701070`, Read `FUN_00702470`; `this` = obj+8 - -| abs | rel | type | save name | notes | -|---|---|---|---|---| -| 0x18 | 0x10 | Vector3 | `Pos` | via StarMapNode | -| 0x4c | 0x44 | Vector3 | `PrvPos` | | -| 0x58 | 0x50 | `ServerPlayer*` | `PID` | handle | -| 0x5c..0x77 | 0x54 | string | `FtName` | | -| 0x78 | 0x70 | bool | `Perm` | | -| 0x7c | 0x74 | `FleetLayout` (inline, ~0x24) | `HLay` + `Lay` | gate = either of its two vectors (+4/+8, +0x14/+0x18) non-empty | -| 0xa0 | 0x98 | `StarSystem*` | `LocID` | handle (legacy `SysID`,`TrdID` ints skipped) | -| 0xa4/0xa8 | 0x9c/0xa0 | `vector` | `NShips` + n×(`ShipID` handle, `Ship` obj) | | -| 0xc4..0xfb | 0xbc | `FlightPlan` (inline, 0x38) | `HFPlan` + `FPlan` | gate = wpts non-empty; §3.1 | -| 0xfc | 0xf4 | int | `FtTrans` | by value (legacy `Caps` int, `GtTrf` short, `FtSens` float, `FtInc` int skipped) | -| 0x100 | 0xf8 | Vector3 | `FtOrig` | (R1 "3 ints" → 3 floats) | -| 0x10c | 0x104 | int | `FtFlg` | | -| 0x110 | 0x108 | int | `Ftae` | | -| 0x114 | 0x10c | int | `Ftpae` | | -| 0x118 | 0x110 | int | `FtEnc` | | -| 0x11c | 0x114 | int | `FtMS` | | - -### 3.1 `Game::FlightPlan` (0x38) / `Waypoint` / `NodeRoute` -FlightPlan: `+0 vptr`, `+4 vector wpts`, `+0x14 float FPsp2`, `+0x18 int FPeta2`, `+0x1c Vector3 FPogn2`, -`+0x28 Vector3 FPdpos`, `+0x34 int pnd`. Waypoint: `+4 int Wpt`, `+8 int Tp`, `+0xc NodeRoute nrt`. -NodeRoute: `+4 nrp`, `+8 nrf`, `+0xc nrt` (ints). Reader also handles legacy `NumWpt`/`path`/`FPognid`. - -## 4. `Game::StarShip` (R1 `SimFleetShipDetails`) — Write `FUN_008291f0`, Read `FUN_00853fa0`; `this` = obj+8 - -| abs | rel | type | save name | notes | -|---|---|---|---|---| -| 0x10 | 0x8 | `ServerPlayer*` | `PlrID` | handle | -| 0x14 | 0xc | `ShipDesign*` | `DesID` | writes `design->+0xa4` (design id) | -| 0x20 | 0x18 | float | `Range` | | -| 0x24..0x33 | 0x1c | `ShipHealth` {vptr; float[3]} | `Health` | 3 unnamed floats (R1 guess command/mission/drive) | -| 0x34 | 0x2c | int | `MineCap` | | -| 0x38/0x3c | 0x30/0x34 | `vector<{float th, thm}>` | `NTH` + n×(`TH`,`THM`) | | -| 0x48 | 0x40 | int | `Plg` | | -| 0x4c | 0x44 | int | `Act` | | -| 0x50 | 0x48 | bool | `Dep` | | -| 0x51 | 0x49 | bool | `Atq` | | -| 0x5c | 0x54 | int | `LCT` | | -| 0x60 | 0x58 | int | `tsd` | | -| 0x64 | 0x5c | `StarFleet*` | `FltID` | handle | -| 0x68 | 0x60 | int | `ConCap` | | -| 0x6c | 0x64 | **float** | `RefCap` | R1 Int32 → float | -| 0x70 | 0x68 | **float** | `RepCap` | R1 Int32 → float | -| 0x7c | 0x74 | int | `EncID` | | -| 0x80 | 0x78 | `PrisonerHold` (inline) | `PrisH` | `+0x14 int*` → `[0]=PrMax`, `[2..8]` per-species counts; disk `PrMax`,`PrNSp`,(`PrSp`=idx,`PrNum`) | -| 0x98 | 0x90 | `BuildQueue*` | `hbq` + `BQ2` | gate = ptr!=NULL | -| 0x9c | 0x94 | `Population*` | `hsp` + `pop` | gate = ptr!=NULL | -| 0xa0 | 0x98 | `Population*` | `ppop` | | -| 0xa8 | 0xa0 | int | `atsp` | by value | -| 0xac | 0xa4 | int | `tblt` | | - ---- - -## 5. Top-level sim block — `Game::StrategyServer` Write `FUN_0079fa70` / Read `FUN_007d27a0` -Tag order confirms R1 §5: KeyPath, NMSz, NMLc, NMnx, ModCount, Frame, GameID, [AIDifficultyID legacy], -Attrib, RNG, GameName, Map, IncMod, ResMod, EnAl, EnTm, GOTurn, GOWinPly, NPCm/o/i/v/a, szadj, rsadj, suadj, -sprjs, RandEncAdj, cmbtid, turnstats, numcreps/crep, ninv/invs/inve/invt/invtb, AllExc×6, AllExcCF, -AllExcCFp×2, NumPlrs/PlayerID/Player, ISsp, ISsu, NumSys/SysID/Sys, NdGr2, trdmgr, spymgr, NumFlts/FltID/Flt, -NumActs/Act, SvSctOb, zdsc, zdsi, zdst. (Offsets not tabulated — out of scope this round.) - ---- - -## 6. Corrections to the community reference (R1/R2) -- Types: `TRM`, `CstR/E/T`, `shrm`, ship `RefCap/RepCap`, PlayerView `Infra` are **floats**; `pswd` is a - **string**; `PvMA`, `AIBn` are bools; `FtOrig` is a Vector3 (3 floats). -- Widths: `Bats2`/`rcex` are true int64 (R2 int is wrong); `Abdn`/`Dstyd` are bools, `ltis` an int - (R2 "short" is an artefact of reading value bytes); `TShn`, `ETS`, all DiplomacyStats counters are - int16 in memory but int32 on disk. -- Missing in R1: `Nexp` entries carry (`xid`,`xmin`,`xmax`,`xper`); SpyReport lists have bodies; Morale - is a sparse (msp,mv) table; `nadct` is followed by sparse (`ads`,`adt`) pairs over a 7-int table. -- `Plg`/`Act`/`EncID`/`OID`/flags are written by value → plain ints (not handles). - -## 7. Confidence & open questions -- **High**: all offsets/types in §1–§4 (direct from Write functions; Read functions agree on every - member address; COL offsets from RTTI; no adjustor thunks). -- **Medium**: nested struct sizes inferred from neighbouring offsets (Population 0x14, Morale 0x20, - FlightPlan 0x38, PlayerView ≈0x9c, IndependenceInfo 0x70); which members belong to `StarSystem` vs - `ServerSystem` (serializer is `ServerSystem`'s; `ClientSystem` shares the StarSystem vftables and only - streams `Pos`). -- **Open**: `FUN_008b9cb0` exact on-disk width for `PlayerColorID` (R1 says int32 — plausible); - `ServerSystem+0x10` owner type (StrategyServer? its `+0x50` is a player-object table); `OID` - allocation (R2's ×16); per-tech body of `TechTree::Write`; `CdPlayer` block not attempted (names are - R1's `unknownN`, nothing to xref). No types were written back into Ghidra (notes only). diff --git a/save-reader/SAVE_FORMAT.md b/verify/save-reader/SAVE_FORMAT.md similarity index 100% rename from save-reader/SAVE_FORMAT.md rename to verify/save-reader/SAVE_FORMAT.md diff --git a/save-reader/__pycache__/save_reader.cpython-310.pyc b/verify/save-reader/__pycache__/save_reader.cpython-310.pyc similarity index 100% rename from save-reader/__pycache__/save_reader.cpython-310.pyc rename to verify/save-reader/__pycache__/save_reader.cpython-310.pyc diff --git a/save-reader/__pycache__/save_writer_stub.cpython-310.pyc b/verify/save-reader/__pycache__/save_writer_stub.cpython-310.pyc similarity index 100% rename from save-reader/__pycache__/save_writer_stub.cpython-310.pyc rename to verify/save-reader/__pycache__/save_writer_stub.cpython-310.pyc diff --git a/save-reader/__pycache__/test_save_reader.cpython-310.pyc b/verify/save-reader/__pycache__/test_save_reader.cpython-310.pyc similarity index 100% rename from save-reader/__pycache__/test_save_reader.cpython-310.pyc rename to verify/save-reader/__pycache__/test_save_reader.cpython-310.pyc diff --git a/save-reader/fixture_joint.sav b/verify/save-reader/fixture_joint.sav similarity index 100% rename from save-reader/fixture_joint.sav rename to verify/save-reader/fixture_joint.sav diff --git a/save-reader/fixture_split.sav b/verify/save-reader/fixture_split.sav similarity index 100% rename from save-reader/fixture_split.sav rename to verify/save-reader/fixture_split.sav diff --git a/save-reader/save_reader.py b/verify/save-reader/save_reader.py similarity index 100% rename from save-reader/save_reader.py rename to verify/save-reader/save_reader.py diff --git a/save-reader/save_writer_stub.py b/verify/save-reader/save_writer_stub.py similarity index 100% rename from save-reader/save_writer_stub.py rename to verify/save-reader/save_writer_stub.py diff --git a/save-reader/test_save_reader.py b/verify/save-reader/test_save_reader.py similarity index 100% rename from save-reader/test_save_reader.py rename to verify/save-reader/test_save_reader.py