Shipwright/ZAPDTR
Christopher Leggett 7b08f98b8c
Dual OTR MQ and Vanilla Support (#1694)
* Changes OTR Extraction to have specific mq and nonmq paths.

Also updates the game to load resources according to whether or not
Master Quest or Vanilla is loaded.

* Removes unneeded code from the last commit.

* Fixes some weird formatting in ZRom.c

* Loads oot-mq.otr and patches oot.otr on top, if both are present.

If only one or the other are present, it becomes the only and main OTR.

* Adds ImGui Logic for whether or an MQ Checkbox.

Checkbox checked only specifies whether new saves should be MQ or not.
Checkbox is disabled or force-enabled according to which OTRs are loaded.
Also as a necessity includes tracking what game versions have been loaded
from the OTRs.

* Adds MQ settings logic for Randomizer's ImGui menu.

* Writes Master Quest dungeons to the spoiler log

* Loads MQ Dungeons from spoiler, persists in save, and loads when appropriate.

* Adds logic to prevent loading or creating incompatible rando saves.

* Fixdes some linux build issues and new rando save issues

* Makes appimage create both vanilla and mq otrs

If either rom is present, it makes the corresponding OTR. If both are present,
it will make both. If one OTR is present but both roms are present, it will
create the missing OTR.

* Makes it so a randomized save file will not be marked as MQ.

* Refactors to load all OTRs from MainPath or a specific list.

Also adds the ability to take a std::unordered_set of hashes to
validate each OTR's version file against.

* Fixes a syntax error

* Makes ExtractAssets output Vanilla and MQ OTRs if both roms are present

* Fixes asset generation bug.

* Partially working fix for dual OTR extract_assets

Currently the cmake ExtractAssets target will return with a 1 if you
only end up exporting one type of OTR isntead of both. Haven't found
a great way to only attempt to copy a file if it exists from within
cmake. It does actually correctly copy the OTR that is generated,
despite the error from copying the other one.

Pushing as is for now but will keep investigating.

* Adds oot-mq.otr to the gitignore.

* Makes ExtractAssets not fail on only one rom/OTR.

* Removes PatchesPath from the constructors requiring OTRFiles vector.

* Renames OOT_UNKNOWN to just UNKNOWN to remove OOT specific reference.

* Removes randomizing MQ Dungeons and re-disables MQ rando.

Doing this so the PR can get merged quicker with just the Dual OTR
support and won't need to wait on rando logic to be updated. That
will happen in another PR directly after the merge.

* Update mac startup script for dual otr

* Update soh/macosx/soh-macos.sh

* Update soh/macosx/soh-macos.sh

* Update soh/macosx/soh-macos.sh

* Implements new BinaryReader to fix Linux build issue.

BinaryReader itself comes from https://github.com/Moneyl/BinaryTools
I added a wrapper to adapt it to the ABI from ZAPD's Binary Reader and
add Endianness checking. I also had to copy a handful of other bits and
pieces from ZAPD to make it all function as expected.

* A few edits to the updatream BinaryReader to compile it on Linux.

* Adds the Endianness to the first byte of the version file.

* Fixes Jenkins

* Addresses some of Kenix's comments

* Renames `ReadNullTerminatedString` to `ReadCString`

* Refactors Archive::LoadFile into a private method with more arguments.

* Removes BitConverter and extends existing endianness.h instead.

* Fixes an endianness issue with the version file.

Co-authored-by: Garrett Cox <garrettjcox@gmail.com>
2022-10-16 23:07:35 -04:00
..
ExporterTest Linux/GCC Support (#28) 2022-05-11 13:18:24 -04:00
ZAPD Dual OTR MQ and Vanilla Support (#1694) 2022-10-16 23:07:35 -04:00
ZAPDUtils LUS Cleanup: Merge Window and GlobalCtx2 classes. (#1259) 2022-08-21 21:08:01 -04:00
docs Update XMLs from upstream (#1448) 2022-09-18 23:29:24 -04:00
lib Add Support for macOS (#441) 2022-06-22 14:59:21 -04:00
.clang-format git subrepo clone https://github.com/HarbourMasters/ZAPDTR.git 2022-03-22 02:54:48 +01:00
.gitignore git subrepo clone https://github.com/HarbourMasters/ZAPDTR.git 2022-03-22 02:54:48 +01:00
.gitrepo git subrepo clone https://github.com/HarbourMasters/ZAPDTR.git 2022-03-22 02:54:48 +01:00
Jenkinsfile git subrepo clone https://github.com/HarbourMasters/ZAPDTR.git 2022-03-22 02:54:48 +01:00
LICENSE git subrepo clone https://github.com/HarbourMasters/ZAPDTR.git 2022-03-22 02:54:48 +01:00
README.md git subrepo clone https://github.com/HarbourMasters/ZAPDTR.git 2022-03-22 02:54:48 +01:00
copycheck.py git subrepo clone https://github.com/HarbourMasters/ZAPDTR.git 2022-03-22 02:54:48 +01:00

README.md

ZAPD: Zelda Asset Processor for Decomp

Compiling

Dependencies

ZAPD needs a compiler with C++17 support.

ZAPD has the following library dependencies:

  • libpng

In a Debian/Ubuntu based environment, those could be installed with the following command:

sudo apt install libpng-dev

On a Mac, you will need to install libpng with Homebrew or MacPorts; we currently only support Homebrew. You can run

brew install libpng

to install it via Homebrew.

Building

Linux / *nix

ZAPD uses the clasic Makefile approach. To build just run make (or even better make -j for faster compilations).

You can configure a bit your ZAPD build with the following options:

  • OPTIMIZATION_ON: If set to 0 optimizations will be disabled (compile with -O0). Any other value compiles with -O2. Defaults to 1.
  • ASAN: If it is set to a non-zero then ZAPD will be compiled with Address Sanitizer enabled (-fsanitize=address). Defaults to 0.
  • DEPRECATION_ON: If it is set to a zero then deprecation warnings will be disabled. Defaults to 1.
  • DEBUG: If non-zero, ZAPD will be compiled in development mode. This implies the following:
    • Debugging symbols enabled (-g3). They are disabled by default.
    • OPTIMIZATION_ON=0: Disables optimizations (-O0).
    • DEPRECATION_ON=0: Disables deprecation warnings.
  • LLD=1: builds with the LLVM linker ld.lld instead of the system default.

As an example, if you want to build ZAPD with optimizations disabled and use the address sanitizer, you could use the following command:

make -j OPTIMIZATION_ON=0 ASAN=1

Windows

This repository contains vcxproj files for compiling under Visual Studio environments. See ZAPD/ZAPD.vcxproj.

Invoking ZAPD

ZAPD needs a File parsing mode to be passed as first parameter. The options are:

  • e: "Extraction" mode.
    • In this mode, ZAPD expects a XML file as input, a folder as ouput and a path to the baserom files.
    • ZAPD will read the XML and use it as a guide to extract the contents of the specified asset file from the baserom folder.
  • bsf: "Build source file" mode.
    • This is an experimental mode.
    • It was going to be used to let you have XMLs that aren't just for extraction. Might get used, might not. Still need to experiment on that.
  • btex: "Build texture" mode.
    • In this mode, ZAPD expects a PNG file as input, a filename as ouput and a texture type parameter (-tt).
    • ZAPD will try to convert the given PNG into the contents of a uint64_t C array.
  • bren: "Build (render) background" mode.
    • In this mode, ZAPD expects a JPG file as input and a filename as ouput.
    • ZAPD will try to convert the given JPG into the contents of a uint64_t C array.
  • blb: "Build blob" mode.
    • In this mode, ZAPD expects a BIN file as input and a filename as ouput.
    • ZAPD will try to convert the given BIN into the contents of a uint8_t C array.
  • bovl: "Build overlay" mode.
    • In this mode, ZAPD expects an overlay C file as input, a filename as ouput and an overlay configuration path (-cfg).
    • ZAPD will generate a reloc .s file.

ZAPD also accepts the following list of extra parameters:

  • -i PATH / --inputpath PATH: Set input path.
  • -o PATH / --outputpath PATH: Set output path.
  • -b PATH / --baserompath: Set baserom path.
    • Can be used only in e or bsf modes.
  • -osf PATH: Set source output path. This is the path where the .c and .h files will be extracted to. If omitted, it will use the value passed to --outputpath parameter.
  • -gsf MODE: Generate source file during extraction. If MODE is 1, C source files will be generated.
    • Can be used only in e mode.
  • -crc / --output-crc: Outputs a CRC file for each extracted texture.
    • Can be used only in e or bsf modes.
  • -ulzdl MODE: Use "Legacy ZDisplayList" instead of libgfxd. Set MODE to 1 to enable it.
    • Can be used only in e or bsf modes.
  • -profile MODE: Enable profiling. Set MODE to 1 to enable it.
  • -uer MODE: Split resources into their individual components (enabled by default). Set MODE to non-1 to disable it.
  • -tt TYPE: Set texture type.
    • Can be used only in mode btex.
    • Valid values:
      • rgba32
      • rgb5a1
      • i4
      • i8
      • ia4
      • ia8
      • ia16
      • ci4
      • ci8
  • -cfg PATH: Set cfg path (for overlays).
    • Can be used only in bovl mode.
  • -rconf PATH Read Config File.
  • -eh: Enable error handler.
    • Only available in non-Windows environments.
  • -v MODE: Enable verbosity. Currently there are 3 possible values:
    • 0: Default. Completely silent (except for warnings and errors).
    • 1: Information.
    • 2 (and higher): Debug.
  • -wu / --warn-unaccounted: Enable warnings for each unaccounted block of data found.
    • Can be used only in e or bsf modes.
  • -vu / --verbose-unaccounted: Changes how unaccounteds are outputted. Max 4 bytes per line (a word) and add a comment with the offset of each of those lines.
    • Could be useful for looking at raw data or testing.
    • Can be used only in e or bsf modes.
  • -tm MODE: Test Mode (enables certain experimental features). To enable it, set MODE to 1.
  • -se / --set-exporter : Sets which exporter to use.
  • --gcc-compat : Enables GCC compatibly mode. Slower.
  • -us / --unaccounted-static : Mark unaccounted data as static
  • -s / --static : Mark every asset as static.
    • This behaviour can be overridden per asset using Static= in the respective XML node.
  • -W...: warning flags, see below

Additionally, you can pass the flag --version to see the current ZAPD version. If that flag is passed, ZAPD will ignore any other parameter passed.

Warning flags

ZAPD contains a variety of warning types, with similar syntax to GCC or Clang's compiler warnings. Warnings can have three levels:

  • Off (does not display anything)
  • Warn (print a warning but continue processing)
  • Err (behave like an error, i.e. print and throw an exception to crash ZAPD when occurs)

Each warning type uses one of these by default, but can be modified with flags, similarly to GCC or Clang:

  • -Wfoo enables warnings of type foo
  • -Wno-foo disables warnings of type foo
  • -Werror=foo escalates foo to behave like an error
  • -Weverything enables all warnings (they may be turned off using -Wno- flags afterwards)
  • -Werror escalates all enabled warnings to errors

All warning types currently implemented, with their default levels:

Warning type Default level Description
-Wdeprecated Warn Deprecated features
-Whardcoded-pointer Warn ZAPD lacks the info to make a symbol, so must output a hardcoded pointer
-Wintersection Warn Two assets intersect
-Winvalid-attribute-value Err Attribute declared in XML is wrong
-Winvalid-extracted-data Err Extracted data does not have correct form
-Winvalid-jpeg Err JPEG file does not conform to the game's format requirements
-Winvalid-png Err Issues arising when processing PNG data
-Winvalid-xml Err XML has syntax errors
-Wmissing-attribute Warn Required attribute missing in XML tag
-Wmissing-offsets Warn Offset attribute missing in XML tag
-Wmissing-segment Warn Segment not given in File tag in XML
-Wnot-implemented Warn ZAPD does not currently support this feature
-Wunaccounted Off Large blocks of unaccounted
-Wunknown-attribute Warn Unknown attribute in XML entry tag

There are also errors that do not have a type, and cannot be disabled.

For example, here we have invoked ZAPD in the usual way to extract using a (rather badly-written) XML, but escalating -Wintersection to an error:

ZAPD warnings example