Dual OTR MQ and Vanilla Support (#1694)

* Changes OTR Extraction to have specific mq and nonmq paths.

Also updates the game to load resources according to whether or not
Master Quest or Vanilla is loaded.

* Removes unneeded code from the last commit.

* Fixes some weird formatting in ZRom.c

* Loads oot-mq.otr and patches oot.otr on top, if both are present.

If only one or the other are present, it becomes the only and main OTR.

* Adds ImGui Logic for whether or an MQ Checkbox.

Checkbox checked only specifies whether new saves should be MQ or not.
Checkbox is disabled or force-enabled according to which OTRs are loaded.
Also as a necessity includes tracking what game versions have been loaded
from the OTRs.

* Adds MQ settings logic for Randomizer's ImGui menu.

* Writes Master Quest dungeons to the spoiler log

* Loads MQ Dungeons from spoiler, persists in save, and loads when appropriate.

* Adds logic to prevent loading or creating incompatible rando saves.

* Fixdes some linux build issues and new rando save issues

* Makes appimage create both vanilla and mq otrs

If either rom is present, it makes the corresponding OTR. If both are present,
it will make both. If one OTR is present but both roms are present, it will
create the missing OTR.

* Makes it so a randomized save file will not be marked as MQ.

* Refactors to load all OTRs from MainPath or a specific list.

Also adds the ability to take a std::unordered_set of hashes to
validate each OTR's version file against.

* Fixes a syntax error

* Makes ExtractAssets output Vanilla and MQ OTRs if both roms are present

* Fixes asset generation bug.

* Partially working fix for dual OTR extract_assets

Currently the cmake ExtractAssets target will return with a 1 if you
only end up exporting one type of OTR isntead of both. Haven't found
a great way to only attempt to copy a file if it exists from within
cmake. It does actually correctly copy the OTR that is generated,
despite the error from copying the other one.

Pushing as is for now but will keep investigating.

* Adds oot-mq.otr to the gitignore.

* Makes ExtractAssets not fail on only one rom/OTR.

* Removes PatchesPath from the constructors requiring OTRFiles vector.

* Renames OOT_UNKNOWN to just UNKNOWN to remove OOT specific reference.

* Removes randomizing MQ Dungeons and re-disables MQ rando.

Doing this so the PR can get merged quicker with just the Dual OTR
support and won't need to wait on rando logic to be updated. That
will happen in another PR directly after the merge.

* Update mac startup script for dual otr

* Update soh/macosx/soh-macos.sh

* Update soh/macosx/soh-macos.sh

* Update soh/macosx/soh-macos.sh

* Implements new BinaryReader to fix Linux build issue.

BinaryReader itself comes from https://github.com/Moneyl/BinaryTools
I added a wrapper to adapt it to the ABI from ZAPD's Binary Reader and
add Endianness checking. I also had to copy a handful of other bits and
pieces from ZAPD to make it all function as expected.

* A few edits to the updatream BinaryReader to compile it on Linux.

* Adds the Endianness to the first byte of the version file.

* Fixes Jenkins

* Addresses some of Kenix's comments

* Renames `ReadNullTerminatedString` to `ReadCString`

* Refactors Archive::LoadFile into a private method with more arguments.

* Removes BitConverter and extends existing endianness.h instead.

* Fixes an endianness issue with the version file.

Co-authored-by: Garrett Cox <garrettjcox@gmail.com>

.gitignore

@@ -410,6 +410,7 @@ ReleaseObj/*
 .tags
 tags
 oot.otr
+oot-mq.otr
 *.sav
 shipofharkinian.ini
 shipofharkinian.json

CMakeLists.txt

@@ -136,14 +136,13 @@ find_package(Python3 COMPONENTS Interpreter)
 add_custom_target(
     ExtractAssets
     # CMake versions prior to 3.17 do not have the rm command, use remove instead for older versions
-    COMMAND ${CMAKE_COMMAND} -E $<IF:$<VERSION_LESS:${CMAKE_VERSION},3.17>,remove,rm> -f oot.otr
-    COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/OTRExporter/extract_assets.py -z "$<TARGET_FILE:ZAPD>"
-    COMMAND ${CMAKE_COMMAND} -E copy oot.otr ${CMAKE_SOURCE_DIR}
-    COMMAND ${CMAKE_COMMAND} -E copy oot.otr ${CMAKE_BINARY_DIR}/soh
+    COMMAND ${CMAKE_COMMAND} -E $<IF:$<VERSION_LESS:${CMAKE_VERSION},3.17>,remove,rm> -f oot.otr oot-mq.otr
+    COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/OTRExporter/extract_assets.py -z "$<TARGET_FILE:ZAPD>" --non-interactive
+    COMMAND ${CMAKE_COMMAND} -DSOURCE_DIR=${CMAKE_CURRENT_SOURCE_DIR} -DBINARY_DIR=${CMAKE_BINARY_DIR} -P ${CMAKE_CURRENT_SOURCE_DIR}/copy-existing-otrs.cmake
     WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/OTRExporter
     COMMENT "Running asset extraction..."
     DEPENDS ZAPD
-    BYPRODUCTS oot.otr ${CMAKE_SOURCE_DIR}/oot.otr
+    BYPRODUCTS oot.otr ${CMAKE_SOURCE_DIR}/oot.otr oot-mq.otr ${CMAKE_SOURCE_DIR}/oot-mq.otr
 )
 
 if(CMAKE_SYSTEM_NAME MATCHES "Linux")

OTRExporter/OTRExporter/DisplayListExporter.cpp

@@ -882,8 +882,14 @@ std::string OTRExporter_DisplayList::GetPrefix(ZResource* res)
 	std::string prefix = "";
 	std::string xmlPath = StringHelper::Replace(res->parent->GetXmlFilePath().string(), "\\", "/");
 
-	if (StringHelper::Contains(oName, "_scene") || StringHelper::Contains(oName, "_room"))
+	if (StringHelper::Contains(oName, "_scene") || StringHelper::Contains(oName, "_room")) {
 		prefix = "scenes";
+        if (Globals::Instance->rom->IsMQ()) {
+            prefix += "/mq";
+        } else {
+            prefix += "/nonmq";
+        }
+    }
 	else if (StringHelper::Contains(xmlPath, "objects/"))
 		prefix = "objects";
 	else if (StringHelper::Contains(xmlPath, "textures/"))

OTRExporter/OTRExporter/Main.cpp

@@ -21,6 +21,7 @@
 #include <Utils/Directory.h>
 #include <Utils/MemoryStream.h>
 #include <Utils/BinaryWriter.h>
+#include <bit>
 
 std::string otrFileName = "oot.otr";
 std::shared_ptr<Ship::Archive> otrArchive;
@@ -69,13 +70,20 @@ static void ExporterProgramEnd()
 		std::string romPath = Globals::Instance->baseRomPath.string();
 		std::vector<uint8_t> romData = File::ReadAllBytes(romPath);
 		uint32_t crc = BitConverter::ToUInt32BE(romData, 0x10);
+        uint8_t endianness = (uint8_t)Endianness::Big;
 
 		// Write crc to version file
 		fs::path versionPath("Extract/version");
+        MemoryStream* versionStream = new MemoryStream();
+        BinaryWriter writer(versionStream);
+        writer.SetEndianness(Endianness::Big);
+        writer.Write(endianness);
+        writer.Write(crc);
 		std::ofstream versionFile(versionPath.c_str(), std::ios::out | std::ios::binary);
-		versionFile.write((char*)&crc, sizeof(crc));
+        versionFile.write(versionStream->ToVector().data(), versionStream->GetLength());
 		versionFile.flush();
 		versionFile.close();
+        writer.Close();
 
 		printf("Created version file.\n");
 

OTRExporter/extract_assets.py

@@ -16,7 +16,8 @@ def BuildOTR(xmlPath, rom, zapd_exe=None):
 
     exec_cmd = [zapd_exe, "ed", "-i", xmlPath, "-b", rom, "-fl", "CFG/filelists",
             "-o", "placeholder", "-osf", "placeholder", "-gsf", "1",
-            "-rconf", "CFG/Config.xml", "-se", "OTR"]
+            "-rconf", "CFG/Config.xml", "-se", "OTR", "--otrfile", 
+            "oot-mq.otr" if Z64Rom.isMqRom(rom) else "oot.otr"]
 
     print(exec_cmd)
     exitValue = subprocess.call(exec_cmd)
@@ -30,16 +31,18 @@ def main():
     parser = argparse.ArgumentParser()
     parser.add_argument("-z", "--zapd", help="Path to ZAPD executable", dest="zapd_exe", type=str)
     parser.add_argument("rom", help="Path to the rom", type=str, nargs="?")
+    parser.add_argument("--non-interactive", help="Runs the script non-interactively for use in build scripts.", dest="non_interactive", action="store_true")
 
     args = parser.parse_args()
 
-    rom_path = args.rom if args.rom else rom_chooser.chooseROM()
-    rom = Z64Rom(rom_path)
+    rom_paths = [ args.rom ] if args.rom else rom_chooser.chooseROM(args.non_interactive)
+    for rom_path in rom_paths:
+        rom = Z64Rom(rom_path)
 
-    if (os.path.exists("Extract")):
-        shutil.rmtree("Extract")
+        if (os.path.exists("Extract")):
+            shutil.rmtree("Extract")
 
-    BuildOTR("../soh/assets/xml/" + rom.version.xml_ver + "/", rom_path, zapd_exe=args.zapd_exe)
+        BuildOTR("../soh/assets/xml/" + rom.version.xml_ver + "/", rom_path, zapd_exe=args.zapd_exe)
 
 if __name__ == "__main__":
     main()

OTRExporter/rom_chooser.py

@@ -2,7 +2,7 @@ import os, sys, glob
 
 from rom_info import Z64Rom
 
-def chooseROM():
+def chooseROM(non_interactive=False):
     roms = []
 
     for file in glob.glob("*.z64"):
@@ -14,7 +14,21 @@ def chooseROM():
         sys.exit(1)
 
     if (len(roms) == 1):
-        return roms[0]
+        return roms
+
+    if non_interactive:
+        romsToExtract = []
+        foundMq = False
+        foundOot = False
+        for rom in roms:
+            isMq = Z64Rom.isMqRom(rom)
+            if isMq and not foundMq:
+                romsToExtract.append(rom)
+                foundMq = True
+            elif not isMq and not foundOot:
+                romsToExtract.append(rom)
+                foundOot = True
+        return romsToExtract
 
     print(str(len(roms))+ " roms found, please select one by pressing 1-"+str(len(roms)))
 
@@ -34,4 +48,4 @@ def chooseROM():
 
         else: break
 
-    return roms[selection - 1]
+    return [ roms[selection - 1] ]

OTRExporter/rom_info.py

@@ -73,6 +73,11 @@ class Z64Rom:
         if self.checksum == Checksums.OOT_UNKNOWN:
             self.is_valid = False
             return
+        
+        if self.checksum in [Checksums.OOT_NTSC_JP_MQ, Checksums.OOT_NTSC_US_MQ, Checksums.OOT_PAL_GC_MQ_DBG, Checksums.OOT_PAL_MQ]:
+            self.isMq = True
+        else:
+            self.isMq = False
 
         # get rom version
         self.version = ROM_INFO_TABLE[self.checksum]
@@ -86,3 +91,7 @@ class Z64Rom:
     @staticmethod
     def isValidRom(rom_path):
         return Z64Rom(rom_path).is_valid
+    
+    @staticmethod
+    def isMqRom(rom_path):
+        return Z64Rom(rom_path).isMq

ZAPDTR/ZAPD/ZResource.cpp

@@ -328,7 +328,7 @@ std::string ZResource::GetSourceOutputHeader([[maybe_unused]] const std::string&
 		std::string xmlPath = StringHelper::Replace(parent->GetXmlFilePath().string(), "\\", "/");
 
 		if (StringHelper::Contains(outName, "_room_") || StringHelper::Contains(outName, "_scene"))
-			prefix = "scenes";
+			prefix = "scenes/nonmq";
 		else if (StringHelper::Contains(xmlPath, "objects/"))
 			prefix = "objects";
 		else if (StringHelper::Contains(xmlPath, "textures/"))
@@ -342,8 +342,9 @@ std::string ZResource::GetSourceOutputHeader([[maybe_unused]] const std::string&
 		else if (StringHelper::Contains(xmlPath, "text/"))
 			prefix = "text";
 
-		if (prefix != "")
+		if (prefix != "") {
 			str += StringHelper::Sprintf("#define d%s \"__OTR__%s/%s/%s\"", name.c_str(), prefix.c_str(), outName.c_str(), nameStr.c_str());
+        }
 		else
 			str += StringHelper::Sprintf("#define d%s \"__OTR__%s/%s\"", name.c_str(), outName.c_str(), nameStr.c_str());
 

ZAPDTR/ZAPD/ZRom.cpp

@@ -64,7 +64,33 @@ namespace fs = std::filesystem;
 #define OOT_PAL_GC_MQ_DBG 0x917D18F6
 #define OOT_IQUE_TW 0x3D81FB3E
 #define OOT_IQUE_CN 0xB1E1E07B
-#define OOT_UNKNOWN 0xFFFFFFFF
+#define UNKNOWN 0xFFFFFFFF
+
+bool ZRom::IsMQ() {
+    int crc = BitConverter::ToInt32BE(romData, 0x10);
+    switch (crc) {
+        case OOT_NTSC_10:
+        case OOT_NTSC_11:
+        case OOT_NTSC_12:
+        case OOT_PAL_10:
+        case OOT_PAL_11:
+        case OOT_NTSC_JP_GC:
+        case OOT_NTSC_JP_GC_CE:
+        case OOT_NTSC_US_GC:
+        case OOT_PAL_GC:
+        case OOT_PAL_GC_DBG1:
+        case OOT_PAL_GC_DBG2:
+        case OOT_IQUE_CN:
+        case OOT_IQUE_TW:
+        default:
+            return false;
+        case OOT_NTSC_JP_MQ:
+        case OOT_NTSC_US_MQ:
+        case OOT_PAL_MQ:
+        case OOT_PAL_GC_MQ_DBG:
+            return true;
+    }
+}
 
 ZRom::ZRom(std::string romPath)
 {

ZAPDTR/ZAPD/ZRom.h

@@ -11,6 +11,7 @@ public:
 	ZRom(std::string romPath);
 
 	std::vector<uint8_t> GetFile(std::string fileName);
+    bool IsMQ();
 
 protected:
 	std::vector<uint8_t> romData;

copy-existing-otrs.cmake

@@ -0,0 +1,11 @@
+if(EXISTS ${SOURCE_DIR}/OTRExporter/oot.otr)
+    execute_process(COMMAND ${CMAKE_COMMAND} -E copy oot.otr ${SOURCE_DIR})
+    execute_process(COMMAND ${CMAKE_COMMAND} -E copy oot.otr ${BINARY_DIR}/soh/)
+endif()
+if(EXISTS ${SOURCE_DIR}/OTRExporter/oot-mq.otr)
+    execute_process(COMMAND ${CMAKE_COMMAND} -E copy oot-mq.otr ${SOURCE_DIR})
+    execute_process(COMMAND ${CMAKE_COMMAND} -E copy oot-mq.otr ${BINARY_DIR}/soh/)
+endif()
+if(NOT EXISTS ${SOURCE_DIR}/oot.otr AND NOT EXISTS ${SOURCE_DIR}/oot-mq.otr)
+    message(FATAL_ERROR, "No OTR files found.")
+endif()

libultraship/libultraship/Animation.cpp

@@ -48,7 +48,7 @@ void Ship::AnimationV0::ParseFileBinary(BinaryReader* reader, Resource* res)
 			data.unk_02 = reader->ReadInt16();
 			data.unk_04 = reader->ReadInt16();
 			data.unk_06 = reader->ReadInt16();
-			data.unk_08 = reader->ReadSingle();
+			data.unk_08 = reader->ReadFloat();
 
 			anim->transformDataArr.push_back(data);
 		}

libultraship/libultraship/Archive.cpp

@@ -1,27 +1,38 @@
 #include "Archive.h"
 #include "Resource.h"
 #include "File.h"
+#include "Window.h"
+#include "ResourceMgr.h"
 #include "spdlog/spdlog.h"
 #include "Utils/StringHelper.h"
 #include "Lib/StrHash64.h"
 #include <filesystem>
+#include "Lib/BinaryTools/BinaryTools/BinaryReader.h"
 
 #ifdef __SWITCH__
 #include "SwitchImpl.h"
 #endif
 
 namespace Ship {
-	Archive::Archive(const std::string& MainPath, bool enableWriting) : Archive(MainPath, "", enableWriting)
+	Archive::Archive(const std::string& MainPath, bool enableWriting) : Archive(MainPath, "", std::unordered_set<uint32_t>(), enableWriting)
 	{
 		mainMPQ = nullptr;
 	}
 
-	Archive::Archive(const std::string& MainPath, const std::string& PatchesPath, bool enableWriting, bool genCRCMap) : MainPath(MainPath), PatchesPath(PatchesPath) {
+	Archive::Archive(const std::string& MainPath, const std::string& PatchesPath, const std::unordered_set<uint32_t>& ValidHashes, bool enableWriting, bool genCRCMap)
+        : MainPath(MainPath), PatchesPath(PatchesPath), OTRFiles({}), ValidHashes(ValidHashes) {
 		mainMPQ = nullptr;
 		Load(enableWriting, genCRCMap);
 	}
 
-	Archive::~Archive() {
+    Archive::Archive(const std::vector<std::string> &OTRFiles, const std::unordered_set<uint32_t> &ValidHashes, bool enableWriting, bool genCRCMap)
+        : OTRFiles(OTRFiles), ValidHashes(ValidHashes)
+    {
+        mainMPQ = nullptr;
+        Load(enableWriting, genCRCMap);
+    }
+
+    Archive::~Archive() {
 		Unload();
 	}
 
@@ -55,48 +66,63 @@ namespace Ship {
 		}
 	}
 
-	std::shared_ptr<File> Archive::LoadFile(const std::string& filePath, bool includeParent, std::shared_ptr<File> FileToLoad) {
-		HANDLE fileHandle = NULL;
-
-		if (FileToLoad == nullptr) {
-			FileToLoad = std::make_shared<File>();
-			FileToLoad->path = filePath;
-		}
-
-		bool attempt = SFileOpenFileEx(mainMPQ, filePath.c_str(), 0, &fileHandle);
-
-		if (!attempt) {
-			SPDLOG_ERROR("({}) Failed to open file {} from mpq archive  {}.", GetLastError(), filePath.c_str(), MainPath.c_str());
-			std::unique_lock<std::mutex> Lock(FileToLoad->FileLoadMutex);
-			FileToLoad->bHasLoadError = true;
-			return FileToLoad;
-		}
-
-		DWORD dwFileSize = SFileGetFileSize(fileHandle, 0);
-		std::shared_ptr<char[]> fileData(new char[dwFileSize]);
-		DWORD dwBytes;
+    std::shared_ptr<File> Archive::LoadFileFromHandle(const std::string& filePath, bool includeParent, std::shared_ptr<File> FileToLoad, HANDLE mpqHandle)
+    {
+        HANDLE fileHandle = NULL;
+
+        if (FileToLoad == nullptr)
+        {
+            FileToLoad = std::make_shared<File>();
+            FileToLoad->path = filePath;
+        }
+
+        if (mpqHandle == nullptr)
+        {
+            mpqHandle = mainMPQ;
+        }
+
+        bool attempt = SFileOpenFileEx(mpqHandle, filePath.c_str(), 0, &fileHandle);
+
+        if (!attempt)
+        {
+            SPDLOG_ERROR("({}) Failed to open file {} from mpq archive  {}.", GetLastError(), filePath.c_str(), MainPath.c_str());
+            std::unique_lock<std::mutex> Lock(FileToLoad->FileLoadMutex);
+            FileToLoad->bHasLoadError = true;
+            return FileToLoad;
+        }
+
+        DWORD dwFileSize = SFileGetFileSize(fileHandle, 0);
+        std::shared_ptr<char[]> fileData(new char[dwFileSize]);
+        DWORD dwBytes;
+
+        if (!SFileReadFile(fileHandle, fileData.get(), dwFileSize, &dwBytes, NULL))
+        {
+            SPDLOG_ERROR("({}) Failed to read file {} from mpq archive {}", GetLastError(), filePath.c_str(), MainPath.c_str());
+            if (!SFileCloseFile(fileHandle))
+            {
+                SPDLOG_ERROR("({}) Failed to close file {} from mpq after read failure in archive {}", GetLastError(), filePath.c_str(), MainPath.c_str());
+            }
+            std::unique_lock<std::mutex> Lock(FileToLoad->FileLoadMutex);
+            FileToLoad->bHasLoadError = true;
+            return FileToLoad;
+        }
+
+        if (!SFileCloseFile(fileHandle))
+        {
+            SPDLOG_ERROR("({}) Failed to close file {} from mpq archive {}", GetLastError(), filePath.c_str(), MainPath.c_str());
+        }
+
+        std::unique_lock<std::mutex> Lock(FileToLoad->FileLoadMutex);
+        FileToLoad->parent = includeParent ? shared_from_this() : nullptr;
+        FileToLoad->buffer = fileData;
+        FileToLoad->dwBufferSize = dwFileSize;
+        FileToLoad->bIsLoaded = true;
+
+        return FileToLoad;
+    }
 
-		if (!SFileReadFile(fileHandle, fileData.get(), dwFileSize, &dwBytes, NULL)) {
-			SPDLOG_ERROR("({}) Failed to read file {} from mpq archive {}", GetLastError(), filePath.c_str(), MainPath.c_str());
-			if (!SFileCloseFile(fileHandle)) {
-				SPDLOG_ERROR("({}) Failed to close file {} from mpq after read failure in archive {}", GetLastError(), filePath.c_str(), MainPath.c_str());
-			}
-			std::unique_lock<std::mutex> Lock(FileToLoad->FileLoadMutex);
-			FileToLoad->bHasLoadError = true;
-			return FileToLoad;
-		}
-
-		if (!SFileCloseFile(fileHandle)) {
-			SPDLOG_ERROR("({}) Failed to close file {} from mpq archive {}", GetLastError(), filePath.c_str(), MainPath.c_str());
-		}
-
-		std::unique_lock<std::mutex> Lock(FileToLoad->FileLoadMutex);
-		FileToLoad->parent = includeParent ? shared_from_this() : nullptr;
-		FileToLoad->buffer = fileData;
-		FileToLoad->dwBufferSize = dwFileSize;
-		FileToLoad->bIsLoaded = true;
-
-		return FileToLoad;
+	std::shared_ptr<File> Archive::LoadFile(const std::string& filePath, bool includeParent, std::shared_ptr<File> FileToLoad) {
+		return LoadFileFromHandle(filePath, includeParent, FileToLoad, nullptr);
 	}
 
 	std::shared_ptr<File> Archive::LoadPatchFile(const std::string& filePath, bool includeParent, std::shared_ptr<File> FileToLoad) {
@@ -320,53 +346,129 @@ namespace Ship {
 		return true;
 	}
 
+    void Archive::GenerateCRCMap() {
+        auto listFile = LoadFile("(listfile)", false);
+
+        std::vector<std::string> lines = StringHelper::Split(std::string(listFile->buffer.get(), listFile->dwBufferSize), "\n");
+
+        for (size_t i = 0; i < lines.size(); i++)
+        {
+            std::string line = StringHelper::Replace(StringHelper::Strip(lines[i], "\r"), "/", "\\");
+            std::string line2 = StringHelper::Replace(line, "\\", "/");
+
+            uint64_t hash = CRC64(StringHelper::Replace(line, "/", "\\").c_str());
+            uint64_t hash2 = CRC64(StringHelper::Replace(line, "\\", "/").c_str());
+            hashes[hash] = line;
+            hashes[hash2] = line2;
+        }
+    }
+
+    bool Archive::PushGameVersion(HANDLE mpqHandle) {
+        auto t = LoadFileFromHandle("version", false, nullptr, mpqHandle);
+        if (!t->bHasLoadError)
+        {
+            BinaryReader *reader = new BinaryReader(t->buffer.get(), t->dwBufferSize);
+            Ship::Endianness endianness = (Ship::Endianness)reader->ReadUByte();
+            reader->SetEndianness(endianness);
+            uint32_t version = reader->ReadUInt32();
+            if (ValidHashes.empty() || ValidHashes.contains(version)) {
+                gameVersions.push_back(version);
+                return true;
+            }
+        }
+        return false;
+    }
+
 	bool Archive::LoadMainMPQ(bool enableWriting, bool genCRCMap) {
-		HANDLE mpqHandle = NULL;
+        HANDLE mpqHandle = NULL;
+        if (OTRFiles.empty()) {
+            if (MainPath.length() > 0) {
+                if (std::filesystem::is_directory(MainPath)) {
+                    for (const auto &p : std::filesystem::recursive_directory_iterator(MainPath)) {
+                        if (StringHelper::IEquals(p.path().extension().string(), ".otr")) {
+                            SPDLOG_ERROR("Reading {} mpq", p.path().string().c_str());
+                            OTRFiles.push_back(p.path().string());
+                        }
+                    }
+                } else if (std::filesystem::is_regular_file(MainPath)) {
+                    OTRFiles.push_back(MainPath);
+                } else {
+                    SPDLOG_ERROR("The directory {} does not exist", MainPath.c_str());
+                    return false;
+                }
+            } else {
+                SPDLOG_ERROR("No OTR file list or Main Path provided.");
+                return false;
+            }
+            if (OTRFiles.empty()) {
+                SPDLOG_ERROR("No OTR files present in {}", MainPath.c_str());
+                return false;
+            }
+        }
+        bool baseLoaded = false;
+        int i = 0;
+        while (!baseLoaded && i < OTRFiles.size()) {
 #ifdef _WIN32
-		std::wstring wfullPath = std::filesystem::absolute(MainPath).wstring();
+            std::wstring wfullPath = std::filesystem::absolute(OTRFiles[i]).wstring();
 #endif
 #if defined(__SWITCH__)
-		std::string fullPath = MainPath;
+            std::string fullPath = OTRFiles[0];
+#else  
+            std::string fullPath = std::filesystem::absolute(OTRFiles[i]).string();
+#endif
+#ifdef _WIN32
+            if (SFileOpenArchive(wfullPath.c_str(), 0, enableWriting ? 0 : MPQ_OPEN_READ_ONLY, &mpqHandle))
+            {
 #else
-		std::string fullPath = std::filesystem::absolute(MainPath).string();
+            if (SFileOpenArchive(fullPath.c_str(), 0, enableWriting ? 0 : MPQ_OPEN_READ_ONLY, &mpqHandle))
+            {
 #endif
-
+                SPDLOG_INFO("Opened mpq file {}.", fullPath.c_str());
+                mainMPQ = mpqHandle;
+                if (!PushGameVersion()) {
+                    SPDLOG_WARN("Attempted to load invalid OTR file {}", OTRFiles[i].c_str());
+                    SFileCloseArchive(mpqHandle);
+                    mainMPQ = nullptr;
+                } else {
+                    mpqHandles[fullPath] = mpqHandle;
+                    if (genCRCMap)
+                    {
+                        GenerateCRCMap();
+                    }
+                    baseLoaded = true;
+                }
+            }
+            i++;
+        }
+        // If we exited the above loop without setting baseLoaded to true, then we've
+        // attemtped to load all the OTRs available to us.
+        if (!baseLoaded) {
+            SPDLOG_ERROR("No valid OTR file was provided.");
+            return false;
+        }
+        for (int j = i; j < OTRFiles.size(); j++) {
 #ifdef _WIN32
-		if (!SFileOpenArchive(wfullPath.c_str(), 0, enableWriting ? 0 : MPQ_OPEN_READ_ONLY, &mpqHandle)) {
+            std::wstring wfullPath = std::filesystem::absolute(OTRFiles[i]).wstring();
+#endif
+#if defined(__SWITCH__)
+            std::string fullPath = OTRFiles[i];
 #else
-		if (!SFileOpenArchive(fullPath.c_str(), 0, enableWriting ? 0 : MPQ_OPEN_READ_ONLY, &mpqHandle)) {
+            std::string fullPath = std::filesystem::absolute(OTRFiles[i]).string();
 #endif
-
-	#ifdef __SWITCH__
-			Switch::ThrowMissingOTR(fullPath);
-	#endif
-			SPDLOG_ERROR("({}) Failed to open main mpq file {}.", GetLastError(), fullPath.c_str());
-			return false;
-		}
-
-		mpqHandles[fullPath] = mpqHandle;
-		mainMPQ = mpqHandle;
-
-		if (genCRCMap) {
-			auto listFile = LoadFile("(listfile)", false);
-
-			std::vector<std::string> lines = StringHelper::Split(std::string(listFile->buffer.get(), listFile->dwBufferSize), "\n");
-
-			for (size_t i = 0; i < lines.size(); i++) {
-				std::string line = StringHelper::Replace(StringHelper::Strip(lines[i], "\r"), "/", "\\");
-				std::string line2 = StringHelper::Replace(line, "\\", "/");
-
-				uint64_t hash = CRC64(StringHelper::Replace(line, "/", "\\").c_str());
-				uint64_t hash2 = CRC64(StringHelper::Replace(line, "\\", "/").c_str());
-				hashes[hash] = line;
-				hashes[hash2] = line2;
-			}
-		}
-
+            if (LoadPatchMPQ(fullPath, true))
+            {
+                SPDLOG_INFO("({}) Patched in mpq file.", fullPath.c_str());
+            }
+            if (genCRCMap)
+            {
+                GenerateCRCMap();
+            }
+        }
+        
 		return true;
 	}
 
-	bool Archive::LoadPatchMPQ(const std::string& path) {
+	bool Archive::LoadPatchMPQ(const std::string& path, bool validateVersion) {
 		HANDLE patchHandle = NULL;
 #if defined(__SWITCH__)
 		std::string fullPath = path;
@@ -386,7 +488,16 @@ namespace Ship {
 #endif
 			SPDLOG_ERROR("({}) Failed to open patch mpq file {} while applying to {}.", GetLastError(), path.c_str(), MainPath.c_str());
 			return false;
-		}
+		} else {
+            // We don't always want to validate the "version" file, only when we're loading standalone OTRs as patches
+            // i.e. Ocarina of Time along with Master Quest.
+            if (validateVersion) {
+                if (!PushGameVersion(patchHandle)) {
+                    SPDLOG_WARN("({}) Invalid MQP file.", path.c_str());
+                    return false;
+                }
+            }
+        }
 #ifdef _WIN32
 		if (!SFileOpenPatchArchive(mainMPQ, wPath.c_str(), "", 0)) {
 #else

libultraship/libultraship/Archive.h

@@ -9,6 +9,7 @@
 #include <unordered_map>
 #include <string>
 #include <vector>
+#include <unordered_set>
 #include "Resource.h"
 #include "StormLib.h"
 
@@ -21,8 +22,9 @@ namespace Ship
 	{
 	public:
 		Archive(const std::string& MainPath, bool enableWriting);
-		Archive(const std::string& MainPath, const std::string& PatchesPath, bool enableWriting, bool genCRCMap = true);
-		~Archive();
+        Archive(const std::string &MainPath, const std::string &PatchesPath, const std::unordered_set<uint32_t> &ValidHashes, bool enableWriting, bool genCRCMap = true);
+        Archive(const std::vector<std::string> &OTRFiles, const std::unordered_set<uint32_t> &ValidHashes, bool enableWriting, bool genCRCMap = true);
+        ~Archive();
 
 		bool IsMainMPQValid();
 
@@ -37,19 +39,25 @@ namespace Ship
 		std::vector<SFILE_FIND_DATA> ListFiles(const std::string& searchMask) const;
 		bool HasFile(const std::string& searchMask) const;
 		const std::string* HashToString(uint64_t hash) const;
+        std::vector<uint32_t> gameVersions;
 	protected:
 		bool Load(bool enableWriting, bool genCRCMap);
 		bool Unload();
 	private:
 		std::string MainPath;
 		std::string PatchesPath;
-		std::map<std::string, HANDLE> mpqHandles;
+        std::vector<std::string> OTRFiles;
+        std::unordered_set<uint32_t> ValidHashes;
+        std::map<std::string, HANDLE> mpqHandles;
 		std::vector<std::string> addedFiles;
 		std::unordered_map<uint64_t, std::string> hashes;
 		HANDLE mainMPQ;
 
 		bool LoadMainMPQ(bool enableWriting, bool genCRCMap);
 		bool LoadPatchMPQs();
-		bool LoadPatchMPQ(const std::string& path);
-	};
+		bool LoadPatchMPQ(const std::string& path, bool validateVersion = false);
+        void GenerateCRCMap();
+        bool PushGameVersion(HANDLE mpqHandle = nullptr);
+        std::shared_ptr<File> LoadFileFromHandle(const std::string &filePath, bool includeParent = true, std::shared_ptr<File> FileToLoad = nullptr, HANDLE mpqHandle = nullptr);
+    };
 }

libultraship/libultraship/Audio.cpp

@@ -31,10 +31,10 @@ namespace Ship
 
 		ResourceFile::ParseFileBinary(reader, res);
 
-		entry->codec = reader->ReadByte();
-		entry->medium = reader->ReadByte();
-		entry->unk_bit26 = reader->ReadByte();
-		entry->unk_bit25 = reader->ReadByte();
+		entry->codec = reader->ReadInt8();
+		entry->medium = reader->ReadInt8();
+		entry->unk_bit26 = reader->ReadInt8();
+		entry->unk_bit25 = reader->ReadInt8();
 
 		uint32_t dataSize = reader->ReadInt32();
 
@@ -66,8 +66,8 @@ namespace Ship
 		ResourceFile::ParseFileBinary(reader, res);
 
 		soundFont->id = reader->ReadInt32();
-		soundFont->medium = reader->ReadByte();
-		soundFont->cachePolicy = reader->ReadByte();
+		soundFont->medium = reader->ReadInt8();
+		soundFont->cachePolicy = reader->ReadInt8();
 		soundFont->data1 = reader->ReadInt16();
 		soundFont->data2 = reader->ReadInt16();
 		soundFont->data3 = reader->ReadInt16();
@@ -85,9 +85,9 @@ namespace Ship
 
 			drum.env = ReadEnvelopeData(reader);
 
-			bool hasSample = reader->ReadByte();
+			bool hasSample = reader->ReadInt8();
 			drum.sampleFileName = reader->ReadString();
-			drum.tuning = reader->ReadSingle();
+			drum.tuning = reader->ReadFloat();
 
 			soundFont->drums.push_back(drum);
 		}
@@ -105,38 +105,38 @@ namespace Ship
 			entry.env = ReadEnvelopeData(reader);
 
 			{
-				bool hasSFEntry = reader->ReadByte();
+				bool hasSFEntry = reader->ReadInt8();
 
 				if (hasSFEntry)
 				{
 					entry.lowNotesSound = new SoundFontEntry();
-					bool hasSampleRef = reader->ReadByte();
+					bool hasSampleRef = reader->ReadInt8();
 					entry.lowNotesSound->sampleFileName = reader->ReadString();
-					entry.lowNotesSound->tuning = reader->ReadSingle();
+					entry.lowNotesSound->tuning = reader->ReadFloat();
 				}
 			}
 
 			{
-				bool hasSFEntry = reader->ReadByte();
+				bool hasSFEntry = reader->ReadInt8();
 
 				if (hasSFEntry)
 				{
 					entry.normalNotesSound = new SoundFontEntry();
-					bool hasSampleRef = reader->ReadByte();
+					bool hasSampleRef = reader->ReadInt8();
 					entry.normalNotesSound->sampleFileName = reader->ReadString();
-					entry.normalNotesSound->tuning = reader->ReadSingle();
+					entry.normalNotesSound->tuning = reader->ReadFloat();
 				}
 			}
 
 			{
-				bool hasSFEntry = reader->ReadByte();
+				bool hasSFEntry = reader->ReadInt8();
 
 				if (hasSFEntry)
 				{
 					entry.highNotesSound = new SoundFontEntry();
-					bool hasSampleRef = reader->ReadByte();
+					bool hasSampleRef = reader->ReadInt8();
 					entry.highNotesSound->sampleFileName = reader->ReadString();
-					entry.highNotesSound->tuning = reader->ReadSingle();
+					entry.highNotesSound->tuning = reader->ReadFloat();
 				}
 			}
 
@@ -147,13 +147,13 @@ namespace Ship
 		{
 			SoundFontEntry* entry = new SoundFontEntry();
 
-			bool hasSFEntry = reader->ReadByte();
+			bool hasSFEntry = reader->ReadInt8();
 
 			if (hasSFEntry)
 			{
-				bool hasSampleRef = reader->ReadByte();
+				bool hasSampleRef = reader->ReadInt8();
 				entry->sampleFileName = reader->ReadString();
-				entry->tuning = reader->ReadSingle();
+				entry->tuning = reader->ReadFloat();
 			}
 
 			soundFont->soundEffects.push_back(entry);

libultraship/libultraship/BinaryReader.cpp

@@ -0,0 +1,216 @@
+#include "BinaryReader.h"
+#include <cmath>
+#include "Lib/BinaryTools/BinaryTools/BinaryReader.h"
+
+Ship::BinaryReader::BinaryReader(char* buffer, uint32_t size) 
+{
+    this->buffer = std::vector<char>(buffer, buffer + size);
+    this->reader = std::make_shared<::BinaryReader>(this->buffer.data(), size);
+}
+
+void Ship::BinaryReader::SetEndianness(Endianness endianness) 
+{
+    this->endianness = endianness;
+}
+
+Ship::Endianness Ship::BinaryReader::GetEndianness() const
+{
+    return endianness;
+}
+
+void Ship::BinaryReader::Seek(int32_t offset, SeekOffsetType seekType) {
+    switch(seekType) {
+        case SeekOffsetType::Current:
+            if (offset < 0) {
+                reader->SeekReverse(-1 * offset);
+                break;
+            }
+            reader->SeekCur(offset);
+            break;
+        case SeekOffsetType::Start:
+            reader->SeekBeg(offset);
+            break;
+        case SeekOffsetType::End:
+            reader->SeekReverse(offset);
+            break;
+    }
+}
+
+uint32_t Ship::BinaryReader::GetBaseAddress() {
+    return reader->Position();
+}
+
+void Ship::BinaryReader::Read(int32_t length)
+{
+    reader->ReadFixedLengthString(length);
+}
+
+void Ship::BinaryReader::Read(char *buffer, int32_t length)
+{
+    reader->ReadToMemory(buffer, length);
+}
+
+char Ship::BinaryReader::ReadChar()
+{
+    return reader->ReadChar();
+}
+
+int8_t Ship::BinaryReader::ReadInt8()
+{
+    return reader->ReadInt8();
+}
+
+int16_t Ship::BinaryReader::ReadInt16()
+{
+    int16_t result = 0;
+    this->Read((char*)&result, sizeof(int16_t));
+    if (endianness != Endianness::Native)
+        result = BSWAP16(result);
+
+    return result;
+}
+
+int32_t Ship::BinaryReader::ReadInt32()
+{
+    int32_t result = 0;
+
+    this->Read((char *)&result, sizeof(int32_t));
+
+    if (endianness != Endianness::Native)
+        result = BSWAP32(result);
+
+    return result;
+}
+
+uint8_t Ship::BinaryReader::ReadUByte()
+{
+    return reader->ReadUint8();
+}
+
+uint16_t Ship::BinaryReader::ReadUInt16()
+{
+    uint16_t result = 0;
+
+    this->Read((char *)&result, sizeof(uint16_t));
+
+    if (endianness != Endianness::Native)
+        result = BSWAP16(result);
+
+    return result;
+}
+
+uint32_t Ship::BinaryReader::ReadUInt32()
+{
+    uint32_t result = 0;
+
+    this->Read((char *)&result, sizeof(uint32_t));
+
+    if (endianness != Endianness::Native)
+        result = BSWAP32(result);
+
+    return result;
+}
+
+uint64_t Ship::BinaryReader::ReadUInt64()
+{
+    uint64_t result = 0;
+
+    this->Read((char *)&result, sizeof(uint64_t));
+
+    if (endianness != Endianness::Native)
+        result = BSWAP64(result);
+
+    return result;
+}
+
+float Ship::BinaryReader::ReadFloat()
+{
+    float result = NAN;
+
+    this->Read((char *)&result, sizeof(float));
+
+    if (endianness != Endianness::Native)
+    {
+        float tmp;
+        char *dst = (char *)&tmp;
+        char *src = (char *)&result;
+        dst[3] = src[0];
+        dst[2] = src[1];
+        dst[1] = src[2];
+        dst[0] = src[3];
+        result = tmp;
+    }
+
+    if (std::isnan(result))
+        throw std::runtime_error("BinaryReader::ReadSingle(): Error reading stream");
+
+    return result;
+}
+
+double Ship::BinaryReader::ReadDouble()
+{
+    double result = NAN;
+
+    this->Read((char *)&result, sizeof(double));
+
+    if (endianness != Endianness::Native)
+    {
+        double tmp;
+        char *dst = (char *)&tmp;
+        char *src = (char *)&result;
+        dst[7] = src[0];
+        dst[6] = src[1];
+        dst[5] = src[2];
+        dst[4] = src[3];
+        dst[3] = src[4];
+        dst[2] = src[5];
+        dst[1] = src[6];
+        dst[0] = src[7];
+        result = tmp;
+    }
+
+    if (std::isnan(result))
+        throw std::runtime_error("BinaryReader::ReadDouble(): Error reading stream");
+
+    return result;
+}
+
+Vec3f Ship::BinaryReader::ReadVec3f()
+{
+    return Vec3f();
+}
+
+Vec3s Ship::BinaryReader::ReadVec3s()
+{
+    return Vec3s(0, 0, 0);
+}
+
+Vec3s Ship::BinaryReader::ReadVec3b()
+{
+    return Vec3s(0, 0, 0);
+}
+
+Vec2f Ship::BinaryReader::ReadVec2f()
+{
+    return Vec2f();
+}
+
+Color3b Ship::BinaryReader::ReadColor3b()
+{
+    return Color3b();
+}
+
+std::string Ship::BinaryReader::ReadString()
+{
+    std::string res;
+    int numChars = reader->ReadInt32();
+    for (int i = 0; i < numChars; i++) {
+        res += reader->ReadChar();
+    }
+    return res;
+}
+
+std::string Ship::BinaryReader::ReadCString()
+{
+    return reader->ReadNullTerminatedString();
+}

libultraship/libultraship/BinaryReader.h

@@ -0,0 +1,61 @@
+#pragma once
+
+#include <string>
+#include <memory>
+#include <vector>
+#include "endianness.h"
+#include "Vec2f.h"
+#include "Vec3f.h"
+#include "Vec3s.h"
+#include "Color3b.h"
+
+class BinaryReader;
+
+namespace Ship {
+
+enum class SeekOffsetType
+{
+    Start,
+    Current,
+    End
+};
+
+class BinaryReader
+{
+public:
+    BinaryReader(char* buffer, uint32_t size);
+
+    void SetEndianness(Endianness endianness);
+    Endianness GetEndianness() const;
+
+    void Seek(int32_t offset, SeekOffsetType seekType);
+    uint32_t GetBaseAddress();
+
+    void Read(int32_t length);
+    void Read(char *buffer, int32_t length);
+    char ReadChar();
+    int8_t ReadInt8();
+    int16_t ReadInt16();
+    int32_t ReadInt32();
+    uint8_t ReadUByte();
+    uint16_t ReadUInt16();
+    uint32_t ReadUInt32();
+    uint64_t ReadUInt64();
+    float ReadFloat();
+    double ReadDouble();
+    Vec3f ReadVec3f();
+    Vec3s ReadVec3s();
+    Vec3s ReadVec3b();
+    Vec2f ReadVec2f();
+    Color3b ReadColor3b();
+    std::string ReadString();
+    std::string ReadCString();
+
+protected:
+    Endianness endianness = Endianness::Native;
+
+private:
+    std::vector<char> buffer;
+    std::shared_ptr<::BinaryReader> reader;
+};
+}

libultraship/libultraship/CMakeLists.txt

@@ -77,6 +77,21 @@ endif ()
 
 source_group("Source Files\\Audio" FILES ${Source_Files__Audio} ${Source_Files__Audio__extra})
 
+set (Source_Files__BinaryTools
+    "BinaryReader.cpp"
+    "BinaryReader.h"
+    "Lib/BinaryTools/BinaryTools/Binary.h"
+    "Lib/BinaryTools/BinaryTools/Binary.cpp"
+    "Lib/BinaryTools/BinaryTools/BinaryReader.cpp"
+    "Lib/BinaryTools/BinaryTools/BinaryReader.h"
+    "Lib/BinaryTools/BinaryTools/BinaryWriter.cpp"
+    "Lib/BinaryTools/BinaryTools/BinaryWriter.h"
+    "Lib/BinaryTools/BinaryTools/MemoryBuffer.h"
+    "Lib/BinaryTools/BinaryTools/Span.h"
+)
+
+source_group("Source Files\\BinaryTools" FILES ${Source_Files__BinaryTools})
+
 set(Source_Files__Controller
     "ControlDeck.cpp"
     "ControlDeck.h"
@@ -376,6 +391,7 @@ set(ALL_FILES
     ${Header_Files__Resources__Files}
     ${Source_Files__Audio}
     ${Source_Files__Audio__extra}
+    ${Source_Files__BinaryTools}
     ${Source_Files__Controller}
     ${Source_Files__Controller__Attachment}
     ${Source_Files__CustomImpl}

libultraship/libultraship/CrashHandler.cpp

@@ -55,7 +55,7 @@ static const char* GetGameVersionString() {
                 return "IQUE_TW";
             case OOT_IQUE_CN:
                 return "IQUE_CN";
-            case OOT_UNKNOWN:
+            case UNKNOWN:
                 return "UNKNOWN";
         }
 

libultraship/libultraship/Cutscene.cpp

@@ -37,7 +37,7 @@ enum class CutsceneCommands
 	Error = 0xFEAF,
 };
 
-static inline uint32_t read_CMD_BBBB(BinaryReader* reader)
+static inline uint32_t read_CMD_BBBB(Ship::BinaryReader* reader)
 {
 	uint32_t v;
 	reader->Read((char*)&v, sizeof(uint32_t));
@@ -45,13 +45,13 @@ static inline uint32_t read_CMD_BBBB(BinaryReader* reader)
 	return v;
 }
 
-static inline uint32_t read_CMD_BBH(BinaryReader* reader)
+static inline uint32_t read_CMD_BBH(Ship::BinaryReader* reader)
 {
 	uint32_t v;
 	reader->Read((char*)&v, sizeof(uint32_t));
 
 	// swap the half word to match endianness
-	if (reader->GetEndianness() != Endianness::Native)
+	if (reader->GetEndianness() != Ship::Endianness::Native)
 	{
 		uint8_t* b = (uint8_t*)&v;
 		uint8_t tmp = b[2];
@@ -62,13 +62,13 @@ static inline uint32_t read_CMD_BBH(BinaryReader* reader)
 	return v;
 }
 
-static inline uint32_t read_CMD_HBB(BinaryReader* reader)
+static inline uint32_t read_CMD_HBB(Ship::BinaryReader* reader)
 {
 	uint32_t v;
 	reader->Read((char*)&v, sizeof(uint32_t));
 
 	// swap the half word to match endianness
-	if (reader->GetEndianness() != Endianness::Native)
+	if (reader->GetEndianness() != Ship::Endianness::Native)
 	{
 		uint8_t* b = (uint8_t*)&v;
 		uint8_t tmp = b[0];
@@ -79,13 +79,13 @@ static inline uint32_t read_CMD_HBB(BinaryReader* reader)
 	return v;
 }
 
-static inline uint32_t read_CMD_HH(BinaryReader* reader)
+static inline uint32_t read_CMD_HH(Ship::BinaryReader* reader)
 {
 	uint32_t v;
 	reader->Read((char*)&v, sizeof(uint32_t));
 
 	// swap the half words to match endianness
-	if (reader->GetEndianness() != Endianness::Native)
+	if (reader->GetEndianness() != Ship::Endianness::Native)
 	{
 		uint8_t* b = (uint8_t*)&v;
 		uint8_t tmp = b[0];
@@ -99,7 +99,7 @@ static inline uint32_t read_CMD_HH(BinaryReader* reader)
 	return v;
 }
 
-void Ship::CutsceneV0::ParseFileBinary(BinaryReader* reader, Resource* res)
+void Ship::CutsceneV0::ParseFileBinary(Ship::BinaryReader* reader, Resource* res)
 {
 	Cutscene* cs = (Cutscene*)res;
 

libultraship/libultraship/DisplayList.cpp

@@ -10,7 +10,7 @@ namespace Ship
 		ResourceFile::ParseFileBinary(reader, res);
 
 		while (reader->GetBaseAddress() % 8 != 0)
-			reader->ReadByte();
+			reader->ReadInt8();
 
 		while (true)
 		{