Dual OTR MQ and Vanilla Support (#1694)

* Changes OTR Extraction to have specific mq and nonmq paths.

Also updates the game to load resources according to whether or not
Master Quest or Vanilla is loaded.

* Removes unneeded code from the last commit.

* Fixes some weird formatting in ZRom.c

* Loads oot-mq.otr and patches oot.otr on top, if both are present.

If only one or the other are present, it becomes the only and main OTR.

* Adds ImGui Logic for whether or an MQ Checkbox.

Checkbox checked only specifies whether new saves should be MQ or not.
Checkbox is disabled or force-enabled according to which OTRs are loaded.
Also as a necessity includes tracking what game versions have been loaded
from the OTRs.

* Adds MQ settings logic for Randomizer's ImGui menu.

* Writes Master Quest dungeons to the spoiler log

* Loads MQ Dungeons from spoiler, persists in save, and loads when appropriate.

* Adds logic to prevent loading or creating incompatible rando saves.

* Fixdes some linux build issues and new rando save issues

* Makes appimage create both vanilla and mq otrs

If either rom is present, it makes the corresponding OTR. If both are present,
it will make both. If one OTR is present but both roms are present, it will
create the missing OTR.

* Makes it so a randomized save file will not be marked as MQ.

* Refactors to load all OTRs from MainPath or a specific list.

Also adds the ability to take a std::unordered_set of hashes to
validate each OTR's version file against.

* Fixes a syntax error

* Makes ExtractAssets output Vanilla and MQ OTRs if both roms are present

* Fixes asset generation bug.

* Partially working fix for dual OTR extract_assets

Currently the cmake ExtractAssets target will return with a 1 if you
only end up exporting one type of OTR isntead of both. Haven't found
a great way to only attempt to copy a file if it exists from within
cmake. It does actually correctly copy the OTR that is generated,
despite the error from copying the other one.

Pushing as is for now but will keep investigating.

* Adds oot-mq.otr to the gitignore.

* Makes ExtractAssets not fail on only one rom/OTR.

* Removes PatchesPath from the constructors requiring OTRFiles vector.

* Renames OOT_UNKNOWN to just UNKNOWN to remove OOT specific reference.

* Removes randomizing MQ Dungeons and re-disables MQ rando.

Doing this so the PR can get merged quicker with just the Dual OTR
support and won't need to wait on rando logic to be updated. That
will happen in another PR directly after the merge.

* Update mac startup script for dual otr

* Update soh/macosx/soh-macos.sh

* Update soh/macosx/soh-macos.sh

* Update soh/macosx/soh-macos.sh

* Implements new BinaryReader to fix Linux build issue.

BinaryReader itself comes from https://github.com/Moneyl/BinaryTools
I added a wrapper to adapt it to the ABI from ZAPD's Binary Reader and
add Endianness checking. I also had to copy a handful of other bits and
pieces from ZAPD to make it all function as expected.

* A few edits to the updatream BinaryReader to compile it on Linux.

* Adds the Endianness to the first byte of the version file.

* Fixes Jenkins

* Addresses some of Kenix's comments

* Renames `ReadNullTerminatedString` to `ReadCString`

* Refactors Archive::LoadFile into a private method with more arguments.

* Removes BitConverter and extends existing endianness.h instead.

* Fixes an endianness issue with the version file.

Co-authored-by: Garrett Cox <garrettjcox@gmail.com>
pull/1705/head
Christopher Leggett 2022-10-16 23:07:35 -04:00 committed by GitHub
parent 350315a5d1
commit 7b08f98b8c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
72 changed files with 3149 additions and 485 deletions

1
.gitignore vendored
View File

@ -410,6 +410,7 @@ ReleaseObj/*
.tags
tags
oot.otr
oot-mq.otr
*.sav
shipofharkinian.ini
shipofharkinian.json

View File

@ -136,14 +136,13 @@ find_package(Python3 COMPONENTS Interpreter)
add_custom_target(
ExtractAssets
# CMake versions prior to 3.17 do not have the rm command, use remove instead for older versions
COMMAND ${CMAKE_COMMAND} -E $<IF:$<VERSION_LESS:${CMAKE_VERSION},3.17>,remove,rm> -f oot.otr
COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/OTRExporter/extract_assets.py -z "$<TARGET_FILE:ZAPD>"
COMMAND ${CMAKE_COMMAND} -E copy oot.otr ${CMAKE_SOURCE_DIR}
COMMAND ${CMAKE_COMMAND} -E copy oot.otr ${CMAKE_BINARY_DIR}/soh
COMMAND ${CMAKE_COMMAND} -E $<IF:$<VERSION_LESS:${CMAKE_VERSION},3.17>,remove,rm> -f oot.otr oot-mq.otr
COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/OTRExporter/extract_assets.py -z "$<TARGET_FILE:ZAPD>" --non-interactive
COMMAND ${CMAKE_COMMAND} -DSOURCE_DIR=${CMAKE_CURRENT_SOURCE_DIR} -DBINARY_DIR=${CMAKE_BINARY_DIR} -P ${CMAKE_CURRENT_SOURCE_DIR}/copy-existing-otrs.cmake
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/OTRExporter
COMMENT "Running asset extraction..."
DEPENDS ZAPD
BYPRODUCTS oot.otr ${CMAKE_SOURCE_DIR}/oot.otr
BYPRODUCTS oot.otr ${CMAKE_SOURCE_DIR}/oot.otr oot-mq.otr ${CMAKE_SOURCE_DIR}/oot-mq.otr
)
if(CMAKE_SYSTEM_NAME MATCHES "Linux")

View File

@ -882,8 +882,14 @@ std::string OTRExporter_DisplayList::GetPrefix(ZResource* res)
std::string prefix = "";
std::string xmlPath = StringHelper::Replace(res->parent->GetXmlFilePath().string(), "\\", "/");
if (StringHelper::Contains(oName, "_scene") || StringHelper::Contains(oName, "_room"))
if (StringHelper::Contains(oName, "_scene") || StringHelper::Contains(oName, "_room")) {
prefix = "scenes";
if (Globals::Instance->rom->IsMQ()) {
prefix += "/mq";
} else {
prefix += "/nonmq";
}
}
else if (StringHelper::Contains(xmlPath, "objects/"))
prefix = "objects";
else if (StringHelper::Contains(xmlPath, "textures/"))

View File

@ -21,6 +21,7 @@
#include <Utils/Directory.h>
#include <Utils/MemoryStream.h>
#include <Utils/BinaryWriter.h>
#include <bit>
std::string otrFileName = "oot.otr";
std::shared_ptr<Ship::Archive> otrArchive;
@ -69,13 +70,20 @@ static void ExporterProgramEnd()
std::string romPath = Globals::Instance->baseRomPath.string();
std::vector<uint8_t> romData = File::ReadAllBytes(romPath);
uint32_t crc = BitConverter::ToUInt32BE(romData, 0x10);
uint8_t endianness = (uint8_t)Endianness::Big;
// Write crc to version file
fs::path versionPath("Extract/version");
MemoryStream* versionStream = new MemoryStream();
BinaryWriter writer(versionStream);
writer.SetEndianness(Endianness::Big);
writer.Write(endianness);
writer.Write(crc);
std::ofstream versionFile(versionPath.c_str(), std::ios::out | std::ios::binary);
versionFile.write((char*)&crc, sizeof(crc));
versionFile.write(versionStream->ToVector().data(), versionStream->GetLength());
versionFile.flush();
versionFile.close();
writer.Close();
printf("Created version file.\n");

View File

@ -16,7 +16,8 @@ def BuildOTR(xmlPath, rom, zapd_exe=None):
exec_cmd = [zapd_exe, "ed", "-i", xmlPath, "-b", rom, "-fl", "CFG/filelists",
"-o", "placeholder", "-osf", "placeholder", "-gsf", "1",
"-rconf", "CFG/Config.xml", "-se", "OTR"]
"-rconf", "CFG/Config.xml", "-se", "OTR", "--otrfile",
"oot-mq.otr" if Z64Rom.isMqRom(rom) else "oot.otr"]
print(exec_cmd)
exitValue = subprocess.call(exec_cmd)
@ -30,16 +31,18 @@ def main():
parser = argparse.ArgumentParser()
parser.add_argument("-z", "--zapd", help="Path to ZAPD executable", dest="zapd_exe", type=str)
parser.add_argument("rom", help="Path to the rom", type=str, nargs="?")
parser.add_argument("--non-interactive", help="Runs the script non-interactively for use in build scripts.", dest="non_interactive", action="store_true")
args = parser.parse_args()
rom_path = args.rom if args.rom else rom_chooser.chooseROM()
rom = Z64Rom(rom_path)
rom_paths = [ args.rom ] if args.rom else rom_chooser.chooseROM(args.non_interactive)
for rom_path in rom_paths:
rom = Z64Rom(rom_path)
if (os.path.exists("Extract")):
shutil.rmtree("Extract")
if (os.path.exists("Extract")):
shutil.rmtree("Extract")
BuildOTR("../soh/assets/xml/" + rom.version.xml_ver + "/", rom_path, zapd_exe=args.zapd_exe)
BuildOTR("../soh/assets/xml/" + rom.version.xml_ver + "/", rom_path, zapd_exe=args.zapd_exe)
if __name__ == "__main__":
main()

View File

@ -2,7 +2,7 @@ import os, sys, glob
from rom_info import Z64Rom
def chooseROM():
def chooseROM(non_interactive=False):
roms = []
for file in glob.glob("*.z64"):
@ -14,7 +14,21 @@ def chooseROM():
sys.exit(1)
if (len(roms) == 1):
return roms[0]
return roms
if non_interactive:
romsToExtract = []
foundMq = False
foundOot = False
for rom in roms:
isMq = Z64Rom.isMqRom(rom)
if isMq and not foundMq:
romsToExtract.append(rom)
foundMq = True
elif not isMq and not foundOot:
romsToExtract.append(rom)
foundOot = True
return romsToExtract
print(str(len(roms))+ " roms found, please select one by pressing 1-"+str(len(roms)))
@ -34,4 +48,4 @@ def chooseROM():
else: break
return roms[selection - 1]
return [ roms[selection - 1] ]

View File

@ -73,6 +73,11 @@ class Z64Rom:
if self.checksum == Checksums.OOT_UNKNOWN:
self.is_valid = False
return
if self.checksum in [Checksums.OOT_NTSC_JP_MQ, Checksums.OOT_NTSC_US_MQ, Checksums.OOT_PAL_GC_MQ_DBG, Checksums.OOT_PAL_MQ]:
self.isMq = True
else:
self.isMq = False
# get rom version
self.version = ROM_INFO_TABLE[self.checksum]
@ -86,3 +91,7 @@ class Z64Rom:
@staticmethod
def isValidRom(rom_path):
return Z64Rom(rom_path).is_valid
@staticmethod
def isMqRom(rom_path):
return Z64Rom(rom_path).isMq

View File

@ -328,7 +328,7 @@ std::string ZResource::GetSourceOutputHeader([[maybe_unused]] const std::string&
std::string xmlPath = StringHelper::Replace(parent->GetXmlFilePath().string(), "\\", "/");
if (StringHelper::Contains(outName, "_room_") || StringHelper::Contains(outName, "_scene"))
prefix = "scenes";
prefix = "scenes/nonmq";
else if (StringHelper::Contains(xmlPath, "objects/"))
prefix = "objects";
else if (StringHelper::Contains(xmlPath, "textures/"))
@ -342,8 +342,9 @@ std::string ZResource::GetSourceOutputHeader([[maybe_unused]] const std::string&
else if (StringHelper::Contains(xmlPath, "text/"))
prefix = "text";
if (prefix != "")
if (prefix != "") {
str += StringHelper::Sprintf("#define d%s \"__OTR__%s/%s/%s\"", name.c_str(), prefix.c_str(), outName.c_str(), nameStr.c_str());
}
else
str += StringHelper::Sprintf("#define d%s \"__OTR__%s/%s\"", name.c_str(), outName.c_str(), nameStr.c_str());

View File

@ -64,7 +64,33 @@ namespace fs = std::filesystem;
#define OOT_PAL_GC_MQ_DBG 0x917D18F6
#define OOT_IQUE_TW 0x3D81FB3E
#define OOT_IQUE_CN 0xB1E1E07B
#define OOT_UNKNOWN 0xFFFFFFFF
#define UNKNOWN 0xFFFFFFFF
bool ZRom::IsMQ() {
int crc = BitConverter::ToInt32BE(romData, 0x10);
switch (crc) {
case OOT_NTSC_10:
case OOT_NTSC_11:
case OOT_NTSC_12:
case OOT_PAL_10:
case OOT_PAL_11:
case OOT_NTSC_JP_GC:
case OOT_NTSC_JP_GC_CE:
case OOT_NTSC_US_GC:
case OOT_PAL_GC:
case OOT_PAL_GC_DBG1:
case OOT_PAL_GC_DBG2:
case OOT_IQUE_CN:
case OOT_IQUE_TW:
default:
return false;
case OOT_NTSC_JP_MQ:
case OOT_NTSC_US_MQ:
case OOT_PAL_MQ:
case OOT_PAL_GC_MQ_DBG:
return true;
}
}
ZRom::ZRom(std::string romPath)
{

View File

@ -11,6 +11,7 @@ public:
ZRom(std::string romPath);
std::vector<uint8_t> GetFile(std::string fileName);
bool IsMQ();
protected:
std::vector<uint8_t> romData;

11
copy-existing-otrs.cmake Normal file
View File

@ -0,0 +1,11 @@
if(EXISTS ${SOURCE_DIR}/OTRExporter/oot.otr)
execute_process(COMMAND ${CMAKE_COMMAND} -E copy oot.otr ${SOURCE_DIR})
execute_process(COMMAND ${CMAKE_COMMAND} -E copy oot.otr ${BINARY_DIR}/soh/)
endif()
if(EXISTS ${SOURCE_DIR}/OTRExporter/oot-mq.otr)
execute_process(COMMAND ${CMAKE_COMMAND} -E copy oot-mq.otr ${SOURCE_DIR})
execute_process(COMMAND ${CMAKE_COMMAND} -E copy oot-mq.otr ${BINARY_DIR}/soh/)
endif()
if(NOT EXISTS ${SOURCE_DIR}/oot.otr AND NOT EXISTS ${SOURCE_DIR}/oot-mq.otr)
message(FATAL_ERROR, "No OTR files found.")
endif()

View File

@ -48,7 +48,7 @@ void Ship::AnimationV0::ParseFileBinary(BinaryReader* reader, Resource* res)
data.unk_02 = reader->ReadInt16();
data.unk_04 = reader->ReadInt16();
data.unk_06 = reader->ReadInt16();
data.unk_08 = reader->ReadSingle();
data.unk_08 = reader->ReadFloat();
anim->transformDataArr.push_back(data);
}

View File

@ -1,27 +1,38 @@
#include "Archive.h"
#include "Resource.h"
#include "File.h"
#include "Window.h"
#include "ResourceMgr.h"
#include "spdlog/spdlog.h"
#include "Utils/StringHelper.h"
#include "Lib/StrHash64.h"
#include <filesystem>
#include "Lib/BinaryTools/BinaryTools/BinaryReader.h"
#ifdef __SWITCH__
#include "SwitchImpl.h"
#endif
namespace Ship {
Archive::Archive(const std::string& MainPath, bool enableWriting) : Archive(MainPath, "", enableWriting)
Archive::Archive(const std::string& MainPath, bool enableWriting) : Archive(MainPath, "", std::unordered_set<uint32_t>(), enableWriting)
{
mainMPQ = nullptr;
}
Archive::Archive(const std::string& MainPath, const std::string& PatchesPath, bool enableWriting, bool genCRCMap) : MainPath(MainPath), PatchesPath(PatchesPath) {
Archive::Archive(const std::string& MainPath, const std::string& PatchesPath, const std::unordered_set<uint32_t>& ValidHashes, bool enableWriting, bool genCRCMap)
: MainPath(MainPath), PatchesPath(PatchesPath), OTRFiles({}), ValidHashes(ValidHashes) {
mainMPQ = nullptr;
Load(enableWriting, genCRCMap);
}
Archive::~Archive() {
Archive::Archive(const std::vector<std::string> &OTRFiles, const std::unordered_set<uint32_t> &ValidHashes, bool enableWriting, bool genCRCMap)
: OTRFiles(OTRFiles), ValidHashes(ValidHashes)
{
mainMPQ = nullptr;
Load(enableWriting, genCRCMap);
}
Archive::~Archive() {
Unload();
}
@ -55,48 +66,63 @@ namespace Ship {
}
}
std::shared_ptr<File> Archive::LoadFileFromHandle(const std::string& filePath, bool includeParent, std::shared_ptr<File> FileToLoad, HANDLE mpqHandle)
{
HANDLE fileHandle = NULL;
if (FileToLoad == nullptr)
{
FileToLoad = std::make_shared<File>();
FileToLoad->path = filePath;
}
if (mpqHandle == nullptr)
{
mpqHandle = mainMPQ;
}
bool attempt = SFileOpenFileEx(mpqHandle, filePath.c_str(), 0, &fileHandle);
if (!attempt)
{
SPDLOG_ERROR("({}) Failed to open file {} from mpq archive {}.", GetLastError(), filePath.c_str(), MainPath.c_str());
std::unique_lock<std::mutex> Lock(FileToLoad->FileLoadMutex);
FileToLoad->bHasLoadError = true;
return FileToLoad;
}
DWORD dwFileSize = SFileGetFileSize(fileHandle, 0);
std::shared_ptr<char[]> fileData(new char[dwFileSize]);
DWORD dwBytes;
if (!SFileReadFile(fileHandle, fileData.get(), dwFileSize, &dwBytes, NULL))
{
SPDLOG_ERROR("({}) Failed to read file {} from mpq archive {}", GetLastError(), filePath.c_str(), MainPath.c_str());
if (!SFileCloseFile(fileHandle))
{
SPDLOG_ERROR("({}) Failed to close file {} from mpq after read failure in archive {}", GetLastError(), filePath.c_str(), MainPath.c_str());
}
std::unique_lock<std::mutex> Lock(FileToLoad->FileLoadMutex);
FileToLoad->bHasLoadError = true;
return FileToLoad;
}
if (!SFileCloseFile(fileHandle))
{
SPDLOG_ERROR("({}) Failed to close file {} from mpq archive {}", GetLastError(), filePath.c_str(), MainPath.c_str());
}
std::unique_lock<std::mutex> Lock(FileToLoad->FileLoadMutex);
FileToLoad->parent = includeParent ? shared_from_this() : nullptr;
FileToLoad->buffer = fileData;
FileToLoad->dwBufferSize = dwFileSize;
FileToLoad->bIsLoaded = true;
return FileToLoad;
}
std::shared_ptr<File> Archive::LoadFile(const std::string& filePath, bool includeParent, std::shared_ptr<File> FileToLoad) {
HANDLE fileHandle = NULL;
if (FileToLoad == nullptr) {
FileToLoad = std::make_shared<File>();
FileToLoad->path = filePath;
}
bool attempt = SFileOpenFileEx(mainMPQ, filePath.c_str(), 0, &fileHandle);
if (!attempt) {
SPDLOG_ERROR("({}) Failed to open file {} from mpq archive {}.", GetLastError(), filePath.c_str(), MainPath.c_str());
std::unique_lock<std::mutex> Lock(FileToLoad->FileLoadMutex);
FileToLoad->bHasLoadError = true;
return FileToLoad;
}
DWORD dwFileSize = SFileGetFileSize(fileHandle, 0);
std::shared_ptr<char[]> fileData(new char[dwFileSize]);
DWORD dwBytes;
if (!SFileReadFile(fileHandle, fileData.get(), dwFileSize, &dwBytes, NULL)) {
SPDLOG_ERROR("({}) Failed to read file {} from mpq archive {}", GetLastError(), filePath.c_str(), MainPath.c_str());
if (!SFileCloseFile(fileHandle)) {
SPDLOG_ERROR("({}) Failed to close file {} from mpq after read failure in archive {}", GetLastError(), filePath.c_str(), MainPath.c_str());
}
std::unique_lock<std::mutex> Lock(FileToLoad->FileLoadMutex);
FileToLoad->bHasLoadError = true;
return FileToLoad;
}
if (!SFileCloseFile(fileHandle)) {
SPDLOG_ERROR("({}) Failed to close file {} from mpq archive {}", GetLastError(), filePath.c_str(), MainPath.c_str());
}
std::unique_lock<std::mutex> Lock(FileToLoad->FileLoadMutex);
FileToLoad->parent = includeParent ? shared_from_this() : nullptr;
FileToLoad->buffer = fileData;
FileToLoad->dwBufferSize = dwFileSize;
FileToLoad->bIsLoaded = true;
return FileToLoad;
return LoadFileFromHandle(filePath, includeParent, FileToLoad, nullptr);
}
std::shared_ptr<File> Archive::LoadPatchFile(const std::string& filePath, bool includeParent, std::shared_ptr<File> FileToLoad) {
@ -320,53 +346,129 @@ namespace Ship {
return true;
}
void Archive::GenerateCRCMap() {
auto listFile = LoadFile("(listfile)", false);
std::vector<std::string> lines = StringHelper::Split(std::string(listFile->buffer.get(), listFile->dwBufferSize), "\n");
for (size_t i = 0; i < lines.size(); i++)
{
std::string line = StringHelper::Replace(StringHelper::Strip(lines[i], "\r"), "/", "\\");
std::string line2 = StringHelper::Replace(line, "\\", "/");
uint64_t hash = CRC64(StringHelper::Replace(line, "/", "\\").c_str());
uint64_t hash2 = CRC64(StringHelper::Replace(line, "\\", "/").c_str());
hashes[hash] = line;
hashes[hash2] = line2;
}
}
bool Archive::PushGameVersion(HANDLE mpqHandle) {
auto t = LoadFileFromHandle("version", false, nullptr, mpqHandle);
if (!t->bHasLoadError)
{
BinaryReader *reader = new BinaryReader(t->buffer.get(), t->dwBufferSize);
Ship::Endianness endianness = (Ship::Endianness)reader->ReadUByte();
reader->SetEndianness(endianness);
uint32_t version = reader->ReadUInt32();
if (ValidHashes.empty() || ValidHashes.contains(version)) {
gameVersions.push_back(version);
return true;
}
}
return false;
}
bool Archive::LoadMainMPQ(bool enableWriting, bool genCRCMap) {
HANDLE mpqHandle = NULL;
HANDLE mpqHandle = NULL;
if (OTRFiles.empty()) {
if (MainPath.length() > 0) {
if (std::filesystem::is_directory(MainPath)) {
for (const auto &p : std::filesystem::recursive_directory_iterator(MainPath)) {
if (StringHelper::IEquals(p.path().extension().string(), ".otr")) {
SPDLOG_ERROR("Reading {} mpq", p.path().string().c_str());
OTRFiles.push_back(p.path().string());
}
}
} else if (std::filesystem::is_regular_file(MainPath)) {
OTRFiles.push_back(MainPath);
} else {
SPDLOG_ERROR("The directory {} does not exist", MainPath.c_str());
return false;
}
} else {
SPDLOG_ERROR("No OTR file list or Main Path provided.");
return false;
}
if (OTRFiles.empty()) {
SPDLOG_ERROR("No OTR files present in {}", MainPath.c_str());
return false;
}
}
bool baseLoaded = false;
int i = 0;
while (!baseLoaded && i < OTRFiles.size()) {
#ifdef _WIN32
std::wstring wfullPath = std::filesystem::absolute(MainPath).wstring();
std::wstring wfullPath = std::filesystem::absolute(OTRFiles[i]).wstring();
#endif
#if defined(__SWITCH__)
std::string fullPath = MainPath;
#else
std::string fullPath = std::filesystem::absolute(MainPath).string();
std::string fullPath = OTRFiles[0];
#else
std::string fullPath = std::filesystem::absolute(OTRFiles[i]).string();
#endif
#ifdef _WIN32
if (!SFileOpenArchive(wfullPath.c_str(), 0, enableWriting ? 0 : MPQ_OPEN_READ_ONLY, &mpqHandle)) {
if (SFileOpenArchive(wfullPath.c_str(), 0, enableWriting ? 0 : MPQ_OPEN_READ_ONLY, &mpqHandle))
{
#else
if (!SFileOpenArchive(fullPath.c_str(), 0, enableWriting ? 0 : MPQ_OPEN_READ_ONLY, &mpqHandle)) {
if (SFileOpenArchive(fullPath.c_str(), 0, enableWriting ? 0 : MPQ_OPEN_READ_ONLY, &mpqHandle))
{
#endif
#ifdef __SWITCH__
Switch::ThrowMissingOTR(fullPath);
#endif
SPDLOG_ERROR("({}) Failed to open main mpq file {}.", GetLastError(), fullPath.c_str());
return false;
}
mpqHandles[fullPath] = mpqHandle;
mainMPQ = mpqHandle;
if (genCRCMap) {
auto listFile = LoadFile("(listfile)", false);
std::vector<std::string> lines = StringHelper::Split(std::string(listFile->buffer.get(), listFile->dwBufferSize), "\n");
for (size_t i = 0; i < lines.size(); i++) {
std::string line = StringHelper::Replace(StringHelper::Strip(lines[i], "\r"), "/", "\\");
std::string line2 = StringHelper::Replace(line, "\\", "/");
uint64_t hash = CRC64(StringHelper::Replace(line, "/", "\\").c_str());
uint64_t hash2 = CRC64(StringHelper::Replace(line, "\\", "/").c_str());
hashes[hash] = line;
hashes[hash2] = line2;
}
}
SPDLOG_INFO("Opened mpq file {}.", fullPath.c_str());
mainMPQ = mpqHandle;
if (!PushGameVersion()) {
SPDLOG_WARN("Attempted to load invalid OTR file {}", OTRFiles[i].c_str());
SFileCloseArchive(mpqHandle);
mainMPQ = nullptr;
} else {
mpqHandles[fullPath] = mpqHandle;
if (genCRCMap)
{
GenerateCRCMap();
}
baseLoaded = true;
}
}
i++;
}
// If we exited the above loop without setting baseLoaded to true, then we've
// attemtped to load all the OTRs available to us.
if (!baseLoaded) {
SPDLOG_ERROR("No valid OTR file was provided.");
return false;
}
for (int j = i; j < OTRFiles.size(); j++) {
#ifdef _WIN32
std::wstring wfullPath = std::filesystem::absolute(OTRFiles[i]).wstring();
#endif
#if defined(__SWITCH__)
std::string fullPath = OTRFiles[i];
#else
std::string fullPath = std::filesystem::absolute(OTRFiles[i]).string();
#endif
if (LoadPatchMPQ(fullPath, true))
{
SPDLOG_INFO("({}) Patched in mpq file.", fullPath.c_str());
}
if (genCRCMap)
{
GenerateCRCMap();
}
}
return true;
}
bool Archive::LoadPatchMPQ(const std::string& path) {
bool Archive::LoadPatchMPQ(const std::string& path, bool validateVersion) {
HANDLE patchHandle = NULL;
#if defined(__SWITCH__)
std::string fullPath = path;
@ -386,7 +488,16 @@ namespace Ship {
#endif
SPDLOG_ERROR("({}) Failed to open patch mpq file {} while applying to {}.", GetLastError(), path.c_str(), MainPath.c_str());
return false;
}
} else {
// We don't always want to validate the "version" file, only when we're loading standalone OTRs as patches
// i.e. Ocarina of Time along with Master Quest.
if (validateVersion) {
if (!PushGameVersion(patchHandle)) {
SPDLOG_WARN("({}) Invalid MQP file.", path.c_str());
return false;
}
}
}
#ifdef _WIN32
if (!SFileOpenPatchArchive(mainMPQ, wPath.c_str(), "", 0)) {
#else

View File

@ -9,6 +9,7 @@
#include <unordered_map>
#include <string>
#include <vector>
#include <unordered_set>
#include "Resource.h"
#include "StormLib.h"
@ -21,8 +22,9 @@ namespace Ship
{
public:
Archive(const std::string& MainPath, bool enableWriting);
Archive(const std::string& MainPath, const std::string& PatchesPath, bool enableWriting, bool genCRCMap = true);
~Archive();
Archive(const std::string &MainPath, const std::string &PatchesPath, const std::unordered_set<uint32_t> &ValidHashes, bool enableWriting, bool genCRCMap = true);
Archive(const std::vector<std::string> &OTRFiles, const std::unordered_set<uint32_t> &ValidHashes, bool enableWriting, bool genCRCMap = true);
~Archive();
bool IsMainMPQValid();
@ -37,19 +39,25 @@ namespace Ship
std::vector<SFILE_FIND_DATA> ListFiles(const std::string& searchMask) const;
bool HasFile(const std::string& searchMask) const;
const std::string* HashToString(uint64_t hash) const;
std::vector<uint32_t> gameVersions;
protected:
bool Load(bool enableWriting, bool genCRCMap);
bool Unload();
private:
std::string MainPath;
std::string PatchesPath;
std::map<std::string, HANDLE> mpqHandles;
std::vector<std::string> OTRFiles;
std::unordered_set<uint32_t> ValidHashes;
std::map<std::string, HANDLE> mpqHandles;
std::vector<std::string> addedFiles;
std::unordered_map<uint64_t, std::string> hashes;
HANDLE mainMPQ;
bool LoadMainMPQ(bool enableWriting, bool genCRCMap);
bool LoadPatchMPQs();
bool LoadPatchMPQ(const std::string& path);
};
bool LoadPatchMPQ(const std::string& path, bool validateVersion = false);
void GenerateCRCMap();
bool PushGameVersion(HANDLE mpqHandle = nullptr);
std::shared_ptr<File> LoadFileFromHandle(const std::string &filePath, bool includeParent = true, std::shared_ptr<File> FileToLoad = nullptr, HANDLE mpqHandle = nullptr);
};
}

View File

@ -31,10 +31,10 @@ namespace Ship
ResourceFile::ParseFileBinary(reader, res);
entry->codec = reader->ReadByte();
entry->medium = reader->ReadByte();
entry->unk_bit26 = reader->ReadByte();
entry->unk_bit25 = reader->ReadByte();
entry->codec = reader->ReadInt8();
entry->medium = reader->ReadInt8();
entry->unk_bit26 = reader->ReadInt8();
entry->unk_bit25 = reader->ReadInt8();
uint32_t dataSize = reader->ReadInt32();
@ -66,8 +66,8 @@ namespace Ship
ResourceFile::ParseFileBinary(reader, res);
soundFont->id = reader->ReadInt32();
soundFont->medium = reader->ReadByte();
soundFont->cachePolicy = reader->ReadByte();
soundFont->medium = reader->ReadInt8();
soundFont->cachePolicy = reader->ReadInt8();
soundFont->data1 = reader->ReadInt16();
soundFont->data2 = reader->ReadInt16();
soundFont->data3 = reader->ReadInt16();
@ -85,9 +85,9 @@ namespace Ship
drum.env = ReadEnvelopeData(reader);
bool hasSample = reader->ReadByte();
bool hasSample = reader->ReadInt8();
drum.sampleFileName = reader->ReadString();
drum.tuning = reader->ReadSingle();
drum.tuning = reader->ReadFloat();
soundFont->drums.push_back(drum);
}
@ -105,38 +105,38 @@ namespace Ship
entry.env = ReadEnvelopeData(reader);
{
bool hasSFEntry = reader->ReadByte();
bool hasSFEntry = reader->ReadInt8();
if (hasSFEntry)
{
entry.lowNotesSound = new SoundFontEntry();
bool hasSampleRef = reader->ReadByte();
bool hasSampleRef = reader->ReadInt8();
entry.lowNotesSound->sampleFileName = reader->ReadString();
entry.lowNotesSound->tuning = reader->ReadSingle();
entry.lowNotesSound->tuning = reader->ReadFloat();
}
}
{
bool hasSFEntry = reader->ReadByte();
bool hasSFEntry = reader->ReadInt8();
if (hasSFEntry)
{
entry.normalNotesSound = new SoundFontEntry();
bool hasSampleRef = reader->ReadByte();
bool hasSampleRef = reader->ReadInt8();
entry.normalNotesSound->sampleFileName = reader->ReadString();
entry.normalNotesSound->tuning = reader->ReadSingle();
entry.normalNotesSound->tuning = reader->ReadFloat();
}
}
{
bool hasSFEntry = reader->ReadByte();
bool hasSFEntry = reader->ReadInt8();
if (hasSFEntry)
{
entry.highNotesSound = new SoundFontEntry();
bool hasSampleRef = reader->ReadByte();
bool hasSampleRef = reader->ReadInt8();
entry.highNotesSound->sampleFileName = reader->ReadString();
entry.highNotesSound->tuning = reader->ReadSingle();
entry.highNotesSound->tuning = reader->ReadFloat();
}
}
@ -147,13 +147,13 @@ namespace Ship
{
SoundFontEntry* entry = new SoundFontEntry();
bool hasSFEntry = reader->ReadByte();
bool hasSFEntry = reader->ReadInt8();
if (hasSFEntry)
{
bool hasSampleRef = reader->ReadByte();
bool hasSampleRef = reader->ReadInt8();
entry->sampleFileName = reader->ReadString();
entry->tuning = reader->ReadSingle();
entry->tuning = reader->ReadFloat();
}
soundFont->soundEffects.push_back(entry);

View File

@ -0,0 +1,216 @@
#include "BinaryReader.h"
#include <cmath>
#include "Lib/BinaryTools/BinaryTools/BinaryReader.h"
Ship::BinaryReader::BinaryReader(char* buffer, uint32_t size)
{
this->buffer = std::vector<char>(buffer, buffer + size);
this->reader = std::make_shared<::BinaryReader>(this->buffer.data(), size);
}
void Ship::BinaryReader::SetEndianness(Endianness endianness)
{
this->endianness = endianness;
}
Ship::Endianness Ship::BinaryReader::GetEndianness() const
{
return endianness;
}
void Ship::BinaryReader::Seek(int32_t offset, SeekOffsetType seekType) {
switch(seekType) {
case SeekOffsetType::Current:
if (offset < 0) {
reader->SeekReverse(-1 * offset);
break;
}
reader->SeekCur(offset);
break;
case SeekOffsetType::Start:
reader->SeekBeg(offset);
break;
case SeekOffsetType::End:
reader->SeekReverse(offset);
break;
}
}
uint32_t Ship::BinaryReader::GetBaseAddress() {
return reader->Position();
}
void Ship::BinaryReader::Read(int32_t length)
{
reader->ReadFixedLengthString(length);
}
void Ship::BinaryReader::Read(char *buffer, int32_t length)
{
reader->ReadToMemory(buffer, length);
}
char Ship::BinaryReader::ReadChar()
{
return reader->ReadChar();
}
int8_t Ship::BinaryReader::ReadInt8()
{
return reader->ReadInt8();
}
int16_t Ship::BinaryReader::ReadInt16()
{
int16_t result = 0;
this->Read((char*)&result, sizeof(int16_t));
if (endianness != Endianness::Native)
result = BSWAP16(result);
return result;
}
int32_t Ship::BinaryReader::ReadInt32()
{
int32_t result = 0;
this->Read((char *)&result, sizeof(int32_t));
if (endianness != Endianness::Native)
result = BSWAP32(result);
return result;
}
uint8_t Ship::BinaryReader::ReadUByte()
{
return reader->ReadUint8();
}
uint16_t Ship::BinaryReader::ReadUInt16()
{
uint16_t result = 0;
this->Read((char *)&result, sizeof(uint16_t));
if (endianness != Endianness::Native)
result = BSWAP16(result);
return result;
}
uint32_t Ship::BinaryReader::ReadUInt32()
{
uint32_t result = 0;
this->Read((char *)&result, sizeof(uint32_t));
if (endianness != Endianness::Native)
result = BSWAP32(result);
return result;
}
uint64_t Ship::BinaryReader::ReadUInt64()
{
uint64_t result = 0;
this->Read((char *)&result, sizeof(uint64_t));
if (endianness != Endianness::Native)
result = BSWAP64(result);
return result;
}
float Ship::BinaryReader::ReadFloat()
{
float result = NAN;
this->Read((char *)&result, sizeof(float));
if (endianness != Endianness::Native)
{
float tmp;
char *dst = (char *)&tmp;
char *src = (char *)&result;
dst[3] = src[0];
dst[2] = src[1];
dst[1] = src[2];
dst[0] = src[3];
result = tmp;
}
if (std::isnan(result))
throw std::runtime_error("BinaryReader::ReadSingle(): Error reading stream");
return result;
}
double Ship::BinaryReader::ReadDouble()
{
double result = NAN;
this->Read((char *)&result, sizeof(double));
if (endianness != Endianness::Native)
{
double tmp;
char *dst = (char *)&tmp;
char *src = (char *)&result;
dst[7] = src[0];
dst[6] = src[1];
dst[5] = src[2];
dst[4] = src[3];
dst[3] = src[4];
dst[2] = src[5];
dst[1] = src[6];
dst[0] = src[7];
result = tmp;
}
if (std::isnan(result))
throw std::runtime_error("BinaryReader::ReadDouble(): Error reading stream");
return result;
}
Vec3f Ship::BinaryReader::ReadVec3f()
{
return Vec3f();
}
Vec3s Ship::BinaryReader::ReadVec3s()
{
return Vec3s(0, 0, 0);
}
Vec3s Ship::BinaryReader::ReadVec3b()
{
return Vec3s(0, 0, 0);
}
Vec2f Ship::BinaryReader::ReadVec2f()
{
return Vec2f();
}
Color3b Ship::BinaryReader::ReadColor3b()
{
return Color3b();
}
std::string Ship::BinaryReader::ReadString()
{
std::string res;
int numChars = reader->ReadInt32();
for (int i = 0; i < numChars; i++) {
res += reader->ReadChar();
}
return res;
}
std::string Ship::BinaryReader::ReadCString()
{
return reader->ReadNullTerminatedString();
}

View File

@ -0,0 +1,61 @@
#pragma once
#include <string>
#include <memory>
#include <vector>
#include "endianness.h"
#include "Vec2f.h"
#include "Vec3f.h"
#include "Vec3s.h"
#include "Color3b.h"
class BinaryReader;
namespace Ship {
enum class SeekOffsetType
{
Start,
Current,
End
};
class BinaryReader
{
public:
BinaryReader(char* buffer, uint32_t size);
void SetEndianness(Endianness endianness);
Endianness GetEndianness() const;
void Seek(int32_t offset, SeekOffsetType seekType);
uint32_t GetBaseAddress();
void Read(int32_t length);
void Read(char *buffer, int32_t length);
char ReadChar();
int8_t ReadInt8();
int16_t ReadInt16();
int32_t ReadInt32();
uint8_t ReadUByte();
uint16_t ReadUInt16();
uint32_t ReadUInt32();
uint64_t ReadUInt64();
float ReadFloat();
double ReadDouble();
Vec3f ReadVec3f();
Vec3s ReadVec3s();
Vec3s ReadVec3b();
Vec2f ReadVec2f();
Color3b ReadColor3b();
std::string ReadString();
std::string ReadCString();
protected:
Endianness endianness = Endianness::Native;
private:
std::vector<char> buffer;
std::shared_ptr<::BinaryReader> reader;
};
}

View File

@ -77,6 +77,21 @@ endif ()
source_group("Source Files\\Audio" FILES ${Source_Files__Audio} ${Source_Files__Audio__extra})
set (Source_Files__BinaryTools
"BinaryReader.cpp"
"BinaryReader.h"
"Lib/BinaryTools/BinaryTools/Binary.h"
"Lib/BinaryTools/BinaryTools/Binary.cpp"
"Lib/BinaryTools/BinaryTools/BinaryReader.cpp"
"Lib/BinaryTools/BinaryTools/BinaryReader.h"
"Lib/BinaryTools/BinaryTools/BinaryWriter.cpp"
"Lib/BinaryTools/BinaryTools/BinaryWriter.h"
"Lib/BinaryTools/BinaryTools/MemoryBuffer.h"
"Lib/BinaryTools/BinaryTools/Span.h"
)
source_group("Source Files\\BinaryTools" FILES ${Source_Files__BinaryTools})
set(Source_Files__Controller
"ControlDeck.cpp"
"ControlDeck.h"
@ -376,6 +391,7 @@ set(ALL_FILES
${Header_Files__Resources__Files}
${Source_Files__Audio}
${Source_Files__Audio__extra}
${Source_Files__BinaryTools}
${Source_Files__Controller}
${Source_Files__Controller__Attachment}
${Source_Files__CustomImpl}

View File

@ -55,7 +55,7 @@ static const char* GetGameVersionString() {
return "IQUE_TW";
case OOT_IQUE_CN:
return "IQUE_CN";
case OOT_UNKNOWN:
case UNKNOWN:
return "UNKNOWN";
}

View File

@ -37,7 +37,7 @@ enum class CutsceneCommands
Error = 0xFEAF,
};
static inline uint32_t read_CMD_BBBB(BinaryReader* reader)
static inline uint32_t read_CMD_BBBB(Ship::BinaryReader* reader)
{
uint32_t v;
reader->Read((char*)&v, sizeof(uint32_t));
@ -45,13 +45,13 @@ static inline uint32_t read_CMD_BBBB(BinaryReader* reader)
return v;
}
static inline uint32_t read_CMD_BBH(BinaryReader* reader)
static inline uint32_t read_CMD_BBH(Ship::BinaryReader* reader)
{
uint32_t v;
reader->Read((char*)&v, sizeof(uint32_t));
// swap the half word to match endianness
if (reader->GetEndianness() != Endianness::Native)
if (reader->GetEndianness() != Ship::Endianness::Native)
{
uint8_t* b = (uint8_t*)&v;
uint8_t tmp = b[2];
@ -62,13 +62,13 @@ static inline uint32_t read_CMD_BBH(BinaryReader* reader)
return v;
}
static inline uint32_t read_CMD_HBB(BinaryReader* reader)
static inline uint32_t read_CMD_HBB(Ship::BinaryReader* reader)
{
uint32_t v;
reader->Read((char*)&v, sizeof(uint32_t));
// swap the half word to match endianness
if (reader->GetEndianness() != Endianness::Native)
if (reader->GetEndianness() != Ship::Endianness::Native)
{
uint8_t* b = (uint8_t*)&v;
uint8_t tmp = b[0];
@ -79,13 +79,13 @@ static inline uint32_t read_CMD_HBB(BinaryReader* reader)
return v;
}
static inline uint32_t read_CMD_HH(BinaryReader* reader)
static inline uint32_t read_CMD_HH(Ship::BinaryReader* reader)
{
uint32_t v;
reader->Read((char*)&v, sizeof(uint32_t));
// swap the half words to match endianness
if (reader->GetEndianness() != Endianness::Native)
if (reader->GetEndianness() != Ship::Endianness::Native)
{
uint8_t* b = (uint8_t*)&v;
uint8_t tmp = b[0];
@ -99,7 +99,7 @@ static inline uint32_t read_CMD_HH(BinaryReader* reader)
return v;
}
void Ship::CutsceneV0::ParseFileBinary(BinaryReader* reader, Resource* res)
void Ship::CutsceneV0::ParseFileBinary(Ship::BinaryReader* reader, Resource* res)
{
Cutscene* cs = (Cutscene*)res;

View File

@ -10,7 +10,7 @@ namespace Ship
ResourceFile::ParseFileBinary(reader, res);
while (reader->GetBaseAddress() % 8 != 0)
reader->ReadByte();
reader->ReadInt8();
while (true)
{

View File

@ -22,13 +22,12 @@ namespace Ship
{
Resource* ResourceLoader::LoadResource(std::shared_ptr<File> FileToLoad)
{
auto memStream = std::make_shared<MemoryStream>(FileToLoad->buffer.get(), FileToLoad->dwBufferSize);
auto reader = std::make_shared<BinaryReader>(memStream);
auto reader = std::make_shared<BinaryReader>(FileToLoad->buffer.get(), FileToLoad->dwBufferSize);
Endianness endianness = (Endianness)reader->ReadByte();
Endianness endianness = (Endianness)reader->ReadInt8();
for (int i = 0; i < 3; i++)
reader->ReadByte();
reader->ReadInt8();
reader->SetEndianness(endianness);

View File

@ -20,6 +20,6 @@
#define OOT_PAL_GC_MQ_DBG 0x917D18F6
#define OOT_IQUE_TW 0x3D81FB3E
#define OOT_IQUE_CN 0xB1E1E07B
#define OOT_UNKNOWN 0xFFFFFFFF
#define UNKNOWN 0xFFFFFFFF
#endif

View File

@ -0,0 +1,276 @@
## Ignore Visual Studio temporary files, build results, and
## files generated by popular Visual Studio add-ons.
# User-specific files
*.suo
*.user
*.userosscache
*.sln.docstates
# User-specific files (MonoDevelop/Xamarin Studio)
*.userprefs
# Build results
[Dd]ebug/
[Dd]ebugPublic/
[Rr]elease/
[Rr]eleases/
bld/
[Bb]in/
[Oo]bj/
[Ll]og/
# Visual Studio 2015 cache/options directory
.vs/
# Uncomment if you have tasks that create the project's static files in wwwroot
#wwwroot/
# MSTest test Results
[Tt]est[Rr]esult*/
[Bb]uild[Ll]og.*
# NUNIT
*.VisualState.xml
TestResult.xml
# Build Results of an ATL Project
[Dd]ebugPS/
[Rr]eleasePS/
dlldata.c
# DNX
project.lock.json
project.fragment