Base64 Encode/Decode

[Godlike-Productions]

Does anyone know if there is base64 decode and encode functions in Ctrlr?

I'm building a chain manager for the Eventide H9000 and as part of that there are small gzip files that are encoded in base64 and saved within a json file. I need to be able to decode them, change a couple of parameters, and then re-encode them.

[damiensellier]

There's a base64 class in Juce with encode/decode functions:
https://docs.juce.com/master/structjuce_1_1Base64.html

If it's not bound to Lua yet I'll take a look at it tonight.

[damiensellier]

The encode decode from/to base 64 is already available to Lua when processed from a memory block :
https://github.com/damiensellier/CtrlrX/blob/1ffa0a27ecaf5fd45fd57ed64df9981ae5387bd8/Source/Lua/JuceClasses/LMemory.cpp#L260

I did not see the whole base 64 class bound so I'll add it to the to do list.

Let me know if you can convert it if loaded as memory block 😉

[Godlike-Productions]

I'll try to look tomorrow. I'm not sure if I'll parse it through a memory block, though I may need to in order to write the binary file.

[damiensellier]

these are the base64 converters available in JUCE (v8) :

https://docs.juce.com/master/structjuce_1_1Base64.html

[damiensellier]

Hi,
Did you get it working?

[Godlike-Productions]

I have been out with Influenza, so haven't done much work on this. I've come up with another problem with the json.decode not being able to unpack the json for these Chain files. This is a bigger and more complex json file than the other ones I was working on (Preset). I'm still trying to identify why, but it looks like it's consistently dropping the closing brackets off the json file in the file open process. This maybe down to how the H9000 encodes the json file (there may be a LF, CR or some other termination character that triggers the parser) or it could be another issue, such as the json structure being too deeply nested, or too long

Anyway, the long and short is that I haven't been able to extract the base64 strings yet to test. I may need to manually parse the files as strings and do a bunch of search's if I can't identify the problem.

It's so frustrating as I've worked out the method after 3 years on how to modify these files, and am temporarily being defeated by some weirdness. I'll look at it this week.

[damiensellier]

No problem, I read on the JUCE forum that the decode from base64 is very strict and fails easily if the syntax is not perfect from the source file. Someone there suggested a more tolerant class for this particular task.

Take care, get well soon!

[dnaldoog]

Maybe just do it all in lua?

https://gist.github.com/mmurdoch/3806239

[dnaldoog]

It will be possible to use JUCE zlib in CtrlrX.

I can also import GZIP files (.gz) but cannot write to .gz, so the user could in theory load your JSON files now contained in .gz and then the user would resave as .zlib in future versions of your software. You could also perhaps create your own extension, or even use .gz although it wouldn't really be .gz of course.

Have also implemented base_64 encoding/decoding for CtrlrX 5.6.34

Note, from google:

It is a well-documented fact that the JUCE framework, 
including version 6.0.8, uses zlib for its compression, 
despite the naming of certain classes. 
For many years, users have pointed out that JUCE's 
GZIPCompressorOutputStream and GZIPDecompressorInputStream 
classes actually implement zlib compression.
 

[Godlike-Productions]

The H9000's files are json files that are gzipped (using .9kp and 9ks extemsons), so its important to be able to write to them. What complicates it is that for some file algorithms (for example Black Hole, Time Factor) and for their chain representation there is a second version of the preset that is saved to a json key. That preset is another gzipped json file that have been encoded by base64 so that it will appear as a string for the json encoding. So the H9000 is expecting to load a gzip file.

Because of the way that Eventide do their checking the algorithm ID needs to match up both in the outer json file, but also for the inner json file.

For factory objects algorithms don't move, but for custom algorithms, like the ones I make, the presets and chains will not load if the algorithm ID and algorithm name don't match at both levels, and users can load them anywhere they want. On top of that they also have a checksum, which fortunately doesn't seem to include the name and ID's, because for the life of me I cannot determine how they calculate the CRC32. It's important to be able to help my customers load presets and chains in the easiest way possible because in addition to the dsp algorithm they really, really want presets and chains that I build to demonstrate them.

Being able to do it all without launching an external gzip application is by far the preference, but I have found a solution launching gzip externally that works for the presets.

My challenge right now is being able to parse the json file for the chain object as it fails to parse every time. I've been out with the flu for the last 2 weeks so haven't been able to work on this, but it looks like it is dropping the closing brackets off the json file during the file load process (prior to attempting the json.decode), so the json decode fails and I can't even attempt to load the base64 object to a variable to decode, gunzip and then load and parse as a json file for me to modify 1 integer, export, gzip, load, base64 encode and replace in the original json file before gzipping again.

If I can't solve the json parsing issue, I'll just use string functions to chop up the json file as a string and do it that way. It's way more work, but should be pretty bullet proof. The comments in the json.decode and json.encode modules in ctrlr acknowledge that the function may not work for all json files, particularly with deep heirarchies, so it could just be that limitation that I'm butting up against, or it could be some type of bug in how eventide build their json files.

It's horrifically convoluted and I think is borne from older technology and they way they have ported code from plugins and other products into the H9000 including preset parsing for those particular algorithms.

[dnaldoog]

I might as well do a PR for the zlib binding anyway, but it looks like you won't be able to use it.

The latest build of CtrlrX now supports XML. Maybe ditch JSON and use XML instead?

I've got a panel somewhere that will show you how to implement it. Will check when I get home.

[Godlike-Productions]

For this application I'm stuck with json as that is the format that the H9000 loads

[dnaldoog]

The whole thing sounds pretty scary 😨

[dnaldoog]

How are you base64 decoding and encoding?
How are you using gzip?

Maybe there’s a problem with the base64 library you are using?

you could try this build:
https://github.com/user-attachments/files/22709283/CtrlrX.5.6.34.Base64.support.zip

    local base_64_string = utils.base64_decode(input)
    local base_64_string = utils.base64_encode(input)

[Godlike-Productions]

I haven't got to base64 yet. I'm still working out how to parse the json file.

The structure is like this

gzip file
-> json file
--> Bunch of tags
--> id
--> base64 string
---> gzip file
----> json file
-----> bunch of tags
-----> id

--> Bunch of tags
--> id
--> base64 string
---> gzip file
----> json file
-----> bunch of tags
-----> id

This is how I'm doing gunzip

tempOutput = io.popen('gzip -d -S 9kf -k "'..fileToConvertPath..'"')

To gzip
tempOutput = io.popen('gzip -S .9kp -k -f "'..destinationPath..'*."')

This is the library I'm looking at using for base64

--#!/usr/bin/env lua
-- Lua 5.1+ base64 v3.0 (c) 2009 by Alex Kloss <alexthkloss@web.de>
-- licensed under the terms of the LGPL2

-- character table string
local b='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'

-- encoding
function enc(data)
    return ((data:gsub('.', function(x) 
        local r,b='',x:byte()
        for i=8,1,-1 do r=r..(b%2^i-b%2^(i-1)>0 and '1' or '0') end
        return r;
    end)..'0000'):gsub('%d%d%d?%d?%d?%d?', function(x)
        if (#x < 6) then return '' end
        local c=0
        for i=1,6 do c=c+(x:sub(i,i)=='1' and 2^(6-i) or 0) end
        return b:sub(c+1,c+1)
    end)..({ '', '==', '=' })[#data%3+1])
end

-- decoding
function dec(data)
    data = string.gsub(data, '[^'..b..'=]', '')
    return (data:gsub('.', function(x)
        if (x == '=') then return '' end
        local r,f='',(b:find(x)-1)
        for i=6,1,-1 do r=r..(f%2^i-f%2^(i-1)>0 and '1' or '0') end
        return r;
    end):gsub('%d%d%d?%d?%d?%d?%d?%d?', function(x)
        if (#x ~= 8) then return '' end
        local c=0
        for i=1,8 do c=c+(x:sub(i,i)=='1' and 2^(8-i) or 0) end
        return string.char(c)
    end))
end

-- command line if not called as library
if (arg ~= nil) then
	local func = 'enc'
	for n,v in ipairs(arg) do
		if (n > 0) then
			if (v == "-h") then print "base64.lua [-e] [-d] text/data" break
			elseif (v == "-e") then func = 'enc'
			elseif (v == "-d") then func = 'dec'
			else print(_G[func](v)) end
		end
	end
--else
--	module('base64',package.seeall)
end

[dnaldoog]

So for Windows, you're using something like this?
https://gnuwin32.sourceforge.net/packages/gzip.htm
and then somehow distributing that with the Ctrlr app?

[damiensellier]

Don't forget the gzip class in juce works on a data stream. It's not like a file zip/unzip funtion.
It uses zlib which can also introduce issue to decode gzip files with a header.

On the juce forum there's a lot of discussion about how not to use it.

If your source files are true gzip I wonder if I would not use an external library to inflate/deflate them instead of juce gzip class. There's also a zip class in nice if needed.

https://forum.juce.com/t/gzipped-file-to-xml-back-to-gzip-help-needed/51349

https://forum.juce.com/t/gzipdecompressorinputstream-from-binarybuilder/13000/6

[Godlike-Productions]

Yeah, I instruct my users to install gnuwin32 and then add a path statement so that I can call it from a Dos Window without needing the path.

[dnaldoog]

Don't forget the gzip class in juce works on a data stream. It's not like a file zip/unzip funtion. It uses zlib which can also introduce issue to decode gzip files with a header.

This is the PR I made for Zlib in CtrlrX lua. Not really of any use in this situation, but anyway ...

[dnaldoog]

Some interesting reading on Zlib and Gzip and friends here.

[Godlike-Productions]

I've managed to fix the string, decode the json and extract the base64 string. Directly decoding the string doesn't work with either base64 algorithm, but I think I first need to declare a MemoryBlock and decode into that, then save that memory block to a gzip file.

It's late, so I'll try to find some time tomorrow to test it.

If anyone wants to have a go the base64 encoded gzip is

H4sIAAAAAAAEA5RSy06FMBDd+xWmayAtIK+lMe5cGO/OGDKUBqqUYltUcsO/296k6ApxVjPnnJnM64xupdQmlM1rhFGVRgV2RkqcJ2mRJQG6V+zd0wVOIxygB9kyD9nw0ftxdGPDU6+Y7j0WktilwNDVHfARVWfULBNo7Tw9N5oqPhkuLTPOwxCgDxhmZtln/LIGiI9b2r44usjFbNixwnI2/6psgA8HezZMTLIWdknHWvlkpm7VUgv+9WcCwdhNajcDxgB982vGkWUuRtISE5IRr6jInqZlFJY9Qb9ow+xBufaqxN1TMAMtGHANT0p2CkRNFU1iVIWkyPK0LFKcBxvHW1RlWVwmefYDjiDsitDdYh1Or5/Y2CI7maanZdoeLLbArzdxN1jXq28AAAD//wMAoflsYLgCAAA=

A base64 decode should give a gzip file. When it is unzipped it should give a json file

{"Boost-obj.0":4.800000190734863,"Freq-obj.0":804.0,"Mode-obj.0":0,"Q-obj.0":2.50,"Thresh-obj.0":-12.0,"alg_gain":{"bypass":{"subscription":null,"values":[0]},"in_gain":{"subscription":null,"values":[0.0]},"mute":{"subscription":null,"values":[0]},"out_gain":{"subscription":null,"values":[0.0]},"tails":{"subscription":null,"values":[0]},"tempo_mode":{"subscription":null,"values":[0]},"wet_dry_mix":{"subscription":null,"values":[100.0]}},"attack-obj.0":0.1000000014901161,"attack:1-obj.0":0.1000000014901161,"decay-obj.0":0.1000000014901161,"hysteresis-obj.0":3.0,"metadata":{"program_crc32":-1867498407,"program_id":6629376,"program_name":"Dynamic Send"},"scType-obj.0":2,"subscriptions":{}}

This tool does the base64 decode and gunzip in 1 step - https://codebeautify.org/gzip-decompress-online

[damiensellier]

Check memory output stream class, it has a function to copy the stream to memoryblock.

Writes data to an internal memory buffer, which grows as required.

The data that was written into the stream can then be accessed later as a contiguous block of memory.

My bet is all that JUCE gzip required that:
https://docs.juce.com/master/classjuce_1_1MemoryOutputStream.html

you should use juce::MemoryOutputStream or a similar juce::OutputStream in combination with JUCE's GZIP classes for your compression/decompression tasks.

juce::MemoryOutputStream is the correct destination when you want to compress data into a memory buffer (juce::MemoryBlock) instead of directly to a file.

For compression, you'll wrap your MemoryOutputStream with a juce::GZIPCompressorOutputStream. For decompression, you'll use a juce::MemoryInputStream to read the compressed data and wrap it with juce::GZIPDecompressorInputStream.

GZIP Compression Example (Data to Memory)

​To compress data (like a juce::ValueTree or a juce::String) and store the resulting GZIP binary data in memory:

• ​Create a destination MemoryOutputStream: This will hold the compressed GZIP data.
• ​Wrap it with a GZIPCompressorOutputStream: This stream performs the compression.
• ​Write your uncompressed data to the GZIPCompressorOutputStream.
• ​Flush and retrieve the compressed data.

​GZIP Decompression Example (Memory to Data)

​To decompress the GZIP data stored in a juce::MemoryBlock back into its original form:

• ​Create a MemoryInputStream: This feeds the compressed data.
• ​Wrap it with a GZIPDecompressorInputStream: This stream performs the decompression.
• ​Read the decompressed data into a new MemoryOutputStream or directly into a juce::String.

There's also juce::FileInputStream and juce::FileOutputStream instead of the Memory...Stream classes if you want to read/write directly to disk.

[dnaldoog]

[damiensellier]

Awesome!
If you implement your new code to my branch https://github.com/damiensellier/CtrlrX/tree/5.6.34-Maintenance-Build-Workflow instead of the older 5.6.34 maintenance you'll get all the latest updates + the automatic online compilation/installer process via actions. You won't need to build the exe on your computer they are going to be available in your action page.

I'll try to include all your pr to this branch ASAP but i'm pretty busy right now.

[damiensellier]

I mean, to make it short, instead of starting from a copy of 5.6.34-maintenance, start from a copy of https://github.com/damiensellier/CtrlrX/tree/5.6.34-Maintenance-Build-Workflow

[dnaldoog]

Finally got it working.

1. Loads and Extracts base64_encoded Eventide H9000 9ka gzip files from JSON and prints to console.
2. Can recompress globally extracted data and save to file.

H9000 Settings_5_1.zip

You can to build CtrlrX from this branch or download prebuilt exe

• https://github.com/dnaldoog/CtrlrX/tree/5.6.34-Maintenance_ZLIB

---

Because of the way that Eventide do their checking the algorithm ID needs to match up both in the outer json file, but also for the inner json file.

For factory objects algorithms don't move, but for custom algorithms, like the ones I make, the presets and chains will not load if the algorithm ID and algorithm name don't match at both levels, and users can load them anywhere they want. On top of that they also have a checksum, which fortunately doesn't seem to include the name and ID's, because for the life of me I cannot determine how they calculate the CRC32. It's important to be able to help my customers load presets and chains in the easiest way possible because in addition to the dsp algorithm they really, really want presets and chains that I build to demonstrate them.

I haven't checked any of this functionality.

[damiensellier]

@Godlike-Productions you can compile on your PC to be 100% sure about the source.
The source code including the base64 and zlib support is here : https://github.com/damiensellier/CtrlrX/tree/5.6.34-Maintenance
Hope it helps ;)

[Godlike-Productions]

I don't for a second think there's a trojan, just wanted to let you know it got flagged.

[Godlike-Productions]

I think there's bug with the decompressGzip in the exe provided by @dnaldoog

This line
str = inputBlock:decompressGzip()

seems to drop some characters from the end of the memoryBlock during the gunzip process, sometimes 4 characters, up to 12 characters.

I've verified this by opening the file in LUA, outputing the file to a HexString to the console prior to the decompressGzip command, copying that into a file editor, converting to ASCII, and saving it back to be manually gunzipped and compared and all of the data is there prior to this line, but after the line, the json file has dropped the last few characters in some cases, when I do

-- output file to console as hex fileStr = inputBlock:toHexString(1):upper() console (fileStr)

I'm wondering if it has something to do with not decoding a block of data that isn't full, or not writing the final data into the memoryBlock.

Here's the whole file open process up to where I find the error.

savePath = L(panel:getLabelComponent("filePathLabel"):getPropertyString("uiLabelText"))


fileToConvert = utils.openFileWindow("Open Chain File to read (*.9kf)",File(savePath),"*.9kf",true)
fileToConvertPath = fileToConvert:getFullPathName()
chainPath, chainFile, chainExtension = SplitFilename(fileToConvertPath) -- this is needed for saving
if fileToConvert:existsAsFile() then
    --console ("Chain File Exists: "..fileToConvertPath)
    tempFileName = fileToConvert:getFileName()
    --tempOutput = io.popen('gzip -d -S 9kf -k "'..fileToConvertPath..'"')
    
else
    return
end --if

local inputBlock = MemoryBlock()
local testHeader = MemoryBlock("1F 8B 08 00")

    --console ("File Exists")
    fileToConvert:loadFileAsData(inputBlock)
    if inputBlock:getSize() > 0 then
        if inputBlock:getRange(0,4):toHexString(1):upper() == testHeader:toHexString(1):upper() then
            --console("Step 1: Decompressing outer gzip file...")
            str = inputBlock:decompressGzip()
        end --if
    end --if

    -- output file to console as hex
    fileStr = inputBlock:toHexString(1):upper()
    console (fileStr)

    console (str)
    strLen = string.len(str)
    console (String(strLen))

I'm not sure if you guys have any insight, or ability to help resolve the error, but I'm thinking I may need to go back to the dos command method, if we can't get to the bottom of it.

I'm happy to post the bpanelz file and some test data if its helpful (though the code is still messy and full of commented test cases right now).

It is so close to working.

I'm 99% sure this has been the source of the problems I've been seeing below where I need to essentially repair the file, but as json files aren't ordered, what gets dropped at the end can pretty much be any data, so it's not possible to build a repair for any file.

[dnaldoog]

Hi Jason,

Can you post a test panel that duplicates the error?

I'm pretty sure I just luabinded the JUCE code for gunzip to lua, but maybe there’s a bug in there somewhere?

if you don’t want to publish the code, I can send you my email address.

[Godlike-Productions]

Yep, I'll build a test panel tomorrow amd post it.

[Godlike-Productions]

Big Drum Room.zip
This is an example chain file.

When you load it the json file fails because it doesn't load the empty slot ,{}]} at the end. I'm currently running a check for this case, and concatenate that string at the end if it doesn't end with that.

This file has 3 of the 4 slots filled.

I still haven't tested chains with 1, 2 or 4 slots filled, but I suspect 1 will end with ,{},{},{}]}, 2 will end with ,{},{}]} which will require a more intensive check (I was going to count the number of times "preset":" appears to determine how the string should end.

For the zip file included here, if you add ,{}]} to the end of the string, then the json file will load fine and you can extract the json data.

This is the gunzipped json file (below)

I'm most interested in the section starting with "slots"

{"dsp_mode":0,"functions":[{"subscription":null,"value":0.0},{"subscription":null,"value":0.0},{"subscription":null,"value":0.0},{"subscription":null,"value":0.0},{"subscription":null,"value":0.0},{"subscription":null,"value":0.0},{"subscription":null,"value":0.0},{"subscription":null,"value":0.0}],"master_gain":{"bypass":{"subscription":null,"values":[0]},"in_gain":{"subscription":null,"values":[0.0]},"mute":{"subscription":null,"values":[0]},"out_gain":{"subscription":null,"values":[0.0]},"tails":{"subscription":null,"values":[0]},"wet_dry_mix":{"subscription":null,"values":[100.0]}},"metadata":{"crc32":-412343991,"effect type":{"Delay":0.8999999761581421,"Distortion":1.0,"Modulation":1.0,"Reverb":1.0},"fxchain_id":0,"instrument type":{"Bass":1.0,"Brass":1.0,"Guitar":0.8999999761581421,"Keyboard":1.0,"Vocal":1.0},"is_factory":false,"name":"Big Drum Room","product type":{"Emulation":1.0,"H8000":1.0,"H9":0.8999999761581421,"H9000":1.0,"Space":1.0},"program_names":"Dynamic Send, Shimmer, Dream Chamber","software":"2.0.5[2] (ade503dd6)","version":2},"routing":{"locations":[0,5,4,-1],"routes":[[[0,[0]],[1,[1]]],[[0,[0]],[1,[1]],[8,[2]],[9,[3]]],[[0,[0,8]],[1,[1,9]]]]},"slots":[{"preset":"H4sIAAAAAAAEA5RSy06FMBDd+xWmayAtIK+lMe5cGO/OGDKUBqqUYltUcsO/296k6ApxVjPnnJnM64xupdQmlM1rhFGVRgV2RkqcJ2mRJQG6V+zd0wVOIxygB9kyD9nw0ftxdGPDU6+Y7j0WktilwNDVHfARVWfULBNo7Tw9N5oqPhkuLTPOwxCgDxhmZtln/LIGiI9b2r44usjFbNixwnI2/6psgA8HezZMTLIWdknHWvlkpm7VUgv+9WcCwdhNajcDxgB982vGkWUuRtISE5IRr6jInqZlFJY9Qb9ow+xBufaqxN1TMAMtGHANT0p2CkRNFU1iVIWkyPK0LFKcBxvHW1RlWVwmefYDjiDsitDdYh1Or5/Y2CI7maanZdoeLLbArzdxN1jXq28AAAD//wMAoflsYLgCAAA=","preset_metadata":null,"program_metadata":{"crc32":-1867498407,"dsp_mode":0,"factory":false,"favorite":false,"id":6629376,"inputs":2,"inputs / outputs":{"2/4":1.0},"is_exportable":true,"name":"Dynamic Send","outputs":4,"rating":0}},{"preset":"H4sIAAAAAAAEA5RUW2+jRhR+76+oeI4qGMBd/Ob4gnE9g2ITg6kqa7iEDB4uMhBgVvnvPUPaJKuVutu38bl8l3OO+apQnl0yykpl/lWJxpo2jXw1XdTEN1a3rIJM2XF+p7xQ3qWQ/VP96/VOYeV7238X/zaVF12b/hxw1bX/C7mljP+k5jYt6upSVMmPpWjSY5+2l+Q2Xgo2/FC7pqrSKXQVaUsT2lLZwprLE43b6jYq8/bWpXdKSQtgV45bB+P1QblT6lvaAA9LlPlMm+mG+buMVUkXt7+2Yz1J3VrKXANoiGc3WlziW6wjZa5/QbMv5kyffWQ+YD4F/+V8ZkWR3hQJxLsMNvjGDXLScfcc2clLXHCV2tpTVGza8OjMHE4q6mttGDhsv9w9xzapnLwacL5Q9/micYpdf0YZc/P14Hq4w8uencv7nG6sKtIPT7Ge1MlbT7/Pzz3OcY+Fg0j+OBLPQe7qLDHM2N4ApzVG/qYBfKh9RHiVCcDUyOpqYu8KnNlb7fbwbe1ydz2jjQl9iCyNgYhFj1ePuutlAourQUBTaA/PKfjBR6MHHzzd3o9hQAT1rS7Q73lov/PKfBXaROLp7tEwSC7xFgh7C0G8tUZYz6hvDIFOeFRi6X3yDbHxuxjiTWQfxol7OXE3/2APgDn1Rf4hn/LeAnzLmWotDeRMsbnPHcndE/BDxFmQPB5B30u0Dd/mBO8kePhOS6SfVMwtEfqmkPuacMv761m/SlwdfmuuB3MVsYpFZpL8wQDcXmoL0KkLNxbMaAG1i85drUcs1ogI8C7WOrw1MvYstkH3yRKJvRmTT/WAOYBeRGBesF9Dzk3eRbzd8Rid1ABpdbqEXSwNDWZrwI5grg+m661hz86kIy5OIgzknt/v0Dk+auRwmu6wjwvrmvgEsPgV/LEzstp9cULgV6X+ge+LzNyX39yuoEHNoXaUO8VwU+BfhTs0ycoRrtSHhpdEs9oIHaCuLs7+IMJjzxJb6xMGej1H3oYa67z7mGnI45LUETKYyxzhlOof8J/+/EWU36bX11/+BgAA//8DACfgIChqBQAA","preset_metadata":{"is_factory":true,"name":"SHIMMER","preset_id":6163457,"product type":{"H9":1},"program_crc32":382685636,"program_id":6163456,"program_name":"Shimmer"},"program_metadata":{"crc32":382685636,"dsp_mode":0,"effect type":{"Reverb":1.0},"factory":true,"favorite":false,"id":6163456,"info":"We don’t have proof, but we’re pretty sure this is what the guitars sound like in heaven. Set the [A-PCH] and [B-PCH] to just above and below 1200c, turn the [DELAY] all the way down, and everything else all the way up. Oh, and remember to walk toward the light. \n\n \nMix - [MIX] : wet/dry mixer, 100% is all wet signal \n \nDecay - [DECAY] : arbitrary 0-100 decay (less decay will also take away reverb attack) \n \nSize - [SIZE] : size of the reverb \n \nDelay - [DELAY] : post reverb and pre pitch-shift delay time in milliseconds or note-based in tempo mode \n \nLow Band Decay - [LO-DCY] : amount of post reverb and pitch-shifter low band signal (this is in the feedback path) \n \nHigh Band Decay - [HI-DCY] : amount of post reverb and pitch-shifter high band signal (this is in the feedback path) \n \nPitch Shift A - [PICH-A] : Pitch-shifter A pitch in cents (500c=P4th, 700c=P5th, 1200c=1 Octave, 1900=1 Octave+P5, 2400=2 Octaves) \n \nPitch Shift B - [PICH-B] : Pitch-shifter B pitch in cents (500c=P4th, 700c=P5th, 1200c=1 Octave, 1900=1 Octave+P5, 2400=2 Octaves) \n \nPitch Decay - [PITCH] : The PITCH-DECAY knob controls the amount of pitch shifting in the reverb tail. It increases from 0 to 100. Beyond 100 are two FREEZE modes. PITCH FREEZE locks out the pitch shifters, but feeds the reverb, allowing you to freeze the Shimmer pitch climb at opportune times. PITCH+VERB FREEZE freezes everything (pitch and reverb) for dry soloing on top of the frozen reverb. \n \nMid Band Decay - [MIDDCY] : amount of post reverb and pitch-shifter mid band signal (this is in the feedback path) \n \n\n","inputs":2,"inputs / outputs":{"2/2":1.0},"is_exportable":false,"name":"Shimmer","outputs":2,"product type":{"H9":0.90,"Space":1.0},"rating":0,"runs_at_96k":false,"six_character_abbreviation":"Shimmr","weight":4}},{"preset":"H4sIAAAAAAAEA5RSy27DIBC89ysqzrFlsI0f1/YvqsrChjhUYCOM06ZR/r1ACs6pSTnBzM7uzi5nQMTYjYRPoD2D/qTIsrjbsvbLoLkyfLbMtAqxA0ciVmbZt+z9sgN8irK/g1MfLlfDHks8r+ZfmQ3h4sGeDZNq7uRM77cCncdPZjqqT53kX3d7L6EzakWU7/fJ3H+kGWhrC1pEnGBA8rTK/CkqXOcVxJ5GgYb5ryAPSACKAECfcxScspi1zJzssNW1JTxyDCr/FDEAFrXjReSTMsWNO3XWIFzCvNoBSZkyh1gEpqWVSGYIJYa4gSg9j5rIbtBDjkCbNLgqMcobGxYoTkGLamTzWacBnIi0GwCvmhH5/HIgsmca2NFJTcxmCvqe5aLIFFzAqy1lJxqhwhnRdNhmjALSbb27mIV/syCrfXK9+fei2wjsI27/tvtltstNg12hY1xe0vh3XF1y5ePmHH95+gEAAP//AwAnF1mFcwMAAA==","preset_metadata":{"program_crc32":-967562390,"program_id":2821376,"program_name":"Dream Chamber"},"program_metadata":{"crc32":-967562390,"dsp_mode":0,"effect type":{"Delay":0.90,"Reverb":0.60},"factory":true,"favorite":false,"id":2821376,"info":"Chamber effect (delays between diffusion and verb). Stereo in, stereo out.","inputs":2,"inputs / outputs":{"2/2":1.0},"instrument type":{"Guitar":0.90},"is_exportable":true,"name":"Dream Chamber","outputs":2,"product type":{"H8000":1.0},"rating":7,"runs_at_96k":true,"weight":1}},{}]}

json key "preset" gives the base64 encoded gzip file of the preset json file, which I need to also get into to modify the id (I call it the inner id) and I also need to set the other "id" tag (not in the base64 bit) which I call outer id. These both need to be set to user definable id's (there's a formula for decoding these which is below)
I only need to process presets where oldBank is >100
And in those cases the user changes oldAlgNum and oldPreset to the new values in the range of 00 to 99. These are determined by where the user loads a particular algorithm into the H9000 (these just get incremented as each new algorithm is loaded, so if users load them in different orders, they will get different id numbers.) If the algorithm numbers of preset numbers don't match, then the chain or preset isn't loaded.

You can navigate to the different slots by just referring to table elements 1, 2, 3, 4 in place of a specific json key.

Currently the H9000 has a maximum of 4 algorithms loaded in a chain, but this may change in the future. An algorithm can be saved without a preset in the chain (a pseudo unnamed preset is created in the preset tag), but it may also be associated with a preset, which are saved parameters of an algorithm, and can have a preset name and preset id.

The code below is from processing preset files (not chain files) so the json tags won't work for chain files, but the logic for the bank and algorithm numbers is the same.

oldProgramID = jStr[metaKey][programKey]
    oldPresetID = jStr[metaKey][presetKey]
    presetName = jStr[metaKey][presetNameKey]
    algoName = jStr[metaKey][algoNameKey]

    oldBank = bit.band(bit.rshift(oldProgramID,16),0xff)
    oldAlgNum = bit.band(bit.rshift(oldProgramID,8), 0xff)
    oldPreset = bit.band(oldPresetID, 0xff)

[dnaldoog]

Will look at it tomorrow- maybe json.decode is failing like you had suggested originally, because the nesting is too deep?

[dnaldoog]

Perhaps you already read this?

I tried some fixes but no dice! Tricky stuff all right.

[Godlike-Productions]

I think I've finally managed to crack it open, at least for reading the files. Thanks everyone for your help and inspiration. I've just about finished the 1 case where there are 3 of the 4 slots filled, for reading the data. I still have heaps to do for the other cases, and still need to do all the modulator interaction, and then the daunting task of modifying the relevant bits of data and reassembling them all into a file that can be loaded into the H9000.

I did find there is actually a 3rd level of encoding which is an encapsulated preset from the Eventide pedals/plugins, which will be handy if I ever want to migrate patches between Eventide products, but that will be a future enterprise. The in line gzip and base64 decoding is working a treat and I'll probably migrate my old code across to make things more steamlined.

I'll post some more code once I've cleaned it up, and look forward to the next 5.6.34 ctrlrX distribution so I have it on OSX as well. I'm not configured for compiling from source at the moment.

[dnaldoog]

That’s great to hear that you are making progress. It’s an incredibly complex implementation for sure. Good luck with it.

[Godlike-Productions]

Nearly there - just got to write the file save portion and then test the re-encoding.

Decoding is working well.

[Godlike-Productions]

I have stumbled into another issue.

The exported file contains escape characters, so / instead of / and \n

I also have an issue with assembling the final file, but that one I'll track down.

[Godlike-Productions]

The escaped characters don't seem to be an issue. The H9000 correctly interprets them, even though it doesn't write fully compliant json files (the characters don't seem to be escaped when it outputs them)

[dnaldoog]

I created an installer for VST3,VST2 and EXE (64 bit) here:

https://github.com/dnaldoog/CtrlrX/releases/tag/5.6.34_ZLIB_Oct24

[Godlike-Productions]

For json.encode and json.decode, are these bound to juce functions directly?

I wanted to try the http://dkolf.de/dkjson-lua/ library as it doesn't havhe same empty array issues as json4lua as per http://lua-users.org/wiki/JsonModules, but I can't get the global module json to invoke the dkjson functions, and when I commented out json4lua and dkjson modules in their entirety, the json.encode and json.decode still ran indicating that these are calling from somewhere else, not from the lua modules.

I've managed to get the chain editor working for some of the chain files, but it seems Eventide are not consistent about how they encode the json.

Rant mode on

For some files the slots are array elements, but for others, they are tagged as keys within a single array element, and what's worse, is that they don't always use the lowest numbers of slots, instead creating a sparse array/table. I have one chain that has slots 0, 1 and 3 with entries but slot 2 is empty, which vastly complicates the checking. I think if some string elements on the H9000 contain square brackets, Eventide encode the files differently. This comes from legacy code in an info tag that probably harks back to the H3000. I wish they had just fixed the text elements in their database and standardized the data format rather than just building edge cases for encoding old strings that actually have nothing to do with the dsp code and is essentially just cosmetic.

I'm having to build additional logic to check stuff on the file as a string before I can decode the json file for each of these issues I come across which is infutiating. I was hoping to try a more robust/tolerant json.decode function to hopefully open the files and allow me to do the checks on content of json keys rather than very involved string searches, but I think there is generally an issue with opening json (and probably ascii, and maybe data) files that have square brackets before we even get to the decoding.

A few things I've learnt about json files:

1. The ordering of the json file and where the keys appear does not matter. If a decoder/encoder is compliant it should find the data from the key alone. This makes doing string checks far more challenging prior to decoding the json file.
2. In Lua the most recently changed data seems to get encoded first from the table that contains the decoded json data, so if you change data, your encoded json file will look very different to the original file.
3. Compliant json files should have escape characters for a bunch of characters (such as ASCII characters 0-31, /, , new line.) Sometimes people build noncompliant json files.
4. JSON files support a null data element that json4lua doesn't necessarily handle well. The dkjson library does have a function for handling null characters (comparison to the read file, and insertion on the file encoding as well as creation from scratch.
5. Numerical data is not well supported in json. Generally it should be encoded as a string.

Anyway, I'm calling it a night and will get back to elaborate string searches tomorrow to deal with how Eventide have built their json files.

Rant mode off.