make ECUID-keylist db external

[fenugrec]

needing to recompile to add keysets is pain.

• .ini file with name-value pairs ? mediocre
• any "noSQL" key-value store is not really a good fit. gdbm, tdb, berkeley db etc. fall in this category
• full relational db ? sqlite, etc... nice API for queries etc, but needs work to import/export human-readable dumps
• csv ? better probably, but needs some supporting code for queries
• some format that has at least some library support for "real" db queries, maybe https://github.com/PsychoCod3r/libcsv/ ? or https://github.com/rgamble/libcsv (low-level backend, callbacks per field and per record) ?
• storing as a "proper" db format ? cons = overhead, not easily (?) text-versionable... could be a challenge to automate.

Requirements :

• fields that nisrom can populate: romsize, cpuid (to determine correct kernel), keyset
• extra fields that may need manual entry : ECUID, eepr function addrs
• MD5 of stock ROM, to identify a dump (this would be used on the nissutils or RR side)
• must be text-versionable

Tasks :

• tweak nisrom to produce clean, CSV output ( improve output formatting nissutils#9 )
• extract ECUID from dump filename if possible, see nisrom: extract ECUID from filename nissutils#14
• move ECUID and keyset query code and internal db to separate file (could be a different repo eventually)
• prepare tools to generate csv (probably from a batch run of nisrom)
• populate db from external csv and query that instead

[Pytrex]

I'm heavily biased, but using an external csv would be perfect! They're extremely easy to work with, both within the code and for the user. Especially if you'd want to increase the supplied data to every field listed in the bonus section, csv would be a breeze.

[fenugrec]

external csv would be perfect!

It is a strong candidate. I'm still thinking of factors like evolution of the contents (i.e. adding / removing "columns") , ease of querying the db (beyond just search-by-ECUID), how to make it the "authoritative DB" of known ROMs yet not overlap too much with the defs XMLs which IMO should remain separate, and how to automate maintenance of this DB - sure I can batch run nisrom on my 160+ ROMs , but if there is manually added content that need to be preserved, this needs some thought.

One example of "manual content" could be for the trickier stuff like the EEPROM addresses. nisrom is pretty good at this but not perfect and doesn't find the _write() func, and can't determine EEPROM size and organization (8- vs 16-bit access). It also sometimes fails guess the keyset although that is rare now.

[Pytrex]

Well, can confirm that batch running Nisrom works perfectly. Apart from some wonky formatting from Nisrom, it's pretty simple to export to a csv. Stuff like this really works well in Excel or Sheets. Plus it can stay as a csv file (at least in excel) rather than having to be converted. Modifying the csv file would be best done in excel/sheets as well rather than a generic text editor, for obvious reasons.

I guess my idea is that you make the original csv with known to be accurate ROMs, then any new ROMs, columns, or data that needs to be added would just be added onto that csv file. Rather than just trying to recreate the entire csv file again.

If I'm misunderstanding what you're saying, just let me know :P

[fenugrec]

then any new ROMs, columns, or data that needs to be added would just be added onto that csv file. Rather than just trying to recreate the entire csv file again.

No, I specifically don't want to add ROMs one by one - neither now, nor later. Ability to recreate and update the db automatically is a must. Look at the "Stock ROM list" thread on RR - I absolutely hate updating that post, because I need to take each new ROM rotting in my todo/ subfolder, copy + validate the fields to my private "master" spreadsheet, then export to text and mess with BBCode formatting. So I haven't been updating that list often at all.

I want to be able to add ROM dumps in my private dump, and not need to keep track of which ones are not yet in the db. Then just generate a new db, do a quick diff to make sure nisrom isn't suddenly broken (or overwriting manually-corrected data, if that ever happens), and keep any special fields intact (e.g. like those eeprom addrs I mentioned). The easier this process is, the more likely I am to keep the db up-to-date.

[Pytrex]

Ahh, I get what you mean! So you could have it check if a ECUID already exists, and if so, don't add another line. But if it doesn't, then add the line. That's assuming you can program it to append rows to an existing file rather than having to generate a new file. So then you'd be able to keep the manual corrections without it overwriting, while also being able to avoid duplicates. Utilizing ECUID for this would allow you to prevent it from overwriting manually adjusted data, which might occur if you tried checking if the entire row matched rather than just a specific value in the row.

So you'd generate the original file, then anytime you add new ROMs it'll generate the same file with new ROMs, compare the two files to find new ECUIDs, then add the new ECUID rows to the original file/create a new file with them added. That seems like it would work pretty decently. But again, I have exactly 0 experience with C programming haha

I honestly don't know how headers are normally handled. But I do believe you can just add the column names to the top of the csv file, even without any ROMs actually having data for it. So depending on how exactly you'd go about parsing, you'd just add any extra columns to the header.

[fenugrec]

this ticket also depends on fenugrec/nissutils#13 (done)