Performance issue in packager

[gmh04]

Here are some times for creating bags with a small number of files (generally 1-3 files) on my workstation:

1GB - 00:26
5GB - 01:20
10GB - 02:60
50GB - 20:08
100GB - 42:34

[gmh04]

A breakdown of the bagInPlace method of BagCreator https://github.com/LibraryOfCongress/bagit-java/blob/v5.0.3/src/main/java/gov/loc/repository/bagit/creator/BagCreator.java#L120-L137 shows 99% (or 134 seconds) of time of this function is createPayloadManifests. This creates the checksums of the files and compares with a manual command-line creation of a md5 checksum of 110 seconds.

Really the place for improvement here is to remove the unnecessary call to isValid

datavault/datavault-worker/src/main/java/org/datavaultplatform/worker/operations/Packager.java

Line 51 in 5a2a7e0

return Packager.isValid(bag);

which is taking another 130 seconds (it is redoing the lengthy checksums!). Any issues with bagInPlace should throw an exception, therefore we can assume the bag is valid if a Bag object is returned. This would half the time of createBag.