Refactor and optimize archive and manifest handling with benchmarks

[thesprockee]

Introduce a new idiomatic Go project structure, enhancing organization and documentation. Implement significant performance optimizations in binary encoding/decoding and extraction processes, achieving substantial speed and memory allocation improvements. Add benchmark tests for various operations to facilitate optimization analysis. Introduce a CLI option for decimal filenames and clean up legacy code for better maintainability. Update dependencies and streamline performance analysis by removing obsolete files.

In general, to fix this kind of issue you should avoid converting a potentially large int64 (from ParseInt with size 64) to a smaller integer type without checking that the value lies within the target type’s range. Either (a) parse directly into the appropriate size, or (b) enforce explicit bounds before casting.

Here, the problem is the conversion from int64 (chunkNum) to int in the loop condition for int(chunkNum) >= len(files). The simplest fix that preserves existing behavior is to avoid the narrowing conversion entirely by working in int64 consistently and using int64(len(files)) in the comparison. That way, no unchecked narrowing occurs, and the code is safe regardless of the platform’s int size.

Concretely in pkg/manifest/scanner.go:

• Replace the loop condition for int(chunkNum) >= len(files) { with a comparison that casts len(files) to int64: for chunkNum >= int64(len(files)) {.
• Replace the subsequent use of files[chunkNum] with files[chunkNumInt] where chunkNumInt is a safely converted int with checked range or, more simply, keep using files[chunkNum] by converting chunkNum to int after proving it fits via the loop bound. However, because slice indices must be int, we need one conversion.
• The best approach: keep the loop condition in int64, and inside the loop and before indexing, convert chunkNum once after a bounds check that ensures chunkNum is within the int range (at least non‑negative and not exceeding math.MaxInt on this platform). In practice, we can leverage the invariant that chunkNum >= 0 (directory names for chunks should be non‑negative); we should add an explicit check to enforce this and to ensure chunkNum <= math.MaxInt. This makes the index conversion safe.

So:

1. Add an import of math (already used earlier in other examples but not in this file).
2. After parsing chunkNum, add checks:
   • if chunkNum < 0 { return fmt.Errorf("invalid chunk number: %d", chunkNum) }
   • if chunkNum > int64(math.MaxInt) { return fmt.Errorf("chunk number too large: %d", chunkNum) }
3. Then we can safely use int(chunkNum) in the loop and index expressions, since we’ve guaranteed it is within int’s range.

This approach retains the structure of the code and satisfies CodeQL by adding explicit upper (and lower) bound checks before the narrowing conversion.

[thesprockee]

just switch to using uint64 from the start, chunkNum will never be negative

[Copilot]

Pull request overview

This PR refactors the project into an idiomatic Go structure with significant performance optimizations and enhanced functionality. The changes move from a monolithic main.go to a well-organized package structure, implement direct binary encoding/decoding for better performance, and add comprehensive benchmarks.

Key Changes:

• Reorganized codebase into pkg/ and cmd/ structure following Go conventions
• Optimized binary encoding/decoding and extraction with context reuse and pre-allocated buffers
• Added CLI option for decimal filenames and improved command-line interface

Reviewed changes

Copilot reviewed 22 out of 26 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pkg/manifest/manifest.go	Core manifest types and optimized binary encoding/decoding
pkg/manifest/package.go	Multi-part package extraction with performance optimizations
pkg/manifest/builder.go	Package building functionality from scanned files
pkg/manifest/scanner.go	Directory scanning for package building
pkg/archive/header.go	ZSTD archive header handling with zero-allocation encoding
pkg/archive/reader.go	Streaming decompression with reusable buffers
pkg/archive/writer.go	Streaming compression with header management
cmd/evrtools/main.go	Refactored CLI application with cleaner flag handling
README.md	Updated documentation with library usage examples and project structure
Makefile	Build automation and development workflow targets
go.mod	Updated module path and Go version
Legacy files	Removed obsolete code from previous implementation

Comments suppressed due to low confidence (2)

ready-at-dawn-echo-arena:1

• This file contains an absolute path to a local installation directory. This appears to be a development/local configuration file that should not be committed to version control as it exposes system paths and won't work for other users.
  _data:1
• This file contains an absolute path to a local installation directory. This appears to be a development/local configuration file that should not be committed to version control as it exposes system paths and won't work for other users.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[thesprockee]

@copilot implement the comments.

[Copilot]

@thesprockee I've opened a new pull request, #3, to work on those changes. Once the pull request is ready, I'll request review from you.