Fix gcc compilation error

[amir-wyvern]

Fix GCC compilation error with variable length array parameter

Summary

This PR fixes a compilation error that occurs when building vdexExtractor with newer GCC versions. The error is caused by a mismatch between the function declaration and implementation for dexInstr_getVarArgs.

Problem

When compiling with ./make.sh gcc, the build fails with:

dex_instruction.c:655:43: error: argument 2 of type 'u4[kMaxVarArgRegs]' {aka 'unsigned int[kMaxVarArgRegs]'} declared as a variable length array [-Werror=vla-parameter]
  655 | void dexInstr_getVarArgs(u2 *code_ptr, u4 arg[kMaxVarArgRegs]) {
      |                                        ~~~^~~~~~~~~~~~~~~~~~~
In file included from dex_instruction.c:23:
dex_instruction.h:278:32: note: previously declared as an ordinary array 'u4[]' {aka 'unsigned int[]'}
  278 | void dexInstr_getVarArgs(u2 *, u4[]);
      |                                ^~~~

Root Cause

The header file declares void dexInstr_getVarArgs(u2 *, u4[]); but the implementation uses void dexInstr_getVarArgs(u2 *code_ptr, u4 arg[kMaxVarArgRegs]). Newer GCC versions treat this mismatch as an error when using -Werror flag.

Solution

Update the function declaration in src/dex_instruction.h to match the implementation:

Before:

void dexInstr_getVarArgs(u2 *, u4[]);

After:

void dexInstr_getVarArgs(u2 *, u4[kMaxVarArgRegs]);

Changes Made

• File: src/dex_instruction.h
• Line: 278
• Change: Updated function declaration to use explicit array size instead of incomplete array type

Testing

✅ Compilation: Successfully compiles with ./make.sh gcc
✅ Functionality: All existing functionality maintained
✅ Compatibility: No breaking changes to the API
✅ Standards: Passes all compilation flags including -Wall -Wextra -Werror

Impact

• Positive: Fixes compilation issues on systems with newer GCC versions
• Neutral: No functional changes to the codebase
• Risk: Minimal - only affects declaration, not implementation

Environment Tested

• OS: Linux 6.8.0-64-generic
• Compiler: GCC with -Werror=vla-parameter support
• Build Command: ./make.sh gcc
• Result: ✅ Successful compilation and execution

Related

This fix addresses compilation issues that prevent users from building vdexExtractor on modern systems, improving the tool's accessibility for Android security research and reverse engineering.

---

Note: This is a safe, backward-compatible change that only affects the function declaration to match the existing implementation.

[RJMultiDev]

Is this made by AI?(sorry but it looks like)

[amir-wyvern]

Is this made by AI?(sorry but it looks like)

Yeah, I used Cursor AI to help write the content in this commit

[zxhubo]

@amir-wyvern

Yes, this solution can resolve this error .But when use vdexExtractor ,the new error is:

./vdexExtractor -i GameAssistantApp.vdex -o ./out/
[INFO] Processing 1 file(s) from GameAssistantApp.vdex
[ERROR] Unsupported Vdex version
[WARNING] Invalid Vdex header - skipping 'GameAssistantApp.vdex'
[INFO] 0 out of 0 Vdex files have been processed
[INFO] 0 Dex files have been extracted in total
[INFO] Extracted Dex files are available in './out/'

Of course, GameAssistantApp.vdex is a vdex file on an Android 16.0 devices.

[vamsishankar2711763]

@amir-wyvern The fix is bringing up new issues