Skip to content

Fix GGUF tensor alignment and validate model config#20

Open
dipeshbabu wants to merge 2 commits intoRightNow-AI:mainfrom
dipeshbabu:fix/gguf-alignment-and-config-validation
Open

Fix GGUF tensor alignment and validate model config#20
dipeshbabu wants to merge 2 commits intoRightNow-AI:mainfrom
dipeshbabu:fix/gguf-alignment-and-config-validation

Conversation

@dipeshbabu
Copy link

This fixes a crash-prone bug in GGUF parsing where tensor data base offset was computed using a bitmask that only works when general.alignment is a non-zero power of two. If alignment is missing/malformed (or non power-of-two), the model can read incorrect mmap offsets and crash.

Changes:

  • Use a safe align_up_any() implementation for tensor data base alignment.
  • Initialize config with memset to avoid stale/uninitialized fields when metadata is missing.
  • Validate required GGUF fields (n_embd/n_heads/n_layers/vocab_size).
  • Default n_kv_heads to n_heads when omitted.
  • Validate n_embd % n_heads == 0 before computing head_dim.

Tests:

  • make clean && make native (Linux x86_64)
  • Load and run generation on TinyLlama Q4_K_M gguf

@dipeshbabu
Copy link
Author

This also fixes for avoiding per-token malloc/leak by using mmap string views.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant