Gemma 2 model lineage (mlx_lm.models.gemma2 )

[uogbuji]

Notes, observations, considerations, etc. around models derived from Google Gemma2

[uogbuji]

A Reddit thread spurred me to try the SPPO variation, but it doesn't seem to follow tool-calling cues.

❯ python test/quick_check.py $HOME/.local/share/models/mlx/Gemma-2-9B-It-SPPO-Iter3-4bit
Model type: gemma2
As  an  AI,  I  don't  have  feelings  or  experiences  like  humans  do.  However,  I'm  here  and  ready  to  assist  you!  How  can  I  help  you  today?<end_of_turn>

 ======================================== Country extraction
```json
{"type":  "array",  "items":{"type":  "object",  "properties":  {"name":  {"type":  "string"},  "continent":  {"type":  "string"}}}}
```<end_of_turn><end_of_turn>
 ======================================== Square root of 256, pt 1
The  square  root  of  256  is  **16**.  

Here's  why:

*  The  square  root  of  a  number  is  a  value  that,  when  multiplied  by  itself,  equals  the  original  number.
*  16  *  16  =  256<end_of_turn><end_of_turn>
 ======================================== Square root of 256, pt 2
⚙️ Calling tool square_root with args {'square': 256}
⚙️ Tool call result: 16.0
<end_of_turn>

[uogbuji]

Turns out the last result was a bug in quick_check.py. Tried again and it's calling the square root tool but not generating a final response. It's also getting confused about how to call the calculator tool.

❯ python test/quick_check.py $HOME/.local/share/models/mlx/Gemma-2-9B-It-SPPO-Iter3-4bit

Model type: gemma2
As  an  AI,  I  don't  have  feelings  or  experiences  like  humans  do.  However,  I'm  here  and  ready  to  assist  you!  How  can  I  help  you  today?<end_of_turn>

 ======================================== Country extraction
```json
{"type":  "array",  "items":{"type":  "object",  "properties":  {"name":  {"type":  "string"},  "continent":  {"type":  "string"}}}}
```<end_of_turn><end_of_turn>
 ======================================== Square root of 256, pt 1
⚙️ Calling tool square_root with args {'square': 256}
⚙️ Tool call result: 16.0
<end_of_turn>
 ======================================== Square root of 256, pt 2
⚙️ Calling tool square_root with args {'square': 256}
⚙️ Tool call result: 16.0
<end_of_turn>
 ======================================== Usain bolt
⚙️ Calling tool calculator with args {'expr': 'Usain  Bolt  ran  the  100m  race  in  9.58s'}
Traceback (most recent call last):
  File "/Users/uche/dev/OoriData/Toolio/test/quick_check.py", line 89, in <module>
    resp = asyncio.run(amain(mm))
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/uche/dev/OoriData/Toolio/test/quick_check.py", line 58, in amain
    async for chunk in extract_content(mm.complete_with_tools(msgs, toolset=['calculator'])):
  File "/Users/uche/.local/venv/toolio/lib/python3.11/site-packages/toolio/llm_helper.py", line 318, in extract_content
    async for chunk in resp_stream:
  File "/Users/uche/.local/venv/toolio/lib/python3.11/site-packages/toolio/llm_helper.py", line 197, in complete_with_tools
    tool_responses = await self._execute_tool_calls(resp, req_tools)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/uche/.local/venv/toolio/lib/python3.11/site-packages/toolio/llm_helper.py", line 266, in _execute_tool_calls
    result = tool(**callee_args)
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/uche/.local/venv/toolio/lib/python3.11/site-packages/toolio/tool/schematics.py", line 53, in tool_inner
    value = func(*args, **processed_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/uche/.local/venv/toolio/lib/python3.11/site-packages/toolio/tool/math.py", line 27, in calculator
    raise ValueError(f'Disallowed characters encountered in mathematical expression {expr}')
ValueError: Disallowed characters encountered in mathematical expression Usain  Bolt  ran  the  100m  race  in  9.58s

[uogbuji]

Gemma 2b seems a bit flaky in terms of post tool-call completion. Instruct version:

❯ python test/quick_check.py mlx-community/gemma-2-2b-it-8bit
Model type: gemma2


I'm  doing  well,  thank  you!  How  can  I  help  you  today?  😊  
<end_of_turn>
 ======================================== Country extraction


```json
[
   {
     "name":  "Nigeria",
     "continent":  "Africa"
   }
]
```<end_of_turn>
 ======================================== Square root of 256, pt 1
⚙️ Calling tool square_root with args {'square': 256}
⚙️ Tool call result: 16.0
  
  The  square  root  of  256  is  **16**.  
<end_of_turn>
 ======================================== Square root of 256, pt 2
⚙️ Calling tool square_root with args {'square': 256}
⚙️ Tool call result: 16.0
  
The  square  root  of  256  is  **16**.  
<end_of_turn>
 ======================================== Usain bolt

 ======================================== END CHECK

Non-instruct flavor gave the following warning, then performed even worse.

No chat template is set for this tokenizer, falling back to a default class-level template. This is very error-prone, because models are often trained with templates different from the class default! Default chat templates are a legacy feature and will be removed in Transformers v4.43, at which point any code depending on them will stop working. We recommend setting a valid chat template before then to ensure that this model continues working without issues.

[uogbuji]

Retrying some models after we've tweaked add_generation_prompt.

gemma-2-9b-it-8bit now tries hader, but in some cases tries too hard.

❯ python test/quick_check.py mlx-community/gemma-2-9b-it-8bit
Model type: gemma2
As  an  AI,  I  don't  have  feelings  or  experiences  like  humans  do.  However,  I'm  here  and  ready  to  assist  you!  How  can  I  help  you  today?<end_of_turn>


 ======================================== Country extraction
[
   {
     "name":  "Nigeria"
   
   
   
   
   
   
   
   
   
   
,
   
   
   
   
   
   
   
   
   
   
"continent":  "Africa"
   }
]

 ======================================== Square root of 256, pt 1
```json
[
   {
     "type":  "object",
     "name":  "toolio.final_response",
     "arguments":  {
       "response":  "The  square  root  of  256  is  16.0"
     }
   }
]
```<end_of_turn>

 ======================================== Square root of 256, pt 2
The  square  root  of  256  is  16.0

 ======================================== Square root of 256, pt 3
```json
[
   {
     "type":  "object",
     "name":  "toolio.final_response",
     "description":  "Give  a  final  response  once  you  have  all  the  info  you  need,  and  have  completed  reasoning.",
     "properties":  {
       "name":  {
         "type":  "const",
         "const":  "toolio.final_response"
       },
       "arguments":  {
         "type":  "object",
         "properties":  {
           "response":  {
             "type":  "string",
             "description":  "

 ======================================== Usain bolt
```json
[
   {
     "type":  "object",
     "name":  "toolio.final_response",
     "arguments":  {
       "response":  "His  average  velocity  was  10.44  m/s."
     }
   }
]
```<end_of_turn>
 ======================================== END CHECK```

[chimezie]

Hmm.. I'm curious about the preference-optimized version of Gemma 9 IT that has been all the rage: UCLA-AGI/Gemma-2-9B-It-SPPO-Iter3 . I also seem to recall Gemma being one of those models whose token configurations were problematic WRT special tokens and that I would always have to explicitly indicate the EOS token to its tokenizer (i.e., via the --eos-token option in mlx_lm.generate). I can't find a good reference to that issue from my brief search.

Many downstream fine tunes (like those in the mix-community) would fix the issue by using their own tokenization configuration instead of the one from the original IT model.