Question

[KOVVURISATYANARAYANAREDDY]

Hello @jonppe, Thanks for this great work.

i am running

      funsearch run examples/cap_set_spec.py 11 --sandbox_type ExternalProcessSandbox

I have a small doubt, how much time or how many samples does it take to improve the island scores from 2048?

20 samples are done and score is not yet improved.

Can you please throw some light on this.

Thank you.

[jonppe]

Hello!

I just added a small related note on the README.md:

The search uses gpt-3.5-turbo-instruct by default, but other models can be used with the --model_name argument
and possibly installing extensions to the llm package.
As an example of performance, with gpt-3.5-turbo-instruct on dimension 8 it usually around 20 tries to find a few
improvements to the naive algorithm.

On the other hand, using orca-mini-3b-gguf2-q4_0 doesn't seem to work quite well.
The latest version has a bit improved parsing to find the last priority_vX method from the LLM response
even if it contains other content like Markdown formatting. Anyway, the model seems to often
use strange indentation of 1 space and thus might require some customization to be useful at all.
Lastly, even with correct Python syntax, orca-mini-3b does not seem to find improvements (in 60 runs) and mostly
generates code that throws "IndexError: tuple index out of range". The situation changes a bit
if the search is started using a database generated by gpt-3.5-turbo prompts.

Overall, all models would probably require some prompt engineering, temperatures tuning, and such for the tool
to be useful at all except for very simple problems.
Also, the implementation is currently lacking good tools to analyze large amount of responses properly which
makes any prompt engineering more difficult.

I can't quite remember if dimension 11 is much different to 8.

I might be doing some improvements to allow asyncio to make prompting faster in the first place, and then add better statistics to make it easier to improve prompts too... but I'm not sure if I find time. If anyone else makes such improvements feel free to create PRs.

[timneumann1]

@jonppe Would adding asyncio entail changing all the implementation files, or only your core.py file (where the last loop calls the samplers)?

[jonppe]

I suppose asyncio implementation would mostly change the core.py although I haven't analyzed it thoroughly. But I'm not sure I'll do any work on it. What are you up to @timneumann1?

[timneumann1]

I am trying to adapt the original research to a somewhat related problem in extremal combinatorics for my Bachelor's thesis in Math. Your edits have helped me a lot, but when running the experiments I am still very limited by computational power (so far, I can only run around 15,000 new prompts/responses everyday, which makes prompt engineering very difficult). So I am trying to increase the computational speed, but lack some knowledge in parallel computing.

[timneumann1]

Hi @jonppe. I noticed that in your implementation, a timeout variable is set but never used, so that the timeout is not implemented right now. Hence just a very quick question: Where exactly in the sandbox.py file would I have to add a timeout mechanism? I don't fully understand which of the functions (run, _exec etc.) actually executes the code. Some clarification would be greatly appreciated!

[jonppe]

Yes it seems I didn't implement the timeout for ExternalProcessSandbox.
ContainerSandbox uses the value.

I think there could be a few places where that could be added in ExternalProcessSandox.
At least the call:
return os.system(cmd)

could be replaced by something like:
import subprocess
...
try:
result = subprocess.run(cmd, shell=True, timeout=timeout, check=True)
except subprocess.TimeoutExpired:
print(f"Command timed out after {timeout} seconds.")
except subprocess.CalledProcessError:
print("Command failed")

Or alternatively, the one could use the linux "timeout" command like:
cmd = f"timeout {timeout_secs} {cmd}"