Skip to content

Added mlflow#74

Open
larryyin wants to merge 16 commits intoCambioML:mainfrom
larryyin:ly-mlflow-231004
Open

Added mlflow#74
larryyin wants to merge 16 commits intoCambioML:mainfrom
larryyin:ly-mlflow-231004

Conversation

@larryyin
Copy link

@larryyin larryyin commented Oct 5, 2023

Added draft example code to train with mlflow. It encountered some GPU issue. We can discuss it when we meet, maybe on Friday.

This is not the final version, just a point for review.

CambioML
CambioML reviewed Oct 5, 2023
View reviewed changes
example/rlhf/demo_supervised_finetuning.ipynb
Copy link
Collaborator

@CambioML CambioML Oct 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we keep this file as it is and move all MLflow related example to the mlflow folder?

Copy link
Member

@goldmermaid goldmermaid Oct 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems that the new notebook has better training results, so worth keep new one :)

example/rlhf/mlflow/demo_rl.py Outdated
Copy link
Collaborator

@CambioML CambioML Oct 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: does this file suppose to be here because I do not see any MLflow related logic?

example/rlhf/mlflow/demo_rw_finetuning.py
Copy link
Collaborator

@CambioML CambioML Oct 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: same as this file. Are you planning to add MLflow on this in the future but make a copy of the current training script first.

example/rlhf/mlflow/supervised_finetuning_with_mlflow_demo.py
from pykoi.chat import QuestionAnswerDatabase
from pykoi.rlhf import RLHFConfig
from pykoi.rlhf import SupervisedFinetuning
import mlflow
Copy link
Collaborator

@CambioML CambioML Oct 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make sure other people from the community can benefit from your amazing work.

  • Make sure you add MLflow as a dependency for the pykoi package after here following the same pattern.
  • Then, add it to the subpackage after here

@larryyin
Copy link
Author

larryyin commented Oct 6, 2023

All corrected. But please hold on and don't approve and merge until we meet. There are some issues. We can discuss them in person.

@larryyin
Copy link
Author

larryyin commented Oct 7, 2023

This is the final version. All the added scripts have been tested.

goldmermaid
goldmermaid reviewed Oct 7, 2023
View reviewed changes
.gitignore
# data
*.csv
!example/rlhf/mlflow/input_rw/ranking.csv
!example/rlhf/ranking.csv
Copy link
Member

@goldmermaid goldmermaid Oct 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this two csv can be kept for future users?

@larryyin
Copy link
Author

larryyin commented Oct 7, 2023 via email

goldmermaid
goldmermaid reviewed Oct 7, 2023
View reviewed changes
example/rlhf/demo_rl.ipynb Outdated
"\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb Cell 7\u001b[0m line \u001b[0;36m1\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0'>1</a>\u001b[0m \u001b[39mfrom\u001b[39;00m \u001b[39maccelerate\u001b[39;00m \u001b[39mimport\u001b[39;00m notebook_launcher\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=2'>3</a>\u001b[0m config \u001b[39m=\u001b[39m RLHFConfig(base_model_path\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m\u001b[39melinas/llama-7b-hf-transformers-4.29\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39m# \"elinas/llama-7b-hf-transformers-4.29\", \u001b[39;00m\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=3'>4</a>\u001b[0m dataset_type\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mlocal_db\u001b[39m\u001b[39m\"\u001b[39m,\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=4'>5</a>\u001b[0m reward_model_path\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mgoldmermaid/rlhf_reward_model\u001b[39m\u001b[39m\"\u001b[39m,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=8'>9</a>\u001b[0m \n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=9'>10</a>\u001b[0m )\n\u001b[0;32m---> <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=10'>11</a>\u001b[0m rlhf_step3_rl \u001b[39m=\u001b[39m RL(config)\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=11'>12</a>\u001b[0m rlhf_step3_rl\u001b[39m.\u001b[39mtrain(\u001b[39m\"\u001b[39m\u001b[39m./models/rlhf_step3_rl\u001b[39m\u001b[39m\"\u001b[39m, num_processes\u001b[39m=\u001b[39m\u001b[39m1\u001b[39m)\n",
"\u001b[1;32m/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb Cell 7\u001b[0m line \u001b[0;36m9\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=4'>5</a>\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mnum_proc \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_rlhf_config\u001b[39m.\u001b[39mnum_workers \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_rlhf_config\u001b[39m.\u001b[39mstreaming \u001b[39melse\u001b[39;00m \u001b[39mNone\u001b[39;00m\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=5'>6</a>\u001b[0m set_seed(rlhf_config\u001b[39m.\u001b[39mseed) \u001b[39m## TODO: how to set seed properly in __init__?\u001b[39;00m\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=7'>8</a>\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mppo_config\u001b[39m=\u001b[39mPPOConfig(\n\u001b[0;32m----> <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=8'>9</a>\u001b[0m steps\u001b[39m=\u001b[39m\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_rlhf_config\u001b[39m.\u001b[39;49mtotal_ppo_epochs,\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=9'>10</a>\u001b[0m model_name\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_rlhf_config\u001b[39m.\u001b[39mbase_model_path,\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=10'>11</a>\u001b[0m learning_rate\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_rlhf_config\u001b[39m.\u001b[39mlearning_rate,\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=11'>12</a>\u001b[0m batch_size\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_rlhf_config\u001b[39m.\u001b[39mppo_batch_size,\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=12'>13</a>\u001b[0m mini_batch_size\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_rlhf_config\u001b[39m.\u001b[39mmini_batch_size,\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=13'>14</a>\u001b[0m gradient_accumulation_steps\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_rlhf_config\u001b[39m.\u001b[39mgradient_accumulation_steps,\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=14'>15</a>\u001b[0m optimize_cuda_cache\u001b[39m=\u001b[39m\u001b[39mTrue\u001b[39;00m,\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=15'>16</a>\u001b[0m early_stopping\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_rlhf_config\u001b[39m.\u001b[39mearly_stopping,\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=16'>17</a>\u001b[0m target_kl\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_rlhf_config\u001b[39m.\u001b[39mtarget_kl,\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=17'>18</a>\u001b[0m ppo_epochs\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_rlhf_config\u001b[39m.\u001b[39mppo_epochs,\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=18'>19</a>\u001b[0m seed\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_rlhf_config\u001b[39m.\u001b[39mseed,\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=19'>20</a>\u001b[0m init_kl_coef\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_rlhf_config\u001b[39m.\u001b[39minit_kl_coef,\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=20'>21</a>\u001b[0m adap_kl_ctrl\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_rlhf_config\u001b[39m.\u001b[39madap_kl_ctrl,\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=21'>22</a>\u001b[0m \u001b[39m# accelerator_kwargs=self._rlhf_config.accelerator_kwargs,\u001b[39;00m\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=22'>23</a>\u001b[0m )\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=24'>25</a>\u001b[0m \u001b[39m## Load the base model and tokenizer and define the PPO Trainer for RL\u001b[39;00m\n\u001b[1;32m <a href='vscode-notebook-cell://ssh-remote%2B34.215.248.75/home/ubuntu/pykoi/example/rlhf/demo_rl.ipynb#W6sdnNjb2RlLXJlbW90ZQ%3D%3D?line=25'>26</a>\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mbase_tokenizer \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mcreate_tokenizer(rlhf_config\u001b[39m.\u001b[39mbase_model_path)\n",
"\u001b[0;31mAttributeError\u001b[0m: 'RLHFConfig' object has no attribute 'total_ppo_epochs'"
Copy link
Member

@goldmermaid goldmermaid Oct 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that there is still an error here. Can you clean up the output and write a to do note?

Copy link
Author

@larryyin larryyin Oct 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, can we remove this step 3 rl notebook example? For step 3, we use the py file, not the notebook file, as the example.
In the mlflow folder, I removed the notebook for step 3 rl.

Copy link
Author

@larryyin larryyin Oct 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error commented. Added a to-do comment in that block.

@larryyin
Copy link
Author

larryyin commented Oct 8, 2023 via email

goldmermaid
goldmermaid approved these changes Oct 8, 2023
View reviewed changes
Copy link
Member

@goldmermaid goldmermaid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love the ML Flow example! @larryyin Please made the minor changes above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@goldmermaid goldmermaid goldmermaid approved these changes

+1 more reviewer

@CambioML CambioML CambioML left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@larryyin @goldmermaid @CambioML