-
-
-## Supported Pipelines
-
-| Pipelines | Status |
-|----------|:-------------:|
-| Task Tuning | :white_check_mark: Supported |
-| Instruction Tuning | :white_check_mark: Supported |
-| Parameter-Efficient Tuning | :white_check_mark: Supported |
-| Alignment Tuning | :white_check_mark: Supported |
-| Large Model Inference | :white_check_mark: Supported |
-
-## Supported Models
-
-Seamlessly supported all the [decoder models](https://huggingface.co/models?pipeline_tag=text-generation&sort=downloads) in 🤗 Hugging Face.
-LLaMA, GPT2, GPT-Neo, Galactica, have been fully tested. We will support encoder models soon.
-
-
-## 1.Setup
+### Setup
Our package has been fully tested on Linux OS (Ubuntu 20.04). Other OS platforms (MacOS, Windows) are not fully tested.
You may encounter some unexpected errors. You may try it first on a Linux machine or use Google Colab to experience it.
```bash
-git clone https://github.com/OptimalScale/LMFlow.git
+git clone -b v0.0.4 https://github.com/OptimalScale/LMFlow.git
cd LMFlow
conda create -n lmflow python=3.9 -y
conda activate lmflow
@@ -217,100 +93,42 @@ conda install mpi4py
./install.sh
```
-## 2.Prepare Dataset
+### Prepare Dataset
Please refer to our [doc](https://optimalscale.github.io/LMFlow/examples/DATASETS.html).
-## 3. Running Scripts
-### 3.1 Finetuning
-
-You can run `scripts/run_finetune.sh` to finetune a GPT-2 base model
+### Finetuning (Full)
+Full training updates all the parameters to finetune a language model.
+Here is an example to finetune a GPT-2 base model
```sh
./scripts/run_finetune.sh
```
-If you would like to provide arguments for deepspeed to reflect your machine
-settings, you may pass the corresponding deepspeed arguments to the script. For
-example,
-```sh
-./scripts/run_finetune.sh "--num_gpus=8 --master_port 10001"
-```
-
-To enable LoRA finetuning, you may refer to
+### Finetuning (LoRA)
+LoRA is a parameter-efficient finetuning algorithm and is more efficient than full finetuning.
```sh
./scripts/run_finetune_with_lora.sh
```
-which can be run in similar manner.
-
-For detailed configurations, one may modify these scripts directly. These
-scripts actually just call python script `examples/finetune.py`, which can
-be run in following manner,
+### Inference
+After finetuning, you can run the following command to chat with the model.
```sh
-deepspeed ${deepspeed_args} \
- examples/finetune.py \
- --deepspeed configs/ds_config_zero3.json \
- --bf16 \
- --run_name finetune_with_lora \
- --model_name_or_path facebook/galactica-1.3b \
- --num_train_epochs 0.01 \
- --learning_rate 2e-5 \
- --dataset_path ${dataset_path} \
- --per_device_train_batch_size 1 \
- --per_device_eval_batch_size 1 \
- --validation_split_percentage 0 \
- --logging_steps 20 \
- --block_size 512 \
- --do_train \
- --output_dir output_models/finetune \
- --overwrite_output_dir \
- --ddp_timeout 72000 \
- --save_steps 5000 \
- --dataloader_num_workers 1
+./scripts/run_chatbot.sh {finetuned-checkpoints-path}
```
-Here we set number of epochs `--num_train_epochs` to `0.01` so that the
-finetuning process can be finished quickly. If you wish to obtain a model with
-better performance, feel free to adjust those hyperparameters. You may run
-```python
-python examples/finetune.py -h
-```
-Note: In the case of a small training data set, the value of ``block_size`` needs to be reduced, otherwise there will be no samples available in the Epoch iterator.
-
-to view all possible finetuning arguments. The finetuned model checkpoint will
-be saved in the argument specified by `--output_dir`, which is
-`output_models/finetune` in the above example.
-We follow [Alpaca](https://github.com/tatsu-lab/stanford_alpaca) and [Vicuna](https://github.com/lm-sys/FastChat) in the model tuning process and serve the model in our web service.
-### 3.2 Evaluation
-
-One can directly run evaluation with an existing Hugging Face model, e.g. to run
-GPT2 large, one may execute
-```sh
-./scripts/run_evaluation.sh
-```
-or run the corresponding python script
-```python
-CUDA_VISIBLE_DEVICES=0 \
- deepspeed examples/evaluate.py \
- --answer_type medmcqa \
- --model_name_or_path gpt2-large \
- --dataset_path data/MedQA-USMLE/validation \
- --deepspeed examples/ds_config.json
-```
-To load the finetuned model, specify `--model_name_or_path` with the saved
-model checkpoint directory path.
+### Deployment
+If you want to deploy your own model locally, we provide a gradio-based UI for building chatbots.
+Running the following command will launch the demo for robin-7b:
-For LoRA finetuned models, one may refer to
```sh
-./scripts/run_evaluation_with_lora.sh
+pip install gradio
+python ./examples/chatbot_gradio.py --deepspeed configs/ds_config_chatbot.json --model_name_or_path YOUR-LLAMA --lora_model_path ./robin-7b --prompt_structure "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: {input_text}###Assistant:" --end_string "#" --max_new_tokens 200
```
+We also hosted it on Hugging Face [Space](https://huggingface.co/spaces/OptimalScale/Robin-7b).
-Those scripts invoke the examples `examples/*.py` built based on our APIs. For
-more API-related examples, one may refer to the methods in the unittest
-`tests`.
-### 3.3 LMFlow Benchmark
-LMFlow Benchmark is an automatic evaluation framework for open-source large language models.
+### Evaluation
+[LMFlow Benchmark](https://blog.gopenai.com/lmflow-benchmark-an-automatic-evaluation-framework-for-open-source-llms-ef5c6f142418) is an automatic evaluation framework for open-source large language models.
We use negative log likelihood (NLL) as the metric to evaluate different aspects of a language model: chitchat, commonsense reasoning, and instruction following abilities.
You can directly run the LMFlow benchmark evaluation to obtain the results to participate in the
@@ -324,96 +142,81 @@ For example, to run GPT2 XL, one may execute
To check the evaluation results, you may check `benchmark.log` in `./output_dir/gpt2-xl_lmflow_chat_nll_eval`,
`./output_dir/gpt2-xl_all_nll_eval` and `./output_dir/gpt2-xl_commonsense_qa_eval`.
-## 4. Additional Notes
-### 4.1 LLaMA Checkpoint
+## Supported Features
-Please refer to our [doc](https://optimalscale.github.io/LMFlow/examples/checkpoints.html).
+
-
