Phi-3 and Llama-3 tutorial

phi3-finetune/README.md

@@ -0,0 +1,58 @@
+# Phi-3 Model Fine-tuning Demo
+
+This demo will show how to use ACPT (Azure Container for PyTorch) along with accelerators such as onnxruntime training (through ORTModule) and DeepSpeed to fine-tune Phi-3 model.
+
+## Background
+
+[Phi-3](https://www.microsoft.com/en-us/research/blog/phi-3-the-surprising-power-of-small-language-models/)  is 2.7 billion-parameter language model with nex-t word prediction objective. It has been trained using mixture of Synthetic and Web datasets.
+
+## Set up
+
+### AzureML
+The easiest option to run the demo will be using AzureML as the environment details are already included, there is another option to run directly on the machine which is provided later. For AzureML, please complete the following prerequisites:
+
+#### Local environment
+Set up your local environment with az-cli and azureml dependency for script submission:
+
+```
+az-cli && az login
+pip install azure-ai-ml azure-identity
+```
+
+#### AzureML Workspace
+- An AzureML workspace is required to run this demo. Download the config.json file ([How to get config.json file from Azure Portal](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-environment#workspace)) for your workspace. Make sure to put this config file in this folder and name it ws_config.json.
+- The workspace should have a gpu cluster. This demo was tested with GPU cluster of SKU [Standard_ND40rs_v2](https://docs.microsoft.com/en-us/azure/virtual-machines/ndv2-series). See this document for [creating gpu cluster](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-create-attach-compute-cluster?tabs=python). We do not recommend running this demo on `NC` series VMs which uses old architecture (K80).
+- Additionally, you'll need to create a [Custom Curated Environment ACPT](https://learn.microsoft.com/en-us/azure/machine-learning/resource-curated-environments) with PyTorch >=2.2.0 and the steps in the Dockerfile.
+
+## Run Experiments
+The demo is ready to be run.
+
+#### `aml_submit.py` submits an training job to AML for both Pytorch+DeepSpeedStage2 and ORT+DeepSpeedStage2. This job builds the training environment and runs the fine-tuning script in it.
+
+```bash
+python aml_submit.py
+```
+
+The above script will generate two URLs, one for Pytorch and another for ONNX Runtime training.
+
+We observe **~<TBD>% speedup** for Phi-3 trained leveraging ONNX Runtime Training with 8 V100 GPUs with 32GB memory, with a batch size of <TBD>.
+
+### Run directly on your compute
+
+If you are using CLI by directly logging into your machine then you can follow the below instructions. The below steps assume you have the required packages like Pytorch, ONNX Runtime training, Transformers and more already installed in your system. For easier setup, you can look at the environment folder.
+
+```bash
+cd finetune-clm
+
+# To run the model using Pytorch
+torchrun --nproc_per_node 8 run_clm.py --model_name_or_path microsoft/phi-3 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --save_strategy 'no' --fp16 --block_size 2048 --max_steps -1 --per_device_train_batch_size 1 --num_train_epochs 2 --output_dir output_dir --overwrite_output_dir --deepspeed zero_stage_2.json --evaluation_strategy no --remove_unused_columns False
+
+# To run the model using ONNX Runtime training, you need to export couple of variables and run the same command above, overall these would be your steps:
+export APPLY_ORT="True"
+export ORTMODULE_FALLBACK_POLICY="FALLBACK_DISABLE"
+export ORTMODULE_DEEPCOPY_BEFORE_MODEL_EXPORT=0
+# Optionally you can enable/disable Triton, for faster performance it is turned on
+export ORTMODULE_USE_TRITON=1
+torchrun --nproc_per_node 8 run_clm.py --model_name_or_path microsoft/phi-3 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --save_strategy 'no' --fp16 --block_size 2048 --max_steps -1 --per_device_train_batch_size 1 --num_train_epochs 2 --output_dir output_dir --overwrite_output_dir --deepspeed zero_stage_2.json --evaluation_strategy no --remove_unused_columns False
+```
+

phi3-finetune/aml_submit_clm.py

@@ -0,0 +1,137 @@
+#!/usr/bin/env python
+# coding=utf-8
+# Copyright 2023 Microsoft Corp. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+
+import argparse
+from pathlib import Path
+import json
+import os
+
+from azure.ai.ml import MLClient, command
+from azure.ai.ml.entities import Environment, BuildContext 
+from azure.identity import AzureCliCredential
+
+# run test on automode workspace
+ws_config = json.load(open("ws_config.json"))
+subscription_id = ws_config["subscription_id"]
+resource_group = ws_config["resource_group"]
+workspace_name = ws_config["workspace_name"]
+compute = ws_config["compute"]
+nproc_per_node = ws_config["nproc_per_node"]
+
+def get_args(raw_args=None):
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument("--experiment_name", default="Phi-2-ORT-CLM-Stage2-Experiment", help="Experiment name for AML Workspace")
+
+    args = parser.parse_args(raw_args)
+    return args
+
+def main(raw_args=None):
+    args = get_args(raw_args)
+
+    ml_client = MLClient(
+        AzureCliCredential(), subscription_id, resource_group, workspace_name
+    )
+
+    root_dir = Path(__file__).resolve().parent
+    environment_dir = root_dir / "environment"
+    code_dir = root_dir / "finetune-clm"
+
+    model = "microsoft/phi-3"
+    num_train_epochs = 2
+    bsz = 3
+    max_steps = -1
+
+    dataset_name = "wikitext"
+    dataset_config_name = "wikitext-2-raw-v1"
+    text_column_name = "text"
+    label_column_name = "label"
+
+    pytorch_job = command(
+        code=code_dir,  # local path where the code is stored
+        command=f"torchrun --nproc_per_node {nproc_per_node} run_clm.py \
+            --model_name_or_path {model} \
+            --dataset_name {dataset_name} \
+            --dataset_config_name {dataset_config_name} \
+            --do_train \
+            --save_strategy 'no' \
+            --per_device_train_batch_size {bsz} \
+            --num_train_epochs {num_train_epochs} \
+            --output_dir results --overwrite_output_dir \
+            --fp16 --max_steps {max_steps} \
+            --block_size 2048 \
+            --deepspeed zero_stage_2.json \
+            --evaluation_strategy no --remove_unused_columns False",
+        environment=Environment(build=BuildContext(path=environment_dir)),
+        experiment_name="Phi-3-Pytorch-CLM-LORA-Stage2-Experiment",
+        compute=compute,
+        display_name=model.replace(
+            "microsoft/phi-2",
+            f"pytorch+DS2-{bsz}"
+        ),
+        description=f"Finetune HuggingFace's Phi-3 using PyTorch",
+        tags={"model": model,
+              "bsz": bsz,
+              "dataset_name": dataset_name},
+        shm_size="16g"
+    )
+
+    print("submitting PyTorch job for " + model)
+    pytorch_returned_job = ml_client.create_or_update(pytorch_job)
+    print("submitted job")
+
+    pytorch_aml_url = pytorch_returned_job.studio_url
+    print("job link:", pytorch_aml_url)
+
+    ort_job = command(
+        code=code_dir,  # local path where the code is stored
+        command=f"torchrun --nproc_per_node {nproc_per_node} run_clm.py \
+            --model_name_or_path {model} \
+            --dataset_name {dataset_name} \
+            --dataset_config_name {dataset_config_name} \
+            --do_train \
+            --save_strategy 'no' \
+            --per_device_train_batch_size {bsz} \
+            --num_train_epochs {num_train_epochs} \
+            --output_dir results --overwrite_output_dir \
+            --fp16 --max_steps {max_steps} \
+            --block_size 2048 \
+            --deepspeed zero_stage_2.json \
+            --evaluation_strategy no --remove_unused_columns False",
+        environment=Environment(build=BuildContext(path=environment_dir)),
+        environment_variables={"APPLY_ORT": "True",
+                               "ORTMODULE_FALLBACK_POLICY": "FALLBACK_DISABLE"},
+        experiment_name="Phi-3-ORT-CLM-Stage2-Experiment",
+        compute=compute,
+        display_name=model.replace(
+            "microsoft/phi-3",
+            f"ort+DS2-{bsz}"
+        ),
+        description=f"Finetune HuggingFace's Phi-3 using ONNX Runtime",
+        tags={"model": model,
+              "bsz": bsz,
+              "dataset_name": dataset_name},
+        shm_size="16g"
+    )
+
+    print("submitting ORT job for " + model)
+    ort_returned_job = ml_client.create_or_update(ort_job)
+    print("submitted job")
+
+    ort_aml_url = ort_returned_job.studio_url
+    print("job link:", ort_aml_url)
+
+if __name__ == "__main__":
+    main()

phi3-finetune/environment/Dockerfile

@@ -0,0 +1,14 @@
+FROM mcr.microsoft.com/aifx/acpt/stable-ubuntu2004-cu118-py38-torch220
+
+RUN pip uninstall onnxruntime-training -y && \
+    pip install -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT/pypi/simple/ onnxruntime-training && \
+    TORCH_CUDA_ARCH_LIST="5.2 6.0 6.1 7.0 7.5 8.0 8.6+PTX" python -m onnxruntime.training.ortmodule.torch_cpp_extensions.install
+
+RUN pip install -U datasets evaluate accelerate scikit-learn transformers==4.36.2
+RUN pip install git+https://github.com/huggingface/optimum.git
+
+RUN pip install einops
+RUN pip install --upgrade pytest
+RUN pip install peft
+
+RUN pip list