Validate QLoRA vs LoRA comparison fix by albertoperdomo2 · Pull Request #592 · openshift-psap/topsail

albertoperdomo2 · 2024-11-20T15:57:32Z

No description provided.

psap-forge-bot · 2024-11-20T17:00:52Z

Jenkins Job #1691

🔴 Test of 'rhoai test test_ci' failed after 00 hours 52 minutes 51 seconds. 🔴

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ibm_lora_qlora_models
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ibm_lora_qlora_models

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/000__matbenchmarking/FAILURE | MatrixBenchmark benchmark failed.
/logs/artifacts/000_test_ci/000__matbenchmarking/qlora/000__qlora/002__fms_fine_tuning/000__fine_tuning__run_fine_tuning_job/FAILURE | [000__fine_tuning__run_fine_tuning_job] ./run_toolbox.py from_config fine_tuning run_fine_tuning_job --extra={'name': 'qlora', 'model_name': 'mistral-7b-v0.3-gptq', 'dataset_name': 'alpaca_data.json', 'gpu': 4, 'dataset_replication': 0.5, 'pod_count': 1, 'hyper_parameters': {'fp16': True, 'gradient_accumulation_steps': 4, 'gradient_checkpointing': False, 'lora_alpha': 16, 'max_seq_length': 1024, 'max_steps': -1, 'num_train_epochs': 1, 'packing': False, 'peft_method': 'lora', 'per_device_train_batch_size': 16, 'r': 4, 'torch_dtype': 'float16', 'use_flash_attn': True, 'warmup_ratio': 0.03, 'auto_gptq': ['triton_v2'], 'target_modules': ['all-linear']}, 'dataset_transform': 'convert_alpaca.py'} --> 2
/logs/artifacts/000_test_ci/000__matbenchmarking/qlora/000__qlora/002__fms_fine_tuning/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/000_test_ci/000__matbenchmarking/qlora/000__qlora/002__fms_fine_tuning" ./run_toolbox.py from_config fine_tuning run_fine_tuning_job --extra="{'name': 'qlora', 'model_name': 'mistral-7b-v0.3-gptq', 'dataset_name': 'alpaca_data.json', 'gpu': 4, 'dataset_replication': 0.5, 'pod_count': 1, 'hyper_parameters': {'fp16': True, 'gradient_accumulation_steps': 4, 'gradient_checkpointing': False, 'lora_alpha': 16, 'max_seq_length': 1024, 'max_steps': -1, 'num_train_epochs': 1, 'packing': False, 'peft_method': 'lora', 'per_device_train_batch_size': 16, 'r': 4, 'torch_dtype': 'float16', 'use_flash_attn': True, 'warmup_ratio': 0.03, 'auto_gptq': ['triton_v2'], 'target_modules': ['all-linear']}, 'dataset_transform': 'convert_alpaca.py'}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 141, in _run_test
    run.run_toolbox_from_config("fine_tuning", "run_fine_tuning_job",
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)

[...]

[Test ran on the internal Perflab CI]

psap-forge-bot · 2024-11-21T14:07:28Z

Jenkins Job #1692

🟢 Test of 'rhoai test test_ci' succeeded after 04 hours 01 minutes 34 seconds. 🟢

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ibm_lora_qlora_models
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ibm_lora_qlora_models

• Link to the Rebuild page.

[Test ran on the internal Perflab CI]

psap-forge-bot · 2024-11-21T17:15:28Z

Jenkins Job #1693

🟢 Test of 'rhoai test test_ci' succeeded after 02 hours 09 minutes 45 seconds. 🟢

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ibm_lora_qlora_models
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ibm_lora_qlora_models

• Link to the Rebuild page.

[Test ran on the internal Perflab CI]

openshift-ci · 2025-01-08T08:03:02Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from albertoperdomo2. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

psap-forge-bot · 2025-01-08T10:19:44Z

Jenkins Job #1821

🔴 Test of 'rhoai test test_ci' failed after 00 hours 18 minutes 32 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ibm_lora_qlora_models
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ibm_lora_qlora_models

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/001__matbenchmarking/FAILURE | MatrixBenchmark benchmark failed.
/logs/artifacts/000_test_ci/001__matbenchmarking/qlora/000__qlora/001__prepare_data/000__storage__download_to_pvc/FAILURE | [000__storage__download_to_pvc] ./run_toolbox.py from_config storage download_to_pvc --extra={'source': 'dmf://rhoai/mistral-7b-v0.3-gptq', 'storage_dir': '/model', 'name': 'mistral-7b-v0.3-gptq', 'image': 'quay.io/modh/fms-hf-tuning:v2.0.1', 'creds': '/tmp/secrets/dmf.token'} --> 2

[Test ran on the internal Perflab CI]

psap-forge-bot · 2025-01-09T11:52:50Z

Jenkins Job #1827

🟢 Test of 'rhoai test test_ci' succeeded after 00 hours 50 minutes 21 seconds. 🟢

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ibm_lora_qlora_models
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ibm_lora_qlora_models

• Link to the Rebuild page.

[Test ran on the internal Perflab CI]

albertoperdomo2 · 2025-01-09T14:02:21Z

/test jump-ci rhoai-4xh100 fine_tuning ibm_lora_qlora_models

psap-forge-bot · 2025-01-09T14:26:52Z

🔴 Test of 'fine_tuning test pre_cleanup_ci' failed after 00 hours 00 minutes 00 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

PR_POSITIONAL_ARGS: jump-ci
PR_POSITIONAL_ARG_1: ibm_lora_qlora_models

Failure indicator: Empty. (See run.log)

albertoperdomo2 · 2025-01-09T14:45:25Z

/test jump-ci rhoai-4xh100 fine_tuning ibm_lora_qlora_models

psap-forge-bot · 2025-01-09T15:03:01Z

🔴 Test of 'fine_tuning test pre_cleanup_ci' failed after 00 hours 00 minutes 04 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

PR_POSITIONAL_ARGS: jump-ci
PR_POSITIONAL_ARG_1: ibm_lora_qlora_models

Failure indicator: Empty. (See run.log)

albertoperdomo2 · 2025-01-09T15:30:37Z

/test jump-ci rhoai-4xh100 fine_tuning ibm_lora_qlora_models cluster_rhoai_h100

psap-forge-bot · 2025-01-09T15:55:33Z

🔴 Test of 'fine_tuning test pre_cleanup_ci' failed after 00 hours 00 minutes 03 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

PR_POSITIONAL_ARGS: jump-ci
PR_POSITIONAL_ARG_1: ibm_lora_qlora_models
PR_POSITIONAL_ARG_2: cluster_rhoai_h100

Failure indicator: Empty. (See run.log)

albertoperdomo2 · 2025-01-09T16:05:07Z

/test jump-ci rhoai-4xh100 fine_tuning ibm_lora_qlora_models

psap-forge-bot · 2025-01-09T16:31:41Z

🔴 Test of 'fine_tuning test pre_cleanup_ci' failed after 00 hours 00 minutes 03 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

PR_POSITIONAL_ARGS: jump-ci
PR_POSITIONAL_ARG_1: ibm_lora_qlora_models

Failure indicator: Empty. (See run.log)

albertoperdomo2 · 2025-01-10T11:49:16Z

/test jump-ci rhoai-4xh100 fine_tuning ibm_lora_qlora_models

psap-forge-bot · 2025-01-10T12:10:09Z

🔴 Test of 'fine_tuning test prepare_ci' failed after 00 hours 12 minutes 34 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

PR_POSITIONAL_ARGS: jump-ci
PR_POSITIONAL_ARG_1: ibm_lora_qlora_models

Failure indicator:

/logs/artifacts/001__prepare2/001__prepare_data/000__storage__download_to_pvc/FAILURE | [000__storage__download_to_pvc] ./run_toolbox.py from_config storage download_to_pvc --extra={'source': 'https://raw.githubusercontent.com/gururise/AlpacaDataCleaned/main/alpaca_data.json', 'storage_dir': '/dataset', 'name': 'alpaca_data.json'} --> 2

albertoperdomo2 · 2025-01-10T15:17:44Z

/test jump-ci rhoai-4xh100 fine_tuning ibm_lora_qlora_models

psap-forge-bot · 2025-01-10T15:36:58Z

🔴 Test of 'fine_tuning test test_ci' failed after 00 hours 00 minutes 22 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

PR_POSITIONAL_ARGS: jump-ci
PR_POSITIONAL_ARG_1: ibm_lora_qlora_models

Failure indicator:

/logs/artifacts/001__matbenchmarking/FAILURE | MatrixBenchmark benchmark failed.
/logs/artifacts/001__matbenchmarking/qlora/000__qlora/FAILURE | KeyError: 'secondary_nic_count'
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 537, in matbench_run_one
    failed = _run_test([None], test_config) \
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 104, in _run_test
    if secondary_nic_count := test_settings.pop("secondary_nic_count"):
KeyError: 'secondary_nic_count'

/logs/artifacts/001__matbenchmarking/qlora_1/000__qlora/FAILURE | KeyError: 'secondary_nic_count'

[...]

albertoperdomo2 · 2025-01-10T15:47:12Z

/test jump-ci rhoai-4xh100 fine_tuning ibm_lora_qlora_models

psap-forge-bot · 2025-01-10T16:58:43Z

🔴 Test of 'fine_tuning test test_ci' failed after 00 hours 33 minutes 24 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

PR_POSITIONAL_ARGS: jump-ci
PR_POSITIONAL_ARG_1: ibm_lora_qlora_models

Failure indicator:

/logs/artifacts/001__matbenchmarking/FAILURE | MatrixBenchmark benchmark failed.
/logs/artifacts/001__matbenchmarking/qlora_1/000__qlora/003__fms_fine_tuning/000__fine_tuning__run_fine_tuning_job/FAILURE | [000__fine_tuning__run_fine_tuning_job] ./run_toolbox.py from_config fine_tuning run_fine_tuning_job --extra={'name': 'qlora', 'pod_count': 1, 'model_name': 'mistral-7b-v0.3', 'dataset_name': 'alpaca_data.json', 'gpu': 8, 'dataset_replication': 0.5, 'hyper_parameters': {'gradient_accumulation_steps': 4, 'gradient_checkpointing': False, 'lora_alpha': 16, 'max_seq_length': 2048, 'max_steps': -1, 'num_train_epochs': 1, 'packing': False, 'peft_method': 'lora', 'per_device_train_batch_size': 4, 'r': 4, 'use_flash_attn': True, 'warmup_ratio': 0.03, 'target_modules': ['all-linear']}, 'dataset_transform': 'convert_alpaca.py', 'dataset_response_template': '\n### Label:'} --> 2
/logs/artifacts/001__matbenchmarking/qlora_1/000__qlora/003__fms_fine_tuning/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/001__matbenchmarking/qlora_1/000__qlora/003__fms_fine_tuning" ./run_toolbox.py from_config fine_tuning run_fine_tuning_job --extra="{'name': 'qlora', 'pod_count': 1, 'model_name': 'mistral-7b-v0.3', 'dataset_name': 'alpaca_data.json', 'gpu': 8, 'dataset_replication': 0.5, 'hyper_parameters': {'gradient_accumulation_steps': 4, 'gradient_checkpointing': False, 'lora_alpha': 16, 'max_seq_length': 2048, 'max_steps': -1, 'num_train_epochs': 1, 'packing': False, 'peft_method': 'lora', 'per_device_train_batch_size': 4, 'r': 4, 'use_flash_attn': True, 'warmup_ratio': 0.03, 'target_modules': ['all-linear']}, 'dataset_transform': 'convert_alpaca.py', 'dataset_response_template': '\n### Label:'}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 156, in _run_test
    run.run_toolbox_from_config("fine_tuning", "run_fine_tuning_job",
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)

[...]

kpouget · 2025-01-10T17:05:46Z

/test jump-ci rhoai-4xh100 fine_tuning ibm_lora_qlora_models gpu
/skip pre_cleanup_ci

kpouget · 2025-01-10T17:43:50Z

/test jump-ci rhoai-4xh100 fine_tuning ibm_lora_qlora_models gpu
/only prepare_ci test_ci

psap-forge-bot · 2025-01-10T18:17:21Z

🔴 Test of 'fine_tuning test test_ci' failed after 00 hours 25 minutes 56 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

PR_POSITIONAL_ARGS: jump-ci
PR_POSITIONAL_ARG_1: ibm_lora_qlora_models
PR_POSITIONAL_ARG_2: gpu
exec_list._only_: True
exec_list.prepare_ci: True
exec_list.test_ci: True

Failure indicator:

/logs/artifacts/001__matbenchmarking/FAILURE | MatrixBenchmark benchmark failed.

kpouget · 2025-01-10T19:27:24Z

/test jump-ci rhoai-4xh100 fine_tuning ibm_lora_qlora_models gpu
/only prepare_ci test_ci

kpouget · 2025-01-10T19:52:30Z

/test jump-ci rhoai-4xh100 fine_tuning ibm_lora_qlora_models gpu
/only test_ci

psap-forge-bot · 2025-01-10T21:01:11Z

🟢 Test of 'fine_tuning test test_ci' succeeded after 00 hours 53 minutes 28 seconds. 🟢

• Link to the test results.

• Link to the reports index.

Test configuration:

PR_POSITIONAL_ARGS: jump-ci
PR_POSITIONAL_ARG_1: ibm_lora_qlora_models
PR_POSITIONAL_ARG_2: gpu
exec_list._only_: True
exec_list.test_ci: True

albertoperdomo2 force-pushed the mistral_qlora_lora_comparison branch from 7793ee2 to 70043ec Compare January 8, 2025 08:02

albertoperdomo2 force-pushed the mistral_qlora_lora_comparison branch from c97ebb3 to 27f5f0b Compare January 9, 2025 14:01

albertoperdomo2 force-pushed the mistral_qlora_lora_comparison branch from b50700c to df21709 Compare January 9, 2025 16:01

albertoperdomo2 force-pushed the mistral_qlora_lora_comparison branch from df21709 to cc82c0f Compare January 10, 2025 11:48

albertoperdomo2 force-pushed the mistral_qlora_lora_comparison branch from cc82c0f to 8b93a37 Compare January 10, 2025 15:17

albertoperdomo2 added 7 commits January 10, 2025 15:46

[fine_tuning]: disable gradient checkpointing

8bd6a4a

[fine_tuning]: simpler settings for testing purposes

886b8a1

[fine_tuning]: test on 8 gpus

e59c401

[fine_tuning]: replicate exact IBM settings

683515a

[fine_tuning]: test only one case scenario

ce0e454

[fine_tuning]: rename cleanup states

50117c6

[fine_tuning]: add new metal profile

5dc805c

albertoperdomo2 force-pushed the mistral_qlora_lora_comparison branch from 8b93a37 to 5dc805c Compare January 10, 2025 15:46

albertoperdomo2 mentioned this pull request Jan 13, 2025

[fine_tuning]: Refactor QLoRA settings #628

Open

albertoperdomo2 closed this Jan 13, 2025

albertoperdomo2 deleted the mistral_qlora_lora_comparison branch January 13, 2025 08:47

Conversation

albertoperdomo2 commented Nov 20, 2024

Uh oh!

psap-forge-bot Bot commented Nov 20, 2024

Uh oh!

psap-forge-bot Bot commented Nov 21, 2024

Uh oh!

psap-forge-bot Bot commented Nov 21, 2024

Uh oh!

openshift-ci Bot commented Jan 8, 2025

Uh oh!

psap-forge-bot Bot commented Jan 8, 2025

Uh oh!

psap-forge-bot Bot commented Jan 9, 2025

Uh oh!

albertoperdomo2 commented Jan 9, 2025 • edited by kpouget Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

psap-forge-bot Bot commented Jan 9, 2025

Uh oh!

albertoperdomo2 commented Jan 9, 2025

Uh oh!

psap-forge-bot Bot commented Jan 9, 2025

Uh oh!

albertoperdomo2 commented Jan 9, 2025

Uh oh!

psap-forge-bot Bot commented Jan 9, 2025

Uh oh!

albertoperdomo2 commented Jan 9, 2025

Uh oh!

psap-forge-bot Bot commented Jan 9, 2025

Uh oh!

albertoperdomo2 commented Jan 10, 2025

Uh oh!

psap-forge-bot Bot commented Jan 10, 2025

Uh oh!

albertoperdomo2 commented Jan 10, 2025

Uh oh!

psap-forge-bot Bot commented Jan 10, 2025

Uh oh!

albertoperdomo2 commented Jan 10, 2025

Uh oh!

psap-forge-bot Bot commented Jan 10, 2025

Uh oh!

kpouget commented Jan 10, 2025

Uh oh!

kpouget commented Jan 10, 2025

Uh oh!

psap-forge-bot Bot commented Jan 10, 2025

Uh oh!

kpouget commented Jan 10, 2025

Uh oh!

kpouget commented Jan 10, 2025

Uh oh!

psap-forge-bot Bot commented Jan 10, 2025

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

albertoperdomo2 commented Jan 9, 2025 •

edited by kpouget

Loading