From 383314a84598f0e1d8a0534485509d20abbf6752 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 10:05:50 +0200 Subject: [PATCH 01/31] feat: add documentation improvement issue template --- .../ISSUE_TEMPLATE/documentation_improvement.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/documentation_improvement.md diff --git a/.github/ISSUE_TEMPLATE/documentation_improvement.md b/.github/ISSUE_TEMPLATE/documentation_improvement.md new file mode 100644 index 00000000..8c5db8fb --- /dev/null +++ b/.github/ISSUE_TEMPLATE/documentation_improvement.md @@ -0,0 +1,14 @@ +--- +name: Documentation improvement +about: Create a report to help us improve +title: '[DOC] ' +labels: 'documentation' +assignees: '' + +--- + +## What part of the documentation do you want to update? + + +## What changes do you want to make? + \ No newline at end of file From 4d9b7b740ba081f240a0406dda1bfa5d16f3f5f3 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 10:08:29 +0200 Subject: [PATCH 02/31] docs: update README for clarity and remove telemetry code snippets --- README.md | 340 ++++-------------------------------------------------- 1 file changed, 20 insertions(+), 320 deletions(-) diff --git a/README.md b/README.md index 006898fe..a3340636 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Element - **Simply make AI models faster, cheaper, smaller, greener!** + **Simply make AI models faster, cheaper, smaller, greener!** Element
@@ -86,17 +86,8 @@ pip install -e . ## Pruna Cool Quick Start -Before we start: Pruna allows to collect [a minimal set of aggregated, non-personal telemetry data](https://docs.pruna.ai/en/stable/docs_pruna/user_manual/telemetry.html) to help us identify popular algorithms and improve the product. Telemetry is enabled by default because your participation helps us make Pruna better. However, if you'd prefer not to share this, you can always disable telemetry with: -```python -from pruna.telemetry import set_telemetry_metrics - -set_telemetry_metrics(False) # disable telemetry for current session -set_telemetry_metrics(False, set_as_default=True) # disable telemetry globally -``` - - -Getting started with Pruna is easy-peasy pruna-squeezy! +Getting started with Pruna is easy-peasy pruna-squeezy! First, load any pre-trained model. Here's an example using Stable Diffusion: @@ -105,7 +96,7 @@ from diffusers import StableDiffusionPipeline base_model = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") ``` -Then, use Pruna's `smash` function to optimize your model. You can customize the optimization process using `SmashConfig`: +Then, use Pruna's `smash` function to optimize your model. Pruna provides a variety of different compression and optimization algorithms, allowing you to combine different algorithms to get the best possible results. You can customize the optimization process using `SmashConfig`: ```python from pruna import smash, SmashConfig @@ -113,6 +104,7 @@ from pruna import smash, SmashConfig # Create and smash your model smash_config = SmashConfig() smash_config["cacher"] = "deepcache" +smash_config["compiler"] = "stable_fast" smashed_model = smash(model=base_model, smash_config=smash_config) ``` @@ -124,18 +116,6 @@ smashed_model("An image of a cute prune.").images[0]
-Pruna provides a variety of different compression and optimization algorithms, allowing you to combine different algorithms to get the best possible results: - -```python -from pruna import smash, SmashConfig - -# Create and smash your model -smash_config = SmashConfig() -smash_config["cacher"] = "deepcache" -smash_config["compiler"] = "stable_fast" -smashed_model = smash(model=base_model, smash_config=smash_config) -``` - You can then use our evaluation interface to measure the performance of your model: ```python @@ -143,16 +123,14 @@ from pruna.evaluation.task import Task from pruna.evaluation.evaluation_agent import EvaluationAgent from pruna.data.pruna_datamodule import PrunaDataModule -task = Task("image_generation_quality", datamodule=PrunaDataModule.from_string("LAION256")) -eval_agent = EvaluationAgent(task) +task = Task("image_generation_quality", datamodule=PrunaDataModule.from_string("LAION256")) +eval_agent = EvaluationAgent(task) eval_agent.evaluate(smashed_model) ``` - This was the minimal example, but you are looking for the maximal example? You can check out our [documentation][documentation] for an overview of all supported [algorithms][docs-algorithms] as well as our tutorials for more use-cases and examples. - ## Pruna Heart Pruna Pro Pruna has everything you need to get started on optimizing your own models. To push the efficiency of your models even further, we offer Pruna Pro. To give you a glimpse of what is possible with Pruna Pro, let us consider three of the most widely used diffusers pipelines and see how much smaller and faster we can make them. In addition to popular open-source algorithms, we use our proprietary Auto Caching algorithm. We compare the fidelity of the compressed models. Fidelity measures the similarity between the images of the compressed models and the images of the original model. @@ -177,298 +155,20 @@ For [HunyuanVideo](https://huggingface.co/tencent/HunyuanVideo), we compare Auto ## Pruna Cool Algorithm Overview -Since Pruna offers a broad range of compression algorithms, the following table provides an overview of all methods available in Pruna and those exclusive to Pruna Pro. For a detailed description of each algorithm, have a look at our [documentation](https://docs.pruna.ai/en/stable/). - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Algorithm
Pruna Pro
Type
Hardware
Model Format
CPUGPUπŸ€— Transformers CausalLMπŸ€— Diffusers PipelineπŸ€— Transformers Whispertorch Module
AWQquantizerCheckCheck
GPTQquantizerCheckCheck
HQQquantizerCheckCheckCheck
Int8quantizerCheckCheckCheck
QUANTOquantizerCheckCheckCheckCheck
Torch DynamicquantizerCheckCheckCheckCheck
HIGGSCheckquantizerCheckCheck
torchaoCheckquantizerCheckCheckCheckCheckCheckCheck
PERPCheckrecovererCheckCheckCheckCheck
c_translatecompilerCheckCheck
IPEXCheckcompilerCheckCheck
Stable FastcompilerCheckCheck
torch.compilecompilerCheckCheckCheckCheckCheckCheck
x-fastCheckcompilerCheckCheckCheckCheckCheck
DeepCache1cacherCheckCheckCheck
Adaptive CachingCheckcacherCheckCheckCheck
Auto CachingCheckcacherCheckCheckCheck
FLUX Caching2CheckcacherCheckCheckCheck
Periodic CachingCheckcacherCheckCheckCheck
HYPER3CheckdistillerCheckCheckCheck
Structured PruningprunerCheckCheckCheck
Unstructured PruningprunerCheckCheckCheckCheck
ifwbatcherCheckCheck
ws2tbatcherCheckCheck
- -1. Only available for unet-based diffusers pipelines.
-2. Only available for FLUX models.
-3. Only available for FLUX, SD-XL, SD-v1-4, SD-v1-5, SD-3.5. +Since Pruna offers a broad range of compression algorithms, the following table provides a high-level overview of all methods available in Pruna. For a detailed description of each algorithm, have a look at our [documentation](https://docs.pruna.ai/en/stable/). + + +| Technique | Description | Speed | Memory | Accuracy | +| --- | --- | --- | --- | --- | +| Batching | Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing overall processing time. | βœ… | ❌ | 〰️ | +| Caching | Stores intermediate results of computations to speed up subsequent operations, reducing inference time by reusing previously computed results. | βœ… | 〰️ | 〰️ | +| Compilation | Compilation optimises the model with instructions for specific hardware. | βœ… | βž– | 〰️ | +| Distillation | Trains a smaller, simpler model to mimic a larger, more complex model. | βœ… | βœ… | ❌ | +| Quantization | Reduces the precision of weights and activations, lowering memory requirements. | βœ… | βœ… | ❌ | +| Pruning | Removes less important or redundant connections and neurons, resulting in a sparser, more efficient network. | βœ… | βœ… | ❌ | +| Recovering | Restores the performance of a model after compression. | 〰️ | 〰️ | βœ… | + +βœ…(improves), βž–(stays the same), 〰️(could worsen), ❌(worsens)

From 9967983ef980d86114adf075fec52415e455b8b3 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 10:10:17 +0200 Subject: [PATCH 03/31] docs: remove contributions table and update how to contribute section with algorithm overview --- docs/contributions/contributions_toc.rst | 10 -- docs/contributions/how_to_contribute.rst | 129 +++--------------- docs/contributions/opening_an_issue.rst | 23 +++- .../telemetry.rst | 0 4 files changed, 38 insertions(+), 124 deletions(-) delete mode 100644 docs/contributions/contributions_toc.rst rename docs/{user_manual => contributions}/telemetry.rst (100%) diff --git a/docs/contributions/contributions_toc.rst b/docs/contributions/contributions_toc.rst deleted file mode 100644 index 90bcb5f6..00000000 --- a/docs/contributions/contributions_toc.rst +++ /dev/null @@ -1,10 +0,0 @@ -.. toctree:: - :hidden: - :maxdepth: 1 - :caption: Contributing - - /docs_pruna/contributions/how_to_contribute - /docs_pruna/contributions/opening_an_issue - /docs_pruna/contributions/adding_algorithm - /docs_pruna/contributions/adding_metric - /docs_pruna/contributions/adding_dataset \ No newline at end of file diff --git a/docs/contributions/how_to_contribute.rst b/docs/contributions/how_to_contribute.rst index a18614c6..2da784e0 100644 --- a/docs/contributions/how_to_contribute.rst +++ b/docs/contributions/how_to_contribute.rst @@ -1,121 +1,24 @@ -How to Contribute πŸ’œ -=============================== +Since Pruna offers a broad range of compression algorithms, the following table provides a high-level overview of all methods available in Pruna. For a detailed description of each algorithm, have a look at our [documentation](https://docs.pruna.ai/en/stable/). -Since you landed on this part of the documentation, we want to first of all say thank you! πŸ’œ -Contributions from the community are essential to improving |pruna|, we appreciate your effort in making the repository better for everyone! - -Please make sure to review and adhere to the `Pruna Code of Conduct `_ before contributing to Pruna. -Any violations will be handled accordingly and result in a ban from the Pruna community and associated platforms. -Contributions that do not adhere to the code of conduct will be ignored. - -There are various ways you can contribute: - -- Have a question? Discuss with us on `Discord `_ or check out the :doc:`/resources/faq` -- Have an idea for a new tutorial? Open an issue with a :ref:`feature-request` or chat with us on `Discord `_ -- Found a bug? Open an issue with a :ref:`bug-report` -- Want a new feature? Open an issue with a :ref:`feature-request` -- Have a new algorithm to add? Check out: :doc:`adding_algorithm` -- Have a new metric to add? Check out: :doc:`adding_metric` -- Have a new dataset to add? Check out: :doc:`adding_dataset` - - -.. _how-to-contribute: - -Setup ------ - -If you want to contribute to |pruna| with a Pull Request, you can do so by following these steps. -If it is your very first time contributing to an open source project, we recommend to start with `this guide `_ for some generally helpful tips. - -1. Clone the repository -^^^^^^^^^^^^^^^^^^^^^^^^ - -First, fork the repository by navigating to the original `pruna repository `_ on GitHub and click the **Fork** button at the top-right. -This creates a copy of the repository in your own GitHub account. -Then, clone the forked repository from your account to your local machine and change into its directory: - -.. code-block:: bash - - git clone https://github.com/your_username/pruna.git - cd pruna - -To keep your fork up to date with the original repository, add the upstream remote: - -.. code-block:: bash - - git remote add upstream https://github.com/PrunaAI/pruna.git - -Always work on a new branch rather than the main branch. You can create a new branch for your feature or fix: - -.. code-block:: bash - - git checkout -b feat/new-feature - - - -2. Installation -^^^^^^^^^^^^^^^^^^^^^^ - -You can now set up a virtual environment of your choice and install the dependencies by running the following command: - -.. code-block:: bash - - pip install -e . - pip install -e .[dev] - pip install -e .[tests] - -You can then also install the pre-commit hooks with - -.. code-block:: bash - - pre-commit install - - -3. Develop your contribution -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -You are now ready to work on your contribution. Check out a branch on your forked repository and start coding! -When committing your changes, we recommend to follow the `Conventional Commit Guidelines `_. - -.. code-block:: bash - - git checkout -b feat/new-feature - git add . - git commit -m "feat: new amazing feature setup" - git push origin feat/new-feature - -Make sure to develop your contribution in a way that is well documented, concise and easy to maintain. -We will do our best to have your contribution integrated and maintained into |pruna| but reserve the right to reject contributions that we do not feel are in the best interest of the project. - -4. Run the tests -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -We have a comprehensive test suite that is designed to catch potential issues before they are merged into |pruna|. -When you make a contribution, it is highly recommended to not only run the existing tests but also to add new tests that cover your contribution. - -You can run the tests by running the following command: - -.. code-block:: bash - - pytest - -If you want to run only the tests with a specific marker, e.g. fast CPU tests, you can do so by running: - -.. code-block:: bash - - pytest -m "cpu and not slow" - - -5. Create a Pull Request -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Once you have made your changes and tested them, you can create a Pull Request. -We will then review your Pull Request and get back to you as soon as possible. -If there are any questions along the way, please do not hesitate to reach out on `Discord `_. +| Technique | Description | Speed | Memory | Accuracy | +| --- | --- | --- | --- | --- | +| Batching | Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing overall processing time. | βœ… | ❌ | 〰️ | +| Caching | Stores intermediate results of computations to speed up subsequent operations, reducing inference time by reusing previously computed results. | βœ… | 〰️ | 〰️ | +| Compilation | Compilation optimises the model with instructions for specific hardware. | βœ… | βž– | 〰️ | +| Distillation | Trains a smaller, simpler model to mimic a larger, more complex model. | βœ… | βœ… | ❌ | +| Quantization | Reduces the precision of weights and activations, lowering memory requirements. | βœ… | βœ… | ❌ | +| Pruning | Removes less important or redundant connections and neurons, resulting in a sparser, more efficient network. | βœ… | βœ… | ❌ | +| Recovering | Restores the performance of a model after compression. | 〰️ | 〰️ | βœ… | +βœ…(improves), βž–(stays the same), 〰️(could worsen), ❌(worsens) +

+

Pruna AI Logo

+
+## Pruna Sad FAQ and Troubleshooting +If you can not find an answer to your question or problem in our [documentation][documentation], in our [FAQs][docs-faq] or in an existing issue, we are happy to help you! You can either get help from the Pruna community on [Discord][discord], join our [Office Hours][docs-office-hours] or open an issue on GitHub. \ No newline at end of file diff --git a/docs/contributions/opening_an_issue.rst b/docs/contributions/opening_an_issue.rst index ae513e98..acda6b8f 100644 --- a/docs/contributions/opening_an_issue.rst +++ b/docs/contributions/opening_an_issue.rst @@ -1,6 +1,27 @@ Opening an Issue =============================== + +.. _documentation-improvement: + +Documentation Improvement +------------------------ + +All bits help! We appreciate your interest in improving |pruna|’s documentation. + +Our documentation is built with `Sphinx `_ and `Read the Docs `_. + +The current set-up relies on restructured text (rst) files for the documentation and forces us to evaluate and build the documentation on our side. +This means you cannot directly evaluate the documentation changes on your local machine, however, you can still make changes to the documentation and create a pull request based on the changes. + +When opening a pull request for a documentation improvement, you will encounter the following template to help you structure your suggestion. Make sure to fill out all sections applicable to your feature request so that we can integrate it in Pruna as fast as possible: + +.. literalinclude:: issue_templates/documentation_improvement.md + :language: markdown + :linenos: + :lines: 9- + + .. _bug-report: Bug Report @@ -33,7 +54,7 @@ When opening a bug report on GitHub, you will encounter the following template t Feature Request --------------- -We appreciate your interest in improving |pruna|! Feature requests help shape the project, and we welcome ideas that align with our mission. +We appreciate your interest in improving |pruna|! Feature requests help shape the project, and we welcome ideas that align with our mission. Before submitting your feature request, consider the following points to ensure your request is clear and actionable: diff --git a/docs/user_manual/telemetry.rst b/docs/contributions/telemetry.rst similarity index 100% rename from docs/user_manual/telemetry.rst rename to docs/contributions/telemetry.rst From cc922bd25a97e9329e3c3bb3fc959edcef5551fa Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 10:11:05 +0200 Subject: [PATCH 04/31] docs: remove outdated ASR Whisper tutorial and add new tutorial index --- docs/tutorials/asr_whisper.ipynb | 194 ------------------------------- docs/tutorials/index.rst | 65 +++++++++++ 2 files changed, 65 insertions(+), 194 deletions(-) delete mode 100644 docs/tutorials/asr_whisper.ipynb create mode 100644 docs/tutorials/index.rst diff --git a/docs/tutorials/asr_whisper.ipynb b/docs/tutorials/asr_whisper.ipynb deleted file mode 100644 index a8896a0d..00000000 --- a/docs/tutorials/asr_whisper.ipynb +++ /dev/null @@ -1,194 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 100% faster Whisper Transcription" - ] - }, - { - "cell_type": "raw", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, - "source": [ - "\n", - " \"Open\n", - "" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This tutorial demonstrates how to use the `pruna` package to optimize any custom whisper model. We will use the `openai/whisper-large-v3` model as an example." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# if you are not running the latest version of this tutorial, make sure to install the matching version of pruna\n", - "# the following command will install the latest version of pruna\n", - "!pip install pruna" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 1. Loading the ASR model\n", - "\n", - "First, load your ASR model." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import torch\n", - "from transformers import AutoModelForSpeechSeq2Seq\n", - "\n", - "device = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\n", - "torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32\n", - "\n", - "model_id = \"openai/whisper-large-v3\"\n", - "\n", - "model = AutoModelForSpeechSeq2Seq.from_pretrained(\n", - " model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True\n", - ")\n", - "model.to(device)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 2. Initializing the Smash Config\n", - "\n", - "Next, initialize the smash_config. Since the compiler requires a processor, we add it to the smash_config." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "from pruna import SmashConfig\n", - "\n", - "# Initialize the SmashConfig\n", - "smash_config = SmashConfig()\n", - "smash_config.add_processor(model_id)\n", - "smash_config['compiler'] = 'c_whisper'\n", - "# uncomment the following line to quantize the model to 8 bits\n", - "# smash_config['c_whisper_weight_bits'] = 8" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 3. Smashing the Model\n", - "\n", - "Now, you can smash the model, which will take approximately 2 minutes on a T4 GPU." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from pruna import smash\n", - "\n", - "# Smash the model\n", - "smashed_model = smash(\n", - " model=model,\n", - " smash_config=smash_config,\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 4. Preparing the Input" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from datasets import load_dataset\n", - "from transformers import AutoProcessor\n", - "\n", - "processor = AutoProcessor.from_pretrained(model_id)\n", - "\n", - "dataset = load_dataset(\"distil-whisper/librispeech_long\", \"clean\", split=\"validation\")\n", - "sample = dataset[0][\"audio\"]\n", - "input_features = processor(sample[\"array\"], sampling_rate=sample[\"sampling_rate\"], return_tensors=\"pt\").input_features\n", - "input_features = input_features.cuda().half()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 5. Running the Model\n", - "\n", - "Finally, run the model to transcribe the audio file." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "# Display the result\n", - "results = smashed_model(input_features)\n", - "processor.decode(results, skip_special_tokens=False)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Wrap Up\n", - "\n", - "Congratulations! You have successfully smashed an ASR model. You can now use the `pruna` package to optimize any custom ASR model. The only parts that you should modify are step 1, 4 and 5 to fit your use case." - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "pruna", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.11" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/docs/tutorials/index.rst b/docs/tutorials/index.rst new file mode 100644 index 00000000..235866fa --- /dev/null +++ b/docs/tutorials/index.rst @@ -0,0 +1,65 @@ +.. _pruna_tutorials: + +Tutorials Pruna +=============== + +This tutorial will guide you through the process of using |pruna| to optimize your model. Looking for |pruna_pro| tutorials? Check out the :ref:`pruna_pro_tutorials` page. + +.. grid:: 1 2 2 2 + + .. grid-item-card:: Transcribe 2 hour of audio in 2 minutes with Whisper + :text-align: center + :link: ./asr_tutorial.ipynb + + Speed up ASR using the ``c_whisper`` ``compilation`` and ``whisper_s2t`` ``batching``. + + .. grid-item-card:: Smash your Computer Vision model with a CPU only + :text-align: center + :link: ./cv_cpu.ipynb + + ``Compile`` your model with ``torch_compile`` and ``openvino`` for faster inference. + + .. grid-item-card:: Speedup and Quantize any Diffusion Model + :text-align: center + :link: ./diffusion_quantization_acceleration.ipynb + + Speed up ``diffusers`` with ``torch_compile`` ``compilation`` and ``hqq_diffusers`` ``quantization``. + + .. grid-item-card:: Evaluating with CMMD using EvaluationAgent + :text-align: center + :link: ./evaluation_agent_cmmd.ipynb + + ``Evaluate`` image generation quality with ``CMMD`` and ``EvaluationAgent``. + + .. grid-item-card:: Run your Flux model with half the memory + :text-align: center + :link: ./flux_small.ipynb + + Speed up your image generation model with ``torch_compile`` ``compilation`` and ``hqq_diffusers`` ``quantization``. + + .. grid-item-card:: Making your LLMs 4x smaller + :text-align: center + :link: ./llms.ipynb + + Speed up your LLM inference with ``gptq`` ``quantization``. + + .. grid-item-card:: x2 smaller Sana diffusers in action + :text-align: center + :link: ./sana_diffusers_int8.ipynb + + Optimize your ``diffusion`` model with ``hqq_diffusers`` ``quantization`` in 8 bits. + + .. grid-item-card:: Make Stable Diffusion 3x Faster with DeepCache + :text-align: center + :link: ./sd_deepcache.ipynb + + Optimize your ``diffusion`` model with ``deepcache`` ``caching``. + + +.. toctree:: + :hidden: + :maxdepth: 1 + :caption: Pruna + :glob: + + ./* \ No newline at end of file From 6d12eacc447cf2ad1cf0bff11f1f554a09903896 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 10:17:54 +0200 Subject: [PATCH 05/31] docs: add user manual sections for customizing algorithms, datasets, metrics, and model optimization --- .../adding_algorithm.rst | 186 +++++----- .../adding_dataset.rst | 33 +- .../adding_metric.rst | 0 docs/user_manual/configure.rst | 320 +++++++++++++++++ docs/user_manual/customize.rst | 39 +++ docs/user_manual/dataset.rst | 106 ------ docs/user_manual/evaluate.rst | 219 ++++++++++++ docs/user_manual/evaluation.rst | 330 ------------------ docs/user_manual/optimize.rst | 207 +++++++++++ docs/user_manual/save_load.rst | 32 +- docs/user_manual/smash.rst | 57 --- docs/user_manual/smash_config.rst | 67 ---- 12 files changed, 912 insertions(+), 684 deletions(-) rename docs/{contributions => user_manual}/adding_algorithm.rst (92%) rename docs/{contributions => user_manual}/adding_dataset.rst (89%) rename docs/{contributions => user_manual}/adding_metric.rst (100%) create mode 100644 docs/user_manual/configure.rst create mode 100644 docs/user_manual/customize.rst delete mode 100644 docs/user_manual/dataset.rst create mode 100644 docs/user_manual/evaluate.rst delete mode 100644 docs/user_manual/evaluation.rst create mode 100644 docs/user_manual/optimize.rst delete mode 100644 docs/user_manual/smash.rst delete mode 100644 docs/user_manual/smash_config.rst diff --git a/docs/contributions/adding_algorithm.rst b/docs/user_manual/adding_algorithm.rst similarity index 92% rename from docs/contributions/adding_algorithm.rst rename to docs/user_manual/adding_algorithm.rst index 9f9da18e..5c90dbbd 100644 --- a/docs/contributions/adding_algorithm.rst +++ b/docs/user_manual/adding_algorithm.rst @@ -1,29 +1,28 @@ -Adding an Algorithm +Customize Algorithms ==================== -Adding the Algorithm to ``pruna.algorithms`` --------------------------------------------- - -If you’ve developed a new method or want to integrate a missing algorithm into |pruna|, we welcome your contribution! This tutorial guides you through the steps to integrate a new compression algorithm, making it available for all users. +If you’ve developed a new method or want to integrate a missing algorithm into |pruna|, we welcome your contribution! This tutorial guides you through the steps to integrate a new compression algorithm, making it available for all users. If anything is unclear or you want to discuss your contribution before opening a PR, please reach out on `Discord `_ anytime! If this is your first time contributing to |pruna|, please refer to the :ref:`how-to-contribute` guide for more information. +Add a Custom Algorithm +---------------------- + We’ll use **Superfast**, an example compiler, to demonstrate the process. -0. Identifying the Algorithm Group -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Step 1. Identify the Algorithm Group +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The first step is to identify the algorithm group. This is important because it determines the folder in which the algorithm should be placed. You can find the list of all algorithm groups in the :doc:`Compression Algorithms <../../compression>` section and determine which group fits your algorithm best by reviewing the algorithm group descriptions. -1. Creating the Compiler Class -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Step 2. Create the Compiler Class +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ First, navigate to ``pruna/algorithms/compilation/`` and create ``superfast.py``. - -2. Defining Compiler Attributes -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Step 3. Define Compiler Attributes +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Define the new compiler by inheriting from ``PrunaCompiler`` and define key attributes for the compiler. These attributes are used to provide information about the algorithm to the user, other functions in the package and even the documentation. @@ -40,10 +39,10 @@ These attributes are used to provide information about the algorithm to the user class SuperfastCompiler(PrunaCompiler): """ Implement Superfast Compiler using the superfast package. - + This compiler compiles anything with zero compilation time and 100x speedup. """ - + algorithm_name = "superfast" references = {"GitHub": "/url/to/GitHub"} tokenizer_required = False @@ -54,8 +53,9 @@ These attributes are used to provide information about the algorithm to the user compatible_algorithms = dict(quantizer=["quanto"]) -Explanation -^^^^^^^^^^^^ +Step 4. Add Algorithm Attributes +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + - docstring: The docstring should be concise and describe the algorithm in a way that is easy to understand. The description paragraph of the algorithm will be used to automatically generate the algorithm's documentation. - ``algorithm_name``: Identifier used to activate the algorithm, name should be in snake case. - ``references``: A dictionary of any references that can be provided for the algorithm, typically a link to the GitHub repository or a paper. @@ -65,8 +65,8 @@ Explanation - Additionally, you might have to specify a saving function. We provide more details on this in the section below. -Defining Hyperparameters -^^^^^^^^^^^^^^^^^^^^^^^^ +Step 5. Define Hyperparameters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Define hyperparameters using `ConfigSpace `_, allowing users to configure the backend and mode. Everything that configures the algorithm or specifies the algorithm's behavior should be a hyperparameter. @@ -81,13 +81,13 @@ Everything that configures the algorithm or specifies the algorithm's behavior s CategoricalHyperparameter("mode", choices=["mode1", "mode2"], default_value="mode1", meta=dict(desc="The mode to use for the Superfast compiler.")), ] -Users can now configure hyperparameters via ``smash_config["superfast_backend"] = "backend2"``. +Users can now configure hyperparameters via ``smash_config["superfast_backend"] = "backend2"``. Make sure to include descriptions of the hyperparameters with the ``desc`` key in the ``meta`` dictionary. This will be used later to document the hyperparameters in the algorithm's documentation. -Checking Model Compatibility -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Step 6. Check Model Compatibility +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Ensure the compiler only runs on supported models. In our example, the Superfast compiler is compatible with any model that is a subclass of ``torch.nn.Module``: @@ -101,8 +101,8 @@ Ensure the compiler only runs on supported models. In our example, the Superfast Users can bypass this check using ``experimental=True`` when calling ``smash``, but results may be unpredictable. -Handling External Dependencies -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Step 7. Handle External Dependencies +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If the compiler requires external packages, isolate their imports: @@ -116,8 +116,9 @@ If the compiler requires external packages, isolate their imports: Make sure that the dependencies are listed in ``pyproject.toml`` if they are not already included. -Implementing the Compilation Process -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Step 8. Implement the Compilation Process +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + The ``_apply()`` function integrates superfast with Pruna: @@ -132,63 +133,9 @@ The ``_apply()`` function integrates superfast with Pruna: Note that the ``smash_config`` prefix wrapper automatically prefixes hyperparameters with the algorithm name (``superfast_``). If a user sets ``smash_config["superfast_backend"]``, it will be mapped correctly to ``"backend"`` in ``get_hyperparameters()``. -Full Implementation -^^^^^^^^^^^^^^^^^^^^ - -Here’s the complete ``superfast.py`` implementation: - -.. code-block:: python - - from typing import Any, Dict - import torch - from ConfigSpace import CategoricalHyperparameter - from pruna.algorithms.compilation import PrunaCompiler - from pruna.config.smash_config import SmashConfigPrefixWrapper - - class SuperfastCompiler(PrunaCompiler): - """ - Implement Superfast Compiler using the superfast package. - - This compiler compiles anything with zero compilation time and 100x speedup. - """ - - algorithm_name = "superfast" - references = {"GitHub": "/url/to/GitHub"} - tokenizer_required = False - processor_required = False - dataset_required = False - run_on_cpu = True - run_on_cuda = True - compatible_algorithms = dict(quantizer=["quanto"]) - - def get_hyperparameters(self) -> list: - return [ - CategoricalHyperparameter("backend", choices=["backend1", "backend2"], default_value="backend1"), - CategoricalHyperparameter("mode", choices=["mode1", "mode2"], default_value="mode1"), - ] - - def model_check_fn(self, model: Any) -> bool: - return isinstance(model, torch.nn.Module) - - def import_algorithm_packages(self) -> Dict[str, Any]: - from superfast import compile_func - return dict(compile_func=compile_func) - - def _apply(self, model: Any, smash_config: SmashConfigPrefixWrapper) -> Any: - compile_func = self.import_algorithm_packages()["compile_func"] - return compile_func(model, smash_config["backend"], smash_config["mode"]) - -.. container:: hidden_code - - .. code-block:: python - - # test instantiation of compiler - SuperfastCompiler() - - +Step 9. Determine the Saving Function +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Determining a Suitable Saving Function ----------------------------------------- Saving e.g. a compiled or quantized model can be tricky and requires careful consideration. To determine the correct saving function for your algorithm, consider the decision tree below. .. mermaid:: @@ -206,21 +153,19 @@ Saving e.g. a compiled or quantized model can be tricky and requires careful con G -->|Yes| L["SAVE_FUNCTIONS.pickled"] G -->|No| M["Introduce new saving function."] -The first decision is whether the original saving function can be retained. +The first decision is whether the original saving function can be retained. For example, GPTQ-quantized transformers models still support ``.from_pretrained`` and ``.save_pretrained``, making retention possible. -If the original function cannot be retained, we consider how long the algorithm takes to apply. -If it is quick (e.g., a caching helper), we can reapply it after loading. -The key distinction is whether the modifications persist when saving. For instance, β€œstep caching cacher” attaches a helper that is discarded by ``diffusers`` upon saving, so the model can be saved and reloaded normally before reapplying the function. +If the original function cannot be retained, we consider how long the algorithm takes to apply. +If it is quick (e.g., a caching helper), we can reapply it after loading. +The key distinction is whether the modifications persist when saving. For instance, β€œstep caching cacher” attaches a helper that is discarded by ``diffusers`` upon saving, so the model can be saved and reloaded normally before reapplying the function. In contrast, compilation is irreversibleβ€”once compiled, a model cannot be saved in its compiled form, so we must save it beforehand and reapply compilation after loading. -If neither approach works, we must introduce a new saving function or use ``SAVE_FUNCTIONS.pickled``. We implement a new saving function following the existing saving-function pattern as well as introducing a matching loading function. +If neither approach works, we must introduce a new saving function or use ``SAVE_FUNCTIONS.pickled``. We implement a new saving function following the existing saving-function pattern as well as introducing a matching loading function. Otherwise, we can resort to saving the model in pickled format, but be aware that pickled models pose security risks and are generally not trusted by the community. - - -Testing the Algorithm ----------------------- +Step 10. Test the Algorithm +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To integrate the algorithm into the test suite, we navigate to ``tests/algorithms/testers/compilation.py`` and add the following Tester Class: @@ -239,14 +184,14 @@ To integrate the algorithm into the test suite, we navigate to ``tests/algorithm dummy_algorithm_tester = types.ModuleType("pruna.algorithms.testers.compilation") dummy_algorithm_tester.AlgorithmTesterBase = ABC sys.modules["base_tester"] = dummy_algorithm_tester - + .. code-block:: python from base_tester import AlgorithmTesterBase from pruna.algorithms.compilation.superfast import SuperfastCompiler from pruna import PrunaModel - + class TestSuperfast(AlgorithmTesterBase): """Tester class for the Superfast algorithm.""" @@ -271,9 +216,62 @@ This Tester class automatically parametrizes an integration test at ``tests/algo Additionally, a test is created to check that ``model_check_fn`` rejects a non-compatible model. Before opening a PR, make sure to run the test suite locally to ensure the algorithm is working as expected. + +Full Implementation +------------------- + +Here’s the complete ``superfast.py`` implementation: + +.. code-block:: python + + from typing import Any, Dict + import torch + from ConfigSpace import CategoricalHyperparameter + from pruna.algorithms.compilation import PrunaCompiler + from pruna.config.smash_config import SmashConfigPrefixWrapper + + class SuperfastCompiler(PrunaCompiler): + """ + Implement Superfast Compiler using the superfast package. + + This compiler compiles anything with zero compilation time and 100x speedup. + """ + + algorithm_name = "superfast" + references = {"GitHub": "/url/to/GitHub"} + tokenizer_required = False + processor_required = False + dataset_required = False + run_on_cpu = True + run_on_cuda = True + compatible_algorithms = dict(quantizer=["quanto"]) + + def get_hyperparameters(self) -> list: + return [ + CategoricalHyperparameter("backend", choices=["backend1", "backend2"], default_value="backend1"), + CategoricalHyperparameter("mode", choices=["mode1", "mode2"], default_value="mode1"), + ] + + def model_check_fn(self, model: Any) -> bool: + return isinstance(model, torch.nn.Module) + + def import_algorithm_packages(self) -> Dict[str, Any]: + from superfast import compile_func + return dict(compile_func=compile_func) + + def _apply(self, model: Any, smash_config: SmashConfigPrefixWrapper) -> Any: + compile_func = self.import_algorithm_packages()["compile_func"] + return compile_func(model, smash_config["backend"], smash_config["mode"]) + +.. container:: hidden_code + + .. code-block:: python + + # test instantiation of compiler + SuperfastCompiler() + Conclusion ---------- You’ve successfully integrated a new compiler into Pruna! πŸš€ -Now, users can utilize Superfast for model compilation, configure its hyperparameters, and ensure compatibility. - +Now, users can utilize Superfast for model compilation, configure its hyperparameters, and ensure compatibility. \ No newline at end of file diff --git a/docs/contributions/adding_dataset.rst b/docs/user_manual/adding_dataset.rst similarity index 89% rename from docs/contributions/adding_dataset.rst rename to docs/user_manual/adding_dataset.rst index b3d82345..e6fb7e78 100644 --- a/docs/contributions/adding_dataset.rst +++ b/docs/user_manual/adding_dataset.rst @@ -1,18 +1,21 @@ -Adding a Dataset -=============================== +Customize Datasets +================== -Our interface makes it easy to add :doc:`your own dataset <../user_manual/dataset>`. +Our interface makes it easy to add :doc:`your own dataset <../user_manual/dataset>`. Additionally, we provide a variety of :doc:`preconfigured datasets <../user_manual/dataset>` that can be readily used in SmashConfig for calibration or evaluation. -If you’d like to contribute a new dataset to our supported list, follow these two quick steps. +If you’d like to contribute a new dataset to our supported list, follow these two quick steps. If anything is unclear or you want to discuss your contribution before opening a PR, please reach out on `Discord `_ anytime! If this is your first time contributing to |pruna|, please refer to the :ref:`how-to-contribute` guide for more information. -1. Define the Dataset Setup -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Add a Custom Dataset +-------------------- -First, create a setup method to prepare the training, validation, and test splits. -This usually involves downloading or generating the dataset. +Step 1. Define the Dataset Setup +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +First, create a setup method to prepare the training, validation, and test splits. +This usually involves downloading or generating the dataset. For a text generation dataset, add the setup method in ``pruna/data/datasets/text_generation.py``: .. code-block:: python @@ -59,7 +62,7 @@ with the matching collate function and any defaults (e.g. the default image size base_datasets["NewDataset"] = (setup_new_dataset, "text_generation_collate", {}) -Ensure the dataset follows the expected format specified in the :doc:`collate function <../user_manual/dataset>`. +Ensure the dataset follows the expected format specified in the :doc:`collate function <../user_manual/dataset>`. The collate function aggregates several samples into a batch and converts them to the expected format. Now, users can add the dataset like this: @@ -74,16 +77,16 @@ Now, users can add the dataset like this: .. container:: hidden_code - + .. code-block:: python - + # test if dataloader works as expected for batch in smash_config.test_dataloader(): break -2. Add a Test +Step 2. Add a Test ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To verify that the dataset loads correctly, add it to ``tests/data/test_datamodule.py`` by parameterizing ``test_dm_from_string`` @@ -94,11 +97,11 @@ To verify that the dataset loads correctly, add it to ``tests/data/test_datamodu pytest.param("NewDataset", dict(img_size=512), marks=pytest.mark.slow) -Include necessary arguments for the collate function and mark the test as slow if needed. +Include necessary arguments for the collate function and mark the test as slow if needed. We categorize a test as slow if it requires several minutes to download and prepare the dataset. This ensures it runs appropriately in CI, either on GitHub Actions or nightly tests. Conclusion -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +---------- -That’s it! Your dataset is now available for everyone to use in Pruna. πŸ’œ +That’s it! Your dataset is now available for everyone to use in |pruna|. πŸ’œ \ No newline at end of file diff --git a/docs/contributions/adding_metric.rst b/docs/user_manual/adding_metric.rst similarity index 100% rename from docs/contributions/adding_metric.rst rename to docs/user_manual/adding_metric.rst diff --git a/docs/user_manual/configure.rst b/docs/user_manual/configure.rst new file mode 100644 index 00000000..00602b3e --- /dev/null +++ b/docs/user_manual/configure.rst @@ -0,0 +1,320 @@ +Define a SmashConfig +==================== + +This guide provides an introduction to configuring model optimization strategies with |pruna|. + +Model optimization configuration relies on the ``SmashConfig`` class. +The ``SmashConfig`` class provides a flexible dictionary-like interface for configuring model optimization strategies. +It manages algorithms, hyperparameters, and additional components like tokenizers, processors and datasets. + +Basic Configuration Workflow +---------------------------- + +|pruna| follows a simple workflow for configuring model optimization strategies: + +.. mermaid:: + :align: center + + graph LR + User -->|creates| SmashConfig + User -->|loads| PreTrainedModel["Pre-trained Model"] + + subgraph "Configuration Components" + SmashConfig --- Algorithm["Algorithm Selection"] + SmashConfig --- Hyperparameters + SmashConfig --- Tokenizer["Tokenizer (optional)"] + SmashConfig --- Processor["Processor (optional)"] + SmashConfig --- Dataset["Dataset (optional)"] + end + + SmashConfig -->|configures| SmashFn["smash() function"] + PreTrainedModel -->|input to| SmashFn + SmashFn -->|returns| OptimizedModel["Optimized PrunaModel"] + + style User fill:#bbf,stroke:#333,stroke-width:2px + style PreTrainedModel fill:#bbf,stroke:#333,stroke-width:2px + style SmashConfig fill:#bbf,stroke:#333,stroke-width:2px + style SmashFn fill:#bbf,stroke:#333,stroke-width:2px + style OptimizedModel fill:#bbf,stroke:#333,stroke-width:2px + +Let's see what that looks like in code. + +.. code-block:: python + + from pruna import SmashConfig + + smash_config = SmashConfig() + + # Activate IFW batching + smash_config['batcher'] = 'ifw' + + # Set IFW batching parameters + smash_config['ifw_weight_bits'] = 16 + smash_config['ifw_group_size'] = 4 + + # Add a tokenizer + smash_config.add_tokenizer('bert-base-uncased') + +Configure Algorithms +-------------------- + +|pruna| implements a extensible architecture for optimization algorithms. +Each algorithm has its own impact on the model in terms of speed, memory and accuracy. +The table underneath provides a general overview of the impact of each algorithm group. + +.. list-table:: + :widths: 10 60 10 10 10 + :header-rows: 1 + + * - Technique + - Description + - Speed + - Memory + - Accuracy + * - ``batcher`` + - Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing processing time. + - βœ… + - ❌ + - ～ + * - ``cacher`` + - Stores intermediate results of computations to speed up subsequent operations. + - βœ… + - ～ + - ～ + * - ``compiler`` + - Optimises the model with instructions for specific hardware. + - βœ… + - βž– + - ～ + * - ``distiller`` + - Trains a smaller, simpler model to mimic a larger, more complex model. + - βœ… + - βœ… + - ❌ + * - ``quantizer`` + - Reduces the precision of weights and activations, lowering memory requirements. + - βœ… + - βœ… + - ❌ + * - ``pruner`` + - Removes less important or redundant connections and neurons, resulting in a sparser, more efficient network. + - βœ… + - βœ… + - ❌ + * - ``recoverer`` + - Restores the performance of a model after compression. + - ～ + - ～ + - βœ… + +βœ…(improves), βž–(stays the same), ～(could worsen), ❌(worsens) + +.. tip:: + + The :doc:`Algorithm Overview ` page provides a more detailed overview of each algorithm within the different groups. + As well as additional information on the hardware requirements, compatibility with other algorithms and required components for each algorithm. + +Configure Algorithm Groups +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To activate an algorithm, you assign its name to the corresponding algorithm group in the ``SmashConfig``. +The group names are outlined in the table above and the specific algorithms are shown in the :doc:`Algorithm Overview ` page. + +Let's activate the ``ifw`` algorithm as a ``batcher``: + +.. code-block:: python + + from pruna import SmashConfig + + smash_config = SmashConfig() + + # Activate IFW batching + smash_config['batcher'] = 'ifw' + +Configure Algorithm Hyperparameters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Each algorithm has its own set of hyperparameters that control its behavior. +These are automatically prefixed with the algorithm name and can also be found underneath each algorithm in the :doc:`Algorithm Overview `. + +Let's add the ``ifw_weight_bits`` and ``ifw_group_size`` hyperparameters for the ``ifw`` we defined above: + +.. code-block:: python + + from pruna import SmashConfig + + smash_config = SmashConfig() + + # Activate IFW batching + smash_config['batcher'] = 'ifw' + + # Set IFW batching parameters + smash_config['ifw_weight_bits'] = 16 + smash_config['ifw_group_size'] = 4 + +Configure Components +-------------------- + +Some algorithms require a tokenizer, processor or dataset to be passed to the SmashConfig. +For example, looking at the :doc:`Algorithm Overview ` we see that the ``gptq`` quantizer requires a dataset and a tokenizer. + +.. list-table:: + :widths: 10 90 10 + :header-rows: 1 + + * - Component + - Description + - Function + * - ``tokenizer`` + - Tokenizes the input text. + - ``add_tokenizer()`` + * - ``processor`` + - Processes the input data. + - ``add_processor()`` + * - ``data`` + - Loads a dataset. + - ``add_dataset()`` + +.. note:: + + If you try to activate a algorithm that requires a dataset, tokenizer or processor and haven’t added them to the ``SmashConfig``, you will receive an error. + Make sure to add them before activating the algorithm! If you want to know which algorithms require a dataset, tokenizer or processor, you can look at the :doc:`Algorithm Overview `. + +Configure Tokenizers, Processors +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +|pruna| provides a directly inherits from the ``transformers`` library. +This means, we can use the same tokenizers and processors as the ones used in the ``transformers`` library. + +.. tabs:: + + .. tab:: String Identifier + + Use a string identifier to use a tokenizer or processor from the Hugging Face Hub. + + .. code-block:: python + + from pruna import SmashConfig + + smash_config = SmashConfig() + + # Add a built-in dataset using a string identifier + smash_config.add_tokenizer('facebook/opt-125m') + smash_config.add_processor('openai/whisper-large-v3') + + .. tab:: Loading Directly + + Load a tokenizer or processor directly from the Hugging Face Hub with your own configuration. + + .. code-block:: python + + from pruna import SmashConfig + from transformers import AutoTokenizer + + smash_config = SmashConfig() + + # Load a tokenizer directly from the Hugging Face Hub + tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m") + smash_config.add_tokenizer(tokenizer) + + # Load a processor directly from the Hugging Face Hub + processor = AutoProcessor.from_pretrained("openai/whisper-large-v3") + smash_config.add_processor(processor) + +Now we've set up the tokenizer and processor, we can use them to process our data. + +Configure Datasets +^^^^^^^^^^^^^^^^^^ + +|pruna| provides a variety of pre-configured datasets for different tasks. +We can use string identifiers to use a built-in dataset or use collate functions to use a custom dataset. +Underneath you can find the list of all the available datasets. + +.. list-table:: + :header-rows: 1 + + * - Task + - Built-in Dataset ID + - Custom Collate Function + - Collate Function Arguments + * - Text Generation + - `WikiText `_, `SmolTalk `_, `SmolSmolTalk `_, `PubChem `_, `OpenAssistant `_, `C4 `_ + - ``text_generation_collate`` + - ``text: str`` + * - Image Generation + - `LAION256 `_, `OpenImage `_, `COCO `_ + - ``image_generation_collate`` + - ``image: PIL.Image.Image``, ``text: str`` + * - Image Classification + - `ImageNet `_, `MNIST `_, `CIFAR10 `_ + - ``image_classification_collate`` + - ``image: PIL.Image.Image``, ``label: int`` + * - Audio Processing + - `CommonVoice `_, `AIPodcast `_ + - ``audio_processing_collate`` + - ``audio: Optional[torch.Tensor]``, ``path: Optional[str]``, ``sentence: str`` + * - Question Answering + - `Polyglot `_ + - ``question_answering_collate`` + - ``question: str``, ``answer: str`` + +Similar to the tokenizers and processors, we can use string identifiers to use a built-in dataset or use a more custom approach, i.e. using a collate function. +Additionallly, you can create a fully custom ``PrunaDataModule`` use it in your workflow. + +.. tabs:: + + .. tab:: String Identifier + + Use a string identifier to use a built-in dataset as defined in the table above. + + .. code-block:: python + + from pruna import SmashConfig + + smash_config = SmashConfig() + + # Add a built-in dataset using a string identifier + smash_config.add_dataset('WikiText') + + .. tab:: Custom Collate Function + + Use a custom collate function to use a custom dataset as ``(train, val, test)`` tuples. + + In this case, you need to specify the ``collate_fn`` to use for the dataset. + The ``collate_fn`` is a function that takes a list of individual data samples and returns a batch of data in a unified format. + Your dataset will have to adhere to the formats expected by the ``collate_fn`` as defined in the table above. + + .. code-block:: python + + from pruna import SmashConfig + from pruna.data.utils import split_train_into_train_val_test + from datasets import load_dataset + + # Load custom datasets + train_ds = load_dataset("SamuelYang/bookcorpus")["train"] + train_ds, val_ds, test_ds = split_train_into_train_val_test(train_ds, seed=42) + + # Add to SmashConfig + smash_config = SmashConfig() + smash_config.add_tokenizer("bert-base-uncased") + smash_config.add_data( + (train_ds, val_ds, test_ds), + collate_fn="text_generation_collate" + ) + + .. tab:: PrunaDataModule + + You can also create a fully custom ``PrunaDataModule`` use it in your workflow. + This process is more flexible but also more complex. It allows for more control over the dataset and the data loading process. + The process for defining a ``PrunaDataModule`` is highlighted in the :doc:`Evaluation ` page but a basic example of adding it to the ``SmashConfig`` is shown below. + + .. code-block:: python + + from pruna import SmashConfig, PrunaDataModule + + # Load PrunaDataModule + data = PrunaDataModule(...) + + # Add to SmashConfig + smash_config = SmashConfig() + smash_config.add_data(data) \ No newline at end of file diff --git a/docs/user_manual/customize.rst b/docs/user_manual/customize.rst new file mode 100644 index 00000000..336872eb --- /dev/null +++ b/docs/user_manual/customize.rst @@ -0,0 +1,39 @@ +Customize components +==================== + +|pruna| is designed to be customizable. You can add your own algorithms, datasets, and metrics to the package. + +.. grid:: 1 3 3 3 + + .. grid-item-card:: Add an algorithm + :text-align: center + :link: ./adding_algorithm.rst + + Steps to integrate a new compression algorithm, making it available in the ``SmashConfig``. + + .. grid-item-card:: Add a dataset + :text-align: center + :link: ./adding_dataset.rst + + Steps to integrate a new dataset, making it available in the ``SmashConfig``. + + .. grid-item-card:: Add a metric + :text-align: center + :link: ./adding_metric.rst + + Steps to integrate a new metric, making it available in the ``SmashConfig``. + +.. tip:: + + You can also customize the package by adding your own algorithms, datasets, and metrics. + Take a look at the :doc:`contributing guide ` to learn more. + If anything is unclear or you want to discuss your contribution before opening a PR, please reach out on `Discord `_ anytime! + +.. toctree:: + :maxdepth: 1 + :caption: Customize components + :hidden: + + adding_algorithm + adding_dataset + adding_metric \ No newline at end of file diff --git a/docs/user_manual/dataset.rst b/docs/user_manual/dataset.rst deleted file mode 100644 index 8094ad37..00000000 --- a/docs/user_manual/dataset.rst +++ /dev/null @@ -1,106 +0,0 @@ -Datasets -========================= - -|pruna| provides a variety of pre-configured datasets for different tasks. This guide will help you understand how to use datasets in your |pruna| workflow. - -Available Datasets -------------------- - -|pruna| currently supports the following datasets categorized by task: - -Text Generation -^^^^^^^^^^^^^^^ - -| ``WikiText``: Wikipedia text dataset for language modeling -| ``SmolTalk``: Everyday conversation dataset -| ``SmolSmolTalk``: Lightweight version of SmolTalk -| ``PubChem``: Chemical compound dataset in SELFIES format -| ``OpenAssistant``: Instruction-following dataset -| ``C4``: Large-scale web text dataset - -Image Classification -^^^^^^^^^^^^^^^^^^^^ - -| ``ImageNet``: Large-scale image classification dataset -| ``MNIST``: Handwritten digit classification dataset - -Text-to-Image -^^^^^^^^^^^^^^^^^^^^ - -| ``COCO``: Image captioning dataset -| ``LAION256``: Subset of LAION artwork dataset -| ``OpenImage``: Image quality preferences dataset - -Audio Processing -^^^^^^^^^^^^^^^^^^^^ - -| ``CommonVoice``: Multi-language speech dataset -| ``AIPodcast``: AI-focused podcast audio dataset - -Question Answering -^^^^^^^^^^^^^^^^^^^^ - -| ``Polyglot``: Fact completion dataset - -Using Datasets ---------------- - -There are two main ways to use datasets in |pruna|: - -1. Using String Identifier -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -What makes using the already implemented datasets so easy is that you can simply use the dataset's string identifier to add it to your :doc:`SmashConfig `: - -.. code-block:: python - - from pruna import SmashConfig - - smash_config = SmashConfig() - smash_config.add_tokenizer("bert-base-uncased") - smash_config.add_data("WikiText") - -2. Using Custom Datasets -^^^^^^^^^^^^^^^^^^^^^^^^^ - -You can also pass your own datasets as a tuple of ``(train, validation, test)`` datasets: - -.. code-block:: python - - from pruna import SmashConfig - from pruna.data.utils import split_train_into_train_val_test - from datasets import load_dataset - - # Load custom datasets - train_ds = load_dataset("SamuelYang/bookcorpus")["train"] - train_ds, val_ds, test_ds = split_train_into_train_val_test(train_ds, seed=42) - - # Add to SmashConfig - smash_config = SmashConfig() - smash_config.add_tokenizer("bert-base-uncased") - smash_config.add_data( - (train_ds, val_ds, test_ds), - collate_fn="text_generation_collate" - ) - -In this case, you need to specify the ``collate_fn`` to use for the dataset. The ``collate_fn`` is a function that takes a list of individual data samples and returns a batch of data in a unified format. -Your dataset will have to adhere to the formats expected by the ``collate_fn`` and this will be checked during a quick compatibility check when adding the dataset to the ``smash_config``. - - -.. autofunction:: pruna.data.collate.text_generation_collate -.. autofunction:: pruna.data.collate.image_generation_collate -.. autofunction:: pruna.data.collate.image_classification_collate -.. autofunction:: pruna.data.collate.audio_collate -.. autofunction:: pruna.data.collate.question_answering_collate - - - -.. _prunadatamodule: - -Accessing the PrunaDataModule directly -------------------------------------- - -You can also create and access the PrunaDataModule directly and use it in your workflow, e.g., if you want to pass it to the :doc:`evaluation agent `. - -.. autoclass:: pruna.data.pruna_datamodule.PrunaDataModule - :members: from_string, from_datasets diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst new file mode 100644 index 00000000..d6de59b4 --- /dev/null +++ b/docs/user_manual/evaluate.rst @@ -0,0 +1,219 @@ +Evaluate optimizations with the Evaluation Agent +================================================ + +This guide provides an introduction to evaluating model optimizations with |pruna|. + +Evaluation helps you understand how compression affects your models across different dimensions - from output quality to resource requirements. +This knowledge is essential for making informed decisions about which compression techniques work best for your specific needs. + +Basic Evaluation Workflow +------------------------- + +|pruna| follows a simple workflow for evaluating model optimizations: + +.. mermaid:: + :align: center + + graph LR + User -->|creates| Task + User -->|creates| EvaluationAgent + Task -->|uses| PrunaDataModule + Task -->|defines| Metrics + Metrics -->|includes| StatefulMetric + Metrics -->|includes| StatelessMetric + PrunaDataModule -->|provides data| EvaluationAgent + PrunaModel -->|provides predictions| EvaluationAgent + EvaluationAgent -->|evaluates| PrunaModel + EvaluationAgent -->|returns| Evaluation_Results + User -->|configures| EvaluationAgent + + subgraph Metric_Types + StatefulMetric + StatelessMetric + end + + style User fill:#bbf,stroke:#333,stroke-width:2px + style Task fill:#bbf,stroke:#333,stroke-width:2px + style EvaluationAgent fill:#bbf,stroke:#333,stroke-width:2px + style PrunaDataModule fill:#bbf,stroke:#333,stroke-width:2px + style PrunaModel fill:#bbf,stroke:#333,stroke-width:2px + style Evaluation_Results fill:#bbf,stroke:#333,stroke-width:2px + style Metrics fill:#bbf,stroke:#333,stroke-width:2px + +Let's see what that looks like in code. + +.. code-block:: python + + from pruna.evaluation.evaluation_agent import EvaluationAgent + from pruna.evaluation.task import Task + from pruna.data.pruna_datamodule import PrunaDataModule + + # Load the optimized model + optimized_model = PrunaModel.from_pretrained("CompVis/stable-diffusion-v1-4") + + # Create and configure Task + task = Task( + requests=["clip_score", "psnr"], + datamodule=PrunaDataModule.from_string('LAION256'), + device="cpu" + ) + + # Create and configure EvaluationAgent + eval_agent = EvaluationAgent(task) + + # Evaluate the model + eval_agent.evaluate(optimized_model) + +Evaluation Components +--------------------- + +The |pruna| package provides a variety of evaluation metrics to assess your models. +In this section, we’ll introduce the evaluation metrics you can use. + +Task +^^^^ + +The ``Task`` is a class that defines the task you want to evaluate your model on and it requires a set of metrics and a :ref:`PrunaDataModule ` to perform the evaluation. + +.. code-block:: python + + from pruna.evaluation.task import Task + from pruna.data.pruna_datamodule import PrunaDataModule + + task = Task( + requests=["image_generation_quality"], + datamodule=PrunaDataModule.from_string('LAION256'), + device="cpu" + ) + +Metrics +~~~~~~~ + +The ``Metrics`` is a class that defines the metrics you want to evaluate your model on. + +Metrics are the core components that calculate specific performance indicators. There are two main types of metrics: + +- **Stateful Metrics**: These metrics compute values directly from inputs without maintaining state across batches. +- **Stateless Metrics**: Metrics that maintain internal state and accumulate information across multiple batches. These are typically used for quality assessment. + +The ``Task`` accepts ``Metrics`` in three ways: + +.. tabs:: + + .. tab:: Predefined Options + + As a plain text request from predefined options (e.g., ``image_generation_quality``) + + .. code-block:: python + + from pruna.evaluation.task import Task + from pruna.data.pruna_datamodule import PrunaDataModule + + # Create the task + task = Task( + request="image_generation_quality", + datamodule=PrunaDataModule.from_string('LAION256'), + device="cpu" + ) + + .. tab:: List of Metric Names + + As a list of metric names (e.g., [``"clip_score"``, ``"psnr"``]) + + .. code-block:: python + + from pruna.evaluation.task import Task + from pruna.data.pruna_datamodule import PrunaDataModule + + # Create the task + task = Task( + metrics=["clip_score", "psnr"], + datamodule=PrunaDataModule.from_string('LAION256'), + device="cpu" + ) + + .. tab:: List of Metric Instances + + As a list of metric instances, which provides more flexibility in configuring the metrics. + + .. code-block:: python + + from pruna.evaluation.task import Task + from pruna.data.pruna_datamodule import PrunaDataModule + from pruna.evaluation.metrics.metric_psnr import PSNR + + # Initialize the metrics + metrics = [ + PSNR() + ] + + # Create the task + task = Task( + metrics=metrics, + datamodule=PrunaDataModule.from_string('LAION256'), + device="cpu" + ) + +.. note:: + + You can find the full list of available metrics in the :ref:`Metric Overview ` section. + +PrunaDataModule +~~~~~~~~~~~~~~~ + +The ``PrunaDataModule`` is a class that defines the data you want to evaluate your model on. +Data modules are a core component of the evaluation framework, providing standardized access to datasets for evaluating model performance before and after optimization. + +They offer the following functionality: + +- Standard dataloaders for training, validation, and testing +- Integration with appropriate collate functions for different data types +- Support for dataset size limitations for faster evaluation +- Compatibility with tokenizers for text-based tasks + +The ``Task`` accepts ``PrunaDataModule`` in three ways: + +.. tabs:: + + .. tab:: From String + + As a plain text request from predefined options (e.g., ``LAION256``) + + .. code-block:: python + + from pruna.data.pruna_datamodule import PrunaDataModule + + # Create the data Module + datamodule = PrunaDataModule.from_string('LAION256') + + .. tab:: From Datasets + + As a list of datasets, which provides more flexibility in configuring the data module. + + .. code-block:: python + + from pruna.data.pruna_datamodule import prunadatamodule + from transformers import AutoTokenizer + from datasets import load_dataset + + # Load a built-in dataset + tokenizer = AutoTokenizer.from_pretrained("gpt2") + + # Load custom datasets + train_ds = load_dataset("SamuelYang/bookcorpus")["train"] + train_ds, val_ds, test_ds = split_train_into_train_val_test(train_ds, seed=42) + + # Create the data module + datamodule = PrunaDataModule.from_datasets( + datasets=(train_ds, val_ds, test_ds), + collate_fn="text_generation_collate", + tokenizer=tokenizer, + collate_fn_args={"max_seq_len": 512}, + dataloader_args={"batch_size": 16, "num_workers": 4} + ) + +EvaluationAgent +^^^^^^^^^^^^^^^ + +The ``EvaluationAgent`` is a class that evaluates the performance of your model. +It is a subclass of ``pl.LightningModule`` and ``pruna.SmashConfig``. \ No newline at end of file diff --git a/docs/user_manual/evaluation.rst b/docs/user_manual/evaluation.rst deleted file mode 100644 index 7e4e2bf3..00000000 --- a/docs/user_manual/evaluation.rst +++ /dev/null @@ -1,330 +0,0 @@ -.. _evaluation: - -Evaluation Metrics -=================== - -The |pruna| package provides helpful evaluation tools to assess your models. In this section, we'll introduce the evaluation metrics you can use with the package. - -Evaluation helps you understand how compression affects your models across different dimensions - from output quality to resource requirements. This knowledge is essential for making informed decisions about which compression techniques work best for your specific needs. - -.. _quicktutorial: - -Quick Tutorial --------------- - -Before we start, here's a simple example showing how to evaluate your models using |pruna|. - -The rest of this guide provides more detailed explanations of each component and additional features available for model evaluation. - -.. code-block:: python - - import copy - - from diffusers import StableDiffusionPipeline - - from pruna import smash, SmashConfig - from pruna.data.pruna_datamodule import PrunaDataModule - from pruna.evaluation.evaluation_agent import EvaluationAgent - from pruna.evaluation.task import Task - - # Load data and set up smash config - smash_config = SmashConfig() - smash_config['cacher'] = 'deepcache' - - # Load the base model - model_path = "CompVis/stable-diffusion-v1-4" - pipe = StableDiffusionPipeline.from_pretrained(model_path) - - # Smash the model - copy_pipe = copy.deepcopy(pipe) - smashed_pipe = smash(copy_pipe, smash_config) - - # Define the task and the evaluation agent - metrics = ['clip_score', 'psnr'] - task = Task(metrics, datamodule=PrunaDataModule.from_string('LAION256')) - eval_agent = EvaluationAgent(task) - - # Evaluate base model, all models need to be wrapped in a PrunaModel before passing them to the EvaluationAgent - first_results = eval_agent.evaluate(pipe) - print(first_results) - - # Evaluate smashed model - smashed_results = eval_agent.evaluate(smashed_pipe) - print(smashed_results) - - -.. code-block:: python - - # Base model result output - {'clip_score_y_x': 28.0828} - - # Smashed model result output - {'clip_score_y_x': 28.4500, 'psnr_pairwise_y_gt': 18.7465} - -Evaluation Framework --------------------- - -The evaluation framework in |pruna| consists of several key components: - -Task -^^^^ -Processes user requests and converts them into a set of metrics. The ``Task`` accepts metrics in three ways: - -- As a plain text request from predefined options (e.g., ``image_generation_quality``) -- As a list of metric names (e.g., [``"clip_score"``, ``"psnr"``]) (see :ref:`Available Metrics ` below) -- As a list of metric instances - -In addition to metrics, ``Task`` requires a :ref:`PrunaDataModule ` to perform the evaluation. - -.. autoclass:: pruna.evaluation.task.Task - -Currently, ``Task`` supports the following plain textrequests: - -- ``image_generation_quality``: Creates metrics for evaluating image generation models (``clip_score``, ``pairwise_clip_score``, ``psnr``) - - -.. code-block:: python - - from pruna.evaluation.task import Task - from pruna.data.pruna_datamodule import PrunaDataModule - - task = Task("image_generation_quality", datamodule=PrunaDataModule.from_string('LAION256')) - -EvaluationAgent -^^^^^^^^^^^^^^^ -The main entry point for evaluating models. The ``EvaluationAgent``: - -- Takes a ``Task`` object that defines what metrics to use -- Provides methods to evaluate any model -- Handles the evaluation process, including separating metrics by execution strategy -- Runs inference on the model to generate predictions -- Caches predictions to avoid redundant computations -- Passes ground truth data and predictions to the appropriate metrics -- Collects and returns results from all metrics - -.. autoclass:: pruna.evaluation.evaluation_agent.EvaluationAgent - :members: evaluate - -.. container:: hidden_code - - .. code-block:: python - - from pruna.evaluation.task import Task - from pruna.data.pruna_datamodule import PrunaDataModule - - data_module = PrunaDataModule.from_string('LAION256') - data_module.limit_datasets(10) - - task = Task("image_generation_quality", datamodule=data_module) - -.. code-block:: python - - from pruna.evaluation.evaluation_agent import EvaluationAgent - - eval_agent = EvaluationAgent(task) - - -For the full example running evaluation please see :ref:`Quick Tutorial ` above. - -.. _metrics: - -Metrics -------- - -Metrics help quantify different aspects of model performance, from output quality to resource requirements. The |pruna| package includes metrics for both quality assessment and resource utilization. - -When using the ``EvaluationAgent``, all metrics are executed automatically as part of the evaluation pipeline. The agent handles model inference, data preparation, and passing the appropriate inputs to each metric, eliminating the need to run metrics individually. - -Metrics can operate in both single-model and pairwise modes: - -- In single-model mode, each evaluation produces independent scores for the model being evaluated. -- In pairwise mode, metrics compare a subsequent model against the first model evaluated by the agent. Usually, this is used to compare the base model (first model) with its smashed version (subsequent model). The first model's outputs are cached and used as a reference point for all following evaluations. The pairwise comparison produces a single score that quantifies the relationship (e.g., similarity or difference) between the two models. - -Our metrics fall into two implementation categories that work differently under the hood: - -Base Metrics -^^^^^^^^^^^^^ -Simple metrics that compute values directly from inputs without maintaining state across batches. Examples include: -- Model Architecture metrics -- Energy consumption metrics -- Memory usage metrics - -elapsed_time -"""""""""""" - -Measures inference time, latency, and throughput. - -:Evaluation on CPU: Yes. -:Required: - A PrunaModel object that defines the model to evaluate. - A DataLoader object that defines the dataloader to evaluate the model on. -:Parameters: - - | ``n_iterations``: Number of inference iterations to measure (default 100). - | ``n_warmup_iterations``: Number of warmup iterations before measurement (default 10). - | ``device``: Device to run inference on (default "cuda"). - | ``timing_type``: Type of timing to use ("sync" or "async", default "sync"). - -`gpu_memory `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -Measures peak GPU memory usage during model loading and execution. - -:Evaluation on CPU: No. -:Required: - Path to the PrunaModel to evaluate. - A DataLoader object that defines the dataloader to evaluate the model on. - The model class to load the model from the path. -:Parameters: - - | ``mode``: Memory measurement mode ("disk", "inference", or "training"). - | ``gpu_indices``: List of GPU indices to monitor (default all available GPUs). - -`energy `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -Measures energy consumption in kilowatt-hours (kWh) and CO2 emissions in kilograms (kg). - -:Evaluation on CPU: Yes. -:Description: Measures energy consumption in kilowatt-hours (kWh) and CO2 emissions in kilograms (kg). -:Required: - A PrunaModel object that defines the model to evaluate. - A DataLoader object that defines the dataloader to evaluate the model on. -:Parameters: - - | ``n_iterations``: Number of inference iterations to measure (default 100). - | ``n_warmup_iterations``: Number of warmup iterations before measurement (default 10). - | ``device``: Device to run inference on (default "cuda"). - -`model_architecture `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -Measures the number of parameters and MACs (multiply-accumulate operations) in the model. - -:Evaluation on CPU: Yes. -:Required: - A PrunaModel object that defines the model to evaluate. - A DataLoader object that defines the dataloader to evaluate the model on. -:Parameters: - - | ``device``: Device to evaluate the model on (default "cuda"). - -Stateful Metrics -^^^^^^^^^^^^^^^^^ -Metrics that maintain internal state and accumulate information across multiple batches. These are typically used for quality assessment. - -Most of our stateful metrics are implemented using the TorchMetricsWrapper, which adapts metrics from the `TorchMetrics `_ library to work within our evaluation framework. This allows us to leverage the robust implementations provided by TorchMetrics while maintaining a consistent API. - -`clip_score `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -Measures the similarity between images and text using the CLIP model. - - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. -:Parameters: Accepts all parameters from the TorchMetrics CLIPScore implementation. - -`pairwise_clip_score `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -Measures the similarity between images of first and subsequent models using the CLIP model. - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. -:Parameters: Accepts all parameters from the TorchMetrics CLIPScore implementation. - -`cmmd `_ -"""""""""""""""""""""""""""""""""""""""""""" - -CMMD measures the distributional discrepancy between two sets of images or text by computing Maximum Mean Discrepancy (MMD) in the CLIP embedding space. It captures both semantic and visual alignment. - -Key Benefits: - -- **Distribution-Free:** Does not rely on any assumptions about the underlying feature distribution. -- **Unbiased Estimation:** Provides a statistically unbiased measure of the discrepancy between two image sets. -- **Sample Efficiency:** Achieves reliable estimates even with smaller image samples, making it suitable for rapid evaluations. -- **Human-Aligned:** Demonstrates better agreement with human perceptual assessments of image quality compared to FID. - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. -:Parameters: - - | ``clip_model_name``: Name of the CLIP model to use (default "openai/clip-vit-large-patch14-336"). - | ``call_type``: Call type to use for the metric (default "gt_y"). For pairwise evaluation pass "pairwise" or "pairwise_gt_y". - | ``device``: Device to run the metric on (default "cuda"). - - - -`accuracy `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures the proportion of correct predictions in classification tasks. - - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. TorchMetrics requires a 'task' parameter to be set to 'binary', 'multiclass', or 'multilabel'. Each task type may have additional specific requirements - please refer to the TorchMetrics documentation for details. -:Parameters: Accepts all parameters from the TorchMetrics Accuracy implementation (task, num_classes, threshold, etc.). - -`precision `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures the proportion of positive identifications that were actually correct. - - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. TorchMetrics requires a 'task' parameter to be set to 'binary', 'multiclass', or 'multilabel'. Each task type may have additional specific requirements - please refer to the TorchMetrics documentation for details. -:Parameters: Accepts all parameters from the TorchMetrics Precision implementation (task, num_classes, threshold, etc.). - -`recall `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures the proportion of actual positives that were identified correctly. - - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. TorchMetrics requires a 'task' parameter to be set to 'binary', 'multiclass', or 'multilabel'. Each task type may have additional specific requirements - please refer to the TorchMetrics documentation for details. -:Parameters: Accepts all parameters from the TorchMetrics Recall implementation (task, num_classes, threshold, etc.). - -`perplexity `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures how well a probability model predicts a text sample. - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. -:Parameters: Accepts all parameters from the TorchMetrics Perplexity implementation. - -`fid `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures the similarity between generated and real image distributions using the Frechet Distance between Gaussian distributions fitted to the Inception embeddings of the generated and real images. - -FID compares the **distribution** of real and generated images in a high-dimensional feature space. Since it estimates **mean and covariance statistics**, smaller sample sizes can introduce high variance, making the metric less stable. Large-scale evaluations often use **tens of thousands of images**, but for practical use, smaller sample sizes may still provide a reasonable approximation. - -**Computation Considerations** - -When generating images and computing FID on **thousands to tens of thousands of samples**, the process can take **multiple hours to several days**, even on a high-end GPU like an **A100 or RTX 4090**. On mid-range GPUs like a **3060 or 4060**, it can take **significantly longer**. A rough approximation using **a few thousand images** may still take **several hours**, even with strong hardware. - -:Evaluation on CPU: No (impractical due to the high computational cost) -:Required: Inputs, ground truth and predictions. -:Parameters: Accepts all parameters from the TorchMetrics FrechetInceptionDistance implementation (feature extraction parameters, etc.). - -`psnr `_ -""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures the peak signal-to-noise ratio (PSNR) between two images. - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. -:Parameters: Accepts all parameters from the TorchMetrics PSNR implementation. - -`ssim `_ -""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures the structural similarity index (SSIM) between two images. - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. -:Parameters: Accepts all parameters from the TorchMetrics SSIM implementation. - -`lpips `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures the Learned Perceptual Image Patch Similarity (LPIPS) between two images. - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. -:Parameters: Accepts all parameters from the TorchMetrics LPIPS implementation. \ No newline at end of file diff --git a/docs/user_manual/optimize.rst b/docs/user_manual/optimize.rst new file mode 100644 index 00000000..c8b8740d --- /dev/null +++ b/docs/user_manual/optimize.rst @@ -0,0 +1,207 @@ +Optimize your first model +========================= + +This guide provides a quick introduction to optimizing AI models with |pruna|. + +You'll learn how to use Pruna's core functionality to make your models faster, smaller, cheaper, and greener. +For installation instructions, see :doc:`Installation `. + +Basic Optimization Workflow +--------------------------- + +|pruna| follows a simple workflow for optimizing models: + +.. mermaid:: + :align: center + + graph LR + A[Load Model] --> B[Define SmashConfig] + B --> C[Optimize Model] + C --> D[Evaluate Model] + D --> E[Run Inference] + style A fill:#bbf,stroke:#333,stroke-width:2px + style B fill:#bbf,stroke:#333,stroke-width:2px + style C fill:#bbf,stroke:#333,stroke-width:2px + style D fill:#bbf,stroke:#333,stroke-width:2px + style E fill:#bbf,stroke:#333,stroke-width:2px + +Let's see what that looks like in code. + +.. code-block:: python + + from pruna import smash, SmashConfig + from diffusers import StableDiffusionPipeline + from pruna.data.pruna_datamodule import PrunaDataModule + from pruna.evaluation.evaluation_agent import EvaluationAgent + from pruna.evaluation.task import Task + + # Load the model + model = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") + + # Create and configure SmashConfig + smash_config = SmashConfig() + smash_config["cacher"] = "deepcache" + + # Optimize the model + optimized_model = smash(model=model, smash_config=smash_config) + + # Evaluate the model + metrics = ['clip_score', 'psnr'] + task = Task(metrics, datamodule=PrunaDataModule.from_string('LAION256')) + eval_agent = EvaluationAgent(task) + eval_agent.evaluate(optimized_model) + + # Run inference + optimized_model("A serene landscape with mountains").images[0] + +Step-by-Step Optimization Workflow +---------------------------------- + +Step 1: Load a pretrained model +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +First, load any model using its original library, like ``transformers`` or ``diffusers``: + +.. code-block:: python + + from diffusers import StableDiffusionPipeline + + base_model = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") + + +Step 2: Define optimizations with a ``SmashConfig`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +After loading the model, we can define a ``SmashConfig`` to customize the optimizations we want to apply. +This ``SmashConfig`` is a dictionary-like object that configures which optimizations to apply to your model. +You can specify multiple optimization algorithms from different categories like batching, caching and quantization. + +For now, let's just use a ``cacher`` to accelerate the model during inference. + +.. code-block:: python + + from pruna import SmashConfig + + smash_config = SmashConfig() + smash_config["cacher"] = "deepcache" # Accelerate the model with caching + +Pruna support a wide range of algorithms for specific optimizations, all with different trade-offs. +To understand how to configure the right one for your scenario, see :doc:`Define a SmashConfig `. + +Step 3: Apply optimizations with ``smash`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``smash()`` function is the core of Pruna. It takes your model and ``SmashConfig``, applies the specified optimizations. +Let's use the ``smash()`` function to apply the configured optimizations: + +.. code-block:: python + + from pruna import smash + + optimized_model = smash(model=base_model, smash_config=smash_config) + + +The ``smash()`` function returns a ``PrunaModel`` - a wrapper that provides a standardized interface for the optimized model. So, we can still use the model as we would use the original one. + +Step 4: Evaluate the optimized model with the ``EvaluationAgent`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To evaluate the optimized model, we can use the same interface as the original model. + +.. code-block:: python + + from pruna.data.pruna_datamodule import PrunaDataModule + from pruna.evaluation.evaluation_agent import EvaluationAgent + + metrics = ['clip_score', 'psnr'] + task = Task(metrics, datamodule=PrunaDataModule.from_string('LAION256')) + eval_agent = EvaluationAgent(task) + eval_agent.evaluate(optimized_model) + +To understand how to run more complex evaluation workflows, see :doc:`Evaluate a model `. + +Step 5: Run inference with the optimized model +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To run inference with the optimized model, we can use the same interface as the original model. + +.. code-block:: python + + optimized_model("A serene landscape with mountains").images[0] + +Example use cases +----------------- + +Let's look at some specific examples for different model types. + +Example 1: Diffusion Model Optimization +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: python + + from diffusers import StableDiffusionPipeline + from pruna import smash, SmashConfig + + # Load the model + model = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") + + # Create and configure SmashConfig + smash_config = SmashConfig() + smash_config["cacher"] = "deepcache" + smash_config["compiler"] = "stable_fast" + + # Optimize the model + optimized_model = smash(model=model, smash_config=smash_config) + + # Generate an image + optimized_model("A serene landscape with mountains").images[0] + +Example 2: Large Language Model Optimization +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: python + + from transformers import AutoModelForCausalLM + from pruna import smash, SmashConfig + + # Load the model + model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m") + + # Create and configure SmashConfig + smash_config = SmashConfig() + smash_config["quantizer"] = "gptq" # Apply GPTQ quantization + + # Optimize the model + optimized_model = smash(model=model, smash_config=smash_config) + + # Use the model for generation + input_text = "The best way to learn programming is" + optimized_model(input_text) + + +Example 3: Speech Recognition Optimization +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: python + + from transformers import AutoModelForSpeechSeq2Seq + from pruna import smash, SmashConfig + import torch + + # Load the model + model_id = "openai/whisper-large-v3" + model = AutoModelForSpeechSeq2Seq.from_pretrained( + model_id, torch_dtype=torch.float16, low_cpu_mem_usage=True + ).to("cuda") + + # Create and configure SmashConfig + smash_config = SmashConfig() + smash_config.add_processor(model_id) # Required for Whisper + smash_config["compiler"] = "c_whisper" + smash_config["batcher"] = "whisper_s2t" + + # Optimize the model + optimized_model = smash(model=model, smash_config=smash_config) + + # Use the model for transcription + optimized_model("audio_file.wav") \ No newline at end of file diff --git a/docs/user_manual/save_load.rst b/docs/user_manual/save_load.rst index 4ab1b0cb..01377a60 100644 --- a/docs/user_manual/save_load.rst +++ b/docs/user_manual/save_load.rst @@ -1,10 +1,12 @@ -Saving and Loading Pruna Models -=============================== +Save and Load Models +===================== -After smashing a model using |pruna|, you can save it to disk and load it later using the built-in save and load functionality. +This guide provides a quick introduction to saving and loading optimized AI models with |pruna|. -Saving and Loading Models -------------------------- +You will learn how to save and load a ``PrunaModel`` after smashing a model using |pruna|. Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. + +Saving a ``PrunaModel`` +----------------------- To save a smashed model, use the ``PrunaModel.save_pretrained()`` or ``PrunaModel.save_to_hub()`` method. This method saves all necessary model files and as well as the smash configuration to the specified directory: @@ -46,13 +48,20 @@ To save a smashed model, use the ``PrunaModel.save_pretrained()`` or ``PrunaMode smashed_model = smash(model=base_model, smash_config=smash_config) # Save the model - smashed_model.save_to_hub("PrunaAI/smashed-stable-diffusion-v1-4") + smashed_model.save_to_hub("PrunaAI/smashed-stable-diffusion-v1-4-smashed") + + .. tip:: + + When saving models to the hub, we recommend to use a suffix like ``-smashed`` to indicate that the model has been smashed with |pruna|. The save operation will: 1. Save the model weights and architecture, including information on how to load the model later on 2. Save the ``smash_config`` (including tokenizer and processor if present, data will be detached and not reloaded) +Loading a ``PrunaModel`` +------------------------ + To load a previously saved ``PrunaModel``, use the ``PrunaModel.from_pretrained()`` or ``PrunaModel.from_hub()`` class method: .. tabs:: @@ -74,6 +83,7 @@ To load a previously saved ``PrunaModel``, use the ``PrunaModel.from_pretrained( loaded_model = PrunaModel.from_hub("PrunaAI/smashed-stable-diffusion-v1-4") The load operation will: + 1. Load the model architecture and weights and cast them to the device specified in the SmashConfig 2. Restore the smash configuration @@ -103,8 +113,6 @@ you should also load the smashed model as follows: Depending on the saving function of the algorithm combination not all keyword arguments are required for loading (e.g. some are set by the algorithm combination itself). In that case, we discard and log a warning about unused keyword arguments. - - Algorithm Reapplication ~~~~~~~~~~~~~~~~~~~~~~~~ Some algorithms, particularly compilers and certain quantization methods, need to be reapplied after loading, as, for example, a compiled model can be rarely saved in its compiled state. @@ -118,10 +126,4 @@ Set ``verbose=True`` when loading if you want to see warning messages as well as from pruna import PrunaModel - loaded_model = PrunaModel.from_pretrained("saved_model/", verbose=True) - -``PrunaModel`` Function Documentation ---------------------------------------------- - -.. autoclass:: pruna.engine.pruna_model.PrunaModel - :members: from_pretrained, from_hub, save_to_hub, save_pretrained \ No newline at end of file + loaded_model = PrunaModel.from_pretrained("saved_model/", verbose=True) \ No newline at end of file diff --git a/docs/user_manual/smash.rst b/docs/user_manual/smash.rst deleted file mode 100644 index fd6aff7d..00000000 --- a/docs/user_manual/smash.rst +++ /dev/null @@ -1,57 +0,0 @@ -smash -========================= - -The ``smash`` function is the main function in |pruna| for optimizing models. In the following sections we will show you how to use it. - -Calling the ``smash`` Function ---------------------------------------------- - -In preparation to using ``smash``, we have to load our model and define a ``SmashConfig``. As an example, we will take a simple model by loading the ``ViT-B/16`` model from ``torchvision``. - -.. code-block:: python - - import torchvision - - base_model = torchvision.models.vit_b_16(weights="ViT_B_16_Weights.DEFAULT").cuda() - -Next, we will define a :doc:`SmashConfig ` and activate the ``torch_compile`` compiler. - -.. code-block:: python - - from pruna import SmashConfig - smash_config = SmashConfig() - smash_config['compiler'] = 'torch_compile' - -We are now ready to call the ``smash`` function! - -We can pass the model and the ``SmashConfig`` to the ``smash`` function as follows: - -.. code-block:: python - - from pruna import smash - - smashed_model = smash( - model=base_model, - smash_config=smash_config, - ) - -The resulting smashed model can be used in the same way as the original one. - -We perform compatibility checks to ensure that the model is compatible with the algorithms that you have selected at the beginning of the ``smash`` process. If you wish to skip these checks, you can set the ``experimental`` flag to ``True``: - -.. code-block:: python - - smashed_model = smash( - model=base_model, - smash_config=smash_config, - experimental=True, - ) - -Please note that this can lead to undefined behavior or difficult-to-debug errors. - -Importantly, the returned model offers save and load functionality that allows you to save the model and load it in its smashed state, see :doc:`save_load`. - -``smash`` Function Documentation ---------------------------------------------- - -.. autofunction:: pruna.smash.smash \ No newline at end of file diff --git a/docs/user_manual/smash_config.rst b/docs/user_manual/smash_config.rst deleted file mode 100644 index f6509d67..00000000 --- a/docs/user_manual/smash_config.rst +++ /dev/null @@ -1,67 +0,0 @@ -SmashConfig -========================= - -``SmashConfig`` is an essential tool in |pruna| for configuring parameters to optimize your models. This manual explains how to define and use a ``SmashConfig``. - -Defining a simple ``SmashConfig`` ---------------------------------- - -Define a ``SmashConfig`` using the following snippet: - -.. code-block:: python - - from pruna import SmashConfig - smash_config = SmashConfig() - -After creating an empty ``SmashConfig``, you can set activate a algorithm by adding it to the ``SmashConfig``: - -.. code-block:: python - - smash_config['quantizer'] = 'hqq' - -Additionally, you can overwrite :doc:`the defaults of the algorithm ` you have added by setting the hyperparameters in the ``SmashConfig``: - -.. code-block:: python - - smash_config['hqq_weight_bits'] = 4 - -You're done! You created your ``SmashConfig`` and can now :doc:`pass it to the smash function. ` - - -Adding a Dataset, Tokenizer or Processor ----------------------------------------- - -Some algorithms require a dataset, tokenizer or processor to be passed to the ``SmashConfig``. -For example, the ``gptq`` quantizer requires a dataset and a tokenizer. We can pass them to the ``SmashConfig`` e.g. as follows: - -.. code-block:: python - - from pruna import SmashConfig - smash_config = SmashConfig() - smash_config.add_tokenizer("facebook/opt-125m") - smash_config.add_data("WikiText") - -As you can see in this example, we can add a dataset simply by passing the name of the dataset. However, the ``add_data()`` function also supports other input formats. For more information, see the :doc:`dataset documentation `. - -We can now activate the ``gptq`` quantizer by adding it to the ``SmashConfig``: - -.. code-block:: python - - smash_config['quantizers'] = 'gptq' - -Similarly, we can add a processor to the ``SmashConfig`` if required, like for example by the ``c_whisper`` compiler: - -.. code-block:: python - - from pruna import SmashConfig - smash_config = SmashConfig() - smash_config.add_processor("openai/whisper-large-v3") - smash_config['compiler'] = 'c_whisper' - -If you try to activate a algorithm that requires a dataset, tokenizer or processor and haven't added them to the ``SmashConfig``, you will receive an error. Make sure to add them before activating the algorithm! If you want to know which algorithms require a dataset, tokenizer or processor, you can look at :doc:`the compression algorithm overview `. - -``SmashConfig`` Documentation ---------------------------------------------- - -.. autoclass:: pruna.config.smash_config.SmashConfig - :members: add_data, add_tokenizer, add_processor, save_to_json, load_from_json, flush_configuration, load_dict From dffd0bde5174cd000479797e99b7df841f7a2723 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 10:24:23 +0200 Subject: [PATCH 06/31] docs: enhance 'How to Contribute' section with detailed guidelines and community engagement options --- docs/contributions/how_to_contribute.rst | 129 ++++++++++++++++++++--- 1 file changed, 113 insertions(+), 16 deletions(-) diff --git a/docs/contributions/how_to_contribute.rst b/docs/contributions/how_to_contribute.rst index 2da784e0..0772d68c 100644 --- a/docs/contributions/how_to_contribute.rst +++ b/docs/contributions/how_to_contribute.rst @@ -1,24 +1,121 @@ -Since Pruna offers a broad range of compression algorithms, the following table provides a high-level overview of all methods available in Pruna. For a detailed description of each algorithm, have a look at our [documentation](https://docs.pruna.ai/en/stable/). +How to Contribute πŸ’œ +==================== +Since you landed on this part of the documentation, we want to first of all say thank you! πŸ’œ +Contributions from the community are essential to improving |pruna|, we appreciate your effort in making the repository better for everyone! -| Technique | Description | Speed | Memory | Accuracy | -| --- | --- | --- | --- | --- | -| Batching | Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing overall processing time. | βœ… | ❌ | 〰️ | -| Caching | Stores intermediate results of computations to speed up subsequent operations, reducing inference time by reusing previously computed results. | βœ… | 〰️ | 〰️ | -| Compilation | Compilation optimises the model with instructions for specific hardware. | βœ… | βž– | 〰️ | -| Distillation | Trains a smaller, simpler model to mimic a larger, more complex model. | βœ… | βœ… | ❌ | -| Quantization | Reduces the precision of weights and activations, lowering memory requirements. | βœ… | βœ… | ❌ | -| Pruning | Removes less important or redundant connections and neurons, resulting in a sparser, more efficient network. | βœ… | βœ… | ❌ | -| Recovering | Restores the performance of a model after compression. | 〰️ | 〰️ | βœ… | +Please make sure to review and adhere to the `Pruna Code of Conduct `_ before contributing to Pruna. +Any violations will be handled accordingly and result in a ban from the Pruna community and associated platforms. +Contributions that do not adhere to the code of conduct will be ignored. + +There are various ways you can contribute: + +- Have a question? Discuss with us on `Discord `_ or check out the :doc:`/resources/faq` +- Have an idea for a new tutorial? Open an issue with a :ref:`feature-request` or chat with us on `Discord `_ +- Found a bug? Open an issue with a :ref:`bug-report` +- Documentation improvements? Open an issue with a :ref:`documentation-improvement` +- Want a new feature? Open an issue with a :ref:`feature-request` +- Have a new algorithm to add? Check out: :doc:`adding_algorithm` +- Have a new metric to add? Check out: :doc:`adding_metric` +- Have a new dataset to add? Check out: :doc:`adding_dataset` + + +.. _how-to-contribute: + +Setup +----- + +If you want to contribute to |pruna| with a Pull Request, you can do so by following these steps. +If it is your very first time contributing to an open source project, we recommend to start with `this guide `_ for some generally helpful tips. + +1. Clone the repository +^^^^^^^^^^^^^^^^^^^^^^^^ + +First, fork the repository by navigating to the original `pruna repository `_ on GitHub and click the **Fork** button at the top-right. +This creates a copy of the repository in your own GitHub account. +Then, clone the forked repository from your account to your local machine and change into its directory: + +.. code-block:: bash + + git clone https://github.com/your_username/pruna.git + cd pruna + +To keep your fork up to date with the original repository, add the upstream remote: + +.. code-block:: bash + + git remote add upstream https://github.com/PrunaAI/pruna.git + +Always work on a new branch rather than the main branch. You can create a new branch for your feature or fix: + +.. code-block:: bash + + git checkout -b feat/new-feature + + + +2. Installation +^^^^^^^^^^^^^^^^^^^^^^ + +You can now set up a virtual environment of your choice and install the dependencies by running the following command: + +.. code-block:: bash + + pip install -e . + pip install -e .[dev] + pip install -e .[tests] + +You can then also install the pre-commit hooks with + +.. code-block:: bash + + pre-commit install + + +3. Develop your contribution +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +You are now ready to work on your contribution. Check out a branch on your forked repository and start coding! +When committing your changes, we recommend to follow the `Conventional Commit Guidelines `_. + +.. code-block:: bash + + git checkout -b feat/new-feature + git add . + git commit -m "feat: new amazing feature setup" + git push origin feat/new-feature + +Make sure to develop your contribution in a way that is well documented, concise and easy to maintain. +We will do our best to have your contribution integrated and maintained into |pruna| but reserve the right to reject contributions that we do not feel are in the best interest of the project. + +4. Run the tests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We have a comprehensive test suite that is designed to catch potential issues before they are merged into |pruna|. +When you make a contribution, it is highly recommended to not only run the existing tests but also to add new tests that cover your contribution. + +You can run the tests by running the following command: + +.. code-block:: bash + + pytest + +If you want to run only the tests with a specific marker, e.g. fast CPU tests, you can do so by running: + +.. code-block:: bash + + pytest -m "cpu and not slow" + + +5. Create a Pull Request +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Once you have made your changes and tested them, you can create a Pull Request. +We will then review your Pull Request and get back to you as soon as possible. +If there are any questions along the way, please do not hesitate to reach out on `Discord `_. -βœ…(improves), βž–(stays the same), 〰️(could worsen), ❌(worsens) -

-

Pruna AI Logo

-
-## Pruna Sad FAQ and Troubleshooting -If you can not find an answer to your question or problem in our [documentation][documentation], in our [FAQs][docs-faq] or in an existing issue, we are happy to help you! You can either get help from the Pruna community on [Discord][discord], join our [Office Hours][docs-office-hours] or open an issue on GitHub. \ No newline at end of file From 1436a264f9923e09b7c831652238797ca4a2c667 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 10:31:32 +0200 Subject: [PATCH 07/31] docs: update contribution guidelines with correct paths for adding algorithms, metrics, and datasets --- docs/contributions/how_to_contribute.rst | 6 +++--- docs/user_manual/evaluate.rst | 3 +-- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/contributions/how_to_contribute.rst b/docs/contributions/how_to_contribute.rst index 0772d68c..4a51f8da 100644 --- a/docs/contributions/how_to_contribute.rst +++ b/docs/contributions/how_to_contribute.rst @@ -15,9 +15,9 @@ There are various ways you can contribute: - Found a bug? Open an issue with a :ref:`bug-report` - Documentation improvements? Open an issue with a :ref:`documentation-improvement` - Want a new feature? Open an issue with a :ref:`feature-request` -- Have a new algorithm to add? Check out: :doc:`adding_algorithm` -- Have a new metric to add? Check out: :doc:`adding_metric` -- Have a new dataset to add? Check out: :doc:`adding_dataset` +- Have a new algorithm to add? Check out: :doc:`/docs_pruna/user_manual/adding_algorithm` +- Have a new metric to add? Check out: :doc:`/docs_pruna/user_manual/adding_metric` +- Have a new dataset to add? Check out: :doc:`/docs_pruna/user_manual/adding_dataset` .. _how-to-contribute: diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst index d6de59b4..596a8608 100644 --- a/docs/user_manual/evaluate.rst +++ b/docs/user_manual/evaluate.rst @@ -215,5 +215,4 @@ The ``Task`` accepts ``PrunaDataModule`` in three ways: EvaluationAgent ^^^^^^^^^^^^^^^ -The ``EvaluationAgent`` is a class that evaluates the performance of your model. -It is a subclass of ``pl.LightningModule`` and ``pruna.SmashConfig``. \ No newline at end of file +The ``EvaluationAgent`` is a class that evaluates the performance of your model. \ No newline at end of file From 1d993e1a6e9e5394ceae17e47a6e9bcb9b566faa Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 16:14:51 +0200 Subject: [PATCH 08/31] docs: update user manual with additional guidance on model evaluation and saving/loading processes --- docs/user_manual/configure.rst | 1 + docs/user_manual/evaluate.rst | 142 ++++++++++++++++++++++++--------- docs/user_manual/save_load.rst | 74 ++++++++++++++++- 3 files changed, 177 insertions(+), 40 deletions(-) diff --git a/docs/user_manual/configure.rst b/docs/user_manual/configure.rst index 00602b3e..77f601fd 100644 --- a/docs/user_manual/configure.rst +++ b/docs/user_manual/configure.rst @@ -6,6 +6,7 @@ This guide provides an introduction to configuring model optimization strategies Model optimization configuration relies on the ``SmashConfig`` class. The ``SmashConfig`` class provides a flexible dictionary-like interface for configuring model optimization strategies. It manages algorithms, hyperparameters, and additional components like tokenizers, processors and datasets. +Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. Basic Configuration Workflow ---------------------------- diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst index 596a8608..feb4da70 100644 --- a/docs/user_manual/evaluate.rst +++ b/docs/user_manual/evaluate.rst @@ -5,6 +5,7 @@ This guide provides an introduction to evaluating model optimizations with |prun Evaluation helps you understand how compression affects your models across different dimensions - from output quality to resource requirements. This knowledge is essential for making informed decisions about which compression techniques work best for your specific needs. +Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. Basic Evaluation Workflow ------------------------- @@ -49,12 +50,12 @@ Let's see what that looks like in code. from pruna.data.pruna_datamodule import PrunaDataModule # Load the optimized model - optimized_model = PrunaModel.from_pretrained("CompVis/stable-diffusion-v1-4") + optimized_model = PrunaModel.from_pretrained("PrunaAI/opt-125m-smashed") # Create and configure Task task = Task( - requests=["clip_score", "psnr"], - datamodule=PrunaDataModule.from_string('LAION256'), + requests=["accuracy"], + datamodule=PrunaDataModule.from_string('WikiText'), device="cpu" ) @@ -73,24 +74,11 @@ In this section, we’ll introduce the evaluation metrics you can use. Task ^^^^ -The ``Task`` is a class that defines the task you want to evaluate your model on and it requires a set of metrics and a :ref:`PrunaDataModule ` to perform the evaluation. - -.. code-block:: python - - from pruna.evaluation.task import Task - from pruna.data.pruna_datamodule import PrunaDataModule - - task = Task( - requests=["image_generation_quality"], - datamodule=PrunaDataModule.from_string('LAION256'), - device="cpu" - ) +The ``Task`` is a class that defines the task you want to evaluate your model on and it requires a set of :doc:`Metrics ` and a :doc:`PrunaDataModule ` to perform the evaluation. Metrics ~~~~~~~ -The ``Metrics`` is a class that defines the metrics you want to evaluate your model on. - Metrics are the core components that calculate specific performance indicators. There are two main types of metrics: - **Stateful Metrics**: These metrics compute values directly from inputs without maintaining state across batches. @@ -109,7 +97,6 @@ The ``Task`` accepts ``Metrics`` in three ways: from pruna.evaluation.task import Task from pruna.data.pruna_datamodule import PrunaDataModule - # Create the task task = Task( request="image_generation_quality", datamodule=PrunaDataModule.from_string('LAION256'), @@ -125,7 +112,6 @@ The ``Task`` accepts ``Metrics`` in three ways: from pruna.evaluation.task import Task from pruna.data.pruna_datamodule import PrunaDataModule - # Create the task task = Task( metrics=["clip_score", "psnr"], datamodule=PrunaDataModule.from_string('LAION256'), @@ -134,22 +120,16 @@ The ``Task`` accepts ``Metrics`` in three ways: .. tab:: List of Metric Instances - As a list of metric instances, which provides more flexibility in configuring the metrics. + As a list of metric (e.g., ``CMMD()``), which provides more flexibility in configuring the metrics. .. code-block:: python from pruna.evaluation.task import Task from pruna.data.pruna_datamodule import PrunaDataModule - from pruna.evaluation.metrics.metric_psnr import PSNR - - # Initialize the metrics - metrics = [ - PSNR() - ] + from pruna.evaluation.metrics import CMMD, TorchMetricWrapper - # Create the task task = Task( - metrics=metrics, + metrics=[CMMD(), TorchMetricWrapper(metric_name="accuracy")], datamodule=PrunaDataModule.from_string('LAION256'), device="cpu" ) @@ -164,27 +144,32 @@ PrunaDataModule The ``PrunaDataModule`` is a class that defines the data you want to evaluate your model on. Data modules are a core component of the evaluation framework, providing standardized access to datasets for evaluating model performance before and after optimization. -They offer the following functionality: +A more detailed overview of the ``PrunaDataModule``, its datasets and their corresponding collate functions can be found in the :doc:`Data Module Overview ` section. -- Standard dataloaders for training, validation, and testing -- Integration with appropriate collate functions for different data types -- Support for dataset size limitations for faster evaluation -- Compatibility with tokenizers for text-based tasks - -The ``Task`` accepts ``PrunaDataModule`` in three ways: +The ``Task`` accepts ``PrunaDataModule`` in two different ways: .. tabs:: .. tab:: From String - As a plain text request from predefined options (e.g., ``LAION256``) + As a plain text request from predefined options (e.g., ``WikiText``) .. code-block:: python from pruna.data.pruna_datamodule import PrunaDataModule + from transformers import AutoTokenizer + + # Load the tokenizer + tokenizer = AutoTokenizer.from_pretrained("gpt2") # Create the data Module - datamodule = PrunaDataModule.from_string('LAION256') + datamodule = PrunaDataModule.from_string( + dataset_name='WikiText', + tokenizer=tokenizer, + collate_fn="text_generation_collate", + collate_fn_args={"max_seq_len": 512}, + dataloader_args={"batch_size": 16, "num_workers": 4} + ) .. tab:: From Datasets @@ -196,7 +181,7 @@ The ``Task`` accepts ``PrunaDataModule`` in three ways: from transformers import AutoTokenizer from datasets import load_dataset - # Load a built-in dataset + # Load the tokenizer tokenizer = AutoTokenizer.from_pretrained("gpt2") # Load custom datasets @@ -212,7 +197,86 @@ The ``Task`` accepts ``PrunaDataModule`` in three ways: dataloader_args={"batch_size": 16, "num_workers": 4} ) +.. tip:: + + You can find the full list of available datasets in the :doc:`Dataset Overview ` section. + +Lastly, you can limit the number of samples in the dataset by using the ``PrunaDataModule.limit_samples`` method. + +.. code-block:: python + + from pruna.data.pruna_datamodule import PrunaDataModule + + # Create the data module + datamodule = PrunaDataModule.from_string('WikiText') + + # Limit all splits to 100 samples + datamodule.limit_datasets(100) + + # Use different limits for each split + datamodule.limit_datasets([500, 100, 200]) # train, val, test + EvaluationAgent ^^^^^^^^^^^^^^^ -The ``EvaluationAgent`` is a class that evaluates the performance of your model. \ No newline at end of file +The ``EvaluationAgent`` is a class that evaluates the performance of your model. + +To evaluate a model with the ``EvaluationAgent``, you need to create a ``Task`` with ``Metrics`` and a ``PrunaDataModule``. +Then, initialize an ``EvaluationAgent`` with that task and call the ``evaluate()`` method with your model. + +We can then chose to evaluate a single model or a pair of models. + +- **Single-Model Evaluation**: each model is evaluated independently, producing metrics that only pertain to that model's performance. The metrics are computed from the model's outputs without reference to any other model. +- **Pairwise Evaluation**: metrics compare the outputs of the current model against the first model evaluated by the agent. The first model's outputs are cached by the EvaluationAgent and used as a reference for subsequent evaluations. + +Let's see how this works in code. + +.. code-block:: python + + import copy + + from diffusers import StableDiffusionPipeline + + from pruna import smash, SmashConfig + from pruna.data.pruna_datamodule import PrunaDataModule + from pruna.evaluation.evaluation_agent import EvaluationAgent + from pruna.evaluation.task import Task + + # Load data and set up smash config + smash_config = SmashConfig() + smash_config['cacher'] = 'deepcache' + + # Load the base model + model_path = "CompVis/stable-diffusion-v1-4" + pipe = StableDiffusionPipeline.from_pretrained(model_path) + + # Smash the model + copy_pipe = copy.deepcopy(pipe) + smashed_pipe = smash(copy_pipe, smash_config) + + # Define the task and the evaluation agent + metrics = ['clip_score', 'psnr'] + task = Task(metrics, datamodule=PrunaDataModule.from_string('LAION256')) + eval_agent = EvaluationAgent(task) + + # Evaluate base model, all models need to be wrapped in a PrunaModel before passing them to the EvaluationAgent + first_results = eval_agent.evaluate(pipe) + print(first_results) + + # Evaluate smashed model + smashed_results = eval_agent.evaluate(smashed_pipe) + print(smashed_results) + + +Best Practices +-------------- + +Start with a small dataset +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When first setting up evaluation, limit the dataset size with ``datamodule.limit_datasets(n)`` to make debugging faster. + +Use pairwise metrics for comparison +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When comparing an optimized model against the baseline, use pairwise metrics to get direct comparison scores. \ No newline at end of file diff --git a/docs/user_manual/save_load.rst b/docs/user_manual/save_load.rst index 01377a60..fb38bc62 100644 --- a/docs/user_manual/save_load.rst +++ b/docs/user_manual/save_load.rst @@ -3,7 +3,79 @@ Save and Load Models This guide provides a quick introduction to saving and loading optimized AI models with |pruna|. -You will learn how to save and load a ``PrunaModel`` after smashing a model using |pruna|. Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. +You will learn how to save and load a ``PrunaModel`` after smashing a model using |pruna|. +Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. + +Basic Save and Load Workflow +---------------------------- + +|pruna| follows a simple workflow for saving and loading optimized models: + +.. mermaid:: + :align: center + + flowchart TB + subgraph LoadFlow["Load Flow"] + direction LR + F["Model Files"] --> G{"Load Method"} + G --> H1["from_pretrained('saved_model/')"] + G --> H2["from_hub('PrunaAI/saved_model')"] + H1 --> I["Pruna Model"] + H2 --> I + end + + subgraph Model["Model Files"] + direction TB + E1["Model Weights"] + E2["Architecture"] + E3["Smash Config"] + E4["Tokenizer/Processor (if present)"] + end + + subgraph SaveFlow["Save Flow"] + direction LR + A["PrunaModel"] --> B{"Save Method"} + B --> C1["save_pretrained('saved_model/')"] + B --> C2["save_to_hub('PrunaAI/saved_model')"] + C1 --> D["Model Files"] + C2 --> D + end + + SaveFlow --- Model + Model --- LoadFlow + + style A fill:#bbf,stroke:#333,stroke-width:2px + style F fill:#f9f,stroke:#333,stroke-width:2px + style G fill:#bbf,stroke:#333,stroke-width:2px + style H1 fill:#bbf,stroke:#333,stroke-width:2px + style H2 fill:#bbf,stroke:#333,stroke-width:2px + style I fill:#bbf,stroke:#333,stroke-width:2px + style B fill:#bbf,stroke:#333,stroke-width:2px + style C1 fill:#bbf,stroke:#333,stroke-width:2px + style C2 fill:#bbf,stroke:#333,stroke-width:2px + style D fill:#f9f,stroke:#333,stroke-width:2px + +Let's see what that looks like in code. + +.. code-block:: python + + from pruna import smash, SmashConfig + from diffusers import StableDiffusionPipeline + + # prepare the base model + base_model = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") + + # Create and smash your model + smash_config = SmashConfig() + smash_config["cacher"] = "deepcache" + smash_config["compiler"] = "diffusers2" + smashed_model = smash(model=base_model, smash_config=smash_config) + + # Save the model + smashed_model.save_pretrained("saved_model/") # or save_to_hub + + # Load the model + loaded_model = PrunaModel.from_pretrained("saved_model/") # or from_hub Saving a ``PrunaModel`` ----------------------- From 049278ce83fc18343706a27c3f60e9d0b5c7a76b Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 16:17:32 +0200 Subject: [PATCH 09/31] docs: add spacing for improved readability in user manual sections on configuration, evaluation, and saving/loading --- docs/user_manual/configure.rst | 1 + docs/user_manual/evaluate.rst | 1 + docs/user_manual/save_load.rst | 1 + 3 files changed, 3 insertions(+) diff --git a/docs/user_manual/configure.rst b/docs/user_manual/configure.rst index 77f601fd..cec19d79 100644 --- a/docs/user_manual/configure.rst +++ b/docs/user_manual/configure.rst @@ -6,6 +6,7 @@ This guide provides an introduction to configuring model optimization strategies Model optimization configuration relies on the ``SmashConfig`` class. The ``SmashConfig`` class provides a flexible dictionary-like interface for configuring model optimization strategies. It manages algorithms, hyperparameters, and additional components like tokenizers, processors and datasets. + Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. Basic Configuration Workflow diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst index feb4da70..cc77cb9c 100644 --- a/docs/user_manual/evaluate.rst +++ b/docs/user_manual/evaluate.rst @@ -5,6 +5,7 @@ This guide provides an introduction to evaluating model optimizations with |prun Evaluation helps you understand how compression affects your models across different dimensions - from output quality to resource requirements. This knowledge is essential for making informed decisions about which compression techniques work best for your specific needs. + Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. Basic Evaluation Workflow diff --git a/docs/user_manual/save_load.rst b/docs/user_manual/save_load.rst index fb38bc62..8bfd0124 100644 --- a/docs/user_manual/save_load.rst +++ b/docs/user_manual/save_load.rst @@ -4,6 +4,7 @@ Save and Load Models This guide provides a quick introduction to saving and loading optimized AI models with |pruna|. You will learn how to save and load a ``PrunaModel`` after smashing a model using |pruna|. + Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. Basic Save and Load Workflow From 4ecdc0392941f378bd91b811dac9bb580b8bb8e4 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Tue, 6 May 2025 14:09:13 +0200 Subject: [PATCH 10/31] refactor: streamline test parameterization for code blocks - Updated test cases in `test_code_blocks.py` to dynamically generate parameters for user manuals and contributions, excluding the 'dataset' tutorial. - Improved maintainability by leveraging glob patterns to include all relevant `.rst` files. --- tests/documentation/test_code_blocks.py | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/tests/documentation/test_code_blocks.py b/tests/documentation/test_code_blocks.py index f0469bb9..1108112c 100644 --- a/tests/documentation/test_code_blocks.py +++ b/tests/documentation/test_code_blocks.py @@ -12,16 +12,14 @@ @pytest.mark.parametrize( "rst_name", [ - pytest.param("user_manual/smash_config", marks=pytest.mark.cuda), - pytest.param("user_manual/smash", marks=pytest.mark.cuda), - pytest.param("user_manual/dataset", marks=(pytest.mark.cpu, pytest.mark.high)), - pytest.param("user_manual/telemetry", marks=pytest.mark.cpu), - pytest.param("user_manual/save_load", marks=pytest.mark.cuda), - pytest.param("contributions/adding_dataset", marks=pytest.mark.cuda), - pytest.param("contributions/adding_algorithm", marks=pytest.mark.cuda), - pytest.param("contributions/adding_metric", marks=pytest.mark.cuda), - pytest.param("user_manual/evaluation", marks=pytest.mark.cuda), - ], + pytest.param(path.stem, marks=pytest.mark.cuda) + for path in TUTORIAL_PATH.glob("user_manual/*.rst") + if path.stem not in ["dataset"] + ] + + [ + pytest.param(path.stem, marks=pytest.mark.cuda) + for path in TUTORIAL_PATH.glob("contributions/*.rst") + ] ) def test_codeblocks_cuda(rst_name: str) -> None: """Test to ensure the notebook runs without errors.""" From f33a304a1a86fdf9274f6771b22caca260f09e31 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Tue, 6 May 2025 20:13:42 +0200 Subject: [PATCH 11/31] docs: enhance user manual with new sections on customizing algorithms, datasets, metrics, and model optimization - Added detailed guides for adding custom algorithms, datasets, and metrics to the user manual. - Introduced a comprehensive tutorial on optimizing models with Pruna, including step-by-step workflows and example use cases. - Updated existing documentation for clarity and consistency across sections. --- docs/contributions/how_to_contribute.rst | 10 +- ...opening_an_issue.rst => open_an_issue.rst} | 8 +- docs/user_manual/configure.rst | 28 +-- docs/user_manual/customize.rst | 18 +- ..._algorithm.rst => customize_algorithm.rst} | 0 ...ding_dataset.rst => customize_dataset.rst} | 0 ...adding_metric.rst => customize_metric.rst} | 32 ++- docs/user_manual/evaluate.rst | 185 ++++++++++++++---- docs/user_manual/save_load.rst | 8 +- docs/user_manual/{optimize.rst => smash.rst} | 19 +- 10 files changed, 209 insertions(+), 99 deletions(-) rename docs/contributions/{opening_an_issue.rst => open_an_issue.rst} (98%) rename docs/user_manual/{adding_algorithm.rst => customize_algorithm.rst} (100%) rename docs/user_manual/{adding_dataset.rst => customize_dataset.rst} (100%) rename docs/user_manual/{adding_metric.rst => customize_metric.rst} (97%) rename docs/user_manual/{optimize.rst => smash.rst} (93%) diff --git a/docs/contributions/how_to_contribute.rst b/docs/contributions/how_to_contribute.rst index 4a51f8da..e402b22d 100644 --- a/docs/contributions/how_to_contribute.rst +++ b/docs/contributions/how_to_contribute.rst @@ -1,4 +1,4 @@ -How to Contribute πŸ’œ +How to Contribute ? πŸ’œ ==================== Since you landed on this part of the documentation, we want to first of all say thank you! πŸ’œ @@ -112,10 +112,4 @@ If you want to run only the tests with a specific marker, e.g. fast CPU tests, y Once you have made your changes and tested them, you can create a Pull Request. We will then review your Pull Request and get back to you as soon as possible. -If there are any questions along the way, please do not hesitate to reach out on `Discord `_. - - - - - - +If there are any questions along the way, please do not hesitate to reach out on `Discord `_. \ No newline at end of file diff --git a/docs/contributions/opening_an_issue.rst b/docs/contributions/open_an_issue.rst similarity index 98% rename from docs/contributions/opening_an_issue.rst rename to docs/contributions/open_an_issue.rst index acda6b8f..166e75b7 100644 --- a/docs/contributions/opening_an_issue.rst +++ b/docs/contributions/open_an_issue.rst @@ -1,11 +1,10 @@ -Opening an Issue -=============================== - +Open an Issue +============= .. _documentation-improvement: Documentation Improvement ------------------------- +------------------------- All bits help! We appreciate your interest in improving |pruna|’s documentation. @@ -54,6 +53,7 @@ When opening a bug report on GitHub, you will encounter the following template t Feature Request --------------- + We appreciate your interest in improving |pruna|! Feature requests help shape the project, and we welcome ideas that align with our mission. Before submitting your feature request, consider the following points to ensure your request is clear and actionable: diff --git a/docs/user_manual/configure.rst b/docs/user_manual/configure.rst index cec19d79..a88c9fba 100644 --- a/docs/user_manual/configure.rst +++ b/docs/user_manual/configure.rst @@ -54,8 +54,10 @@ Let's see what that looks like in code. smash_config['ifw_weight_bits'] = 16 smash_config['ifw_group_size'] = 4 - # Add a tokenizer - smash_config.add_tokenizer('bert-base-uncased') + # Add a tokenizer and processor + model_id = 'openai/whisper-large-v3' + smash_config.add_tokenizer(model_id) + smash_config.add_processor(model_id) Configure Algorithms -------------------- @@ -72,22 +74,22 @@ The table underneath provides a general overview of the impact of each algorithm - Description - Speed - Memory - - Accuracy + - Quality * - ``batcher`` - Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing processing time. - βœ… - ❌ - - ～ + - - * - ``cacher`` - Stores intermediate results of computations to speed up subsequent operations. - βœ… - - ～ - - ～ + - - + - - * - ``compiler`` - Optimises the model with instructions for specific hardware. - βœ… - βž– - - ～ + - - * - ``distiller`` - Trains a smaller, simpler model to mimic a larger, more complex model. - βœ… @@ -105,11 +107,11 @@ The table underneath provides a general overview of the impact of each algorithm - ❌ * - ``recoverer`` - Restores the performance of a model after compression. - - ～ - - ～ + - - + - - - βœ… -βœ…(improves), βž–(stays the same), ～(could worsen), ❌(worsens) +βœ…(improves), βž–(stays the same), -(Approx. the same), ❌(worsens) .. tip:: @@ -278,7 +280,7 @@ Additionallly, you can create a fully custom ``PrunaDataModule`` use it in your # Add a built-in dataset using a string identifier smash_config.add_dataset('WikiText') - .. tab:: Custom Collate Function + .. tab:: Custom Dataset Use a custom collate function to use a custom dataset as ``(train, val, test)`` tuples. @@ -306,8 +308,8 @@ Additionallly, you can create a fully custom ``PrunaDataModule`` use it in your .. tab:: PrunaDataModule - You can also create a fully custom ``PrunaDataModule`` use it in your workflow. - This process is more flexible but also more complex. It allows for more control over the dataset and the data loading process. + You can also create a ``PrunaDataModule`` use it in your workflow. + This process is more flexible but also more complex and need to adhere to certain configuration limitations. The process for defining a ``PrunaDataModule`` is highlighted in the :doc:`Evaluation ` page but a basic example of adding it to the ``SmashConfig`` is shown below. .. code-block:: python diff --git a/docs/user_manual/customize.rst b/docs/user_manual/customize.rst index 336872eb..0533d390 100644 --- a/docs/user_manual/customize.rst +++ b/docs/user_manual/customize.rst @@ -5,21 +5,21 @@ Customize components .. grid:: 1 3 3 3 - .. grid-item-card:: Add an algorithm + .. grid-item-card:: Customize an algorithm :text-align: center - :link: ./adding_algorithm.rst + :link: ./customize_algorithm.rst Steps to integrate a new compression algorithm, making it available in the ``SmashConfig``. - .. grid-item-card:: Add a dataset + .. grid-item-card:: Customize a dataset :text-align: center - :link: ./adding_dataset.rst + :link: ./customize_dataset.rst Steps to integrate a new dataset, making it available in the ``SmashConfig``. - .. grid-item-card:: Add a metric + .. grid-item-card:: Customize a metric :text-align: center - :link: ./adding_metric.rst + :link: ./customize_metric.rst Steps to integrate a new metric, making it available in the ``SmashConfig``. @@ -34,6 +34,6 @@ Customize components :caption: Customize components :hidden: - adding_algorithm - adding_dataset - adding_metric \ No newline at end of file + customize_algorithm + customize_dataset + customize_metric \ No newline at end of file diff --git a/docs/user_manual/adding_algorithm.rst b/docs/user_manual/customize_algorithm.rst similarity index 100% rename from docs/user_manual/adding_algorithm.rst rename to docs/user_manual/customize_algorithm.rst diff --git a/docs/user_manual/adding_dataset.rst b/docs/user_manual/customize_dataset.rst similarity index 100% rename from docs/user_manual/adding_dataset.rst rename to docs/user_manual/customize_dataset.rst diff --git a/docs/user_manual/adding_metric.rst b/docs/user_manual/customize_metric.rst similarity index 97% rename from docs/user_manual/adding_metric.rst rename to docs/user_manual/customize_metric.rst index 34339c20..0984ae4b 100644 --- a/docs/user_manual/adding_metric.rst +++ b/docs/user_manual/customize_metric.rst @@ -12,20 +12,20 @@ Understanding Pruna's Metric System |pruna| has two main types of metrics that live under ``pruna/evaluation/metrics``: -1. **Base Metrics** - Inherit from ``BaseMetric`` and compute values directly without maintaining state. These metrics usually require isolated inference computation. Examples: ``GPUMemoryMetric``, ``ElapsedTimeMetric``. -2. **Stateful Metrics** - Inherit from ``StatefulMetric`` and maintain internal state across multiple computations. State here refers to the information that is accumulated across multiple batches. Examples: all metrics under ``TorchMetricWrapper`` like ``Accuracy``, ``CLIPScore``. +1. **Base Metrics** - Inherit from ``BaseMetric`` and compute values directly without maintaining state. These metrics usually require isolated inference computation. Examples: ``GPUMemoryMetric``, ``ElapsedTimeMetric``. +2. **Stateful Metrics** - Inherit from ``StatefulMetric`` and maintain internal state across multiple computations. State here refers to the information that is accumulated across multiple batches. Examples: all metrics under ``TorchMetricWrapper`` like ``Accuracy``, ``CLIPScore``. When adding a new metric to |pruna|, you should place your implementation in ``pruna/evaluation/metrics`` directory to ensure it's properly integrated with the rest of the system. Use snake_case for the file name (e.g., ``your_new_metric.py``). In |pruna|, we evaluate metrics by sharing inference runs across multiple metrics whenever possible. This means that |pruna| runs inference once for all compatible metrics. - + - **Stateful metrics** are preferred for most use cases, especially quality metrics, as they can share inference results across multiple metrics - **Base metrics** are primarily used when isolated inference is required (e.g., for GPU memory metrics where sharing inference would distort results) .. note:: If you are confused about which type of metric to implement, you will likely need to implement stateful metrics. Base metrics are typically only used for specialized performance measurements that require isolated inference. -We use PascalCase for the class names (e.g, ``YourNewMetric``) and NumPy style docstrings for documentation. +We use PascalCase for the class names (e.g, ``YourNewMetric``) and NumPy style docstrings for documentation. Base Metrics ~~~~~~~~~~~~ @@ -34,8 +34,6 @@ Base metrics inherit from the ``BaseMetric`` class and implement the ``compute() |pruna| ``EvaluationAgent`` (`documentation <../user_manual/evaluation.html#evaluationagent>`_) requires all ``BaseMetric`` s to implement the ``compute`` method with two specific parameters: ``model`` and ``dataloader``. Please take note that the ``EvaluationAgent`` does not handle inference for base metrics. You will need to handle inference computations yourself. - - .. code-block:: python from pruna.evaluation.metrics.metric_base import BaseMetric @@ -44,10 +42,10 @@ Base metrics inherit from the ``BaseMetric`` class and implement the ``compute() def __init__(self): super().__init__() # Initialize any parameters your metric needs - + def compute(self, model, dataloader): '''Run inference on the model and compute the metric value.''' - + outputs = run_inference(model, dataloader) result = some_calculation(outputs) return result @@ -93,11 +91,11 @@ Here's a complete example showing all required methods: self.metric_name = "your_metric_name" self.default_call_type = "y_gt" self.call_type = call_type if call_type else self.default_call_type - + # Initialize state variables self.add_state("total", torch.zeros(1)) self.add_state("count", torch.zeros(1)) - + def update(self, inputs, ground_truths, predictions): # Update the state variables based on the current batch # Pass the inputs, ground_truths and predictions and the call_type to the metric_data_processor to get the data in the correct format @@ -105,13 +103,13 @@ Here's a complete example showing all required methods: batch_result = some_calculation(*metric_data) self.total += batch_result self.count += 1 - + def compute(self): # Compute the final metric value using the accumulated state if self.count == 0: return 0 return self.total / self.count - + When to Use Each Type ~~~~~~~~~~~~~~~~~~~~~ @@ -119,12 +117,12 @@ When to Use Each Type - **Use Stateful Metrics when**: Your metric can share inference with other metrics without affecting results (most quality metrics fall into this category) - **Use Basic Metrics when**: Your metric requires isolated inference or would produce incorrect results if inference were shared (e.g., performance metrics like GPU memory usage) -By using stateful metrics whenever possible, |pruna| can efficiently evaluate multiple metrics with just a single inference pass. +By using stateful metrics whenever possible, |pruna| can efficiently evaluate multiple metrics with just a single inference pass. Registering Your Metric ----------------------- -After implementing your metric, you need to register it with Pruna's ``MetricRegistry`` system. +After implementing your metric, you need to register it with Pruna's ``MetricRegistry`` system. The simplest way to do this is with the ``@MetricRegistry.register`` decorator: @@ -140,7 +138,7 @@ The simplest way to do this is with the ``@MetricRegistry.register`` decorator: self.param1 = param1 self.param2 = param2 self.metric_name = "your_metric_name" - + Thanks to this registry system, everyone using |pruna| can now refer to your metric by name without having to create instances directly! One important thing: the registration happens when your module is imported. To ensure your metric is always available, we suggest importing it in ``pruna/evaluation/metrics/__init__.py`` file. @@ -194,7 +192,7 @@ Once you've implemented your metric, everyone can use it in Pruna's evaluation p metrics = [ 'clip_score', - 'your_new_metric_name' + 'your_new_metric_name' ] data_module = PrunaDataModule.from_string('LAION256') @@ -206,6 +204,6 @@ Once you've implemented your metric, everyone can use it in Pruna's evaluation p results = eval_agent.evaluate(model) - + diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst index cc77cb9c..befb8538 100644 --- a/docs/user_manual/evaluate.rst +++ b/docs/user_manual/evaluate.rst @@ -1,7 +1,7 @@ -Evaluate optimizations with the Evaluation Agent +Evaluate quality with the Evaluation Agent ================================================ -This guide provides an introduction to evaluating model optimizations with |pruna|. +This guide provides an introduction to evaluating models with |pruna|. Evaluation helps you understand how compression affects your models across different dimensions - from output quality to resource requirements. This knowledge is essential for making informed decisions about which compression techniques work best for your specific needs. @@ -19,19 +19,26 @@ Basic Evaluation Workflow graph LR User -->|creates| Task User -->|creates| EvaluationAgent - Task -->|uses| PrunaDataModule + Task -->|defines| PrunaDataModule Task -->|defines| Metrics + Task -->|uses| PrunaModel Metrics -->|includes| StatefulMetric Metrics -->|includes| StatelessMetric - PrunaDataModule -->|provides data| EvaluationAgent PrunaModel -->|provides predictions| EvaluationAgent EvaluationAgent -->|evaluates| PrunaModel - EvaluationAgent -->|returns| Evaluation_Results + EvaluationAgent -->|returns| C["Evaluation Results"] User -->|configures| EvaluationAgent subgraph Metric_Types - StatefulMetric - StatelessMetric + StatefulMetric + StatelessMetric + end + + subgraph Task_Definition + Task + PrunaDataModule + Metrics + Metric_Types end style User fill:#bbf,stroke:#333,stroke-width:2px @@ -39,7 +46,7 @@ Basic Evaluation Workflow style EvaluationAgent fill:#bbf,stroke:#333,stroke-width:2px style PrunaDataModule fill:#bbf,stroke:#333,stroke-width:2px style PrunaModel fill:#bbf,stroke:#333,stroke-width:2px - style Evaluation_Results fill:#bbf,stroke:#333,stroke-width:2px + style C fill:#bbf,stroke:#333,stroke-width:2px style Metrics fill:#bbf,stroke:#333,stroke-width:2px Let's see what that looks like in code. @@ -82,7 +89,7 @@ Metrics Metrics are the core components that calculate specific performance indicators. There are two main types of metrics: -- **Stateful Metrics**: These metrics compute values directly from inputs without maintaining state across batches. +- **Base Metrics**: These metrics compute values directly from inputs without maintaining state across batches. - **Stateless Metrics**: Metrics that maintain internal state and accumulate information across multiple batches. These are typically used for quality assessment. The ``Task`` accepts ``Metrics`` in three ways: @@ -121,7 +128,7 @@ The ``Task`` accepts ``Metrics`` in three ways: .. tab:: List of Metric Instances - As a list of metric (e.g., ``CMMD()``), which provides more flexibility in configuring the metrics. + As a list of metric instances (e.g., ``CMMD()``), which provides more flexibility in configuring the metrics. .. code-block:: python @@ -130,7 +137,7 @@ The ``Task`` accepts ``Metrics`` in three ways: from pruna.evaluation.metrics import CMMD, TorchMetricWrapper task = Task( - metrics=[CMMD(), TorchMetricWrapper(metric_name="accuracy")], + metrics=[CMMD(call_type="pairwise"), TorchMetricWrapper(metric_name="accuracy")], datamodule=PrunaDataModule.from_string('LAION256'), device="cpu" ) @@ -139,6 +146,69 @@ The ``Task`` accepts ``Metrics`` in three ways: You can find the full list of available metrics in the :ref:`Metric Overview ` section. +Metric Call Types +^^^^^^^^^^^^^^^^ + +|pruna| metrics can operate in both single-model and pairwise modes. + +- **Single-Model mode**: Each evaluation produces independent scores for the model being evaluated. +- **Pairwise mode**: Metrics compare a subsequent model against the first model evaluated by the agent and produce a single comparison score. + +Underneath the hood, the ``StatefulMetric`` class uses the ``call_type`` parameter to determine the order of the inputs. + +The following table shows the different call types supported by |pruna| metrics and the metrics that support each call type. + +.. list-table:: + :widths: 10 60 10 + :header-rows: 1 + + * - Call Type + - Description + - Example Metrics + + * - ``y_gt`` + - Model's output first, then ground truth + - ``fid``, ``cmmd``, ``accuracy``, ``recall``, ``precision`` + + * - ``gt_y`` + - Ground truth first, then model's output + - ``fid``, ``cmmd``, ``accuracy``, ``recall``, ``precision`` + + * - ``x_gt`` + - Input data first, then ground truth + - ``clip_score`` + + * - ``gt_x`` + - Ground truth first, then input data + - ``clip_score`` + + * - ``pairwise_y_gt`` + - Base model's output first, then subsequent model's output + - ``psnr``, ``ssim``, ``lpips``, ``cmmd`` + + * - ``pairwise_gt_y`` + - Subsequent model's output first, then base model's output + - ``psnr``, ``ssim``, ``lpips``, ``cmmd`` + +Each metric has a default ``call_type`` but you can switch the mode of the metric despite your default ``call_type``. + +.. tabs:: + + .. tab:: Single-Model mode + + .. code-block:: python + + from pruna.evaluation.metrics import CMMD + + metric = CMMD() # or ["cmmd"] + + .. tab:: Pairwise mode + + .. code-block:: python + + from pruna.evaluation.metrics import CMMD + metric = CMMD(call_type="pairwise") + PrunaDataModule ~~~~~~~~~~~~~~~ @@ -227,47 +297,84 @@ Then, initialize an ``EvaluationAgent`` with that task and call the ``evaluate() We can then chose to evaluate a single model or a pair of models. -- **Single-Model Evaluation**: each model is evaluated independently, producing metrics that only pertain to that model's performance. The metrics are computed from the model's outputs without reference to any other model. -- **Pairwise Evaluation**: metrics compare the outputs of the current model against the first model evaluated by the agent. The first model's outputs are cached by the EvaluationAgent and used as a reference for subsequent evaluations. +- **Single-Model mode**: each model is evaluated independently, producing metrics that only pertain to that model's performance. The metrics are computed from the model's outputs without reference to any other model. +- **Pairwise mode**: metrics compare the outputs of the current model against the first model evaluated by the agent. The first model's outputs are cached by the EvaluationAgent and used as a reference for subsequent evaluations. Let's see how this works in code. -.. code-block:: python +.. tabs:: - import copy + .. tab:: Single-Model Evaluation - from diffusers import StableDiffusionPipeline + .. code-block:: python - from pruna import smash, SmashConfig - from pruna.data.pruna_datamodule import PrunaDataModule - from pruna.evaluation.evaluation_agent import EvaluationAgent - from pruna.evaluation.task import Task + import copy - # Load data and set up smash config - smash_config = SmashConfig() - smash_config['cacher'] = 'deepcache' + from diffusers import StableDiffusionPipeline - # Load the base model - model_path = "CompVis/stable-diffusion-v1-4" - pipe = StableDiffusionPipeline.from_pretrained(model_path) + from pruna import smash, SmashConfig + from pruna.data.pruna_datamodule import PrunaDataModule + from pruna.evaluation.evaluation_agent import EvaluationAgent + from pruna.evaluation.task import Task + from pruna.evaluation.metrics import CMMD + # Load data and set up smash config + smash_config = SmashConfig() + smash_config['cacher'] = 'deepcache' - # Smash the model - copy_pipe = copy.deepcopy(pipe) - smashed_pipe = smash(copy_pipe, smash_config) + # Load the base model + model_path = "CompVis/stable-diffusion-v1-4" + pipe = StableDiffusionPipeline.from_pretrained(model_path) - # Define the task and the evaluation agent - metrics = ['clip_score', 'psnr'] - task = Task(metrics, datamodule=PrunaDataModule.from_string('LAION256')) - eval_agent = EvaluationAgent(task) + # Smash the model + copy_pipe = copy.deepcopy(pipe) + smashed_pipe = smash(copy_pipe, smash_config) + + # Define the task and the evaluation agent + metrics = [CMMD()] + task = Task(metrics, datamodule=PrunaDataModule.from_string('LAION256')) + eval_agent = EvaluationAgent(task) + + # Evaluate base model, all models need to be wrapped in a PrunaModel before passing them to the EvaluationAgent + first_results = eval_agent.evaluate(pipe) + print(first_results) - # Evaluate base model, all models need to be wrapped in a PrunaModel before passing them to the EvaluationAgent - first_results = eval_agent.evaluate(pipe) - print(first_results) + .. tab:: Pairwise Evaluation - # Evaluate smashed model - smashed_results = eval_agent.evaluate(smashed_pipe) - print(smashed_results) + .. code-block:: python + + import copy + + from diffusers import StableDiffusionPipeline + from pruna import smash, SmashConfig + from pruna.data.pruna_datamodule import PrunaDataModule + from pruna.evaluation.evaluation_agent import EvaluationAgent + from pruna.evaluation.task import Task + from pruna.evaluation.metrics import CMMD + # Load data and set up smash config + smash_config = SmashConfig() + smash_config['cacher'] = 'deepcache' + + # Load the base model + model_path = "CompVis/stable-diffusion-v1-4" + pipe = StableDiffusionPipeline.from_pretrained(model_path) + + # Smash the model + copy_pipe = copy.deepcopy(pipe) + smashed_pipe = smash(copy_pipe, smash_config) + + # Define the task and the evaluation agent + metrics = [CMMD(call_type="pairwise")] + task = Task(metrics, datamodule=PrunaDataModule.from_string('LAION256')) + eval_agent = EvaluationAgent(task) + + # Evaluate base model, all models need to be wrapped in a PrunaModel before passing them to the EvaluationAgent + first_results = eval_agent.evaluate(pipe) + print(first_results) + + # Evaluate smashed model + smashed_results = eval_agent.evaluate(smashed_pipe) + print(smashed_results) Best Practices -------------- diff --git a/docs/user_manual/save_load.rst b/docs/user_manual/save_load.rst index 8bfd0124..988d79d7 100644 --- a/docs/user_manual/save_load.rst +++ b/docs/user_manual/save_load.rst @@ -27,10 +27,10 @@ Basic Save and Load Workflow subgraph Model["Model Files"] direction TB - E1["Model Weights"] - E2["Architecture"] - E3["Smash Config"] - E4["Tokenizer/Processor (if present)"] + E1["Model Weights (.safetensors)"] + E2["Architecture (.json)"] + E3["Smash Config (.json)"] + E4["Tokenizer/Processor (original directory)"] end subgraph SaveFlow["Save Flow"] diff --git a/docs/user_manual/optimize.rst b/docs/user_manual/smash.rst similarity index 93% rename from docs/user_manual/optimize.rst rename to docs/user_manual/smash.rst index c8b8740d..623b06db 100644 --- a/docs/user_manual/optimize.rst +++ b/docs/user_manual/smash.rst @@ -1,5 +1,5 @@ -Optimize your first model -========================= +Smash your first model +====================== This guide provides a quick introduction to optimizing AI models with |pruna|. @@ -16,7 +16,7 @@ Basic Optimization Workflow graph LR A[Load Model] --> B[Define SmashConfig] - B --> C[Optimize Model] + B --> C[Smash Model] C --> D[Evaluate Model] D --> E[Run Inference] style A fill:#bbf,stroke:#333,stroke-width:2px @@ -42,7 +42,7 @@ Let's see what that looks like in code. smash_config = SmashConfig() smash_config["cacher"] = "deepcache" - # Optimize the model + # Smash the model optimized_model = smash(model=model, smash_config=smash_config) # Evaluate the model @@ -52,9 +52,14 @@ Let's see what that looks like in code. eval_agent.evaluate(optimized_model) # Run inference + optimized_model.set_progress_bar_config(disable=True) + optimized_model.inference_handler.model_args.update( + {"num_inference_steps": 1, "guidance_scale": 0.0} + ) optimized_model("A serene landscape with mountains").images[0] -Step-by-Step Optimization Workflow + +Step-by-Step Optimisation Workflow ---------------------------------- Step 1: Load a pretrained model @@ -127,6 +132,10 @@ To run inference with the optimized model, we can use the same interface as the .. code-block:: python + optimized_model.set_progress_bar_config(disable=True) + optimized_model.inference_handler.model_args.update( + {"num_inference_steps": 1, "guidance_scale": 0.0} + ) optimized_model("A serene landscape with mountains").images[0] Example use cases From fbf8f602b2b95c8372799daa059c0919f6af1575 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Tue, 6 May 2025 22:49:55 +0200 Subject: [PATCH 12/31] docs: update algorithm impact table in user manual for clarity and consistency MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replaced empty entries with a standardized symbol (βž–) to indicate algorithms that have an approximate same impact. - Adjusted the legend to reflect the updated symbols for improved understanding. --- docs/user_manual/configure.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/user_manual/configure.rst b/docs/user_manual/configure.rst index a88c9fba..7406d0af 100644 --- a/docs/user_manual/configure.rst +++ b/docs/user_manual/configure.rst @@ -79,17 +79,17 @@ The table underneath provides a general overview of the impact of each algorithm - Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing processing time. - βœ… - ❌ - - - + - βž– * - ``cacher`` - Stores intermediate results of computations to speed up subsequent operations. - βœ… - - - - - - + - βž– + - βž– * - ``compiler`` - Optimises the model with instructions for specific hardware. - βœ… - βž– - - - + - βž– * - ``distiller`` - Trains a smaller, simpler model to mimic a larger, more complex model. - βœ… @@ -107,11 +107,11 @@ The table underneath provides a general overview of the impact of each algorithm - ❌ * - ``recoverer`` - Restores the performance of a model after compression. - - - - - - + - βž– + - βž– - βœ… -βœ…(improves), βž–(stays the same), -(Approx. the same), ❌(worsens) +βœ…(improves), βž–(Approx. the same), ❌(worsens) .. tip:: From 2d950a7ad0681f1f6666e00f1f0f8496134a0cfe Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Thu, 8 May 2025 10:29:47 +0200 Subject: [PATCH 13/31] docs: update README for clarity and accuracy in algorithm descriptions - Changed references from "compression algorithms" to "optimization algorithms" for consistency. - Updated installation link for pip to direct users to the correct page. - Removed redundant line breaks and improved formatting for better readability. - Enhanced the algorithm overview table with clearer descriptions and standardized symbols. --- README.md | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index a3340636..b153a7fa 100644 --- a/README.md +++ b/README.md @@ -29,17 +29,12 @@ [![Huggingface](https://img.shields.io/badge/Huggingface-models-yellow?style=flat-square)][huggingface] [![Replicate](https://img.shields.io/badge/replicate-black?style=flat-square)][replicate] - - - -
Pruna AI Logo - ## Pruna Cool Introduction Pruna is a model optimization framework built for developers, enabling you to deliver faster, more efficient models with minimal overhead. It provides a comprehensive suite of compression algorithms including [caching](https://docs.pruna.ai/en/stable/compression.html#cachers), [quantization](https://docs.pruna.ai/en/stable/compression.html#quantizers), [pruning](https://docs.pruna.ai/en/stable/compression.html#pruners), [distillation](https://docs.pruna.ai/en/stable/compression.html#distillers) and [compilation](https://docs.pruna.ai/en/stable/compression.html#compilers) techniques to make your models: @@ -68,7 +63,7 @@ Before installing, ensure you have: #### Option 1: Install Pruna using pip -Pruna is available on PyPI, so you can [install it using pip](https://docs.pruna.ai/en/stable/setup/pip.html): +Pruna is available on PyPI, so you can [install it using pip](https://docs.pruna.ai/en/stable/setup/install.html): ```bash pip install pruna @@ -96,7 +91,7 @@ from diffusers import StableDiffusionPipeline base_model = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") ``` -Then, use Pruna's `smash` function to optimize your model. Pruna provides a variety of different compression and optimization algorithms, allowing you to combine different algorithms to get the best possible results. You can customize the optimization process using `SmashConfig`: +Then, use Pruna's `smash` function to optimize your model. Pruna provides a variety of different optimization algorithms, allowing you to combine different algorithms to get the best possible results. You can customize the optimization process using `SmashConfig`: ```python from pruna import smash, SmashConfig @@ -130,7 +125,6 @@ eval_agent.evaluate(smashed_model) This was the minimal example, but you are looking for the maximal example? You can check out our [documentation][documentation] for an overview of all supported [algorithms][docs-algorithms] as well as our tutorials for more use-cases and examples. - ## Pruna Heart Pruna Pro Pruna has everything you need to get started on optimizing your own models. To push the efficiency of your models even further, we offer Pruna Pro. To give you a glimpse of what is possible with Pruna Pro, let us consider three of the most widely used diffusers pipelines and see how much smaller and faster we can make them. In addition to popular open-source algorithms, we use our proprietary Auto Caching algorithm. We compare the fidelity of the compressed models. Fidelity measures the similarity between the images of the compressed models and the images of the original model. @@ -155,20 +149,19 @@ For [HunyuanVideo](https://huggingface.co/tencent/HunyuanVideo), we compare Auto ## Pruna Cool Algorithm Overview -Since Pruna offers a broad range of compression algorithms, the following table provides a high-level overview of all methods available in Pruna. For a detailed description of each algorithm, have a look at our [documentation](https://docs.pruna.ai/en/stable/). +Since Pruna offers a broad range of optimization algorithms, the following table provides a high-level overview of all methods available in Pruna. For a detailed description of each algorithm, have a look at our [documentation](https://docs.pruna.ai/en/stable/). +| Technique | Description | Speed | Memory | Quality | +|--------------|-----------------------------------------------------------------------------------------------|:-----:|:------:|:-------:| +| `batcher` | Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing processing time. | βœ… | ❌ | βž– | +| `cacher` | Stores intermediate results of computations to speed up subsequent operations. | βœ… | βž– | βž– | +| `compiler` | Optimises the model with instructions for specific hardware. | βœ… | βž– | βž– | +| `distiller` | Trains a smaller, simpler model to mimic a larger, more complex model. | βœ… | βœ… | ❌ | +| `quantizer` | Reduces the precision of weights and activations, lowering memory requirements. | βœ… | βœ… | ❌ | +| `pruner` | Removes less important or redundant connections and neurons, resulting in a sparser, more efficient network. | βœ… | βœ… | ❌ | +| `recoverer` | Restores the performance of a model after compression. | βž– | βž– | βœ… | -| Technique | Description | Speed | Memory | Accuracy | -| --- | --- | --- | --- | --- | -| Batching | Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing overall processing time. | βœ… | ❌ | 〰️ | -| Caching | Stores intermediate results of computations to speed up subsequent operations, reducing inference time by reusing previously computed results. | βœ… | 〰️ | 〰️ | -| Compilation | Compilation optimises the model with instructions for specific hardware. | βœ… | βž– | 〰️ | -| Distillation | Trains a smaller, simpler model to mimic a larger, more complex model. | βœ… | βœ… | ❌ | -| Quantization | Reduces the precision of weights and activations, lowering memory requirements. | βœ… | βœ… | ❌ | -| Pruning | Removes less important or redundant connections and neurons, resulting in a sparser, more efficient network. | βœ… | βœ… | ❌ | -| Recovering | Restores the performance of a model after compression. | 〰️ | 〰️ | βœ… | - -βœ…(improves), βž–(stays the same), 〰️(could worsen), ❌(worsens) +βœ… (improves), βž– (approx. the same), ❌ (worsens)

@@ -208,7 +201,6 @@ If you use Pruna in your research, feel free to cite the project! πŸ’œ

Pruna AI Logo

- [discord]: https://discord.gg/Tun8YgzxZ9 [reddit]: https://www.reddit.com/r/PrunaAI/ [x]: https://x.com/PrunaAI From f50a09bf0f3bba95fa7e394f0bac9cd7eaa5d7e3 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 10:05:50 +0200 Subject: [PATCH 14/31] feat: add documentation improvement issue template --- .../ISSUE_TEMPLATE/documentation_improvement.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/documentation_improvement.md diff --git a/.github/ISSUE_TEMPLATE/documentation_improvement.md b/.github/ISSUE_TEMPLATE/documentation_improvement.md new file mode 100644 index 00000000..8c5db8fb --- /dev/null +++ b/.github/ISSUE_TEMPLATE/documentation_improvement.md @@ -0,0 +1,14 @@ +--- +name: Documentation improvement +about: Create a report to help us improve +title: '[DOC] ' +labels: 'documentation' +assignees: '' + +--- + +## What part of the documentation do you want to update? + + +## What changes do you want to make? + \ No newline at end of file From 963c5660121a2c9a1b7d749d26a241a61f4f5398 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 10:08:29 +0200 Subject: [PATCH 15/31] docs: update README for clarity and remove telemetry code snippets --- README.md | 340 ++++-------------------------------------------------- 1 file changed, 20 insertions(+), 320 deletions(-) diff --git a/README.md b/README.md index 006898fe..a3340636 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Element - **Simply make AI models faster, cheaper, smaller, greener!** + **Simply make AI models faster, cheaper, smaller, greener!** Element
@@ -86,17 +86,8 @@ pip install -e . ## Pruna Cool Quick Start -Before we start: Pruna allows to collect [a minimal set of aggregated, non-personal telemetry data](https://docs.pruna.ai/en/stable/docs_pruna/user_manual/telemetry.html) to help us identify popular algorithms and improve the product. Telemetry is enabled by default because your participation helps us make Pruna better. However, if you'd prefer not to share this, you can always disable telemetry with: -```python -from pruna.telemetry import set_telemetry_metrics - -set_telemetry_metrics(False) # disable telemetry for current session -set_telemetry_metrics(False, set_as_default=True) # disable telemetry globally -``` - - -Getting started with Pruna is easy-peasy pruna-squeezy! +Getting started with Pruna is easy-peasy pruna-squeezy! First, load any pre-trained model. Here's an example using Stable Diffusion: @@ -105,7 +96,7 @@ from diffusers import StableDiffusionPipeline base_model = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") ``` -Then, use Pruna's `smash` function to optimize your model. You can customize the optimization process using `SmashConfig`: +Then, use Pruna's `smash` function to optimize your model. Pruna provides a variety of different compression and optimization algorithms, allowing you to combine different algorithms to get the best possible results. You can customize the optimization process using `SmashConfig`: ```python from pruna import smash, SmashConfig @@ -113,6 +104,7 @@ from pruna import smash, SmashConfig # Create and smash your model smash_config = SmashConfig() smash_config["cacher"] = "deepcache" +smash_config["compiler"] = "stable_fast" smashed_model = smash(model=base_model, smash_config=smash_config) ``` @@ -124,18 +116,6 @@ smashed_model("An image of a cute prune.").images[0]
-Pruna provides a variety of different compression and optimization algorithms, allowing you to combine different algorithms to get the best possible results: - -```python -from pruna import smash, SmashConfig - -# Create and smash your model -smash_config = SmashConfig() -smash_config["cacher"] = "deepcache" -smash_config["compiler"] = "stable_fast" -smashed_model = smash(model=base_model, smash_config=smash_config) -``` - You can then use our evaluation interface to measure the performance of your model: ```python @@ -143,16 +123,14 @@ from pruna.evaluation.task import Task from pruna.evaluation.evaluation_agent import EvaluationAgent from pruna.data.pruna_datamodule import PrunaDataModule -task = Task("image_generation_quality", datamodule=PrunaDataModule.from_string("LAION256")) -eval_agent = EvaluationAgent(task) +task = Task("image_generation_quality", datamodule=PrunaDataModule.from_string("LAION256")) +eval_agent = EvaluationAgent(task) eval_agent.evaluate(smashed_model) ``` - This was the minimal example, but you are looking for the maximal example? You can check out our [documentation][documentation] for an overview of all supported [algorithms][docs-algorithms] as well as our tutorials for more use-cases and examples. - ## Pruna Heart Pruna Pro Pruna has everything you need to get started on optimizing your own models. To push the efficiency of your models even further, we offer Pruna Pro. To give you a glimpse of what is possible with Pruna Pro, let us consider three of the most widely used diffusers pipelines and see how much smaller and faster we can make them. In addition to popular open-source algorithms, we use our proprietary Auto Caching algorithm. We compare the fidelity of the compressed models. Fidelity measures the similarity between the images of the compressed models and the images of the original model. @@ -177,298 +155,20 @@ For [HunyuanVideo](https://huggingface.co/tencent/HunyuanVideo), we compare Auto ## Pruna Cool Algorithm Overview -Since Pruna offers a broad range of compression algorithms, the following table provides an overview of all methods available in Pruna and those exclusive to Pruna Pro. For a detailed description of each algorithm, have a look at our [documentation](https://docs.pruna.ai/en/stable/). - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Algorithm
Pruna Pro
Type
Hardware
Model Format
CPUGPUπŸ€— Transformers CausalLMπŸ€— Diffusers PipelineπŸ€— Transformers Whispertorch Module
AWQquantizerCheckCheck
GPTQquantizerCheckCheck
HQQquantizerCheckCheckCheck
Int8quantizerCheckCheckCheck
QUANTOquantizerCheckCheckCheckCheck
Torch DynamicquantizerCheckCheckCheckCheck
HIGGSCheckquantizerCheckCheck
torchaoCheckquantizerCheckCheckCheckCheckCheckCheck
PERPCheckrecovererCheckCheckCheckCheck
c_translatecompilerCheckCheck
IPEXCheckcompilerCheckCheck
Stable FastcompilerCheckCheck
torch.compilecompilerCheckCheckCheckCheckCheckCheck
x-fastCheckcompilerCheckCheckCheckCheckCheck
DeepCache1cacherCheckCheckCheck
Adaptive CachingCheckcacherCheckCheckCheck
Auto CachingCheckcacherCheckCheckCheck
FLUX Caching2CheckcacherCheckCheckCheck
Periodic CachingCheckcacherCheckCheckCheck
HYPER3CheckdistillerCheckCheckCheck
Structured PruningprunerCheckCheckCheck
Unstructured PruningprunerCheckCheckCheckCheck
ifwbatcherCheckCheck
ws2tbatcherCheckCheck
- -1. Only available for unet-based diffusers pipelines.
-2. Only available for FLUX models.
-3. Only available for FLUX, SD-XL, SD-v1-4, SD-v1-5, SD-3.5. +Since Pruna offers a broad range of compression algorithms, the following table provides a high-level overview of all methods available in Pruna. For a detailed description of each algorithm, have a look at our [documentation](https://docs.pruna.ai/en/stable/). + + +| Technique | Description | Speed | Memory | Accuracy | +| --- | --- | --- | --- | --- | +| Batching | Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing overall processing time. | βœ… | ❌ | 〰️ | +| Caching | Stores intermediate results of computations to speed up subsequent operations, reducing inference time by reusing previously computed results. | βœ… | 〰️ | 〰️ | +| Compilation | Compilation optimises the model with instructions for specific hardware. | βœ… | βž– | 〰️ | +| Distillation | Trains a smaller, simpler model to mimic a larger, more complex model. | βœ… | βœ… | ❌ | +| Quantization | Reduces the precision of weights and activations, lowering memory requirements. | βœ… | βœ… | ❌ | +| Pruning | Removes less important or redundant connections and neurons, resulting in a sparser, more efficient network. | βœ… | βœ… | ❌ | +| Recovering | Restores the performance of a model after compression. | 〰️ | 〰️ | βœ… | + +βœ…(improves), βž–(stays the same), 〰️(could worsen), ❌(worsens)

From 3c3d30147ab9bf3a07b2eb4a6794c53f528ae3b1 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 10:10:17 +0200 Subject: [PATCH 16/31] docs: remove contributions table and update how to contribute section with algorithm overview --- docs/contributions/contributions_toc.rst | 10 -- docs/contributions/how_to_contribute.rst | 129 +++--------------- docs/contributions/opening_an_issue.rst | 23 +++- .../telemetry.rst | 0 4 files changed, 38 insertions(+), 124 deletions(-) delete mode 100644 docs/contributions/contributions_toc.rst rename docs/{user_manual => contributions}/telemetry.rst (100%) diff --git a/docs/contributions/contributions_toc.rst b/docs/contributions/contributions_toc.rst deleted file mode 100644 index 90bcb5f6..00000000 --- a/docs/contributions/contributions_toc.rst +++ /dev/null @@ -1,10 +0,0 @@ -.. toctree:: - :hidden: - :maxdepth: 1 - :caption: Contributing - - /docs_pruna/contributions/how_to_contribute - /docs_pruna/contributions/opening_an_issue - /docs_pruna/contributions/adding_algorithm - /docs_pruna/contributions/adding_metric - /docs_pruna/contributions/adding_dataset \ No newline at end of file diff --git a/docs/contributions/how_to_contribute.rst b/docs/contributions/how_to_contribute.rst index a18614c6..2da784e0 100644 --- a/docs/contributions/how_to_contribute.rst +++ b/docs/contributions/how_to_contribute.rst @@ -1,121 +1,24 @@ -How to Contribute πŸ’œ -=============================== +Since Pruna offers a broad range of compression algorithms, the following table provides a high-level overview of all methods available in Pruna. For a detailed description of each algorithm, have a look at our [documentation](https://docs.pruna.ai/en/stable/). -Since you landed on this part of the documentation, we want to first of all say thank you! πŸ’œ -Contributions from the community are essential to improving |pruna|, we appreciate your effort in making the repository better for everyone! - -Please make sure to review and adhere to the `Pruna Code of Conduct `_ before contributing to Pruna. -Any violations will be handled accordingly and result in a ban from the Pruna community and associated platforms. -Contributions that do not adhere to the code of conduct will be ignored. - -There are various ways you can contribute: - -- Have a question? Discuss with us on `Discord `_ or check out the :doc:`/resources/faq` -- Have an idea for a new tutorial? Open an issue with a :ref:`feature-request` or chat with us on `Discord `_ -- Found a bug? Open an issue with a :ref:`bug-report` -- Want a new feature? Open an issue with a :ref:`feature-request` -- Have a new algorithm to add? Check out: :doc:`adding_algorithm` -- Have a new metric to add? Check out: :doc:`adding_metric` -- Have a new dataset to add? Check out: :doc:`adding_dataset` - - -.. _how-to-contribute: - -Setup ------ - -If you want to contribute to |pruna| with a Pull Request, you can do so by following these steps. -If it is your very first time contributing to an open source project, we recommend to start with `this guide `_ for some generally helpful tips. - -1. Clone the repository -^^^^^^^^^^^^^^^^^^^^^^^^ - -First, fork the repository by navigating to the original `pruna repository `_ on GitHub and click the **Fork** button at the top-right. -This creates a copy of the repository in your own GitHub account. -Then, clone the forked repository from your account to your local machine and change into its directory: - -.. code-block:: bash - - git clone https://github.com/your_username/pruna.git - cd pruna - -To keep your fork up to date with the original repository, add the upstream remote: - -.. code-block:: bash - - git remote add upstream https://github.com/PrunaAI/pruna.git - -Always work on a new branch rather than the main branch. You can create a new branch for your feature or fix: - -.. code-block:: bash - - git checkout -b feat/new-feature - - - -2. Installation -^^^^^^^^^^^^^^^^^^^^^^ - -You can now set up a virtual environment of your choice and install the dependencies by running the following command: - -.. code-block:: bash - - pip install -e . - pip install -e .[dev] - pip install -e .[tests] - -You can then also install the pre-commit hooks with - -.. code-block:: bash - - pre-commit install - - -3. Develop your contribution -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -You are now ready to work on your contribution. Check out a branch on your forked repository and start coding! -When committing your changes, we recommend to follow the `Conventional Commit Guidelines `_. - -.. code-block:: bash - - git checkout -b feat/new-feature - git add . - git commit -m "feat: new amazing feature setup" - git push origin feat/new-feature - -Make sure to develop your contribution in a way that is well documented, concise and easy to maintain. -We will do our best to have your contribution integrated and maintained into |pruna| but reserve the right to reject contributions that we do not feel are in the best interest of the project. - -4. Run the tests -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -We have a comprehensive test suite that is designed to catch potential issues before they are merged into |pruna|. -When you make a contribution, it is highly recommended to not only run the existing tests but also to add new tests that cover your contribution. - -You can run the tests by running the following command: - -.. code-block:: bash - - pytest - -If you want to run only the tests with a specific marker, e.g. fast CPU tests, you can do so by running: - -.. code-block:: bash - - pytest -m "cpu and not slow" - - -5. Create a Pull Request -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Once you have made your changes and tested them, you can create a Pull Request. -We will then review your Pull Request and get back to you as soon as possible. -If there are any questions along the way, please do not hesitate to reach out on `Discord `_. +| Technique | Description | Speed | Memory | Accuracy | +| --- | --- | --- | --- | --- | +| Batching | Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing overall processing time. | βœ… | ❌ | 〰️ | +| Caching | Stores intermediate results of computations to speed up subsequent operations, reducing inference time by reusing previously computed results. | βœ… | 〰️ | 〰️ | +| Compilation | Compilation optimises the model with instructions for specific hardware. | βœ… | βž– | 〰️ | +| Distillation | Trains a smaller, simpler model to mimic a larger, more complex model. | βœ… | βœ… | ❌ | +| Quantization | Reduces the precision of weights and activations, lowering memory requirements. | βœ… | βœ… | ❌ | +| Pruning | Removes less important or redundant connections and neurons, resulting in a sparser, more efficient network. | βœ… | βœ… | ❌ | +| Recovering | Restores the performance of a model after compression. | 〰️ | 〰️ | βœ… | +βœ…(improves), βž–(stays the same), 〰️(could worsen), ❌(worsens) +

+

Pruna AI Logo

+
+## Pruna Sad FAQ and Troubleshooting +If you can not find an answer to your question or problem in our [documentation][documentation], in our [FAQs][docs-faq] or in an existing issue, we are happy to help you! You can either get help from the Pruna community on [Discord][discord], join our [Office Hours][docs-office-hours] or open an issue on GitHub. \ No newline at end of file diff --git a/docs/contributions/opening_an_issue.rst b/docs/contributions/opening_an_issue.rst index ae513e98..acda6b8f 100644 --- a/docs/contributions/opening_an_issue.rst +++ b/docs/contributions/opening_an_issue.rst @@ -1,6 +1,27 @@ Opening an Issue =============================== + +.. _documentation-improvement: + +Documentation Improvement +------------------------ + +All bits help! We appreciate your interest in improving |pruna|’s documentation. + +Our documentation is built with `Sphinx `_ and `Read the Docs `_. + +The current set-up relies on restructured text (rst) files for the documentation and forces us to evaluate and build the documentation on our side. +This means you cannot directly evaluate the documentation changes on your local machine, however, you can still make changes to the documentation and create a pull request based on the changes. + +When opening a pull request for a documentation improvement, you will encounter the following template to help you structure your suggestion. Make sure to fill out all sections applicable to your feature request so that we can integrate it in Pruna as fast as possible: + +.. literalinclude:: issue_templates/documentation_improvement.md + :language: markdown + :linenos: + :lines: 9- + + .. _bug-report: Bug Report @@ -33,7 +54,7 @@ When opening a bug report on GitHub, you will encounter the following template t Feature Request --------------- -We appreciate your interest in improving |pruna|! Feature requests help shape the project, and we welcome ideas that align with our mission. +We appreciate your interest in improving |pruna|! Feature requests help shape the project, and we welcome ideas that align with our mission. Before submitting your feature request, consider the following points to ensure your request is clear and actionable: diff --git a/docs/user_manual/telemetry.rst b/docs/contributions/telemetry.rst similarity index 100% rename from docs/user_manual/telemetry.rst rename to docs/contributions/telemetry.rst From 654a25035320ba8f27063cddaa2ff7d5c44d05e3 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 10:11:05 +0200 Subject: [PATCH 17/31] docs: remove outdated ASR Whisper tutorial and add new tutorial index --- docs/tutorials/asr_whisper.ipynb | 194 ------------------------------- docs/tutorials/index.rst | 65 +++++++++++ 2 files changed, 65 insertions(+), 194 deletions(-) delete mode 100644 docs/tutorials/asr_whisper.ipynb create mode 100644 docs/tutorials/index.rst diff --git a/docs/tutorials/asr_whisper.ipynb b/docs/tutorials/asr_whisper.ipynb deleted file mode 100644 index a8896a0d..00000000 --- a/docs/tutorials/asr_whisper.ipynb +++ /dev/null @@ -1,194 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 100% faster Whisper Transcription" - ] - }, - { - "cell_type": "raw", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, - "source": [ - "\n", - " \"Open\n", - "" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This tutorial demonstrates how to use the `pruna` package to optimize any custom whisper model. We will use the `openai/whisper-large-v3` model as an example." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# if you are not running the latest version of this tutorial, make sure to install the matching version of pruna\n", - "# the following command will install the latest version of pruna\n", - "!pip install pruna" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 1. Loading the ASR model\n", - "\n", - "First, load your ASR model." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import torch\n", - "from transformers import AutoModelForSpeechSeq2Seq\n", - "\n", - "device = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\n", - "torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32\n", - "\n", - "model_id = \"openai/whisper-large-v3\"\n", - "\n", - "model = AutoModelForSpeechSeq2Seq.from_pretrained(\n", - " model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True\n", - ")\n", - "model.to(device)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 2. Initializing the Smash Config\n", - "\n", - "Next, initialize the smash_config. Since the compiler requires a processor, we add it to the smash_config." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "from pruna import SmashConfig\n", - "\n", - "# Initialize the SmashConfig\n", - "smash_config = SmashConfig()\n", - "smash_config.add_processor(model_id)\n", - "smash_config['compiler'] = 'c_whisper'\n", - "# uncomment the following line to quantize the model to 8 bits\n", - "# smash_config['c_whisper_weight_bits'] = 8" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 3. Smashing the Model\n", - "\n", - "Now, you can smash the model, which will take approximately 2 minutes on a T4 GPU." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from pruna import smash\n", - "\n", - "# Smash the model\n", - "smashed_model = smash(\n", - " model=model,\n", - " smash_config=smash_config,\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 4. Preparing the Input" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from datasets import load_dataset\n", - "from transformers import AutoProcessor\n", - "\n", - "processor = AutoProcessor.from_pretrained(model_id)\n", - "\n", - "dataset = load_dataset(\"distil-whisper/librispeech_long\", \"clean\", split=\"validation\")\n", - "sample = dataset[0][\"audio\"]\n", - "input_features = processor(sample[\"array\"], sampling_rate=sample[\"sampling_rate\"], return_tensors=\"pt\").input_features\n", - "input_features = input_features.cuda().half()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 5. Running the Model\n", - "\n", - "Finally, run the model to transcribe the audio file." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "# Display the result\n", - "results = smashed_model(input_features)\n", - "processor.decode(results, skip_special_tokens=False)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Wrap Up\n", - "\n", - "Congratulations! You have successfully smashed an ASR model. You can now use the `pruna` package to optimize any custom ASR model. The only parts that you should modify are step 1, 4 and 5 to fit your use case." - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "pruna", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.11" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/docs/tutorials/index.rst b/docs/tutorials/index.rst new file mode 100644 index 00000000..235866fa --- /dev/null +++ b/docs/tutorials/index.rst @@ -0,0 +1,65 @@ +.. _pruna_tutorials: + +Tutorials Pruna +=============== + +This tutorial will guide you through the process of using |pruna| to optimize your model. Looking for |pruna_pro| tutorials? Check out the :ref:`pruna_pro_tutorials` page. + +.. grid:: 1 2 2 2 + + .. grid-item-card:: Transcribe 2 hour of audio in 2 minutes with Whisper + :text-align: center + :link: ./asr_tutorial.ipynb + + Speed up ASR using the ``c_whisper`` ``compilation`` and ``whisper_s2t`` ``batching``. + + .. grid-item-card:: Smash your Computer Vision model with a CPU only + :text-align: center + :link: ./cv_cpu.ipynb + + ``Compile`` your model with ``torch_compile`` and ``openvino`` for faster inference. + + .. grid-item-card:: Speedup and Quantize any Diffusion Model + :text-align: center + :link: ./diffusion_quantization_acceleration.ipynb + + Speed up ``diffusers`` with ``torch_compile`` ``compilation`` and ``hqq_diffusers`` ``quantization``. + + .. grid-item-card:: Evaluating with CMMD using EvaluationAgent + :text-align: center + :link: ./evaluation_agent_cmmd.ipynb + + ``Evaluate`` image generation quality with ``CMMD`` and ``EvaluationAgent``. + + .. grid-item-card:: Run your Flux model with half the memory + :text-align: center + :link: ./flux_small.ipynb + + Speed up your image generation model with ``torch_compile`` ``compilation`` and ``hqq_diffusers`` ``quantization``. + + .. grid-item-card:: Making your LLMs 4x smaller + :text-align: center + :link: ./llms.ipynb + + Speed up your LLM inference with ``gptq`` ``quantization``. + + .. grid-item-card:: x2 smaller Sana diffusers in action + :text-align: center + :link: ./sana_diffusers_int8.ipynb + + Optimize your ``diffusion`` model with ``hqq_diffusers`` ``quantization`` in 8 bits. + + .. grid-item-card:: Make Stable Diffusion 3x Faster with DeepCache + :text-align: center + :link: ./sd_deepcache.ipynb + + Optimize your ``diffusion`` model with ``deepcache`` ``caching``. + + +.. toctree:: + :hidden: + :maxdepth: 1 + :caption: Pruna + :glob: + + ./* \ No newline at end of file From 7a62772036ef8dcbd9dd7ec9b335ae7c07a634ce Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 10:17:54 +0200 Subject: [PATCH 18/31] docs: add user manual sections for customizing algorithms, datasets, metrics, and model optimization --- .../adding_algorithm.rst | 186 +++++----- .../adding_dataset.rst | 33 +- .../adding_metric.rst | 0 docs/user_manual/configure.rst | 320 +++++++++++++++++ docs/user_manual/customize.rst | 39 +++ docs/user_manual/dataset.rst | 106 ------ docs/user_manual/evaluate.rst | 219 ++++++++++++ docs/user_manual/evaluation.rst | 330 ------------------ docs/user_manual/optimize.rst | 207 +++++++++++ docs/user_manual/save_load.rst | 53 ++- docs/user_manual/smash.rst | 57 --- docs/user_manual/smash_config.rst | 67 ---- 12 files changed, 935 insertions(+), 682 deletions(-) rename docs/{contributions => user_manual}/adding_algorithm.rst (92%) rename docs/{contributions => user_manual}/adding_dataset.rst (89%) rename docs/{contributions => user_manual}/adding_metric.rst (100%) create mode 100644 docs/user_manual/configure.rst create mode 100644 docs/user_manual/customize.rst delete mode 100644 docs/user_manual/dataset.rst create mode 100644 docs/user_manual/evaluate.rst delete mode 100644 docs/user_manual/evaluation.rst create mode 100644 docs/user_manual/optimize.rst delete mode 100644 docs/user_manual/smash.rst delete mode 100644 docs/user_manual/smash_config.rst diff --git a/docs/contributions/adding_algorithm.rst b/docs/user_manual/adding_algorithm.rst similarity index 92% rename from docs/contributions/adding_algorithm.rst rename to docs/user_manual/adding_algorithm.rst index 9f9da18e..5c90dbbd 100644 --- a/docs/contributions/adding_algorithm.rst +++ b/docs/user_manual/adding_algorithm.rst @@ -1,29 +1,28 @@ -Adding an Algorithm +Customize Algorithms ==================== -Adding the Algorithm to ``pruna.algorithms`` --------------------------------------------- - -If you’ve developed a new method or want to integrate a missing algorithm into |pruna|, we welcome your contribution! This tutorial guides you through the steps to integrate a new compression algorithm, making it available for all users. +If you’ve developed a new method or want to integrate a missing algorithm into |pruna|, we welcome your contribution! This tutorial guides you through the steps to integrate a new compression algorithm, making it available for all users. If anything is unclear or you want to discuss your contribution before opening a PR, please reach out on `Discord `_ anytime! If this is your first time contributing to |pruna|, please refer to the :ref:`how-to-contribute` guide for more information. +Add a Custom Algorithm +---------------------- + We’ll use **Superfast**, an example compiler, to demonstrate the process. -0. Identifying the Algorithm Group -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Step 1. Identify the Algorithm Group +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The first step is to identify the algorithm group. This is important because it determines the folder in which the algorithm should be placed. You can find the list of all algorithm groups in the :doc:`Compression Algorithms <../../compression>` section and determine which group fits your algorithm best by reviewing the algorithm group descriptions. -1. Creating the Compiler Class -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Step 2. Create the Compiler Class +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ First, navigate to ``pruna/algorithms/compilation/`` and create ``superfast.py``. - -2. Defining Compiler Attributes -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Step 3. Define Compiler Attributes +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Define the new compiler by inheriting from ``PrunaCompiler`` and define key attributes for the compiler. These attributes are used to provide information about the algorithm to the user, other functions in the package and even the documentation. @@ -40,10 +39,10 @@ These attributes are used to provide information about the algorithm to the user class SuperfastCompiler(PrunaCompiler): """ Implement Superfast Compiler using the superfast package. - + This compiler compiles anything with zero compilation time and 100x speedup. """ - + algorithm_name = "superfast" references = {"GitHub": "/url/to/GitHub"} tokenizer_required = False @@ -54,8 +53,9 @@ These attributes are used to provide information about the algorithm to the user compatible_algorithms = dict(quantizer=["quanto"]) -Explanation -^^^^^^^^^^^^ +Step 4. Add Algorithm Attributes +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + - docstring: The docstring should be concise and describe the algorithm in a way that is easy to understand. The description paragraph of the algorithm will be used to automatically generate the algorithm's documentation. - ``algorithm_name``: Identifier used to activate the algorithm, name should be in snake case. - ``references``: A dictionary of any references that can be provided for the algorithm, typically a link to the GitHub repository or a paper. @@ -65,8 +65,8 @@ Explanation - Additionally, you might have to specify a saving function. We provide more details on this in the section below. -Defining Hyperparameters -^^^^^^^^^^^^^^^^^^^^^^^^ +Step 5. Define Hyperparameters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Define hyperparameters using `ConfigSpace `_, allowing users to configure the backend and mode. Everything that configures the algorithm or specifies the algorithm's behavior should be a hyperparameter. @@ -81,13 +81,13 @@ Everything that configures the algorithm or specifies the algorithm's behavior s CategoricalHyperparameter("mode", choices=["mode1", "mode2"], default_value="mode1", meta=dict(desc="The mode to use for the Superfast compiler.")), ] -Users can now configure hyperparameters via ``smash_config["superfast_backend"] = "backend2"``. +Users can now configure hyperparameters via ``smash_config["superfast_backend"] = "backend2"``. Make sure to include descriptions of the hyperparameters with the ``desc`` key in the ``meta`` dictionary. This will be used later to document the hyperparameters in the algorithm's documentation. -Checking Model Compatibility -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Step 6. Check Model Compatibility +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Ensure the compiler only runs on supported models. In our example, the Superfast compiler is compatible with any model that is a subclass of ``torch.nn.Module``: @@ -101,8 +101,8 @@ Ensure the compiler only runs on supported models. In our example, the Superfast Users can bypass this check using ``experimental=True`` when calling ``smash``, but results may be unpredictable. -Handling External Dependencies -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Step 7. Handle External Dependencies +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If the compiler requires external packages, isolate their imports: @@ -116,8 +116,9 @@ If the compiler requires external packages, isolate their imports: Make sure that the dependencies are listed in ``pyproject.toml`` if they are not already included. -Implementing the Compilation Process -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Step 8. Implement the Compilation Process +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + The ``_apply()`` function integrates superfast with Pruna: @@ -132,63 +133,9 @@ The ``_apply()`` function integrates superfast with Pruna: Note that the ``smash_config`` prefix wrapper automatically prefixes hyperparameters with the algorithm name (``superfast_``). If a user sets ``smash_config["superfast_backend"]``, it will be mapped correctly to ``"backend"`` in ``get_hyperparameters()``. -Full Implementation -^^^^^^^^^^^^^^^^^^^^ - -Here’s the complete ``superfast.py`` implementation: - -.. code-block:: python - - from typing import Any, Dict - import torch - from ConfigSpace import CategoricalHyperparameter - from pruna.algorithms.compilation import PrunaCompiler - from pruna.config.smash_config import SmashConfigPrefixWrapper - - class SuperfastCompiler(PrunaCompiler): - """ - Implement Superfast Compiler using the superfast package. - - This compiler compiles anything with zero compilation time and 100x speedup. - """ - - algorithm_name = "superfast" - references = {"GitHub": "/url/to/GitHub"} - tokenizer_required = False - processor_required = False - dataset_required = False - run_on_cpu = True - run_on_cuda = True - compatible_algorithms = dict(quantizer=["quanto"]) - - def get_hyperparameters(self) -> list: - return [ - CategoricalHyperparameter("backend", choices=["backend1", "backend2"], default_value="backend1"), - CategoricalHyperparameter("mode", choices=["mode1", "mode2"], default_value="mode1"), - ] - - def model_check_fn(self, model: Any) -> bool: - return isinstance(model, torch.nn.Module) - - def import_algorithm_packages(self) -> Dict[str, Any]: - from superfast import compile_func - return dict(compile_func=compile_func) - - def _apply(self, model: Any, smash_config: SmashConfigPrefixWrapper) -> Any: - compile_func = self.import_algorithm_packages()["compile_func"] - return compile_func(model, smash_config["backend"], smash_config["mode"]) - -.. container:: hidden_code - - .. code-block:: python - - # test instantiation of compiler - SuperfastCompiler() - - +Step 9. Determine the Saving Function +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Determining a Suitable Saving Function ----------------------------------------- Saving e.g. a compiled or quantized model can be tricky and requires careful consideration. To determine the correct saving function for your algorithm, consider the decision tree below. .. mermaid:: @@ -206,21 +153,19 @@ Saving e.g. a compiled or quantized model can be tricky and requires careful con G -->|Yes| L["SAVE_FUNCTIONS.pickled"] G -->|No| M["Introduce new saving function."] -The first decision is whether the original saving function can be retained. +The first decision is whether the original saving function can be retained. For example, GPTQ-quantized transformers models still support ``.from_pretrained`` and ``.save_pretrained``, making retention possible. -If the original function cannot be retained, we consider how long the algorithm takes to apply. -If it is quick (e.g., a caching helper), we can reapply it after loading. -The key distinction is whether the modifications persist when saving. For instance, β€œstep caching cacher” attaches a helper that is discarded by ``diffusers`` upon saving, so the model can be saved and reloaded normally before reapplying the function. +If the original function cannot be retained, we consider how long the algorithm takes to apply. +If it is quick (e.g., a caching helper), we can reapply it after loading. +The key distinction is whether the modifications persist when saving. For instance, β€œstep caching cacher” attaches a helper that is discarded by ``diffusers`` upon saving, so the model can be saved and reloaded normally before reapplying the function. In contrast, compilation is irreversibleβ€”once compiled, a model cannot be saved in its compiled form, so we must save it beforehand and reapply compilation after loading. -If neither approach works, we must introduce a new saving function or use ``SAVE_FUNCTIONS.pickled``. We implement a new saving function following the existing saving-function pattern as well as introducing a matching loading function. +If neither approach works, we must introduce a new saving function or use ``SAVE_FUNCTIONS.pickled``. We implement a new saving function following the existing saving-function pattern as well as introducing a matching loading function. Otherwise, we can resort to saving the model in pickled format, but be aware that pickled models pose security risks and are generally not trusted by the community. - - -Testing the Algorithm ----------------------- +Step 10. Test the Algorithm +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To integrate the algorithm into the test suite, we navigate to ``tests/algorithms/testers/compilation.py`` and add the following Tester Class: @@ -239,14 +184,14 @@ To integrate the algorithm into the test suite, we navigate to ``tests/algorithm dummy_algorithm_tester = types.ModuleType("pruna.algorithms.testers.compilation") dummy_algorithm_tester.AlgorithmTesterBase = ABC sys.modules["base_tester"] = dummy_algorithm_tester - + .. code-block:: python from base_tester import AlgorithmTesterBase from pruna.algorithms.compilation.superfast import SuperfastCompiler from pruna import PrunaModel - + class TestSuperfast(AlgorithmTesterBase): """Tester class for the Superfast algorithm.""" @@ -271,9 +216,62 @@ This Tester class automatically parametrizes an integration test at ``tests/algo Additionally, a test is created to check that ``model_check_fn`` rejects a non-compatible model. Before opening a PR, make sure to run the test suite locally to ensure the algorithm is working as expected. + +Full Implementation +------------------- + +Here’s the complete ``superfast.py`` implementation: + +.. code-block:: python + + from typing import Any, Dict + import torch + from ConfigSpace import CategoricalHyperparameter + from pruna.algorithms.compilation import PrunaCompiler + from pruna.config.smash_config import SmashConfigPrefixWrapper + + class SuperfastCompiler(PrunaCompiler): + """ + Implement Superfast Compiler using the superfast package. + + This compiler compiles anything with zero compilation time and 100x speedup. + """ + + algorithm_name = "superfast" + references = {"GitHub": "/url/to/GitHub"} + tokenizer_required = False + processor_required = False + dataset_required = False + run_on_cpu = True + run_on_cuda = True + compatible_algorithms = dict(quantizer=["quanto"]) + + def get_hyperparameters(self) -> list: + return [ + CategoricalHyperparameter("backend", choices=["backend1", "backend2"], default_value="backend1"), + CategoricalHyperparameter("mode", choices=["mode1", "mode2"], default_value="mode1"), + ] + + def model_check_fn(self, model: Any) -> bool: + return isinstance(model, torch.nn.Module) + + def import_algorithm_packages(self) -> Dict[str, Any]: + from superfast import compile_func + return dict(compile_func=compile_func) + + def _apply(self, model: Any, smash_config: SmashConfigPrefixWrapper) -> Any: + compile_func = self.import_algorithm_packages()["compile_func"] + return compile_func(model, smash_config["backend"], smash_config["mode"]) + +.. container:: hidden_code + + .. code-block:: python + + # test instantiation of compiler + SuperfastCompiler() + Conclusion ---------- You’ve successfully integrated a new compiler into Pruna! πŸš€ -Now, users can utilize Superfast for model compilation, configure its hyperparameters, and ensure compatibility. - +Now, users can utilize Superfast for model compilation, configure its hyperparameters, and ensure compatibility. \ No newline at end of file diff --git a/docs/contributions/adding_dataset.rst b/docs/user_manual/adding_dataset.rst similarity index 89% rename from docs/contributions/adding_dataset.rst rename to docs/user_manual/adding_dataset.rst index b3d82345..e6fb7e78 100644 --- a/docs/contributions/adding_dataset.rst +++ b/docs/user_manual/adding_dataset.rst @@ -1,18 +1,21 @@ -Adding a Dataset -=============================== +Customize Datasets +================== -Our interface makes it easy to add :doc:`your own dataset <../user_manual/dataset>`. +Our interface makes it easy to add :doc:`your own dataset <../user_manual/dataset>`. Additionally, we provide a variety of :doc:`preconfigured datasets <../user_manual/dataset>` that can be readily used in SmashConfig for calibration or evaluation. -If you’d like to contribute a new dataset to our supported list, follow these two quick steps. +If you’d like to contribute a new dataset to our supported list, follow these two quick steps. If anything is unclear or you want to discuss your contribution before opening a PR, please reach out on `Discord `_ anytime! If this is your first time contributing to |pruna|, please refer to the :ref:`how-to-contribute` guide for more information. -1. Define the Dataset Setup -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Add a Custom Dataset +-------------------- -First, create a setup method to prepare the training, validation, and test splits. -This usually involves downloading or generating the dataset. +Step 1. Define the Dataset Setup +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +First, create a setup method to prepare the training, validation, and test splits. +This usually involves downloading or generating the dataset. For a text generation dataset, add the setup method in ``pruna/data/datasets/text_generation.py``: .. code-block:: python @@ -59,7 +62,7 @@ with the matching collate function and any defaults (e.g. the default image size base_datasets["NewDataset"] = (setup_new_dataset, "text_generation_collate", {}) -Ensure the dataset follows the expected format specified in the :doc:`collate function <../user_manual/dataset>`. +Ensure the dataset follows the expected format specified in the :doc:`collate function <../user_manual/dataset>`. The collate function aggregates several samples into a batch and converts them to the expected format. Now, users can add the dataset like this: @@ -74,16 +77,16 @@ Now, users can add the dataset like this: .. container:: hidden_code - + .. code-block:: python - + # test if dataloader works as expected for batch in smash_config.test_dataloader(): break -2. Add a Test +Step 2. Add a Test ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To verify that the dataset loads correctly, add it to ``tests/data/test_datamodule.py`` by parameterizing ``test_dm_from_string`` @@ -94,11 +97,11 @@ To verify that the dataset loads correctly, add it to ``tests/data/test_datamodu pytest.param("NewDataset", dict(img_size=512), marks=pytest.mark.slow) -Include necessary arguments for the collate function and mark the test as slow if needed. +Include necessary arguments for the collate function and mark the test as slow if needed. We categorize a test as slow if it requires several minutes to download and prepare the dataset. This ensures it runs appropriately in CI, either on GitHub Actions or nightly tests. Conclusion -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +---------- -That’s it! Your dataset is now available for everyone to use in Pruna. πŸ’œ +That’s it! Your dataset is now available for everyone to use in |pruna|. πŸ’œ \ No newline at end of file diff --git a/docs/contributions/adding_metric.rst b/docs/user_manual/adding_metric.rst similarity index 100% rename from docs/contributions/adding_metric.rst rename to docs/user_manual/adding_metric.rst diff --git a/docs/user_manual/configure.rst b/docs/user_manual/configure.rst new file mode 100644 index 00000000..00602b3e --- /dev/null +++ b/docs/user_manual/configure.rst @@ -0,0 +1,320 @@ +Define a SmashConfig +==================== + +This guide provides an introduction to configuring model optimization strategies with |pruna|. + +Model optimization configuration relies on the ``SmashConfig`` class. +The ``SmashConfig`` class provides a flexible dictionary-like interface for configuring model optimization strategies. +It manages algorithms, hyperparameters, and additional components like tokenizers, processors and datasets. + +Basic Configuration Workflow +---------------------------- + +|pruna| follows a simple workflow for configuring model optimization strategies: + +.. mermaid:: + :align: center + + graph LR + User -->|creates| SmashConfig + User -->|loads| PreTrainedModel["Pre-trained Model"] + + subgraph "Configuration Components" + SmashConfig --- Algorithm["Algorithm Selection"] + SmashConfig --- Hyperparameters + SmashConfig --- Tokenizer["Tokenizer (optional)"] + SmashConfig --- Processor["Processor (optional)"] + SmashConfig --- Dataset["Dataset (optional)"] + end + + SmashConfig -->|configures| SmashFn["smash() function"] + PreTrainedModel -->|input to| SmashFn + SmashFn -->|returns| OptimizedModel["Optimized PrunaModel"] + + style User fill:#bbf,stroke:#333,stroke-width:2px + style PreTrainedModel fill:#bbf,stroke:#333,stroke-width:2px + style SmashConfig fill:#bbf,stroke:#333,stroke-width:2px + style SmashFn fill:#bbf,stroke:#333,stroke-width:2px + style OptimizedModel fill:#bbf,stroke:#333,stroke-width:2px + +Let's see what that looks like in code. + +.. code-block:: python + + from pruna import SmashConfig + + smash_config = SmashConfig() + + # Activate IFW batching + smash_config['batcher'] = 'ifw' + + # Set IFW batching parameters + smash_config['ifw_weight_bits'] = 16 + smash_config['ifw_group_size'] = 4 + + # Add a tokenizer + smash_config.add_tokenizer('bert-base-uncased') + +Configure Algorithms +-------------------- + +|pruna| implements a extensible architecture for optimization algorithms. +Each algorithm has its own impact on the model in terms of speed, memory and accuracy. +The table underneath provides a general overview of the impact of each algorithm group. + +.. list-table:: + :widths: 10 60 10 10 10 + :header-rows: 1 + + * - Technique + - Description + - Speed + - Memory + - Accuracy + * - ``batcher`` + - Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing processing time. + - βœ… + - ❌ + - ～ + * - ``cacher`` + - Stores intermediate results of computations to speed up subsequent operations. + - βœ… + - ～ + - ～ + * - ``compiler`` + - Optimises the model with instructions for specific hardware. + - βœ… + - βž– + - ～ + * - ``distiller`` + - Trains a smaller, simpler model to mimic a larger, more complex model. + - βœ… + - βœ… + - ❌ + * - ``quantizer`` + - Reduces the precision of weights and activations, lowering memory requirements. + - βœ… + - βœ… + - ❌ + * - ``pruner`` + - Removes less important or redundant connections and neurons, resulting in a sparser, more efficient network. + - βœ… + - βœ… + - ❌ + * - ``recoverer`` + - Restores the performance of a model after compression. + - ～ + - ～ + - βœ… + +βœ…(improves), βž–(stays the same), ～(could worsen), ❌(worsens) + +.. tip:: + + The :doc:`Algorithm Overview ` page provides a more detailed overview of each algorithm within the different groups. + As well as additional information on the hardware requirements, compatibility with other algorithms and required components for each algorithm. + +Configure Algorithm Groups +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To activate an algorithm, you assign its name to the corresponding algorithm group in the ``SmashConfig``. +The group names are outlined in the table above and the specific algorithms are shown in the :doc:`Algorithm Overview ` page. + +Let's activate the ``ifw`` algorithm as a ``batcher``: + +.. code-block:: python + + from pruna import SmashConfig + + smash_config = SmashConfig() + + # Activate IFW batching + smash_config['batcher'] = 'ifw' + +Configure Algorithm Hyperparameters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Each algorithm has its own set of hyperparameters that control its behavior. +These are automatically prefixed with the algorithm name and can also be found underneath each algorithm in the :doc:`Algorithm Overview `. + +Let's add the ``ifw_weight_bits`` and ``ifw_group_size`` hyperparameters for the ``ifw`` we defined above: + +.. code-block:: python + + from pruna import SmashConfig + + smash_config = SmashConfig() + + # Activate IFW batching + smash_config['batcher'] = 'ifw' + + # Set IFW batching parameters + smash_config['ifw_weight_bits'] = 16 + smash_config['ifw_group_size'] = 4 + +Configure Components +-------------------- + +Some algorithms require a tokenizer, processor or dataset to be passed to the SmashConfig. +For example, looking at the :doc:`Algorithm Overview ` we see that the ``gptq`` quantizer requires a dataset and a tokenizer. + +.. list-table:: + :widths: 10 90 10 + :header-rows: 1 + + * - Component + - Description + - Function + * - ``tokenizer`` + - Tokenizes the input text. + - ``add_tokenizer()`` + * - ``processor`` + - Processes the input data. + - ``add_processor()`` + * - ``data`` + - Loads a dataset. + - ``add_dataset()`` + +.. note:: + + If you try to activate a algorithm that requires a dataset, tokenizer or processor and haven’t added them to the ``SmashConfig``, you will receive an error. + Make sure to add them before activating the algorithm! If you want to know which algorithms require a dataset, tokenizer or processor, you can look at the :doc:`Algorithm Overview `. + +Configure Tokenizers, Processors +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +|pruna| provides a directly inherits from the ``transformers`` library. +This means, we can use the same tokenizers and processors as the ones used in the ``transformers`` library. + +.. tabs:: + + .. tab:: String Identifier + + Use a string identifier to use a tokenizer or processor from the Hugging Face Hub. + + .. code-block:: python + + from pruna import SmashConfig + + smash_config = SmashConfig() + + # Add a built-in dataset using a string identifier + smash_config.add_tokenizer('facebook/opt-125m') + smash_config.add_processor('openai/whisper-large-v3') + + .. tab:: Loading Directly + + Load a tokenizer or processor directly from the Hugging Face Hub with your own configuration. + + .. code-block:: python + + from pruna import SmashConfig + from transformers import AutoTokenizer + + smash_config = SmashConfig() + + # Load a tokenizer directly from the Hugging Face Hub + tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m") + smash_config.add_tokenizer(tokenizer) + + # Load a processor directly from the Hugging Face Hub + processor = AutoProcessor.from_pretrained("openai/whisper-large-v3") + smash_config.add_processor(processor) + +Now we've set up the tokenizer and processor, we can use them to process our data. + +Configure Datasets +^^^^^^^^^^^^^^^^^^ + +|pruna| provides a variety of pre-configured datasets for different tasks. +We can use string identifiers to use a built-in dataset or use collate functions to use a custom dataset. +Underneath you can find the list of all the available datasets. + +.. list-table:: + :header-rows: 1 + + * - Task + - Built-in Dataset ID + - Custom Collate Function + - Collate Function Arguments + * - Text Generation + - `WikiText `_, `SmolTalk `_, `SmolSmolTalk `_, `PubChem `_, `OpenAssistant `_, `C4 `_ + - ``text_generation_collate`` + - ``text: str`` + * - Image Generation + - `LAION256 `_, `OpenImage `_, `COCO `_ + - ``image_generation_collate`` + - ``image: PIL.Image.Image``, ``text: str`` + * - Image Classification + - `ImageNet `_, `MNIST `_, `CIFAR10 `_ + - ``image_classification_collate`` + - ``image: PIL.Image.Image``, ``label: int`` + * - Audio Processing + - `CommonVoice `_, `AIPodcast `_ + - ``audio_processing_collate`` + - ``audio: Optional[torch.Tensor]``, ``path: Optional[str]``, ``sentence: str`` + * - Question Answering + - `Polyglot `_ + - ``question_answering_collate`` + - ``question: str``, ``answer: str`` + +Similar to the tokenizers and processors, we can use string identifiers to use a built-in dataset or use a more custom approach, i.e. using a collate function. +Additionallly, you can create a fully custom ``PrunaDataModule`` use it in your workflow. + +.. tabs:: + + .. tab:: String Identifier + + Use a string identifier to use a built-in dataset as defined in the table above. + + .. code-block:: python + + from pruna import SmashConfig + + smash_config = SmashConfig() + + # Add a built-in dataset using a string identifier + smash_config.add_dataset('WikiText') + + .. tab:: Custom Collate Function + + Use a custom collate function to use a custom dataset as ``(train, val, test)`` tuples. + + In this case, you need to specify the ``collate_fn`` to use for the dataset. + The ``collate_fn`` is a function that takes a list of individual data samples and returns a batch of data in a unified format. + Your dataset will have to adhere to the formats expected by the ``collate_fn`` as defined in the table above. + + .. code-block:: python + + from pruna import SmashConfig + from pruna.data.utils import split_train_into_train_val_test + from datasets import load_dataset + + # Load custom datasets + train_ds = load_dataset("SamuelYang/bookcorpus")["train"] + train_ds, val_ds, test_ds = split_train_into_train_val_test(train_ds, seed=42) + + # Add to SmashConfig + smash_config = SmashConfig() + smash_config.add_tokenizer("bert-base-uncased") + smash_config.add_data( + (train_ds, val_ds, test_ds), + collate_fn="text_generation_collate" + ) + + .. tab:: PrunaDataModule + + You can also create a fully custom ``PrunaDataModule`` use it in your workflow. + This process is more flexible but also more complex. It allows for more control over the dataset and the data loading process. + The process for defining a ``PrunaDataModule`` is highlighted in the :doc:`Evaluation ` page but a basic example of adding it to the ``SmashConfig`` is shown below. + + .. code-block:: python + + from pruna import SmashConfig, PrunaDataModule + + # Load PrunaDataModule + data = PrunaDataModule(...) + + # Add to SmashConfig + smash_config = SmashConfig() + smash_config.add_data(data) \ No newline at end of file diff --git a/docs/user_manual/customize.rst b/docs/user_manual/customize.rst new file mode 100644 index 00000000..336872eb --- /dev/null +++ b/docs/user_manual/customize.rst @@ -0,0 +1,39 @@ +Customize components +==================== + +|pruna| is designed to be customizable. You can add your own algorithms, datasets, and metrics to the package. + +.. grid:: 1 3 3 3 + + .. grid-item-card:: Add an algorithm + :text-align: center + :link: ./adding_algorithm.rst + + Steps to integrate a new compression algorithm, making it available in the ``SmashConfig``. + + .. grid-item-card:: Add a dataset + :text-align: center + :link: ./adding_dataset.rst + + Steps to integrate a new dataset, making it available in the ``SmashConfig``. + + .. grid-item-card:: Add a metric + :text-align: center + :link: ./adding_metric.rst + + Steps to integrate a new metric, making it available in the ``SmashConfig``. + +.. tip:: + + You can also customize the package by adding your own algorithms, datasets, and metrics. + Take a look at the :doc:`contributing guide ` to learn more. + If anything is unclear or you want to discuss your contribution before opening a PR, please reach out on `Discord `_ anytime! + +.. toctree:: + :maxdepth: 1 + :caption: Customize components + :hidden: + + adding_algorithm + adding_dataset + adding_metric \ No newline at end of file diff --git a/docs/user_manual/dataset.rst b/docs/user_manual/dataset.rst deleted file mode 100644 index 8094ad37..00000000 --- a/docs/user_manual/dataset.rst +++ /dev/null @@ -1,106 +0,0 @@ -Datasets -========================= - -|pruna| provides a variety of pre-configured datasets for different tasks. This guide will help you understand how to use datasets in your |pruna| workflow. - -Available Datasets -------------------- - -|pruna| currently supports the following datasets categorized by task: - -Text Generation -^^^^^^^^^^^^^^^ - -| ``WikiText``: Wikipedia text dataset for language modeling -| ``SmolTalk``: Everyday conversation dataset -| ``SmolSmolTalk``: Lightweight version of SmolTalk -| ``PubChem``: Chemical compound dataset in SELFIES format -| ``OpenAssistant``: Instruction-following dataset -| ``C4``: Large-scale web text dataset - -Image Classification -^^^^^^^^^^^^^^^^^^^^ - -| ``ImageNet``: Large-scale image classification dataset -| ``MNIST``: Handwritten digit classification dataset - -Text-to-Image -^^^^^^^^^^^^^^^^^^^^ - -| ``COCO``: Image captioning dataset -| ``LAION256``: Subset of LAION artwork dataset -| ``OpenImage``: Image quality preferences dataset - -Audio Processing -^^^^^^^^^^^^^^^^^^^^ - -| ``CommonVoice``: Multi-language speech dataset -| ``AIPodcast``: AI-focused podcast audio dataset - -Question Answering -^^^^^^^^^^^^^^^^^^^^ - -| ``Polyglot``: Fact completion dataset - -Using Datasets ---------------- - -There are two main ways to use datasets in |pruna|: - -1. Using String Identifier -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -What makes using the already implemented datasets so easy is that you can simply use the dataset's string identifier to add it to your :doc:`SmashConfig `: - -.. code-block:: python - - from pruna import SmashConfig - - smash_config = SmashConfig() - smash_config.add_tokenizer("bert-base-uncased") - smash_config.add_data("WikiText") - -2. Using Custom Datasets -^^^^^^^^^^^^^^^^^^^^^^^^^ - -You can also pass your own datasets as a tuple of ``(train, validation, test)`` datasets: - -.. code-block:: python - - from pruna import SmashConfig - from pruna.data.utils import split_train_into_train_val_test - from datasets import load_dataset - - # Load custom datasets - train_ds = load_dataset("SamuelYang/bookcorpus")["train"] - train_ds, val_ds, test_ds = split_train_into_train_val_test(train_ds, seed=42) - - # Add to SmashConfig - smash_config = SmashConfig() - smash_config.add_tokenizer("bert-base-uncased") - smash_config.add_data( - (train_ds, val_ds, test_ds), - collate_fn="text_generation_collate" - ) - -In this case, you need to specify the ``collate_fn`` to use for the dataset. The ``collate_fn`` is a function that takes a list of individual data samples and returns a batch of data in a unified format. -Your dataset will have to adhere to the formats expected by the ``collate_fn`` and this will be checked during a quick compatibility check when adding the dataset to the ``smash_config``. - - -.. autofunction:: pruna.data.collate.text_generation_collate -.. autofunction:: pruna.data.collate.image_generation_collate -.. autofunction:: pruna.data.collate.image_classification_collate -.. autofunction:: pruna.data.collate.audio_collate -.. autofunction:: pruna.data.collate.question_answering_collate - - - -.. _prunadatamodule: - -Accessing the PrunaDataModule directly -------------------------------------- - -You can also create and access the PrunaDataModule directly and use it in your workflow, e.g., if you want to pass it to the :doc:`evaluation agent `. - -.. autoclass:: pruna.data.pruna_datamodule.PrunaDataModule - :members: from_string, from_datasets diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst new file mode 100644 index 00000000..d6de59b4 --- /dev/null +++ b/docs/user_manual/evaluate.rst @@ -0,0 +1,219 @@ +Evaluate optimizations with the Evaluation Agent +================================================ + +This guide provides an introduction to evaluating model optimizations with |pruna|. + +Evaluation helps you understand how compression affects your models across different dimensions - from output quality to resource requirements. +This knowledge is essential for making informed decisions about which compression techniques work best for your specific needs. + +Basic Evaluation Workflow +------------------------- + +|pruna| follows a simple workflow for evaluating model optimizations: + +.. mermaid:: + :align: center + + graph LR + User -->|creates| Task + User -->|creates| EvaluationAgent + Task -->|uses| PrunaDataModule + Task -->|defines| Metrics + Metrics -->|includes| StatefulMetric + Metrics -->|includes| StatelessMetric + PrunaDataModule -->|provides data| EvaluationAgent + PrunaModel -->|provides predictions| EvaluationAgent + EvaluationAgent -->|evaluates| PrunaModel + EvaluationAgent -->|returns| Evaluation_Results + User -->|configures| EvaluationAgent + + subgraph Metric_Types + StatefulMetric + StatelessMetric + end + + style User fill:#bbf,stroke:#333,stroke-width:2px + style Task fill:#bbf,stroke:#333,stroke-width:2px + style EvaluationAgent fill:#bbf,stroke:#333,stroke-width:2px + style PrunaDataModule fill:#bbf,stroke:#333,stroke-width:2px + style PrunaModel fill:#bbf,stroke:#333,stroke-width:2px + style Evaluation_Results fill:#bbf,stroke:#333,stroke-width:2px + style Metrics fill:#bbf,stroke:#333,stroke-width:2px + +Let's see what that looks like in code. + +.. code-block:: python + + from pruna.evaluation.evaluation_agent import EvaluationAgent + from pruna.evaluation.task import Task + from pruna.data.pruna_datamodule import PrunaDataModule + + # Load the optimized model + optimized_model = PrunaModel.from_pretrained("CompVis/stable-diffusion-v1-4") + + # Create and configure Task + task = Task( + requests=["clip_score", "psnr"], + datamodule=PrunaDataModule.from_string('LAION256'), + device="cpu" + ) + + # Create and configure EvaluationAgent + eval_agent = EvaluationAgent(task) + + # Evaluate the model + eval_agent.evaluate(optimized_model) + +Evaluation Components +--------------------- + +The |pruna| package provides a variety of evaluation metrics to assess your models. +In this section, we’ll introduce the evaluation metrics you can use. + +Task +^^^^ + +The ``Task`` is a class that defines the task you want to evaluate your model on and it requires a set of metrics and a :ref:`PrunaDataModule ` to perform the evaluation. + +.. code-block:: python + + from pruna.evaluation.task import Task + from pruna.data.pruna_datamodule import PrunaDataModule + + task = Task( + requests=["image_generation_quality"], + datamodule=PrunaDataModule.from_string('LAION256'), + device="cpu" + ) + +Metrics +~~~~~~~ + +The ``Metrics`` is a class that defines the metrics you want to evaluate your model on. + +Metrics are the core components that calculate specific performance indicators. There are two main types of metrics: + +- **Stateful Metrics**: These metrics compute values directly from inputs without maintaining state across batches. +- **Stateless Metrics**: Metrics that maintain internal state and accumulate information across multiple batches. These are typically used for quality assessment. + +The ``Task`` accepts ``Metrics`` in three ways: + +.. tabs:: + + .. tab:: Predefined Options + + As a plain text request from predefined options (e.g., ``image_generation_quality``) + + .. code-block:: python + + from pruna.evaluation.task import Task + from pruna.data.pruna_datamodule import PrunaDataModule + + # Create the task + task = Task( + request="image_generation_quality", + datamodule=PrunaDataModule.from_string('LAION256'), + device="cpu" + ) + + .. tab:: List of Metric Names + + As a list of metric names (e.g., [``"clip_score"``, ``"psnr"``]) + + .. code-block:: python + + from pruna.evaluation.task import Task + from pruna.data.pruna_datamodule import PrunaDataModule + + # Create the task + task = Task( + metrics=["clip_score", "psnr"], + datamodule=PrunaDataModule.from_string('LAION256'), + device="cpu" + ) + + .. tab:: List of Metric Instances + + As a list of metric instances, which provides more flexibility in configuring the metrics. + + .. code-block:: python + + from pruna.evaluation.task import Task + from pruna.data.pruna_datamodule import PrunaDataModule + from pruna.evaluation.metrics.metric_psnr import PSNR + + # Initialize the metrics + metrics = [ + PSNR() + ] + + # Create the task + task = Task( + metrics=metrics, + datamodule=PrunaDataModule.from_string('LAION256'), + device="cpu" + ) + +.. note:: + + You can find the full list of available metrics in the :ref:`Metric Overview ` section. + +PrunaDataModule +~~~~~~~~~~~~~~~ + +The ``PrunaDataModule`` is a class that defines the data you want to evaluate your model on. +Data modules are a core component of the evaluation framework, providing standardized access to datasets for evaluating model performance before and after optimization. + +They offer the following functionality: + +- Standard dataloaders for training, validation, and testing +- Integration with appropriate collate functions for different data types +- Support for dataset size limitations for faster evaluation +- Compatibility with tokenizers for text-based tasks + +The ``Task`` accepts ``PrunaDataModule`` in three ways: + +.. tabs:: + + .. tab:: From String + + As a plain text request from predefined options (e.g., ``LAION256``) + + .. code-block:: python + + from pruna.data.pruna_datamodule import PrunaDataModule + + # Create the data Module + datamodule = PrunaDataModule.from_string('LAION256') + + .. tab:: From Datasets + + As a list of datasets, which provides more flexibility in configuring the data module. + + .. code-block:: python + + from pruna.data.pruna_datamodule import prunadatamodule + from transformers import AutoTokenizer + from datasets import load_dataset + + # Load a built-in dataset + tokenizer = AutoTokenizer.from_pretrained("gpt2") + + # Load custom datasets + train_ds = load_dataset("SamuelYang/bookcorpus")["train"] + train_ds, val_ds, test_ds = split_train_into_train_val_test(train_ds, seed=42) + + # Create the data module + datamodule = PrunaDataModule.from_datasets( + datasets=(train_ds, val_ds, test_ds), + collate_fn="text_generation_collate", + tokenizer=tokenizer, + collate_fn_args={"max_seq_len": 512}, + dataloader_args={"batch_size": 16, "num_workers": 4} + ) + +EvaluationAgent +^^^^^^^^^^^^^^^ + +The ``EvaluationAgent`` is a class that evaluates the performance of your model. +It is a subclass of ``pl.LightningModule`` and ``pruna.SmashConfig``. \ No newline at end of file diff --git a/docs/user_manual/evaluation.rst b/docs/user_manual/evaluation.rst deleted file mode 100644 index 7e4e2bf3..00000000 --- a/docs/user_manual/evaluation.rst +++ /dev/null @@ -1,330 +0,0 @@ -.. _evaluation: - -Evaluation Metrics -=================== - -The |pruna| package provides helpful evaluation tools to assess your models. In this section, we'll introduce the evaluation metrics you can use with the package. - -Evaluation helps you understand how compression affects your models across different dimensions - from output quality to resource requirements. This knowledge is essential for making informed decisions about which compression techniques work best for your specific needs. - -.. _quicktutorial: - -Quick Tutorial --------------- - -Before we start, here's a simple example showing how to evaluate your models using |pruna|. - -The rest of this guide provides more detailed explanations of each component and additional features available for model evaluation. - -.. code-block:: python - - import copy - - from diffusers import StableDiffusionPipeline - - from pruna import smash, SmashConfig - from pruna.data.pruna_datamodule import PrunaDataModule - from pruna.evaluation.evaluation_agent import EvaluationAgent - from pruna.evaluation.task import Task - - # Load data and set up smash config - smash_config = SmashConfig() - smash_config['cacher'] = 'deepcache' - - # Load the base model - model_path = "CompVis/stable-diffusion-v1-4" - pipe = StableDiffusionPipeline.from_pretrained(model_path) - - # Smash the model - copy_pipe = copy.deepcopy(pipe) - smashed_pipe = smash(copy_pipe, smash_config) - - # Define the task and the evaluation agent - metrics = ['clip_score', 'psnr'] - task = Task(metrics, datamodule=PrunaDataModule.from_string('LAION256')) - eval_agent = EvaluationAgent(task) - - # Evaluate base model, all models need to be wrapped in a PrunaModel before passing them to the EvaluationAgent - first_results = eval_agent.evaluate(pipe) - print(first_results) - - # Evaluate smashed model - smashed_results = eval_agent.evaluate(smashed_pipe) - print(smashed_results) - - -.. code-block:: python - - # Base model result output - {'clip_score_y_x': 28.0828} - - # Smashed model result output - {'clip_score_y_x': 28.4500, 'psnr_pairwise_y_gt': 18.7465} - -Evaluation Framework --------------------- - -The evaluation framework in |pruna| consists of several key components: - -Task -^^^^ -Processes user requests and converts them into a set of metrics. The ``Task`` accepts metrics in three ways: - -- As a plain text request from predefined options (e.g., ``image_generation_quality``) -- As a list of metric names (e.g., [``"clip_score"``, ``"psnr"``]) (see :ref:`Available Metrics ` below) -- As a list of metric instances - -In addition to metrics, ``Task`` requires a :ref:`PrunaDataModule ` to perform the evaluation. - -.. autoclass:: pruna.evaluation.task.Task - -Currently, ``Task`` supports the following plain textrequests: - -- ``image_generation_quality``: Creates metrics for evaluating image generation models (``clip_score``, ``pairwise_clip_score``, ``psnr``) - - -.. code-block:: python - - from pruna.evaluation.task import Task - from pruna.data.pruna_datamodule import PrunaDataModule - - task = Task("image_generation_quality", datamodule=PrunaDataModule.from_string('LAION256')) - -EvaluationAgent -^^^^^^^^^^^^^^^ -The main entry point for evaluating models. The ``EvaluationAgent``: - -- Takes a ``Task`` object that defines what metrics to use -- Provides methods to evaluate any model -- Handles the evaluation process, including separating metrics by execution strategy -- Runs inference on the model to generate predictions -- Caches predictions to avoid redundant computations -- Passes ground truth data and predictions to the appropriate metrics -- Collects and returns results from all metrics - -.. autoclass:: pruna.evaluation.evaluation_agent.EvaluationAgent - :members: evaluate - -.. container:: hidden_code - - .. code-block:: python - - from pruna.evaluation.task import Task - from pruna.data.pruna_datamodule import PrunaDataModule - - data_module = PrunaDataModule.from_string('LAION256') - data_module.limit_datasets(10) - - task = Task("image_generation_quality", datamodule=data_module) - -.. code-block:: python - - from pruna.evaluation.evaluation_agent import EvaluationAgent - - eval_agent = EvaluationAgent(task) - - -For the full example running evaluation please see :ref:`Quick Tutorial ` above. - -.. _metrics: - -Metrics -------- - -Metrics help quantify different aspects of model performance, from output quality to resource requirements. The |pruna| package includes metrics for both quality assessment and resource utilization. - -When using the ``EvaluationAgent``, all metrics are executed automatically as part of the evaluation pipeline. The agent handles model inference, data preparation, and passing the appropriate inputs to each metric, eliminating the need to run metrics individually. - -Metrics can operate in both single-model and pairwise modes: - -- In single-model mode, each evaluation produces independent scores for the model being evaluated. -- In pairwise mode, metrics compare a subsequent model against the first model evaluated by the agent. Usually, this is used to compare the base model (first model) with its smashed version (subsequent model). The first model's outputs are cached and used as a reference point for all following evaluations. The pairwise comparison produces a single score that quantifies the relationship (e.g., similarity or difference) between the two models. - -Our metrics fall into two implementation categories that work differently under the hood: - -Base Metrics -^^^^^^^^^^^^^ -Simple metrics that compute values directly from inputs without maintaining state across batches. Examples include: -- Model Architecture metrics -- Energy consumption metrics -- Memory usage metrics - -elapsed_time -"""""""""""" - -Measures inference time, latency, and throughput. - -:Evaluation on CPU: Yes. -:Required: - A PrunaModel object that defines the model to evaluate. - A DataLoader object that defines the dataloader to evaluate the model on. -:Parameters: - - | ``n_iterations``: Number of inference iterations to measure (default 100). - | ``n_warmup_iterations``: Number of warmup iterations before measurement (default 10). - | ``device``: Device to run inference on (default "cuda"). - | ``timing_type``: Type of timing to use ("sync" or "async", default "sync"). - -`gpu_memory `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -Measures peak GPU memory usage during model loading and execution. - -:Evaluation on CPU: No. -:Required: - Path to the PrunaModel to evaluate. - A DataLoader object that defines the dataloader to evaluate the model on. - The model class to load the model from the path. -:Parameters: - - | ``mode``: Memory measurement mode ("disk", "inference", or "training"). - | ``gpu_indices``: List of GPU indices to monitor (default all available GPUs). - -`energy `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -Measures energy consumption in kilowatt-hours (kWh) and CO2 emissions in kilograms (kg). - -:Evaluation on CPU: Yes. -:Description: Measures energy consumption in kilowatt-hours (kWh) and CO2 emissions in kilograms (kg). -:Required: - A PrunaModel object that defines the model to evaluate. - A DataLoader object that defines the dataloader to evaluate the model on. -:Parameters: - - | ``n_iterations``: Number of inference iterations to measure (default 100). - | ``n_warmup_iterations``: Number of warmup iterations before measurement (default 10). - | ``device``: Device to run inference on (default "cuda"). - -`model_architecture `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -Measures the number of parameters and MACs (multiply-accumulate operations) in the model. - -:Evaluation on CPU: Yes. -:Required: - A PrunaModel object that defines the model to evaluate. - A DataLoader object that defines the dataloader to evaluate the model on. -:Parameters: - - | ``device``: Device to evaluate the model on (default "cuda"). - -Stateful Metrics -^^^^^^^^^^^^^^^^^ -Metrics that maintain internal state and accumulate information across multiple batches. These are typically used for quality assessment. - -Most of our stateful metrics are implemented using the TorchMetricsWrapper, which adapts metrics from the `TorchMetrics `_ library to work within our evaluation framework. This allows us to leverage the robust implementations provided by TorchMetrics while maintaining a consistent API. - -`clip_score `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -Measures the similarity between images and text using the CLIP model. - - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. -:Parameters: Accepts all parameters from the TorchMetrics CLIPScore implementation. - -`pairwise_clip_score `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -Measures the similarity between images of first and subsequent models using the CLIP model. - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. -:Parameters: Accepts all parameters from the TorchMetrics CLIPScore implementation. - -`cmmd `_ -"""""""""""""""""""""""""""""""""""""""""""" - -CMMD measures the distributional discrepancy between two sets of images or text by computing Maximum Mean Discrepancy (MMD) in the CLIP embedding space. It captures both semantic and visual alignment. - -Key Benefits: - -- **Distribution-Free:** Does not rely on any assumptions about the underlying feature distribution. -- **Unbiased Estimation:** Provides a statistically unbiased measure of the discrepancy between two image sets. -- **Sample Efficiency:** Achieves reliable estimates even with smaller image samples, making it suitable for rapid evaluations. -- **Human-Aligned:** Demonstrates better agreement with human perceptual assessments of image quality compared to FID. - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. -:Parameters: - - | ``clip_model_name``: Name of the CLIP model to use (default "openai/clip-vit-large-patch14-336"). - | ``call_type``: Call type to use for the metric (default "gt_y"). For pairwise evaluation pass "pairwise" or "pairwise_gt_y". - | ``device``: Device to run the metric on (default "cuda"). - - - -`accuracy `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures the proportion of correct predictions in classification tasks. - - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. TorchMetrics requires a 'task' parameter to be set to 'binary', 'multiclass', or 'multilabel'. Each task type may have additional specific requirements - please refer to the TorchMetrics documentation for details. -:Parameters: Accepts all parameters from the TorchMetrics Accuracy implementation (task, num_classes, threshold, etc.). - -`precision `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures the proportion of positive identifications that were actually correct. - - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. TorchMetrics requires a 'task' parameter to be set to 'binary', 'multiclass', or 'multilabel'. Each task type may have additional specific requirements - please refer to the TorchMetrics documentation for details. -:Parameters: Accepts all parameters from the TorchMetrics Precision implementation (task, num_classes, threshold, etc.). - -`recall `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures the proportion of actual positives that were identified correctly. - - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. TorchMetrics requires a 'task' parameter to be set to 'binary', 'multiclass', or 'multilabel'. Each task type may have additional specific requirements - please refer to the TorchMetrics documentation for details. -:Parameters: Accepts all parameters from the TorchMetrics Recall implementation (task, num_classes, threshold, etc.). - -`perplexity `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures how well a probability model predicts a text sample. - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. -:Parameters: Accepts all parameters from the TorchMetrics Perplexity implementation. - -`fid `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures the similarity between generated and real image distributions using the Frechet Distance between Gaussian distributions fitted to the Inception embeddings of the generated and real images. - -FID compares the **distribution** of real and generated images in a high-dimensional feature space. Since it estimates **mean and covariance statistics**, smaller sample sizes can introduce high variance, making the metric less stable. Large-scale evaluations often use **tens of thousands of images**, but for practical use, smaller sample sizes may still provide a reasonable approximation. - -**Computation Considerations** - -When generating images and computing FID on **thousands to tens of thousands of samples**, the process can take **multiple hours to several days**, even on a high-end GPU like an **A100 or RTX 4090**. On mid-range GPUs like a **3060 or 4060**, it can take **significantly longer**. A rough approximation using **a few thousand images** may still take **several hours**, even with strong hardware. - -:Evaluation on CPU: No (impractical due to the high computational cost) -:Required: Inputs, ground truth and predictions. -:Parameters: Accepts all parameters from the TorchMetrics FrechetInceptionDistance implementation (feature extraction parameters, etc.). - -`psnr `_ -""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures the peak signal-to-noise ratio (PSNR) between two images. - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. -:Parameters: Accepts all parameters from the TorchMetrics PSNR implementation. - -`ssim `_ -""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures the structural similarity index (SSIM) between two images. - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. -:Parameters: Accepts all parameters from the TorchMetrics SSIM implementation. - -`lpips `_ -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Measures the Learned Perceptual Image Patch Similarity (LPIPS) between two images. - -:Evaluation on CPU: Yes. -:Required: Inputs, ground truth and predictions. -:Parameters: Accepts all parameters from the TorchMetrics LPIPS implementation. \ No newline at end of file diff --git a/docs/user_manual/optimize.rst b/docs/user_manual/optimize.rst new file mode 100644 index 00000000..c8b8740d --- /dev/null +++ b/docs/user_manual/optimize.rst @@ -0,0 +1,207 @@ +Optimize your first model +========================= + +This guide provides a quick introduction to optimizing AI models with |pruna|. + +You'll learn how to use Pruna's core functionality to make your models faster, smaller, cheaper, and greener. +For installation instructions, see :doc:`Installation `. + +Basic Optimization Workflow +--------------------------- + +|pruna| follows a simple workflow for optimizing models: + +.. mermaid:: + :align: center + + graph LR + A[Load Model] --> B[Define SmashConfig] + B --> C[Optimize Model] + C --> D[Evaluate Model] + D --> E[Run Inference] + style A fill:#bbf,stroke:#333,stroke-width:2px + style B fill:#bbf,stroke:#333,stroke-width:2px + style C fill:#bbf,stroke:#333,stroke-width:2px + style D fill:#bbf,stroke:#333,stroke-width:2px + style E fill:#bbf,stroke:#333,stroke-width:2px + +Let's see what that looks like in code. + +.. code-block:: python + + from pruna import smash, SmashConfig + from diffusers import StableDiffusionPipeline + from pruna.data.pruna_datamodule import PrunaDataModule + from pruna.evaluation.evaluation_agent import EvaluationAgent + from pruna.evaluation.task import Task + + # Load the model + model = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") + + # Create and configure SmashConfig + smash_config = SmashConfig() + smash_config["cacher"] = "deepcache" + + # Optimize the model + optimized_model = smash(model=model, smash_config=smash_config) + + # Evaluate the model + metrics = ['clip_score', 'psnr'] + task = Task(metrics, datamodule=PrunaDataModule.from_string('LAION256')) + eval_agent = EvaluationAgent(task) + eval_agent.evaluate(optimized_model) + + # Run inference + optimized_model("A serene landscape with mountains").images[0] + +Step-by-Step Optimization Workflow +---------------------------------- + +Step 1: Load a pretrained model +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +First, load any model using its original library, like ``transformers`` or ``diffusers``: + +.. code-block:: python + + from diffusers import StableDiffusionPipeline + + base_model = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") + + +Step 2: Define optimizations with a ``SmashConfig`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +After loading the model, we can define a ``SmashConfig`` to customize the optimizations we want to apply. +This ``SmashConfig`` is a dictionary-like object that configures which optimizations to apply to your model. +You can specify multiple optimization algorithms from different categories like batching, caching and quantization. + +For now, let's just use a ``cacher`` to accelerate the model during inference. + +.. code-block:: python + + from pruna import SmashConfig + + smash_config = SmashConfig() + smash_config["cacher"] = "deepcache" # Accelerate the model with caching + +Pruna support a wide range of algorithms for specific optimizations, all with different trade-offs. +To understand how to configure the right one for your scenario, see :doc:`Define a SmashConfig `. + +Step 3: Apply optimizations with ``smash`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``smash()`` function is the core of Pruna. It takes your model and ``SmashConfig``, applies the specified optimizations. +Let's use the ``smash()`` function to apply the configured optimizations: + +.. code-block:: python + + from pruna import smash + + optimized_model = smash(model=base_model, smash_config=smash_config) + + +The ``smash()`` function returns a ``PrunaModel`` - a wrapper that provides a standardized interface for the optimized model. So, we can still use the model as we would use the original one. + +Step 4: Evaluate the optimized model with the ``EvaluationAgent`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To evaluate the optimized model, we can use the same interface as the original model. + +.. code-block:: python + + from pruna.data.pruna_datamodule import PrunaDataModule + from pruna.evaluation.evaluation_agent import EvaluationAgent + + metrics = ['clip_score', 'psnr'] + task = Task(metrics, datamodule=PrunaDataModule.from_string('LAION256')) + eval_agent = EvaluationAgent(task) + eval_agent.evaluate(optimized_model) + +To understand how to run more complex evaluation workflows, see :doc:`Evaluate a model `. + +Step 5: Run inference with the optimized model +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To run inference with the optimized model, we can use the same interface as the original model. + +.. code-block:: python + + optimized_model("A serene landscape with mountains").images[0] + +Example use cases +----------------- + +Let's look at some specific examples for different model types. + +Example 1: Diffusion Model Optimization +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: python + + from diffusers import StableDiffusionPipeline + from pruna import smash, SmashConfig + + # Load the model + model = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") + + # Create and configure SmashConfig + smash_config = SmashConfig() + smash_config["cacher"] = "deepcache" + smash_config["compiler"] = "stable_fast" + + # Optimize the model + optimized_model = smash(model=model, smash_config=smash_config) + + # Generate an image + optimized_model("A serene landscape with mountains").images[0] + +Example 2: Large Language Model Optimization +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: python + + from transformers import AutoModelForCausalLM + from pruna import smash, SmashConfig + + # Load the model + model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m") + + # Create and configure SmashConfig + smash_config = SmashConfig() + smash_config["quantizer"] = "gptq" # Apply GPTQ quantization + + # Optimize the model + optimized_model = smash(model=model, smash_config=smash_config) + + # Use the model for generation + input_text = "The best way to learn programming is" + optimized_model(input_text) + + +Example 3: Speech Recognition Optimization +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: python + + from transformers import AutoModelForSpeechSeq2Seq + from pruna import smash, SmashConfig + import torch + + # Load the model + model_id = "openai/whisper-large-v3" + model = AutoModelForSpeechSeq2Seq.from_pretrained( + model_id, torch_dtype=torch.float16, low_cpu_mem_usage=True + ).to("cuda") + + # Create and configure SmashConfig + smash_config = SmashConfig() + smash_config.add_processor(model_id) # Required for Whisper + smash_config["compiler"] = "c_whisper" + smash_config["batcher"] = "whisper_s2t" + + # Optimize the model + optimized_model = smash(model=model, smash_config=smash_config) + + # Use the model for transcription + optimized_model("audio_file.wav") \ No newline at end of file diff --git a/docs/user_manual/save_load.rst b/docs/user_manual/save_load.rst index 74c4404c..01377a60 100644 --- a/docs/user_manual/save_load.rst +++ b/docs/user_manual/save_load.rst @@ -1,10 +1,12 @@ -Saving and Loading Pruna Models -=============================== +Save and Load Models +===================== -After smashing a model using |pruna|, you can save it to disk and load it later using the built-in save and load functionality. +This guide provides a quick introduction to saving and loading optimized AI models with |pruna|. -Saving and Loading Models -------------------------- +You will learn how to save and load a ``PrunaModel`` after smashing a model using |pruna|. Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. + +Saving a ``PrunaModel`` +----------------------- To save a smashed model, use the ``PrunaModel.save_pretrained()`` or ``PrunaModel.save_to_hub()`` method. This method saves all necessary model files and as well as the smash configuration to the specified directory: @@ -46,13 +48,20 @@ To save a smashed model, use the ``PrunaModel.save_pretrained()`` or ``PrunaMode smashed_model = smash(model=base_model, smash_config=smash_config) # Save the model - smashed_model.save_to_hub("PrunaAI/smashed-stable-diffusion-v1-4") + smashed_model.save_to_hub("PrunaAI/smashed-stable-diffusion-v1-4-smashed") + + .. tip:: + + When saving models to the hub, we recommend to use a suffix like ``-smashed`` to indicate that the model has been smashed with |pruna|. The save operation will: 1. Save the model weights and architecture, including information on how to load the model later on 2. Save the ``smash_config`` (including tokenizer and processor if present, data will be detached and not reloaded) +Loading a ``PrunaModel`` +------------------------ + To load a previously saved ``PrunaModel``, use the ``PrunaModel.from_pretrained()`` or ``PrunaModel.from_hub()`` class method: .. tabs:: @@ -74,12 +83,36 @@ To load a previously saved ``PrunaModel``, use the ``PrunaModel.from_pretrained( loaded_model = PrunaModel.from_hub("PrunaAI/smashed-stable-diffusion-v1-4") The load operation will: + 1. Load the model architecture and weights and cast them to the device specified in the SmashConfig 2. Restore the smash configuration Special Considerations ---------------------- +Loading Keyword Arguments +~~~~~~~~~~~~~~~~~~~~~~~~~ +We generally recommend to load the smashed model in the same configuration as the base model, **in particular** if the two should be compared in terms of efficiency and quality. +So, when the base model was loaded with e.g. a specific precision: + +.. code-block:: python + + import torch + from diffusers import StableDiffusionPipeline + + base_model = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", torch_dtype=torch.float16) + +you should also load the smashed model as follows: + +.. code-block:: python + + from pruna import PrunaModel + + loaded_model = PrunaModel.from_pretrained("saved_model/", torch_dtype=torch.float16) + +Depending on the saving function of the algorithm combination not all keyword arguments are required for loading (e.g. some are set by the algorithm combination itself). +In that case, we discard and log a warning about unused keyword arguments. + Algorithm Reapplication ~~~~~~~~~~~~~~~~~~~~~~~~ Some algorithms, particularly compilers and certain quantization methods, need to be reapplied after loading, as, for example, a compiled model can be rarely saved in its compiled state. @@ -93,10 +126,4 @@ Set ``verbose=True`` when loading if you want to see warning messages as well as from pruna import PrunaModel - loaded_model = PrunaModel.from_pretrained("saved_model/", verbose=True) - -``PrunaModel`` Function Documentation ---------------------------------------------- - -.. autoclass:: pruna.engine.pruna_model.PrunaModel - :members: from_pretrained, from_hub, save_to_hub, save_pretrained \ No newline at end of file + loaded_model = PrunaModel.from_pretrained("saved_model/", verbose=True) \ No newline at end of file diff --git a/docs/user_manual/smash.rst b/docs/user_manual/smash.rst deleted file mode 100644 index fd6aff7d..00000000 --- a/docs/user_manual/smash.rst +++ /dev/null @@ -1,57 +0,0 @@ -smash -========================= - -The ``smash`` function is the main function in |pruna| for optimizing models. In the following sections we will show you how to use it. - -Calling the ``smash`` Function ---------------------------------------------- - -In preparation to using ``smash``, we have to load our model and define a ``SmashConfig``. As an example, we will take a simple model by loading the ``ViT-B/16`` model from ``torchvision``. - -.. code-block:: python - - import torchvision - - base_model = torchvision.models.vit_b_16(weights="ViT_B_16_Weights.DEFAULT").cuda() - -Next, we will define a :doc:`SmashConfig ` and activate the ``torch_compile`` compiler. - -.. code-block:: python - - from pruna import SmashConfig - smash_config = SmashConfig() - smash_config['compiler'] = 'torch_compile' - -We are now ready to call the ``smash`` function! - -We can pass the model and the ``SmashConfig`` to the ``smash`` function as follows: - -.. code-block:: python - - from pruna import smash - - smashed_model = smash( - model=base_model, - smash_config=smash_config, - ) - -The resulting smashed model can be used in the same way as the original one. - -We perform compatibility checks to ensure that the model is compatible with the algorithms that you have selected at the beginning of the ``smash`` process. If you wish to skip these checks, you can set the ``experimental`` flag to ``True``: - -.. code-block:: python - - smashed_model = smash( - model=base_model, - smash_config=smash_config, - experimental=True, - ) - -Please note that this can lead to undefined behavior or difficult-to-debug errors. - -Importantly, the returned model offers save and load functionality that allows you to save the model and load it in its smashed state, see :doc:`save_load`. - -``smash`` Function Documentation ---------------------------------------------- - -.. autofunction:: pruna.smash.smash \ No newline at end of file diff --git a/docs/user_manual/smash_config.rst b/docs/user_manual/smash_config.rst deleted file mode 100644 index f6509d67..00000000 --- a/docs/user_manual/smash_config.rst +++ /dev/null @@ -1,67 +0,0 @@ -SmashConfig -========================= - -``SmashConfig`` is an essential tool in |pruna| for configuring parameters to optimize your models. This manual explains how to define and use a ``SmashConfig``. - -Defining a simple ``SmashConfig`` ---------------------------------- - -Define a ``SmashConfig`` using the following snippet: - -.. code-block:: python - - from pruna import SmashConfig - smash_config = SmashConfig() - -After creating an empty ``SmashConfig``, you can set activate a algorithm by adding it to the ``SmashConfig``: - -.. code-block:: python - - smash_config['quantizer'] = 'hqq' - -Additionally, you can overwrite :doc:`the defaults of the algorithm ` you have added by setting the hyperparameters in the ``SmashConfig``: - -.. code-block:: python - - smash_config['hqq_weight_bits'] = 4 - -You're done! You created your ``SmashConfig`` and can now :doc:`pass it to the smash function. ` - - -Adding a Dataset, Tokenizer or Processor ----------------------------------------- - -Some algorithms require a dataset, tokenizer or processor to be passed to the ``SmashConfig``. -For example, the ``gptq`` quantizer requires a dataset and a tokenizer. We can pass them to the ``SmashConfig`` e.g. as follows: - -.. code-block:: python - - from pruna import SmashConfig - smash_config = SmashConfig() - smash_config.add_tokenizer("facebook/opt-125m") - smash_config.add_data("WikiText") - -As you can see in this example, we can add a dataset simply by passing the name of the dataset. However, the ``add_data()`` function also supports other input formats. For more information, see the :doc:`dataset documentation `. - -We can now activate the ``gptq`` quantizer by adding it to the ``SmashConfig``: - -.. code-block:: python - - smash_config['quantizers'] = 'gptq' - -Similarly, we can add a processor to the ``SmashConfig`` if required, like for example by the ``c_whisper`` compiler: - -.. code-block:: python - - from pruna import SmashConfig - smash_config = SmashConfig() - smash_config.add_processor("openai/whisper-large-v3") - smash_config['compiler'] = 'c_whisper' - -If you try to activate a algorithm that requires a dataset, tokenizer or processor and haven't added them to the ``SmashConfig``, you will receive an error. Make sure to add them before activating the algorithm! If you want to know which algorithms require a dataset, tokenizer or processor, you can look at :doc:`the compression algorithm overview `. - -``SmashConfig`` Documentation ---------------------------------------------- - -.. autoclass:: pruna.config.smash_config.SmashConfig - :members: add_data, add_tokenizer, add_processor, save_to_json, load_from_json, flush_configuration, load_dict From f80f28ff08a3b4aa7eaca5774a879b2567c85f6c Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 10:24:23 +0200 Subject: [PATCH 19/31] docs: enhance 'How to Contribute' section with detailed guidelines and community engagement options --- docs/contributions/how_to_contribute.rst | 129 ++++++++++++++++++++--- 1 file changed, 113 insertions(+), 16 deletions(-) diff --git a/docs/contributions/how_to_contribute.rst b/docs/contributions/how_to_contribute.rst index 2da784e0..0772d68c 100644 --- a/docs/contributions/how_to_contribute.rst +++ b/docs/contributions/how_to_contribute.rst @@ -1,24 +1,121 @@ -Since Pruna offers a broad range of compression algorithms, the following table provides a high-level overview of all methods available in Pruna. For a detailed description of each algorithm, have a look at our [documentation](https://docs.pruna.ai/en/stable/). +How to Contribute πŸ’œ +==================== +Since you landed on this part of the documentation, we want to first of all say thank you! πŸ’œ +Contributions from the community are essential to improving |pruna|, we appreciate your effort in making the repository better for everyone! -| Technique | Description | Speed | Memory | Accuracy | -| --- | --- | --- | --- | --- | -| Batching | Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing overall processing time. | βœ… | ❌ | 〰️ | -| Caching | Stores intermediate results of computations to speed up subsequent operations, reducing inference time by reusing previously computed results. | βœ… | 〰️ | 〰️ | -| Compilation | Compilation optimises the model with instructions for specific hardware. | βœ… | βž– | 〰️ | -| Distillation | Trains a smaller, simpler model to mimic a larger, more complex model. | βœ… | βœ… | ❌ | -| Quantization | Reduces the precision of weights and activations, lowering memory requirements. | βœ… | βœ… | ❌ | -| Pruning | Removes less important or redundant connections and neurons, resulting in a sparser, more efficient network. | βœ… | βœ… | ❌ | -| Recovering | Restores the performance of a model after compression. | 〰️ | 〰️ | βœ… | +Please make sure to review and adhere to the `Pruna Code of Conduct `_ before contributing to Pruna. +Any violations will be handled accordingly and result in a ban from the Pruna community and associated platforms. +Contributions that do not adhere to the code of conduct will be ignored. + +There are various ways you can contribute: + +- Have a question? Discuss with us on `Discord `_ or check out the :doc:`/resources/faq` +- Have an idea for a new tutorial? Open an issue with a :ref:`feature-request` or chat with us on `Discord `_ +- Found a bug? Open an issue with a :ref:`bug-report` +- Documentation improvements? Open an issue with a :ref:`documentation-improvement` +- Want a new feature? Open an issue with a :ref:`feature-request` +- Have a new algorithm to add? Check out: :doc:`adding_algorithm` +- Have a new metric to add? Check out: :doc:`adding_metric` +- Have a new dataset to add? Check out: :doc:`adding_dataset` + + +.. _how-to-contribute: + +Setup +----- + +If you want to contribute to |pruna| with a Pull Request, you can do so by following these steps. +If it is your very first time contributing to an open source project, we recommend to start with `this guide `_ for some generally helpful tips. + +1. Clone the repository +^^^^^^^^^^^^^^^^^^^^^^^^ + +First, fork the repository by navigating to the original `pruna repository `_ on GitHub and click the **Fork** button at the top-right. +This creates a copy of the repository in your own GitHub account. +Then, clone the forked repository from your account to your local machine and change into its directory: + +.. code-block:: bash + + git clone https://github.com/your_username/pruna.git + cd pruna + +To keep your fork up to date with the original repository, add the upstream remote: + +.. code-block:: bash + + git remote add upstream https://github.com/PrunaAI/pruna.git + +Always work on a new branch rather than the main branch. You can create a new branch for your feature or fix: + +.. code-block:: bash + + git checkout -b feat/new-feature + + + +2. Installation +^^^^^^^^^^^^^^^^^^^^^^ + +You can now set up a virtual environment of your choice and install the dependencies by running the following command: + +.. code-block:: bash + + pip install -e . + pip install -e .[dev] + pip install -e .[tests] + +You can then also install the pre-commit hooks with + +.. code-block:: bash + + pre-commit install + + +3. Develop your contribution +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +You are now ready to work on your contribution. Check out a branch on your forked repository and start coding! +When committing your changes, we recommend to follow the `Conventional Commit Guidelines `_. + +.. code-block:: bash + + git checkout -b feat/new-feature + git add . + git commit -m "feat: new amazing feature setup" + git push origin feat/new-feature + +Make sure to develop your contribution in a way that is well documented, concise and easy to maintain. +We will do our best to have your contribution integrated and maintained into |pruna| but reserve the right to reject contributions that we do not feel are in the best interest of the project. + +4. Run the tests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We have a comprehensive test suite that is designed to catch potential issues before they are merged into |pruna|. +When you make a contribution, it is highly recommended to not only run the existing tests but also to add new tests that cover your contribution. + +You can run the tests by running the following command: + +.. code-block:: bash + + pytest + +If you want to run only the tests with a specific marker, e.g. fast CPU tests, you can do so by running: + +.. code-block:: bash + + pytest -m "cpu and not slow" + + +5. Create a Pull Request +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Once you have made your changes and tested them, you can create a Pull Request. +We will then review your Pull Request and get back to you as soon as possible. +If there are any questions along the way, please do not hesitate to reach out on `Discord `_. -βœ…(improves), βž–(stays the same), 〰️(could worsen), ❌(worsens) -

-

Pruna AI Logo

-
-## Pruna Sad FAQ and Troubleshooting -If you can not find an answer to your question or problem in our [documentation][documentation], in our [FAQs][docs-faq] or in an existing issue, we are happy to help you! You can either get help from the Pruna community on [Discord][discord], join our [Office Hours][docs-office-hours] or open an issue on GitHub. \ No newline at end of file From 631abaf0ef20f34fe79fd0d094bec2eb89632d5c Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 10:31:32 +0200 Subject: [PATCH 20/31] docs: update contribution guidelines with correct paths for adding algorithms, metrics, and datasets --- docs/contributions/how_to_contribute.rst | 6 +++--- docs/user_manual/evaluate.rst | 3 +-- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/contributions/how_to_contribute.rst b/docs/contributions/how_to_contribute.rst index 0772d68c..4a51f8da 100644 --- a/docs/contributions/how_to_contribute.rst +++ b/docs/contributions/how_to_contribute.rst @@ -15,9 +15,9 @@ There are various ways you can contribute: - Found a bug? Open an issue with a :ref:`bug-report` - Documentation improvements? Open an issue with a :ref:`documentation-improvement` - Want a new feature? Open an issue with a :ref:`feature-request` -- Have a new algorithm to add? Check out: :doc:`adding_algorithm` -- Have a new metric to add? Check out: :doc:`adding_metric` -- Have a new dataset to add? Check out: :doc:`adding_dataset` +- Have a new algorithm to add? Check out: :doc:`/docs_pruna/user_manual/adding_algorithm` +- Have a new metric to add? Check out: :doc:`/docs_pruna/user_manual/adding_metric` +- Have a new dataset to add? Check out: :doc:`/docs_pruna/user_manual/adding_dataset` .. _how-to-contribute: diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst index d6de59b4..596a8608 100644 --- a/docs/user_manual/evaluate.rst +++ b/docs/user_manual/evaluate.rst @@ -215,5 +215,4 @@ The ``Task`` accepts ``PrunaDataModule`` in three ways: EvaluationAgent ^^^^^^^^^^^^^^^ -The ``EvaluationAgent`` is a class that evaluates the performance of your model. -It is a subclass of ``pl.LightningModule`` and ``pruna.SmashConfig``. \ No newline at end of file +The ``EvaluationAgent`` is a class that evaluates the performance of your model. \ No newline at end of file From 05ec40261f411ed574065454e134f7b3ff7ce725 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 16:14:51 +0200 Subject: [PATCH 21/31] docs: update user manual with additional guidance on model evaluation and saving/loading processes --- docs/user_manual/configure.rst | 1 + docs/user_manual/evaluate.rst | 142 ++++++++++++++++++++++++--------- docs/user_manual/save_load.rst | 74 ++++++++++++++++- 3 files changed, 177 insertions(+), 40 deletions(-) diff --git a/docs/user_manual/configure.rst b/docs/user_manual/configure.rst index 00602b3e..77f601fd 100644 --- a/docs/user_manual/configure.rst +++ b/docs/user_manual/configure.rst @@ -6,6 +6,7 @@ This guide provides an introduction to configuring model optimization strategies Model optimization configuration relies on the ``SmashConfig`` class. The ``SmashConfig`` class provides a flexible dictionary-like interface for configuring model optimization strategies. It manages algorithms, hyperparameters, and additional components like tokenizers, processors and datasets. +Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. Basic Configuration Workflow ---------------------------- diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst index 596a8608..feb4da70 100644 --- a/docs/user_manual/evaluate.rst +++ b/docs/user_manual/evaluate.rst @@ -5,6 +5,7 @@ This guide provides an introduction to evaluating model optimizations with |prun Evaluation helps you understand how compression affects your models across different dimensions - from output quality to resource requirements. This knowledge is essential for making informed decisions about which compression techniques work best for your specific needs. +Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. Basic Evaluation Workflow ------------------------- @@ -49,12 +50,12 @@ Let's see what that looks like in code. from pruna.data.pruna_datamodule import PrunaDataModule # Load the optimized model - optimized_model = PrunaModel.from_pretrained("CompVis/stable-diffusion-v1-4") + optimized_model = PrunaModel.from_pretrained("PrunaAI/opt-125m-smashed") # Create and configure Task task = Task( - requests=["clip_score", "psnr"], - datamodule=PrunaDataModule.from_string('LAION256'), + requests=["accuracy"], + datamodule=PrunaDataModule.from_string('WikiText'), device="cpu" ) @@ -73,24 +74,11 @@ In this section, we’ll introduce the evaluation metrics you can use. Task ^^^^ -The ``Task`` is a class that defines the task you want to evaluate your model on and it requires a set of metrics and a :ref:`PrunaDataModule ` to perform the evaluation. - -.. code-block:: python - - from pruna.evaluation.task import Task - from pruna.data.pruna_datamodule import PrunaDataModule - - task = Task( - requests=["image_generation_quality"], - datamodule=PrunaDataModule.from_string('LAION256'), - device="cpu" - ) +The ``Task`` is a class that defines the task you want to evaluate your model on and it requires a set of :doc:`Metrics ` and a :doc:`PrunaDataModule ` to perform the evaluation. Metrics ~~~~~~~ -The ``Metrics`` is a class that defines the metrics you want to evaluate your model on. - Metrics are the core components that calculate specific performance indicators. There are two main types of metrics: - **Stateful Metrics**: These metrics compute values directly from inputs without maintaining state across batches. @@ -109,7 +97,6 @@ The ``Task`` accepts ``Metrics`` in three ways: from pruna.evaluation.task import Task from pruna.data.pruna_datamodule import PrunaDataModule - # Create the task task = Task( request="image_generation_quality", datamodule=PrunaDataModule.from_string('LAION256'), @@ -125,7 +112,6 @@ The ``Task`` accepts ``Metrics`` in three ways: from pruna.evaluation.task import Task from pruna.data.pruna_datamodule import PrunaDataModule - # Create the task task = Task( metrics=["clip_score", "psnr"], datamodule=PrunaDataModule.from_string('LAION256'), @@ -134,22 +120,16 @@ The ``Task`` accepts ``Metrics`` in three ways: .. tab:: List of Metric Instances - As a list of metric instances, which provides more flexibility in configuring the metrics. + As a list of metric (e.g., ``CMMD()``), which provides more flexibility in configuring the metrics. .. code-block:: python from pruna.evaluation.task import Task from pruna.data.pruna_datamodule import PrunaDataModule - from pruna.evaluation.metrics.metric_psnr import PSNR - - # Initialize the metrics - metrics = [ - PSNR() - ] + from pruna.evaluation.metrics import CMMD, TorchMetricWrapper - # Create the task task = Task( - metrics=metrics, + metrics=[CMMD(), TorchMetricWrapper(metric_name="accuracy")], datamodule=PrunaDataModule.from_string('LAION256'), device="cpu" ) @@ -164,27 +144,32 @@ PrunaDataModule The ``PrunaDataModule`` is a class that defines the data you want to evaluate your model on. Data modules are a core component of the evaluation framework, providing standardized access to datasets for evaluating model performance before and after optimization. -They offer the following functionality: +A more detailed overview of the ``PrunaDataModule``, its datasets and their corresponding collate functions can be found in the :doc:`Data Module Overview ` section. -- Standard dataloaders for training, validation, and testing -- Integration with appropriate collate functions for different data types -- Support for dataset size limitations for faster evaluation -- Compatibility with tokenizers for text-based tasks - -The ``Task`` accepts ``PrunaDataModule`` in three ways: +The ``Task`` accepts ``PrunaDataModule`` in two different ways: .. tabs:: .. tab:: From String - As a plain text request from predefined options (e.g., ``LAION256``) + As a plain text request from predefined options (e.g., ``WikiText``) .. code-block:: python from pruna.data.pruna_datamodule import PrunaDataModule + from transformers import AutoTokenizer + + # Load the tokenizer + tokenizer = AutoTokenizer.from_pretrained("gpt2") # Create the data Module - datamodule = PrunaDataModule.from_string('LAION256') + datamodule = PrunaDataModule.from_string( + dataset_name='WikiText', + tokenizer=tokenizer, + collate_fn="text_generation_collate", + collate_fn_args={"max_seq_len": 512}, + dataloader_args={"batch_size": 16, "num_workers": 4} + ) .. tab:: From Datasets @@ -196,7 +181,7 @@ The ``Task`` accepts ``PrunaDataModule`` in three ways: from transformers import AutoTokenizer from datasets import load_dataset - # Load a built-in dataset + # Load the tokenizer tokenizer = AutoTokenizer.from_pretrained("gpt2") # Load custom datasets @@ -212,7 +197,86 @@ The ``Task`` accepts ``PrunaDataModule`` in three ways: dataloader_args={"batch_size": 16, "num_workers": 4} ) +.. tip:: + + You can find the full list of available datasets in the :doc:`Dataset Overview ` section. + +Lastly, you can limit the number of samples in the dataset by using the ``PrunaDataModule.limit_samples`` method. + +.. code-block:: python + + from pruna.data.pruna_datamodule import PrunaDataModule + + # Create the data module + datamodule = PrunaDataModule.from_string('WikiText') + + # Limit all splits to 100 samples + datamodule.limit_datasets(100) + + # Use different limits for each split + datamodule.limit_datasets([500, 100, 200]) # train, val, test + EvaluationAgent ^^^^^^^^^^^^^^^ -The ``EvaluationAgent`` is a class that evaluates the performance of your model. \ No newline at end of file +The ``EvaluationAgent`` is a class that evaluates the performance of your model. + +To evaluate a model with the ``EvaluationAgent``, you need to create a ``Task`` with ``Metrics`` and a ``PrunaDataModule``. +Then, initialize an ``EvaluationAgent`` with that task and call the ``evaluate()`` method with your model. + +We can then chose to evaluate a single model or a pair of models. + +- **Single-Model Evaluation**: each model is evaluated independently, producing metrics that only pertain to that model's performance. The metrics are computed from the model's outputs without reference to any other model. +- **Pairwise Evaluation**: metrics compare the outputs of the current model against the first model evaluated by the agent. The first model's outputs are cached by the EvaluationAgent and used as a reference for subsequent evaluations. + +Let's see how this works in code. + +.. code-block:: python + + import copy + + from diffusers import StableDiffusionPipeline + + from pruna import smash, SmashConfig + from pruna.data.pruna_datamodule import PrunaDataModule + from pruna.evaluation.evaluation_agent import EvaluationAgent + from pruna.evaluation.task import Task + + # Load data and set up smash config + smash_config = SmashConfig() + smash_config['cacher'] = 'deepcache' + + # Load the base model + model_path = "CompVis/stable-diffusion-v1-4" + pipe = StableDiffusionPipeline.from_pretrained(model_path) + + # Smash the model + copy_pipe = copy.deepcopy(pipe) + smashed_pipe = smash(copy_pipe, smash_config) + + # Define the task and the evaluation agent + metrics = ['clip_score', 'psnr'] + task = Task(metrics, datamodule=PrunaDataModule.from_string('LAION256')) + eval_agent = EvaluationAgent(task) + + # Evaluate base model, all models need to be wrapped in a PrunaModel before passing them to the EvaluationAgent + first_results = eval_agent.evaluate(pipe) + print(first_results) + + # Evaluate smashed model + smashed_results = eval_agent.evaluate(smashed_pipe) + print(smashed_results) + + +Best Practices +-------------- + +Start with a small dataset +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When first setting up evaluation, limit the dataset size with ``datamodule.limit_datasets(n)`` to make debugging faster. + +Use pairwise metrics for comparison +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When comparing an optimized model against the baseline, use pairwise metrics to get direct comparison scores. \ No newline at end of file diff --git a/docs/user_manual/save_load.rst b/docs/user_manual/save_load.rst index 01377a60..fb38bc62 100644 --- a/docs/user_manual/save_load.rst +++ b/docs/user_manual/save_load.rst @@ -3,7 +3,79 @@ Save and Load Models This guide provides a quick introduction to saving and loading optimized AI models with |pruna|. -You will learn how to save and load a ``PrunaModel`` after smashing a model using |pruna|. Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. +You will learn how to save and load a ``PrunaModel`` after smashing a model using |pruna|. +Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. + +Basic Save and Load Workflow +---------------------------- + +|pruna| follows a simple workflow for saving and loading optimized models: + +.. mermaid:: + :align: center + + flowchart TB + subgraph LoadFlow["Load Flow"] + direction LR + F["Model Files"] --> G{"Load Method"} + G --> H1["from_pretrained('saved_model/')"] + G --> H2["from_hub('PrunaAI/saved_model')"] + H1 --> I["Pruna Model"] + H2 --> I + end + + subgraph Model["Model Files"] + direction TB + E1["Model Weights"] + E2["Architecture"] + E3["Smash Config"] + E4["Tokenizer/Processor (if present)"] + end + + subgraph SaveFlow["Save Flow"] + direction LR + A["PrunaModel"] --> B{"Save Method"} + B --> C1["save_pretrained('saved_model/')"] + B --> C2["save_to_hub('PrunaAI/saved_model')"] + C1 --> D["Model Files"] + C2 --> D + end + + SaveFlow --- Model + Model --- LoadFlow + + style A fill:#bbf,stroke:#333,stroke-width:2px + style F fill:#f9f,stroke:#333,stroke-width:2px + style G fill:#bbf,stroke:#333,stroke-width:2px + style H1 fill:#bbf,stroke:#333,stroke-width:2px + style H2 fill:#bbf,stroke:#333,stroke-width:2px + style I fill:#bbf,stroke:#333,stroke-width:2px + style B fill:#bbf,stroke:#333,stroke-width:2px + style C1 fill:#bbf,stroke:#333,stroke-width:2px + style C2 fill:#bbf,stroke:#333,stroke-width:2px + style D fill:#f9f,stroke:#333,stroke-width:2px + +Let's see what that looks like in code. + +.. code-block:: python + + from pruna import smash, SmashConfig + from diffusers import StableDiffusionPipeline + + # prepare the base model + base_model = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") + + # Create and smash your model + smash_config = SmashConfig() + smash_config["cacher"] = "deepcache" + smash_config["compiler"] = "diffusers2" + smashed_model = smash(model=base_model, smash_config=smash_config) + + # Save the model + smashed_model.save_pretrained("saved_model/") # or save_to_hub + + # Load the model + loaded_model = PrunaModel.from_pretrained("saved_model/") # or from_hub Saving a ``PrunaModel`` ----------------------- From f02105e62ba2fa6cbc4a6f6579e3f66d2850a623 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Sun, 4 May 2025 16:17:32 +0200 Subject: [PATCH 22/31] docs: add spacing for improved readability in user manual sections on configuration, evaluation, and saving/loading --- docs/user_manual/configure.rst | 1 + docs/user_manual/evaluate.rst | 1 + docs/user_manual/save_load.rst | 1 + 3 files changed, 3 insertions(+) diff --git a/docs/user_manual/configure.rst b/docs/user_manual/configure.rst index 77f601fd..cec19d79 100644 --- a/docs/user_manual/configure.rst +++ b/docs/user_manual/configure.rst @@ -6,6 +6,7 @@ This guide provides an introduction to configuring model optimization strategies Model optimization configuration relies on the ``SmashConfig`` class. The ``SmashConfig`` class provides a flexible dictionary-like interface for configuring model optimization strategies. It manages algorithms, hyperparameters, and additional components like tokenizers, processors and datasets. + Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. Basic Configuration Workflow diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst index feb4da70..cc77cb9c 100644 --- a/docs/user_manual/evaluate.rst +++ b/docs/user_manual/evaluate.rst @@ -5,6 +5,7 @@ This guide provides an introduction to evaluating model optimizations with |prun Evaluation helps you understand how compression affects your models across different dimensions - from output quality to resource requirements. This knowledge is essential for making informed decisions about which compression techniques work best for your specific needs. + Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. Basic Evaluation Workflow diff --git a/docs/user_manual/save_load.rst b/docs/user_manual/save_load.rst index fb38bc62..8bfd0124 100644 --- a/docs/user_manual/save_load.rst +++ b/docs/user_manual/save_load.rst @@ -4,6 +4,7 @@ Save and Load Models This guide provides a quick introduction to saving and loading optimized AI models with |pruna|. You will learn how to save and load a ``PrunaModel`` after smashing a model using |pruna|. + Haven't smashed a model yet? Check out the :doc:`optimize guide ` to learn how to do that. Basic Save and Load Workflow From 5ef34813cb4a737efa33354d8316258056b112df Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Tue, 6 May 2025 14:09:13 +0200 Subject: [PATCH 23/31] refactor: streamline test parameterization for code blocks - Updated test cases in `test_code_blocks.py` to dynamically generate parameters for user manuals and contributions, excluding the 'dataset' tutorial. - Improved maintainability by leveraging glob patterns to include all relevant `.rst` files. --- tests/documentation/test_code_blocks.py | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/tests/documentation/test_code_blocks.py b/tests/documentation/test_code_blocks.py index f0469bb9..1108112c 100644 --- a/tests/documentation/test_code_blocks.py +++ b/tests/documentation/test_code_blocks.py @@ -12,16 +12,14 @@ @pytest.mark.parametrize( "rst_name", [ - pytest.param("user_manual/smash_config", marks=pytest.mark.cuda), - pytest.param("user_manual/smash", marks=pytest.mark.cuda), - pytest.param("user_manual/dataset", marks=(pytest.mark.cpu, pytest.mark.high)), - pytest.param("user_manual/telemetry", marks=pytest.mark.cpu), - pytest.param("user_manual/save_load", marks=pytest.mark.cuda), - pytest.param("contributions/adding_dataset", marks=pytest.mark.cuda), - pytest.param("contributions/adding_algorithm", marks=pytest.mark.cuda), - pytest.param("contributions/adding_metric", marks=pytest.mark.cuda), - pytest.param("user_manual/evaluation", marks=pytest.mark.cuda), - ], + pytest.param(path.stem, marks=pytest.mark.cuda) + for path in TUTORIAL_PATH.glob("user_manual/*.rst") + if path.stem not in ["dataset"] + ] + + [ + pytest.param(path.stem, marks=pytest.mark.cuda) + for path in TUTORIAL_PATH.glob("contributions/*.rst") + ] ) def test_codeblocks_cuda(rst_name: str) -> None: """Test to ensure the notebook runs without errors.""" From f7893a81b5d56b4256a37c1886b74d26b63daac3 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Tue, 6 May 2025 20:13:42 +0200 Subject: [PATCH 24/31] docs: enhance user manual with new sections on customizing algorithms, datasets, metrics, and model optimization - Added detailed guides for adding custom algorithms, datasets, and metrics to the user manual. - Introduced a comprehensive tutorial on optimizing models with Pruna, including step-by-step workflows and example use cases. - Updated existing documentation for clarity and consistency across sections. --- docs/contributions/how_to_contribute.rst | 10 +- ...opening_an_issue.rst => open_an_issue.rst} | 8 +- docs/user_manual/configure.rst | 28 +-- docs/user_manual/customize.rst | 18 +- ..._algorithm.rst => customize_algorithm.rst} | 0 ...ding_dataset.rst => customize_dataset.rst} | 0 ...adding_metric.rst => customize_metric.rst} | 32 ++- docs/user_manual/evaluate.rst | 185 ++++++++++++++---- docs/user_manual/save_load.rst | 8 +- docs/user_manual/{optimize.rst => smash.rst} | 19 +- 10 files changed, 209 insertions(+), 99 deletions(-) rename docs/contributions/{opening_an_issue.rst => open_an_issue.rst} (98%) rename docs/user_manual/{adding_algorithm.rst => customize_algorithm.rst} (100%) rename docs/user_manual/{adding_dataset.rst => customize_dataset.rst} (100%) rename docs/user_manual/{adding_metric.rst => customize_metric.rst} (97%) rename docs/user_manual/{optimize.rst => smash.rst} (93%) diff --git a/docs/contributions/how_to_contribute.rst b/docs/contributions/how_to_contribute.rst index 4a51f8da..e402b22d 100644 --- a/docs/contributions/how_to_contribute.rst +++ b/docs/contributions/how_to_contribute.rst @@ -1,4 +1,4 @@ -How to Contribute πŸ’œ +How to Contribute ? πŸ’œ ==================== Since you landed on this part of the documentation, we want to first of all say thank you! πŸ’œ @@ -112,10 +112,4 @@ If you want to run only the tests with a specific marker, e.g. fast CPU tests, y Once you have made your changes and tested them, you can create a Pull Request. We will then review your Pull Request and get back to you as soon as possible. -If there are any questions along the way, please do not hesitate to reach out on `Discord `_. - - - - - - +If there are any questions along the way, please do not hesitate to reach out on `Discord `_. \ No newline at end of file diff --git a/docs/contributions/opening_an_issue.rst b/docs/contributions/open_an_issue.rst similarity index 98% rename from docs/contributions/opening_an_issue.rst rename to docs/contributions/open_an_issue.rst index acda6b8f..166e75b7 100644 --- a/docs/contributions/opening_an_issue.rst +++ b/docs/contributions/open_an_issue.rst @@ -1,11 +1,10 @@ -Opening an Issue -=============================== - +Open an Issue +============= .. _documentation-improvement: Documentation Improvement ------------------------- +------------------------- All bits help! We appreciate your interest in improving |pruna|’s documentation. @@ -54,6 +53,7 @@ When opening a bug report on GitHub, you will encounter the following template t Feature Request --------------- + We appreciate your interest in improving |pruna|! Feature requests help shape the project, and we welcome ideas that align with our mission. Before submitting your feature request, consider the following points to ensure your request is clear and actionable: diff --git a/docs/user_manual/configure.rst b/docs/user_manual/configure.rst index cec19d79..a88c9fba 100644 --- a/docs/user_manual/configure.rst +++ b/docs/user_manual/configure.rst @@ -54,8 +54,10 @@ Let's see what that looks like in code. smash_config['ifw_weight_bits'] = 16 smash_config['ifw_group_size'] = 4 - # Add a tokenizer - smash_config.add_tokenizer('bert-base-uncased') + # Add a tokenizer and processor + model_id = 'openai/whisper-large-v3' + smash_config.add_tokenizer(model_id) + smash_config.add_processor(model_id) Configure Algorithms -------------------- @@ -72,22 +74,22 @@ The table underneath provides a general overview of the impact of each algorithm - Description - Speed - Memory - - Accuracy + - Quality * - ``batcher`` - Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing processing time. - βœ… - ❌ - - ～ + - - * - ``cacher`` - Stores intermediate results of computations to speed up subsequent operations. - βœ… - - ～ - - ～ + - - + - - * - ``compiler`` - Optimises the model with instructions for specific hardware. - βœ… - βž– - - ～ + - - * - ``distiller`` - Trains a smaller, simpler model to mimic a larger, more complex model. - βœ… @@ -105,11 +107,11 @@ The table underneath provides a general overview of the impact of each algorithm - ❌ * - ``recoverer`` - Restores the performance of a model after compression. - - ～ - - ～ + - - + - - - βœ… -βœ…(improves), βž–(stays the same), ～(could worsen), ❌(worsens) +βœ…(improves), βž–(stays the same), -(Approx. the same), ❌(worsens) .. tip:: @@ -278,7 +280,7 @@ Additionallly, you can create a fully custom ``PrunaDataModule`` use it in your # Add a built-in dataset using a string identifier smash_config.add_dataset('WikiText') - .. tab:: Custom Collate Function + .. tab:: Custom Dataset Use a custom collate function to use a custom dataset as ``(train, val, test)`` tuples. @@ -306,8 +308,8 @@ Additionallly, you can create a fully custom ``PrunaDataModule`` use it in your .. tab:: PrunaDataModule - You can also create a fully custom ``PrunaDataModule`` use it in your workflow. - This process is more flexible but also more complex. It allows for more control over the dataset and the data loading process. + You can also create a ``PrunaDataModule`` use it in your workflow. + This process is more flexible but also more complex and need to adhere to certain configuration limitations. The process for defining a ``PrunaDataModule`` is highlighted in the :doc:`Evaluation ` page but a basic example of adding it to the ``SmashConfig`` is shown below. .. code-block:: python diff --git a/docs/user_manual/customize.rst b/docs/user_manual/customize.rst index 336872eb..0533d390 100644 --- a/docs/user_manual/customize.rst +++ b/docs/user_manual/customize.rst @@ -5,21 +5,21 @@ Customize components .. grid:: 1 3 3 3 - .. grid-item-card:: Add an algorithm + .. grid-item-card:: Customize an algorithm :text-align: center - :link: ./adding_algorithm.rst + :link: ./customize_algorithm.rst Steps to integrate a new compression algorithm, making it available in the ``SmashConfig``. - .. grid-item-card:: Add a dataset + .. grid-item-card:: Customize a dataset :text-align: center - :link: ./adding_dataset.rst + :link: ./customize_dataset.rst Steps to integrate a new dataset, making it available in the ``SmashConfig``. - .. grid-item-card:: Add a metric + .. grid-item-card:: Customize a metric :text-align: center - :link: ./adding_metric.rst + :link: ./customize_metric.rst Steps to integrate a new metric, making it available in the ``SmashConfig``. @@ -34,6 +34,6 @@ Customize components :caption: Customize components :hidden: - adding_algorithm - adding_dataset - adding_metric \ No newline at end of file + customize_algorithm + customize_dataset + customize_metric \ No newline at end of file diff --git a/docs/user_manual/adding_algorithm.rst b/docs/user_manual/customize_algorithm.rst similarity index 100% rename from docs/user_manual/adding_algorithm.rst rename to docs/user_manual/customize_algorithm.rst diff --git a/docs/user_manual/adding_dataset.rst b/docs/user_manual/customize_dataset.rst similarity index 100% rename from docs/user_manual/adding_dataset.rst rename to docs/user_manual/customize_dataset.rst diff --git a/docs/user_manual/adding_metric.rst b/docs/user_manual/customize_metric.rst similarity index 97% rename from docs/user_manual/adding_metric.rst rename to docs/user_manual/customize_metric.rst index 34339c20..0984ae4b 100644 --- a/docs/user_manual/adding_metric.rst +++ b/docs/user_manual/customize_metric.rst @@ -12,20 +12,20 @@ Understanding Pruna's Metric System |pruna| has two main types of metrics that live under ``pruna/evaluation/metrics``: -1. **Base Metrics** - Inherit from ``BaseMetric`` and compute values directly without maintaining state. These metrics usually require isolated inference computation. Examples: ``GPUMemoryMetric``, ``ElapsedTimeMetric``. -2. **Stateful Metrics** - Inherit from ``StatefulMetric`` and maintain internal state across multiple computations. State here refers to the information that is accumulated across multiple batches. Examples: all metrics under ``TorchMetricWrapper`` like ``Accuracy``, ``CLIPScore``. +1. **Base Metrics** - Inherit from ``BaseMetric`` and compute values directly without maintaining state. These metrics usually require isolated inference computation. Examples: ``GPUMemoryMetric``, ``ElapsedTimeMetric``. +2. **Stateful Metrics** - Inherit from ``StatefulMetric`` and maintain internal state across multiple computations. State here refers to the information that is accumulated across multiple batches. Examples: all metrics under ``TorchMetricWrapper`` like ``Accuracy``, ``CLIPScore``. When adding a new metric to |pruna|, you should place your implementation in ``pruna/evaluation/metrics`` directory to ensure it's properly integrated with the rest of the system. Use snake_case for the file name (e.g., ``your_new_metric.py``). In |pruna|, we evaluate metrics by sharing inference runs across multiple metrics whenever possible. This means that |pruna| runs inference once for all compatible metrics. - + - **Stateful metrics** are preferred for most use cases, especially quality metrics, as they can share inference results across multiple metrics - **Base metrics** are primarily used when isolated inference is required (e.g., for GPU memory metrics where sharing inference would distort results) .. note:: If you are confused about which type of metric to implement, you will likely need to implement stateful metrics. Base metrics are typically only used for specialized performance measurements that require isolated inference. -We use PascalCase for the class names (e.g, ``YourNewMetric``) and NumPy style docstrings for documentation. +We use PascalCase for the class names (e.g, ``YourNewMetric``) and NumPy style docstrings for documentation. Base Metrics ~~~~~~~~~~~~ @@ -34,8 +34,6 @@ Base metrics inherit from the ``BaseMetric`` class and implement the ``compute() |pruna| ``EvaluationAgent`` (`documentation <../user_manual/evaluation.html#evaluationagent>`_) requires all ``BaseMetric`` s to implement the ``compute`` method with two specific parameters: ``model`` and ``dataloader``. Please take note that the ``EvaluationAgent`` does not handle inference for base metrics. You will need to handle inference computations yourself. - - .. code-block:: python from pruna.evaluation.metrics.metric_base import BaseMetric @@ -44,10 +42,10 @@ Base metrics inherit from the ``BaseMetric`` class and implement the ``compute() def __init__(self): super().__init__() # Initialize any parameters your metric needs - + def compute(self, model, dataloader): '''Run inference on the model and compute the metric value.''' - + outputs = run_inference(model, dataloader) result = some_calculation(outputs) return result @@ -93,11 +91,11 @@ Here's a complete example showing all required methods: self.metric_name = "your_metric_name" self.default_call_type = "y_gt" self.call_type = call_type if call_type else self.default_call_type - + # Initialize state variables self.add_state("total", torch.zeros(1)) self.add_state("count", torch.zeros(1)) - + def update(self, inputs, ground_truths, predictions): # Update the state variables based on the current batch # Pass the inputs, ground_truths and predictions and the call_type to the metric_data_processor to get the data in the correct format @@ -105,13 +103,13 @@ Here's a complete example showing all required methods: batch_result = some_calculation(*metric_data) self.total += batch_result self.count += 1 - + def compute(self): # Compute the final metric value using the accumulated state if self.count == 0: return 0 return self.total / self.count - + When to Use Each Type ~~~~~~~~~~~~~~~~~~~~~ @@ -119,12 +117,12 @@ When to Use Each Type - **Use Stateful Metrics when**: Your metric can share inference with other metrics without affecting results (most quality metrics fall into this category) - **Use Basic Metrics when**: Your metric requires isolated inference or would produce incorrect results if inference were shared (e.g., performance metrics like GPU memory usage) -By using stateful metrics whenever possible, |pruna| can efficiently evaluate multiple metrics with just a single inference pass. +By using stateful metrics whenever possible, |pruna| can efficiently evaluate multiple metrics with just a single inference pass. Registering Your Metric ----------------------- -After implementing your metric, you need to register it with Pruna's ``MetricRegistry`` system. +After implementing your metric, you need to register it with Pruna's ``MetricRegistry`` system. The simplest way to do this is with the ``@MetricRegistry.register`` decorator: @@ -140,7 +138,7 @@ The simplest way to do this is with the ``@MetricRegistry.register`` decorator: self.param1 = param1 self.param2 = param2 self.metric_name = "your_metric_name" - + Thanks to this registry system, everyone using |pruna| can now refer to your metric by name without having to create instances directly! One important thing: the registration happens when your module is imported. To ensure your metric is always available, we suggest importing it in ``pruna/evaluation/metrics/__init__.py`` file. @@ -194,7 +192,7 @@ Once you've implemented your metric, everyone can use it in Pruna's evaluation p metrics = [ 'clip_score', - 'your_new_metric_name' + 'your_new_metric_name' ] data_module = PrunaDataModule.from_string('LAION256') @@ -206,6 +204,6 @@ Once you've implemented your metric, everyone can use it in Pruna's evaluation p results = eval_agent.evaluate(model) - + diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst index cc77cb9c..befb8538 100644 --- a/docs/user_manual/evaluate.rst +++ b/docs/user_manual/evaluate.rst @@ -1,7 +1,7 @@ -Evaluate optimizations with the Evaluation Agent +Evaluate quality with the Evaluation Agent ================================================ -This guide provides an introduction to evaluating model optimizations with |pruna|. +This guide provides an introduction to evaluating models with |pruna|. Evaluation helps you understand how compression affects your models across different dimensions - from output quality to resource requirements. This knowledge is essential for making informed decisions about which compression techniques work best for your specific needs. @@ -19,19 +19,26 @@ Basic Evaluation Workflow graph LR User -->|creates| Task User -->|creates| EvaluationAgent - Task -->|uses| PrunaDataModule + Task -->|defines| PrunaDataModule Task -->|defines| Metrics + Task -->|uses| PrunaModel Metrics -->|includes| StatefulMetric Metrics -->|includes| StatelessMetric - PrunaDataModule -->|provides data| EvaluationAgent PrunaModel -->|provides predictions| EvaluationAgent EvaluationAgent -->|evaluates| PrunaModel - EvaluationAgent -->|returns| Evaluation_Results + EvaluationAgent -->|returns| C["Evaluation Results"] User -->|configures| EvaluationAgent subgraph Metric_Types - StatefulMetric - StatelessMetric + StatefulMetric + StatelessMetric + end + + subgraph Task_Definition + Task + PrunaDataModule + Metrics + Metric_Types end style User fill:#bbf,stroke:#333,stroke-width:2px @@ -39,7 +46,7 @@ Basic Evaluation Workflow style EvaluationAgent fill:#bbf,stroke:#333,stroke-width:2px style PrunaDataModule fill:#bbf,stroke:#333,stroke-width:2px style PrunaModel fill:#bbf,stroke:#333,stroke-width:2px - style Evaluation_Results fill:#bbf,stroke:#333,stroke-width:2px + style C fill:#bbf,stroke:#333,stroke-width:2px style Metrics fill:#bbf,stroke:#333,stroke-width:2px Let's see what that looks like in code. @@ -82,7 +89,7 @@ Metrics Metrics are the core components that calculate specific performance indicators. There are two main types of metrics: -- **Stateful Metrics**: These metrics compute values directly from inputs without maintaining state across batches. +- **Base Metrics**: These metrics compute values directly from inputs without maintaining state across batches. - **Stateless Metrics**: Metrics that maintain internal state and accumulate information across multiple batches. These are typically used for quality assessment. The ``Task`` accepts ``Metrics`` in three ways: @@ -121,7 +128,7 @@ The ``Task`` accepts ``Metrics`` in three ways: .. tab:: List of Metric Instances - As a list of metric (e.g., ``CMMD()``), which provides more flexibility in configuring the metrics. + As a list of metric instances (e.g., ``CMMD()``), which provides more flexibility in configuring the metrics. .. code-block:: python @@ -130,7 +137,7 @@ The ``Task`` accepts ``Metrics`` in three ways: from pruna.evaluation.metrics import CMMD, TorchMetricWrapper task = Task( - metrics=[CMMD(), TorchMetricWrapper(metric_name="accuracy")], + metrics=[CMMD(call_type="pairwise"), TorchMetricWrapper(metric_name="accuracy")], datamodule=PrunaDataModule.from_string('LAION256'), device="cpu" ) @@ -139,6 +146,69 @@ The ``Task`` accepts ``Metrics`` in three ways: You can find the full list of available metrics in the :ref:`Metric Overview ` section. +Metric Call Types +^^^^^^^^^^^^^^^^ + +|pruna| metrics can operate in both single-model and pairwise modes. + +- **Single-Model mode**: Each evaluation produces independent scores for the model being evaluated. +- **Pairwise mode**: Metrics compare a subsequent model against the first model evaluated by the agent and produce a single comparison score. + +Underneath the hood, the ``StatefulMetric`` class uses the ``call_type`` parameter to determine the order of the inputs. + +The following table shows the different call types supported by |pruna| metrics and the metrics that support each call type. + +.. list-table:: + :widths: 10 60 10 + :header-rows: 1 + + * - Call Type + - Description + - Example Metrics + + * - ``y_gt`` + - Model's output first, then ground truth + - ``fid``, ``cmmd``, ``accuracy``, ``recall``, ``precision`` + + * - ``gt_y`` + - Ground truth first, then model's output + - ``fid``, ``cmmd``, ``accuracy``, ``recall``, ``precision`` + + * - ``x_gt`` + - Input data first, then ground truth + - ``clip_score`` + + * - ``gt_x`` + - Ground truth first, then input data + - ``clip_score`` + + * - ``pairwise_y_gt`` + - Base model's output first, then subsequent model's output + - ``psnr``, ``ssim``, ``lpips``, ``cmmd`` + + * - ``pairwise_gt_y`` + - Subsequent model's output first, then base model's output + - ``psnr``, ``ssim``, ``lpips``, ``cmmd`` + +Each metric has a default ``call_type`` but you can switch the mode of the metric despite your default ``call_type``. + +.. tabs:: + + .. tab:: Single-Model mode + + .. code-block:: python + + from pruna.evaluation.metrics import CMMD + + metric = CMMD() # or ["cmmd"] + + .. tab:: Pairwise mode + + .. code-block:: python + + from pruna.evaluation.metrics import CMMD + metric = CMMD(call_type="pairwise") + PrunaDataModule ~~~~~~~~~~~~~~~ @@ -227,47 +297,84 @@ Then, initialize an ``EvaluationAgent`` with that task and call the ``evaluate() We can then chose to evaluate a single model or a pair of models. -- **Single-Model Evaluation**: each model is evaluated independently, producing metrics that only pertain to that model's performance. The metrics are computed from the model's outputs without reference to any other model. -- **Pairwise Evaluation**: metrics compare the outputs of the current model against the first model evaluated by the agent. The first model's outputs are cached by the EvaluationAgent and used as a reference for subsequent evaluations. +- **Single-Model mode**: each model is evaluated independently, producing metrics that only pertain to that model's performance. The metrics are computed from the model's outputs without reference to any other model. +- **Pairwise mode**: metrics compare the outputs of the current model against the first model evaluated by the agent. The first model's outputs are cached by the EvaluationAgent and used as a reference for subsequent evaluations. Let's see how this works in code. -.. code-block:: python +.. tabs:: - import copy + .. tab:: Single-Model Evaluation - from diffusers import StableDiffusionPipeline + .. code-block:: python - from pruna import smash, SmashConfig - from pruna.data.pruna_datamodule import PrunaDataModule - from pruna.evaluation.evaluation_agent import EvaluationAgent - from pruna.evaluation.task import Task + import copy - # Load data and set up smash config - smash_config = SmashConfig() - smash_config['cacher'] = 'deepcache' + from diffusers import StableDiffusionPipeline - # Load the base model - model_path = "CompVis/stable-diffusion-v1-4" - pipe = StableDiffusionPipeline.from_pretrained(model_path) + from pruna import smash, SmashConfig + from pruna.data.pruna_datamodule import PrunaDataModule + from pruna.evaluation.evaluation_agent import EvaluationAgent + from pruna.evaluation.task import Task + from pruna.evaluation.metrics import CMMD + # Load data and set up smash config + smash_config = SmashConfig() + smash_config['cacher'] = 'deepcache' - # Smash the model - copy_pipe = copy.deepcopy(pipe) - smashed_pipe = smash(copy_pipe, smash_config) + # Load the base model + model_path = "CompVis/stable-diffusion-v1-4" + pipe = StableDiffusionPipeline.from_pretrained(model_path) - # Define the task and the evaluation agent - metrics = ['clip_score', 'psnr'] - task = Task(metrics, datamodule=PrunaDataModule.from_string('LAION256')) - eval_agent = EvaluationAgent(task) + # Smash the model + copy_pipe = copy.deepcopy(pipe) + smashed_pipe = smash(copy_pipe, smash_config) + + # Define the task and the evaluation agent + metrics = [CMMD()] + task = Task(metrics, datamodule=PrunaDataModule.from_string('LAION256')) + eval_agent = EvaluationAgent(task) + + # Evaluate base model, all models need to be wrapped in a PrunaModel before passing them to the EvaluationAgent + first_results = eval_agent.evaluate(pipe) + print(first_results) - # Evaluate base model, all models need to be wrapped in a PrunaModel before passing them to the EvaluationAgent - first_results = eval_agent.evaluate(pipe) - print(first_results) + .. tab:: Pairwise Evaluation - # Evaluate smashed model - smashed_results = eval_agent.evaluate(smashed_pipe) - print(smashed_results) + .. code-block:: python + + import copy + + from diffusers import StableDiffusionPipeline + from pruna import smash, SmashConfig + from pruna.data.pruna_datamodule import PrunaDataModule + from pruna.evaluation.evaluation_agent import EvaluationAgent + from pruna.evaluation.task import Task + from pruna.evaluation.metrics import CMMD + # Load data and set up smash config + smash_config = SmashConfig() + smash_config['cacher'] = 'deepcache' + + # Load the base model + model_path = "CompVis/stable-diffusion-v1-4" + pipe = StableDiffusionPipeline.from_pretrained(model_path) + + # Smash the model + copy_pipe = copy.deepcopy(pipe) + smashed_pipe = smash(copy_pipe, smash_config) + + # Define the task and the evaluation agent + metrics = [CMMD(call_type="pairwise")] + task = Task(metrics, datamodule=PrunaDataModule.from_string('LAION256')) + eval_agent = EvaluationAgent(task) + + # Evaluate base model, all models need to be wrapped in a PrunaModel before passing them to the EvaluationAgent + first_results = eval_agent.evaluate(pipe) + print(first_results) + + # Evaluate smashed model + smashed_results = eval_agent.evaluate(smashed_pipe) + print(smashed_results) Best Practices -------------- diff --git a/docs/user_manual/save_load.rst b/docs/user_manual/save_load.rst index 8bfd0124..988d79d7 100644 --- a/docs/user_manual/save_load.rst +++ b/docs/user_manual/save_load.rst @@ -27,10 +27,10 @@ Basic Save and Load Workflow subgraph Model["Model Files"] direction TB - E1["Model Weights"] - E2["Architecture"] - E3["Smash Config"] - E4["Tokenizer/Processor (if present)"] + E1["Model Weights (.safetensors)"] + E2["Architecture (.json)"] + E3["Smash Config (.json)"] + E4["Tokenizer/Processor (original directory)"] end subgraph SaveFlow["Save Flow"] diff --git a/docs/user_manual/optimize.rst b/docs/user_manual/smash.rst similarity index 93% rename from docs/user_manual/optimize.rst rename to docs/user_manual/smash.rst index c8b8740d..623b06db 100644 --- a/docs/user_manual/optimize.rst +++ b/docs/user_manual/smash.rst @@ -1,5 +1,5 @@ -Optimize your first model -========================= +Smash your first model +====================== This guide provides a quick introduction to optimizing AI models with |pruna|. @@ -16,7 +16,7 @@ Basic Optimization Workflow graph LR A[Load Model] --> B[Define SmashConfig] - B --> C[Optimize Model] + B --> C[Smash Model] C --> D[Evaluate Model] D --> E[Run Inference] style A fill:#bbf,stroke:#333,stroke-width:2px @@ -42,7 +42,7 @@ Let's see what that looks like in code. smash_config = SmashConfig() smash_config["cacher"] = "deepcache" - # Optimize the model + # Smash the model optimized_model = smash(model=model, smash_config=smash_config) # Evaluate the model @@ -52,9 +52,14 @@ Let's see what that looks like in code. eval_agent.evaluate(optimized_model) # Run inference + optimized_model.set_progress_bar_config(disable=True) + optimized_model.inference_handler.model_args.update( + {"num_inference_steps": 1, "guidance_scale": 0.0} + ) optimized_model("A serene landscape with mountains").images[0] -Step-by-Step Optimization Workflow + +Step-by-Step Optimisation Workflow ---------------------------------- Step 1: Load a pretrained model @@ -127,6 +132,10 @@ To run inference with the optimized model, we can use the same interface as the .. code-block:: python + optimized_model.set_progress_bar_config(disable=True) + optimized_model.inference_handler.model_args.update( + {"num_inference_steps": 1, "guidance_scale": 0.0} + ) optimized_model("A serene landscape with mountains").images[0] Example use cases From d4b09ec98b2863a4b50ea4b499c2f4195358b8f9 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Tue, 6 May 2025 22:49:55 +0200 Subject: [PATCH 25/31] docs: update algorithm impact table in user manual for clarity and consistency MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replaced empty entries with a standardized symbol (βž–) to indicate algorithms that have an approximate same impact. - Adjusted the legend to reflect the updated symbols for improved understanding. --- docs/user_manual/configure.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/user_manual/configure.rst b/docs/user_manual/configure.rst index a88c9fba..7406d0af 100644 --- a/docs/user_manual/configure.rst +++ b/docs/user_manual/configure.rst @@ -79,17 +79,17 @@ The table underneath provides a general overview of the impact of each algorithm - Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing processing time. - βœ… - ❌ - - - + - βž– * - ``cacher`` - Stores intermediate results of computations to speed up subsequent operations. - βœ… - - - - - - + - βž– + - βž– * - ``compiler`` - Optimises the model with instructions for specific hardware. - βœ… - βž– - - - + - βž– * - ``distiller`` - Trains a smaller, simpler model to mimic a larger, more complex model. - βœ… @@ -107,11 +107,11 @@ The table underneath provides a general overview of the impact of each algorithm - ❌ * - ``recoverer`` - Restores the performance of a model after compression. - - - - - - + - βž– + - βž– - βœ… -βœ…(improves), βž–(stays the same), -(Approx. the same), ❌(worsens) +βœ…(improves), βž–(Approx. the same), ❌(worsens) .. tip:: From 42d9fab5344ba8e2e2d7ecaff77e771d0da8bcbe Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Thu, 8 May 2025 10:29:47 +0200 Subject: [PATCH 26/31] docs: update README for clarity and accuracy in algorithm descriptions - Changed references from "compression algorithms" to "optimization algorithms" for consistency. - Updated installation link for pip to direct users to the correct page. - Removed redundant line breaks and improved formatting for better readability. - Enhanced the algorithm overview table with clearer descriptions and standardized symbols. --- README.md | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index a3340636..b153a7fa 100644 --- a/README.md +++ b/README.md @@ -29,17 +29,12 @@ [![Huggingface](https://img.shields.io/badge/Huggingface-models-yellow?style=flat-square)][huggingface] [![Replicate](https://img.shields.io/badge/replicate-black?style=flat-square)][replicate] - - - -
Pruna AI Logo - ## Pruna Cool Introduction Pruna is a model optimization framework built for developers, enabling you to deliver faster, more efficient models with minimal overhead. It provides a comprehensive suite of compression algorithms including [caching](https://docs.pruna.ai/en/stable/compression.html#cachers), [quantization](https://docs.pruna.ai/en/stable/compression.html#quantizers), [pruning](https://docs.pruna.ai/en/stable/compression.html#pruners), [distillation](https://docs.pruna.ai/en/stable/compression.html#distillers) and [compilation](https://docs.pruna.ai/en/stable/compression.html#compilers) techniques to make your models: @@ -68,7 +63,7 @@ Before installing, ensure you have: #### Option 1: Install Pruna using pip -Pruna is available on PyPI, so you can [install it using pip](https://docs.pruna.ai/en/stable/setup/pip.html): +Pruna is available on PyPI, so you can [install it using pip](https://docs.pruna.ai/en/stable/setup/install.html): ```bash pip install pruna @@ -96,7 +91,7 @@ from diffusers import StableDiffusionPipeline base_model = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") ``` -Then, use Pruna's `smash` function to optimize your model. Pruna provides a variety of different compression and optimization algorithms, allowing you to combine different algorithms to get the best possible results. You can customize the optimization process using `SmashConfig`: +Then, use Pruna's `smash` function to optimize your model. Pruna provides a variety of different optimization algorithms, allowing you to combine different algorithms to get the best possible results. You can customize the optimization process using `SmashConfig`: ```python from pruna import smash, SmashConfig @@ -130,7 +125,6 @@ eval_agent.evaluate(smashed_model) This was the minimal example, but you are looking for the maximal example? You can check out our [documentation][documentation] for an overview of all supported [algorithms][docs-algorithms] as well as our tutorials for more use-cases and examples. - ## Pruna Heart Pruna Pro Pruna has everything you need to get started on optimizing your own models. To push the efficiency of your models even further, we offer Pruna Pro. To give you a glimpse of what is possible with Pruna Pro, let us consider three of the most widely used diffusers pipelines and see how much smaller and faster we can make them. In addition to popular open-source algorithms, we use our proprietary Auto Caching algorithm. We compare the fidelity of the compressed models. Fidelity measures the similarity between the images of the compressed models and the images of the original model. @@ -155,20 +149,19 @@ For [HunyuanVideo](https://huggingface.co/tencent/HunyuanVideo), we compare Auto ## Pruna Cool Algorithm Overview -Since Pruna offers a broad range of compression algorithms, the following table provides a high-level overview of all methods available in Pruna. For a detailed description of each algorithm, have a look at our [documentation](https://docs.pruna.ai/en/stable/). +Since Pruna offers a broad range of optimization algorithms, the following table provides a high-level overview of all methods available in Pruna. For a detailed description of each algorithm, have a look at our [documentation](https://docs.pruna.ai/en/stable/). +| Technique | Description | Speed | Memory | Quality | +|--------------|-----------------------------------------------------------------------------------------------|:-----:|:------:|:-------:| +| `batcher` | Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing processing time. | βœ… | ❌ | βž– | +| `cacher` | Stores intermediate results of computations to speed up subsequent operations. | βœ… | βž– | βž– | +| `compiler` | Optimises the model with instructions for specific hardware. | βœ… | βž– | βž– | +| `distiller` | Trains a smaller, simpler model to mimic a larger, more complex model. | βœ… | βœ… | ❌ | +| `quantizer` | Reduces the precision of weights and activations, lowering memory requirements. | βœ… | βœ… | ❌ | +| `pruner` | Removes less important or redundant connections and neurons, resulting in a sparser, more efficient network. | βœ… | βœ… | ❌ | +| `recoverer` | Restores the performance of a model after compression. | βž– | βž– | βœ… | -| Technique | Description | Speed | Memory | Accuracy | -| --- | --- | --- | --- | --- | -| Batching | Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing overall processing time. | βœ… | ❌ | 〰️ | -| Caching | Stores intermediate results of computations to speed up subsequent operations, reducing inference time by reusing previously computed results. | βœ… | 〰️ | 〰️ | -| Compilation | Compilation optimises the model with instructions for specific hardware. | βœ… | βž– | 〰️ | -| Distillation | Trains a smaller, simpler model to mimic a larger, more complex model. | βœ… | βœ… | ❌ | -| Quantization | Reduces the precision of weights and activations, lowering memory requirements. | βœ… | βœ… | ❌ | -| Pruning | Removes less important or redundant connections and neurons, resulting in a sparser, more efficient network. | βœ… | βœ… | ❌ | -| Recovering | Restores the performance of a model after compression. | 〰️ | 〰️ | βœ… | - -βœ…(improves), βž–(stays the same), 〰️(could worsen), ❌(worsens) +βœ… (improves), βž– (approx. the same), ❌ (worsens)

@@ -208,7 +201,6 @@ If you use Pruna in your research, feel free to cite the project! πŸ’œ

Pruna AI Logo

- [discord]: https://discord.gg/Tun8YgzxZ9 [reddit]: https://www.reddit.com/r/PrunaAI/ [x]: https://x.com/PrunaAI From 2089e623c371a8e6296e2d7e23de334416b23823 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Thu, 8 May 2025 16:44:05 +0200 Subject: [PATCH 27/31] docs: update evaluation workflow diagram in user manual for accuracy - Changed the reference from "PrunaModel" to "EvaluationAgent" to accurately reflect the evaluation process. - Ensured consistency in terminology within the evaluation workflow section. --- docs/user_manual/evaluate.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst index befb8538..6fe38e9b 100644 --- a/docs/user_manual/evaluate.rst +++ b/docs/user_manual/evaluate.rst @@ -21,7 +21,7 @@ Basic Evaluation Workflow User -->|creates| EvaluationAgent Task -->|defines| PrunaDataModule Task -->|defines| Metrics - Task -->|uses| PrunaModel + Task -->|uses| EvaluationAgent Metrics -->|includes| StatefulMetric Metrics -->|includes| StatelessMetric PrunaModel -->|provides predictions| EvaluationAgent From 998f9465d4b56ad5f94924c769eed359ecace945 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Thu, 8 May 2025 16:44:42 +0200 Subject: [PATCH 28/31] docs: correct terminology in evaluation metrics section of user manual - Changed "Stateless Metrics" to "Stateful Metrics" for accurate representation of metrics that maintain internal state across batches. --- docs/user_manual/evaluate.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst index 6fe38e9b..35b71d3e 100644 --- a/docs/user_manual/evaluate.rst +++ b/docs/user_manual/evaluate.rst @@ -90,7 +90,7 @@ Metrics Metrics are the core components that calculate specific performance indicators. There are two main types of metrics: - **Base Metrics**: These metrics compute values directly from inputs without maintaining state across batches. -- **Stateless Metrics**: Metrics that maintain internal state and accumulate information across multiple batches. These are typically used for quality assessment. +- **Stateful Metrics**: Metrics that maintain internal state and accumulate information across multiple batches. These are typically used for quality assessment. The ``Task`` accepts ``Metrics`` in three ways: From cef438d89ded9c311a9f4e71bc7eda2f6bc5bf3d Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Mon, 12 May 2025 11:21:40 +0200 Subject: [PATCH 29/31] docs: update user manual to enhance clarity on metric customization and evaluation modes - Renamed section "Adding a Metric" to "Customize a Metric" for improved clarity. - Added details on pairwise evaluation mode options to the evaluation metrics section. --- docs/user_manual/customize_metric.rst | 2 +- docs/user_manual/evaluate.rst | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/user_manual/customize_metric.rst b/docs/user_manual/customize_metric.rst index 0984ae4b..9be15ae8 100644 --- a/docs/user_manual/customize_metric.rst +++ b/docs/user_manual/customize_metric.rst @@ -1,4 +1,4 @@ -Adding a Metric +Customize a Metric =============================== This guide will walk you through the process of adding a new metric to Pruna's evaluation system. diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst index 35b71d3e..c90ca6cf 100644 --- a/docs/user_manual/evaluate.rst +++ b/docs/user_manual/evaluate.rst @@ -182,6 +182,10 @@ The following table shows the different call types supported by |pruna| metrics - Ground truth first, then input data - ``clip_score`` + * - ``pairwise`` + - Pairwise mode to default to ``pairwise_y_gt`` or ``pairwise_gt_y`` + - ``psnr``, ``ssim``, ``lpips``, ``cmmd`` + * - ``pairwise_y_gt`` - Base model's output first, then subsequent model's output - ``psnr``, ``ssim``, ``lpips``, ``cmmd`` From c5b91805c58d7c54ee8932af2fbebbef4714b0bd Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Tue, 13 May 2025 09:38:07 +0200 Subject: [PATCH 30/31] docs: refine user manual for clarity in evaluation workflow and metric terminology - Updated the evaluation workflow diagram to clarify the relationships between components. - Changed terminology from "uses" to "is used by" for better accuracy. - Standardized metric type references and improved subgraph labeling for consistency. --- docs/user_manual/configure.rst | 2 +- docs/user_manual/evaluate.rst | 26 +++++++++++++------------- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/user_manual/configure.rst b/docs/user_manual/configure.rst index 7406d0af..d5e18d30 100644 --- a/docs/user_manual/configure.rst +++ b/docs/user_manual/configure.rst @@ -111,7 +111,7 @@ The table underneath provides a general overview of the impact of each algorithm - βž– - βœ… -βœ…(improves), βž–(Approx. the same), ❌(worsens) +βœ…(improves), βž–(approx. the same), ❌(worsens) .. tip:: diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst index c90ca6cf..0c76b142 100644 --- a/docs/user_manual/evaluate.rst +++ b/docs/user_manual/evaluate.rst @@ -21,24 +21,24 @@ Basic Evaluation Workflow User -->|creates| EvaluationAgent Task -->|defines| PrunaDataModule Task -->|defines| Metrics - Task -->|uses| EvaluationAgent - Metrics -->|includes| StatefulMetric - Metrics -->|includes| StatelessMetric + Task -->|is used by| EvaluationAgent + Metrics -->|includes| B["Base Metrics"] + Metrics -->|includes| C["Stateless Metric"] PrunaModel -->|provides predictions| EvaluationAgent EvaluationAgent -->|evaluates| PrunaModel - EvaluationAgent -->|returns| C["Evaluation Results"] + EvaluationAgent -->|returns| D["Evaluation Results"] User -->|configures| EvaluationAgent - subgraph Metric_Types - StatefulMetric - StatelessMetric + subgraph A["Metric Types"] + B + C end - subgraph Task_Definition - Task - PrunaDataModule - Metrics - Metric_Types + subgraph E["Task Definition"] + Task + PrunaDataModule + Metrics + A end style User fill:#bbf,stroke:#333,stroke-width:2px @@ -46,7 +46,7 @@ Basic Evaluation Workflow style EvaluationAgent fill:#bbf,stroke:#333,stroke-width:2px style PrunaDataModule fill:#bbf,stroke:#333,stroke-width:2px style PrunaModel fill:#bbf,stroke:#333,stroke-width:2px - style C fill:#bbf,stroke:#333,stroke-width:2px + style D fill:#bbf,stroke:#333,stroke-width:2px style Metrics fill:#bbf,stroke:#333,stroke-width:2px Let's see what that looks like in code. From a41ab8249929dd1a377f6a5cd8a51da65e7b42d6 Mon Sep 17 00:00:00 2001 From: davidberenstein1957 Date: Tue, 13 May 2025 09:48:05 +0200 Subject: [PATCH 31/31] docs: enhance evaluation workflow diagram in user manual - Added new styles for components B and C in the evaluation workflow diagram for better visual distinction. - Improved overall clarity of the diagram to aid user understanding of the evaluation process. --- docs/user_manual/evaluate.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/user_manual/evaluate.rst b/docs/user_manual/evaluate.rst index 0c76b142..0cc44a10 100644 --- a/docs/user_manual/evaluate.rst +++ b/docs/user_manual/evaluate.rst @@ -48,6 +48,8 @@ Basic Evaluation Workflow style PrunaModel fill:#bbf,stroke:#333,stroke-width:2px style D fill:#bbf,stroke:#333,stroke-width:2px style Metrics fill:#bbf,stroke:#333,stroke-width:2px + style B fill:#f9f,stroke:#333,stroke-width:2px + style C fill:#f9f,stroke:#333,stroke-width:2px Let's see what that looks like in code.