Skip to content

Add GenAI packages #7169

Closed
Closed
Add GenAI packages#7169
Labels
area-GenAIGenerative AI models and APIsenhancementNew feature or request

Description

@LittleLittleCloud
LittleLittleCloud
opened on Jun 6, 2024

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
The GenAI packages will provide torchsharp implementation for a series of popular GenAI models. The goal is to load the same weight from the corresponding python regular model.

The following models will be added in the first wave

MEAI intergration

Fine-tuning #7287

Along with the benchmark

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Activity

  1. LittleLittleCloud
    added
    enhancementNew feature or request
    on Jun 6, 2024
  2. dotnet-policy-service
    added
    untriagedNew issue has not been triaged
    on Jun 6, 2024
  3. luisquintanilla
    added
    Deep Learning
    NLP
    on Aug 27, 2024

  4. lostmsu commented on Sep 14, 2024

    @lostmsu
    lostmsu
    on Sep 14, 2024

    Can you guys publish a preview for Microsoft.ML.GenAI.LLaMA package?

  5. LittleLittleCloud commented on Sep 15, 2024

    @LittleLittleCloud
    LittleLittleCloud
    on Sep 15, 2024
    MemberAuthor

    @lostmsu You should be able to consume it from the daily build below

    Oh, just notice that the GenAI package hasn't been set to IsPackable to true so it's not available on daily build. Will publish a PR to enable the package flag

  6. aforoughi1 commented on Sep 27, 2024

    @aforoughi1
    aforoughi1
    on Sep 27, 2024

    Can you please publish a preview for Microsoft.ML.GenAI.Core package?
    It is not available

    https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-libraries/nuget/v3/index.json

    The sample Microsoft.ML.GenAI.Samples/Llama/LLaMA3_1.cs is broken without it .

    Furthermore, the sample has hard coded weight folder
    var weightFolder = @"C:\Users\xiaoyuz\source\repos\Meta-Llama-3.1-8B-Instruct";
    I have downloaded the model and config from Meta site. May be a few comments will be helpful.

  7. LittleLittleCloud commented on Sep 27, 2024

    @LittleLittleCloud
    LittleLittleCloud
    on Sep 27, 2024
    MemberAuthor

    Can you please publish a preview for Microsoft.ML.GenAI.Core package? It is not available

    https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-libraries/nuget/v3/index.json

    The sample Microsoft.ML.GenAI.Samples/Llama/LLaMA3_1.cs is broken without it .

    Furthermore, the sample has hard coded weight folder var weightFolder = @"C:\Users\xiaoyuz\source\repos\Meta-Llama-3.1-8B-Instruct"; I have downloaded the model and config from Meta site. May be a few comments will be helpful.

    Oh, sorry, I'll make the fix

  8. aforoughi1 commented on Oct 1, 2024

    @aforoughi1
    aforoughi1
    on Oct 1, 2024

    I am getting System.IO.FileNotFoundException
    couldn't find model.safetensors.index.json
    calling at Microsoft.ML.GenAI.LLaMA.LlamaForCausalLM.FromPretrained(String modelFolder, String configName, String checkPointName, ScalarType torchDtype, String device)
    I can't get the example working, please explain where/what this file is?

  9. LittleLittleCloud commented on Oct 1, 2024

    @LittleLittleCloud
    LittleLittleCloud
    on Oct 1, 2024
    MemberAuthor

    @aforoughi1 Which llama, I suppose you are runnning llama 3.2 1B?

  10. aforoughi1 commented on Oct 1, 2024

    @aforoughi1
    aforoughi1
    on Oct 1, 2024

    Llama3.1-8B

  11. LittleLittleCloud commented on Oct 1, 2024

    @LittleLittleCloud
    LittleLittleCloud
    on Oct 1, 2024
    MemberAuthor

    @aforoughi1

    The error basically say it can't find the {ModelFolder}/model.safetensors.index.json, could you share the full code to call the model, stacktrace and a screenshot of the llama 3.1 8B model folder

  12. aforoughi1 commented on Oct 1, 2024

    @aforoughi1
    aforoughi1
    on Oct 1, 2024

    model folder

    // issue 7169
    //Meta-Llama-3.1-8B-Instruct/orginial
    string weightFolder = @"C:\Users\abbas.llama\checkpoints\Llama3.1-8B";
    string configName = "params.json";
    string modelFile = "tokenizer.model";

    TiktokenTokenizer tokenizer = LlamaTokenizerHelper.FromPretrained(weightFolder, modelFile);
    LlamaForCausalLM model = LlamaForCausalLM.FromPretrained(weightFolder, configName, layersOnTargetDevice: -1 ,targetDevice: "cpu");
    Console.WriteLine("Loading Llama from model weight folder");

    var pipeline = new CausalLMPipeline<TiktokenTokenizer, LlamaForCausalLM>(tokenizer, model, "cpu");

    System.IO.FileNotFoundException
    HResult=0x80070002
    Message=Could not find file 'C:\Users\abbas.llama\checkpoints\Llama3.1-8B\model.safetensors.index.json'.
    Source=System.Private.CoreLib
    StackTrace:
    at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
    at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
    at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
    at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
    at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
    at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
    at System.IO.File.InternalReadAllText(String path, Encoding encoding)
    at System.IO.File.ReadAllText(String path)
    at TorchSharp.PyBridge.PyBridgeModuleExtensions.load_checkpoint(Module module, String path, String checkpointName, Boolean strict, IList1 skip, Dictionary2 loadedParameters, Boolean useTqdm)
    at Microsoft.ML.GenAI.LLaMA.LlamaForCausalLM.FromPretrained(String modelFolder, String configName, String checkPointName, ScalarType torchDtype, String device)
    at Test.GenAITest.LLaMATest1() in C:\Users\abbas\OneDrive\Documents\WorkingProgress\MLStcokMarketPrediction\Test\GenAITest.cs:line 35
    at Test.Program.GenAI() in C:\Users\abbas\OneDrive\Documents\WorkingProgress\MLStcokMarketPrediction\Test\Program.cs:line 425
    at Test.Program.Main(String[] args) in C:\Users\abbas\OneDrive\Documents\WorkingProgress\MLStcokMarketPrediction\Test\Program.cs:line 54

  13. LittleLittleCloud commented on Oct 1, 2024

    @LittleLittleCloud
    LittleLittleCloud
    on Oct 1, 2024
    MemberAuthor

    @aforoughi1
    LlamaForCausalLM loads .safetensor model weight while in your code, you are targeting the original .pth model weight folder.

    The .safetensor model weight should be located in Meta-Llama-3.1-8B-Instruct, maybe update the weight folder to that path when loading LlamaForCausalLM?

    LlamaForCausalLM model = LlamaForCausalLM.FromPretrained("Meta-Llama-3.1-8B-Instruct", configName, layersOnTargetDevice: -1 ,targetDevice: "cpu");

  14. aforoughi1 commented on Oct 7, 2024

    @aforoughi1
    aforoughi1
    on Oct 7, 2024

    I sorted the following missing files and the directory structure:
    model.safetensors.index
    model-00004-of-00004.safetensors
    model-00001-of-00004.safetensors
    model-00002-of-00004.safetensors
    model-00003-of-00004.safetensors
    model-00004-of-00004.safetensors

    The model is loaded successfully ONLY if I use the defaults
    layersOnTargetDevice: -1,
    quantizeToInt8: false
    quantizeToInt4 = false

    Setting layersOnTargetDevice: 26, quantizeToInt8: true causes memory corruptions exception.

    The example also missing stopWatch.Stop();

    I also don't see RegisterPrintMessage(), print any messages to the console.

  15. LittleLittleCloud commented on Oct 7, 2024

    @LittleLittleCloud
    LittleLittleCloud
    on Oct 7, 2024
    MemberAuthor

    @aforoughi1 Are you using nightly build or trying the example from main branch

  16. aforoughi1 commented on Oct 7, 2024

    @aforoughi1
    aforoughi1
    on Oct 7, 2024 via email

  17. LittleLittleCloud commented on Oct 7, 2024

    @LittleLittleCloud
    LittleLittleCloud
    on Oct 7, 2024
    MemberAuthor

    @aforoughi1 And your GPU device/platform?

  18. aforoughi1 commented on Oct 7, 2024

    @aforoughi1
    aforoughi1
    on Oct 7, 2024 via email

  19. LittleLittleCloud commented on Oct 7, 2024

    @LittleLittleCloud
    LittleLittleCloud
    on Oct 7, 2024
    MemberAuthor

    The layersOnTargetDevice is for GPU, so I haven't test values other than -1 in CPU scenario. For the quantizeToInt8 and quantizeToInt4, you probably also won't gain benefits on CPU scenarios. So maybe just keep it as false.

  20. rosebyte
    added
    area-GenAIGenerative AI models and APIs
    and removed
    Deep Learning
    NLP
    on Sep 8, 2026

  21. rosebyte commented on Oct 14, 2026

    @rosebyte
    rosebyte
    on Oct 14, 2026
    Member

    Triage: we have discontinued the GenAI packages.

  22. rosebyte
    closed this as completedon Oct 14, 2026
  23. dotnet-policy-service
    removed
    untriagedNew issue has not been triaged
    on Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-GenAIGenerative AI models and APIsenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions