Note: This issue was copied from ggml-org#3294
Original Author: @OhadRubin
Original Issue Number: ggml-org#3294
Created: 2023-09-21T06:29:29Z
Request: Nougat OCR Integration
I suggest adding Nougat OCR into llama.cpp to enable the processing of scientific PDF documents.
This can act as a first step towards adding multimodal models to this project!
Implementation:
It seems that Nougat is based on standard transformer architecture (like Bart and Swin Transformer) and most of the work would be on figuring out how to add the image processing.
Let me know what you think!
P.S.: Love this repo! I hope to add my own retrieval-pretrained transformer at some point to this repo.
Note: This issue was copied from ggml-org#3294
Original Author: @OhadRubin
Original Issue Number: ggml-org#3294
Created: 2023-09-21T06:29:29Z
Request: Nougat OCR Integration
I suggest adding Nougat OCR into llama.cpp to enable the processing of scientific PDF documents.
This can act as a first step towards adding multimodal models to this project!
Implementation:
It seems that Nougat is based on standard transformer architecture (like Bart and Swin Transformer) and most of the work would be on figuring out how to add the image processing.
Let me know what you think!
P.S.: Love this repo! I hope to add my own retrieval-pretrained transformer at some point to this repo.