Repository navigation
how much does better models help writing the final output? #5
Description
I'd like pluggable LLM use
I have it running on my rtx 3090, I guess all the VRAM use is in embedding ? it seems to not use much GPU compute. I put ollama on my 4060ti - so the 3090 was dedicated to NaneSage
I am keen to use openroute, and gemini flash 2 - it's nearly free and very good at writing documents.
I am keen on getting it to even more deep dive. more levels of research / summarizing etc. But I quickly run out of VRAM. any tips ?
Activity
masterFoad commentedon Feb 12, 2025
I am working on adding a generic llm interface soon : openai api support etc
Is that good?
Also will improve the report output
Working on many things, hopefully new updates soon
More deep dive is better, i want to add more subqueries to the subqueries also
sfarrell5123 commentedon Feb 12, 2025
openai api would be great, just leave the base URL open to change, that way I can use openrouter (which allows every LLM on the planet - using the openai interface - so you dont have to recode if someone say wants to use sonnet).
This is an awesome project, I had already drafted up ideas of writing my own, but this seems better than what I might do.
I love your focus on research + embedding, and local GPU usage. very smart.
sfarrell5123 commentedon Feb 12, 2025
I tried to fork and add changes, but the changes ended in your code.
I don't care if you keep them or remove them.