StoryToolkitAI
patreon
Using free local Assistant models via Ollama
🕑 Added 2024-02-29 14:11:47 +0000 UTCWe get asked a lot if there are any free GPT-like models available for the Assistant and the answer was always, "yes, but it's complicated".
Well... thanks to the work done by the people working on Ollama, we finally have a painless solution to run large language models on our own machines. So, we decided to quickly work on a solution that allows StoryToolkitAI to connect to Ollama and make use of its models in the Assistant window.
This means that you can now have an Assistant that is free and, if you were concerned about privacy, you no longer have to send data to any LLM provider on the Internet.
Connecting Ollama's models to StoryToolkitAI
1. Go to ollama.com and download the latest version.
2. Install the Ollama app on your computer.
3. Open terminal (macOS) or command prompt (Windows) and run "ollama run llama2" - this will download the llama2 model on your computer. You only need to do this once and then it becomes available on your machine.
4. Go to your StoryToolkitAI configuration folder (in the tool: File -> Open configuration folder).
5. Create the additional_llm_models.json file if it doesn't exist and add the example code found here.
6. Now restart the tool, go to Preferences -> Assistant, and choose Assistant Provider "ollama" and Assistant Model "llama2"
7. Open the Assistant window and you should see something like "Using llama2 (ollama)" or similar.
That's it! Now simply prompt the assistant and you should get replies from your locally stored llama2 model which should work even if you're unplugged from the internet. If you're interested in a deeper dive into Ollama, this tutorial shows how the works and how to customize it further.
Disclaimer: we're not affiliated with Ollama or any of these model providers in any way, so use them cautiously and responsibly on your own risk.
Experiment and send us your feedback
Although OpenAI's GPT 3.5 and 4 are still performing considerably better than open source models for the Assistant, there are tons of free models available with Ollama (see the list here). For tasks like summarization, research or digging into transcriptions, some are quite decent!
Please let us know how this works on your end!
We'll slowly add more features and also try to figure out which open source models are best for the transcription and story responses that work well with GPT.
One of the next steps is to let GPT Vision (and hopefully the free llava models) "see" our footage and do some advanced analysis during ingest.
Also in this update: Custom Transcription Exports
We know that different production and post-production houses manage transcripts in different formats than just SRT, AVID DS, plain text or whatever is available in the tool, so we added the ability to create custom transcription exports to fit the file format you have in your own workflow.
If you want to add speakers to each line, or use timecodes, or export to a pre-defined text format - like CSV with custom columns, or a custom file format for a software you use in-house, or anything that's plain text - see this for more info on how it works. There, you'll also see an example yaml file that shows all the available variables that you can use on export.
We tried to make this as less technical as possible, so let us know if we can improve the process in any way, or if you want to export other variables, that are not available in the given example.
Use git to update
This update is now available only via git update, but the early standalone release is coming soon for Frequent Users!!
Thank you again for your support!
Comments
vg
ah yes "Starting with version 0.24.1, you can add additional LLM models to the Assistant. For this, you need to add each model in the additional_llm_models.json file in your configuration folder."
vg
should this work on 19.5? i see no difference in Preferences -> Assistant window nor there is any new info in the console after creating additional_llm_models.json file...
Octavian Mot
Almost there!! We've been pushing fixes on the git version, but will come up with a new early standalone in the next days! Since we aimed to change the underlying Python version to 3.11, some changes needed to be made to support the differences before being able to compile it.
Scott Napier
It’s almost been a year, and there has been no update on the project as a whole.
Octavian Mot
We haven't tested it on 19 yet ourselves, but technically there should't be any reason not to work.
Roy
Is StoryToolkitAI also working with Davinci Resolve 19 beta?
