tblankensmith
patreon
ChatGPT Plugin for TouchDesigner supports GPT-4o + Vision
🕑 Added 2024-05-13 21:11:50 +0000 UTCI'm very excited to launch this new plugin that supports OpenAI's latest GPT-4o model which you can read more about here.
This new plugin update also includes the ability to have GPT process images, which you can easily turn on and off with each new message via the 'Upload Image With Message' toggle. You can attach any TOP to the component and the second you press 'Generate Response' it will snag the current frame from the input TOP and pass it along to GPT along with your message.
Here's a link to my previous tutorial that shows how to get an API key and set up your project (new tutorial coming soon)
Here's another tutorial showing how to combine the plugin with other AI models to create a voice-activated talking 3D character that also follows the viewer's movement.
If you're using the vision feature don't forget to clear your chat history eventually, because it will keep the images encoded as text inside the component, which can take up a decent amount of space over time.
⚠️ Don't forget to include the TDAsyncIO.tox in your same project to allow the component to run⚠️
Comments
Phil Jungschlaeger
Is it possible to refer to an online chat with chatgpt. or export/import a chat into touchdesigenr?
Torin Blankensmith
Just uploaded :) https://www.patreon.com/posts/118027526
Filippo Mirandola
Hi! Thank you for the incredible work. Just a question: would it be possible to use the 4o-mini model instead of the "full" 4o?
Viola Liang
It's working now. I think the issue is probably caused by the td version I used. Thanks
Viola Liang
Thanks for your response! I’m currently working with the OpenAI.170.toe project file.
Torin Blankensmith
Ok sweet, which project file are you using?
Viola Liang
Hi Torin, I’m using macOS 15.1.1 and TouchDesigner version 2023.11220. I’m working with a .toe project file. The other parts of the project are working pretty well. I’d really appreciate your guidance on how to make the whisper component work. Thanks
Torin Blankensmith
What OS are you on, and what version of TouchDesigner? Also are you using a .toe project file, or are you dragging the whisper.tox into your project?
Viola Liang
Great work on this tutorial. Super helpful! But I ran into an issue where the Whisper component doesn't seem to work for me. Every time I hit the Record Audio button, my TD crashes. Any ideas on what might be causing this? Thanks!
Elrashid Ahmed
I have been working on a project where I'm integrating ChatGPT into a custom application. While everything is functioning well, I've encountered a challenge. I'm trying to configure the model to respond in Arabic instead of English, but despite various approaches, the responses still come back in English. Could you please guide me on the best way to ensure that the model consistently responds in Arabic? Any specific parameters or prompt configurations that I should consider would be really helpful.
7337 Productions
I swear every time I'm looking to add a feature you beat me to it and do it better :) Was just looking at adding image context for chat interactions with the new 4o model.
