tblankensmith
patreon
Face, Hand, Pose Tracking & More with MediaPipe GPU Plugin for TouchDesigner
🕑 Added 2023-09-09 00:18:36 +0000 UTCHey! In this tutorial, we'll go over the new free open-sourced MediaPipe plugin for TouchDesigner that's GPU accelerated and works on Mac and PC with no instillation. The plugin supports face, hand, pose and object tracking with multi-person face detection, object detection, and image segmentation / background removal.
Link to the GitHub to download the Plugin: https://github.com/torinmb/mediapipe-touchdesigner
Dom Scott's website: https://art.domscott.ca/home
Link to Torin's Patreon: https://patreon.com/tblankensmith
Link to the body tracking alternative: https://www.youtube.com/watch?v=83StND-y4fY
Link to OBS for using video with MediaPipe on Mac: https://obsproject.com/
Link to SpoutCam for using video on PC: https://github.com/leadedge/SpoutCam/releases
Comments
David Henderson
Is there a way to track the entire body? I need to track a body across a stage and if the person turns, then MediaPipe loses the point on the body I was tracking (left shoulder for example.) I need to track the body as an entire object.
HeavenArmy
Or to link to a 3d model e.g. fbx
HeavenArmy
How can I create/rig a 2D character and have mapped to Mediapipe to create a avatar (like the mouse renderer example on Mediapipe's demo)
Akane Hiraoka
This is great! But I'm still new to TD and can't figure out how I connect this to Particle GPU. I want to control particles by using your plugin, is there anyway you can make a tutorial about it in the future?
Runa Renard
Hi,Torin. I used the hand tracking part of the plugin and noticed that TD has some trouble with distinguishing the left and right hand(I mean, I attached different sounds for left and right gestures but while tracking it more likely triggers the sound for the left hand than for the right even if I show the gesture for the right side). What is the way to make side differentiation more accurate?
Desert Fox
Is there a Discord for troubleshooting or is here the best place? I'm having trouble matching the cameras in both MediaPipe and Video Device In. Is that what needs to happen? When I have MP set to Spoutcam and VDI set to my actual webcam, I'm able to play with Image Segmentation. However, when I set them both to Spoutcam, it goes black. I noticed this when trying to refine Image Segmentation as instructed in this video at 23:18. I also get errors that say "webcam failed to start. is it being used somewhere else?". I'm on PC, TD 2023.11880.
Behshad Jalalian
My software version is this(2022.32660) and I need to use this MediaPipe but i cant , please guide me, because I don't want to update it.
Alec Johnson
Thanks Tori! This has been a lifesaver for me. I bought the Orbbec Gemini 2 camera thinking I could do skeleton tracking in TD on Mac M1, and when that did not work, thought I'd bought the wrong depth camera. I've got everything up and running now thanks to your Mediapipe .toe.
Mecha MIO
Thanks! I'm a little stuck on the tutorial because I can't find the way to add the box with the "hand_trancking" (2:53 of the video) and continue with the visual possibilities. Sorry for my amateur question. If you have the way to share me, it would be great! thanks a lot!
Torin Blankensmith
Hey! This is a quick example showing how to work with the hand tracking data: https://www.youtube.com/watch?v=NwqLnNyfOb4
Mercedes Invernizzi Oviedo
Hi Torin. Im new here! I from Argentina, buenos aires. i neet some help to this tutorial. i want to know if you can explain us how take the data of mediapipe to create some visuals with our hands or face, around Interactive Segmentation and Image Embedding. do you have that parte of the video to practice?
Torin Blankensmith
Yeah totally check out the helper component that we use to put a ring above the person's face
Ben Alderman
For the face tracking, can I use a different 3D model, is it possible to use an obj file I made in Blender?
Torin Blankensmith
We're about to release an update that adds a webcam flip 😄
Torin Blankensmith
Hey! I haven't tested it out personally, but in theory it should still work with a black and white texture that you get from an infrared camera
Ben Alderman
This might be a silly question, but I am wondering if this works with depth cameras such as Kinect V2. The reason I am asking is that the installation for trying to build is in the dark. If I use a regular webcam, the visual information won't register.
Torin Blankensmith
glad you got it working :)
Andrea Bucci
Great! But I had the first issue!! https://github.com/torinmb/mediapipe-touchdesigner/issues/35#issue-1906853027
Mini UV
Amazing! Thank you so much!
Adrien Oneiga
this looks amazing ! one question, is there an easy way to implement image flip? my webcam is flipped right now usually i just apply a transform but in this case it doesn't seem like you use a video in TOP. either that, or perhaps just make it so there is a TOP in and we can ever have the option to track people from a media file?
