Digital puppets using AR (augmented reality) … some apps, companies, and 14-ish years of trying things … @2020cv_inc @thinkoco @creativecloud #CharacterAnimator @adafruit #circuitplayground
Thanks to our new bluetooth LE-powered, sensor-packed development boards, there’s a lot of neat interactive projects that are now really easy to build and extend. For example, Adafruit has some super-cute digital AR puppets as part of our Bluefruit Playground app, it will come out as an update soon, all you’ll need is a Circuit Playground Express Bluefruit and an iPhone or iPad (listing of all apps here).
It’s all wireless (Bluetooth Low Energy, BLE), there are no wires needed to move the puppet around, tilting the head up and down, opens and closes the mouth. With all the sensors from the Adafruit ecosystem you can connect up all sorts of things to make the puppet do even more.
Circuit Playground Express Bluefruit has a microphone, so the mouth movement and more could be voice activated as well.
The Circuit Playground Express Bluefruit is small, runs off a battery, and packed with sensors. Kids can decorate a glove, or paper bag and have a real-life puppet, and also have a digital AR version on the phone/tablet.
For this demo (video), Sparky the Blue Smoke monster has a green background behind them in the app. This is so the puppet can live in the digital world for live streaming.
All together at Adafruit we have 4+ weekly shows and being able to add a digital puppet for live interaction is some I’d like to do. To do it well, we need it to be easy to set up for production, easy to keep the frequency of publishing, and interactive with the community. Pictured above is the set up for this quickie demo.
You could even add a “sip and puff” sensor (Adafruit MPRLS Ported Pressure Sensor Breakout – 0 to 25 PSI) so when you blow in a tube the digital puppet can expand … digital puppet blowfish, a bird’s feathers can get puffy or shrink! 
Future updates may have different or custom puppets to choose from, use of camera or still image backgrounds, recording, etc. Part of the effort is to teach folks about how sensors work. From visualizing an accelerometer, to learning some Python maybe. Also, we want to make it easy to share stories with puppets on Instagram, TikTok, YouTube etc. Same fun clips, but with puppet friends.
Pictured above are just some of our puppets (physical ones!) for our show “Circuit Playground” which teaches electronics and more.
Theese characters are also in our games, AR apps and more.
The playlist above is our series “Ladyada’s E is for Electronics”, in each episode the topic is based on the letter, A for Ampere, B for Battery, C for Capacitor … we’re up to T is for Transistor, should be out in a few weeks.
The Circuit Playground video series is based on our coloring book you can buy or download for free to print at home.
The physical puppet video series takes a lot of time, and they’re worth it! We also wanted a way for anyone to use these characters to make their own “shows”, share them, and learn about electronics. All while using real electronics that we make here at Adafruit, a 100% women-owned open-source educational electronics company in NYC!
Over the last decade, I’ve had a few on-and-off attempts to make it easier to have avatars, or digital puppets, or VR, or something for content creation. There are also a few new efforts and apps that are using very modern tech to do this. So, I figured I’d add that in this post as well starting with the most recent, going all the way back to over 14 years ago.
YoPuppet: AR Hand Puppet Show
With near-desktop computing power and great cameras on phones, we’re finally seeing a combination of machine learning, vision tech, and puppetry! (video)
YoPuppet by 2020CV brings virtual hand puppets anywhere you want to go. Easily create your own puppet shows, record and share with friends, or just enjoy with your family at home. Endless possibilities and new puppets added every week. YoPuppet uses advanced technology to capture 22 points on your hand in realtime and map them to the puppet’s face.
I like this one a lot! It launched around June, 2019 and, using machine learning to map the points of a human arm + hand is clever, cute, fun, and people of all skill levels can immediately start making the puppet do things. The interface is self-documenting.
Here’s a bit more from the creator, Hart Woolery. There are also some fun examples on Twitter from the YoPuppet account as well as the tag #YoPuppet. I emailed the developer and there are some updates to the app coming soon, so I am going to use it on some live shows soon!
YoPuppet is free in the app store.
Thinko Animation Studio
Thinko Animation Studio, Brooklyn, NY (video) –
Thinko was founded on Halloween of 2016 by Pasquale D’Silva (software designer + animator) & Jacob Bijani (software + hardware engineer).
We left the agency world, and now we’re building a new type of animation studio. Using our experience crafting modern software, we’re building tools to give our artists and collaborators superpowers unlike any other studio.
We’re down to make some new shows, maybe some tv/internet commercials, do live events, music videos, pitch some stuff to the big studios, and whatever else sounds compelling. Animation is a magical medium, and the possibilities are forever expanding.
…
Mr. Puppet is our in house animation system. It allows us to make animation FAST, by using a hybrid approach of puppet style performance, and “traditional” animation. It is valuable at many stages of production to help artists get their ideas onto the screen instantaneously. It’s a great tool for writing, working live, improvising, sketching core performance, and sculpting shots with layered animation. The hardware and software was made by Thinko.
I emailed Thinko and got an updated bit of text and image too!
Thinko is a new type of animation studio, making shows for the internet generation. The studio is enabled by our proprietary Mr. Puppet animation rig that reduces production time for animated content by orders of magnitude. It allows us to make animation FAST, by using a hybrid approach of puppet style performance, and “traditional” animation. It is valuable at many stages of production to help artists get their ideas onto the screen instantaneously. It’s a great tool for writing, working live, improvising, sketching core performance, and sculpting shots with layered animation.
The hardware and software was developed by Thinko. We’re using a combination of Adafruit’s components, our own custom-made PCBs, and our own software tying it all together.
Thinko debuted Mr. Puppet in December of 2019, there is some press coverage (Cartoon Brew, Digital Arts). Additionally, check out their Twitter account @thinkoco and videos on YouTube. There is also a show they are making called Bird News Now, BNN.
Looks like they’re even using Adafruit hardware (video and pictured above), for their Mr. Puppet? The BNO055 sensor is an easy-to-use inertial measurement unit (IMU).
Adobe Character Animator
Adobe Character Animator is part of the Adobe Creative Cloud, looks like it’s only in the “all” plan that’s around $50/month… (video)
Turn any art into an animated character with your webcam, powered by Adobe Sensei AI. Import art from Illustrator and Photoshop, or start with built-in character templates. Your performance can bring a character to life. Using your webcam and microphone, Character Animator matches your expressions — from syncing lips to tracking eyes — to animate in real time. Control gestures like waving with your keyboard or MIDI device. Movements can be saved as buttons, so they’re easier to remember than key commands.
We use Telestream’s Wirecast for our live shows, we also use restream.io which allows us to stream to all the platforms at once using RMTP from Wirecast. This means we send one stream to one server and have it go to: YouTube, Facebook, Twitch, periscope.tv, mixer, and LinkedIn.
So! If you want to use Adobe Character Animated with Wirecast, you can – you’ll need to use NewTek’s Network Device Interface (NDI) protocol. Download the plug-in. After installing, in Adobe Character Animator go to Preferences, then Live Output. Click/check off enable Mercury transport and then click/check off NewTek NDI output. You’re now able to transmit what’s in Adobe Character Animator over your local network, including Wirecast.
Adobe Character Animator will map your face for motion / movement and it uses the mic for mouth movements, all of these things can be mapped to keyboard commands as well.
Close up of the mapping.
In Wirecast, add a source, go to network, you’ll see the Adobe Character Animator NDI source on the local network.
The character comes in, then add whatever backgrounds, overlays, and other cameras you want.
From here, you can broadcast live or record to disk. I have not edited or made a puppet from scratch in Adobe Character Animator, I used the built in cat for this demo.
Second Life – 2006
14-ish years ago (February 2006). I was trying to get some physical hardware to work with Second Life. The idea was if I could move around in the virtual world, I could get that avatar out in some way eventually. The photos above are part of the set up and from the event in DC (fund raiser for the East DC Fund, Advocates for Justice and Education).
There was a virtual version of MAKE magazine, and I had virtual world(s) interacting with RSS to get news in and out of the real world to virtual. We had an area with projects from MAKE (like the Cigar Box Guitar), and one of the things we tried was to see if it was possible to add sensors to it in real-life and have it play in the virtual world.
The tech to get the avatars out was really slow, so it was not good for any type real world video at the time.
Logitech Quickcam Orbit MP – 2006
A few months later, also about 14 years ago (April 2006), I recorded this video while senior editor at MAKE Magazine, it’s the Logitech Quickcam Orbit MP that, at the time, I thought would really get things moving with being able to make your own digital puppets. However, Logitech stopped making it, they did not allow custom avatars, and all traces of the avatars are off their site. The video has 1 million views.
Apple Sock Puppet – 2005
And in April of 2005 Apple had released “iPod socks” so I had made a couple stop-motion puppet videos with a green iPod sock and Apple headphones since it looked like a frog. The idea was it would talk about Apple and iPods news. Here is Steve Jobs announcing the Apple iPod Socks, they were removed from the Apple store in 2012.
The companion was a little robot made out of recycled Apple iPod packaging. It was the iFrog and iRobot news show.
I could keep going with more past efforts, but think I’m gonna end it here
