Second Life Puppetry

[Nalates Urriah]

We have much of the tech needed for Puppetry to work and be useful in SL. But, the Lab still has to figure out how people will use it in SL. Something they repeatedly fail to do. And will they integrate the various techs to make things work together?

The VR hype era is over. That is settling into niche fields where it provides a "better way". It is well adopted in various types of training. Those usually require some level of eye-hand coordination building. So those using the systems are not sitting at a keyboard and pushing a mouse, which the current SL user base is. Which brings up the question, will the Lab develop for new users coming in that want the Puppetry abilities, which I assume will be small numbers similar to those using VR training. Or... will they look at how to adapt the Puppetry to those of us sitting at a desk?

In 2013 VR was way near its hype peak. Users were considering LEAP Motion as the pathway for controlling hands in SL-VR. Several of us were experimenting with LEAP. I blogged my experience here: My Second Life LEAP Motion Experiment.  (11/2013)@Draxtor Despres was playing with it too. The combination of Puppetry and Leap would probably be popular with many in SL if we could control our avatar and have more precise control of our hands. I think it would be great for photos and machinima.

Note: Draxtor provides lots of video of SL in his YouTube channel. I am sad to see his viewership is down to <100 per video. While our political views are opposites we share a strong interest in SL and he makes excellent videos. I would not be surprised to find that Drax and Strawberry were involved in making the new video on the opening page of the website. Check out his work.

LEAP Motion has progressed and is now UltraLEAP. They are integrating with VR headsets. Over the last 9 years since I last paid attention to it, or we heard about it in SL, the tech has been advancing. So I see Puppetry as a base component of a greater SL system supporting these advanced technologies.

On a side note: LEAP controllers are US$60 and up on eBay. New LEAP Controllers sell for £113 (US$130 - I am fascinated they also are offered on eBay for US$300). The new LEAP Inspire with haptic feedback goes for US$4k. A bit of research explains why the high cost for the haptic feedback.

Since Phillip has been playing in and developing in the virtual space where eye-hand coordination is important in controlling an avatar in VR, I find it hard to think he is developing Puppetry for dancers. It also encourages me to think there is a bigger plan we are not being told about.

Edited September 3, 2022 by Nalates Urriah
grammer - spelling - my butterfingers

[Cinnamon Mistwood]

18 hours ago, Nalates Urriah said:

Since Phillip has been playing in and developing in the virtual space where eye-hand coordination is important in controlling an avatar in VR, I find it hard to think he is developing Puppetry for dancers.

He better be developing this for dancers.

Remember that old joke that if someone told me 100 years ago that everyone in the future would have a device they can carry in their pocket that puts all the knowledge of mankind at their fingertips, but all they use it for is looking at cat pics and insulting people they don't know a thousand miles away, I never would have believed it?

Let's be generous and say 10% of SL users are creators with their specialized skills in 3D modeling or scripting.  They are in this small elite club with lofty goals and imagination as to how their product should be used.

Then there is the other 90% of SL residents.  The consumers.  How are they actually going to use your product?  What will the average SL user want to buy?

It's great to have goals for niche products that can help people,  but a creator has to pay the bills, too.  They better find a way to monetize this or it fails.  They better make it for dancers, too.

[Fluffy Sharkfin]

While it's nice to theorize on the possibilities a feature like this may present I doubt things like working collision bones or IK are going to be part of what LL delivers at the end of its development.

On 9/1/2022 at 2:50 PM, Coffee Pancake said:

ZOOM calls but with avatars

I suspect this is essentially what we're getting, turn on your webcam and you can wave at other avatars and nod or shake your head.

I personally don't see any point in developing it beyond that, it's a neat toy that some people will enjoy and a few will use when making machinima or tutorial videos, if LL want to do something to improve avatar interactions or provide new features for creators I think their efforts would be best spent in other areas.

[Modulated]

According to the last third party viewer meeting, puppetry is still 'very very early alpha' and its target audience right now are devs only pretty much.

[Arielle Popstar]

1 hour ago, Cinnamon Mistwood said:

Then there is the other 90% of SL residents.  The consumers.  How are they actually going to use your product?  What will the average SL user want to buy?

It's great to have goals for niche products that can help people,  but a creator has to pay the bills, too.  They better find a way to monetize this or it fails.  They better make it for dancers, too.

Well as one of that 90%, I see this as being a great feature and have been awaiting this since I came to S/L in 2009 and read that they had been working on puppeteering the year before, but had shelved it for that time. The ability to have my avatar make movements not dependent on a pre made or purchased animation is something I think will really enhance the immersivity of the platform and as such I think quite of few of that 90% will wind up using it as naturally as we walk, and fly today already. I certainly hope this is a feature that will be usable for all of us and not just be something for creators and developers to extract more L$ from the general populace. It is high time the Lab started giving us features that do not require it going through a small segment of the business interests in S/L to convert it into something usable for the rest of us.

[Istelathis]

And thus, the day of puppetry lead to the great coffee catastrophe, a significant number of keyboards were lost in those days.  

 

I will not likely use it, simply because I don't want to get a second camera to extend behind my laptop and I am not very social to begin with, but I do think it is pretty neat.  I remember first using this back in EverQuest 2, where you could make your character's face smile, grimace, whatever really.  It was neat, but not worth the resources at the time - plus as with SL most people's cameras are zoomed out so no one really sees the face anyway - I don't think it would work with the rest of the body for EQ2 though.  

I think for a lot of use, you will first have to face the avatar you are interacting with to get the most out of it, so it becomes a bit of a struggle to line up just at the right time to land your wave (or bird) at the right person.  To do that with your hands on the keyboard and mouse, then raise them to perform whatever action you want is probably going to feel a bit off.

It is technology I believe better used in a VR setting, where you are not reliant upon keyboards or a mouse, where you are already standing and have an area cleared and have no worries of spilling your drink on your keyboard 😜

 

Edit:
I do think that this will be great for people who take photos though.  Setting up the perfect scene and posing just right, this will be fantastic for them.  For naturally occurring socializing, it is probably not going to be worth the hassle a lot of the time.  The only problem I see is trying to get the viewer to take a screenshot while you are posed.  I'm sure there are workarounds though.

 

It would also be great to make animations, if SL offered that feature.  A snapshot of each movement, done on the fly would be wonderful.

Edited September 4, 2022 by Istelathis

[Bree Giffen]

On 8/30/2022 at 8:29 PM, animats said:

People who want an audience. Vtubers. DJs. Strawberry Linden.

I watch a lot of youtube vids and I see people using an icon, or a drawing, or a 2d avatar on screen as they talk about things and show the subject matter in the background. If LL can make it so someone can use their SL avatar as a 'talking head' in videos I think there would be a group that would use it and that group has a large youtube audience. 

[Quistess Alpha]

23 minutes ago, Bree Giffen said:

I watch a lot of youtube vids and I see people using an icon, or a drawing, or a 2d avatar on screen as they talk about things and show the subject matter in the background. If LL can make it so someone can use their SL avatar as a 'talking head' in videos I think there would be a group that would use it and that group has a large youtube audience. 

If you can get your avatar's head to move, it's not a stretch to put a fullbright green screen behind yourself. but agreed that it'd be nice if LL/some SL creator were to make that use-case as easy as possible.

[Fluffy Sharkfin]

It should be a relatively simple process to set up a full-bright background in SL and then set the SL window as a source in OBS so you can overlay your puppeteered avatar onto whatever live video you're capturing/streaming.

It will be interesting to see if any streamers will risk the wrath of Twitch moderation and try using their SL avatar as a cheap alternative to the more expensive custom made Vtuber avatars some streamers currently use and just how seriously Twitch takes that infraction (assuming the feature ends up working well enough to be a viable alternative of course).

[CaithLynnSayes]

I suspect someone at LL found 'Rinions SLKinect2 anm_server' based in Japan that did this, and released it in around 2010. Development ended in early 2015 but you can still get SLKinect2 and if you have a Kinect. More on: https://www.nsl.tuis.ac.jp/xoops/modules/xpwiki/?anm_server

 

 

It's a bit of a shame LL didn't mention that and give Network System Laboratory of TUIS credit for developing this over 10 years ago now. Oh well.

[Paul Hexem]

I was playing Arma 3 the other day and another potential use for the technology occurred to me.

Controls, the way a lot of simulation games use their motion trackers. Even just being able to control the camera by moving my head would be something I'd use a lot in SL.

[Cinos Field]

I for one am super looking forward to this, it's going to make our avatars so much more lively.

I was excited enough that I tried to make the project viewer work with it, though, and that was a one way trip into dependency hell the likes of which I don't even see in my day job. And I work with Javascript!

Maybe I'll manage to temper my excitement until they actually integrate it into the client.

[Seicher Rae]

Well, my technology knowledge limits were hit somewhere on page 1 of this. God you guys are knowledgeable nerds (or really good at faking it, and I sure can't tell). So, in the original blog posted in the OP the statement is made that they can't wait to see how SL creators will use this!

For reasons unknown to me, my first thought was, "ooohbaby! Adult stuff!" (not the actual verbiage cuz can't do that here)

A friend of mine with a PhD in really weird stuff, has studied game modding. There is an actual, academic term (don't remember it) for the time it takes for a modifiable game to come out and the time someone makes "male and female bits" as a mod. It is usually measured in nanoseconds. I'm willing to bet that this will hold true here, too.

[CaithLynnSayes]

So i wanted to put more of my thoughts down o this. (LL, Keep your pants on, i know you hate me criticizing you, but hear me out.)

 

Like someone mentioned here, roughly 90% of the regular Residents would likely not use this. Please educate yourself about your userbase (for once!). You have done similar work like this... Animesh. On November 14th 2018 you've announced the official release of it. That's going on 5 years as of me writing this, so why is it still not possible to change the shape of Animesh like a regular avatar?

I know why you do this, why you've announced this now. You want to chase the Zuckerberg but in the process you forget your userbase. It's not only me moaning about Animesh now, there are a hell of a lot of open but accepted jira's. I think i've recently mentioned here i have one open and accepted for going on 7 years now. And it's an easy fix at that, you just won't do it for some reason. You don't even tell me why when i ask. It's a little sad.

While, as an animation maker myself (hence why i knew about SLKinect2), i appreciate the technology, i don't think it's what SL needs right now. Keep it on the backburner or faff with it on the side. See the project i linked previously. The viewer and server is basically already capable of it so it can't be that big of a thing to spend a lot of resources on (i hope). Just... Please try and listen a little bit more to your active userbase and what they want.

Thank you.

[CaithLynnSayes]

13 minutes ago, Seicher Rae said:

A friend of mine with a PhD in really weird stuff, has studied game modding. There is an actual, academic term (don't remember it) for the time it takes for a modifiable game to come out and the time someone makes "male and female bits" as a mod. It is usually measured in nanoseconds. I'm willing to bet that this will hold true here, too.

Oh you bet. I just spoke of Animesh, guess what sells the best, Animesh female attachments to walk around with and "play".

So i was skeptic about it but if it can get people up and "exercise", i'm all in. i change my opinion, lol

[Da5id Weatherwax]

1 hour ago, Seicher Rae said:

Well, my technology knowledge limits were hit somewhere on page 1 of this. God you guys are knowledgeable nerds (or really good at faking it, and I sure can't tell). So, in the original blog posted in the OP the statement is made that they can't wait to see how SL creators will use this!

For reasons unknown to me, my first thought was, "ooohbaby! Adult stuff!" (not the actual verbiage cuz can't do that here)

A friend of mine with a PhD in really weird stuff, has studied game modding. There is an actual, academic term (don't remember it) for the time it takes for a modifiable game to come out and the time someone makes "male and female bits" as a mod. It is usually measured in nanoseconds. I'm willing to bet that this will hold true here, too.

Probably. I mean, who laid (sorry) the foundation that pretty much all scripted furniture uses these days, that spawned AVSitter and all its ilk? Who made it possible for that to interact with attachments first? Stroker Serpentine, that's who - and we all know what he made

[Love Zhaoying]

1 minute ago, Da5id Weatherwax said:

Probably. I mean, who laid (sorry) the foundation that pretty much all scripted furniture uses these days, that spawned AVSitter and all its ilk? Who made it possible for that to interact with attachments first? Stroker Serpentine, that's who - and we all know what he made

Who put the "bomp" in the "bomp, bomp, bomp"?

[Da5id Weatherwax]

1 minute ago, Love Zhaoying said:

Who put the "bomp" in the "bomp, bomp, bomp"?

LOL - but seriously, it's a known fact that whether you admit it or not, if your new technology makes it possible to make better internet erotica that technology is almost guaranteed to succeed. It's never admitted, of course. There is a Victorian layer of prudery over "what it is proper for a company to be involved in" but.. they all know it when it's coded.

[Love Zhaoying]

15 minutes ago, Da5id Weatherwax said:

LOL - but seriously, it's a known fact that whether you admit it or not, if your new technology makes it possible to make better internet erotica that technology is almost guaranteed to succeed. It's never admitted, of course. There is a Victorian layer of prudery over "what it is proper for a company to be involved in" but.. they all know it when it's coded.

Trying to remember the terminology used for the benefits of the NASA Space Programs, which resulted in everything from Tang to miniaturization of electronics. "Benefits to Humanity" comes up a lot, but that's not the familiar phrase. Google fail! 

 

[Nalates Urriah]

1 hour ago, Seicher Rae said:

Well, my technology knowledge limits were hit somewhere on page 1 of this. God you guys are knowledgeable nerds (or really good at faking it, and I sure can't tell). So, in the original blog posted in the OP the statement is made that they can't wait to see how SL creators will use this!

For reasons unknown to me, my first thought was, "ooohbaby! Adult stuff!" (not the actual verbiage cuz can't do that here)

A friend of mine with a PhD in really weird stuff, has studied game modding. There is an actual, academic term (don't remember it) for the time it takes for a modifiable game to come out and the time someone makes "male and female bits" as a mod. It is usually measured in nanoseconds. I'm willing to bet that this will hold true here, too.

TTP is game dev term for what you are mentioning. Time To P__________

Years ago I came across it and wasn't too surprised to find it is actual jargon in the dev community.

4 minutes ago, Love Zhaoying said:

Trying to remember the terminology used for the benefits of the NASA Space Programs, which resulted in everything from Tang to miniaturization of electronics. "Benefits to Humanity" comes up a lot, but that's not the familiar phrase. Google fail!

Yeah when you are trying to remember the name of something Google can be highly disappointing.

NASA has some pages on their website about nutrition in space and the AS10 vitamins developed for them. I recall they have a name for that branch of study/development... which of course I can't remember.

 

It is always interesting to see what the Lindens decide to work on. I suspect they have a preference for new projects over fixing old ones. Fixing always has more complications because the fixes can't break things already made for the feature. Unless the problem is really making a mess of things... which moves the fix into the "No Choice but Fit It" category even if it breaks things. There has been a few of those.

[Love Zhaoying]

3 minutes ago, Nalates Urriah said:

Yeah when you are trying to remember the name of something Google can be highly disappointing.

Finally found "spin-off", which is closer.

[Da5id Weatherwax]

2 minutes ago, Nalates Urriah said:

I recall they have a name for that branch of study/development... which of course I can't remember.

Of course they do. This is NASA - a branch of  the US government that likes to think of itself as a bastion of forward-looking policies and progress - but is so prudish they sometimes make the Taliban look like libertarians!

Can you imagine the hoops NASA engineers had to go through when talking about plumbing connections in space suits, to try and avoid a document that said "Option 1, we clamp a tube around their d*ck"

[Judas Shuffle]

To get a better understanding of what is being  done here, i stuck my nose in their code a little. They seem to be using a plugin or 2 to get this all working i think - mediapipe

https://google.github.io/mediapipe/

This means they havent spent a year coding this from the ground up and that they are just playing with connecting the tracking stuff to their existing puppetry code.

The code comments seem to be mostly about getting the tracking to play nicely with the sl rig, just the head and arms so far. Keeping tracked movments aligned with the avatar can be like making kittens stand in a row.

The existing sl avatar has i think fourty somthing face bones in it. The face mesh part of media pipe tracks over 200. I would hope they are on with attaching one too the other to see if it can be made to work.

[Love Zhaoying]

Be there or be square!

 

[Henri Beauchamp]

6 hours ago, Love Zhaoying said:

Be there or be square

Sadly, it is held in voice, which is a no-no for a non-English *speaker*. Went there, did not understand a third of it all, so... No time to waste, sorry. 🙄