ChatGPT for Second Life

[elleevelyn]

using GPT-4 (ChatGPT) to debug python scripts

is pretty interesting this. It takes debugging to the next level, as it attempts to fix bugs by making edits to the script, rerunning the code until the bugs have been eliminated. Is early days yet (proof-of-concept stage at this time), but as these kinds of tools advance, they will be quite useful I think for programming/scripting

article here:  https://arstechnica.com/information-technology/2023/04/developer-creates-self-healing-programs-that-fix-themselves-thanks-to-gpt-4/

source code here: https://github.com/biobootloader/wolverine

it may be that somebody will have a go at doing something similar for LSL, probably more for fun than anything else at this time

 

 

[animats]

23 hours ago, animats said:

More re Generative Agents: Interactive Simulacra of Human Behavior research paper.

I've been reading more about how they did this, and thinking about how it might be applied to SL.

What they're doing is built entirely on top of standard GPT 3.5. They just have their own system on top of that. It maintains a memory stream of "observations" about what's happened. For each action, a mini search engine picks recent, important, and relevant items from the "observation" history, and those are used to give GPT 3.5 the background for the next question. A few times a day, there's a "reflection" process, which uses GPT to extract "insights" from the observations, which are then added to the observation stream. So the NPCs summarize their life. There's a planning system which makes coarse plans, and then refines them. Again, GPT is used to do that. The machinery added on top of GPT isn't really that complicated. Amazingly, it sort of works.

It doesn't use "The Sims". It just looks sort of looks like an early version of The Sims. The researchers built their own 2D mini-game using a tool for simple browser games, because that's easy. The connection between the NPCs and the world is in English! They can ask about where they are, what objects are nearby, and what those objects can do. The GPT system is operating in a verbal world that's kind of like paragraph roleplay. Or like a classic dungeon manual, where you have text descriptions of everything. The NPCs talk to users and each other in English. They talk to the world in a way that's a bit more formalized, at the level of "Go to Jim's house".

To build something like that for SL, you'd need to have objects with scripts in them that describe themselves. In English. Like "This is the door to Joe's house. You can open it by touching it." Rather than having one big dungeon description, you'd have lots of little ones. Pick a chat channel number for this, and have objects reply with their info when asked. Mostly they'd just be reading a notecard.

Moving around would need a solid pathfinding system. Something like the lower level of my existing NPC control system, which is reasonably good at getting NPCs from A to B. Where to go and what to do is decided by the GPT system.

That's just the mechanics. The psychology is the complicated part. These characters have an inner dialog, reflections on their life, plans, and goals. All in English. They can get confused, and may have emotional states. "Robopsychologist" is going to be a real job soon.

A busy roleplay town could be built this way. These things have emergent behavior.

This is just a thought exercise for now. The compute load for this is huge, as in needing a sizable compute server or servers outside SL for each NPC. Give it a year, and there will probably be more efficient technology and  open source software for this sort of thing. Early versions will work badly. Expect rapid improvement.

[Love Zhaoying]

3 hours ago, animats said:

I've been reading more about how they did this, and thinking about how it might be applied to SL.

What they're doing is built entirely on top of standard GPT 3.5. They just have their own system on top of that. It maintains a memory stream of "observations" about what's happened. For each action, a mini search engine picks recent, important, and relevant items from the "observation" history, and those are used to give GPT 3.5 the background for the next question. A few times a day, there's a "reflection" process, which uses GPT to extract "insights" from the observations, which are then added to the observation stream. So the NPCs summarize their life. There's a planning system which makes coarse plans, and then refines them. Again, GPT is used to do that. The machinery added on top of GPT isn't really that complicated. Amazingly, it sort of works.

It doesn't use "The Sims". It just looks sort of looks like an early version of The Sims. The researchers built their own 2D mini-game using a tool for simple browser games, because that's easy. The connection between the NPCs and the world is in English! They can ask about where they are, what objects are nearby, and what those objects can do. The GPT system is operating in a verbal world that's kind of like paragraph roleplay. Or like a classic dungeon manual, where you have text descriptions of everything. The NPCs talk to users and each other in English. They talk to the world in a way that's a bit more formalized, at the level of "Go to Jim's house".

To build something like that for SL, you'd need to have objects with scripts in them that describe themselves. In English. Like "This is the door to Joe's house. You can open it by touching it." Rather than having one big dungeon description, you'd have lots of little ones. Pick a chat channel number for this, and have objects reply with their info when asked. Mostly they'd just be reading a notecard.

Moving around would need a solid pathfinding system. Something like the lower level of my existing NPC control system, which is reasonably good at getting NPCs from A to B. Where to go and what to do is decided by the GPT system.

That's just the mechanics. The psychology is the complicated part. These characters have an inner dialog, reflections on their life, plans, and goals. All in English. They can get confused, and may have emotional states. "Robopsychologist" is going to be a real job soon.

A busy roleplay town could be built this way. These things have emergent behavior.

This is just a thought exercise for now. The compute load for this is huge, as in needing a sizable compute server or servers outside SL for each NPC. Give it a year, and there will probably be more efficient technology and  open source software for this sort of thing. Early versions will work badly. Expect rapid improvement.

I love some of the specifics and use-cases you gave! It matches well with what I've been working towards (scripting / parser-wise). So, hopefully I'll be able to use my system to create dungeons / maps / NPC chats that leverage this, once ChatGPT becomes more widely available.

Thank you for all your research and commentary on this topic.

[musichero]

I'm building what I think are the most advanced NPC/bot systems in SL where I have information flow in both directions (SL <> GPT).  I'm also running a blog focused on SL robotics, which includes ideas, observations and some useful bits of code. Enjoy! Please feel free to ask if you want demonstrations of any of the systems.