It really is the number of idle scripts that drags down a sim

[Qie Niangao]

7 minutes ago, Profaitchikenz Haiku said:

Another thing has occurred to me, linking this thread to the one hypothesising fewer servers: If the server has say four regions, and something in Region A is causing the choking of scripts, regions B C and D had request they are restarted until they are blue in the face, the problem will only be cleared when something happens to Region A.

Agreed, something this, too, is likely happening, but I'd guess unrelated to the general decline in Scripts Run percentage. Anomalies such as the "Sleep Time not associated with Spare Time" thing seem (to me) good candidates for such dramatic cross-sim effects.

If we see "sheep and goats" sims, some with good performance characteristics and some with bad independent of their own loading factors, those "goats" seem likely victims of resource exhaust on their host-mate sims.

[Israel Schnute]

I want to try to integrate this into the 25 year plan of my metropolis.  I occupy about a half of two sims (from a public Right of Way to the sim border) on each and notice two things that go on in both.

My question is, do you mean total scripts in total?  Or total types of scripts? 

I do not believe that all scripts are equal, as a animated painting is 16kb, a color changing deer is 16kb, but a 66 prim harpsichord that must be 10 years old will use 2128kb or a train i overly scripted(made all the doors workable) is 3568kb.   In the olden days, I thought larger scripts was what creates the lag.

Therefore, I want to participate by *trying* to cut down the scripts and see what happens.  

My lagged sim (which I only have half of mind you), 17024 objects, 7373 scripts, script time 17ms, spare time 0.01ms;  but  118,671kb memory used on my parcels alone.
Lets add in some other stuff.  Scripts Run: 33-58%; Script Events - 2300-2700eps.

My good sim (which I have slightly less than a half I believe); 10977 objects, 3083 scripts, script time 17ms, spare time fluctuating from 1.2 to 0.01ms; but 118,573kb of memory used on my parcels alone.
Lets add in some other stuff.  Scripts Run: fluctuates between 50-95%;  Script Events - 949-19,000eps. Note I see no correlation between these numbers,  the events can go up and the %age goes down.

Am I supposed to go back to the olden days and delete scripts from objects that I'm not going to be using?    Just in front of me, I see trees, it made life easier to change the seasons when  you can click their scripts than to re-rez all of them.  I got hundreds of light bulbs for illumination and traffic; it would be silly to delete most of them.  I do see some birds that have a color changing lantern menu.  I could gut those at least.  Not sure how many scripts I'll save combined...

[Profaitchikenz Haiku]

42 minutes ago, Israel Schnute said:

Am I supposed to go back to the olden days and delete scripts from objects that I'm not going to be using?

One approach is to have a script that you go around lobbing into trees, etc, that makes a foliage change for the season and then deletes itself. It's extra workload for you but less for the sim. That said, I think it best to wait a few weeks. My take on this issue is that it's like the TP problem, that came from nowhere, blew up big, people said "we don't understand exactly why it's happening" and then once they got a handle on it, they were able to fix it. Cross everything and hope for the best.

[animats]

On 6/4/2019 at 2:37 AM, Qie Niangao said:

Agreed, something this, too, is likely happening, but I'd guess unrelated to the general decline in Scripts Run percentage. Anomalies such as the "Sleep Time not associated with Spare Time" thing seem (to me) good candidates for such dramatic cross-sim effects.

If we see "sheep and goats" sims, some with good performance characteristics and some with bad independent of their own loading factors, those "goats" seem likely victims of resource exhaust on their host-mate sims.

I'm thinking that, too. My home sim, Vallone, was restarted overnight, and performance has improved from 20% of scripts running to 60-90%. Pathfinding went up from 7% to 50%. Still at 4183 scripts.

Too many sims on too few servers?

[Israel Schnute]

4 hours ago, Profaitchikenz Haiku said:

One approach is to have a script that you go around lobbing into trees, etc, that makes a foliage change for the season and then deletes itself. It's extra workload for you but less for the sim. That said, I think it best to wait a few weeks. My take on this issue is that it's like the TP problem, that came from nowhere, blew up big, people said "we don't understand exactly why it's happening" and then once they got a handle on it, they were able to fix it. Cross everything and hope for the best.

Interesting idea versus re-rezzing and copying the position.

However, this issue cropped up either last November or December when I was the only one on the sim.  Now I have neighbors, but I I think the lag started before they moved in.  But, I did have someone else tell me that they've seen problems start at about the same time all over the mainland.  Back then I thought I too many bells and whistles rezzed.

Regardless, I am going to see if I delete at least the scripts in the items that I never ever touch, and forget they even had a menu.   I am curious if it will improve things at all.  I'm getting tired of not being able to drive down certain streets because I can barely turn, or watching a train take 3 minutes for a 30 second ride.  I'll report back in a day or two what happens.

BTW, I also remember when they said that megaprims cause lag...

Edited June 5, 2019 by Israel Schnute
megaprimy

[Wulfie Reanimator]

On 6/5/2019 at 8:27 PM, animats said:

I'm thinking that, too. My home sim, Vallone, was restarted overnight, and performance has improved from 20% of scripts running to 60-90%. Pathfinding went up from 7% to 50%. Still at 4183 scripts.

Too many sims on too few servers?

Another weird thing on a sim called Fluffy:

2222 scripts, 20% +/- 5% run each frame with ~16ms spare time.

[animats]

1 hour ago, Wulfie Reanimator said:

Another weird thing on a sim called Fluffy:

2222 scripts, 20% +/- 5% run each frame with ~16ms spare time.

Homestead or open space sim?  There, "spare time" includes the time the other sims on the same CPU are using.

[Profaitchikenz Haiku]

And that sleep time figure, is that just physics sleeping, or scripts? I spotted that same detail in a region that had 45% script run and no spare time

[Aishagain]

It is clear from that stats panel that Fluffy cannot be an Openspace.  An open space simply could not support that many scripts or that many "agents" (read avatars).  It must, I assume therefore be either a Homestead or a Mainland full region and my money is on it being a Homestead.  Those stats are not too dissimilar to those for my own Homestead, Woods of Heaven, though I am not running as many scripts.  With 18 agents on a Homestead I'd be having difficulty running ANYTHING or even moving.

Edited June 10, 2019 by Aishagain
corrected info

[Qie Niangao]

I teleported in to check it out for sure and yeah, Fluffy is a Homestead. That Sleep Time figure is perfectly normal, and the scripts run percentage is pretty expected these days for a homestead with that many scripts running -- it would be like 9000 active scripts on a full region, so... yeah.

"Sleep Time" generally has nothing to do with Physics. The weird cases are when Sleep Time isn't scored to "Spare Time" but rather to something else: In Vallone's case, for example, Sleep Time and Script Time tracked each other almost perfectly. I don't think I've heard a definitive explanation of what that means, or if "Sleep" can contaminate other time categories, too.

[Profaitchikenz Haiku]

3 hours ago, Qie Niangao said:

I don't think I've heard a definitive explanation of what that means, or if "Sleep" can contaminate other time categories, too.

A question for the meeting tomorrow, perhaps? I'm feeling worried that we are black-box testers with lots of gaps in our knowledge of what figures we are seeing coming out of or going into the box.

[Qie Niangao]

1 minute ago, Profaitchikenz Haiku said:

A question for the meeting tomorrow, perhaps? I'm feeling worried that we are black-box testers with lots of gaps in our knowledge of what figures we are seeing coming out of or going into the box.

I can't attend tomorrow but I know y'all will keep good notes, hehehe.

At one point I noted that weird "Sleep Time" == "Script Time" thing and asked Simon if all that "sleeping" could be scripts blocked on resource availability, specifically waiting on paging. He said that is extremely rare these days. I suspect the developers may not know what it means, either -- nor whether it might be a product of idiosyncrasies in the collection and reporting of the metrics.

Occasionally I think I sense some unease about putting customers in the position of trying to diagnose problems we also had to detect, rather than it being discovered in Linden Operations' own data collection. It's not a great look for them, although I haven't heard them say anything. It's just a vibe I get sometimes.

[Aishagain]

1 hour ago, Qie Niangao said:

Occasionally I think I sense some unease about putting customers in the position of trying to diagnose problems we also had to detect, rather than it being discovered in Linden Operations' own data collection. It's not a great look for them, although I haven't heard them say anything. It's just a vibe I get sometimes.

The problem for them is, Qie, that some of us "older" accounts here have a much longer knowledge of the foibles of SL than they do.  While some of the Senior Lindens are coming up for their long service medals, many lower-echelon Lindens have not been on this duty very long and do not have any in-depth knowledge of the cobbled-together nature of the SL codebase.  So occasionally they have to accept that some of us may actually know more about SL and what makes it "tick" in a particular way than them.  That, as you surmise, does not sit well with them, understandably.

Edited June 10, 2019 by Aishagain

[Qie Niangao]

11 hours ago, Aishagain said:

some of us may actually know more about SL and what makes it "tick" in a particular way than them.

Well, speaking for myself, I certainly don't know more about the innards of SL than Simon or Oz, for example, and other than secret ex-Lindens I doubt any of us "normals" do. My thing here isn't really about Development but rather Operations... and not even that they're at fault for not collecting and tracking detailed historical performance trends that would be so valuable for detecting drift that risks platform viability. I'll bet Operations is the most under-funded, under-staffed part of the Lab, and Ebbe's office probably (naively) views the cloud as a way to strip yet more resources from that group. I got no evidence here, just speculating.

[Aishagain]

Indeed, Qie.

Oz and Simon are far more knowledgeable than us oldtimers but, and it's a big but, they are very few and are less "hands on" than they were.

My point was made when you mentioned the collection of data related to significant performance changes.  The underfunding of such operations is what may well be the source of their discomfiture, especially regarding the apparent lure of the cloud for some.

 

It is a personal source of satisfaction that some are prepared to swallow their pride and give credence to the thoughts of some users.

[Sayrah Parx]

I think it would really help for most of the "prim properties" to be editable in the object editor.  Hover text and sit targets are the major ones that people leave scripts in after being set.  Texture animation in the object editor would also be very helpful.  Particles might be a little too complicated for an editor UI, but there could be a button to clear them at least.

[Wulfie Reanimator]

58 minutes ago, Sayrah Parx said:

I think it would really help for most of the "prim properties" to be editable in the object editor.  Hover text and sit targets are the major ones that people leave scripts in after being set.  Texture animation in the object editor would also be very helpful.  Particles might be a little too complicated for an editor UI, but there could be a button to clear them at least.

This. I wouldn't even care if we got a new UI window called "advanced properties" or something with just a text field or drop list for each prim property. It's so time consuming to write a new script just to get a few properties how you want them.

[arabellajones]

One things about scripts in the old days, which changed, but could still be lingering in old items:

There used to be LSL functions which could run in one prim, and do something to all the prims in a linkset. But that action triggered a 1 second delay, and people would program a control script that sent a signal that was detected by a myriad of scripts, one in each prim. I've worn prim hair which had over 70 prims, and to get an instant colour change they needed over 70 scripts listening for the signal.

Most of those delays have gone. Prim hair still has a few useful feature, such as flexiprims, but the reason for a large script count seems to have vanished.

You'll need somebody with more clue than I have on LSL to report just which bits of LSL had this problem, but it's a problem with old stuff on the marketplace. And if you don't have enough permissions on the scripts and the object there's not much you can do about it.

 

[Wulfie Reanimator]

54 minutes ago, arabellajones said:

One things about scripts in the old days, which changed, but could still be lingering in old items:

There used to be LSL functions which could run in one prim, and do something to all the prims in a linkset. But that action triggered a 1 second delay, and people would program a control script that sent a signal that was detected by a myriad of scripts, one in each prim. I've worn prim hair which had over 70 prims, and to get an instant colour change they needed over 70 scripts listening for the signal.

Most of those delays have gone. Prim hair still has a few useful feature, such as flexiprims, but the reason for a large script count seems to have vanished.

You'll need somebody with more clue than I have on LSL to report just which bits of LSL had this problem, but it's a problem with old stuff on the marketplace. And if you don't have enough permissions on the scripts and the object there's not much you can do about it.

The whole "one script in each prim" was just a result of a wide-spread script product created and used by people with no real knowledge on how to do it better.

Forced delays still exist in all of the functions that had them. llSetTexture and llSetLinkTexture are still being used in mesh products today and the delay is extremely noticeable as relatively few hairs are a single piece of mesh. But note how llSetLinkTexture can have an input of -1 which means WHOLE LINKSET, so there's no need for a script in every link even if you want to use it.

Edited June 11, 2019 by Wulfie Reanimator

[Qie Niangao]

The history as I recall:

Pretty sure there was a time early on when there were no llSetLink* functions, so there was no choice but to send link messages to scripts in every adjustable prim in the linkset.

Then we got those specialized link functions as well as the general llSetLinkPrimitiveParams that could do it all from a single script, but with a 200 msec delay after each call, so if one wanted to do different things to different links, it could take a while.

Then we got llSetLinkPrimitiveParamsFast which did away with that forced delay.

There are still reasons to use some of the specialized link functions, including stuff still not available in llS.L.P.P.F. (llLinkParticleSystem and llSetLinkTextureAnim), and unfortunately there are still things that require link messages to separate scripts in individual links (e.g., to emit sound from a specific prim)>

[Wulfie Reanimator]

2 hours ago, Qie Niangao said:

The history as I recall:

Pretty sure there was a time early on when there were no llSetLink* functions, so there was no choice but to send link messages to scripts in every adjustable prim in the linkset.

Then we got those specialized link functions as well as the general llSetLinkPrimitiveParams that could do it all from a single script, but with a 200 msec delay after each call, so if one wanted to do different things to different links, it could take a while.

Then we got llSetLinkPrimitiveParamsFast which did away with that forced delay.

There are still reasons to use some of the specialized link functions, including stuff still not available in llS.L.P.P.F. (llLinkParticleSystem and llSetLinkTextureAnim), and unfortunately there are still things that require link messages to separate scripts in individual links (e.g., to emit sound from a specific prim)>

llSetLinkTexture was released in 2007, so that makes sense, but oof how the legacy of bad decisions live on in current products.

Edited June 11, 2019 by Wulfie Reanimator

[animats]

20 hours ago, Aishagain said:

Oz and Simon are far more knowledgeable than us oldtimers but, and it's a big but, they are very few and are less "hands on" than they were.

Often I get the feeling at Server User Group that the developers are in way over their heads.

LL's tools for diagnosing SL problems seem very weak. Asking people to go to the beta grid and teleport around while someone watches indicates they don't have automatic tools for that. You'd expect them to be running platoons of scriptable bots on the beta grid to exercise things. Especially since there are scriptable bot systems for SL. But apparently not.

The sim overload problem seems to have the developers puzzled. No insight on this has come out at the meetings. SL seems to be gradually slowing down, and we don't know why. That's very serious and not getting fixed. And, of course, region crossings remain a problem.

A big problem for LL is staffing. Finding someone good enough to deal with difficult internal problems sim side will be very tough.

• SL is totally different than web programming. The skills that get you a web-related job today are totally unhelpful. SL is all about persistent state. Web services are all about transaction processing. SL is soft real time, an unusual area. Few people other than MMO server side engine devs do this kind of thing, and there are not many of those people.
• Going to LL is not a career choice that leads to a better job in Silicon Valley. Three years in, you finally understand the internals of this monster. This has little value to your next employer.
• Anybody good enough to fix the hard problems inside SL can probably get into Google or Facebook or Amazon and make tons of money. They have lots of complicated code with scaling problems that needs attention and will pay for people to work on it.

Not that LL is trying very hard. Here are Linden Lab's job ads. Four for Sansar. One, in QA, for Second Life.

 

Edited June 12, 2019 by animats
antecedent

[Profaitchikenz Haiku]

1 hour ago, animats said:

Often I get the feeling at Server User Group that the developers are in way over their heads.

I know it looks bad, but the same sense was growing with the TP problems, and they finally got a grasp of it. My feelings are that

1) They aren't going to release details of weaknesses to us (understandable)

2) They cannot switch everybody available to deal with a problem because there is still other stuff

3) The old problem of change-management whilst fault-finding, only varying one thing at a time to asses the impact it has. (SL is complicated, this is going to be a lengthy operation)

4) The problem might be user-created (Mesh everywhere) and they can't just ban it, too many users/creators would get hurt

I felt happy enough that they weren't simply brushing the problem away. It's painful waiting for a solution but it was just as bad when TPs stuffed you.

[Qie Niangao]

10 minutes ago, Profaitchikenz Haiku said:

2) They cannot switch everybody available to deal with a problem because there is still other stuff

This is super important. As I understand it, Simon continues working the teleport / region-crossing problem with the goal of best-ever region crossing performance. If there's a reasonable probability of that succeeding, I'd much rather that work continue and let the script performance decay further for a while.

1 hour ago, animats said:

Four for Sansar. One, in QA, for Second Life.

We can't see their capitalization, but SL's absurdly positive cash flow can't keep up with this forever. They need a new CEO before they bleed out.

[Israel Schnute]

Not sure if "idle" scripts are worse than farming parcels, but a neighbor just dumped a 4000 parcel which freed 1000 scripts exactly(and this wasn't even the farm part).  Now I'm down to 6288 scripts running.  My 10 year old Train script does a segment in 1 minute and 24 seconds.  It should take no more than 55 seconds (which occurs at 3200 scripts).  At  7400 scripts, that run was almost 3 minutes long.

I'm looking around, and frankly, this means that the servers are underpowered.  I went around deleting all the scripts from lights and things that I never touch, or have never turned on and off for years.  I managed to only get 300 scripts out.  I didn't expect to dump anywhere close to 1000, but overall, those little idling scripts don't come up to a very high percentage.  I think the only way you can have an entire sim run on 3000 scripts, is to be boring, or very static.  Likely, the usual shopping sim.  (ignoring nature sims for a moment as the exception).

If one has buildings with multiple stories, or lights so the streets are like, you know, not dark, maybe some vehicles, rides, animals, noises, ambiance, traffic signals, music, bathtubs, whatever else I can think of, I don't know if these sims are build to handle it.   Maybe I don't expect 100% performance when you start bogging it down, but there is a line.  A slight lag in performance I personally accept for more prims (the last allotment was WAY too little).  But, an upgrade seems to be in order.

One more edit, I had someone idling in a sim, they left.  The script count for them was exactly 301 and they weren't even that fancy looking.  Therefore, using the agreed upon max of 3000 scripts when the sims start getting bogged, you can only have 10 people in a sim?  OTOH, two people in a car towing a boat just drove through.  194 scripts.  About 5%-ish of what a sim can handle before serious lag?

Edited June 12, 2019 by Israel Schnute