Jump to content
Canis Canning

Modern Sim Lag Times

Recommended Posts

What really bothers me about all this is the discrepancy between Top Scripts & Stats making it nearly impossible to track where the issue actually is.

Top Scripts is claiming 2328 scripts, time 6.682ms - the top few individual figures are visitors; most of the scripts appear from TS to be behaving reasonably well, we've only few showing as over 0.010ms

Stats is seeing 7380, time 18.547 ms - & that figure will always fill all spare time.

This is a constant issue on this region. I don't own it all, just ⅛, but I have Estate rights so I can try to look at the figures. I built a region 'tractor' to go round the whole thing, floor to ceiling & try to measure script activity using a sensor. Allowing for the inaccuracy of the method, it pretty much agreed with Top Scripts - so further efforts to find the errant scripts stops there.

I've done the best I can with my own parcel, the only scripts above 0.010 are ones I didn't write, but even with the co-operation of the region's main owner, we are really struggling to find what is causing our constant script lag - because we can't track the numbers in Stats to actual objects on the region.

ade5696ddff3b873dfd588562b602a57.png

ScreenShot 2018-07-03 at 11.51.14 am.png

Share this post


Link to post
Share on other sites

One thing that may help a bit is to lose the temp-rezzed zombies near http://maps.secondlife.com/secondlife/Northfarthing/18/97/606 -- they have like 20 scripts each, and burn a lot of processor time just to get those scripts loaded into memory and slotted into the scheduler. (If you can just make them non-temp, it could help.)

As a Mainlander I'm by no means an expert on Top Scripts, but I might hazard a guess that Statistics and Top Scripts use slightly different time-averaging windows. I know that heavily scripted temp-on-rez objects will cause short bursts of demand -- almost like an avatar teleporting in -- so maybe that's part of the discrepancy between Top Scripts and the Statistics panel.

Share this post


Link to post
Share on other sites

I advise you start by turning on object updates (CTRL+ALT+SHIFT+U), roaming your region, and removing items that meet the following criteria. There will not be "that one thing", expect to be brutal. Take your time. Watch items for a moment

  • If it rezzes or spawns things (A pulse of green). REMOVE.
  • If moves, animates or changes (steady stream of blue). REMOVE.
  • If it's a virtual pet, decorative or breedable .. 

There are so many objects that move and roam about on that region it's really no surprise that performance is suffering.

Thinking there are one or two bad apples you can remove to fix the problem is misunderstanding the nature of the problem. The script engine is spending a little time running the script and a lot of time updating objects on the region. Or in other words .. low script time resulting in high overall usage. By comparison, an object with high script time that does not stream updates to objects could actually be lighter.

 

RGAnIlr.png

52tj13e.pngYiTQPVw.pngkfatvAX.png

  • Thanks 2

Share this post


Link to post
Share on other sites

A very active sim, done well, doesn't go into serious overload. See Janet's Viking sim. There are trains all over the place, freight loading cranes, horses and carts, flying birds, and entire Viking ship experiences. About 85% of scripts are running. Anyone who makes things move in SL should visit.

Edited by animats
  • Thanks 1

Share this post


Link to post
Share on other sites
2 hours ago, CoffeeDujour said:

RGAnIlr.png

I absolutely agree with checking out object updates as a first clue of where scripts are burning cycles, but I specifically didn't mention the skyway devices (whatever they're called) such as the one depicted above because they contain only one script each, and burn less than 1µs per frame on average. They're apparently following a simple, precalculated KFM waypoint trajectory, so there's practically no script processing at playback. They'll certainly contribute some network and viewer-side load but they won't be adding to script overload.

All this said, there is a huge discrepancy in script counts between the Statistics panel and the Top Scripts number reported above. I'm inclined to trust the Statistics panel, but it would sure be nice to know what Top Scripts isn't counting.

Edited by Qie Niangao

Share this post


Link to post
Share on other sites
1 hour ago, Qie Niangao said:

I absolutely agree with checking out object updates as a first clue of where scripts are burning cycles, but I specifically didn't mention the skyway devices (whatever they're called) such as the one depicted above because they contain only one script each, and burn less than 1µs per frame on average. They're apparently following a simple, precalculated KFM waypoint trajectory, so there's practically no script processing at playback. They'll certainly contribute some network and viewer-side load but they won't be adding to script overload.

All this said, there is a huge discrepancy in script counts between the Statistics panel and the Top Scripts number reported above. I'm inclined to trust the Statistics panel, but it would sure be nice to know what Top Scripts isn't counting.

The scripts might be simple and have low times, but that time wont represent the time impact issuing object updates has. There are many ways to max out a region.

Share this post


Link to post
Share on other sites

Help me out here, being a scripter. Does “object updates” mean an object is updating its prim parameters via script, or servicing HTTP requests/responses, or ?

Share this post


Link to post
Share on other sites

Yes, quite literally that .. an object update is when .. an object is updated, typically from a command in a script.

Share this post


Link to post
Share on other sites

An object update is communicated from sim to viewer. Setting prim parameters would do it, and so does KFM, rezzing a new object, llDie(), etc. Strictly server-side stuff such as script-to-script communications wouldn't cause such updates, nor do scene changes that are strictly viewer-side like ongoing particle displays, llTargetOmega, texture animation, etc.

 

Share this post


Link to post
Share on other sites

The sim stats posted a lot earlier in this thread showed a fairly high NET time, and I suggested there was a lot of off grid web database activity going on there...

Now we have a list of scripted items showing multiple "Angels and Demons" items, a scripted Grid-wide RP Combat hud system, that presumably uses an off grid web database to track players scores and levels...

Might be an idea to look at the list of A&D stuff, and remove anything you really don't need, like surplus 'score boards'.

Also, you you really NEED a Grid Status live web feed display?, and so on, I can't remember what the NET time was in your earlier screenshot, 6 or 7 ms?



 

Share this post


Link to post
Share on other sites

Re temp-rezzed object detection, try something like this:

// Report number of temp prims on the current parcel

float SAMPLING_INTERVAL = 0.5;
integer prevCount;

default
{
    state_entry()
    {
        llSetMemoryLimit(8000);
        llRequestPermissions(llGetOwner(), PERMISSION_TAKE_CONTROLS);

    }
    attach(key id)
    {
        if (NULL_KEY != id)
            llResetScript();
    }
    run_time_permissions(integer perms)
    {
        if (PERMISSION_TAKE_CONTROLS & perms)
        {
           llTakeControls(FALSE, FALSE, TRUE);    // Continue running on no-script land
           llSetTimerEvent(SAMPLING_INTERVAL);
        }
    }
    timer()
    {
        integer tempPrimCount = llGetParcelPrimCount(llGetPos(), PARCEL_COUNT_TEMP, FALSE);
        if (tempPrimCount != prevCount)
        {
            llOwnerSay((string)tempPrimCount+" temporary prims on this parcel.");
            prevCount = tempPrimCount;
        }
    }
}

And I just remembered that this thread is split in two different problems, right at the page break; I'd forgotten there was a whole separate thing going on back on page 1. So I imagine this refers to that earlier problem:

7 hours ago, Klytyna said:

The sim stats posted a lot earlier in this thread showed a fairly high NET time [...]

Right, that screenshot from the earlier sim (NARCISO) showed 8.75 msec of Network Time (which is indeed a heck of a lot, even for a dilated 44 msec frame length). In the latter, "Angels & Demons" sim (Northfarthing), the network time was very low (and the frame wasn't extended -- all the server-side lag was script-on-script). I admit I've never been entirely clear what all gets attributed to the "Network Time" accounting category of SL sim statistics.

  • Like 1
  • Thanks 1

Share this post


Link to post
Share on other sites

@Qie Niangao

Nice, Qie. Ty. Frontier Linden's detector goes further and gives the user the locations of any temp objects that are found. I could post the script because it's full perms, but I don't know if that would be right or not, so I won't. I'll pass a copy of the detector to anyone who wants to see it though.

  • Like 1

Share this post


Link to post
Share on other sites
10 hours ago, Klytyna said:

The sim stats posted a lot earlier in this thread showed a fairly high NET time, and I suggested there was a lot of off grid web database activity going on there...

Now we have a list of scripted items showing multiple "Angels and Demons" items, a scripted Grid-wide RP Combat hud system, that presumably uses an off grid web database to track players scores and levels...

Might be an idea to look at the list of A&D stuff, and remove anything you really don't need, like surplus 'score boards'.

Also, you you really NEED a Grid Status live web feed display?, and so on, I can't remember what the NET time was in your earlier screenshot, 6 or 7 ms?



 

The A&D stuff only transmits over HTTP once, at attach. The only time there are ever any updates are when it's in active use, otherwise it's all quiescent, even when worn. The same goes for all the working demos set outside the store.

The 'region servers' are permanently listening, but if no comms are sent to them, that's all.

The only other 'active' objects have 15 min timers [inworld counter, grid status feed etc], so are active once every 15 mins, except for a clock, which is once per second.

Two of the active demos have a long time-gap sensors, so go inactive if there's no-one near [10m]. I've extended those times to 5 mins.

I did a bit of a tidy-up anyway, I lifted all the Hippo stuff, now defunct & the Alika visitor counter I never actually use, so I'm pretty sure now the only script in the parcel not written by me is whatever's in the campfire.

I'm now more than sure it's nothing on my parcel.

Using Top Scripts I have 83 out of 2300 scripts on the region. 0.156ms of 6.510.

 

Phil - could I get a copy of that, thanks

Edited by Norton Burns

Share this post


Link to post
Share on other sites
On 6/27/2018 at 5:42 AM, Phil Deakins said:

Note: I have a Linden-made temporary objects detector if anyone wants a copy (created by Frontier Linden). He gave it to me many years ago, and once in a while someone will find mention of it in the archived forum and ask for a copy.

I would like a copy, please. 

Share this post


Link to post
Share on other sites

Object updates are displayed as red, blue, or green. I see a lot of red as the avatar moves. Those disappear after the avatar is stationary.

Looking around  Vallone sim, everything is rather quiet in terms of updates, even the carnival, except for one carnival ride which has far too many updates when doing nothing. There's one plant on another parcel that moves a bit and has updates. All my parked bikes generate no updates. Nor does the parked helicopter. Once started, there are constant updates, of course.

But there's 16ms of script usage in every 22ms cycle with nothing much going on. About 98% of scripts are running. Whatever is going on isn't generating object motion.

Technical question: is a blue marker really an ImprovedTerseObjectUpdate message, and a red an ObjectUpdate message?

Share this post


Link to post
Share on other sites
3 hours ago, animats said:

Technical question: is a blue marker really an ImprovedTerseObjectUpdate message, and a red an ObjectUpdate message?

A viewer developer would know for sure but that sounds right based on what wiki documentation exists:

Quote

The update indicators show you when packet data is being utilized -- typically, when an object makes an update to the world. Each color has a different meaning:

  • Red: Indicates a full update, such as the creation of a prim. This is a relatively large data packet sent to your computer. If you see objects that are showing a constant stream of red, they're contending for your bandwidth, which may cause other things in the area (like textures) to load slower. If the object is made up of many pieces, the packet is larger.
  • Blue: Indicates a partial update, such as a change of position or color for a prim. These are always smaller than full updates. However, the same rules apply as for the full updates. If you're creating content, it's a good habit to make sure it's not updating many times per second. Changing colors, textures, shape, or particle parameters several times per second causes partial updates that contend for your bandwidth.
  • Green: Indicates an ending update, such as the deletion of a prim. If this packet gets lost on the way from the server to your computer, the object becomes a "ghost" -- your viewer still renders it, but you can walk through it, and when you edit it, the Creator field is blank because the object no longer exists on the server.

It's certainly the case that scripts can do all manner of script-laggy stuff without sending any object updates to the viewer. They can, for example, start and stop and even implant other whole scripts, calculate, communicate, store and retrieve script memory and key-value pairs in Experience persistent store, and generally run amok without needing to send anything to the viewer. It's true that object updates can be especially costly but there are plenty of other ways for scripts to lag other scripts.

Edited by Qie Niangao

Share this post


Link to post
Share on other sites
9 hours ago, Qie Niangao said:

It's certainly the case that scripts can do all manner of script-laggy stuff without sending any object updates to the viewer. ...  there are plenty of other ways for scripts to lag other scripts.

If a script is doing extensive work that's doesn't affect the world, that's probably unnecessary load.

Scanning for other objects is a common way to use script time without accomplishing much. Badly coded "security" programs can lag an entire sim, endlessly getting data with llSensor about objects that aren't doing anything and examining those objects. That may be the problem with some breedables - checking for objects around them in case they want to move, even when not moving.

Communicating can be a problem. There are two DJ speakers in Kama Center which are constantly issuing updates and show "Slave Initializing" as hover text. They're apparently trying to find a music source or a control unit or something. Vendors can get into that mode if their server goes down.

Estate tools which give you info about active sensors or communicators might help when diagnosing lag. There's too much mystery about this and not enough data.

Share this post


Link to post
Share on other sites

A lot of this is beyond me, but if you teleport to the road in the Kuula sim, and then walk into the NCI parcel west of the road, when it's busy, you'll see something about the difference between mesh rezzing and other sources of load. It's set up so you can't see the avatars in the parcel from outside, and when you cross the parcel boundary they all load at the same time.

There is a lot of stuff downloading to your viewer when you TP into a sim. and you have a chunk of data about you being sent from one sim to another. The way things work, it's hard to get a good figure for Download Weight, and complexity is something a bit different anyway, but, for what little it's worth, my usual Complexity is under 15k, and there are other people up around 150k, and the slider for limiting complexity in nearly useless when the complexity is over about 250k

There is a lag indicator in the parcel, it does show red pulses when people TP in.

I have been seeing the same sort of high-script times lately as the OP has described. The mesh rezzing problem isn't new. I think the script-time problem is, and there have been changes of occupier on some other parcels

I make a bit of effort to keep the mesh I make a low-complexity as possible. I look at the LOD distances and triangle counts (use Firestorm to get the info), and try to avoid triangles smaller than a screen pixel. There are rumours of some changes in the work, a change in LOD distances, which might spoil the look of what I make. I can't say I shall be happy about that, but. frankly, the Lindens couldn't document their way out of a soggy-wet paper bag. And the same problem seems to be part of this mess.

Share this post


Link to post
Share on other sites

The biggest cause of this newer level of horrid performance is that LL reduced performance for regions by shoving more regions on a server so they could give resources to Sansar. This happened about a year ago. You didn't realize that your hard earned money you are spending on SL was getting siphoned off to pay for Sansar, did you?

  • Haha 2
  • Confused 4

Share this post


Link to post
Share on other sites
On 8/6/2018 at 6:40 AM, Sharie Criss said:

The biggest cause of this newer level of horrid performance is that LL reduced performance for regions by shoving more regions on a server so they could give resources to Sansar. 

This is the dumbest thing I've read this month.

 

  • Like 3
  • Thanks 1
  • Haha 3

Share this post


Link to post
Share on other sites
On 6/27/2018 at 5:42 AM, Phil Deakins said:

Note: I have a Linden-made temporary objects detector if anyone wants a copy (created by Frontier Linden). He gave it to me many years ago, and once in a while someone will find mention of it in the archived forum and ask for a copy.

I'd also like a copy please :)

Share this post


Link to post
Share on other sites
On 8/6/2018 at 7:40 AM, Sharie Criss said:

The biggest cause of this newer level of horrid performance is that LL reduced performance for regions by shoving more regions on a server so they could give resources to Sansar. This happened about a year ago. You didn't realize that your hard earned money you are spending on SL was getting siphoned off to pay for Sansar, did you?

FYI: SL runs on servers in a Linden-owned data center. Sansar runs on Amazon's cloud service. Totally separate things.

  • Like 3

Share this post


Link to post
Share on other sites
1 hour ago, Parhelion Palou said:

FYI: SL runs on servers in a Linden-owned data center. Sansar runs on Amazon's cloud service. Totally separate things.

That hasn't been correct for a long time now. SL actual regions run on Linden servers. Assets and textures etc is via 3rd party cloud services. The goal is to have regions in the cloud at some point with the option to have regions you own hosted geographically close

  • Like 2

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...