Region FPS stability

[Profaitchikenz Haiku]

I started adding extra logging to a KFM-ed build that seems to show sporadic problems with one of the objects not starting to move when the KFM command is issued, What I got back from the logging were two instances where the object had obviously not moved at all, and when the KFM command was issued, the values for llGetRegionFPS() were 21.131 and 8.193, not the 44 or 45 figures I would have expected. The drops were obviously transitory, because when the 30-second timeout detected and reported there had been no movement, the FPS values were at 44. and 45. as one would wish them to be.

Does anybody else have any experience of how stable mainland should be? I don't know if one could or should expect momentary drops of the region FPS to as low as 21, let alone 8.

I was originally intending to raise a Jira to report sporadic failures of KFM to commence motion, but it looks like this is a different issue, KFM is fine, the server is the cause of the problem.

Edited March 24, 2021 by Profaitchikenz Haiku
wrong function name

[Wulfie Reanimator]

I think @animats might be able to shed some light on this.

[Rachel1206]

Yes contact @animats and let him show your his work and solution in-world, he is a highly skilled excellent programmer and with deep knowledge of how SL internals work.

[Profaitchikenz Haiku]

Thanks.

/me sings "Hey Joe! Would you please lend me your ear? I've got to know if everything's alright"

[animats]

I've seen drops in sim frame rate, but nothing like that. My main location in world, in Vallone, has 5 of my NPCs running around, all using keyframe motion. Plus there are four escalators, also using keyframe motion. Sim FPS is holding steady at 44.9 FPS according to the statistics bar in the viewer.  Script usage is at 69%, which doesn't change much if I shut down all of my stuff.

I have a little device called "Sim Lag Reporter", which checks for delays in sim script responsiveness of more than 200ms. I turned that on, and it's not reporting any problems.

What region is in trouble?

[Profaitchikenz Haiku]

Zephyr. I've just written a monitor to detect FPS changes of more than 10%.

When a mainland region is showing this sort of behaviour, do you raise a support ticker, AR it, or raise a Jira?

[Coffee Pancake]

Realistically, unless it's your stuff that's causing the issues, the best you can do is to contact the region owner.

In the case of mainland, that's a support ticket.

Good Luck!

[Profaitchikenz Haiku]

Interesting session, a big thanks to Animats for helping out. During all the time we were there, with a motorcycle rezzed and going in and out of the region, the 15-second logged FPS never dropped below 43, although I saw a momentary 39 FPS in the viewer stats bar when the bike first arrived.

So the region is healthy most of the time, which my other scripted stuff tends to bear out.  I'm going to do some aggressive logging for a few days and ask them at the SUGM who to hand the results to.

56 minutes ago, Coffee Pancake said:

Good Luck!

Actually, I share your pessimism, after a chat with Joe I think I'm going to switch from defensive programming (assume something might go wrong) to offensive programming (assume nothing is going to go right) and see where that gets me.

With KFM that has already meant adding  a while loop immediately before the KFM command is issued, and only exiting the loop when llGetRegion FPS() returns > 40%

[Coffee Pancake]

1 minute ago, Profaitchikenz Haiku said:

With KFM that has already meant adding  a while loop immediately before the KFM command is issued, and only exiting the loop when llGetRegion FPS() returns > 40%

A while loop based on region FPS doesn't guarantee anything aside from applying more pressure to the region.

General rule of thumb, the more closely you look at performance metrics, the worse the entire system performs.

 

Also, be very wary of over engineering a complex command tracking and receipt system with a stack of issued commands and so on, very easy to end up with a monster scripted project that will still trigger the original issue & add half a dozen more. There are several points of failure when it comes to object motion, not least of which is your viewer simply missing an object update and being wrong. The simplest & lightest solution might be to blindly reissue all commands a second time after a short delay.

[Profaitchikenz Haiku]

2 minutes ago, Coffee Pancake said:

A while loop based on region FPS doesn't guarantee anything aside from applying more pressure to the region.

I agree in principle, but if tomorrow there are no IMs waiting for me, I'm going to take it as a tentative "OK".

You're right also about gathering metrics often exacerbating the issue, but there's no other way I can track this down. There *is* a problem, it's not visible using the normal viewer figures, I don''t have access to the LL logs to cross-reference the errors against other region occurrences. I don't feel upping and offing to a less problematic region is the answer, I've already moved once from a region that showed poor performance and all that happens is that after a few weeks in the new home, the poor performance finds where I've moved to and joins me again.

I've used KFM a lot for railways, canal boats, swinging signs, slicing-peril pendulums, it's always been fairly predictable, but this is something new, and I want to either get it solved, or work around it.

[Coffee Pancake]

1 minute ago, Profaitchikenz Haiku said:

it's always been fairly predictable, but this is something new, and I want to either get it solved, or work around it.

Maybe you've stumbled on a more serious region bug, might be an idea to make simplified test objects to either verify that it's your specific region or something systemic. It's difficult to help more without source code.

 

[Profaitchikenz Haiku]

I will be doing a simple case as the basis for a Jira after presenting the problem to the Server User Group meeting yesterday. Any volunteers to run the tests case and suggest alternatives are very welcome.

[Profaitchikenz Haiku]

Here's the results of the first monitoring, and they are rather surprising.

The monitor takes an entry of the time, number of avatars in the region, region FPS every 10 seconds, and records the results whenever one or two things occur:

The number of avatars has changed from the previous value

The region FPS has changed by more then 0.9 from the previous value

First, the region behaving "normally"

The entries are the object name, (avatars in region), llGetRegionFPS value

• WhatFps: 11:20:35 (1) 44.611770
• WhatFps: 11:50:56 (1) 43.436540
• WhatFps: 11:51:06 (1) 44.839750
• WhatFps: 11:52:56 (2) 45.087070
• WhatFps: 11:55:06 (1) 45.144430
• WhatFps: 11:57:06 (1) 42.909450
• WhatFps: 11:57:16 (1) 44.829430
• WhatFps: 12:09:46 (2) 44.569030
• WhatFps: 12:20:46 (2) 40.782920
• WhatFps: 12:20:56 (2) 45.255450
• WhatFps: 12:27:36 (1) 44.457620

Notice that Region FPS drops to 40 at one point but in general is fine with varying numbers of avatars present

And now, the region behaving in an abnormal way

Note there are no avatars present at all, so  the variations cannot be attributed to avatar actions such as building.

• WhatFps: 23:27:26 (0) 45.255490
• WhatFps: 23:30:56 (0) 42.674430
• WhatFps: 23:31:06 (0) 44.722870
• WhatFps: 23:43:36 (0) 12.144670
• WhatFps: 23:43:46 (0) 8.190929
• WhatFps: 23:44:26 (0) 35.758640
• WhatFps: 23:44:36 (0) 8.200023
• WhatFps: 23:45:06 (0) 44.554590
• WhatFps: 23:45:26 (0) 8.198494
• WhatFps: 23:45:36 (0) 44.572800
• WhatFps: 23:46:26 (0) 8.188798
• WhatFps: 23:46:36 (0) 44.980500
• WhatFps: 23:46:46 (0) 8.202129
• WhatFps: 23:47:06 (0) 45.127930
• WhatFps: 23:47:46 (0) 40.311850
• WhatFps: 23:47:56 (0) 44.848960
• WhatFps: 23:48:16 (0) 8.206425
• WhatFps: 23:48:26 (0) 45.213500
• WhatFps: 23:48:46 (0) 8.174291
• WhatFps: 23:48:56 (0) 44.519040
• WhatFps: 23:49:06 (0) 8.214548
• WhatFps: 23:49:26 (0) 44.640190
• WhatFps: 23:49:36 (0) 8.204894
• WhatFps: 23:49:56 (0) 44.728490
• WhatFps: 23:50:06 (0) 8.199108
• WhatFps: 23:50:16 (0) 44.774190
• WhatFps: 23:50:26 (0) 8.223446
• WhatFps: 23:50:46 (0) 44.961960
• WhatFps: 23:50:56 (0) 7.616540
• WhatFps: 23:51:36 (0) 44.678660
• WhatFps: 23:52:16 (0) 8.196971
• WhatFps: 23:52:26 (0) 44.498580
• WhatFps: 23:52:36 (0) 8.214035
• WhatFps: 23:52:56 (0) 31.833360
• WhatFps: 23:53:06 (0) 44.881900

 

Edited March 25, 2021 by Profaitchikenz Haiku

[Ardy Lay]

Is this a factor?  Region Idling FAQ - Second Life Server - Second Life Community

[Profaitchikenz Haiku]

It doesn't say that moving_start and moving_end events are going to be affected, the main warning is on very short timer loops where the timer might be trying to run faster than the 8-10 frames a second the idling region is now slowed down to. Nothing I'm doing is that tight.

My original KFM code for the funiculars when I had them running 64 metres up and down a cliff used the llGetRegionFPS() as the divisor when working out the speed for the KFM, I assume that this was advised as good practice because it would compensate for idling by allowing for fewer frames to be executed by the KFM? If however the value I get back varies this much it seems pot-luck if I divide by 40.xxx or 8.xxx to get a time for the KFM movement that is appropriate.

ETA on thinking this through, I wonder if llGetRegionFPS() is not able to average enough data to return a meaningful calculation when a region is being idled? I'll have to ask this question of @Simon Lindenor @Rider Linden

Edited March 25, 2021 by Profaitchikenz Haiku