Deploys for the week of 2012-09-24

[Moon Metty]

There's a pattern in how affected regions behave.
You could call this script a "patient timeslice counter", lol.

default
{
    state_entry()
    {
        llResetTime();
        frame = (integer)llGetEnv("frame_number");
        region = llGetRegionName();
        llOwnerSay("Region "
                 + region
                 + "\ncurrent frame: " + (string)frame
                 + "\nNo lag uptime " + (string)(frame / 162000) + " hours");
        while (TRUE)
        {
            llSleep(0.0001);
            integer new_frame = (integer)llGetEnv("frame_number");
            integer elapsed = new_frame - frame;
            if (region != llGetRegionName()) llResetScript();
            if (elapsed > 45)
            {
                llOwnerSay("FPS = " + (string)(elapsed / llGetAndResetTime()) + ", Frame = " + (string)(elapsed));
            }
            frame = new_frame;
            
        }
    }
}

It shows the FPS, and how many frames go by before the script gets a turn to run.
"No lag uptime" means the time to reach the current framenumber in perfectly lagfree conditions.

=======

[8:09]  Object: Region Serpentata
current frame: 3902516
No lag uptime 24 hours
[8:11]  Object: FPS = 21.119050, Frame = 1960
[8:12]  Object: FPS = 18.227080, Frame = 1960
[8:14]  Object: FPS = 21.051160, Frame = 1960
[8:14]  Object: FPS = 22.998090, Frame = 358
[8:16]  Object: FPS = 20.837030, Frame = 1602

Note that 1602 + 358 = 1960 again

=======

[8:25]  Object: Region Nolidae
current frame: 3922795
No lag uptime 24 hours
[8:27]  Object: FPS = 18.511800, Frame = 1961
[8:28]  Object: FPS = 22.537040, Frame = 1961
[8:29]  Object: FPS = 23.074650, Frame = 853
[8:30]  Object: FPS = 22.631170, Frame = 1108
[8:30]  Object: FPS = 18.577030, Frame = 843
[8:31]  Object: FPS = 18.575690, Frame = 1118

Note: 853 + 1108 = 1961 and also 843 + 1118 = 1961
Two different regions, same numbers.

=======

[7:04]  Object: Region Turnip
current frame: 24305650
No lag uptime 150 hours
[7:05]  Object: FPS = 18.101890, Frame = 1400
[7:07]  Object: FPS = 20.265610, Frame = 1400
[7:07]  Object: FPS = 26.697500, Frame = 591
[7:07]  Object: FPS = 23.388500, Frame = 809
[7:08]  Object: FPS = 18.943070, Frame = 591
[7:09]  Object: FPS = 17.409910, Frame = 809

The magic number for this region is 1400.

=======

[6:53]  Object: Region Eggar
current frame: 37910844
No lag uptime 234 hours
[6:53]  Object: FPS = 26.012590, Frame = 1123
[6:56]  Object: FPS = 26.867400, Frame = 1122
[6:57]  Object: FPS = 20.847310, Frame = 909
[6:57]  Object: FPS = 18.683490, Frame = 213
[6:58]  Object: FPS = 18.166260, Frame = 907
[6:58]  Object: FPS = 25.348190, Frame = 213
[6:59]  Object: FPS = 26.396440, Frame = 910
[6:59]  Object: FPS = 21.316050, Frame = 214

All very close to 1122.

=======

[8:41]  Object: Region Laserlight
current frame: 5379152
No lag uptime 33 hours
[8:42]  Object: FPS = 17.480940, Frame = 843
[8:43]  Object: FPS = 16.642640, Frame = 845
[8:43]  Object: FPS = 18.821870, Frame = 508
[8:43]  Object: FPS = 16.920500, Frame = 335
[8:44]  Object: FPS = 19.506290, Frame = 506
[8:44]  Object: FPS = 15.175480, Frame = 339

Here the number is 844.

=======

[5:47]  Object: Region Highflyer
current frame: 3948278
No lag uptime 24 hours
[5:48]  Object: FPS = 17.933130, Frame = 560
[5:48]  Object: FPS = 21.846250, Frame = 561
[5:49]  Object: FPS = 19.775010, Frame = 961
[5:49]  Object: FPS = 17.722290, Frame = 160
[5:50]  Object: FPS = 21.606740, Frame = 392
[5:50]  Object: FPS = 22.683760, Frame = 169
[5:50]  Object: FPS = 22.873470, Frame = 560

Note: (961 + 160) / 2 = 560 and also 392 + 169 = 561
This sim's number is 560.

=======

560, 844, 1122, 1400, 1961 .... What do these numbers have in common?
Well, they're all multiples of 280.

Sorry for the lengthy post.

[Ayesha Askham]

Moon

Could you explain your data in a way that a dunce like me can understand without my brain exploding, please? :smileyembarrassed:

I can see that you have detected a pattern, but what does it mean?:matte-motes-confused:

[Moon Metty]

Hehe, ok Ayesha.

In the most simple words: If your region is in this state, it needs a restart, lol.

But I know you're not the dunce you claim to be! On a healthy and well managed-sim each script gets a turn each frame, 45 times per second. If there's time dilation, the frame rate drops. Then scriptspeed drops too, but it's possible that scripts still get a turn each frame, right?

On the affected regions we see here, scripts only get a turn every 560th, 844th, etc frame. The pattern is clearly in the frame numbers, not in the realtime measurements. So there must be some process on the sim that counts to 280 frames, and it's probably somewhere in the script engine.

We don't know the inner workings of the server, but for a developer this pattern can be a nice clue.

[Ayesha Askham]

Woo

Thanks Moon, I understood that and...nope my brain seems to be somewhere still inside my cranium (which is good!).  I had seen a pattern of behaviour that looked like that but I had no metrics on it!:smileysurprised:

[Qie Niangao]

Moon, that's a pretty phenomenal bit of detective work!

Besides the 280 frame-multiple magic, I'm struck by how recently these sims were rebooted. Even if they were in constant 0.4 dilation (the approximate worst-case for these guys), 24 no-lag hours would stretch to only 60 hours of real time.

That argues against the "sick for lack of a restart" hypothesis... and that means (unless steps were taken to prevent its recurrence) it could happen again anywhere, even on sims that were recently restarted.

(This of course assumes that frame_number behaves as we think it does... and I think it does.)

[Moon Metty]

Actually, maybe we're looking at griefer objects, each containing 280 malicious scripts.

[Qie Niangao]

That's as plausible an explanation as any I can dream up, and seems to fit all the data we have right now.

I'd hate to go hunting for that object, though, given that a scripted sensor would only rarely get to execute.

[Hitomi Tiponi]

@Moon - did all the affected regions have part of them in which scripting enabled for everyone?

[Moon Metty]

Yes Hitomi,

Even under these terrible conditions, Qie has managed to find griefing objects, each with 280 scripts inside ...