Recent updates to SL appear to have caused problems with multiple games.

[Sorina Garrigus]

EDIT UPDATE: The Area 51 game which is a mouse shooter was not working because of a gun attachment on my Avatar. It worked with an alt and after removing the gun. Huds can interfere with such games or have in the past but for some reason the gun was intervering with clicks on that game but not other similar target games. The rest of the issues still remain. which might point to rez time issues. But further testing on that I tested some simple rez scripts of things I made and they seem to be fast enough. Could be a slight delay but nothing like being seen in the games affected

I have at least three games that are broken due apparently due to changes in SL updates. Response time between input and the game reacting have gone from happening to pretty much immediately to happening 10 seconds or more if happening at all. So far I am seeing these issues in scripted arcade games like Komikaze which is basically a scripted version of Asteroids. You shoot and you see the particle effect it has firing but the actual shot comes out 5 seconds later and it happens between 5 to 10 seconds later and often at a different angle. Missle Defence which is a scripted version of the classic Missle command you click on the screen to fire a missile off and it happens maybe 15 seconds later or so. Duck Hunt seems to play fine but it is currently getting a flood of "Too many sound requests. Throttled until average falls." messages. But does play fine otherwise. It didn't do this before. A game called Area 51 which is a mouse view target shooter doesn't work at all. Doesn't register any clicks. I also tested Zombie Hunter another target shooter I believe to be mouseview click but not sure of how they made the game to know if they took a different approach or not. But it seemed to work perfectly fine compared to some of the other target shooter games as duck hunt did. But Duck hunt isn't a mouse view shooter it is just a click shooter. none use projectiles to account for hits that I can tell. I am going through a lot of games today to see if they are affected. Another click shooter arcade (but not mouse view but a sit down camera position type game) called Kill em all seemed to work fine as does another mouse view click shooter called Zombie Forest Run. 

On the board game side of things I am still testing. Haven't done proper play testing but noticed huds in Hirons games which use the experience feature take a few seconds to appear/attached when before it was nearly right away. But was only able to test a couple board games to see if they were affected. Haven't found anything outside of that. 

Not sure what the pattern is causing issues in some games but not others. 

If you have any of these games in your sims or find other problems post about them. There were some form of issues the other day I assume created by updates. I had issues with my anywhere doors not being able to find animations but they seem to be working again fine now. They seem to have had those errors after a sim reset. 

 

The sim has a lot of games in it but seems to be in good shape unless I am missing something as i don't understand all the data of sim statistics. 

I am leaving the games with issues up in case a Linden wanted to come down and see what is being affected and how/why. The affected games are primarily at my Grunge Arcade location but all are in the same sim if that is a factor. Hopefully the issues will be corrected soon. 

 

Edited August 19, 2022 by Sorina Garrigus

[Rowan Amore]

2 hours ago, Sorina Garrigus said:

I have at least three games that are broken due apparently due to changes in SL updates. Response time between input and the game reacting have gone from happening to pretty much immediately to happening 10 seconds or more if happening at all. So far I am seeing these issues in scripted arcade games like Komikaze which is basically a scripted version of Asteroids. You shoot and you see the particle effect it has firing but the actual shot comes out 5 seconds later aand it happens between 5 to 10 seconds later and often at a different angle. Missle Defence which is a scripted version of the classic Missle command you click on the screen to fire a missile off and it happens maybe 15 seconds later or so. Duck Hunt seems to play fine but it is currently getting a flood of "Too many sound requests. Throttled until average falls." messages. But does play fine otherwise. It didn't do this before. A game called Area 51 which is a mouse view target shooter doesn't work at all. Doesn't register any clicks. I also tested Zombie Hunter another target shooter I believe to be mouseview click but not sure of how they made the game to know if they took a different approach or not. But it seemed to work perfectly fine compared to some of the other target shooter games as duck hunt did. But Duck hunt isn't a mouse view shooter it is just a click shooter. none use projectiles to account for hits that I can tell. I am going through a lot of games today to see if they are affected. Another click shooter arcade (but not mouse view but a sit down camera position type game) called Kill em all seemed to work fine as does another mouse view click shooter called Zombie Forest Run. 

On the board game side of things I am still testing. Haven't done proper play testing but noticed huds in Hirons games which use the experience feature take a few seconds to appear/attached when before it was nearly right away. But was only able to test a couple board games to see if they were affected. Haven't found anything outside of that. 

Not sure what the pattern is causing issues in some games but not others. 

If you have any of these games in your sims or find other problems post about them. There were some form of issues the other day I assume created by updates. I had issues with my anywhere doors not being able to find animations but they seem to be working again fine now. They seem to have had those errors after a sim reset. 

 

The sim has a lot of games in it but seems to be in good shape unless I am missing something as i don't understand all the data of sim statistics. 

I am leaving the games with issues up in case a Linden wanted to come down and see what is being affected and how/why. The affected games are primarily at my Grunge Arcade location but all are in the same sim if that is a factor. Hopefully the issues will be corrected soon. 

 

Have you tried rezzing out fresh copies to see if those work?

[Sorina Garrigus]

6 minutes ago, Rowan Amore said:

Have you tried rezzing out fresh copies to see if those work?

Yup. Tried rerezing them. some I can't manually reset scripts but the ones that I did try didn't affect them. 

[Lucia Nightfire]

The main thing I see from stats is pathfinding characters in use in an unoptimized region.

That Lag Meter should not be referenced as it can show normal and the region's timings be through the roof.

4 hours ago, Sorina Garrigus said:

Hopefully the issues will be corrected soon. 

Don't wait on LL. Contact support or file a bug report.

[animats]

Are these instances of the "slow rezzing" problem that appeared about a year ago? Things that shoot projectiles have been having problems for a while. Fourmilab did some measurements and found that the delay is bimodal - either it's very small, or seconds, but nothing in between.

[Lucia Nightfire]

1 hour ago, animats said:

that appeared about a year ago

You mean November 2019? 😉

[Sorina Garrigus]

1 hour ago, animats said:

Are these instances of the "slow rezzing" problem that appeared about a year ago? Things that shoot projectiles have been having problems for a while. Fourmilab did some measurements and found that the delay is bimodal - either it's very small, or seconds, but nothing in between.

I checked for that. I some cases it could be but in other cases they had no rezing going on. 

[Qie Niangao]

I almost never see the "slow-rezzing" problem myself, but those who do: Does it come with side-effects? like maybe this sound throttling, which seems a distinct outlier to everything else.

Was anything added to the region that might contribute to that relatively high Physics time? I realize there's some Spare time and effectively all scripts get run, at least when these stats were snapped. Still, it's finding quite a lot of Havok to do each frame. There hasn't been a new Pathfinding character added or anything?

 

[Rolig Loon]

Just looking at the screen shot of the statistics posted in the OP ..... As Qie noted, there is 5.768 ms of free time per frame available, but I see that there are over 7000 scripts running and that they are using 9.305 ms or almost 42% of the server time. With all the physics time that's also being used, it's not hard to suppose that at any given instant free time is dropping close to zero.  According to that snapshot, 96.6% of scripts are running.  It looks to me as if there's an awful lot going on in the region and that it's not very well optimized. It wouldn't take much to make scripts a bit sluggish.

[Sorina Garrigus]

1 hour ago, Rolig Loon said:

Just looking at the screen shot of the statistics posted in the OP ..... As Qie noted, there is 5.768 ms of free time per frame available, but I see that there are over 7000 scripts running and that they are using 9.305 ms or almost 42% of the server time. With all the physics time that's also being used, it's not hard to suppose that at any given instant free time is dropping close to zero.  According to that snapshot, 96.6% of scripts are running.  It looks to me as if there's an awful lot going on in the region and that it's not very well optimized. It wouldn't take much to make scripts a bit sluggish.

There isn't anything physics related going on currently. it says physics FPS is 45 or so ... but underthat it shows nothing physical going on literall 0. Until a car comes down the mainland road maybe. It appears the mainland servers have been gimped. The last sever update created problems and they seem to continue.  Very little change script wise has happened. I took the games having problems to a sandbox and they were having similar problems. Then I took them to another mainland area and they seemed to work as before or close to it. 

There are plenty of scripts but they are all waiting to be used most are not super active. I pulled one very old build Settlers of Second Life which had massive amount of scripts in it. pulled some SecondAd sign boards and other small additions, shut down the scripts to the animated butterflies and of course it is having no effect. 

Here are my current sim stats after pulling the Settlers games, the games having problems shutting down the butterfly light scripts and other animated lights. So now it should be lower than it was a month ago easily .... same problems with the games affected. Will ask for a reset from LL and see if there is a difference

I had many more games out before in the past and these games were working 100% fine a few days ago.  The server seems to be fine time dilation good etc. 

[Sorina Garrigus]

2 hours ago, Rolig Loon said:

Just looking at the screen shot of the statistics posted in the OP ..... As Qie noted, there is 5.768 ms of free time per frame available, but I see that there are over 7000 scripts running and that they are using 9.305 ms or almost 42% of the server time. With all the physics time that's also being used, it's not hard to suppose that at any given instant free time is dropping close to zero.  According to that snapshot, 96.6% of scripts are running.  It looks to me as if there's an awful lot going on in the region and that it's not very well optimized. It wouldn't take much to make scripts a bit sluggish.

Also scripts in general are not sluggish. physics based games when they are used like pinball are not having any issues. They actually seem to be working better than I recall the last time I tested them. Some other games which are intense on what they do for play like the Cannons game which plays kind of like the worms video game if you know that one but in real time works perfect.  I tested another one which has a lot of animations going on called Super Rally where you drive a car avoiding obstacles jumping over things runs perfectly fine. If it was an issue of too many scripts it would be affected as some other games would be which are heavily scripted real time arcade games ... which play perfectly fine. The issue isn't consistent between various games that would be affected the most. I am fairly sure that it is some gimped issue with servers and or a problem introduced in how scripts run with certain commands etc. 

[Sorina Garrigus]

On script count ... the original data posted must been from an avatar passing through. It is under 6860 ... no clue how it counts active scripts because it alternates and showing fraction of scripts. Not sure how that works. 

[Rolig Loon]

When you look at that statistics readout, you are grabbing a snapshot, an instantaneous view of what's going on at the specific time when you collected the data. Yes, at the time of this new snapshot, physics time was only 0.498 ms, which is really quite small. In the snapshot you posted earlier, it was 6.160 ms, which is about a quarter of the server's time during that time slice.  Your script time was 10.184 ms, which is about 45% of the server's time.  As you might expect, both numbers hop all over the place from one snapshot to the next.  Still, just in those two snapshots you have between 6600 and 7000 scripts active.  Even if they are individually doing very little, they are adding up to quite a lot of the server's attention.  If physics time shoots up at the same time as you get a bit of extra demand form a few big scripts, those few big scripts, free time will drop yo zero and those few big scripts will stall.

The basic point is that everything else takes precedence over scripts for server time.  If you have a lot of avatars in the region, or if the physical objects are moving around a lot, they get whatever free time is available.  If they eat it all up and there are scripts still needing attention, the scripts will slow down.  You simply have a ton of stuff going on at once and it's all competing with the scripts.  

[Rolig Loon]

As a separate issue, you might also notice that your own packet loss in that second screen shot was 0.2% and your own FPS (not the sim's) has dropped from 29.6 (where it was in the first shot) to 10.4.  Neither number is very good, but 10.4 is really poor.  You might look at your own Internet connection.

[Sorina Garrigus]

I wasn't sure where people were looking for physics initially I was looking at Physics FPS but yah physics time was huge when I took that snap shot. Pretty low normally but I am on mainland and there are cars that come on the road so that high physics time number very much not the norm. 

I had an animesh band out playing ... they are animated but not pathfinding/moving. Any idea if that has some kind of impact. I also had two animesh shop helpers walking about but I am not sure if they are pathfinding. They move by heading to rezed prims and don't appear to be affected by physics. They are not phantom but they can walk through walls and objects. I think they are just some kind of scripted move in conjunction with a walk animation. 

On the game issues I think I narrowed down the issue. Both games affected rez scripted objects that are temporary. One game the missles are physical but the other is not. On Komikaze using edit tools I was able to determine the missle was rezing perfectly fine at about the right time ... but the script that causes it to move there is what is delayed from kicking in. I am now wondering if there has been a change with temporary scripted objects. I am doing a lot of testing to narrow down the issue the sim statistics I only understand a bit of it. I am trying to create a detailed report to LL for both sim impact and  a potential issue introduce with the last update. It might be a combination of both but reduction of resources isn't getting the games to work again. Might have reduced the delay from 5 seconds to 3 though. 

I was getting alerts my draw distance was too far is probably why. I am having lower FPS ... plus there are a lot of objects. Likely internet issues 

[Rolig Loon]

Any time you rez an object, it causes a mandatory hiccup in server load. That's one of the big reasons why landowners often prohibit renters from having repeating temp rezzers.

Same with every time an avatar enters or leaves the region.  A lot of come and go traffic slows things down.

[Sorina Garrigus]

15 minutes ago, Rolig Loon said:

Any time you rez an object, it causes a mandatory hiccup in server load. That's one of the big reasons why landowners often prohibit renters from having repeating temp rezzers.

Same with every time an avatar enters or leaves the region.  A lot of come and go traffic slows things down.

On temp rezzers yah those are problematic because they are basically going off every 60 seconds forever. But a game will just have those happened when it is being played so less of an issue. Games that rez objects in part as temp objects to avoid say some kind of issue filling the land with bullets and such in case something fails. This is mostly happens in scripted arcade style games or in shooting combat sims. 

Edited August 20, 2022 by Sorina Garrigus

[Sorina Garrigus]

I can't edit the original post for an update but the original Physics times listed was some weird random anomaly when I captured the screen. Not remotely the norm. it hovers around .45

[Qie Niangao]

19 minutes ago, Sorina Garrigus said:

On the game issues I think I narrowed down the issue. Both games affected rez scripted objects that are temporary. One game the missles are physical but the other is not. On Komikaze using edit tools I was able to determine the missle was rezing perfectly fine at about the right time ... but the script that causes it to move there is what is delayed from kicking in. I am now wondering if there has been a change with temporary scripted objects. I am doing a lot of testing to narrow down the issue the sim statistics I only understand a bit of it. I am trying to create a detailed report to LL for both sim impact and  a potential issue introduce with the last update. It might be a combination of both but reduction of resources isn't getting the games to work again. Might have reduced the delay from 5 seconds to 3 though. 

That thing with stuff rezzing but their scripts taking a while to start, that is the "slow rezzing" problem that kicks in on regions occasionally, some regions much more than others, and only really improves after a restart. Or at least that's how I've understood its description.

As you say, the different practice of constant temp-rezzing causes lag, but the "slow rezzing" thing isn't really specific to temp-rezzed items, they're just the most common things to get rezzed. And it may be that the more scripted rezzing a sim does, the more likely it will fall into the "slow rezzing" trap and need a restart.

If you are able to restart the region, it would be interesting to know if the game delay returns to normal at least for a while.

[Sorina Garrigus]

1 minute ago, Qie Niangao said:

That thing with stuff rezzing but their scripts taking a while to start, that is the "slow rezzing" problem that kicks in on regions occasionally, some regions much more than others, and only really improves after a restart. Or at least that's how I've understood its description.

As you say, the different practice of constant temp-rezzing causes lag, but the "slow rezzing" thing isn't really specific to temp-rezzed items, they're just the most common things to get rezzed. And it may be that the more scripted rezzing a sim does, the more likely it will fall into the "slow rezzing" trap and need a restart.

If you are able to restart the region, it would be interesting to know if the game delay returns to normal at least for a while. I want to keep it out but if it is an issue I will just rez it upon requests. I like to preserve SL game history ... it's a weird hobby I know. 

It was reset a few days ago after the server updates. But I called for another reset to see if that fixes the issue. Also asked them if they can identify anything I might have out that might be causing issues. 

I imagine my Settlers of Second Life Game which is such an SL antique the scripts don't even have aquired dates on them. It was also created at a time when scripting was much more limited. Basically it is 255 prims and the vast majority of the prims have scripts in them. I don't think they had the ability to affect linked prims back then and had to send messages between prims.

[Sorina Garrigus]

I was able to get LL to do a sim reset. The Linden responding also opted to move the region to a new host in case that was the issue. The games affected were working as normal right after the reset. So was't a change to the updates after all