Deploy Plans for the week 2019-08-19

[Bugsly Linden]

Second Life Server:

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/19#19.08.06.529800

Second Life RC BlueSteel:

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/19#19.08.19.530199

Scheduled Wednesday 2019-08-21 07:00-10:30 PDT

Second Life RC LeTigre:

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/19#19.08.19.530199

Scheduled Wednesday 2019-08-21 07:00-10:30 PDT

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/19#19.08.21.530270

Scheduled Thursday 2019-08-22 07:00-10:30 PDT

Second Life RC Magnum:

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/19#19.08.19.530199

Scheduled Wednesday 2019-08-21 07:00-10:30 PDT

On Region Restarts:

Regions will be restarted if they have been running for more than 10 days on Tuesday (Main Channel) and Wednesday (RCs) regardless of whether or not new code is being deployed, for the general health and well being of the Simulators. Nothing beats turning it off and then on again ... once in a while.

As always please plan events around these maintenance windows. However, if you are hosting an event on Tuesday or Wednesday when we're not deploying new code, you can restart your region ahead of time to avoid int err up tions.

Edited August 22, 2019 by Bugsly Linden
Added another release for LeTigre

[Qie Niangao]

Well this is disappointing. Tussock ran great for a week after restart on 06.529800 averaging about 6 msec Spare Time (as I mentioned in last week's deployment thread), but is now showing no spare time and modest script lag with the script improvements in 19.530199. 

I'm very aware that this isn't easy to interpret. For all I know, the script improvements may be a stellar success overall, but in the case of individual regions, there's something about how/when/where the sim is restarted that utterly swamps the effects of improvements in the software itself. It's great to be studying the code for what could be better, but I really think we need some data collection by Operations about what these sims are doing in the wild, to know why a region's performance varies so widely from one restart to another, with no changes to contents.

[Aishagain]

I don't know if this is relevant to Qie's issue, but I see that the GSP is telling us that LeTigre and ONLY LeTigre is to have a further roll tomorrow.. Now what?

[Lucia Nightfire]

1 hour ago, Qie Niangao said:

Well this is disappointing. Tussock ran great for a week after restart on 06.529800 averaging about 6 msec Spare Time (as I mentioned in last week's deployment thread), but is now showing no spare time and modest script lag with the script improvements in 19.530199. 

I'm very aware that this isn't easy to interpret. For all I know, the script improvements may be a stellar success overall, but in the case of individual regions, there's something about how/when/where the sim is restarted that utterly swamps the effects of improvements in the software itself. It's great to be studying the code for what could be better, but I really think we need some data collection by Operations about what these sims are doing in the wild, to know why a region's performance varies so widely from one restart to another, with no changes to contents.

I also had a region gain a whole 3+ ms of available timing when it started up on 529800. I'll be sad to lose that next version change.

For a minute I thought LL pulled a switcheroo with 529800 being the drug and 530199  being the placebo.

[MBeatrix]

11 hours ago, Qie Niangao said:

[...]

I'm very aware that this isn't easy to interpret. For all I know, the script improvements may be a stellar success overall, but in the case of individual regions, there's something about how/when/where the sim is restarted that utterly swamps the effects of improvements in the software itself. It's great to be studying the code for what could be better, but I really think we need some data collection by Operations about what these sims are doing in the wild, to know why a region's performance varies so widely from one restart to another, with no changes to contents.

Tuesday, for the rolling restarts, Innsmouth was restarted twice by the Linden crew. After the second restart, the region was showing Scripts Run around 20%. Later on, a region admin restarted it once more and Scripts Run went up to around 85%. While I'm writing this, they are now around 60% average...

[Aishagain]

This week LL are testing our patience.  Wirtz region on LeTigre RC channel has so far had 4 (four) restarts, the most recent being an hour ago, at approximately 9:00am, 1:00pm, 5:00pm yesterday and 1:00am today.  We are given to understand that LeTigre is to be rolled again today: I wonder how many restarts we will experience over the next 24 hours?

Last week's abortive roll caused 4 ( four) restarts, so this has caused some of our visitors ( this is an RP parcel) to believe that our parcel no longer exists! We are necessarily dependent on footfall (traffic).

Just WHAT is going on, Linden Lab??

[Miller Thor]

@Aishagain : Should Linden turn back to the complete Downtime where SL was down for 6 to 8 Hours and that everytime when a Maintenance was in progess?

And btw. ………………… not only you are affected by that. Others too and when ist necessary to do 4 Restarts than it is so. Other People also have to wait until ist properly done.  And not only you need or want a good Traffic but a Maintenance witch brings up a better Script Runtime Performance is more important than a few Trafficpoints less on 1 or 2 Days and ist more important for all and not only for 1 Person.. 

Edited August 22, 2019 by Miller Thor

[Ron Khondji]

Yes. Let's go back to Wednesdays grid wide restarts.

Edited August 22, 2019 by Ron Khondji

[Willow Wilder]

7 hours ago, Aishagain said:

This week LL are testing our patience.  Wirtz region on LeTigre RC channel has so far had 4 (four) restarts, the most recent being an hour ago, at approximately 9:00am, 1:00pm, 5:00pm yesterday and 1:00am today.  We are given to understand that LeTigre is to be rolled again today: I wonder how many restarts we will experience over the next 24 hours?

Last week's abortive roll caused 4 ( four) restarts, so this has caused some of our visitors ( this is an RP parcel) to believe that our parcel no longer exists! We are necessarily dependent on footfall (traffic).

Just WHAT is going on, Linden Lab??

If you're going to insist on keeping the region on an RC channel, learn to roll with the rolls. 😉 

 

 

[Aishagain]

@Willow Wilder: Indeed.  We are moving as soon as possible.  We were not asked if we minded when we were moved to LeTigre, we did not have a choice over it.

@Miller Thor:  A couple of extra restarts are nothing to comment on and indeed I didn't last week.  Four for a second week and at the last count two more today are.  I don't expect you to agree and I will lose no sleep over it, but this many restarts is not a competent attempt to remove a blight from SL, it is an uncoordinated mess.

[Willow Wilder]

13 minutes ago, Aishagain said:

We were not asked if we minded when we were moved to LeTigre, we did not have a choice over it.

As the region owner, LL doesn't need to ask your permission. It goes with the territory. However, requesting the region be moved to the main channel is an option which has been pointed out at least a few times. Particularly as you've indicated in the past that the scheduled times are an inconvenience for the Euro community. 

[Aishagain]

@Willow Wilder:  why did you post that?  You stated the Blindingly obvious, and I had already pointed out, after much local debate we ARE moving Servers.  Sadly since we are not the only landowners on the region (there is a tiny parcel owned by one of the most uncooperative and uncommunicative individuals I have ever had the misfortune to have encountered), we needed his cooperation.

Further to my previous posts I now see that with the new server version 19.08.21.530270 script-run is down to below 15%!  We await the next restart/rollback

ETA:  Well, we had a restart (7 now) and surprise surprise, script run is back up to 60%!  So maybe, just maybe, it is going to work, or is it just random fluctuations in the SL aether??  Who knows!

2nd ETA:  After 6 hours of running Wirtz is performing better than it has done for more than 6 months.  Script run will never be 100% on that region (between 9.5 and 10K scripts on occasions) so it is heavily overloaded but it has been running at 60% or thereabouts for all this past 6 hours and TP in and out , while noticeable are no longer rubber banding nightmares lasting several seconds.

It may be just serendipity but maybe, just maybe the issue is being quelled.

Edited August 22, 2019 by Aishagain
additional information

[Willow Wilder]

6 hours ago, Aishagain said:

we needed his cooperation

As it turns out, you and I have a very different understanding of the word "cooperation". 😉 

[Israel Schnute]

I'm not sure  how you get the percentages, but I just logged in to another restart at 7:30 SL time.  I wonder what's going on or what's wrong (or right).  

I feel like my trains that I just had to reset (for the 3rd time in 2 days) are running fast now though.  Those old legacy scripts are good litmus tests on timings imo.

[KT Kingsley]

Ctrl+Shift+1 (Advanced/Performance Tools/Statistics Bar) opens the Statistics floater. The percentages referred to are Scripts Run under Simulator.

(I can't remember what's expanded in the default view, but clicking on items will expand or collapse a section, or cycle through the graphic indicators.)

[Qie Niangao]

5 hours ago, Israel Schnute said:

I'm not sure  how you get the percentages, but I just logged in to another restart at 7:30 SL time.  I wonder what's going on or what's wrong (or right).  

I feel like my trains that I just had to reset (for the 3rd time in 2 days) are running fast now though.  Those old legacy scripts are good litmus tests on timings imo.

Yeah, I wanted to check on Tussock's performance and noticed it had restarted about the same time -- and (from Help / About) it reloaded LeTigre 19.08.06.529800, the earlier "unimproved" release that was showing several milliseconds of Spare Time. Indeed it again has that Spare Time, a substantial improvement over the "internal script improvements" version 19.530199 (although as noted before, that may or may not be related to the software, inasmuch as script performance seems to depend on luck-of-the-draw during restarts).

According to the pinned post at the top of this thread, apparently one of the LeTigre (only) restarts was to a different "internal script improvements" version, 19.08.21.530270, so probably that was what was replaced with the most recent restart.

Not that it should be up to me or anything, but I'm fine with this region getting some restarts to try out new releases on a fairly stable Mainland situation; they don't disrupt much here, so it's a good spot for Release Channel loads.

I just wonder how to engage productively in this endeavor of trying to improve how sims run scripts. Are all data worth gathering already being collected and analyzed by Operations and Development, or might residents with some background in systems contribute any insights if we continue watching what we're able to see? If we could serve a useful role, we might benefit from more information about server architecture and practices. At least I have a few questions.

Maybe that would be asking more than I realize, though: with progress on the move to the cloud being kept under wraps.

[Aishagain]

@Willow Wilder:We needed his agreement in order to have region moved to Main Server.  On a given region all occupants/owners must agree and give their assent.

Regarding restarts yes Wirtz had one at c. 7:30pm SLT.  I have not yet been in-world to assess region performance, but from what I read above I am not hopeful.  Shame; last night looked promising.

[Israel Schnute]

4 hours ago, Qie Niangao said:

Not that it should be up to me or anything, but I'm fine with this region getting some restarts to try out new releases on a fairly stable Mainland situation; they don't disrupt much here, so it's a good spot for Release Channel loads.

I just wonder how to engage productively in this endeavor of trying to improve how sims run scripts. Are all data worth gathering already being collected and analyzed by Operations and Development, or might residents with some background in systems contribute any insights if we continue watching what we're able to see? If we could serve a useful role, we might benefit from more information about server architecture and practices. At least I have a few questions.

Maybe that would be asking more than I realize, though: with progress on the move to the cloud being kept under wraps.

That's my feeling as well.  I have a pretty unique set-up and situation versus the traditional abandoned sim or the estate that's just a store.  I liked being able to claim things like the first sim with EEP and other things I've claimed on my own.  But I wonder if they're getting anything out of it or anything I could notice.  I just keep on trucking and hope I even notice the changes.

Scripts Run 96-99%.  But there is sparetime and 3000 scripts.

In my main grid portion not in LaTigre (both mainland) , 5500 scripts, no spare time, but performance has been great for just over a week, script run at like 75-80%.

[Aishagain]

Quick update on Wirtz.  As I feared we're now on the Server Version rolled to Main this week, 529800.  Script run is back down to 20-25% and our memory leak is back, which means I'll be looking to restart every other day again. What a pity.  Still I guess there was a good reason for the roll back.

[Chic Aeon]

Just wanted to mention that this has been the worst server update for me in ages. Along with several hard crashes (viewer just closes), not being able to teleport out of an area and having to relog into where I want to go ---- I have also had major rubberbanding when building (where item reverts to previous position several times and doesn't stick in the new spot) and I have had wearables change colors when I relogged back to the default that came with the product -- not what I was wearing when I logged out. 

 

The abrupt crashing has stopped, but on Sunday still all the other problems are still with me. This is NOT sim or channel dependent and I have made no changes to my viewer or settings or added software or hardware (you get the idea).   I have had no Windows updates since August 14 and none of these issues showed up until  Wednesday.  

[MBeatrix]

32 minutes ago, Chic Aeon said:

Just wanted to mention that this has been the worst server update for me in ages. Along with several hard crashes (viewer just closes), not being able to teleport out of an area and having to relog into where I want to go ---- I have also had major rubberbanding when building (where item reverts to previous position several times and doesn't stick in the new spot) and I have had wearables change colors when I relogged back to the default that came with the product -- not what I was wearing when I logged out. 

 

The abrupt crashing has stopped, but on Sunday still all the other problems are still with me. This is NOT sim or channel dependent and I have made no changes to my viewer or settings or added software or hardware (you get the idea).   I have had no Windows updates since August 14 and none of these issues showed up until  Wednesday.  

It's been bad not only for you, unfortunately. Besides what you mentioned, I must add sim crashes.

A while ago, I was told that Innsmouth crashed earlier, today. And who knows how many other sims... I'm glad I spent the whole day away.

[Lucia Nightfire]

I've crashed on 19.08.06.529800 six times this week, once just standing in an empty sandbox after having been there for half and hour while the viewer was minimized.

[MBeatrix]

8 hours ago, Lucia Nightfire said:

I've crashed on 19.08.06.529800 six times this week, once just standing in an empty sandbox after having been there for half and hour while the viewer was minimized.

Of course I can't be sure, as LL doesn't speak about it, but I suspect those crashes are due to a momentarily unresponsive sim. I've had that happening on region crossings — the viewer just closes without any warning.

[Chic Aeon]

One more super odd event of the week this morning --- which of course I (hopefully) cannot recreate :D.

I was taking a photo for a blog post and needed to derender a bunch of items because they were causing cast shadows on the screenshot area. I derendered over a dozen things. They "disappeared" as always. I used my movement key to turn around back to the scene and noted that I once again had those shadows. Looking back, all the things I had derendered a couple of minutes before (not just ONE) were back and I had to derender them again.  So like with the rubberbanding copying that snaps back, the viewer isn't remembering  what is happening.  

 

Note that these are problems I haven't seen since my time in Opensim. LOL.  And the derendering not sticking I have never seen before and I derender a lot. 

 

Firestorm 6.0.2 (56680) Feb  9 2019 18:55:39 (64bit) (Firestorm-Releasex64) with Havok support

[MBeatrix]

2 hours ago, Chic Aeon said:

One more super odd event of the week this morning --- which of course I (hopefully) cannot recreate :D.

I was taking a photo for a blog post and needed to derender a bunch of items because they were causing cast shadows on the screenshot area. I derendered over a dozen things. They "disappeared" as always. I used my movement key to turn around back to the scene and noted that I once again had those shadows. Looking back, all the things I had derendered a couple of minutes before (not just ONE) were back and I had to derender them again.  So like with the rubberbanding copying that snaps back, the viewer isn't remembering  what is happening.  

 

Note that these are problems I haven't seen since my time in Opensim. LOL.  And the derendering not sticking I have never seen before and I derender a lot. 

 

Firestorm 6.0.2 (56680) Feb  9 2019 18:55:39 (64bit) (Firestorm-Releasex64) with Havok support

Hmmm... There seems to be some bad communication between the viewer and sims, sometimes.
Lately, I've been experiencing some trouble setting normal and specular maps on stuff. As soon as I close the toolbox, the object(s) forget what I did, requiring several attempts, at times. And I'm running the latest Firestorm viewer...
But de-rendering something should be on the viewer only, right? Or am I wrong?

[EDIT] By "de-rendering stuff" I mean objects, not avatars. But maybe the process is identical to blocking an avatar/account...

Edited August 26, 2019 by MBeatrix
adding a comment