Deploy Plans for week of 2019-04-22

[Bugsly Linden]

We are working on the TP & sim crossing disconnect issue, and making several changes over this week.  These may be a little bit more disruptive than our usual grid roll process and we apologize in advance for the inconvenience of those.  We’ll do our best to keep this disruption to a minimum. We have a simulator update which we will roll to BlueSteel and LeTigre on Tuesday.  Depending on the results we will make additional plans for gridwide rolls as soon as practical. We thank you for your patience and fully realize both the urgency and the frustration this has been causing.

 

Monday:

No rolls

Tuesday:

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/19#19.04.22.526534

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/19#19.04.22.526534

Rolls for Second Life RC LeTigre and Second Life RC BlueSteel, ending by about 7 AM Pacific.

Wednesday:

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/19#19.04.22.526534

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/19#19.04.22.526534

Rolls for Second Life RC Magnum and Second Life Server, ending by about 10 AM Pacific

Thursday:

TBD

 

On Region Restarts

Regions will be restarted if they have been running for more than 10 days on Tuesday (Main Channel) and Wednesday (RCs) regardless of whether or not new code is being deployed, for the general health and well being of the Simulators.  Nothing beats turning it off and then on again … once in a while.

As always, please plan events around these maintenance windows.  However, if you are hosting an event on Tuesday or Wednesday when we’re not deploying new code, you can restart your region ahead of time to avoid interruption. 

[Eminy2655]

Things are still not better.  Cannot travel through Blake Sea without constant disconnects.

[ChinRey]

On 4/22/2019 at 10:17 PM, Bugsly Linden said:

These may be a little bit more disruptive than our usual grid roll process and we apologize in advance for the inconvenience of those.

A little bit more disruptive? I didn't know Americans did understatements!

But if you can improve sim crossings, it's more than worth it, so keep up your good work!

[Grumpity Linden]

2 hours ago, ChinRey said:

 A little bit more disruptive? I didn't know Americans did understatements!

But if you can improve sim crossings, it's more than worth it, so keep up your good work!

you know, it may have been initially written by an immigrant...

[Grumpity Linden]

The real answer is, we've been working on this around the clock and, as you may have surmised, it's a really difficult problem to catch.   It's not terribly helpful that the issue is intermittent and therefore we cannot have a bulletproof repro scenario which will save us (and more importantly, all of you) from testing in production.  Until we're able to see a drop in the percentage of failures, we are left scouring the forums for anecdotal reports (hi ).  After the SLS update this morning (this was on LT & BS yesterday), the early numbers are looking promising.  We're waiting with bated breath, while working on additional contingency plans. We will certainly share more detail of how things progressed on the back end once we've got a few good days of metrics! 

[Nalates Urriah]

I've been flying and sailing in Bellissaria this morning. Previously I was getting a disconnect every 3rd to 10th crossing. This morning I haven't had a disconnect. Except when hitting a region that was having a problem. When I would log back in the region would show in red on the map.

So... seems better.

[MBeatrix]

35 minutes ago, Nalates Urriah said:

I've been flying and sailing in Bellissaria this morning. Previously I was getting a disconnect every 3rd to 10th crossing. This morning I haven't had a disconnect. Except when hitting a region that was having a problem. When I would log back in the region would show in red on the map.

So... seems better.

A bit too early to know if it's better or not.
Yesterday, I went through more than 50 regions at very high speed on motorboats without issues, and only two of those regions had gotten the latest server release.

[Sharie Criss]

My TP's are causing logouts about once every 3 attempts. Sometimes right after logging in the first TP causes a logout. It's pretty serious. But even MORE concerning is that today, without getting new server software, my region is performing nearly 4 - 10 times slower than normal (it's hard to quantify.) With the region empty except for me, getting a simple menu from some item like furniture that should be nearly instant is taking nearly 10 seconds. Restarts haven't helped. I'm really close to cancelling my regions and just giving up. I filed a ticket yesterday but apparently there's nobody left at LL to look at it.

[Mazidox Linden]

Was the last time you encountered the disconnect dialog after the roll a few hours ago @Sharie Criss?

[Eminy2655]

I'm in Blake again today.  2nd hour boating with no crashes.  I feel a definite imporovement.  It's currently 4:49 PM SLT.

Edited April 24, 2019 by Eminy2655

[ShamuO77]

I have not had a TP disconnect since my Main Server private region was restarted 12 hrs ago.  And the Teleport progress slider seems to zip instantly to about 90%, a few secs before I am teleported to my houseboat.  Before, TP's I would get stuck right away, with liitle progress in the teleport process, and then sometimes slowly move up to about 1/2 way before I was timed out and disconnected.  TP's are definitely more zippy for me, so far... 

The Houseboat rezzer menu is also back to normal, after the latest region update today.

edits - I keep changing sim to region to keep the LL folks happy

 

Edited April 25, 2019 by ShamuO77

[MasoDevotina]

18 hours ago, Sharie Criss said:

But even MORE concerning is that today, without getting new server software, my region is performing nearly 4 - 10 times slower than normal (it's hard to quantify.) With the region empty except for me, getting a simple menu from some item like furniture that should be nearly instant is taking nearly 10 seconds. Restarts haven't helped. I'm really close to cancelling my regions and just giving up. I filed a ticket yesterday but apparently there's nobody left at LL to look at it.

Exactly what I notice here on mainland at Harriot. Walking is difficult, building and editing stuff nearly impossible, and trying to get a menu is not even worth trying.

Since the rollout on tuesday (?) performance has dropped rock bottom, restarts (3 by now) helped for a limited time (a couple of hours) until Scripts Run drop down as low as 1% with occasional peaks up to 15%. Before it used to be around 50% with occasional drops to 15%.

Currently, with 7 main agents and 13 child agents Time Dilatation occasionally drops as low 0.1, every now and then it goes above 0.95 and is usually moving somewhere between 0.9 and 0.25.

[Aishagain]

I'll give this fix a cautious "thumbs up" on the basis of my time to-ing and fro-ing at Fantasy Faire and a few other regions so far today.  At home the region is much the same as it has been these past few weeks, which is to say not as good as it "should" be (or used to be), but not worse as a result of this roll. 

Edited April 25, 2019 by Aishagain
removing a freudian typo

[MBeatrix]

I've just had the first teleport failure since the grid got the fix. I guess that as it's written on GSP, it's "a fix which should reduce incidence of teleport disconnects." A sort of a band-aid, then.

Still, after a long time stuck on the TP beam and the teleport failure I wasn't disconnected. So, it was a good band-aid. 😄