Deploy Plan for the Week of 2020-11-09

[Mazidox Linden]

Second Life Server

https://releasenotes.secondlife.com/simulator/2020-11-09.551942.html

Deploying version 2020-11-09.551942. Please note not all regions that are in "Second Life Server" will be uplifted this week. We do anticipate this deployment pushing us over the 50% mark for regions in the cloud, however.

Will be deployed from 3 AM to 9 AM Pacific Standard Time on 2020-11-11

Second Life RCs

https://releasenotes.secondlife.com/simulator/2020-11-09.551942.html

Updating existing uplifted simulators to 2020-11-09.551942

Will be deployed from 7 AM to 10:30 AM Pacific Standard Time on 2020-11-10

On Region Restarts

We are (still) currently in the process of uplifting the Grid Poking Bot that handles automatic restarts for regions with high uptime, so you may not see them this week.

 

Edited November 10, 2020 by Mazidox Linden
Added release notes for 551942

[Miller Thor]

@Mazidox Linden Whats the content of the Update 2020-11-09.551942 please? 

[Oz Linden]

8 minutes ago, Miller Thor said:

@Mazidox Linden Whats the content of the Update 2020-11-09.551942 please? 

This update has a couple of cloud configuration changes; these may improve some of the performance metrics, but otherwise should not be anything user visible.

 

[Miller Thor]

@Oz Linden Thanks Oz. But it's interesting for an IT pro and long-established region owner. 😉

Edited November 10, 2020 by Miller Thor

[JoyofRLC Acker]

Has the maintenance ended?  Quite a few sims in Blake arte not opertaional (Jones Locker, Beaufort to name but two ) you can TP in but not sail in.

Also, any issues elsewhere?

Edited November 10, 2020 by JoyofRLC Acker
additional question

[arabellajones]

24 minutes ago, JoyofRLC Acker said:

Has the maintenance ended?  Quite a few sims in Blake arte not opertaional (Jones Locker, Beaufort to name but two ) you can TP in but not sail in.

Also, any issues elsewhere?

I couldn't get into Beaufort, Kendra, or Cattewater. Time of testing is 11:30 to 11:45

[JoyofRLC Acker]

We have filed tickets, with this response "We have escalated the regions with the trouble you have noticed and hope they will resolve the issue soon. Is there anything else that Support could help with today, as we are not able to alter anything in development?"

i wasnt aware that the updates to the main grid are "in development" LOL

my basic question is - how is this update going?   does not seem like "This update has a couple of cloud configuration changes; these may improve some of the performance metrics, but otherwise should not be anything user visible"

[Vicious Hollow]

39 minutes ago, JoyofRLC Acker said:

Has the maintenance ended?  Quite a few sims in Blake arte not opertaional (Jones Locker, Beaufort to name but two ) you can TP in but not sail in.

Also, any issues elsewhere?

There's also  - still -  the animation lock up issue (steering and position change animation stops when the boat leans, yet the boat still functions and so forth ) . Haven't seen it happen out at Blake 'as' much as I have at Eden/Fruit Island and Bellisseria. There was a script "Wind Compass" creator Kim Farleigh was working on, that seemed to help, out at Blake, but not anywhere else. But as noted it hasn't seemed 'as' bad out there as far as that issue went.  

[JoyofRLC Acker]

Cattewater, Kendra  still blocked (from the S) at 12:05 SLT

[arabellajones]

2 hours ago, JoyofRLC Acker said:

Cattewater, Kendra  still blocked (from the S) at 12:05 SLT

Don't know anything useful, but the issue has gone to the engineers. So it must have needed more than a restart.

[Mazidox Linden]

Internal networking issues, we think we resolved them all. Please let us know if you find any more instances where region crossing/neighbor regions (not teleports) have problems.

[JoyofRLC Acker]

As of last night Blake Sea was looking good.  Will be sailing some more there in a little while.  Will keep an eye out.   Thanks for the feed back, its appreciated.

[Phate Shepherd]

Anyone else have issues with L$ balance not showing since the latest restart? Also, map won't load.

You are at 109.7, 179.1, 1002.6 in Kilauea located at ec2-54-244-176-238.us-west-2.compute.amazonaws.com (54.244.176.238:13025)
SLURL: http://maps.secondlife.com/secondlife/Kilauea/110/179/1003
(global coordinates 230510.0, 246963.0, 1002.6)
Second Life Server 2020-11-09.551942

[AlettaMondragon]

1 hour ago, Phate Shepherd said:

Anyone else have issues with L$ balance not showing since the latest restart? Also, map won't load.

You are at 109.7, 179.1, 1002.6 in Kilauea located at ec2-54-244-176-238.us-west-2.compute.amazonaws.com (54.244.176.238:13025)
SLURL: http://maps.secondlife.com/secondlife/Kilauea/110/179/1003
(global coordinates 230510.0, 246963.0, 1002.6)
Second Life Server 2020-11-09.551942

 

I haven't had map and other data issues today, but a few weeks ago, after the first round of region uplifts, there was an issue with "communication of uplifted regions", which resulted in symptoms you mention.

What I am noticing in an increasing rate, and apparently now mostly in regions that got uplifted this week, is the "slow asset loading" issue that @animats reported, so that objects, and textures on objects take much more time to render, and the other issue that is apparently getting more and more serious is that profiles fail to load. Both personal profiles and group profiles, especially at crowded places, but sometimes my own profile doesn't load even if I'm alone in a region (so it's not just lag). At crowded places, even the radar doesn't manage to load people's age data. Trying to open the parcel details of a parcel that is in a neighboring region results in weird artifacts as well, such as not showing any info, showing the parcel where you're at, or the last parcel's info that you checked, or even your home parcel's info if you haven't opened any land info since you logged in. To me it seems like all of these issues were more prominent on datacenter hosted regions, but now the uplifted ones show these symptoms quite as much, so I'd suspect the source of these all isn't in the uplift of a sim, but of something else (other services).

I've experienced relatively much more teleport issues as well (compared to my previous rate of 1 failed TP out of 500), with about 10 failing TP attempts out of 60, but out of all, only 1 resulted in getting disconnected, the other ones were recoverable by cancelling, waiting 3 seconds and then attempting to TP to the same place again. Some of them were uplifted regions.

[Logan Elf]

I've noticed group tag changing can take several seconds to occur sometimes. 

[arabellajones]

I am wondering if some of the early reports, after any roll-out, are essentially misleading. The roll-out process has always put a fair load on the network. Some of that will be changing but regions were moving to the Cloud all day. According to status messages, from 03:00 to 15:00 Pacific time. Which is less than 12 hours ago, as I type.

There's the normal daily load pattern as well. The end of that time block is when load problems are more likely any day, both the data-centre-specific network and the USA in general. That problem on Tuesday looked very different to any of my experience with lag. But how can you tell?

We do have some tools. There are enough data centres in Phoenix that you can get some useful general indicators with tools such as speedtest.net  but I am not sure what will be able to replace that (I did find that Project Sansar had significant AWS use from the start). There are ways, but what is normal? It needs time to learn what is usual.

[Tymus Tenk]

Following the server deployment / uplift of November 11th, all of my 13 Calas Galadhon Park regions (connected) are split between LL AGNI simhosts and AWS, and are now experiencing significant access issues for non Estate Manager users since last night. Some teleports to my regions are being diverted to two other region near my estate on the map.  Some tp's are simply getting are getting an 'no valid parcel'.  One has no problem crossing the sim lines either flying or walking. I submitted a ticket last night with more details. A pretty serious issue.  I submitted a ticket explaining all of this last night. Hopefully at least the mapping/tp issues will be fixed soon.

 

[arabellajones]

23 hours ago, arabellajones said:

I am wondering if some of the early reports, after any roll-out, are essentially misleading. The roll-out process has always put a fair load on the network. Some of that will be changing but regions were moving to the Cloud all day. According to status messages, from 03:00 to 15:00 Pacific time. Which is less than 12 hours ago, as I type.

This pic is for Thursday, but it shows the general pattern. The shaded block is the 12 hours which starts at 3am Pacific Time, the times shown are for Europe, 9 hours ahead. The Lindens will have other data on what is going on, this uses the concurrencu figure that most viewers when they start up.

Does the concurrency matter? It's always a bit higher over the weekend, and there are some things I am reluctant to do near the peaks. There are various groups which arrange vehicle-driving events, and region crossing when concurrency is high is usually bad. I reckon that peak and the following ledge is driven by residents in the Americas, but there is all the rest of the world.