Deploy plan for the week of 2022-01-10

[Maestro Linden]

Second Life Server

The main channel will roll to the same version that it is already on, just to keep the simulators running smoothly.

https://releasenotes.secondlife.com/simulator/2021-12-01.566406.html

Tuesday 2022-01-11 06:00-11:00 PST

Second Life RC:

All of the RC regions will be rolled to a new build, on the same tooling project that they were previously on.

https://releasenotes.secondlife.com/simulator/2022-01-06.567269.html

Wednesday 2022-01-12 07:00-09:30 PST

On Region Restarts:

Given that rolling restarts are happening grid-wide, no simulators with >10 day uptime are anticipated to exist after the rolling restarts.

Edited January 11, 2022 by Maestro Linden
Added release notes link.

[Sayrah Parx]

Just saw a main channel region come back from the restart with the same version in about 4 minutes.  But my restart monitor said it was restarted about 1 hour, 37 minutes ago, and was down for about 1 hour, 37 minutes.

It compares llGetEnv("region_start_time") to the last restart time every 30 seconds, in order to detect restarts.  So the downtime has always been accurate within about 30 seconds.

That seems to suggest that the restart used a backup that was over 90 minutes old, instead of getting a new backup when the region goes offline.  I've never noticed or remember that happening since I've had a monitor set up almost 6 years ago.

[Lucia Nightfire]

19 minutes ago, Sayrah Parx said:

Just saw a main channel region come back from the restart with the same version in about 4 minutes.  But my restart monitor said it was restarted about 1 hour, 37 minutes ago, and was down for about 1 hour, 37 minutes.

It compares llGetEnv("region_start_time") to the last restart time every 30 seconds, in order to detect restarts.  So the downtime has always been accurate within about 30 seconds.

That seems to suggest that the restart used a backup that was over 90 minutes old, instead of getting a new backup when the region goes offline.  I've never noticed or remember that happening since I've had a monitor set up almost 6 years ago.

Unless your monitor device saves and queries(after restart) a timestamp to/from an external database, the best script memory alone can offer you after a restart is the last saved data before a sim state save occurred.

What seems to have occurred is a time loss of over 1.5 hours. Meaning if the region was only down 4 minutes, but your script sees a time difference of over 1.5 hours, the last sim state save had to have occurred over 1.5 hours ago. This is "typically" the max random time period that elapses before a save state is made.

The sim state save point is the point in time where a region is brought back from, excluding other factors, controlled or not.

A sim state save captures(saves) the state of objects, terrain, land ownership, script memory, etc.

Not all restarts, yield instant sim state saves.

Edited January 11, 2022 by Lucia Nightfire

[Sayrah Parx]

Yeah, that's exactly what I mean.  It's supposed to get a new backup after it goes down for a controlled restart.  That's a major security requirement.  Otherwise it would be easy to farm no-copy objects.  I've never seen the backup anywhere close to over 90 minutes old.  So it seems like there was a crash during the restart process, or there was a bug in the restart process that caused it to not get a new backup.  I don't know if we're allowed to mention region names.

[Lucia Nightfire]

41 minutes ago, Sayrah Parx said:

It's supposed to get a new backup after it goes down for a controlled restart.  That's a major security requirement.  Otherwise it would be easy to farm no-copy objects.

To be fair, most users that commonly abuse this know there is a chance a sim state save might not be made during the shutdown and do not wait until a region is in countdown before rezzing things. They do it hours before, else they risk losing things to the ether. Also, the ones that commonly abuse it don't even wait on weekly region restarts. They have their estate renting/owning friends do planned restarts for them repeatedly or they repeatedly crash the sim themself without needing controlled restarts.

This is why the argument of not giving users scripted access to knowing if a region is in countdown mode is silly at best.

41 minutes ago, Sayrah Parx said:

So it seems like there was a crash during the restart process, or there was a bug in the restart process that caused it to not get a new backup.

Sim state saves can fail and fail repeatedly, dragging out the time loss if a restart occurs afterwards. I've seen time losses up to 19 hours without planned rollbacks.

I've mostly seen such times in sandbox regions, so IDK if script lag, physics lag or what is a factor.

Edited January 11, 2022 by Lucia Nightfire

[Profaitchikenz Haiku]

 

 

3 hours ago, Sayrah Parx said:

Just saw a main channel region come back from the restart with the same version in about 4 minutes.  But my restart monitor said it was restarted about 1 hour, 37 minutes ago, and was down for about 1 hour, 37 minutes.

Two of the regions I am keeping tabs on for the pilot user-monitoring study were down for 89 and 149 minutes, based on the two wall-clock times recorded immediately before restart and immediately after.

 

ETA

Those regions were both mainland. I just got the figures back from the private island that is taking part in the study and they were back in 5 minutes.

Does having neighbouring region perhaps make it take longer?

 

Edited January 11, 2022 by Profaitchikenz Haiku

[Jaylinbridges]

I lost 4 no copy breedables after the restart Tuesday morning.  This was on mainland.  The four lost cats were attached to my avatar for an hour before the restart.  I dropped them back on land about 10 minutes before the restart.   It's not my land so I didn't have a region monitor to see the downtime .  But it surely did not restart with a recent safety  backup, as used to be normal.

 

 

 

[Lucia Nightfire]

4 hours ago, Jaylinbridges said:

I lost 4 no copy breedables after the restart Tuesday morning.  This was on mainland.  The four lost cats were attached to my avatar for an hour before the restart.  I dropped them back on land about 10 minutes before the restart.   It's not my land so I didn't have a region monitor to see the downtime .  But it surely did not restart with a recent safety  backup, as used to be normal.

 

 

 

Sim state saves are not guaranteed with every controlled region shutdown and restart.

[Aishagain]

@Lucia NightfireReally?  Why not?  Surely, the implication is, considering the preceeding post from Jaylinbridges, that we are to expect data loss at every restart?  I'm pretty sure that didn't happen pre uplift.  Presumably LL have lost the fine control of the server restart process they used to have at the data centre?

How is that going to be acceptable to users and if users become disenchanted, they will leave and SL will cease.

Edited January 12, 2022 by Aishagain

[Lucia Nightfire]

1 hour ago, Aishagain said:

Surely, the implication is, considering the preceeding post from Jaylinbridges, that we are to expect data loss at every restart?

I don't know how to say it any differently than what I have been.

1 hour ago, Aishagain said:

I'm pretty sure that didn't happen pre uplift.

It's been the case ever since I started tracking region restarts 15 years ago.

1 hour ago, Aishagain said:

How is that going to be acceptable to users and if users become disenchanted, they will leave and SL will cease.

This is nothing new.

[Sayrah Parx]

What Simon Linden said here is still really interesting.  That's what I still think about during every restart.

 

Edited January 12, 2022 by Sayrah Parx
typo

[bigmoe Whitfield]

that topic is from 2012,  I would verify with @Monty Linden or some one that the process is still the same. before taking it with a grain of salt.

[Monty Linden]

Oskar was a bit stressed during that deploy cycle.... 

[Whirly Fizzle]

1.40.2 is still my favorite....    💣

[Monty Linden]

2 hours ago, Whirly Fizzle said:

1.40.2 is still my favorite....    💣

Someday...  someday...  we'll get it right again.  Maybe when our viewer gets pie menus back.

[arabellajones]

Can pie be counted as pudding? (I was involved in an argument about that yesterday, which also tried to deal with cookies.)

[Aishagain]

That all depends on what you mean by "pudding".

One use refers to the generic desert..."how can ye have yer pudding if ye don't  eat yer meat?"

The second use refers to a rich, often baked desert frequently eaten with a sauce.

The 18th century political rhyme "Georgy porgy pudding and pie" would suggest they are different things.

It could also be that I don't know what I am talking about...."Nurse! The screens!"

Exit, stage left, wibbling.

 

Edited January 13, 2022 by Aishagain
Insanity

[Henri Beauchamp]

My home region (Hunburgh) is currently running 2022-01-06.567269 and I noticed (it has been lasting for a few days) that when logging in it, the friends list is not updated (online friends appear as offline in the list), while when logging in in a neighbouring region with a different server version, everything works fine in this respect.

Anyone else noticed it ?

Edited January 17, 2022 by Henri Beauchamp

[Whirly Fizzle]

8 hours ago, Henri Beauchamp said:

My home region (Hunburgh) is currently running 2022-01-06.567269 and I noticed (it has been lasting for a few days) that when logging in it, the friends list is not updated (online friends appear as offline in the list), while when logging in in a neighbouring region with a different server version, everything works fine in this respect.

Anyone else noticed it ?

It sounds like this bug, which as far as I know has never been fixed  BUG-7557 - Online Friends are not showing when login (Once again)

Restarting the affected region should fix it.

Edited January 17, 2022 by Whirly Fizzle

[Monty Linden]

14 minutes ago, Whirly Fizzle said:

It sounds like this bug, which as far as I know has never been fixed  BUG-7557 - Online Friends are not showing when login (Once again)

27 linked Jiras. 

[Aishagain]

@Monty LindenSo...are LL knowingly going to promote a bug to Main Server tomorrow?  After all the fuss in various in-world groups, the noise cannot have failed to register with the Lindens, and now you have heard it from a trusted source in Whirly and Henri.

Come on, it would be stupid to promote a bug that affects nearly all accounts.  Wouldn't it?

Edited January 17, 2022 by Aishagain

[Whirly Fizzle]

26 minutes ago, Aishagain said:

@Monty LindenSo...are LL knowingly going to promote a bug to Main Server tomorrow?  After all the fuss in various in-world groups, the noise cannot have failed to register with the Lindens, and now you have heard it from a trusted source in Whirly and Henri.

Come on, it would be stupid to promote a bug that affects nearly all accounts.  Wouldn't it?

Eh, that bug is over 8 years old. Another year won't hurt 

Edited January 17, 2022 by Whirly Fizzle

[Aishagain]

@Whirly FizzleI am saddened that you are so blasé about this.  It is generating a LOT of issues in SL and if ALL regions are bugged how on earth are we supposed to get the names to show?

Does the issue resolve with a region restart still?

ETA 17.01.2021:  I am corrected by Whirly.  This appears to be a new but outwardly similar bug.

It is surely even more important that it isn't propagated across the Grid.

 

Edited January 18, 2022 by Aishagain

[Lucia Nightfire]

13 minutes ago, Aishagain said:

@Whirly FizzleI am saddened that you are so blasé about this.  It is generating a LOT of issues in SL and if ALL regions are bugged how on earth are we supposed to get the names to show?

Does the issue resolve with a region restart still?

 

 

I thought she was paraphrasing someone from management, personally. 😆

[Whirly Fizzle]

43 minutes ago, Aishagain said:

@Whirly FizzleI am saddened that you are so blasé about this.  It is generating a LOT of issues in SL and if ALL regions are bugged how on earth are we supposed to get the names to show?

Does the issue resolve with a region restart still?

 

 

I think you are fretting about a different bug, which is new & started a few days ago.

I just filed it. BUG-231680 - Friends user names often stuck as (Loading) over the last few days for many users
This one is being complained about a lot in groups.

Edited January 18, 2022 by Whirly Fizzle