Jump to content

Deploy Plans for the week of 2020-05-18

Maestro Linden

By Maestro Linden,
in Second Life Server

 Share
You are about to reply to a thread that has been inactive for 1435 days.

Please take a moment to consider if this thread is worth bumping.

Recommended Posts

  • Lindens
Maestro Linden

Maestro Linden

Posted
  • Lindens
Posted

Second Life Server:

https://releasenotes.secondlife.com/simulator/2020-05-08T19%3A15%3A39.541970.html

Scheduled Tuesday 2020-05-19 03:00-09:00 PDT

 

Second Life RC:

https://releasenotes.secondlife.com/simulator/2020-05-15T18%3A31%3A37.542403.html

https://releasenotes.secondlife.com/simulator/2020-05-15T17%3A44%3A47.542391.html

Scheduled Wednesday 2020-05-20 07:00-10:30 PDT


On Region Restarts:
Regions will be restarted if they have been running for more than 10 days on Tuesday (Main Channel) and Wednesday (RCs) regardless of whether or not new code is being deployed, for the general health and well being of the Simulators. Nothing beats turning it off and then on again ... once in a while.

As always please plan events around these maintenance windows. However, if you are hosting an event on Tuesday or Wednesday when we're not deploying new code, you can restart your region ahead of time to avoid interruptions.

  • Like 1
  • Thanks 2
Link to comment
Share on other sites
  • Lindens
Maestro Linden

Maestro Linden

Posted
  • Author
  • Lindens
Posted
1 hour ago, Whirly Fizzle said:

No fix for BUG-228696 - Group notices do not display proper timestamps ?

angry-cat

The current version of the notes at https://releasenotes.secondlife.com/simulator/2020-05-15T18%3A31%3A37.542403.html picked up this BUG jira instead, which is a dupe of BUG-228696:

  • BUG-228743 Group notices get timestamped with the current time

Let's see if I can nudge the notes-generating script to pick up 228696 instead...

  • Thanks 2
Link to comment
Share on other sites
Willow Wilder

Willow Wilder

Posted
Posted
26 minutes ago, Maestro Linden said:

Let's see if I can nudge the notes-generating script to pick up 228696 instead...

But it was originally reported 2 weeks before that at BUG-228562 Group Notice Timestamps don't report accurately Server Channel 540369/ now 541440 when it only affected 1 RC Channel. 

Just felt the need to put that out there, 'cause I'm still grumpy about this.  😉 

  • Like 5
Link to comment
Share on other sites
Qie Niangao

Qie Niangao

Posted
Posted
9 minutes ago, Willow Wilder said:

Just felt the need to put that out there, 'cause I'm still grumpy about this.  😉 

I wonder if there needs to be a better way to convey urgency. Or maybe it was conveyed, and it just takes this long. Or maybe it was an unrecognized showstopper that should have prevented the RC from rolling as a release.

When this was first described to me, it sounded like no big deal... until I actually saw it in my own viewer while sorting through announcements: absolutely crippling!

Anyway, if we need to improve communications about urgency of regression defects like this -- maybe the process needs some attention -- if so, we should make sure the Lab understands.

  • Like 4
Link to comment
Share on other sites
Willow Wilder

Willow Wilder

Posted
Posted (edited)
18 minutes ago, Qie Niangao said:

I wonder if there needs to be a better way to convey urgency. Or maybe it was conveyed, and it just takes this long. Or maybe it was an unrecognized showstopper that should have prevented the RC from rolling as a release.

When this was first described to me, it sounded like no big deal... until I actually saw it in my own viewer while sorting through announcements: absolutely crippling!

Anyway, if we need to improve communications about urgency of regression defects like this -- maybe the process needs some attention -- if so, we should make sure the Lab understands.

Possibly. Though it wasn't exactly urgent at the time it was reported. It was an annoyance with a relatively inconvenient workaround. Everyone thinks their bug report is important and high priority, and I do understand the need for prioritizing, probably better than many. My report was indeed marked as "minor", but I never anticipated that meant, LL would just let this one run wild on all channels. It's also not just the residents, it creates an unnecessary burden on the support lines, and that bothers me too, when it could have been avoided. 

Edited by Willow Wilder
  • Like 2
Link to comment
Share on other sites
Qie Niangao

Qie Niangao

Posted
Posted (edited)

Not sure how triage works now. Seems they have resident input (besides the text of the jira) and they get it right more often than not I think.

And really not sure how RCs are anointed for release status. It just seems like any fresh regressions should carry a lot of weight in that decision. And maybe they do.

Nuts. Hey, @Oz Linden, maybe this needs a post-mortem on how this particular regression made it to production. It wasn't a roll-back class disaster but it probably shouldn't have happened.

[ETA: It's this regression we're discussing from above]

 

Edited by Qie Niangao
  • Like 1
Link to comment
Share on other sites
Jaylinbridges

Jaylinbridges

Posted
Posted (edited)

Second Life Server:

https://releasenotes.secondlife.com/simulator/2020-05-08T19%3A15%3A39.541970.html

Scheduled Tuesday 2020-05-19 03:00-09:00 PDT

Second Life RC:

https://releasenotes.secondlife.com/simulator/2020-05-15T18%3A31%3A37.542403.html <== only one that fixes timestamp Bug

https://releasenotes.secondlife.com/simulator/2020-05-15T17%3A44%3A47.542391.html

Scheduled Wednesday 2020-05-20 07:00-10:30 PDT

=========================================

2020-05-15T18:31:37.542403
Tuesday, May 19, 2020
Release Notes
Fixes a high profile bug in group notice archive timestamps. Also includes some internal fixes.

Resolved Issues
BUG-228696 Group notices do not display proper timestamps

===========================================

So the timestamp bug fix will only occur on one of the RC channels this week, on Wednesday AM? 

I hope our club sim is the lucky RC channel that gets fixed.  That won't help everyone else on the Main Channel who uses the Group Notices archive to find present events!

 

Edited by Jaylinbridges
  • Like 1
Link to comment
Share on other sites
Logan Elf

Logan Elf

Posted
Posted
25 minutes ago, Jaylinbridges said:

So the timestamp bug fix will only occur on one of the RC channels this week, on Wednesday AM? 

I hope our club sim is the lucky RC channel that gets fixed.  That won't help everyone else on the Main Channel who uses the Group Notices archive to find present events!

 

My guess is that if you're currently on 541970 you'll roll to 542403 (one with the fix), as that RC channel seems to be the one that progresses to Main. 541969 may also roll to 542403, just to give wider exposure to the fix. If you're on 541952 my guess is that will roll to 542391 as that RC channel seems to have been getting the uplift infrastructure improvements.

Link to comment
Share on other sites
Whirly Fizzle

Whirly Fizzle

Posted
Posted
9 hours ago, Willow Wilder said:

As far as I know, @Kyle Linden is the head of QA. 

I may be wrong but I think Kyle is viewer QA lead only.
I'm not sure who is server QA lead now - Caleb?
I think the decision of which server channel gets promoted lies with Product (or Oz) rather then any of the QA Lindens.

  • Thanks 1
Link to comment
Share on other sites
KT Kingsley

KT Kingsley

Posted
Posted

Am I mistaken, or did my region just shut down, update and restart in just under two minutes?

Quote

 

UTC 2020-05-19 14:33:54 Alert: server did not respond.

UTC 2020-05-19 14:35:42 Region restart detected. Simulator updated to version 2020-05-08T19:15:39.541970.

 

 

Link to comment
Share on other sites
Oz Linden

Oz Linden

Posted
Posted
13 hours ago, Qie Niangao said:

Not sure how triage works now. Seems they have resident input (besides the text of the jira) and they get it right more often than not I think.

And really not sure how RCs are anointed for release status. It just seems like any fresh regressions should carry a lot of weight in that decision. And maybe they do.

Nuts. Hey, @Oz Linden, maybe this needs a post-mortem on how this particular regression made it to production. It wasn't a roll-back class disaster but it probably shouldn't have happened.

The build that the group date problem was found in was the same one that made name changes possible. I don't recall whether or not we discussed blocking that release because of the group notice date problem, but I suspect that if we did we'd have decided that getting the name changes out was more important. We do normally weigh regressions fairly heavily, but everything is a tradeoff - there are few hard-and-fast rules.

In general, even a fairly important problem normally takes a couple of weeks to get from the developer to the main channel on the grid. Even after the fix is coded and tested by the developer (the time for which can vary quite a lot), it has to go to QA to be tested and the build has to get a general QA pass to look for regressions; depending on what else is in the queue (more crowded than usual lately due to uplift-related work) that can take from a day to a few days. After that, it goes into an RC channel on a Wednesday. Depending on the level of risk we think the build has (based on what changes are in it), that might be a tiny channel with only a few regions in it to one that's a substantial part of the grid. If it's a riskier change, then after a week in a small channel it will normally be bumped up to one or more of the larger channels and will spend at least a week there. Normally, only a release that's been in at least one of the larger RC channels is eligible to be promoted to the main channel (at the same time, those changes are merged into the builds for all the RC channels, so at that point the change is everywhere).

The decisions about what builds should go on which channels is handled by a team composed of representatives from Product, Development, QA, Operations, and Support; it's never any one person.

 

  • Thanks 5
  • Confused 2
Link to comment
Share on other sites
Jaylinbridges

Jaylinbridges

Posted
Posted
On 5/19/2020 at 3:37 AM, LoganPryor said:

My guess is that if you're currently on 541970 you'll roll to 542403 (one with the fix), as that RC channel seems to be the one that progresses to Main. 541969 may also roll to 542403, just to give wider exposure to the fix. If you're on 541952 my guess is that will roll to 542391 as that RC channel seems to have been getting the uplift infrastructure improvements.

We were on 541970, and we rolled this morning to 542391, which does not have the time/date fix (:

All the RC channel regions I visit are on 542391 now.  Still looking for an RC public region on 542403, so I can at least sent an alt there to check group notices. 

Seems they put the time/date fix on the fewest number of RC regions.  Would be nice if they would reveal RC channel names too, but I guess they change these so often now they would get those wrong.

 

Link to comment
Share on other sites
  • Lindens
Maestro Linden

Maestro Linden

Posted
  • Author
  • Lindens
Posted
36 minutes ago, Jaylinbridges said:

We were on 541970, and we rolled this morning to 542391, which does not have the time/date fix 🙂

All the RC channel regions I visit are on 542391 now.  Still looking for an RC public region on 542403, so I can at least sent an alt there to check group notices. 

Seems they put the time/date fix on the fewest number of RC regions.  Would be nice if they would reveal RC channel names too, but I guess they change these so often now they would get those wrong.

 

About 10% of the grid is on the 542403 RC right now; if you'd like to try the fix, these are a few example mainland regions on that version:

  • Davis Gulf         
  •  Portage Bay        
  •  Falmouth           
  •  Chishonganiro      
  •  Bay City - Rollers 
  •  Bouraq             
  •  Sandbox Island     
  •  Shamwari           
  •  Hiltrude           
  •  West Haven   
  • Thanks 2
Link to comment
Share on other sites
You are about to reply to a thread that has been inactive for 1435 days.

Please take a moment to consider if this thread is worth bumping.

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
 Share
Go to topic listing
×
×
  • Create New...