Deploy Plans for the week of 2019-09-23

[Caleb Linden]

Second Life Server:

https://releasenotes.secondlife.com/simulator/2019-09-13T20%3A04%3A44.530946.html

(2019-09-13T20:04:44.530946)

Scheduled Tuesday 2019-09-24 03:00-09:00 PDT

 

Second Life RC Magnum:

https://releasenotes.secondlife.com/simulator/2019-09-20T23%3A03%3A22.531148.html

(2019-09-20T23:03:22.531148)

Second Life RC BlueSteel and LeTigre:

https://releasenotes.secondlife.com/simulator/2019-09-24T17%3A23%3A00.531200.html

(2019-09-24T17:23:00.531200)

Scheduled Wednesday 2019-09-25 07:00-10:30 PDT

 

On Region Restarts:

Regions will be restarted if they have been running for more than 10 days on Tuesday (Main Channel) and Wednesday (RCs) regardless of whether or not new code is being deployed, for the general health and well being of the Simulators. Nothing beats turning it off and then on again ... once in a while.

As always please plan events around these maintenance windows. However, if you are hosting an event on Tuesday or Wednesday when we're not deploying new code, you can restart your region ahead of time to avoid interruptions.

Edited September 25, 2019 by Bugsly Linden
Showing Release Per Channel

[Aishagain]

Caleb

For the second week in a row the Roll seems to be going on for much longer than anticipated or forecast.  Is this going to be the fashion from now?  I have had unsubstantiated and therefore inconclusive reports that sims are down for much longer than used to be the case this week on Main and last week on the RCs.  May we know if there is a particular non-security reason for this?

I recall Oz saying that the new Roll strategy would result in less downtime for the regions, not more. 

[Caleb Linden]

We should see the effects of the recent changes planned for RC. They include some fixes that may help with the rolls. I will edit the original post as soon we get the final release page generated.

[Oz Linden]

17 hours ago, Aishagain said:

I recall Oz saying that the new Roll strategy would result in less downtime for the regions, not more. 

We're making a series of changes that I hope/believe will result in rolls that will reduce downtime and solve some other roll-related problems (some of which are purely internal). Those changes are being made slowly so that we can measure the effect of each change carefully. The roll this week was deliberately slightly slower than normal (I don't have the numbers yet, but we're collecting data); future changes should reduce the downtime for each region during the roll, but those changes will be made one at a time over the next several weeks. I don't think that any of those intermediate steps will increase the roll time further, but stay tuned...

[Miller Thor]

@Oz Linden Maybe 2 or 3  Minutes it was slower than usual on "Magnum" but that wont let the Earth go down 😉 .

Better a bit slower but working good so we dont need Rollbacks or other Changings. Work needs Time and if someone wont understand it he should come to you and see it live and in Color how much Work it is keeping SL alive.

Edited September 25, 2019 by Miller Thor

[NeoBokrug Elytis]

It hasn't been brought up here, but there are a lot of problems with the latest roll concerning objects being rezzed, http services, and I believe llRegionSayTo().  The most recent bugs on the JIRA highlight a lot of the issues -- but because they're intermittent it's hard to test and nail down exactly what is going on.

A lot of SL content is affected by this, and I think you should consider rolling server versions back.

[MBeatrix]

5 hours ago, NeoBokrug Elytis said:

It hasn't been brought up here, but there are a lot of problems with the latest roll concerning objects being rezzed, http services, and I believe llRegionSayTo().  The most recent bugs on the JIRA highlight a lot of the issues -- but because they're intermittent it's hard to test and nail down exactly what is going on.

A lot of SL content is affected by this, and I think you should consider rolling server versions back.

Not to mention that the inventory service has been slow for weeks... Not always, but enough times along a day to be annoying.

Inventory slowness may be due to a poor quality CDN service, and maybe some of the other issues have the same cause.

Edited September 27, 2019 by MBeatrix

[Oz Linden]

13 hours ago, NeoBokrug Elytis said:

It hasn't been brought up here, but there are a lot of problems with the latest roll concerning objects being rezzed, http services, and I believe llRegionSayTo().  The most recent bugs on the JIRA highlight a lot of the issues -- but because they're intermittent it's hard to test and nail down exactly what is going on.

A lot of SL content is affected by this, and I think you should consider rolling server versions back.

See the Status notice

 

[Oz Linden]

8 hours ago, MBeatrix said:

Not to mention that the inventory service has been slow for weeks... Not always, but enough times along a day to be annoying.

Forgive me for using your post as an example @MBeatrix; I don't want to pick on you at all. We do rely on user problem reports for detecting many kinds of problems, but the quality of the reports make a huge difference in how effective they can be.

This report doesn't have enough information in it for us to even begin to investigate.

• What inventory operations are "slow"?
• Quantify "slow"
• What viewer?
• What regions are you in when they are "slow"?
• When is "for weeks"?

Specifics might allow us to correlate the problem report with changes we've made to the deployed services, including changes to backend systems that may be made at times other then when simulators are updated. Note that if you fill in a BUG report in Jira, and just copy into the Environment for the bug all the information in your viewers About Second Life floater, you'll have provided much of that.

8 hours ago, MBeatrix said:

Inventory slowness may be due to a poor quality CDN service, and maybe some of the other issues have the same cause.

This comment illustrates a common communication problem; this gets a bit geeky, but the better we all understand the same vocabulary, the better we'll be able to help each other to identify and solve problems...

• "Inventory" is a database of items either owned by an agent or inside an object. When you open the inventory floater or the content pane of the build floater, you're looking at that list. But the list does not contain the actual objects - it's essentially just a set of named pointers to the objects.
• "Assets" are the actual objects. They're not stored in the inventory database, they're in a completely different (very large) data store of every object in all of Second Life.

The Inventory database is manipulated by talking to that database through the simulator.  If you move an item from your personal inventory to the contents of an object, for example, you're removing a row from your inventory and adding a row to the inventory of the object, but you're not actually moving any of the data for the object those rows point to.

Most Assets are accessed through the CDN (there are a few exceptions that are managed through the simulator for security reasons).

This is why it's important to understand what operations are "slow" (or have some other problem); some operations are just interacting with the Inventory database, some are interacting just with the Asset store, and others are doing both.

[NeoBokrug Elytis]

1 hour ago, Oz Linden said:

See the Status notice

 

I have made several comments on one of the 4+ JIRA issues relating to this.  I also spent the majority of the day yesterday whittling down which functions and events aren't working.  The issue is intermittent and hard to nail down as to what the primary cause is.  I did notice that the "Scripts Run" stat was much better before the roll, when it had initially became a problem months ago.  After the roll it's doing bad again.

I believe the core of the issue has to do with object to object communications.  Possibly the listen event is broken.

Before I continue Oz: I am not attacking you personally, you're a smart dude who's helped the grid grow in profound ways over these past 16 years.  I am however critical of some of the policies that (to my understanding) you're in charge of.

The average user of SL doesn't know the technicalities of SL, or even know anything about scripting.  However the interactivity of scripts breathe life into SL and make SL what it is.  Because of that most people can only say "It's not working".  Content creators, people who write code in particular are a very VERY small percentage of SL users.  Those who make comments or file JIRA reports are an even smaller percentage of that.  In the past couple days there  have been 4 or more JIRA issues relating to the problems we're having.  I think you should take into account all of the people who are "silent" about what is happening.  I myself am guilty of not reporting bugs on the JIRA, because in the past I have felt my contributions do not help or matter, and that I don't have the time to try to figure things out.  I only report and comment when things are "on fire" anymore.

But -- I think you should make the call to do an emergency server rollback to working versions of the simulator code.  I understand that this is a good opportunity to figure out the problem and try to fix it.  But it is Friday, vast swathes of SL content isn't working and the people who run businesses, create content, and have their own customer base are having a very hard time right now.  We're bearing the brunt of most of the complaints on behalf of LL.  The weekend is almost here, and once 2pm PDT/SLT rolls around, there will be a lot more people on SL.  Second Life has historically had a very hard time on weekends, even when there's no major issues.  I suspect with the concurrency SL will have, and the issues the regions are having now, it's going amplify the issue and make a lot of people very frustrated.  I am planning to cancel at least one of my game events if things aren't better.

All of my regions are on the Main server channel, at least that's what concierge can confirm.  I gotta say that I don't agree with the removal of the RC candidate names on server versions.  This only confuses people, and I think this past rollout is a primary example of why this is a bad idea.  If people complain about being on a RC server because they think it runs worse than a normal server -- I think you just need to bear the brunt of that.  Most mainland folks don't have the option of choosing what server version they're on.  Keep in mind they are paying customers, so they rightfully feel entitled to a working service.

[Aishagain]

@Oz Linden:

I have observed both the Scripted object issue (and have added to the existing JIRA on the matter) and the Inventory "slowness" issue.  Both observed on a Main Server homestead (as described in the JIRA submission)running the current software set (2019-09-13T20%3A04%3A44.530946.html ). In a "typical case I can attach an object without trouble but upon detach nothing happens until I perhaps TP or in extreme cases I relog.  On one occasion I found the item detached when I attached another item  (using ADD not WEAR) to the same attachment point.

The latter is annoying but not critical, the former is a game-breaker.  Please Triage this ASAP and effect a solution...oh and incidentally I echo Neo in his dislike of the removal of the channel ID in the viewer.  Just WHO this is supposed to assist I cannot imagine.

[Whirly Fizzle]

31 minutes ago, Aishagain said:

In a "typical case I can attach an object without trouble but upon detach nothing happens until I perhaps TP or in extreme cases I relog.  On one occasion I found the item detached when I attached another item  (using ADD not WEAR) to the same attachment point.

This sounds like the problem attachment was "ghosted". This has been a problem for many years.
Attachments generally ghost on region change/TP & either:

• The viewer thinks the attachment is attached but the region thinks it's detached - this is the most common form of "ghosting" & when this happens you'll see the attachment worn on your own screen but observers will not see it.  You cannot detach the attachment because the viewer sends the detach request to the region, the region says Nope that thing is not attached so you can't detach it. Also scripts inside the attachment will not work because as far as the region is concerned that attachment is not there.
  Sometimes a further region change will fix this problem because the viewer & region get back in sync & agree what you have attached.
  A relog will always fix it.
  If you happen to use Firestorm, you can save yourself a relog by going to Avatar -> Avatar Health -> Refresh attachments.  This will force a sync up of your attachments between the region & the viewer.
  In the next release of Firestorm (soon (tm)) if the viewer detects your attachments are out of sync, it will auto-refresh them after region change.  This works pretty well & for all intents this ghosting bug will be "fixed" in the next FS realease.
• Or, the viewer thinks your attachment is detached but the region thinks it is attached - in this case you will not see the attachment any longer on your own screen but observers will still see it & scripts in the attachment will still be running.

31 minutes ago, Aishagain said:

On one occasion I found the item detached when I attached another item  (using ADD not WEAR) to the same attachment point.

This is just a side effect of the same thing.
Your attachment was likely ghosted & only visible to you locally but when you attached another item it forced a sync up between viewer & region & the ghosted attachment was detached locally to match the state the region thought your avatar was in.

Edited September 27, 2019 by Whirly Fizzle

[MBeatrix]

5 hours ago, Oz Linden said:

Forgive me for using your post as an example @MBeatrix; I don't want to pick on you at all. We do rely on user problem reports for detecting many kinds of problems, but the quality of the reports make a huge difference in how effective they can be.

This report doesn't have enough information in it for us to even begin to investigate.

• What inventory operations are "slow"?
• Quantify "slow"
• What viewer?
• What regions are you in when they are "slow"?
• When is "for weeks"?

[...]

No worries, Oz, I know that you or anyone from the Lab would never pick on a user. And yes, you are right about my "report".

Please keep in mind that it is an intermittent issue, and it began maybe about a month ago — sorry I can't be precise. Let's see:

What inventory operations are "slow"? Quantify "slow"
detaching or attaching items — many times I have to detach one item at a time, on changing outfits, and for some of them each may take about 20 seconds (maybe a bit more) to detach or attach; and then, suddenly, the issue clears and I can attach/detach a batch of items at once; I've ended up wearing two outfits like it sometimes happened in my newbie days — the one I was wearing and the one I tried to wear, and it's why that many times I just detach item by item;
taking (or copying) an item rezzed in-world back to inventory or dragging one from inventory to rezz on the ground/floor maybe take up to about 30 seconds; it happened to me a couple times copying one of my own builds (all prims created by me) that was rezzed and the copy never went to inventory;
an item delivered by a vendor only shows in inventory after I re-log — this only happened once recently but the region had only 4 avatars, my own included;

What viewer?
Firestorm, latest release;

What regions are you in when they are "slow"?
Most issues happen in Nuggy (Main Channel Server) but I've had slowness rezzing/taking back items at a Premium sandbox (Peritus).

My inventory has 56,480 elements when I write this. All my elements are under protected folders (Objects, Clothing, Body Parts).

[EDIT] I forgot to mention that emptying Trash has the same "slowness" problem, no matter if it contains only one item or half a dozen.

Edited September 27, 2019 by MBeatrix
adding more info

[nerthuschild]

 

https://secondlife.com/my/community/events/edit.php

https://www.dropbox.com/s/v7bawqlhseql5sk/Screenshot 2019-09-27 10.27.09.png?dl=0

So I do not know if this is the right place to get help on this, but I have not been able to do my job is a Club Host posting events for two days now.  this is a notice that I have used for at least six months. I have not changed anything and I cannot see anything that would violate any standards.  so SL is broken and I have not idea what to do

[Whirly Fizzle]

27 minutes ago, nerthuschild said:

 

https://secondlife.com/my/community/events/edit.php

https://www.dropbox.com/s/v7bawqlhseql5sk/Screenshot 2019-09-27 10.27.09.png?dl=0

So I do not know if this is the right place to get help on this, but I have not been able to do my job is a Club Host posting events for two days now.  this is a notice that I have used for at least six months. I have not changed anything and I cannot see anything that would violate any standards.  so SL is broken and I have not idea what to do

By a process of elimination, the word filter objects to the word "like" in the description.
Change Like for another word & it should post.

Also, WTF  lol.

Edited September 27, 2019 by Whirly Fizzle
Can't bloody spell

[Aishagain]

@Whirly Fizzle: No, Whirly, I HAVE seen instances of the issue to which you refer.  This is quite different and on a couple of occasions the item HAS detached so I guess that would simply be considered a delayed effect.  The attachment that is unremovable is not affected by the refresh and is still active until detached.  If, as happened twice it is a box, I can still obtain copies of the contents until it is detached.  I suspect that the mechanism involved is similar and may have some common elements, but it is NOT the same issue.

for further clarification, my inventory on this account is fewer than 25,000, but I use other than the base folders

 

Edited September 27, 2019 by Aishagain

[MBeatrix]

49 minutes ago, nerthuschild said:

 

https://secondlife.com/my/community/events/edit.php

https://www.dropbox.com/s/v7bawqlhseql5sk/Screenshot 2019-09-27 10.27.09.png?dl=0

So I do not know if this is the right place to get help on this, but I have not been able to do my job is a Club Host posting events for two days now.  this is a notice that I have used for at least six months. I have not changed anything and I cannot see anything that would violate any standards.  so SL is broken and I have not idea what to do

I'm not sure, but I suspect that you tried to post a bit late for the scheduled time. It happened to me a couple weeks ago, when I was late posting — it was around noon PST and my event was/is at 2 PM PST.
I think it just happens that the error message is wrong. I wonder if it's the only existing error message, and it covers all cases...

[EDIT] Never mind what I wrote. 😄

Edited September 27, 2019 by MBeatrix

[Whirly Fizzle]

It really is the word "Like"
Try to post any event with just the word Like in the description & "Test" as the title & it gives you the naughty word error.
Change Like to anything else & it posts.
 

[nerthuschild]

OMG Whirly Fizzie you are amazing!  thank you so very very much!

[MBeatrix]

6 minutes ago, Whirly Fizzle said:

It really is the word "Like"
Try to post any event with just the word Like in the description & "Test" as the title & it gives you the naughty word error.
Change Like to anything else & it posts.
 

Oh dear! So that was it?! 😂

[MBeatrix]

DANG!

[Whirly Fizzle]

Just now, MBeatrix said:

DANG!

IT WASN'T ME!

[MBeatrix]

14 minutes ago, Whirly Fizzle said:

IT WASN'T ME!

HAHAHA!

I already reported it to Support.

[EDIT] It's apparently fixed now but only apparently — when browsing events for 2 PM the error comes up at some point.

Edited September 27, 2019 by MBeatrix
adding more info

[Sharie Criss]

I haven't seen this issue mentioned in this thread yet, but I'm seeing increased rate of region crashes since the rollout. Is anyone else noticing the same?

[Whirly Fizzle]

FYI

Lots of info about the recent problems with scripts & rezzing here now: https://status.secondlifegrid.net/incidents/263n6z6w1sp0