Oskar Linden
-
Posts
461 -
Joined
Content Type
Forums
Blogs
Knowledge Base
Posts posted by Oskar Linden
-
-
Solo Mornington wrote:Oskar Linden says: "The second step is to restart the region to a simstate that was before the prims got returned. It is a process that is done by hand. There is no way to automate it."Hire me.I'll make a way to automate it.I'm sure there are about a dozen million other people who could automate a rollback from a *release candidate* software rollout as well.Any of those people could also make a few reasonable suggestions about how to write regression tests.
"There is no way to automate it" given the time that we had. It is technically possible to automate, but we don't currently have a system for it. Most simstate rollbacks are one offs and don't need a massive level of automation.
__Oskar
-
The first step was to remove the buggy code. That was the first rollback restart. The second step is to restart the region to a simstate that was before the prims got returned. It is a process that is done by hand. There is no way to automate it. There are more than a thousand affected regions. The Support Team is working as fast as they can. We appreciate your patience.
__Oskar
-
Skell Dagger wrote:The rollback appears to have fixed the script issues (all sit scripts stopped working on one of my two Le Tigre parcels, and door scripts went doolally, too) but the returned prims didn't re-appear in the sim. I'm now in the process of re-placing them all by hand.As I was doing so, I realised that most of the prims that were returned (I had 15 coalesced objects come back to me in a flood) were ones on which I had set prim physics to Convex Hull to save prim count. Buildings were returned, furniture was not.Since all of the returns were because the parcel was ostensibly 'full' (even though, with CH, I still had about 100 prims free) I'm assuming that something rolled out in today's restarts broke the prim-saving of Convex Hull, so my parcel reported as full because - without the CH benefits - there were technically too many prims on it.
The affected regions will be getting a simstate rollback. That is progressing now. If you wait the Support Team will get to your region as soon as it can.
__Oskar
-
Starheart McMasters wrote:Whatever you did just returned every single thing from my parcel to lost and found and is blocking me from entering! Sim owner directed me here. Outrageous, a year's work trashed in an instant!
The regions on LeTigre are getting a simstate rollback to their state before the deploy this morning. It's a lenghty process but we are doing it now. Your region will be back like it was soon.
__Oskar
-
The regions on the LeTIgre channel are being rolled back. There is a serious issue on those.
__Oskar
-
MB Robonaught wrote:Hi OskarI was wondering if you could elaborate on the "Back end infrastructural changes" listed in the release information for Magnum server. I realize it also says "No intentional change to behaviour" and I'm curious in case we spot something non intentional.Thanks
Infrastructural changes could technically manifest with an in-world change, but I wouldn't be able to tell you where to look. If you can notice specific adverse changes to behaviour that are only tied to Magnum and are not noticeable on the other channels we will look into it.
__Oskar
-
Happy October! Is it Christmas yet?
There is no promotion to main channel this week. Since there was no RC rollout last week we do not have anything to promote Tuesday morning. There will be no rolling restart.
Each of the RC's passed QA and will be on each channel.
UPDATE: LeTigre regions exhibited issues that necessitated a rollback. The regions were rolled forward and are on the same code that is on BlueSteel. Affected regions are getting a simstate rollback.
Second Life Server (main channel)
This code will stay the same. Regions will not be restarted.
2012-10-02, 5:00am:Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/12Second Life RC BlueSteel
- This is a maint-server project.
- Bug Fixes
- Fixed some server crash modes.
2012-10-03, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/12
Second Life RC LeTigre
There were showstopper issues on these regions. They have been updated to the same code that is on BlueSteel.
2012-10-03, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/12
Second Life RC Magnum
- This is a maint-server project.
- Scheduled 2012-10-03
- Bug Fixes
- Back end infrastructural changes.
- No intentional change to behaviour.
- Back end infrastructural changes.
2012-10-03, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/12
We will be monitoring this thread during the next week so please feel free to post issues that you feel have been introduced by the new code. Please file a JIRA for issues you find and post the JIRA link into this thread. It really helps us out. When determining if issues are relevant or not research is key. Tracking down exactly the right situation where an issue is occurring greatly speeds up the development process to get fixes in place.
I appreciate your help. Have a good week!
__Oskar
p.s. If you are interested in helping test SecondLife in beta please join the group "Second Life Beta" in-world. We also have an email list where we communicate upcoming projects and how you can help. (https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta ) Once a week we meet on ADITI to discuss new features, new bugs, new fixes, and other fun stuff. You are more than welcome. Information is here:https://wiki.secondlife.com/wiki/Server_Beta_User_Group
This is not a forum thread for off-topic and unproductive ranting. I will no longer tolerate it. This forum thread is to discuss newly found issues in the Release Candidate channels. If you feel the need to have your opinions heard on topics unrelated to new bugs then please start your own thread. If an individual continues to post unproductive and off-topic comments I will take measures to remedy that.
If you are unsure about your posting please read this : "Linden Lab Official:Community Participation Guidelines"
-
We found blocking issues during our pre-RC smoke tests. These issues will block the Wednesday morning RC releases. Regions will not be restarted.
__Oskar
-
Ayesha Askham wrote:OskarUm...2012-09-124??? I know youmeant24!!:smileytongue:
:catsurprised:
-
There is no promotion to main channel this week. Since there was no RC rollout last week we do not have anything to promote Tuesday morning. There will be no rolling restart.
UPDATE: There were blocking bugs found in both the RC's planned for release this week. There will be no releases Wednesday morning. There will be no rolling restarts.
Second Life Server (main channel)
This code will stay the same. Regions will not be restarted.
2012-09-25, 5:00am:Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/12Second Life RC BlueSteel
This code will stay the same. Regions will not be restarted.
2012-09-26, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/12Second Life RC LeTigre
This code will stay the same. Regions will not be restarted.
2012-09-26, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/12Second Life RC Magnum
This code will stay the same. Regions will not be restarted.
2012-09-26, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/12We will be monitoring this thread during the next week so please feel free to post issues that you feel have been introduced by the new code. Please file a JIRA for issues you find and post the JIRA link into this thread. It really helps us out. When determining if issues are relevant or not research is key. Tracking down exactly the right situation where an issue is occurring greatly speeds up the development process to get fixes in place.
I appreciate your help. Have a good week!
__Oskar
p.s. If you are interested in helping test SecondLife in beta please join the group "Second Life Beta" in-world. We also have an email list where we communicate upcoming projects and how you can help. (https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta ) Once a week we meet on ADITI to discuss new features, new bugs, new fixes, and other fun stuff. You are more than welcome. Information is here:https://wiki.secondlife.com/wiki/Server_Beta_User_Group
This is not a forum thread for off-topic and unproductive ranting. I will no longer tolerate it. This forum thread is to discuss newly found issues in the Release Candidate channels. If you feel the need to have your opinions heard on topics unrelated to new bugs then please start your own thread. If an individual continues to post unproductive and off-topic comments I will take measures to remedy that.
If you are unsure about your posting please read this : "Linden Lab Official:Community Participation Guidelines"
-
Moon Metty wrote:Hi Oskar,Egglebury was indeed restarted, it's now running Second Life RC PF 12.09.07.264510The server version is up to date, but does "RC PF" still exist?I mean, if there were fewer release channels, there wouldn't be so many of them, numerically speaking.
The answer is a bit complicated. We can't at this time easily remove that channel. At a minimum I had the release team update the server code to match trunk. We're in the middle of some datacenter maintennance that has been ongoing for a while now. We don't have the capacity to move RC PF regions off from RC PF without a lot of juggling and multiple region restarts. So for the next few weeks or so RC PF needs to stay, but will be RC PF in name only. It will have trunk code.
__Oskar
-
Moon Metty wrote:Hi Oskar,Last week I asked you about the region Egglebury.Today the region has been restarted, after more than 30 days of (amazingly smooth) uptime.The problem is that Egglebury is still running RC PF 12.07.31.262785, an ancient version.So it looks like one on't cross beams gone owt askew on treadle.
it should be updated to trunk this morning.
__Oskar
-
Marianne McCann wrote:WolfBaginski Bearsfoot wrote:So it looks like one on't cross beams gone owt askew on treadle.Pardon?
It's their only line!(Ah, Monty Python references)NOBODY expects the Spanish Inquisition!
-
I updated the main information. There will be no RC code release tomorrow morning. The code didn't pass QA.
__Oskar
-
We're almost done with the rolling restart this morning. In another hours or so everything should be settled. The issues seen this morning were transient.
__Oskar
-
Yeah, we're working on it. Fun morning!
__Oskar
-
The maint-server intended for the RC channels Wednesday morning did not pass QA and will not be released. They will continue to run the same code.
Second Life Server (main channel)
The code that was on all RC channels last week is getting promoted this week.
- Bug Fixes
2012-09-18, 5:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/12
Second Life RC BlueSteel
There are no changes or rolling restarts in this channel.
2012-09-19, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/12Second Life RC LeTigre
There are no changes or rolling restarts in this channel.
2012-09-19, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/12Second Life RC Magnum
There are no changes or rolling restarts in this channel.
2012-09-19, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/12We will be monitoring this thread during the next week so please feel free to post issues that you feel have been introduced by the new code. Please file a JIRA for issues you find and post the JIRA link into this thread. It really helps us out. When determining if issues are relevant or not research is key. Tracking down exactly the right situation where an issue is occurring greatly speeds up the development process to get fixes in place.
I appreciate your help. Have a good week!
__Oskar
p.s. If you are interested in helping test SecondLife in beta please join the group "Second Life Beta" in-world. We also have an email list where we communicate upcoming projects and how you can help. (https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta ) Once a week we meet on ADITI to discuss new features, new bugs, new fixes, and other fun stuff. You are more than welcome. Information is here:https://wiki.secondlife.com/wiki/Server_Beta_User_Group
This is not a forum thread for off-topic and unproductive ranting. I will no longer tolerate it. This forum thread is to discuss newly found issues in the Release Candidate channels. If you feel the need to have your opinions heard on topics unrelated to new bugs then please start your own thread. If an individual continues to post unproductive and off-topic comments I will take measures to remedy that.
If you are unsure about your posting please read this : "Linden Lab Official:Community Participation Guidelines"
-
Rollbacks only happen under very specific situations that qualify as emergencies. They are very disruptive to the grid. We do not perform them unless there is a very good reason. Widespread content loss is one example.
__Oskar
-
the RC channels have been updated and will be released Thursday AM.
__Oskar
-
Jack Abraham wrote:Oskar, did some main channel regions restart twice? I got an unexpected restart at home. Not a problem, just curious.
It happens sometimes.
__Oskar
-
WolfBaginski Bearsfoot wrote:I don't recall the last time the whole Grid was effectively running on the same server version. It's an odd feeling.I'm pretty sure the bandwidth problem is over. I suspect, from the improvments in such things as general lag and sim-crossing, that there was being enough traffic generated by the bug to interfere with the desired traffic between Linden Labs servers and the customers. Sim crossing is still far from the best I have seen, but there has been a big improvement in reliability.I wouldn't complain if there wasn't new code tomorrow. I sometimes wonder how you can tell what's worked, without some stability in the server code.
The grid has all run the same code probably about 5 times this year. Only for a week or so though. Our metrics gathering system is sorted via channel. Main channel is large enough to provide reliable 'trunk' data.
__Oskar
-
Sassy Romano wrote:Oskar Linden wrote:Please file a JIRA for issues you find and post the JIRA link into this thread. It really helps us out. When determining if issues are relevant or not research is key.
"...and post the JIRA link into this thread"I know that this isn't the place for this discussion, I have no idea where is anymore but clearly there appears to be derived value from sharing such information but this seems in conflict with the new JIRA process and given that sharing a link will no longer be possible for others to view, how is this going to work?It's also interesting to see the list of fixes each with a JIRA link, how is that going to work going forward?This isnt a whine, i'm genuinely interested in how this conflict will work.Oh right... yeah I'm still trying to figure that out. I guess one could file a bug, post the link here with the title and description. Then others can comment on it.
__Oskar
-
WolfBaginski Bearsfoot wrote:And maybe I spoke too soon.Either they forgot to update the Grid Status, or the rollout has taken 8 hours, and counting.
We had some hardware issues that made the release take a lot longer.
We're actually postponing Wednesday mornings RC roll and doing it Thursday instead.
__Oskar
-
Zaphod Kotobide wrote:Oskar, this isn't a rant, but just a curious thing I observe consistently each Tuesday for some time now. After the restart sweeps through, all the regions surrounding my home region (Samoa) come right back to life, while Samoa can often take upwards of 40 minutes before it comes back. Is that something to be expected due to the channel it's in?
Quentin and Clarksburg are the only ones on an RC. Those won't get rolled until the AM on Wednesday. Samoa, and the other regions around her are all part of the main channel and will get rolled on a Tuesday morning. I'm not sure why Samoa will take longer to come back.
It is back online now with the new version. How long did it take today>
__Oskar
Facebook
Instagram
Twitter
YouTube
Flickr
Deploys for the week of 2012-10-08
in Second Life Server
Posted
We have a main channel promotion this week and most likely Wednesday morning RC deploys.
Second Life Server (main channel)
This will get the code that was on BlueSteel last week.
2012-10-09, 5:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/12
Second Life RC BlueSteel
This channel has some backend fixes to improve database access for large groups.
2012-10-10, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/12
Second Life RC LeTigre
This has the same code as is on BlueSteel.
2012-10-10, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/12
Second Life RC Magnum
This will have the same code as last week. There will be no rolling restart.
This is a maint-server project.
2012-10-10, 7-11:00am: Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/12
We will be monitoring this thread during the next week so please feel free to post issues that you feel have been introduced by the new code. Please file a JIRA for issues you find and post the JIRA link into this thread. It really helps us out. When determining if issues are relevant or not research is key. Tracking down exactly the right situation where an issue is occurring greatly speeds up the development process to get fixes in place.
I appreciate your help. Have a good week!
__Oskar
p.s. If you are interested in helping test SecondLife in beta please join the group "Second Life Beta" in-world. We also have an email list where we communicate upcoming projects and how you can help. (https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta ) Once a week we meet on ADITI to discuss new features, new bugs, new fixes, and other fun stuff. You are more than welcome. Information is here:https://wiki.secondlife.com/wiki/Server_Beta_User_Group
This is not a forum thread for off-topic and unproductive ranting. I will no longer tolerate it. This forum thread is to discuss newly found issues in the Release Candidate channels. If you feel the need to have your opinions heard on topics unrelated to new bugs then please start your own thread. If an individual continues to post unproductive and off-topic comments I will take measures to remedy that.
If you are unsure about your posting please read this : "Linden Lab Official:Community Participation Guidelines"