Deploys for the week of 2011-11-07

[Oskar Linden]

It is November! That means that it is almost Christmas. As we approach the holidays in the US our release schedule will lighten. More people take vacations and we have strict No Change Windows during certain periods. I will keep you updated here with any changes we do have.

 

Second Life Server (main channel)

BlueSteel's refactored voice code project is being promoted this week. We fixed some bugs and cleaned up some code. There were some parcel issues with voice dots not showing up. There is more info in the release notes.

2011-11-08, 5:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/11

 

Second Life RC BlueSteel

This contains a project with enhanced threaded region crossing code. You may or may not see region crossing enhancements. This code is a prototype of an infrastructure we may build on in the future. There shouldn't be a noticeable increase in region crossing failures.

2011-11-09, 7-11:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/11

 

Second Life RC LeTigre

This is the same "Maint-server" project as last week.

• Bug Fixes
  • SVC-5927 Temp on Rezzed objects get queued
  • SVC-7360 Driving a vehicle over into a full region gives strange error message: You can't enter this region because the sebehavior is full
  • SVC-7379 For group notices groupID is being sent in the AgentID field
  • SVC-7343 llMinEventDelay Bug
  • SVC-7354 Simulator fails to load notecard asset (Intan won't read config card)

• New Bug Fixes
  • SCR-234 Loops throttled by a short sleep run twice per frame on Letigre
  • SVC-7397 impossible to enter the sim of blossom at elven forest realm. Nobody can enter the sim. message is "failed to grant capabilities"

2011-11-09, 7-11:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/11

 

Second Life RC Magnum

Again this week this project contains an update to the Havok Engine to keep us inline with the latest updates from them. There are no features or bug fixes and there should be no impact to vehicles or physics. Please file a jira if there is.

• Bug Fixes
  • SVC-7424 Error rezzing certain mesh objects: "Unable to rez object because its mesh data is invalid."
  • Fix for root-prim meshes from an 'old server' not permitted to enter a 'new server'

• Upgraded physics engine to Havok 2011.2. This brings in the latest work from Havok. We do not expect any noticeable change from this upgrade, but please let us know via jira if you see any new physics related issues.
• Implemented llSetKeyframedMotion (http://wiki.secondlife.com/wiki/LlSetKeyframedMotion)

2011-11-09, 7-11:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/11

 

We will be monitoring this thread during the next week so please feel free to post issues that you feel have been introduced by the new code. Please file a JIRA for issues you find and post the JIRA link into this thread. It really helps us out. When determining if issues are relevant or not research is key. Tracking down exactly the right situation where an issue is occurring greatly speeds up the development process to get fixes in place.

I appreciate your help. Have a good week!

 

__Oskar

 

p.s. If you are interested in helping test SecondLife in beta please join the group "Second Life Beta" in-world. We also have an email list where we communicate upcoming projects and how you can help. ( https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta ) Once a week we meet on ADITI to discuss new features, new bugs, new fixes, and other fun stuff. You are more than welcome. Information is here: https://wiki.secondlife.com/wiki/Server_Beta_User_Group

[Ayesha Askham]

Oskar

Approaching a holiday?? I know you folks in the USA have Thanksgiving this month, but Christmas (or whatever cobbled-up non-religious name the PC mob wish to give the time) is six weeks away!

I suppose there is Hanukkah, Diwali is past, as is Ede, but come on guys, how many weeks' holiday do you get?:matte-motes-big-grin-wink:

No, I just want this TD spike issue sorted out, and as soon as possible, and Yes...I will be patient while you do it, just ...please do it!

 

Edit:  OK so I can't count!:matte-motes-zipped:

 

[Ayesha Askham]

Oskar

Approaching a holiday?? I know you folks in the USA have Thanksgiving this month, but Christmas (or whatever cobbled-up non-religious name the PC mob wish to give the time) is six weeks away!

I suppose there is Hanukkah, Diwali is past, as is Ede, but come on guys, how many weeks' holiday do you get?:matte-motes-big-grin-wink:

No, I just want this TD spike issue sorted out, and as soon as possible, and Yes...I will be patient while you do it, just ...please do it!

 

Edit:  OK so I can't count!:matte-motes-zipped:

 

[Void Singer]

there's about a dozen holidays this time of year, including the "big four", of thanksgiving, hanukkah, christmas, and new years.... all of which are traditional times to take off work and spend time with family....

and while the first of them isn't for another 2 weeks, travel times means about one week of full workforce left, by which time any major changes that require multiple rollouts need to be done so they aren't left hanging disastrously in the middle.

[Void Singer]

there's about a dozen holidays this time of year, including the "big four", of thanksgiving, hanukkah, christmas, and new years.... all of which are traditional times to take off work and spend time with family....

and while the first of them isn't for another 2 weeks, travel times means about one week of full workforce left, by which time any major changes that require multiple rollouts need to be done so they aren't left hanging disastrously in the middle.

[Flip Idlemind]

The joke is that the "Christmas Season" gets longer and longer every year...and it's actually true. Early November and the Holiday shopping commercials have already started. Pretty much as soon as it's not Halloween anymore, it's Christmas.

[Flip Idlemind]

The joke is that the "Christmas Season" gets longer and longer every year...and it's actually true. Early November and the Holiday shopping commercials have already started. Pretty much as soon as it's not Halloween anymore, it's Christmas.

[Oskar Linden]

---

Flip Idlemind wrote:

The joke is that the "Christmas Season" gets longer and longer every year...and it's actually true. Early November and the Holiday shopping commercials have already started. Pretty much as soon as it's not Halloween anymore, it's Christmas.

---

Does that mean you bought me my presents already Flip?

__Oskar

[Oskar Linden]

---

Flip Idlemind wrote:

The joke is that the "Christmas Season" gets longer and longer every year...and it's actually true. Early November and the Holiday shopping commercials have already started. Pretty much as soon as it's not Halloween anymore, it's Christmas.

---

Does that mean you bought me my presents already Flip?

__Oskar

[Void Singer]

uh! no fair... flip never gives ME presents!!! =D

(nice to see a sense of humor in these threads again though =)

[Void Singer]

uh! no fair... flip never gives ME presents!!! =D

(nice to see a sense of humor in these threads again though =)

[Ayesha Askham]

Oskar

Now, everyone knows that things are not going well with the roll again...but....

While I realise that we are lucky to be back online at all, what is the rationale behind moving our sim to LeTigre?

"You are at 257370.0, 321842.3, 901.8 in Woods of Heaven located at sim21110.agni.lindenlab.com (216.82.30.144:13025) Second Life RC LeTigre 11.10.28.244134."

We were on sim 5338 Main Server and had been on 4 or 5 thousand series sims all year...now 21110?

Please don't anyone think I'm bitching, I'm just pleased to be online, 'cos running RLV if my home isn't online (where I last logged out) I cannot log in!

I just wonder why LeTigre?

[Sayrah Parx]

Are you able to make the maintainance restarts pop up a dialog message for the countdown, like they used to before the change about 6-8 weeks ago?  After the change,  it only shows countdown text in local chat which is easy to miss and doesn't flash the taskbar.  Ideally it would show both.

[Oskar Linden]

---

Sayrah Parx wrote:

Are you able to make the maintainance restarts pop up a dialog message for the countdown, like they used to before the change about 6-8 weeks ago?  After the change,  it only shows countdown text in local chat which is easy to miss and doesn't flash the taskbar.  Ideally it would show both.

---

That sounds like a vwr bug. I'll pass that info on to the team. I agree with you that it should be an intrusive notice. Would you mind filing a jira for it?

Thanks!

__Oskar

[Oskar Linden]

---

Ayesha Askham wrote:

Oskar

Now, everyone knows that things are not going well with the roll again...but....

While I realise that we are lucky to be back online at all, what is the rationale behind moving our sim to LeTigre?

"You are at 257370.0, 321842.3, 901.8 in Woods of Heaven located at sim21110.agni.lindenlab.com (216.82.30.144:13025) Second Life RC LeTigre 11.10.28.244134."

We were on sim 5338 Main Server and had been on 4 or 5 thousand series sims all year...now 21110?

Please don't anyone think I'm bitching, I'm just pleased to be online, 'cos running RLV if my home isn't online (where I last logged out) I cannot log in!

I just wonder why LeTigre?

---

Short answer is "I don't know". A lot of weird stuff happened today. We attempted to balance out region - simulator load. The hope was to make more machines have a batter balance of shared resources instead of having some machines empty and others at full capacity. During the rolling restart some of these re-workings caused some datacenters to run out of spares. This caused the rolling restart queue to get backed up and that makes machines stay down for a while. We had to revert the balancing changes.

After tomorrow's upgrades if you are still on LeTigre I'll get you moved back.

__Oskar

[Oskar Linden]

I updated the notes with the information for the Wednesday morning releases.

__Oskar

[Ayesha Askham]

@Oskar

Thankyou, Oskar for that open and rapid reply.  I know you and your team must be dreading Restart day now, but that explanation is more than good enough!

This issue of server-load is a very germane one to a lot of SL's current woes, perhaps your masters could see that you really cannot squeeze a quart into a pint pot without a lot of spillage and mess.

I'll keep you posted about our server host and channel after the LeTigre restarts today.:smileyhappy:

[Ayesha Askham]

@Oskar

As promised, an update:

About 10am PST our sim was rolled as part of the LeTigre server update.  The situation is now:

You are at 257370.0, 321842.0, 901.8 in Woods of Heaven located at sim21110.agni.lindenlab.com (216.82.30.144:13028) Second Life RC LeTigre 11.11.04.244558.

So we are still on LeTigre.

However the strangeness does not end there.  I have a "Spray" sailboat, made by Lia Woodget, version 5.5, which, after the reintroduction of physics functions after Andrew's security patch killed the existing physics function, sped around our sim like a speedboat, much to the confusion of its creator and the scripter.  Nothing they could do seemed to correct the matter and I could reproduce the effect a couple of weeks later on Magnum sims (at Blackspot Shipyard).

Now it is behaving perfectly again.  So I ask....why?  Is there something in the new release on LeTigre that would explain it?  And moreover, is the boat going to fail again when (and if) we return to the Main Server?

For now, notwithstanding a revelation as to "what gives?" I would like our sim to stay on LeTigre, since I have seen none of those nasty lag spikes in the last 24 hours.

Is that OK with you, Oskar?:smileysurprised:

[Maestro Linden]

Hi Ayesha, the bug you describe sounds like https://jira.secondlife.com/browse/SVC-7413, which was fixed in this week's LeTigre update.  I'm fixing the LeTigre release notes now to include that Jira and the other new fix.

[JoyofRLC Acker]

Regarding the previous post ...

I say this with all due respect, and acknowledging that I am no expert on a lot of this technology, but at a very basic level is this not the kind of thing that should be caugh on testing before the change is released?

My sense is that this kind of thing happens quite a lot.

It's very inefficient, as resources are wasted on the user and the systems side to remedy these "bugs".

I remain to be convinced that in geenral we are doing sufficient testing in the current release regime.   Im not sure that the rationale for the current regime factors in the resources required for adequate testing.   (Basically EVERYTHING that happens on the grid has to be retested EVERY week and more up to 3 times as we have these 3 channels).  In thoery if we could analyse reliably the scope as it were of each change a more structured testing strategy could be employed, but its pretty clear that we have limited succcess in indentifying the scope or potential impact of any given change).

 

 

 

 

[Nalates Urriah]

Your thinking is reasonable. In a perfect world with infinite time and money testing could prevent the problems we see.

In the real world we have contrants and limited resources. Lots of time spent testing delays delivery of the product. If you are in the USA think of the FDA and drug approval. Many dying people go to Mexico to get new drugs the FDA has not yet approved because the FDA's testing is considered overly protective by many. Something similar happens with software developement.

The problem with testing software in SL is the variety of things people do with SL. Previously most testing was done on the Preview Grid then rolled to the main grid. The problem with that is the variety of scenarios there is limited. By comparison to the main grid it is sparse. 

To compensate for that 'sparseness' the release channels were implemented. They contain about 20% of the main grid. There is more variety there and more problems are revealed. Those are fixed and retested. Once they pass they are rolled to the main grid.

The variety in the remaining 80% of the grid is much larger. New problems not found in the release channels are revealed and have to be fixed.

This is just the nature of software developement.

If you think these problems can be avoided and all the problems can be resolved in testing, you are right in theory. But, that thinking ignores the factors of time, money, and especially the variety of things the general public can do to software. Someone has to stike a balance between problems, time lines, and cost. We each have our own ideas about where that balance should be.

In a free society we each get to make our own decisions about how to balance the factors with our projects. In an enlightened society we respect the decisions of others in regard to their projects and anticipate we will be treated with the same respect. 

That does not mean we will agree with their decisions and balance points. Complaining is human nature and we are really good at it. In many ways complains help decide the balancing point. So, free speech is important to the process.

Free speech without respect generates conflict and shows a lack of enlightenment. It means one cannot comprehend what it is to walk a mile in the others shoes.

[JoyofRLC Acker]

Nalates I agree with everything you say (although Im not totally clear as to the relevance of your last paragraph).

[btw yo better hope that  " Lots of time spent testing delays delivery of the product." never gets quoted out fo context LOL].

You are dead right that it is precisely the variety of things that people do that makes testing difficult.

That means the corollary is - minimise the amount of required testing.

So I come back to - is it wise to be relasing fresh bugs every week?  sorry, upgrades.  As opposed to batching things up to bigger, less frequent, releases,  But they would be propoeryl tested.  Or better tested anyway.

Just asking a question.

...

as an aside , its interesting that we now seem to be seeing some focus on the problems which is good.  perhpas they have joined up the dots and realised the contemplated marketing campaign would be much more effective if the inworld experience matches whats being sold.