Deploys for the week of 2011-05-02 [[UPDATED]]

[Oskar Linden]

The coldest April that Seattle, Washington has had for 120 years is now over! I have high hopes for a warm May and trouble free releases this week. Poor LeTigre has been having some troubles. We pulled last weeks release when it was determined that certain scripted objects would disappear on-rez. The issue has been fixed and will be released this week.

 

UPDATE

After the release Tuesday morning we were notified by support that large numbers of residents were having issues with amaretto horses and other physical objects. After a lot of research and discussion we made the decision to push a fix to each of the three RC channels. Affected regions will have the option to move to the RC's for this week. We will push this fix live next Tuesday.

Second Life Server (main channel)

We are promoting Andrew's "maint-server" project from BlueSteel. The "maint-server" branches are where we collect bug fixes. We fixed quite a few bugs in this branch and we also added a new LSL function. The following issues were fixed in this project.

• Avatar position reverts to original sittarget in simcrossing
• Muting a new-style 'Resident' avatar is ineffective when you're offline
• llTeleportAgentHome on an agent not over your land shouldn't result in the _target_ getting the message
• Mini-map shows multiple dots for one avatar, but avatar is not ghosted
• Vehicles "Jumping" when crossing prims
• Group-owned objects located in private estates cannot send IMs to offline avatars
• llGiveInventory is not working!

Additionally LLRegionSayTo() was added to LSL.

 

2011-05-03, 5:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/11

 

Second Life RC BlueSteel

This project is on all three RC channels. It is an Andrew's "maint-server" project. It includes server crash fixes and other bug fixes. It has the physics change removed.

2011-05-04, 8:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/11

 

Second Life RC LeTigre

This project is on all three RC channels. It is an Andrew's "maint-server" project. It includes server crash fixes and other bug fixes. It has the physics change removed.

2011-05-04, 9:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/11

 

Second Life RC Magnum

This project is on all three RC channels. It is an Andrew's "maint-server" project. It includes server crash fixes and other bug fixes. It has the physics change removed.

2011-05-04, 10:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/11

 

 

We will be monitoring this thread during the next week so please feel free to post issues that you feel have been introduced by the new code. Please file a JIRA for issues you find and post the JIRA link into this thread. It really helps us out. When determining if issues are relevant or not research is key. Tracking down exactly the right situation where an issue is occurring greatly speeds up the development process to get fixes in place.

I appreciate your help. Have a good week!

 

__Oskar

 

p.s. If you are interested in helping test SecondLife in beta please join the group "Second Life Beta" in-world. We also have an email list where we communicate upcoming projects and how you can help. ( https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta ) Once a week we meet on ADITI to discuss new features, new bugs, new fixes, and other fun stuff. You are more than welcome. Information is here: https://wiki.secondlife.com/wiki/Server_Beta_User_Group

 

[Kardargo Adamczyk]

Oskar, thanks for the update...

i hoped the prim jumping was fixed, but if anything it got worse, way worse...

Can you please roll back KJ Racing island, as it is currently impossible to transverse any prims without jumping off the track.

Thanks in advance,

Kardargo

[Astarte Artaud]

Well  status page says everything has been completed with standard server roll but Raptors Reef sim is still unreachable at the moment

[Maestro Linden]

Hi Astarte, are you still seeing the problem?  I just tried to TP to "Raptors Reef", and arrived in the region successfully.  It appears that the sim has been up for just over 5 hours.

[Oskar Linden]

@Kardargo

 

We're investigating the issue.

 

__Oskar

[Maestro Linden]

 

Due to physics issues with certain objects, such as Kardargo's karts and the Amaretto horses, we will undo the original fix for SVC-5880, which entered the "Second Life Server" channel today.  A new server version has been built, which removes the original SVC-5880 fix, and also fixes a crash mode related to object deletion.  We plan to deploy this new version to all 3 server RC channels tomorrow morning.  The other two release candidates hit some last minute issues during beta testing and missed the deadline for RC.

In case anybody wants to check out this week's RC before tomorrow's deploy, it can be found on Aditi (the 'beta grid') right now, with version 'DRTSIM-55 11.05.03.228703'.  This server version on the following regions: Aegean, Adriatic, Azov, Badnarik, and Dore.

 

[airiellabella]

I can not get into second life.

Not sure what the problem is.

I was always able to before.

Can someone help!!!

 

[Oskar Linden]

airiellabella, please contact support.

 - http://support.secondlife.com/

 

__Oskar

[Hitomi Tiponi]

 

---

airiellabella wrote:

I can not get into second life.

Not sure what the problem is.

I was always able to before.

Can someone help!!!

 

---

As well as Oskar's advice you may like to look at the Knowledge Base article for 'Login Failure'

 

[Kurt Ludd]

I can only select and change the properties on 6 or less prims when I go to 7 it greys out the option in edit mode...Cant change them in linked mode also...This seems to apply to prims rezzed before today...and doesn't apply to new ones rezzed after update...

Is this Server related....

I am in the LeTigre group in Malra Region...

 

 

Kurt Ludd

[Argus Collingwood]

 

---

Maestro Linden wrote:

 

Due to physics issues with certain objects, such as Kardargo's karts and the Amaretto horses, we will undo the original fix for SVC-5880, which entered the "Second Life Server" channel today.  A new server version has been built, which removes the original SVC-5880 fix, and also fixes a crash mode related to object deletion.  We plan to deploy this new version to all 3 server RC channels tomorrow morning.  The other two release candidates hit some last minute issues during beta testing and missed the deadline for RC.

In case anybody wants to check out this week's RC before tomorrow's deploy, it can be found on Aditi (the 'beta grid') right now, with version 'DRTSIM-55 11.05.03.228703'.  This server version on the following regions: Aegean, Adriatic, Azov, Badnarik, and Dore.

 

---

Would this cause a horse to fall from a sky pasture to the ground level? We are a LeTigre sim @ Van Eyck and I found a horse in my store.:smileysurprised:

 

[Maestro Linden]

Hi Argus, it looks like van Eyck is in the RC LeTigre channel, which skipped the server version with this physics bug.  I suppose it's possible that a horse in a neighboring region (like "Nick Rhodes", which is in "Second Life Server") escaped from its skybox pen and flew/fell into your store.  For what it's worth, the main issue I saw with the Amaretto horses is that they walk about 3x faster than normal.

[Argus Collingwood]

 

---

Maestro Linden wrote:

Hi Argus, it looks like van Eyck is in the RC LeTigre channel, which skipped the server version with this physics bug.  I suppose it's possible that a horse in a neighboring region (like "Nick Rhodes", which is in "Second Life Server") escaped from its skybox pen and flew/fell into your store.  For what it's worth, the main issue I saw with the Amaretto horses is that they walk about 3x faster than normal.

---

Thanks, but it fell from above lol. The owner is checking on the trouble and the horse now will not move once returned to the skypasture.

 

[Alexi Raynier]

Enough is enough, since the weekly restarts began over a year ago there has been a steady diminshment of the quality of sl services, and a steady and constant interruption of normal sl activity.  This has led to a serious drop in the playing time of established player, the closing of stores which had been economic institituions for year, monsterous inventory problem, a less reliable search, unprecedented lag and teleport problems, and a fantastic array of crashes that would make a beta test look. bad.  What is worse we are paying for all this and recieveing for our money services that we no longer get or even expect, but which were taken for granted two or three years ago.  What we have gotten in return if the merger of the teen grid, windows profiles, designer names, and an array of other "improvements" that were eitehr not asked for or openly asked to be reconsideered by the player community.  This inturn has led to a monsterous depletion of resources and an apparent exodus of players from sl due to this dramatic dimishment of services.

Would it be possible rather than foist more "improvements" on us you to spend a period giving us time to recover from the disruptions you have already created, and during that time concentrate of fixing things that woudl lead to an overall improvement of playing quality like search, lag and perhaps doing something substantial with the massive inventory problems that plague everyone.  I realize that this is probably too much to ask, and that player concerns over game quality is not a high linden priority, but I also know that, unless something is done soon to improve quality dramatically you will loose the veteran players that have built this economy and society and with them SL will cease to be even the shell that it is now becoming.  Please listen to this plea, give it some serious consideration and at the very least give us a period of stability rather than the ongoing disaster that has become sl restarts.

[Jack Abraham]

Alexi, we have asked for them.  "We want less lag."  "We want sim crossings to work."  That's what Oskar's pushing out here.  I tend to agree with you about search (and the effects that's had on the economy), display names, the children's invasion, etc., but that's not Oskar's turf.  You're complaining about restarts, but I'm really not seeing the connection between the weekly restarts and the problems you're describing.

The weekly restarts are a result of the fact that an statistically useless portion of SL activity occurs on the beta grid.  They simply can't test very effectively without rolling things out to the main grid.  Yes, that creates problems, but my perception of things has been that teleports are getting more reliable, sim crossings better, sim crashes less frequent, and scripting tools better at a steady pace.  Oskar, Maestro, and their cohorts are very responsive to problems that they can fix.

[Void Singer]

Like jack, I'm not seeing the weekly restarts being a huge issue... 30mins tops (in most cases under 5) on the lowest concurrency days of the week, and the only problems related to the new server version fixes that you mentioned (teleports and lag) have seen marked improvements.... teleports are now completely faster for heavily loaded avatars and and no longer stalling the region for up to 2 mins. region crossings are almost unnoticeable for lightly loaded avatars, and even the severely overloaded ones rarely crash now. ghosted avatars now are freed on a regular basis (not an ideal fix, but workable), as are regions with ghost prims and other accumulation crud that hasn't been sussed out yet. the vast majority of regions are behaving much better.

yes there have been a few misstarts, that are usually cleared within a day or two, and are almost always limited to the beta channel servers, but overall I've personally seen amazing strides in usability and stability across the board for server behavior.

your other complaints, while they may be valid, are misplaced.... and should be directed towards the departments that handle or are responsible for making those issues. Just like any company, specific departments are only responsible for specific things... and if you want to be heard, you need to beat on the right door... this is not the door you were looking for.

[Void Singer]

if you are using IE9 you'll need to be in compatibility mode to post (Guessing that's why we see a blank)

[Ayesha Askham]

@Void

Durr....only just started with IE9....thanks!

Now...as I was saying.........I take the point about weekly restarts being no big deal - at least we know when to expect disruption, but it is inconveniently slap bang in the middle of the European day, so it eats significantly into my online time, despite  my accessing SL at home.  IF, and it's a big IF the recent restarts hadn't been such borked affairs I'd have no worries - Oskar knows I snipe but I'm fair.

The last two attempts to get software changes out to the grid have been outright failures, this latest affair was painful.  I don't live on LeTigre thank the deity, but if I did I'd be a tad miffed at such underdeveloped software being foisted on me time and again.  I KNOW we need RCs to adequately test new stuff, but these last two are so far off beam I cannot seriously believe that sufficient care was taken during the development phase.

I can look at this in two ways:

1)  Linden Lab are now using less competent software developers, who do not consider the impact of their work on Linden Lab's customers, in order to cut costs,

or

2)  Linden Lab is such a poor employer that they cannot attract better devs.

I don't want to believe either.

Alexi has a tendency, in my opinion to yell rather quickly, but recent events, and the general slow degradation of SL services make it more understandable.

The script failures I've seen lately are to say the least inconvenient, and they are definitely related to changes made to the server software - it smacks of poor checking.  I work in a pressured, high-value industry, we cannot afford to make mistakes of the scale we've seen lately in SL. 

My verdict has to be: Could and Must do better.

[Void Singer]

you forgot option...

3) It's impossible to test use cases in an environemt with so much and so varied (and too often poorly made) user generate content... there's literally no environment like this in the wildwith anything approaching a similar scale and scope.

number 3 necessitates that a representative portion of the vast population are used to test for anything missed in the development phase. unfortunately this affects scripted items the most.

that doesn't mean that improvements in the system couldn't be made, or that better test plans wouldn't help (there has been development in that direction thankfully) but I do try to cut them some slack considering that they're treading new ground still, as they are literally creating the industry standard for this type of dev environment as they go.

 

I see it as the lesser of the evils right now, as before even with more cooks, they were paralyzed to make significant changes to the soup. now there is definite and constant movement towards better, with some customers getting a bad batch once in a while. compared to the original paradigm of change stuff with considering consequences, or the previous paradigm of being so careful that nothing ever improves, I think they're closer to a balance now that serves the majority of their customers the best.

yes it sucks to be stuck in the minority when it goes wrong, and I'm right there bitching loud and long too, but I would only begin to worry about it being sloppy development if it becomes a stream of continuous rollbacks and problems... to my eye, I'm not seeing that yet. although the recent string of failures could be the start of such, until the next batch of changes comes around, I'm reserving judgement, and counting it as just some minor missteps.

my personal verdict is: ok, there's landmines here, must be aware of them when proceeding.

[Ayesha Askham]

@Void

Looks like another mine went off today.  If this isn't a stream of rollbacks, perhaps I don't know what is.

I don't like sounding like a doomsayer, but this really is getting too much.  Yes I know software needs to be tested on the main grid, but this last example, where two of three instances have failed is merely a case-in-point.  The point is that too much...far too much is being missed.

More pre-roll work needs to be done or SL will begin haemorraging even more badly, and before long it will be difficult to find another avatar older than a few weeks with whom to have sensible discourse.

 

Perhaps SL is playing to a script...a badly written one?

[Oskar Linden]

There are two very tricky areas of our server code: scripting and physics. These areas need improvement if we are going to get to the glorious future of awesome stability that everyone wants SL to be. However, even tiny updates in either of these areas can and does have dire consequences and is highly controversial. We've received many complaints about vehicle performance and attempted to alleviate those with last weeks physics tweaks. We tested a ton of vehicles on the beta grid and created vehicle testing tracks. We didn't know horses used the physics engine for movements. Those physics changes were on an RC channel for almost three weeks with no complaints. We thought it was good to go on the grid. We were mistaken. We recognized that immediately and made changes to fix things. Sadly our fix introduced a very serious server crashing bug that went undetected in beta. Because of that we had to push an emergency roll this morning. 

We have made many performance improvements to SL since the weekly release process was instituted that have gone off without a hitch. The areas that could use the most improvements are also the trickiest to touch. Every change will have a negative effect on someone. The reward of a better system overall is greater than the risk. This isn't a trade-off we take lightly though. We have long discussions about how to properly institute positive change. We have the ability now to make small incremental bug fixes that for a majority of the time have very little negative impact. We're slowly making things better. That wouldn't have been possible a year ago. With these trickier pieces of our infrastructure we have a larger risk of making things worse for a lot of people. We recognize that and work to patch any issues as soon as we can. I appreciate your patience.

 

__Oskar

[Void Singer]

Kelly recently solicited feedback about running a beta sandbox on AGNI (maingrid) as a way to prevent some of the more dire consequence like rollbacks, by making it more convenient to test without swapping grids, so that testing is more thorough with a better cross section of normal use.

of course it's not without it's perils as it would mean content created there could interact with the regular grid, so there's a question of limiting access (similar to what was done with some mesh regions).

in my eyes this does go a ways to proving the point that this likely isn't a problem of sloppy application, but rather of the limits of internal testing, that the appropriate staff are trying to explore other ways of addressing and preventing it, and I had the distinct impression that Kelly was upset that his changes ended up in a rollback.