Deploys for the week of 2011-07-04

[Oskar Linden]

Since today is a holiday in the US we will not be promoting any code to the main channel. There will be no rolling restart Tuesday morning.

 

__Oskar

 

p.s. If you are interested in helping test SecondLife in beta please join the group "Second Life Beta" in-world. We also have an email list where we communicate upcoming projects and how you can help. ( https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta ) Once a week we meet on ADITI to discuss new features, new bugs, new fixes, and other fun stuff. You are more than welcome. Information is here: https://wiki.secondlife.com/wiki/Server_Beta_User_Group

 

[Ayesha Askham]

@Oskar and the ReleaseTeam

It also gives you folk another week to work out how to improve on the string and sealing-wax fix of the Mono2 upgrade from last week.

I hope all the Team at Linden Lab had a good 4th!:matte-motes-sunglasses-1:

Now, please guys, get it right next time, eh?  I guess a pilot fix willl go out to a RC channel tomorrow, or am I being naiive?:matte-motes-wink:

 

Addition: 

OK now that the calendar is revised and working on the Grid Status page, I see that no releases are scheduled for the RC channels either.  Now should I draw the conclusion that you guys had several days off over the weekend and last week, so no programming was done?  That's not so surprising when you consider how big a holiday the 4th is to the USA!

[Mayalily]

I was just looking for this info because I need to do a Marketplace transaction this morning to get some things and I was wondering "is the a rolling restart" this morning?  I hope all goes well with me this morning trying to get my MP items delivered.  Crossing fingers. 

ETA:  Can't log into SL this morning; it keeps saying region may be experiencing difficulty.  Anyhow, I am so glad I checked the viewer before I checkouted on Marketplace.  I'll try the viewer again this afternoon, and hopefully I will have a successful delivery later this afternoon!  I am so glad I checked viewer first 'cuz no can do as far as log in right now. 

[Sassy Romano]

Frankly there's one word that describes this...

APPALLING!

I'll say it again, link LL salaries to Time Dilation figures as a multiplier.  TD=1.0, full salary etc.

 

[Cincia Singh]

I don't know what the problem is with the main server channel. I don't much care to get involved in long discussions of "metrics" and TD and script times. What matters in a social environment like SL in the clubs and welcome areas is basically "does it work?" I'm sorry to say right now it doesn't. Notecards don't open, objects don't rez, once rezzed you can't delete them or pick them up again, TP's are failing all over the place, rubber banding when trying to move from the lag even on freshly restarted regions both mainland and estate is appalling, audio streams and video streams are jittering, and overall the experience inworld has taken a definite turn for the worse in the last 2 weeks. I don't know or care what broke it, but it needs to be fixed ASAP. Frankly you can't roll new projects (i.e. mesh) to servers running this poorly and expect any semblance of useful data and eventual success. Please. Fix the main server channel.

[Oskar Linden]

The main server channel issues are being investigated. The US vacation schedule this week and the No Change WIndow has complicated things. I hope to have something more to report soon.

 

__Oskar

[Lee Ponzu]

And we can fine you a dollar for each post?

 

[Sassy Romano]

While that may or may not be an attempt at humour, it fails.  Since i'm not responsible for either:-

1. coding the change that has so negatively impacted the platform
2. steadfastly refusing to roll it back to a stable state

Most people in a commercial environment operate in a way such that they're accountable for their results, this is often tied to their income by means of penalty or bonus.  I don't see why this concept should be so alien?

In the case of this disastrous server deployment, we are experiencing a very negative impact to commerce and enjoyment of the service paid for and the response is "we went on holiday so we're going to wait even longer".

In an organisation with an effective CAB, this Change Request would have had a section "if it's bad, do what?", in this case, the answer would have been "ROLLBACK".  Instead we have "well, if it's bad, we'll acknowledge that it's bad, then we'll have a look at fixing it, then a couple of days later we'll deploy, then if it's still bad, because we're on holiday we'll leave it to continue even longer as bad."

Personally, I don't find that professional or acceptable to those of us who operate commercially here.  We all have different ideas for what makes SL worthwhile, if logging in and chatting to someone is all some do, then the impact will be different but some of us are more used to a commercial world where people are held to account for their mistakes and rewarded or not in an appropriate manner.

[Dilbert Dilweg]

Am I to understand there will be no main server roll at all?

Yes I have noticed a large increase of server trouble lately. for the past 2 weeks there has been some serious break downs of our region losing the ability of users to teleport in or out.. if you are not there to catch it. your sim will be unacccessable for hours on end. Other issues include, unable to save new scripts,. several issues of unable to save new notecards,, Editing notecars you already have in inventory are editable and savable but new items you can not manipulatequite as well. Same as objects at times. Terrible lag spikes and rubberbanding again. Time dilation dropping way down, and dragging the sim down.. Before the last coup[le weeks it wasnt too bad

Oh well. Hope it gets sorted soon.

[Cincia Singh]

Hang in there Oskar ... I know you'll get it figured out.

[Maestro Linden]

There will be an emergency rollback in the LeTigre and Magnum channels this morning, due to a high crash rate on those channels.  Until now, these channels were running "mesh prep 2": http://wiki.secondlife.com/wiki/Beta/LeTigre#11.06.28.234198

After the rollback, both channels will be running the same server version as the "Second Life Server" channel, which is 11.06.29.234329.

You can monitor the status of the rollback here:

http://status.secondlifegrid.net/2011/07/05/post1333/

[Oskar Linden]

For those experiencing issues with the main channel please update this jira:

 - https://jira.secondlife.com/browse/SVC-7079

What information we need:

• The region name, the time of the poor performance, and the duration.
• Why the performance is poor: what feels sluggish or what stat is bad.

If you have specific scripts that are not functioning correctly please be sure to read the notes below and file a separate jira bug in the Scripting (SCR) project. Please report both the expected behavior and what is wrong, it is very difficult to tell how a script is broken if you only ever get the broken version.

Notes:

• Increased 'Script ms' time in the stats ctrl-shift-1 stats is expected. The simulator is more aggressive at using available time to run scripts. This does not mean the sim is performing worse by itself. SimFPS and Time Dilation and the general feel of the region are more important factors.
• The time reported to estate owners/managers in Top Scripts is expected to be drastically different than in prior versions. This tool now reports the average script time over the last 30 minutes (or the life of the object, whichever is shorter). Previously it was using broken math and only reporting roughly the last instant of activity, weighted incorrectly by the last few seconds. The new data is more accurate and more
• Script based benchmarks may no longer work or their behavior may be drastically different. We addressed a bug called 'fly wheeling' or 'juicing' that would allow scripts to use inappropriate amounts of script time and hurt simulator performance. Essentially by doing a lot of really simple operations you could prime your script to get a whole lot of time for doing more complex work. Now we are more quick to adjust the time a script gets to the standard allotted time. This will most effect scripts that vary their workload and do tight loops - and it is quite possible script based benchmarks are adversely effected.

[Void Singer]

---

Oskar Linden wrote:

• Script based benchmarks may no longer work or their behavior may be drastically different. We addressed a bug called 'fly wheeling' or 'juicing' that would allow scripts to use inappropriate amounts of script time and hurt simulator performance. Essentially by doing a lot of really simple operations you could prime your script to get a whole lot of time for doing more complex work. Now we are more quick to adjust the time a script gets to the standard allotted time. This will most effect scripts that vary their workload and do tight loops - and
  it is quite possible script based benchmarks are adversely effected.

---

I hope that means an end to negative results for comparison run scripts and not and increase =X

[Bavid Dailey]

I'm am utterly astonished that Linden Labs did not feel the need to make these very significant changes known to developers   BEFORE they were released on the paying customers.  And please dodn't respond that they were mentioned in office hours.  If all the affected  scripters of second life ever turned up at an office  hours meeting ... well there'd have to have  been a lto of other stuff fixed,  no? 

This last debcle has been the last straw for me; scores of hours of careful crafting to have my coral swarms be low cost accordign to  the best available tools, and now you tell me that the figures were always rubbish!  I  look at the results now and I see I am up for starting over; or just giving  up. Guess which I shall choose?

It's  so damnably unproffessional. Shame on you Linden Labs. 

 

[Void Singer]

---

Bavid Dailey wrote:

[...] scores of hours of careful crafting to have my coral swarms be low cost accordign to  the best available tools, and now you tell me that the figures were always rubbish!  [...]

---

so what you're saying is that during the last 2+ years you never questioned and never asked about the accuracy of the figures that scores of people has been complaining were broken / ineffective, and you're angry now that they fixed it so those tools work more accurately? or possibly even worse you are just basing you testing on the raw numbers and not product comparison?

[Zena Juran]

Back to square one.

Everything was running along fairly well on my Homestead (after recompiling all the mono scripts to LSL) until this morning when I decided to perform a Region Reset.

Now, no matter what server my region ends up on, the Time Dialtion is so erratic as to again negatively impact teleports, object rezzing/derezzin, and region performance.... no diff than right after the last server updates.

Not impressed... the lot of you responsible for this need some serious disciplinary action!

 

 

[Bavid Dailey]

Nope, neither.  Provocative remarks don't disguise that Linden Labs have  made  a major change that affects many adversely and they made no attempt to inform people in advance 

As to comparatively  products, mine do very well ty. But customers whose estate managers kow enough to look in top scripts but not enough to evaluate what they see, end up beign told they can't run these scripts and complain to me.  So a big chnage in hwo my scripts get displayed matters to me. 

 

As for complaining about misleading performance figures, sure I've done my share.  Anytiem you can give me a coherent explanation fo what pump I/O is, I'd be fascinated. I've been askign that question for a while...

[Fleth Robonaught]

How do I change the release channel of a sim? We would like to switch to Magnum.

[Void Singer]

well if you products stlill do better than other products in script time then there really isn't a problem is there? those other products will still be above yours in tops scripts neh? and if yours is now above the the others you can now address it. I'm sorry you think that's adverse, but it's like finding out smoking cause cancer... it's not an adverse effect on the tobacco industry, it was always true, now you just have proof. now you can actually do something about it.

ps.
without even looking at the stat i/o pump should be either the difference between received and sent data, or the conglomerate of both, useful for diagnosing information floods or heavy communications respectively... it likely wouldn't be useful for diagnosing single scripts, but would be useful for diagnosing region performance under load.

[WolfBaginski Bearsfoot]

I don't know what you plan to do next week, but I think you've managed to bust things big-time.

The Mono-2 fix seemed to work, at least on Mainland.  Sim crossings were not perfect, but they worked. I was able to drive significant distance with only minor rubber-banding. But there was that scripting problem which was revealed.

Since you fixed that, Sim-crossing has become as bad as it ever has been Sometimes sims behave as if they're just not there, and if the viewer doesn't crash, the only thing to do is log out.

I count myself lucky if I can make a half-dozen sim-crossing without disaster.

 If you can't fix problems as basic as that, why should I expect Mesh to be worth the trouble.

 

 

 

[Ayesha Askham]

@Wolfbaginski

Yes, that is an odd effect, it makes me think that the "fix" for script-sim interactions is intimately involved with sim-crossing vehicles.  I hadn't noticed the improvement you saw, but once the "fix" was rolled it certainly did put the crossing issue back where it used to be.

It's not just vehicles though!  I played a game of boules yesterday and I am considering applying to NASA as a replacement launcher for ISS missons after what a few of my throws did.

I wonder if the way the sim reads vectors has been borked?

There's enough going on here to cause a fair few headaches among the release team.  I wonder what tomorrow will bring?

[WolfBaginski Bearsfoot]

The sim-crossing problem is certainly related to the general Mono-script problem. Physical vehicles need scripts, and however they come into the region, the Mono-problem applied. The Sim hesitated. Add the complexity of actual sim crossing.

The Mono-2 fix does make a big difference. But I've also paid for some thoroughly awkward vehicles, things like scripted colour changes with no way to remove the scripts which do that. It makes for a lot more data to transfer from sim to sim, and that doesn't help.

There's clothing items which do the same. There are better ways of handling the problem now, but a couple of years ago the onlt usable method was to put a script in every prim.

As far as I've been able to find out. it things do run slow, you can temporarily be at one of several "wrong" locations within the new region, depending just where in the transfer cycle yoiu might be. And where the Sim thinks you are may have little to do with where your Viewer thinks you are. It doesn't help if one of those temporary locations is protected by ban lines or security orbs.