Deploys for the week of 2011-10-17

[Oskar Linden]

Second Life Server (main channel)

This week LeTigre gets promoted with the AIS (Agent Inventory Services) updates. Users of viewer 2.5.1 or above (or those using thirrd party viewers which have merged in 2.5.1 or above) should see a speed increase when fetching inventory.  This server version also includes support for Marketplace Direct Delivery, which should improve reliability when buying and selling items on the Marketplace.

 

2011-10-18, 5:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_Server/11

 

Second Life RC BlueSteel

Both this channel and LeTigre have a new project that contains some features to help with estate management. There is a new LSL function called llManageEstateAccess(). Estate Owner or Estate Managers can us it to add or remove agents from the estate's agent access or ban lists or groups from the estate's group access list. There are more details in the release notes.

2011-10-19, 8:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/11

 

Second Life RC LeTigre

LeTigre has the same code that BlueSteel has.

2011-10-19, 9:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/11

 

Second Life RC Magnum

Magnum will have the same maint-server project with lots of bugs fixes and a few server crash fixes.

• Bug Fixes:
  • SCR-186 Collision detection does not work for some children of linked prim set
  • SVC-7251 remove automatic logout of agents that have been Away too long
  • SVC-7260 Phantom, unremovable prims
  • SVC-7297 'excessive instant message' throttle kicks in when it shouldn't
  • SCR-115 State changes considerably slower under Mono-LSL than LSO-LSL
  • SCR-22 "X-SecondLife-Owner-Name" sometimes set to '(waiting)' instead of ownername
  • SVC-7321 llRemoteLoadScriptPin() fails to add O:VMCT script to O:PERM_ALL atchmnt
  • SVC-7294 The simulator is too strict when llGiveInventory() adds restricted-permission items to fullperm attachments which already contained other restricted-permission items
  • SVC-7339 Performance issues on LeTigre 11.09.23.241511 (and Main Channel)
  • SVC-7289 llCastRay() returns RCERR_CAST_TIME_EXCEEDED on small parcels.
  • Fixed various server crash modes.
• New features:
  • New parameters for llGetObjectDetails(), to allow LSL scripts to view mesh prim accounting:
    • OBJECT_STREAMING_COST returns streaming (download) cost
    • OBJECT_PHYSICS_COST returns physics cost
    • OBJECT_SERVER_COST returns server cost
    • OBJECT_PRIM_EQUIVALENCE returns prim equivalent weight

2011-10-19, 10:00am: Rolling Restart - Release Notes: https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/11

 

We will be monitoring this thread during the next week so please feel free to post issues that you feel have been introduced by the new code. Please file a JIRA for issues you find and post the JIRA link into this thread. It really helps us out. When determining if issues are relevant or not research is key. Tracking down exactly the right situation where an issue is occurring greatly speeds up the development process to get fixes in place.

I appreciate your help. Have a good week!

 

__Oskar

 

p.s. If you are interested in helping test SecondLife in beta please join the group "Second Life Beta" in-world. We also have an email list where we communicate upcoming projects and how you can help. ( https://lists.secondlife.com/cgi-bin/mailman/listinfo/server-beta ) Once a week we meet on ADITI to discuss new features, new bugs, new fixes, and other fun stuff. You are more than welcome. Information is here: https://wiki.secondlife.com/wiki/Server_Beta_User_Group

[Jayme Llewellyn]

This is all well and good, Oskar, but I ask you this - what are the linden labs doing about the one consistant thing that they have been consistantly ignoring for four years? I'm talking stability here. For four years, SL has been notorious for poor stability and Linden labs have been equally notorious for ignoring every complaint about stability and adding features that only further decrease stability. I have been on twice in the last 10 days with alternate avi's (DeeAnna Treves and CamiN0rris) only to experience nothing but continual lag. I decreased the quality of my graphics setting - still horrendous lag. This has happened irregardless of what region i'm in and happens for hours non-stop. Then there are the crashes when sl boots you out of the region. I am not the only person who is experiencing this either, only none of the others have the knowledge of guts to actually stand up to you lot and say something. by continually adding new feature that undermine the stability of the grid and game even further, you at linden labs show you have no regard for what happens nor do have any regard for the users of your product. All you are concerned about is the bottom dollar. I'll tell you this - I am glad i do not pay for the premium package because I know there's no chance in hell linden labs will even consider giving a refund or discount to make up for the piss-poor effort you people are making to addressing the core issues of stability. I took an 18 month break from second life because of this very same issue and came back only to find that nothing had been done. and now it is just getting worse, Until i see actual effort from linden labs to address the issue of stability, I will be using sl very infrequently and advising others i meet to do the same thing. If nothing is done, I will start a campaign on public forums such as facebook telling the truth about linden labs efforts and what a bunch of lazy, inconsiderate, money-hungry egotists you are.

[Sassy Romano]

Since the last updates, our sim has been lagging far worse than before and suffering typically twice daily restarts.  We're now logging this to a database so will be able to analyse the patterns.

Laughably, live chat said it was a local user problem with "wireless internet".  Yeah ok.  When local use of wifi causes a sim crash, i'll be impressed.

I hope this weeks update fixes this...

[Norton Burns]

Jayme...

1) Take breaths occasionally... All that invective in one paragraph makes for difficult reading.

2) LL-bashing for 'lag' has been going as long as I've been here. One of the things I have learned over those years is that a great deal of 'lag' can be at the user's end & many things can be done to alleviate it.

I'm not saying for one minute that server-side lag doesn't exist - of course it does & that is what you wish could be fixed.

However, try this... grab a new copy of either LL's latest version or one from your favourite TPV. Make sure you do a full clean reinstall, removing all legacy data from your computer [some folders are 'hidden' by default] not just add/remove program.

Should you, by any chance, choose to use Phoenix or Firestom as your viewer of choice, then come visit the Phoenix/Firestorm Viewer Support Group inworld, which is manned virtually 24/7 & let's see if we can get you up to speed.

[Ayesha Askham]

Oskar, and others in server support:

Just what is going on?  Dozens, nay, hundreds of folk are being badly impacted by these seemingly endless tranches of restarts and upgrades.

Now recently I had the misfortune to TP (one of the few days recently when teleports were functioning), to a region called Nouveau, which blithley announced that it was running server version 11.10.11.0 - an impossible ID and supposedly on the BETA Grid Aditi. How on Earth (real or virtual) did that happen? And BTW it happened twice, it wasn't a fluke - ask Hitomi Tiponi.

Also despite meticulous sim management, our server, despite returning apparently ideal figures for a Homestead (ie TD consistently above 0.95 and script time <5ms, with upwards of 15ms spare time) we have what amounts to server cardiac arrest when someone logs in or out!

Teleports result in failed handoffs almost everywhere, with the poor av ending up under a prim floor somewhere - something I haven't experienced in 2 years.  It's not just me, I have testimony from several others about this, and one av has specifically checked her script memory to ensure it stays low.

Something is not going well with this Upgrade, and it is high time we were given some information about what Linden Lab is doing, and how much longer we must endure this.

I understand the reason for the weekly server updates...you know that I'm not an idiot!  But what is going on now needs some explanation if LL is not to see a mass exodus.

PLEASE JUST TELL US.

 

Update 9am SLT: after 2 solid hours of work I finally managed to get the damned asset server to give me my clothes back and De-Ruthed myself.  Thanks to those in Phoenix/Firestorm Viewer Support.

As far as I can see, teloporting is back to "normal" now, so maybe the New Server software on Main Server works a bit better.  I also notice that our sim-server has changed again, which does not normally occur at a server version roll.  That may also have helped.

[Maestro Linden]

Sassy and Ayesha:

Since the OS upgrade (mentioned here: http://status.secondlifegrid.net/2011/10/17/post1448/ ) takes quite a while to run, regions are migrated off of a host while it's being upgraded. When this happens, it's quite possible for a region to be migrated to a host which hasn't yet received the upgrade, resulting in multiple restarts for a region over the course of the grid-wide upgrade. It's also possible that some of the restarts you see are from the sim crashing - if you provide a list of region names and times of restarts, I can check to see if the region restarted cleanly or actually crashed.

 

Ayesha

There was something funny going on with the version number reporting of Nouveau and other regions running on sim20693.agni; they were really running 11.10.03.242227, despite reporting that funky version number. The rolling restart this morning fixed that - the version reported now is 11.10.07.242626. I see no evidence to suggest that the regions on this host are or were ever running on aditi, however.

 

Aside from SVC-7339, which is only fixed in Magnum, I'm not aware of any specific performance issues which are new in recent server versions. If you can confirm a performance issue which seems to be a bug in the server code (as opposed to just a single slow host), please file a Jira describing the issue, so we can investigate.

[Ayesha Askham]

@Maestro

I didn't say that Nouveau WAS on Aditi (I could've been clearer in what I said!).  However there is no doubt that something screwwy is going on, since I am now very firmly ruthed, and I cannot shift it.

I've just done some performance checks my end and it is not my net or PC.  SL just won't send me my skin, shape or clothes.

I am THIS close to telling LL to stuff it.

[Lydia Craig]

Thank you for not posting notice of today's restarts in grid status, this just verifies that keeping players informed is not a Linden priority as the low numbers of players online is starting to verify that SL is not longer a player priority.  In fact when you subtract the horde of bots, which officailly dont exist, one wonders just how many players are on line anyway.  In all seriousness, restarts are bad enough, but restarts without warning other than in the chat window, and not even on the grid status page are a new low, and if any of you cared a bit about the player population would never happen.  Again your actions speak louder than your words or lack of them and the picture they paint is not a pretty one.

[Oskar Linden]

Lydia, is this the grid status you were looking for? 

http://status.secondlifegrid.net/2011/10/18/post1447/



---

Lydia Craig wrote:

Thank you for not posting notice of today's restarts in grid status, this just verifies that keeping players informed is not a Linden priority as the low numbers of players online is starting to verify that SL is not longer a player priority.  In fact when you subtract the horde of bots, which officailly dont exist, one wonders just how many players are on line anyway.  In all seriousness, restarts are bad enough, but restarts without warning other than in the chat window, and not even on the grid status page are a new low, and if any of you cared a bit about the player population would never happen.  Again your actions speak louder than your words or lack of them and the picture they paint is not a pretty one.

---

 

[Chriss Rosca]

Lydia, Oskar,

The message was there.
They just forgot to move it to the top like they use to do, so it was a bit hidden this week.

Chriss

[Hitomi Tiponi]

@Maestro - thanks for telling us what that weird server number was - a real odd one that.  Personally I have not noticed any unusual problems inworld as a result of the restarts, but I haven't visited every sim there yet :smileyhappy:.

[Oskar Linden]

---

Chriss Rosca wrote:

Lydia, Oskar,

The message was there.

They just forgot to move it to the top like they use to do, so it was a bit hidden this week.

Chriss

---

Mystery solved! Thanks Chriss.

[MB Robonaught]

Personally I've seen major improvements in stability and other issues since last year. While it still needs work most of any sim stability issues are caused by inconsiderate people who load themselves down with enough scripts to crash NORAD But something that would help sim owners now that  we are getting tools to better watch our sim performance ourselves would be a list of what is optimal returns for sim data information so we can deal with a script that has gone wild or other situations

[Oskar Linden]

---

MB Robonaught wrote:

Personally I've seen major improvements in stability and other issues since last year. While it still needs work most of any sim stability issues are caused by inconsiderate people who load themselves down with enough scripts to crash NORAD

But something that would help sim owners now that  we are getting tools to better watch our sim performance ourselves would be a list of what is optimal returns for sim data information so we can deal with a script that has gone wild or other situations

---

Thanks MB. We've been trying to create a more stable server experience. I hope that we can continue to offer increases in stability. I agree with you regarding the sim owner tools and will pass that information on to the rest of the team.

 

__Oskar

[Ayesha Askham]

@MB

I don't usually bother with direct responses, but this post cries out for it.  I don't disagree with you in a lot of instances but....

1) the sudden and inexplicable problems on our sim were not due to script-memory load.  Yes, the server was stable...so stable I suspect a nuclear blast wouldn't have perturbed it - it simply did not respond.

2) the repeated teleport failures were not script load related in any way, neither in terms of script numbers nor memory allocation.  There simply was a problem, which, thankfully seems to have gone away.

Oskar and many of his peers have been working hard to improve our experience of SL, and I understand that as well as anyone.

However.....there is still a yawning gap between the level of communication needed between Linden Lab and it's customers, the users of SL, and that which they currently provide.

Unless and until Linden Lab realise that they need to have a meaningful dialogue with its customer base, that customer base will, by and large, hold Linden Lab in doubt and disbelief.  Such situations do not lead to cooperation or positive outcomes. (Office hours are inconvenient to many and selective in the extreme.)

[Oskar Linden]

---

Ayesha Askham wrote:

However.....there is still a yawning gap between the level of communication needed between Linden Lab and it's customers, the users of SL, and that which they currently provide.

Unless and until Linden Lab realise that they need to have a meaningful dialogue with its customer base, that customer base will, by and large, hold Linden Lab in doubt and disbelief.  Such situations do not lead to cooperation or positive outcomes. (Office hours are inconvenient to many and selective in the extreme.)

---

 

Ayesha I am asking in all seriousness what level of communication would be sufficient for you. If our communications are lacking at the level you claim they are I would like to change that.

 

__Oskar

[MB Robonaught]

I don't think the level of communication Ayesha wants is even possible. I mean there are 60,000+ people online right now and how many lindens?. No way they can keep up. I still think that self help by sim owners is the way to go to not only speed up fixes but to also take some of the load off LL staff so they can concentrate on fixing the harder issues.Oskar tap into that unused rescource of some of the brilliant and imaginative minds that play SL

PS we work real cheap lol

[Oskar Linden]

ok MB :matte-motes-grin: In the vein of increased communications from LL to the residents outline for me everything you would like to see in the form of additional tools from sim/estate owners/managers. From your point of view show me where the fail points are and how you would like to see them improved. Detail the tools that you would like to see and how they can help run a better sim. I'll use that to help the teams that are working on upcoming server changes.

That goes for any sim/estate owners/managers. What tools would make your regions easier to manage?

__Oskar

[Sassy Romano]

---

Oskar Linden wrote:

That goes for any sim/estate owners/managers. What tools would make your regions easier to manage?

__Oskar

---

One that has the region automatically email/text you to get out of bed, any time of the night, when my sim has a fault. *smiles*

How about top scripts that actually works when the sim is so lagged to death that time dilation is at a standstill, top scripts doesn't work.  (So this probably isnt' going to be a great tool anyway but it's hard to investigate anything when one of the tools itself suffers when the region is under load).

I'd like to know when the sim is memory constrained.

I'd like to be able to see url allocations, how http traffic is being throttled (if at all).

I'D REALLY LIKE A METHOD TO DEFER YOUR RESTARTS! 

I accept that LL wants to keep it's platform up to date but your off peak hours are MY peak hours and I don't see that as the customer, that should be forced upon me.  I'd like a window where I can set when the sim will update which is inside your update window and not "Region is restarting in 5 minutes".  I could have a bulid open, scripts open and be afk on the phone or in a different room and come back and i've lost work.

LL often seems to forget who the customer is Oskar, that is a common theme.

[Tristizia Demonista]

Only problem here would be, since SL is a Worldwide thingie, there is allways some customer for which the restart is inconvinient.

I don't see how that particular Prob can be handeled, other than the regular intervals that we have now. 

[Sassy Romano]

Easy, that's why I said for estate owners/sim owners to be able to set the window when the restart should be done by so if LL chooses Tuesday, I get to set when during Tuesday or at least, have sufficient notice as to be able to defer it.

5 minutes notice isn't enough to avoid loss of work especially when you don't know when that 5 minute notice is going to arrive.

How can people reasonably expect to plan events when the sim could be restarted without warning (5 minutes isn't enough)? 

There was a time when LL wanted to attract corporate customers but what corp is going to be interested in using the public facing services when any of their planned usage could be interrupted like this?  This is the sort of thing that illustrates that LL is not engaged with the customer base and treats the customers as if they're just passengers on an LL train set that can be played with at a whim.

[WolfBaginski Bearsfoot]

Oskar, you run one of the better departments, as far as communication goes. I don't know if the OS upgrade process is, strictly, your responsibility, but we don't know what to expect. We're in the third week, we seem to have no way of knowing if a region is running on "safe" hardware, and it just goes on and on and on.

Another example of poor communication, nothing to do with server code or hardware: the default Viewer download changed, to version 3.1.0, and I've no real idea of what has been done. It doesn't get an announcement. There doesn't seem to be much information about what was fixed, if anything. I hope they've fixed the graphics bugs.

It isn't hard to find examples of poor Linden-communication being exposed by the Merchants: sudden changes to the marketplace which aren't clearly announced in advance.

Oskar, some of your fellow-Lindens are embarrassing the company with poor communications. They don't even have the advantage that the LDPW has--their stuff works.

[MB Robonaught]

Boy I feel like a little kid on Santa lindens lap lol j/k But seriously below is a short list of items that I feel will help in self monitoring our sims

1 list of optimal settings/data returns for a sim
example: a sim shows 100mb of allocated memory and if a sim server side allocates a max of 5 gigs we're doing great but not so good if it's 500mbmax
2 paper on which data is processed client side vs. server side
example: there is mass confusion when someone claims lag and quotes stats to say it is a server issue when most time it is their side
3 access with a script of estate script times and collisions
example: when a script or object glitches and causes excessive of either yet memory remains constant
4 ability to open/close a sim via a script
example: security in speed of response to a griefing. By the time you open the estate window and go to the settings it's usually too late to stop the sim from being lagged

offline
5 ability to save a data log of sim stats and other data without using http
example: there is a need to save a text file of data gathered by a script or to write settings either saved server side or the users hard drive. Accessing a web page via http

requests can be extremely laggy depending on internet traffic especially if only one line of text is needed
6 be able to count and acquire the name and location of an object that suddenly spikes in http requests or script time/memory
example: objects script suddenly peak above an average such as when a script may be stuck in a loop. locating a defective script can take days even with estate tools
7 ability to set all estate settings via a script such as blocking physics
example: again this is for security for a faster response to a griefing
8 data for both current and average script times
example: to define if a script has peaked or is running consistently
9 ability for a script to notify a person using excessive scripts automatically
example: a message via a viewer can be seen as an attack on a persons character and the sender doesn't even know it was sent. an object sending is seen more as a function

then an attack
10 way to identify alts (sever security issue in not being able to)
example: one popular method of griefing is to create several alts attack with one then log into the next and to attack again so on and so forth then delete all the accounts

to avoid tracking
11 data stream from the statistics window
example: most of the data provided can't be accessed via scripts and again there isn't any information as to which reading is way high or low
12 scanners that cover the entire region and not limited to 96m
13 data on image/rendering costs per object
example: someone loads a sim with textures of 1024 or higher which can cause sim lag which isn't shown in any report window
14 location and list of objects owned by a specific person along with prim counts(land options)
example: the ability to locate and deal with an object that is lagging a sim
15 notifications of grid status report changes without using http requests
example: a sim suddenly has an issue and having to go to the web page to see if there is a status update. Doing via http requests is bad if half of sl is checking the page at

the same time

[Ayesha Askham]

@Oskar

I'll try to suggest some things that I see as lacking, but I think MB Robonaught is far more learned in some of the more technical aspects of SL and maybe his "Wish-List" contains some things your colleagues could address.

The way I see it is this:

The Grid Status page and the Official Blog are the places where most people go (or should go) to be updated on issues within and occasionally without (your service providers) SL.  I'm guessing that the Status Page is operated on a shift basis, in order to cover the 24 hour nature of SL.  Now it has to be said that the Grid Status Page is very variable in its response times and its content, ranging from the informative to the blankly silent, and I suppose it is due to the fact that whoever updates that page has other, sometimes more pressing duties.  I think it also reflects the variable involvement with SL that some Linden Lab staff have.

Now perhaps it would be good to give responsibility for writing that page to a single role with as close to 24 hour support as possible, that way some consistency would be obtained.  Also it would be wise to give that role to someone with good communication skills so that LL can be put across as caring (as many of you are - I know that), something that doesn't get across at present.

I'm not technically savvy enough to give you details, I'll leave that to the likes of MB Robonaught.  Perhaps more emphasis could be put on pointing folk at the status page (Yes I know some folk would need to be physically dragged there)...but if there was better and more consistent info there perhaps more WOULD use it?

Some hiatuses(?) in SL never get aired at all, despite us knowing full well that there is an issue, and that bugs a lot of folk I talk to.

It IS getting better, Oskar, and maybe that is why I feel more inclined to pester you and your ilk, and there are some folk for whom it will never be good enough - I suspect they like to complain so much they'd be lost if there was nothing to complain about!

Recent issues re a new Viewer UI, script-permission changes, just what the OS upgrade meant...all these would be suitable candidates for Official Blog Posts, yet all I've seen lately are Fashion updates, which, while interesting and somewhat entertaining, are not what a lot of folk want to see.

Some of what I've suggested is probably not terribly practical in these straitened times, but at least in principle I believe they would begin to change the perception of SL as a system made by geeks for geeks and run by geeks.

PS Edit:  Some of the more popular TPVs have the recent Status Page topics as part of the login splash screen, so if the information there was improved Linden Lab would reach a wider audience quite easily.

[SappaDallagio]

You handle the world-wide nature of Second Life by creating an upgrade package for each server and caching it up for the next time it needs a reboot. If estate owners/managers were given the opportunity to schedule their own reboots, start them, and monitor their status with LL support, there should be no reason why owners and customers have to scramble when they get the ominous 5-minute countdown.

Of course, this would mean that LL would need to trust that estate owners/managers will actually schedule the update and initiate it as scheduled.

Many, many years I have spent in IT... I see this as an easily preventable situation but to make it so requires trust and cooperation, something I have not seen offered by either LL or the owners/managers, speaking as one who is strictly a SL consumer at this point.