Second Life Server

[Toysoldier Thor Member]

I am writing this new thread here since creating a JIRA for public input and involvement is no longer possible AND because later today I will be bringing this serious issue up at the Weekly LL Server User Group meeting at 4pm slt.

BACKGROUND:

A major part of my nightly activity almost every night in SL is to attend several live music concerts / events / gigs at a wide variety of venues that host live events.  I attend usually 2 to 4 live events a night.  As such, I have very good experience on being on sims where I frequently see 30 - 50 - 70+ avatars on a sim to attend many of the more popular music artists in SL.  I know what kind of lag to expect over the past years of attending these events. 

I also know what is normal verses very unusual regarding the kind and intensity of lag as well as sim crash frequency at events - which does happen but is usually quite uncommon to happen even when a sim is real full.

THIS WEEK THINGS CHANGED:

I dont know what changed and I am suspecting it was changes that LL has deployed in the server code but clearly something has changed that has caused a serious impact on a sim's ability to be stable during these larger events.

On Tuesday night (Sep 25th) I attended 4 concerts that evening SLT time and all at different sims.  At two of these event, one with about 25 avatars and one with about 45 avatars, I personally witness very unusual and severe situations that personally impacted the club owner and artists that were holding the events.

What was more suspicious was that they both had very similar symptoms:

At the 1st event with only about 25 avatars I arrived at the event almost at the beginning of the show.  I TPed into the club with massive intense lag.  I have been at events far larger with nothing near the lag i experienced at this event on the sim.  No one at the event could move.  When I looked at the lag meter it was ALL SERVER LAG as the Network and Client were both in GREEN.  It appeared that the sim was completely hitting the wall.

the stream was working still so the artists could be heard.  And then about 10 minutes into all this lag, all the avatars started crashing out of the club.  I was one of the last ones to survive without crashing.  The musicians crashed even though the stream stayed up.

Also, those that crashed took could not get back on for a few minutes and even when they got on, they could not return to the club/sim even though there was almost no one left and the lag subsided.  After about another 10 minutes I crashed too and then after returning - people could all start coming back to the sim.  It was like the entire sim had to be rid of all avatars before it could return to normal.

Later that night I went to another very popular concert that was preceded by another popular artist.  When I arrived at this sim, there already were about 40 ppl at the club listening to an artist.  There was no significant lag at all.  THEN... as soon as the stream was switched to the new artist's stream the sim instantly got HAMMERED by the exact same massive lag that I witnessed earlier at the first event.  It hit like a light switch and just as the stream was switched. 

Within minutes, all but 4 of us (I was one of the few that survived the avatar crashes) were kicked out of SL.  None of the 40+ avatars could log back into SL for several minutes.. and even when they got back on - none of them could TP back to the club even though they could TP to other sims.  Then almost at the same time, the remaining 4 of us all crashed off the sim as well.  We could get back pretty quick and we were all allowed to return to the sim after that - even those that crashed 20 minutes earlier.

FAST FORWARD TO LAST NIGHT....

I went to another popular club sim that frequently host very large crowded events.  I arrived to see a popular artist and there was already one on stage.  The club was a bit laggy as would be expected with about 50 avatars on it... but normal.

And then top of hour the club switched streams for the new artist start his show and BOOOM !!  everything that happened on Tuesday night happened again.  MASSIVE Server side lag instantly hit..... all but about 4 of us survived - I was one of them again.  And no one that could relog back into sl could TP to the grid.  Instead of waiting to crash, I logged out gently and re-logged back in... but for the remainder of the hour no one - including me - could return to the sim. 

Once again another major show was disrupted.

I have also heard from another venue owner that they have noticed this severe increase in sim lag freeze / crashes recently.

So I dont know if this is ever since PathFinder was deployed and that it got majorly worse after this Tuesday's code deployment.... but something severe has happened to the grid.

If you are a club / venue owner of big events... have you recently seen similar types of lag and crashes?

I will be asking the Lindens about this tonight.

[Toysoldier Thor Member]

PS....

Attention to the Server User Group Linden staff : 

Simon, Andrew, Baker, Cheesey

Hopefully you will read this thread prior to this afternoon's weekly Server User Group meeting so that I can just point to it and ask for your input / opinions on whats going on and if LL code upgrades could be causing these majore sim lag crashes.

[Ayesha Askham]

Toy S

Seen and similar issue noted at Pine Wood sim, two (UK) nights ago.  It has put off a lot of Club "regulars" since they seem far more "crash-prone" than me.

Typical effects are freezing and "rubber-banding" when moving, extremely slow rezzing, streams faltering, etc.  This seems to have been a result of the rolling of the last set of code.

As a corollary we have seen issues related to sim-server/viewer communication even on lightly-loaded sims, including stream issues.  Also same server version, since it is now Grid wide.

[Theresa Tennyson]

I don't see how any server changes could have caused a sudden-onset problem in the last few days because the main channel hasn't seen any code changes since September 18 and the RC channels were last updated the week before.

[Hitomi Tiponi Member]

There was almost 3 hours of scheduled maintenance early on Tuesday evening (PST) - see http://status.secondlifegrid.net/2012/09/25/post1771/.  Maybe something was changed during that which caused the problems.

[WolfBaginski Bearsfoot]

I don't want this to be read as a specific suggestion, but this could be down to a memory leak, which means that sim servers gradually use more and more memory, until there isn't enough RAM, and the OS starts using virtual memory, and performance plummets.

That sort of problem would explain why the fault doesn't normally become apparent, because the weekly roll-outs reset the system before it gets too bad. I don't know how the location of the bad code could be discovered, It doesn't have to be all that recent a change.

[Jadeclaw Denfu]

I have rented a parcel on the Hubble-SIM and since a few days, I experience heavy Lag-Attacks, slapped a LagMeter on the house and sometimes it drops down to near zero, at the same time it is practically impossible to move. Sometimes up to 200 - 300 of these Lag-Attacks per day. Note, this SIM is low traffic, usually 2 to 6 avatars on it at the same time. According to GridSurvey, these 5 Sims are on the same server:

• Immaculate - surveyed 2012-09-20
• Jemmica - surveyed 2012-09-20
• Yacumama - surveyed 2012-09-20
• Getaway Cove - surveyed 2011-12-31 (older data - Region not visited on last survey)
• Jeremis - surveyed 2011-12-28 (older data - Region not visited on last survey)

Checked these Sims, no avatars on them during the Lag-Attacks.

Another problem: An unusual high number of Sims -especially on the Sansara-Continent- that can't process sim-crossings in a timely manner, with the result in being kicked out of any automated vehicle, or the avatar keeps on walking forever until being stuck somewhere nowhere. Or the error message comes up. Completing one of the longer YavaScript-Tours is practically impossible.

Also new: Sim simply stops responding. You can't move in any direction, only turning around one's axis is possible. Logout/Relogin usually fixes that problem.

( Happens with the default viewer as well as with Firestorm - Both are latest version.)

[Theresa Tennyson]

Right now there are several mainland sims that are running in a constant state of lag - less than half their normal frame rate - while being nearly empty. They're all showing abnormally high script times - often around 50 ms when it's usually in the neighborhood of 15 ms. Even so, they're shown as running only a very small percentage of script events per frame - well under 1%. In a perfectly running sim this should be 100% and even in very crowded sims I haven't seen this much below 40%. Other nearby regions are running completely normally.

I'm wondering if there might be a new griefing script out that can silently lag the servers.

[Jadeclaw Denfu]

Currently I'm on Heterocera, walking along the SLRR-Railway and most of the Sims I passed through are lagging heavily.

e.g Mocis, Tenera. Aglia is especially bad, I'm just watching a train moving slower than a garden snail.

[Toysoldier Thor Member]

I can believe that a memory leak could be a potential cause to this problem seems like problem shows up immediately after reaching some threshold.

Like I mention.i have not seen a sim instantly lag up solidly so fast. It spike a switch was turned on. This problem is a major world experience problem.