Reply
Member
Jadeclaw Denfu
Posts: 362

Re: Major Increase in Instant SIM LAG & Crash during large events

[ Edited ]

Reply to Toysoldier Thor - view message

And it seems, the lagging problem has now caught the attention of the Support guys. Saw one of the Lindens hovering in the Galli-Sim a few minutes ago.

 

Good quality low prim furniture by Phil Deakins
Landowners & Tenants, please activate Autoreturn!


Posts: 2,539
Registered: ‎03-31-2009

Re: Major Increase in Instant SIM LAG & Crash during large events

[ Edited ]

Reply to Jadeclaw Denfu - view message

Here is all the details I have about two of the LAG event (the one on Tuesday and the one last night):

 

EVENT #1

Lag trigger date/time:

Date:  Tuesday Sep 25, 2012

Time:  9pm SLT (almost to the minute as the new artist was taking over the stream from theprevious artist)

Location:  http://maps.secondlife.com/secondlife//20/109/24

Location:  "The Pier at Wild Beach - Live Music Venue"

Description: There was about 40 ppl at the club / sim at the time the event triggered.  It happened just as the stream was changed from the previous singer to the new singer.  The artist did not even get on stage and the sim got hit with a massive server-side lag (not network - not client).  All but have 7 avatars were kicked off the sim.  I survived for about 15 minutes waiting for the audience / friends to return.  None returned while the sim stayed up.  Then the remainder of us were forcefully shut down.  When we logged back in, we could get back to the sim and the audience eventually returned for only about 10-15 minutes of show.

UPDATE: I talked to the event host that reported the problem to LL and LL responded to them that it was a "STALE SERVER".... what ever that means.  User said she did a LIVE CHAT call to report it and she talked to REED LINDEN

 

EVENT #2

Lag trigger date/time:

Date:  Thursday Sep 27, 2012

Time:  10pm SLT (almost to the minute as the new artist was taking over the stream from theprevious artist)

Location:  http://maps.secondlife.com/secondlife/Purple%20Magic/21/21/26

Location:  "Sherie's Gaslight"

Description: There was roughly about 45 ppl on the sim/parcel of land at the time of the triggered massive server-side lag (it was not network or client when I looked).  Within a few minutes only 4 of us were left on the sim.  The sim did not crash as I was able to not crash during the event.  BUT, I decided to log out gracefully and then relog and return.  No one was ever able to return for the rest of the hour.

 

I do not have enough details of the first recorded event on Tuesday (ie. where and exactly when) but I believe it was about 6pm slt and the crowd was a bit smaller than the two recorded above.

I am hoping that LL has sim activity logs and can use the sim location and date/time to see if they saw ANYTHING in their logs for these sims at those times.

Toys SL Marketplace StoreToys SL Art Gallery
Honored Resident
TheaDee
Posts: 2

Re: Major Increase in Instant SIM LAG & Crash during large events

Reply to Toysoldier Thor - view message

I am the owner of a live music venue.  We moved to a different sim and had our first show there on September 13th.  We were hosting three hours of shows that night.  The first hour we had about 30 people on the sim.  The second hour, we had an increase in people and when the sim hit about 40 avatars, the time dialation and sim FPS tanked.  Nobody could move and people crashed off the sim and could not get back.  The sim never went down and several avatars remained on the sim.    We had to move the last 1.5 hours of shows to another venue in order to be able to hold them.  

 

After the shows were over, I filed an offline region ticket with LL since I could not even get on the sim to restart it.  The sim was restarted and seemed to stablilize.  We went ahead with our plans to hold a single hour of live music on Sept. 15.  Once the number of avs on the sim approached 40, the same issue occurred and we moved the show to another venue to finish the show.

 

The sim owner filed a support ticket with LL and the sim was changed to a different server.  We did a massive script and av stress test to see what would happen and the performance seems to have stabillized.  However, I frequently have trouble getting audio streams to work on the land itself and often have to relog to hear sound when I teleport to a different sim, something that I never had an issue with before.

 

My venue has hosted events with as many as 95 people on the sim at one time and then we didn't suffer the kinds of performance issues that I mentioned before.  This issue has started to affect not only my venue, but other art, music and dance activities that are important to me and many others in Second Life.  When we can't perform basic functions such as being able to change media streams and having sim performance issues with a relatively low number of avs on a sim, this affects a lot of things that people enjoy about SL.  

 

~Thea Dee

Co-Owner, Ground Zero Music and Art

Co-Owner, Pumpkin Pictures

Soloist, Guerilla Burlesque

Honored Resident
TheaDee
Posts: 2

Re: Major Increase in Instant SIM LAG & Crash during large events

Reply to TheaDee - view message

The sim where I had the issues is Eyefliez.  We also had some issues with music and sound on Atlantizz the week before that (our former venue location)

Nalates Urriah
Posts: 2,427
Registered: ‎03-26-2009

Re: Major Increase in Instant SIM LAG & Crash during large events

Reply to Jadeclaw Denfu - view message

If you want to help resolve problems, understand that LATEST VERSION does not mean anything. It is way hard and time consuming to try and figure out what version you had installed and whether it was or wasn't the latest version. OPen HRLP -> About in the viewer and copy the specific version, like: Second Life 3.4.0 (264911) Sep 19 2012 11:15:02 (Second Life Release)

Since you are discussing a server problem, include the server version, like: Second Life Server 12.09.07.264510, also in Help -> About.

The Lindens simply are not going to try and estimate which version people are on when they wrote their post and possibly be wrong and make things even more confusing. Be explicit. It will make the whole thing move forward faster.

Posts: 2,539
Registered: ‎03-31-2009

Re: Major Increase in Instant SIM LAG & Crash during large events

Reply to Nalates Urriah - view message

Thanks Nalates....

I will add this Server detail to my events posting.  Simon Linden is going to take a look at the logs for the date/time of these events for these two examples.

Toys SL Marketplace StoreToys SL Art Gallery
Posts: 2,539
Registered: ‎03-31-2009

Re: Major Increase in Instant SIM LAG & Crash during large events

[ Edited ]

Reply to Toysoldier Thor - view message

I got an IM from Grafx Newbold tonight that the very popular INSPIRE SPACE Park  on the Mainland SHINDA sim tonight seems to have had the same thing happen to it.

I asked him to provide all the details but here is what he told me so far:

Sim Location:  http://maps.secondlife.com/secondlife/Shinda/35/218/1560

Date: / Time :  Sept 30, 2012  @ about 7:40pm SLT  (time corrected from 8:40)

I will quote what was posted by the person that told Grafx the details:

 anonymous resident: : if it's any consolation, SL in general is being a big **bleep** this evening
[21:42]  anonymous Resident: Tempura was impossible...changing clothes is asking for a quick logout and half my TPs are failin
[21:43]  anonymous Resident: I got that same 'capabilities not granted' message for other destinations as well  Tempura and my outfits got sticky...so I left to try and force a change...and got logged out instead. Logged back to Smith and when I tried to TP out to Tempura I got the capabilities error.  it was a series of crashes before I could get back to Tempura and then it was still bad enough to crash me again, which is unusual

Since there was no owner available to restart the sim... the SHINDA sim was blocking ppl from tping into it.  Then as we were IMing... just like the other problems i reported.  At time of this reporting - the Mainland Sim was still locked.  Since its mainland and satuday night - no one around to force a reboot of the sim.

Sounds like Tempura has the same issues but i cant confirm that. 

Hope anyone else can report details as well.

 

 

 

Toys SL Marketplace StoreToys SL Art Gallery
Posts: 2,539
Registered: ‎03-31-2009

ORe: Major Increase in Instant SIM LAG & Crash during large events

Reply to Toysoldier Thor - view message

OK LL - you got yourself a big problem somewhere in your Server Code..... 

It happened again at another major music event venue and exact same symptoms as the other music event....

 

DATE / TIME:  Sep 29, 2012 @ 10pm SLT

Location:  http://maps.secondlife.com/secondlife/Musicland%20Isle/117/99/24

Population at time of event:  40-55 avatars (sim can support 65 I was told)

DETAILS:

When they were switching Music Stream from the outgoing singer to the new singer (top of hour).....  The sim SLAMMED into LAG.  And then - just like the other events about 5-9 minutes later almost all the avatars were booted off SL.  When they all could eventually log in - no one could get back to the sim - it was blocking them. 

I arrived at the sim that BORKED about 30 minutes later and it let me onto the sim - only 2 ppl were there.  So it seems like it ended up healing itself - unless the sim owner rebooted it.

 

This is the 2nd time this major incident has happened to the same popular music artist in sl.  Its affecting / disrupting a lot of major events.  The Singer's management had to scramble a get everyone on to another sim and advertise for guests to come to the new sim.

 

Toys SL Marketplace StoreToys SL Art Gallery
Cincia Singh
Posts: 285
Registered: ‎02-25-2009

Re: ORe: Major Increase in Instant SIM LAG & Crash during large events

[ Edited ]

Reply to Toysoldier Thor - view message

There is a reason mainland regions are limited to 40 avatars; not 40 on a parcel, 40 total across a region. There's nothing "enhanced" about the setup of a private region ... it's running the same software, on the same hardware, and sharing server resources with as many other regions, as any mainland region. Private region owners "can" choose to set the avatar limit on their region as high as 100, I believe, but that doesn't mean the region will run well if that many are present. So pretty much anything over 40 avatars present starts to make lag almost an expected experience. A region supporting 65 avatars would be running at more than 160% of design limits for a well managed region. I don't understand why anyone would be surprised when the conditions you describe cause a poor experience.


FYI a mainland region restart can be requested in live chat, but you have to get the avatar count on the region below 10 before they'll do it.

Warranty clause voided by payment of invoice.
Posts: 2,539
Registered: ‎03-31-2009

Re: ORe: Major Increase in Instant SIM LAG & Crash during large events

Reply to Cincia Singh - view message

@Cincia,

Please read my OP posting.  I have gone and been a part of very heavily congested sim countless times as its a very common situation with many of the popular inworld musicans.  During any given week I am at sims during live events where I see 30 40 60 and even over 80+ avatars.

Trust me when I tell you that I know full well what a sim feels like when a sim is overloaded with avatars.  I am not naive to think that sims with 40 or 50 avatars on it will cause lag on a sim.  A month ago I was on a sim during a 2 hour 2 singer event where there was almost 100 avatars.  I stood there and enjoyed the music coming from the sim's stream even though I was not able to basically move duing the entire 2 hour show.  THAT IS NORMAL and everyone that attends events like this is fully expecting lag and just shrugs it off.  BUT... we also know that normally - even though the lag grows in intensity with the increase in arriving avatars.... the sim does not crash or go "1/2 stale" and forces all the avatars to be booted out and then does not even allow them to tp back to the sim.  And then - without any intervention from the sim owner - the sim goes back to normal.

What ever is causing this - its not normal lag or activities caused simply because there are a lot of avatars on the sim.

As I mention in following posts..... in all cases I have witnessed, the sim already had about 40 or 50 avatars on the club and there was no serious lag at all.... until all of a sudden a switch was pulled / trigger was hit and the sim went into DUMB mode.

If there was a public jira for inworld residents / sim owners to share info I would have placed this data there.  Thanks to Rodvik's policy to remove Resident JIRA, this is the only place to solicit resident sharing of larger problems.

If the root cause is because of a recent LL server code issue then I am hoping that providing them as much detail as possible will help them identify the problem. 

If this is being cause by some new griefer generated script (or what ever) then providing LL this detail can hopefully point them to their internal logs that can isolate the method the attack is happening and even isolate the griefer who was either on the sim each time wearing the griefing script or placed the object on the sime that caused it.  Through forensics - they could possible go after the griefer as well as shut down the weakness that he / she is using.

Toys SL Marketplace StoreToys SL Art Gallery