Second Life Server

[Toysoldier Thor Member]

Of those number you pointed out, the last two stats would make a network guy like me take notice...

Pending Downloads

• Bad: 531
• Good: 1

Total Unacked Bytes:

• Bad: 581.2 kb
• Good: 3182.3

Again, I dont exactly know the source of these metrics (i.e. exactly what resource is reporting these network metrics - the sim app instance?  the metric from the OS of the physical server?), but if we assume the metrics are from the OS/kernel or whatever their VM is on the physical server, these two metrics are saying that the server is not able to keep up on servicing network requests and as a result the sim is being forced to hold communications with any of its respective clients.

The BAD sim has much less actual traffic to service yet it is majorly overwhelmed at a network level - the two good healthy sims are processing far greater network traffic loads and not being overwhelmed.  Something here is WRONG.

If this is true, a network stat that we also cannot see but would be very interesting to see the server's "Session Count".  If a max has been reached and/or if communications are UDP, another symptom would be that clients would be dropped.  Could explain the drop of all the avatars on the sim all of a sudden.  With a lot of the avatars dropped, maybe whatever is overwhelming the network layer of the sim gets enough relief that eventually it catches up and lets ppl back on the sim.

LL Engineers need to focus on the root cause of what is causing the network layer of a busy sim to go stale or easily overwhelmed.  This could be the root cause to many sim lag issues.

[Toysoldier Thor Member]

Qie...

Notice posting #13 of this thread - added info to the theory of something specific in the LL DC ... the hardware or different kernel....

One of the GOOD sims ( the one with 65 avi's on it) was one of them that was REAL BAD with crashing until a couple weeks ago the sim owner opened a support ticket and LL resolved the problem when they switched something - the poster suspects they moved the sim to different hardware. 

Ever since they moved the sim - the massive lag issues went away.  and we all can see how well it takes large load.  Specifically, the network stats are very healthy.

LL staff can find this ticket to check the details of what they left behind and check the Network layer / driver differences between the two.

[Tymus Tenk]

Thanks Toy and to the rest who confirm that others are experiencing this unusual increase in SIM lag during large events. As a venue owner this it has become almost impossible to hold an event.

We have been experiencing the exact same issue during our shows over the past several weeks...for no apparent reason.  During a performance I keep watch on my sim stats and have noticed too... that at some point...even with as few as 20 to 45 .. with script counts and running times well within what would normally not cause a problem .. suddenly drop in performance to where the Sim FPS & PHYSICs FPS drops to almost nothing (as the one graphic above of the statistic bars shows).  Not always, but sometimes this can happen when guests are tp'ing on top of one another ...but unlike in the past ...once all have arrived....and rezzed....the sim just doesnt recover ... everyone starts crashing ...then the sim goes down ... ending up with having to submit a ticket as I cant' get into that sim to do a restart on my own. 

All sims in my island estate are on the Second Life Main Servers.  Before giving up on doing what I can to support Live music in SL , I keep hoping that the 'next' server rollout will at least take care of that problem...instead of creation more problems :/ I've done what I can with submitting tickets .. just waiting.

'Normal' SIM FPS & PHYSICS FPS run between 44.5 & 45 on each of the sims.

Calas Galadhon Park sims/owner

OZ Nightclub, Glass Pavilion, The Dolphin Cafe owner

[Toysoldier Thor Member]

I have officially create BUG-355 with LL and pointed them to this public viewable thread

https://jira.secondlife.com/browse/BUG-355

(due to recent LL policy changes - SL Residents cannot see any JIRA's created by another resident.  So I have filled in the formal bug that basically tells LL to come to this thread to get all the details collected by the resident community).

I have asked that LL look into a possible Network related bug in their SIM code or the old/new physical hardware or its kernel related to the network drivers / transport, or a LL DC routing issue.

I am hoping Simon or Andrew Linden will pick this up and focus investigation on this bug.

[Simon Linden]

Thanks for the reports, Toy ... there's some good information there that seems to shows the simulator getting into trouble when there are network problems.  

[Sinshine Moorsider]

Forgive my non-techie status.

I wanted to contribute to this becuase in my 13 sim rp estate (private land), we have been having the same issue. The difference is many of you seem to have problems with many avatars, in clubs (with possibly more scripty stuff) and sometimes stream changes. This has happened on many of my sims with just a few avatars, very low script time, and no stream changes. The sim stats (dilation and fps) either fluctuate or tank completely. Sometimes there are associated crashes - sometimes not.

Also, I wanted to ask all of you if this seems a lot like the kernel bug referred to as "time warp" and if not, could you explain any differences? To me, it looks exactly the same.

[Toysoldier Thor Member]

@sinshine,

Next time one of your sims encounters one of these major lag events.... take a snapshot of the Advanced Performance screen and post it here.  The important stats to look at are related to the NETWORK stats. 

What has been initially noticed (and we would like more evidence of it from stale sims) is that some event triggers the the sim or the server kernel's network driver or something related to the network to not rocess network packets at a normal flow.  Even at a low avi count and with other metrics looking normal, the network stats show unusually high pending downloads and very high un-acked packets.

But, we need more examples (snapshots of the perf stats) of a sim when it has fallen into this state.  Also make shur you mention date/time and the name of the sim. 

[Toysoldier Thor Member]

Hey Simon.... i will ask you at the meeting in about 3ish hours but has there been and digging into this issue with the network anomaly? 

[Toysoldier Thor Member]

SO just to mention that there were two more large events from two sims where - as we are calling it - the sim went stale during the event and several avatars were all booted out of SL and the sim did not crash.  One event happened badly on Saturday but the sim owner didnt capture the stats during the crash.  I informed her what to capture if it happens again.

The other event was last night at a busy event at a sim.  The sim owner doesnt want me to mention his sim since he feels that mentioning it might scare away his visitors (as if music loving SL residents are not fully aware of laggy crashy sims and would not show up to a live event simply because of what was posted in an obscure SL forum).  shrugs...  regardless... the events are still happening.

[hombra]

This was Help Island Public on 2012-10-18 at 11:07AM. It's the second time I find such an enormous lag. The last one was less than a week ago and it was the same kind of lag with the huge net time. I hope this is of some help.