Deploys for the week of 2013-01-21

[Ayesha Askham]

Adding my comment to the stock

I am finding teleports out of LeTigre sim 9030 Woods of Heaven very poor, as of the last restart.  I believe that Linden Lab may have a network issue at the Dallas datacentre that is drastically reducing the available bandwidth and download capacity to residents.  This is resulting in high (c20+%) packetloss on TP, which results in disconnects.

I have examined the connection speed and stability of a number of European and US servers, and the Dallas datacentre is by far the worst both in bandwidth and stability terms.

While many of the reported issues are related to the Magnum RC channel, by no means all are.  This issue is proving to be a showstopper for me at least.

[Ayesha Askham]

Some additional info:

During logins I have seen twice something that I have never seen before, in 5 years in SL:

while the status bar is moving across as the login proceeds it reaches "Requesting Region capabilities..." and it gets to "2nd try" before it completes to "contacting Region" and "Loading world".

These, combined with several abortive attempts to log in suggest that something within Linden Labs' intranet is not performing "as expected".

Now since it is the weekend I do not expect Linden response until 8am SLT Monday, but this is a serious issue that is affecting a good many accounts, inworld numbers are a good 10% down on normal.

I would really like to know what is going on.

[Hetress]

After a ticket the reply from the Linden at my friend for sort her problem with that region on Magnum server (where she pay the rent of 1/2 sim at an agency) was: you need to be Premium account. ??? Only the land owner need it for pay the tax, there is a bug with an avi in a Magnum server and i read here after the rolling restart of the server it's not the only one case. When is possible to sort all this troubles? Thanks for the attention.

[DaleMaison]

Hi there, I have had trouble after the magnum update on the Sim i use

My Bot that i host through Pikku Bot are going crazy after the update, they keep sending commands via IM three or four times, my translator comes on by its self when i seem to return to the sim.

I have found the problem to be the sim and the update, as when i move my bot of the land, its working fine, soon as they on same parcel, script and bot, it goes crazy

my script talks to my bot via LLInstantMessage, and no matter what work around i been using i still get same reults.

so for time being, my bots and scripts have to be kept apart if they are both to be kept on a magnum server, there have been number of ppl reporting teh same duplication in commands been sent in the pikku support group and all after this update

 

[WolfBaginski Bearsfoot]

I can't give any info that would help pin things down to particular servers, I didn't take notes, but movement and sim scrossing have certainly been jerky. Rubber-banding, unexpected changes of heading on sim crossing, and a couple of minutes of completely crazy movement on one sim crossing before things stabilised. Also, object rezzing was sometimes extremely slow, the road not being visible, though apparently enough was there for the Physics Engine. That was particularly obvious for the bridges at Kaminari and Clarksburg and the road between those locations.

That wild sim-crossing included a vehicle moving several hundred metres sideways and continuously pitching, while the camera position remained stable relative to the vehicle centre. As near as I could tell from the minimap. the lateral displacement was parallel to the region boundary being crossed. The vehicle was always close to the visible terrain surface. It lasted long enough that any recovery at all seemed remarkable. Best guess at location is Sven-Grishin, but that could easily be a couple of regions wrong, north or south.

Some of the effect could be down to the Viewer and whatever sort of interpolation is supposed to smooth out glitches. I could see the pitching being the consequence of some bump in the road surface, but the lateral displacement suggests some specific bad data getting into the system.

I'm not sure why I hung on as long as I did: it was the sort of sim-crossing behaviour which prompts despair and log-outs.

 

 

[Hilver]

Sicne  wednesday  i  have  problems  that  i  cant  see  people  at  my  sim  and  who  arrives  cant  see other  people  too.

Linden  came  over  to  check  it  but  they  said  it  all  normal.  Last  night  there was  an event on the  sim a lot of people  werent  able  to  see  each  other  they saw  onle themself  even  there where a lot of  friends.

I noticed  i only  have  it on a  Second Life RC Magnum 13.01.17.26916 if im  on a Second Life Server 12.12.18.268345 its  all  normal.

This is  verry  frustated   also when a   Linden  says  there  nothing wrong

Hilver

[Shayne Hesten]

Just to add my own comments to this thread.  This bug is not affecting me, however, we held a small gathering on our sim last night and 3 of our guests were unable to see rezzed objects and none of the other avatars were visible to them.

On discussing it, it appears all three (plus other avatars they know) are experiencing this same issue when they tp to any sim running Second Life RC Magnum 13.01.17.269162.  Avatars remain invisible, objects remain invisible.

I tp'd to another sim running Second Life RC Magnum 13.01.17.269162 to check and discovered that whilst I could see all avatars present, many of the objects did not rez for me.

I can state that one avatar experimented by doing a full clean install of Firestorm, it did not help.  He then logged in with the official LL viewer, again it did not help.  He owns two full sims and one homestead and two Lindens visited his land but could find nothing at all wrong with the sims.

The avatars I know who are reporting this problem are all based in Europe but not in the same regions, so it is not a wholly geographical bug.  It also affects their alts, so it is not an avatar-based bug.

 

 

[Kochloeffel Pfeffer]

I am one of the Owners of the Club on Hilvers Sim and a friend made some tests with the firestorm viewer team and they said its a network problem on server side, He told me about a Packet loss of 60% and more. Regards Jessi

[Whirly Fizzle]

More reports of problems on Magnum are on this thread

[Chilli Riederer]

I have exactly the same issue, paket loss on magnum around 20% or more. le tigre and main are good

Sie befinden sich in 261.171,0, 344.520,0, 30,7 in Funny Island auf sim9924.agni.lindenlab.com (216.82.46.66:13001)
Second Life RC Magnum 13.01.17.269162

When i log in on magnum server, my whole inventar loads new, starting from 0. When i log in on main, my inventar is already loadet, it only have to load the last few hundred items. i have no bots on any sim.

[Andrew Linden]

Thanks for the info everybody.

Firestorm is off the hook -- it has been tested and works for some people, so the source of the problems must lie elsewhere.

I've got a theory about the bot problems and a corresponding fix lined up for next week.  The problem is that the current simulator on Magnum will initialize the viewer's camera as having a zero draw distance when they first connect to the region.  It expects the SL client to send a non-zero draw distance in the AgentUpdate messages, but some bot implementations may specify zero draw distances.  One might expect that a zero draw distance would minimize data from the server, however by a quirk of the logic it now causes the visibility of some objects to be constantly cleared and then streamed again.  This thrashes the server a bit and causes it to send a lot of packets to the bot.

The fix is to sanitize the "Far" feild of the AgentUpdate messages at the server.  If a client sends an unreasonable value there we will clamp it within an expected range.  Also, the draw distance is initialized to a non-zero value -- just in case the client NEVER sends an AgentUpdate.

If you have a bot on Magnum you probably want to configure it such that it sends a non-zero draw distance in the AgentUpdate messages.  This appears to be possible in libOpenMV bots -- dunno about other varieties.

Some (but not all) of the reports here sound exactly like bad network connectivity: rubber banding and packet loss in conjunction with objects not showing up or taking a long time to appear.  But why are most of these reports being seen on Magnum?  I'm not sure if the Magnum servers are all located in just one of our colocation facilities or are distributed across several.  I'm going to query some of our operations people to see if they have any ideas about this.

One of the other changes currently on Mangum is that the server uses the CameraCenter field of the AgentUpdate message for ALL visibility logic, instead of using the location of the Avatar (previously it was a mixture of the two positions).  If an SL client is sending zero, or some other incorrect value, for CameraCenter then I would expect the visibility calculations to be wrong -- some things woudn't show up because the server thinks the camera is elsewhere and is sorting objects accordingly.  I would expect all TPVs to send the correct CameraCenter, but some simple bots might not.

[Hetress]

That's for the bots but for all the normal avi? Firestorm, SL viever, Kokua or Singularity the problem is the same, can't see the people or the objects on lands with Magnum server, in the other lands there is no problem.

[Taylor Wassep]

I have had this Magnum Server Bug since last Thursday, I've had the following problem:

1. When i first log in, I sometimes will be at the corner of the sim and i wouldn't be able to see my avatar.

2. I would be able to TP home, but when I do.  I wouldn't be able to rezz completely.  It's usually my hair, or some attachments that just won't rezz; weirdly all my clothing layers would rezz fine.

3. If i make a copy of like a pose stand or something, it wouldn't copy, even if it is my own pose stand.

I've tried a couple of clean reinstall, cleaning my cache, trying different viewers.  I just hope that this problem gets fixed ASAP.

[BlackMagi Darkwatch]

Here's a weird observation - for the past couple of days when looking at the mini map the sim in the adjacent (SE) corner isn't there on the mini map and also when looking into the sim on the ground level 95m's. Looking at the world map the sim IS there.  I can fly into the sim though but as soon as I leave it disappears even when looking in from the SE edge of our parcel?

Sim I was looking at:

You are at 292,663.0, 278,215.0, 107.5 in Mists Edge located at sim9377.agni.lindenlab.com (216.82.43.80:13000) Second Life Server 12.12.18.268345 Error fetching server release notes URL.

Home Parcel in SE corner of Le Tigre sim

You are at 292,602.0, 278,278.0, 95.4 in Riback located at

sim7658.agni.lindenlab.com (216.82.36.225:13002) Second Life RC LeTigre 13.01.14.269010 Error fetching server release notes URL.

Thanks Black

 

ETA:  Since the restarts this week (28/01/2013) the sim that was previously 'not' there is back and the mini-map is no longer red!

[Jessica Lyon]

Andrew Linden wrote:
"Thanks for the info everybody.
Firestorm is off the hook -- it has been tested and works for some people, so the source of the problems must lie elsewhere."


I would just like to point out that making an assumption that Firestorm is to blame for something because there is a disproportionate amount of reports coming from Firestorm users as opposed to the Linden viewer users or any other viewer is poor practice. Firestorm has a disproportionately larger number of users than any other viewer and generally a large portion of these users are well educated and willing to report bugs because we make a lot of effort to educate, help, interact and generally encourage them to report issues they find.

Also, I do not regard the Progressive Draw Distance feature in Firestorm to be a ‘hack’. It fixes the long standing failings of the Linden interest list 'without' side effects or bugs. To date, all efforts to improve the interest list system have always resulted in more problems. I've been around long enough to remember all those efforts.

</exit soapbox>

My team and our users are more than happy and honored to continue trying to help you (Linden Lab) improve the user experience, truly happy to do it. But I do resent being blamed...

Thanks.

Jessica Lyon

Project Manager

The Phoenix Firestorm Project, Inc

[Whirly Fizzle]

Hi BlackMagi,

That sounds like you may be seeing SVC-8130 - some neighboring regions are not visible

 

[gaetano Kidd]

hello

 

Since thursday i can't see people ( avatars and names) in the sim "Energy" but they can see me. I can see local chat, buldings and objects.

i'm on the server Second Life Server 12.12.18.268345 with Second Life 3.4.4 .

 

Thanks

[Qie Niangao]

Just to be sure: the problem is only on that "Energy" sim, and you can see avatars normally elsewhere?

(If, rather, you can't see avatars anywhere, Control-Alt-Shift-4 might be worth a try.)

[gaetano Kidd]

the issue is just on this sim... i can see avatars in all others sim

[DaleMaison]

i Use my Bot through a hosted service, so not sure what draw distance is set at

Would this problem regarding draw distance fix it duplicating IM chat 3 to 4 times a second

Just wondering as thats my only problem i hav faced with the update

and with been in advertising and trying to stick with TOS while this is happening is difficult at times, but have found a work around aslong as the bots and script dont work on the same magnum sim

[Blake Burger]

I have seen a lot of the problems in my magnum sim also. With the avatars not appearing to bots not working properly to the objects not moving to their updated locations until u look at them.  I am just conferming this is occouring on my sim also.  Hopefully I can use my bot again soon...   I also wonder what the "Error fetching server release notes URL." is about as noted below.....

Second Life 3.4.0 (264911) Sep 19 2012 11:15:02 (Second Life Release)
Release Notes

You are at 128,465.0, 397,895.0, 987.0 in Phantom Island located at sim8945.agni.lindenlab.com (216.82.41.121:13025)
Second Life RC Magnum 13.01.17.269162
Error fetching server release notes URL.

[Whirly Fizzle]

---

Blake Burger wrote:

   I also wonder what the "Error fetching server release notes URL." is about as noted below.....

.

---

---

This has been broken on and off for a long time.

See SVC-7598 - Error fetching server release notes URL

[Andrew Linden]

Sorry Jessica, I didn't mean "hack" in any pejorative way.  I meant it as a "solution to a problem using unexpected methods or tools" -- I'm definitely not against such "hacks", and I was quite intrigued with this TPV innovation when I heard about it years ago. For some people "hack" has a negative connotation, so perhaps a better word would be "workaround" since the expanding draw distance feature is to reduce a "bug" that is external to the viewer, namely the poor near-to-far sorting by the server.

Meanwhile, I've been thinking about the symptoms and also reviewing some of the changes in the server code looking for clues....  I have a tentative theory that I'm not satisfied with, but I'll share it anyway.

The new interestlist code packs the same average bandwidth per viewer over a simulator frame, however it builds the individual packets a little faster and thus hands them to the network hardware in tighter temporal clumps.  Picture, if you will, bursts of data that have the same area and frequency as before, but higher peak and narrower width.  My most favorable tests for sending a storm of full updates showed a speedup of about 50% less time (say 105 msec to pack instead of 200 msec), so while the average bandwidth is the same the momentary bandwidth on the wire might double.   The theory is that these narrower bursts of data represent higher momentary bandwidth that is overwhelming some packet buffers of network equipment between the servers and some viewers.

One way to test this theory is for those afflicted to try the following:

(1) Via the preferences reduce the total bandwidth.  Preferences --> Setup --> Total Bandwidth

(2) Reduce the draw distance. Preferences --> Graphics --> Advanced --> Draw Distance

(3) Relog

I don't have a lot of confidence in this theory, but if someone could test it and report back we can either elevate or retire the idea.

One last thing I'll mention.  Although I can't find the post now I recall someone mentioning in this thread that one of the symptoms they noticed was that some linked sets were only showing up as their root prims.  When the visible prims were selected the rest of the object showed up.  This problem can be caused by out of order packets where the child prim info arrives before the viewer knows anything about the root -- the viewer is not very smart about connecting the child prims to their roots in this situation.  Out of order packets don't explain all the problems reported here, but are indicative of general network problems.

Edited to fix bad formatting.

[BlackMagi Darkwatch]

Tbanks Whirly

 

Ok, I see that it is being worked on - perhaps the server restarts will work this one out as normally it's there so will wait and see.

 

Thanks

Black

[Ardy Lay]

Andrew,

My round trip time to the host I first reported this issue against is quite brief:

Ping statistics for 216.82.42.19:
Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 79ms, Maximum = 95ms, Average = 87ms

Most of that is burned on a rather startling number of hops with Level3 in their names:

4 15 ms 15 ms 15 ms cer-edge-17.inet.qwest.net [63.149.0.249]
5 12 ms 8 ms 15 ms chp-brdr-03.inet.qwest.net [67.14.8.194]
6 31 ms 30 ms 32 ms 63.146.27.18
7 95 ms 96 ms 100 ms vlan52.ebr2.Chicago2.Level3.net [4.69.138.190]
8 114 ms 111 ms 111 ms ae-14-14.ebr1.Dallas1.Level3.net [4.69.151.118]
9 104 ms 130 ms 99 ms ae-1-8.bar1.Phoenix1.Level3.net [4.69.133.29]
10 67 ms 63 ms 63 ms ae-5-5.car1.Phoenix1.Level3.net [4.69.148.117]
11 105 ms 103 ms 102 ms LINDEN-RESE.car1.Phoenix1.Level3.net [4.53.104.14]
12 122 ms 130 ms 125 ms sw-core0-84.phx.lindenlab.com [216.82.7.138]
13 105 ms 101 ms 104 ms sim9083.agni.lindenlab.com [216.82.42.19]

There is some variablility but one has to consider that ICMP ECHO and TTL exceeded processing is very low priority on modern networking hardware.