Sign in to follow this  
Followers 0
Maestro Linden

Deploys for the week of 2013-01-21

95 posts in this topic

Two previously-RCed projects are coming back this week, with bug fixes. 

 

Second Life Server (main channel)

Since there was a problem with last week's RC projects (BS and LT had a crasher and Magnum had an issue with parcel search), there is no update for the main channel this week.

 

Second Life RC BlueSteel and Second Life RC LeTigre:

BlueSteel and LeTigre are getting the region crossing improvement project.  This project makes sim performance smoother when objects and avatars cross between regions.  This is the same project which was on BS and LT 2 weeks ago; the only change is a fix for a crasher.

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_BlueSteel/13#13.01.14.269010

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_LeTigre/13#13.01.14.269010

Scheduled Wednesday 2013-01-23 07:00-11:00 PST

 

Second Life RC Magnum:

Magnum is getting the interest list improvement project.  This update should reduce the bandwidth usage of viewers due to object updates, and should improve simulator performance, especially in sims with many connected avatars.  This is the same project which was on BS and LT last week, briefly; the only change is a fix for two crash modes.

https://wiki.secondlife.com/wiki/Release_Notes/Second_Life_RC_Magnum/13#13.01.17.269162  

Scheduled Wednesday 2013-01-23 07:00-11:00 PST

 

We will be monitoring this thread as the code gets released, so feel free to note any observations you have about the server updates.  If you have a specific bug you'd like to report, please file a Jira

Share this post


Link to post
Share on other sites

Simulator and viewer are NOT getting along well after "the roll" on Wednesday.  This is a mainland region so we have been having to get support to look at it and trying to talk them into restarting.  We find that the restarts, which did change the region's IP address by, presumably moving it to another host computer.  After clearing cache, group cache and allowing time for things to settle, conditions are not improved.

Symptoms:

  • Chat duplication
  • Attachments displayed as worn in inventory not appearing at all
  • Attachments not appearing completely, i.e., some parts of worn link-sets are not visible.  No pattern detected.
  • Some objects in world not appearing at all or incompletely
  • Avatar not always able to enter region "in a timely manner" failures common
  • Upon entering region, ones own avatar appears frozen in place but is probably actually moving when controls are operated (see next item)
  • After entering region by teleport, my avatar hovered at the teleport destination for many seconds before suddenly "warping" in the direction I was trying to travel by walking.  This is not the normal "object interpolation" jump seen when traveling linearly and receiving sparse course updates.
  • Outfit changes are MUCH slower in the affected region than in the baseline region

Affected region:

You are at 255,719.0, 256,661.0, 59.7 in Lusk located at sim9083.agni.lindenlab.com (216.82.42.19:13005)
Second Life RC Magnum 13.01.17.269162
Error fetching server release notes URL.

Baseline region:

You are at 255,767.0, 256,632.0, 56.2 in Perry located at sim10057.agni.lindenlab.com (216.82.48.67:13009)
Second Life Server 12.12.18.268345
Error fetching server release notes URL.

Viewer used in testing:

Second Life 3.4.3 (268262) Dec 17 2012 12:23:54 (Second Life Release)
Release Notes

Share this post


Link to post
Share on other sites

Maestro,

Bots in Magnum areas are downloading 100kB/sec. Just 1 bot had downloaded over 5GB of data before I caught it. Restarting the bot doesn't help. The usage climbs back to exactly 100kB/sec within minutes.

This image show the bot's usage yesterday. Before/after the roll is obvious:

Usage02.jpg

 

Here is another view that shows the bot's usage for the first 4 minutes after logging into or TP'ing into a Magnum region. The left side is download, the right side is upload:

Usage03.jpg

 

Second Life 3.3.4 (264214) Aug 31 2012 06:30:36 (Second Life Release) Release Notes

You are at 253,238.0, 256,264.0, 25.6 in Bay City - Falconmoon located at sim8018.agni.lindenlab.com (216.82.37.85:13002) Second Life RC Magnum 13.01.17.269162 Error fetching server release notes URL.

CPU: Intel® Core i7 CPU         950  @ 3.07GHz (3064.53 MHz) Memory: 12287 MB OS Version: Microsoft Windows 7 64-bit Service Pack 1 (Build 7601) Graphics Card Vendor: NVIDIA Corporation Graphics Card: GeForce GTX 260/PCIe/SSE2

Windows Graphics Driver Version: 9.18.0013.1090 OpenGL Version: 3.3.0

libcurl Version: libcurl/7.21.1 OpenSSL/0.9.8q zlib/1.2.5 c-ares/1.7.1 J2C Decoder Version: KDU v6.4.1 Audio Driver Version: FMOD version 3.750000 Qt Webkit Version: 4.7.1 (version number hard-coded) Voice Server Version: Not Connected Built with MSVC version 1600 Packets Lost: 8/75,422 (0.0%)

 

Share this post


Link to post
Share on other sites

I have the same problems here:

Sie befinden sich in 279.680,0, 311.658,0, 23,9 in Red Roses auf sim9095.agni.lindenlab.com (216.82.42.31:12035)
Second Life RC Magnum 13.01.17.269162

 

  •  I can't move on Magnum Sim my avatar seems to be frozen, but it actually moves but i don't see it
  • The world is loading very slow
  • Other Avatars don't load @ all
  • High Paket lose all the time : Paketverlust: 10.156/69.062 (14,7%)

It run fine on  Letigre and secondlife server Release version.

Share this post


Link to post
Share on other sites

Ardy, Triple, NogarDrevlis: thanks for the reports.  We are investigating these issues on Magnum.  At a glance, the issues do seem like they could be triggered by bugs in the interest list changes.

Share this post


Link to post
Share on other sites

There is an issure with an avatar with the EM located in sim10040.agni.lindenlab.com (216.82.48.50:13000)
Second Life RC Magnum 13.01.17.269162 ( Bonta ), after the rolling restart that persons is ejexted offsim at  0,0,0 point, all avatar and object disappear but only in that specific Magnum region and only that avatar (i hope to don't be OT, i write for her cause she don't talk english very well), thanks for the attention.

Share this post


Link to post
Share on other sites

Triple -

I've been examining the code to figure out why the simulator might start sending lots of data to a bot.  I think I identified one mode:  if the bot never sends a valid AgentUpdate UDP message (which includes info about the camera's view angles and draw distance) then the server-side culling will be using some initialized values that might trigger resends of data.  I'll change the initialized camera properties to be more sane defaults and try to get this out in an update, but if you happen to know if your bot implements the AgentUpdate message or not I'd love to hear back.  At the moment this is my only theory for the cause of the problem.

Share this post


Link to post
Share on other sites

Sorry Andrew, I don't have any way to tell that that I know of. The bot's console screen remains silent, and the bot even continues to function normally as far as I can tell.

If you can set up a test region somewhere (on agni) it would be easy enough for me to TP the bot there and watch the traffic meter.

ETA: Does the 100kB/sec cap the data appears to reach after 2 minutes give a clue? Is anything capped at that rate?

 

Share this post


Link to post
Share on other sites

Andrew and Maestro,

I just realized I gave some incorrect information. This problem is occurring with the LibOpenMetaverse bot, not a JVA bot. I used to use JVA but switched. There is another thread concerning a problem with JVA bots.

This bot is open source, so if there if is a good search string or another way to tell if valid AgentUpdate UDP messages are being sent let me know.

 

Share this post


Link to post
Share on other sites

Triple -

I downloaded the LibOpenMetaverse code.  It appears that libOpenMV's "bot" code is their "TestClient" program.  I did some quick searches through the codebase for certain keywords and it appears that the TestClient does indeed support sending AgentUpdate messages, which makes sense since these messages must be sent if your bot is going to move around.  However it is not clear whether a valid camera "Far" clip is sent in your case, or if it is sending zero.  The values that are put into the AgentUpdate message depend on some config files.  It makes sense to me that someone running a chat bot, or a parcel search bot, would set the camera's Far clip to zero to minimize streaming of visual objects which the bot would probably just ignore.


I also looked over our own code again, but more carefully, and I'm prepared to put more confidence into my theory.  I will try to submit a patched version of the code for update next week.  It will have to pass QA over the weekend so there is no guarantee that it will get released, but I'll be working on that today.

Share this post


Link to post
Share on other sites

Since the restarts my Magnum region has been experiencing a number of issues. Avatars not loading (mine and others), items rezzing very slowly or not at all, slow fall through the floor of skyboxes, any item with movement "freezes" as soon as it's out of camera view and then snaps back to where it should be when the camera is panned back around, scripts not responding and unable to reset them, unable to "open" objects, and other issues.

You are at 245,572.0, 241,715.0, 30.5 in Grand Pacific located at sim7400.agni.lindenlab.com (216.82.34.147:13001)

Second Life RC Magnum 13.01.17.269162

Share this post


Link to post
Share on other sites

Sky -

I'll number your symptoms so we can specify:

(1) Objects and avatars not showing up

(2) Objects showing up but slowly

(3) Slow fall through floor of skyboxes (this is about avatars specifically?)

(4) Objects not updating position until they are back in camera view

(5) Unable to open objects to view their contents


Everything could be explained by a bad net connection except (4) which is known new behavior in the Mangum RC.  I'm not positive that there was a network glitch, but some more info may be able to rule it out.  Are symptoms (1)-(3) and (5) happening constantly or sometimes?  Do they affect everyone at the same time or only some people? Item (3) sounds like "rubberbanding" which can happen when experiencing very long ping times (packet travel time) to the server -- did you notice other objects rubberbanding at the time?  and do you see horizontal rubberbanding as well as verticall "fall through"?

Item (5) sounds like outright network failure or extreme packet loss.  Did you notice packetloss in the stats (CTRL + SHIFT + 1) when this was happening?

BTW, I've got a fix that will reduce (4) significantly, but it is not yet ready for RC.

Share this post


Link to post
Share on other sites

My first thought was also network. But I'm having identical issues on 2 different networks and 2 different computers. Both networks and both computers have been reset and speed tests show normal. I'm usually the only one in the region, but I did have a friend over yesterday who said there was more lag here than normal. 

Symptoms 1 & 2 have been pretty consistent since the restarts. Prior to the restarts I would occasionally have items not show until I right clicked on them but nowhere near as much as it's happening now.

Symptom 3 is most of the time since the restart but not every time. Yes, it's my avatar and it just slowly sinks through the floor of the skybox until it's 5-10m below the floor. Then it usually pops back up to the floor but sometimes just hovers there below it.

Symptom 4 is constant since the restarts and is only happening on objects that move (ie: breedables). It's like the movement scripts keep going but the prims themselves stop as soon as they are out of camera view. When the camera is turned back to them they shoot horizontally to where their script has actually moved them to. I'm having a hard time explaining it.

Symptom 5 - I can "edit" to view contents, the issue is when I try to use "Open" to copy the contents to inventory. This can be worked around with Edit and dragging the contents to my inventory but it's not an issue I had before. This is only happening on objects that have a fair number of items in the contents, ones with only a few still "open" fine. But prior to the restarts I was able to open objects with many items inside with no problem. I just tested that a bit further. By dragging about half of the contents out of the object using edit and then clicking open and waiting several minutes it did finally open. So it's possible that there was just too much stuff in that object although I did have others last night that did the same and they weren't nearly as full. Packet loss was 0% just now while I did that.

Edit to add: It should also be noted that I am not experiencing these issues on other regions, just the magnum one.

Share this post


Link to post
Share on other sites

Could some of the issues that appear to be network issues be caused by the bot bandwidth problem? I know that on my home network I was going crazy trying to figure out what my strange problems were until I discovered all the bandwidth that was being hogged by the bot I host.

Seems to me if there were enough bots in the same sim that they could be using so much of the server resources that it could explain some of these other issues.

 

Share this post


Link to post
Share on other sites

If someone could send me a landmark to a region or skybox where these problems are showing up I'd appreciate it.  I'm going to check out Lusk to see if I can witness any of Ardy's reports.

Tiple, I don't think Sky's problem is related to bots since  that region is usually empty.

Sky, your region on the Magnum channel is public, but you've got some parcel setting that prevents teleports.  I'm going to temporarily add myself to the Estare access list so I can look around to try to reproduce the problems.

Share this post


Link to post
Share on other sites

Just wanted to add that everyone we are seeing coming to Firestorm Support having the problems on Magnum have horrendous packetloss - 30-40% in some cases. They are only getting the high packetloss issues on Magnum regions.

Share this post


Link to post
Share on other sites

I notice that although teleports out of my home Blue Steel Server (iaconelli) have been a little troublesome of late they are now nigh near impossible since the roll.

Share this post


Link to post
Share on other sites

I'm on the Magnum server and have been having trouble attching objects to myself.  I've tried other sims that are on different severs, like Pooley, and I do not have a problem attaching anything there.  

Share this post


Link to post
Share on other sites

We ran one test with Firestorm from a residential connection but were not able to reproduce the problem. The test involved teleporting from pointA to pointB and back to pointA to see if all the pile of scripted objects in pointA would all show up on return -- they did.  We noticed that the content took longer to show up in Firestorm, whereas it shows up almost immediately in the official SL client.  I wonder if Firestorm is using the a gradually increasing draw distance hack to help content get sorted near to far?  I've heard rumors of this feature in TPV's but do not know if Firestorm is using it.

For this test we had a 64m draw distance.  I believe Sky Linneaus is typically running with a draw distance that is larger than the official viewer even allows (512m).  We should repeat the test with a maximized draw distance in a region with a lot of content.

Whirly, I take it that the packetloss is primarily happening on TP arrival?

Share this post


Link to post
Share on other sites

If you were using our latest official release, it does not have the new HTTP texture fetch code that Monty Linden wrote, which may be why things take longer to appear.  This was added to Firestorm after that release, and will be in the next official release.  As for the increased draw distance 'hack', it is nothing more than modifying panel_preferences_graphics1.xml to increase the max value from 512 to 1024 for the draw distance slider, something anyone can do to any viewer at any time.

Share this post


Link to post
Share on other sites


Andrew Linden wrote:

 I wonder if Firestorm is using the a gradually increasing draw distance hack to help content get sorted near to far?  I've heard rumors of this feature in TPV's but do not know if Firestorm is using it.

 


Yes Firestorm has this

Under Avatar -> Preferences -> Firestorm -> General -> Enable Progressive Draw Distance Stepping

Code is here: http://hg.phoenixviewer.com/phoenix-firestorm-lgpl/rev/f6b1dc9c6d32

 


Andrew Linden wrote:

 

Whirly, I take it that the packetloss is primarily happening on TP arrival?

Not as far as I can tell. Their packetloss appears to stay constantly high while on Magnum regions.

 

 

I am at the Skybox from BUG-1460 currently and I am unable to reproduce the problems on Firestorm. I have tried with a 1024 draw distance, a 64m draw distance and with both progressive draw distance enabled and disabled.

I have been sending people affected by this to test on Viewer 3 also and they have the same symptoms.

I have been unable to find anything in common between the people affected by this issue and those that arnt. But those affected appear to be able to consistantly reproduce on any Magnum region and those of us who cant reproduce it have not been able to find a way to trigger it yet.

Could it possibly be only affecting people with certain router hardware? I have seen a couple of reports now of internet connections cutting out when entering  Magnum regions only.

 

Share this post


Link to post
Share on other sites

I have a Magnum Sim and had badly problems today, everything happens with a delay, moving not visible, then i found me back 50m from my position standing against a wall, i can turn but not move, it comes in waves, not permantly.starting a script, whos scrolled a text, is stucked, never happened befor, and suddenly its done.

Now im hunting, jumping from Sim to Sim, everything OK, until i landed now at a Magnum Sim, same problems here.

after some minutes i can move

 

Share this post


Link to post
Share on other sites

It might be good to note here that recent viewer-development (Snowstorm) and CHUI builds have some issues of their own with object updates.

It will be best to run something else while looking for problems on Magnum.

 

Share this post


Link to post
Share on other sites

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
Sign in to follow this  
Followers 0