Triple Peccable

One of my items had become unlisted also. Everyone should check their listings for items that have suddenly been unlisted.

I believe the diagonal sim visibility problem is not a simulator<>simulator or a viewer<>simulator communication problem, it is a simulator<>central server communication problem. The problem started over a year ago, when I filed a JIRA about TCP connections remaining open, causing 1000's of connections to pile up as one moved through different sims. At the time the issue was resolved, it was stated the problem wasn't with the server code, it was with the central servers code. That is when the diagonal sim visibility problem initially started. I believe a patch was applied back then that actually didn't fix the problem, it just masked it. I say that because ever since then a series of different problems have cropped up, all seemingly related to the simulator getting the wrong information from the central servers. Since then more patches have been applied to fix these different anomalies as they crop up, to the point now where it might not be fixable. Here is a little demonstration that illustrates the corrupt data coming from the central servers: Go to an area with lots of adjoining sims. I used Blake Sea - Nelson (128,128) because most of the surrounding area is open water. Open your mini-map, and zoom all the way out. Then turn your draw distance all the way up. As they connect, you will see all the surrounding sims on the mini-map painted as blue squares. Now, turn your draw distance all the way down, and wait until the connections with the distant sims are dropped (at least one minute, each square changes from blue to black). What you should see are only the surrounding sims in blue, but what I saw just now when performing this test is only 2 of the adjoining sims staying connected (the ones to the west and the south). There are 2 other sims in the wrong spots staying connected, which I assume should be the ones to the east and north. I submit that these sims are staying alive in the wrong spots because the data coming from the central servers is for the wrong sims (or the simulator is requesting the wrong data), and that it has been that way for over a year. Even though I am the only avatar in the sim, I see lots of green dots plastered against the border to the north. Those avatars are in the sim to the north, but because that sim is missing the viewer has no other place to draw the green dots. That is just one example of the many different types of problems this central server problem is causing. Red mini-maps is another.

Maki Guyot wrote: If any Linden Lab employee is reading this; you're pushing your userbase away with your constant changes. You're effectively pushing your product on the road to bankruptcy. Have fun with it. I'll just sit on the sidelines and watch the ship sink. You clearly don't want my business. There is an angle you might not have considered: LL is indeed allowing older residents to drop out of SL without worrying about the losses, possibly even encouraging us to leave. Why? Because we are used to the old Linden culture, we feel like residents, and we expect to be treated like residents. That is not what LL wants anymore, in my opinion. I believe they want "players", not "residents". Residents demand too many rights, making them too difficult to support, and they no longer have the staff for such support. They are hanging their hat on the successful conversion of SL from a virtual world to a gaming platform (again, in my opinion). They already lost the vast majority of my money. My tier + advertising was $350USD/month at one point. Now is it $8USD/month. Just as in your case, they made it clear that they don't care one bit about that loss: http://community.secondlife.com/t5/Mainland/Farewell-Dobinson/m-p/1528457

Here is a little info on a connection between the missing prim issue and some of the problems with scripts people have been reporting. I have a bot that wears a HUD, and on a daily basis TP's to a lot of sims. About the time the interest list project was first introduced, script errors started occurring on a once-in-a-great-while basis. The problem turns out to be related to the missing prim issue. Every once in a great while, one prim in the HUD the bot wears fails to rez after a teleport (it is always the same prim). That prim has 1 script in it, and when the problem occurs this mono error is displayed: [2013/04/25 06:21] Crosshairs: Crosshairs [script:Ban Line HUD Crosshairs Script] Script run-time error [2013/04/25 06:21] Crosshairs: System.NullReferenceException: Object reference not set to an instance of an object at LindenLab.SecondLife.UThread.UThread.Run () [0x00000] in <filename unknown>:0 at LindenLab.SecondLife.Script.Run (ScriptEvent evt) [0x00000] in <filename unknown>:0 At that point the HUD will not function again until I log the bot in with a normal viewer in order to reset the scripts in the HUD. A similar type of problem could have been what happened to Theresa's cat. If there is a JIRA that is relevant to this and someone can get this info into it, please do so. ETA: It seems to me that if the missing prim issue was strictly a viewer-related bug that it couldn't crash server side scripts. OTOH, since this is a HUD maybe this bug is completely unrelated to the missing prim issue. But still something to think about.

Maestro and Andrew, I wanted to report on the bot's usage. Fixed! Before this incident the bot's "normal" usage was 5 MB / hr. That is so normal no one would suspect anything. But now it is 1 MB / hr! It has never been that low before, ever. The improvement might be from the interest list changes, but since the bot is parked 3300m up with a very limited draw distance, I think it is from this UDP bug fix, and will help with more than just bots.

WolfBaginski Bearsfoot wrote: 5: The old bugs are still there, such as the parcel-full problem for vehicles when they cross a region boundary. Can you point to a specific place where this is still a problem? I have done a lot of testing, and from what I can tell that problem is fixed. It has been replaced with a new problem: Crossing into or out of a private parcel while crossing a sim boundry. I was so sure the full parcel problem was fixed that I removed that check from the Ban Line HUD, and replaced it with private parcel detection.

I think you are fighting a SL problem. I ran into the exact same problem yesterday. Fortunately the customer was in my support group and I was able to pull her up and send her an IM from there. But I told her about being missing from search and she knew nothing about it. She had not changed her privacy settings in any way, so I am pretty sure LL has dropped a bunch of residents from search. You probably will want to file a JIRA. I don't spend enough time in SL any more to justify filing one myself.

Very good! I can easily park the bot in a non-Magnum region when it is idle, so I'm ok. I am more worried about the users that don't know about this. There is no outward sign of a problem, so anyone on a metered internet connection is in for a (quite large) shock when they get their next bill. Thanks to you and Latif for looking into it.

Maestro and Andrew, Unfortunately today's roll did not fix the bot excessive bandwidth/resend problem. It still shows the same pattern as before: downloaded data reaches 100 kB/sec within 2 to 3 minutes of landing in a Magnum sim. Notice that is 100 kilobytes/sec, not 100 kilobits/sec. The data transferred reaches over 10GB daily. Pretty major if you ask me. This chart shows 6 minutes of time, with download data on the left and upload data on the right. Time is shown along the bottom. The first minute or so shows the bot's normal usage while sitting in a main server channel region. The first spike you see is when I TP'ed the bot into a Magnum region. The interesting part to me is that the upload data (maybe some type of request?) continues to rise for a full minute even after the download data (responses?) saturates at 100kB/sec. The upload data eventually levels off too, at around 8 to 10 kB/sec. Second Life 3.3.4 (264214) Aug 31 2012 06:30:36 (Second Life Release) Release Notes You are at 253,191.0, 256,260.0, 25.2 in Bay City - Falconmoon located at sim8018.agni.lindenlab.com (216.82.37.85:12035) Second Life RC Magnum 13.01.28.269574 Error fetching server release notes URL. CPU: Intel® Core i7 CPU 950 @ 3.07GHz (3064.54 MHz) Memory: 12287 MB OS Version: Microsoft Windows 7 64-bit Service Pack 1 (Build 7601) Graphics Card Vendor: NVIDIA Corporation Graphics Card: GeForce GTX 260/PCIe/SSE2 Windows Graphics Driver Version: 9.18.0013.1090 OpenGL Version: 3.3.0 libcurl Version: libcurl/7.21.1 OpenSSL/0.9.8q zlib/1.2.5 c-ares/1.7.1 J2C Decoder Version: KDU v6.4.1 Audio Driver Version: FMOD version 3.750000 Qt Webkit Version: 4.7.1 (version number hard-coded) Voice Server Version: Not Connected Built with MSVC version 1600 Packets Lost: 86/269,306 (0.0%)

Whirly Fizzle wrote: For those of you having this problem and have a laptop, if you take your laptop round to a RL friends and test on their connection, can you still reproduce it? And if I can add to that request: If anyone affected who has bandwidth monitoring software could go to a Magnum region, with no other avatars and draw distance set to minimum, and compare the background bandwidth being used (after a couple minutes settling time) to that of a non-Magnum region under the same conditions?

Andrew, here is my take on it. Please take it with a grain of salt. What I have read here seems to indicate that all the resends being sent to bots are not limited to bots, it seems some normal residents are affected also. My theory is that the residents having all the problems with duplicated chat, moving, packet loss, etc. are affected by this resend bug. So, in effect their bandwidth is being clogged with all the resends. If I am correct, then once the "bot resend bug" is isolated and fixed then these other network related problems will go away also. This should be easy enough to check. All we need is one of the affected residents to monitor their bandwidth usage after TP'ing into a quiet Magnum sim. If my theory is correct, then they will start seeing heavy network usage within 2 minutes of landing.

Could some of the issues that appear to be network issues be caused by the bot bandwidth problem? I know that on my home network I was going crazy trying to figure out what my strange problems were until I discovered all the bandwidth that was being hogged by the bot I host. Seems to me if there were enough bots in the same sim that they could be using so much of the server resources that it could explain some of these other issues.

Sounds good Andrew. Thanks for your attention to this matter.

Andrew and Maestro, I just realized I gave some incorrect information. This problem is occurring with the LibOpenMetaverse bot, not a JVA bot. I used to use JVA but switched. There is another thread concerning a problem with JVA bots. This bot is open source, so if there if is a good search string or another way to tell if valid AgentUpdate UDP messages are being sent let me know.

Sorry Andrew, I don't have any way to tell that that I know of. The bot's console screen remains silent, and the bot even continues to function normally as far as I can tell. If you can set up a test region somewhere (on agni) it would be easy enough for me to TP the bot there and watch the traffic meter. ETA: Does the 100kB/sec cap the data appears to reach after 2 minutes give a clue? Is anything capped at that rate?