Gradual viewer slowdown to a crawl when testing KFM

[animats]

I've been working on my own pathfinding system, and it's coming along nicely. So I wanted to see how a crowd worked. I set up a test situation with 20 cubes under scripted pathfinding control, in Sandbox Bricker, which was otherwise empty. Everything is running fine. The cubes are maneuvering around each other like they're supposed to, and the sim load is about 2ms/frame, which is not bad compared to regular pathfinding.

1 minute after login.

But then things start to slow down. The viewer gets slower, and slower.

 

10 minutes after the first screenshot. Ping time 21 seconds!

After about 15 minutes, I get logged out. If I log back in, everything is fine, and the cubes are moving around as before.

I've had this happen several times now, with both the LL viewer and Firestorm, and on both Windows  7 and Ubuntu 16.04 LTS.

Notes:

• The graphics load is trivial. There are 10 cubes and bare ground on screen.
• Network load is very low. That "ping time" thing is the viewer falling behind, not real network traffic. A ping from a ping program is consistently around 23ms. I have 50mb/sec bandwidth available.
• No significant disk traffic.
• Firestorm is compute-bound when this is happening.
• These cubes are just keyframe-animated, as far as the viewer is concerned. My pathfinding scripts sim side are generating the keyframe list, but the viewer doesn't see that.
• Script time hovers around 2-3ms/frame. Lots of spare time; this is an empty sandbox.
• There's a lot of printing from llOwnerSay. I pushed the chat window off  screen here. Can that all by itself jam things up, as the window scrolls to keep up?
• I have a Firestorm log of this, but since it happens with the Linden viewer too, it's not Firestorm's fault.

This is strange. I was worried about overloading the sim. That's not a problem. How did I overload the viewer with 20 moving cubes?

 

 

Edited August 22, 2019 by animats

[animats]

I see something suspicious. Under Viewer/Memory usage/UI, that number starts out around 3000KB and continually increases to 36141 KB. That doesn't happen elsewhere in SL. There's lots of debug print here, via llOwnerSay. You can scroll all the way back in the chat window and look at it, so it's using viewer resources. llSay has some throttling, but you can do all the llOwnerSay you want, I think.  You can only impact your own viewer.

I wonder if adding text in the chat window gets slower as more text is added. More testing needed. Any known problems in that area?

Edited August 22, 2019 by animats

[Qie Niangao]

One possible test: try an alt looking at the same scene. They'd get all the object updates (which KFM generates in vast quantity) but none of the llOwnerSay traffic.

[Alyona Su]

3 hours ago, Qie Niangao said:

One possible test: try an alt looking at the same scene. They'd get all the object updates (which KFM generates in vast quantity) but none of the llOwnerSay traffic.

I would think this to be a good test. You know how group chat can start lagging like crazy when there are a lot of people? Or even open chat when there are a lot of people? I've always thought it had to do with a lot of people, maybe it's the chat system itself? Full disclosure: I am a GENIUS at knowing just barely enough to be dangerous.

[Love Zhaoying]

1 hour ago, Alyona Su said:

You know how group chat can start lagging

And when chat starts to come out of order!

[Alyona Su]

15 minutes ago, Love Zhaoying said:

And when chat starts to come out of order!

Yes, I often post "OH FUN! Write today, read tomorrow!"

[animats]

It's not KFM or scripting. It's llOwnerSay. All by themselves, enough llOwnerSay calls can crash the viewer.

I wrote a trivial script that makes thousands of llOwnerSay calls. Rez that object, and the viewer slows down to a crawl within minutes, and then "You have been disconnected from Second Life". Doesn't seem to hurt the sim, just the owner's viewer.

Test in empty Sandbox Goyer. This is Firestorm on Linux, but I've had similar problems with the LL viewer on Windows.

Sim ping over 5 seconds, frame rate 4.6/sec. Forced logout shortly thereafter.

UI memory is not becoming large, unlike when I tested with my keyframe animated objects.

At first, the viewer can keep up, but over a few minutes, it falls behind and gets slower and slower.

llSay, unlike llOwnerSay, has a throttle. ("Messages sent on channel zero and DEBUG_CHANNEL are throttled to a rate of <200/10sec, per region, per owner/user.") So you can't grief others this way.

This is a relief. My pathfinding system isn't breaking anything. It's just the debug output choking the viewer.

Incidentally, notice that the messages, which are numbered, are out of order. Way out of order, like 3000 places out of order.

(Technically, this looks like an O(n^2) bug. Adding to the queue runs slower as the queue grows, so processing time increases with the square of the message traffic. I had to avoid this once in the logging for a hard real time robotics application. I had the log message queue set up so that when it hit a limit, the output showed "..." and the queue was cleared. That way, the real-time part creating the messages never had to wait.)     

Edited August 22, 2019 by animats

[Lucia Nightfire]

Ownersay spam is a known spam vector. There have been "combat HUD's" in the past that offer obfuscated titled objects to targets to get them to rez/wear that would crash them with ownersay spam.

[animats]

5 hours ago, Lucia Nightfire said:

Ownersay spam is a known spam vector. There have been "combat HUD's" in the past that offer obfuscated titled objects to targets to get them to rez/wear that would crash them with ownersay spam.

Ah. No surprise. Anyway, llOwnerSay was definitely the problem.

Turned off most llOwnerSay debug output. Working now. 20 objects in the sandbox, all trying to go back and forth between the same two points while avoiding collisions. Viewer steady at 30 FPS, the max I have set. Script load between 1 and 3 ms/frame, less than regular SL pathfinding. And it drops when objects are not moving. Huge amounts of KFM and llCastRay don't seem to hurt anything. I didn't expect they would; Janet's Viking sim, Folkvang, must have several hundred KFM tasks going.