Pathfinding Q&A

[Innula Zenovka]

I know there's a big problem with traffic being misreported (and I also know for a fact that several of the sims suffering from it have pathfinding turned off), but I just don't think it's helpful, absent any better evidence than the fact the problem manifested itself around the same time as the main pathfinding roll-out, to suggest any particular relationship between pathfinding and the traffic problem.   

What I'd be interested in knowing is what the sims suffering from it have in common that they don't have in common with sims that aren't affected.    I noticed that a lot of the affected sims are Adult ones,  and wondered about that, but it doesn't look as if it's confined to adult sims.   Anyway, as we know, there have been problems for ages with sims vanishing from search and traffic being misreported (several sims in the DG were reporting their population as 0 on the log-in page for several months, after all).  

I guess what I'm saying is that barely a month goes by without something going badly wrong with search and traffic, so I don't think too much can be read into what else happened around the same time in any particular month something goes wrong.

As to the bandwidth, that's what I thought you meant, but I don't really see what data the sim is going to be sending to my computer about pathfinding, other that objects are moving about.   But that's no different from it having to tell me avatars or vehicles are moving around, so it shouldn't really have much effect (and I don't think I've seen a pathfinding object in world yet, apart from in sandboxes).    I can understand concerns -- indeed, I shared them for a while -- about the sim having to devote so much time to pathfinding-related tasks that other stuff gets delayed (though apparently it's set up so it's just pathfinding-related stuff that suffers) but I just don't see what extra data the sim has to push to my PC.   

 

ETA -- about the traffic issue.   It's just been pointed out on the jira that this sounds very much indeed like a repeat of SVC-7459  from last November.

[Talia Davidov]

---

Innula Zenovka wrote:

ETA -- about the traffic issue.   It's just been pointed out on the jira that this sounds very much indeed like a repeat of

SVC-7459

  from last November.

---

I would have to agree it mimics last years bug number almost exactly on my sim.

[Perrie Juran]

---

Innula Zenovka wrote:

According to

Lorca Linden's reply to Nalates

 ,Perrie,

---

We do not consider pathfinding to be fully released until the pathfinding viewer tools are out of beta. This is why we have not yet made an official announcement. I agree that we need to do a pass on our pathfinding related wiki as some of the information there has not been updated since we were in alpha. We plan to make a blog post in the near future that will address some common misconceptions we have heard about pathfinding. We also plan to continue updating the “Good Building Practices” guide so that it will be a useful resource for Residents looking to make optimized content.

---

I don't particularly agree with the reasoning, but there will be an announcement shortly, it seems, once they're satisfied the viewer tools are working OK and they've given the wiki articles a much-needed update.

What is the massive impact it's supposed to have?  Everything I've read recently suggests the technical Lindens involved are saying that, under normal circumstances, the only things that are going to take a hit from pathfinding performance issues are pathfinding objects, but I've probably missed something.

---

I don't know enough actually / haven't taken the time to sort out what code changes in all the recent Server upgrades are specific to Pathfinding or are actually related to other things.  So I may be lumping some things together.  I am assuming for instance, that the new Havok engine was necessary for Pathfinding to work.  So all one has to do is look at the impact on Vehicles and yes, it's a big change.

A couple of weeks ago after a server upgrade my Rez times went through the roof.  Four minutes on a busy SIM that used to rez in under 30 seconds.  On an educated whim I turned off HTTP textures (which supposedly are supposed to work better) and my Rez time sped up back to what it used to be.  Why this is so is technically 100% outside my personal technical understanding of how things work.  All  I know are the results.

It may be a mistake to lump other current problems I and/or Friends have started having under the umbrella of Pathfinding.  I know other fixes, etc are also being implemented.  But Pathfinding and the new User Experience tools are the Biggies going on right now.

Trying to decipher some of this / what is the actual cause can be a hair puller.  Like why three days in a row now I have completely lost No Copy objects that I rezzed.  For all due and intent purposes they are gone for good. 

 

[Darien Caldwell]

I've been keeping a close eye on my LeTigre sim since Pathfinding came to it. And Whenever there's 3-4 avatars in the sim, I'm seeing *huge* spikes in Agent Time. By Huge I mean spikes into the 200-400 ms range (and recall, frame time is only 23 ms max).

Of course this completely freezes the sim for a few seconds. I really can't figure out what causes it.  Also being LeTigre, there's some mysterious project running on it that LL won't discuss, so I don't know if that's the cause or if it's the Havok/Pathfinding.  I just know it never happened until after this wednesday's roll.

[Innula Zenovka]

The http textures issue has been going on for ages, and I think it's more to do with the viewer code than the sim.

As to vehicles, one issue I know some people have had is that when stuff that was using llVolumeDetect automatically converts to physics shape none (llVolumeDetect prims) and convex hull (everything else), this obviously puts you into the mesh accounting system, and it doesn't necessarily do it the most economical way, in that (for example) tortured prims -- especially toruses -- can artificially inflate the LI unless at least some of them converted to physics shape none, too (the number of prims with a high rendering cost makes a difference).   But the automatic conversion process can't know what to convert and what not.

That might be something to consider when adjusting vehicles, if the vehicles are mod, that is.

 

[Perrie Juran]

---

Innula Zenovka wrote:

The http textures issue has been going on for ages, and I think it's more to do with the viewer code than the sim.

 

---

 

It may be more related to the Viewer, but the thing is I changed NOTHING on my computer.  I did not install, upgrade or downgrade anything except for AntiVirus updates.  Maybe my ISP changed something in their backbone, but that I have to my knowledge no way of knowing.  So at least from an experiential point of view the least common denominator is the Server changes.

 

---

Innula Zenovka wrote:

 

As to vehicles.......

That might be something to consider when adjusting vehicles, if the vehicles are mod, that is.

 

---

 

As a "consumer" I am already dealing with this.  I haven't been through all my vehicles yet, but my favorite ones are No Mod.  Which puts me currently at the mercy of the good will of the Merchant.

 

I know you are a Merchant and maybe in the long run this is going to make you  more money.  Absolutely nothing wrong with that.

But in the short term, how much time, effort and money are all these changes costing you?  Do you really know yet the effect all these changes are having on all your products?  And the effects they are having on your customers?

Please don't get me wrong here.  I am all for upgrades and improvements.  And with those upgrades there may be some growing pains.  I understand this.  But in following all these issues it appears that  LL either willfully chose to ignore them or chose to push through with this any ways. 

I do love Second Life, but wow, sometimes what we deal with.

 

 

[Innula Zenovka]

These changes haven't really had much of an effect on me, because I've had several months to prepare for them.   I've never relied on the llVolumeDetect hack (which is what it is) too much, having been caught out by a very unexpected change in its functionality a couple of years or so ago, and because I knew this was coming I've been keeping an eye on things.   My business partner and I don't normally sell things no-mod anyway (unless modifying them would actually break them) , and one of the main reasons for this is that we know LL tend to change things now and again.  And we do have an update system, and have done for a long time.   It's not expensive to buy one (not difficult to script one, either), and we very soon realised any serious content creator needs one.

LL, in point of fact, are very good about not breaking legacy content.  They're too good about it, at times -- there's some changes to scripting that everyone wants to see implemented, just about, but can't be because it would break some older stuff.   And when they do break things -- posJump, for example -- they do give us substitutes.   

So it's not really hurt me that much.  I don't make vehicles, so I'm not at the sharp end, but I do know that many vehicle makers have been involved in testing and debugging stuff related to pathfinding and that LL have been very good about tryiing to meet their concerns.

As a content creator and merchant, I really wish that the people running the marketplace and search were half as responsive and approachable as the technical Lindens like Oskar, Falcon, Maestro, Kelly and Lorca.    It's search and the MP that have me tearing my hair out, not the technical people (who are used to dealing with problems that can't be solved by blarney and soft-soap).

[Falcon Linden]

---

BlackMagi Darkwatch wrote:

Why do I now need to unlink buildings (remove the floor prim) to make it walkable?  This is especially pertinant as mesh builds can't simply be unlinked.  The only option (if I've read correctly) is to add another prim to act as the physics shape?! Or re-upload the mesh... 

---

You don't need to unlink and relink. Honestly, I'm not sure where that idea came from but I know it made its way into the documentation somewhere (which should be fixed). Here's the deal:

In a perfect world, walkability would be a per prim property not a per linkset property. For technical reasons, this wasn't achievable (and how would you have liked to have to set properties not just on all your linksets but on individual prims?!). So you have a couple options if you'd like to make a complex linkset like a house walkable (ignoring issues surrounding special prims like doors):

1) Make the whole house walkable. This is a perfectly good first pass solution. The only negative effect is that rebaking the navmesh will take slightly longer (I very much doubt it will ever be noticeable).

2) Unlink the walkable surfaces, link them together as a second linkset, and make just that walkable (and the rest Static Obstacle). This is understandably annoying, sometimes impossible, and generally unnecessary.

3) As a step better than (1), make whole house walkable and then drop an exclusion volume in that encompasses only the walls/roof. This will produce a result very similar to (2) (though not identical).

Falcon

[Falcon Linden]

---

Ayesha Askham wrote:

Innula

I'm going to take the second point first:

"it can't be to do with pathfinding?" Well, perhaps to a degree it isn't, but the basic error in traffic calculation was certainly introduced to server code around the time Pathfinding infected the Grid, and it certainly first surfaced as an issue on Magnum RC.  Guilt by association?

There is another aspect though, and this is more insidious: I suspect that as Pathfinding and Havok 7 was rolled to the Grid some aspects of sim operation were changed, maybe "reverted" in Lindenspeak.  It now seems that bots are being counted as genuine avatar visitors again.  That is definitely an aspect that is NOT "working as expected"...unless...I recall that back when Rod announced the advent of Pathfinding that "empty spaces" are not inviting.  So maybe there is an algorithm that is intended to count NPCs as vistors to give ranking in search to Pathfinding sims and this code also allows bots to be counted in seach ranking?  The sounds a tad Macchiavellian, but it's the best explanation I have for what is a very real issue, yet we are told all is "working as expected"

As regards the bandwidth issue I am a bit out of my depth here.  However this may be a consequence of sims that have little or no "spare time".  The connection has held on all the sims that I have seen this issue.  I had noticed in times past that these sims had very little on no spare time, so Havok 7 physics may well be causing this. 

Note I am not blaming Linden Lab for this, it is a consequence of the change that we were warned might occur.  I am old enough to recall the manifold issues that accompanied the introduction of Havok 4 to the Grid some years ago.

---

We are investigating the dwell time issue but I can say with confidence that it was not directly related to pathfinding (although it may have been introduced as a merge error/regression). We definitely are not trying to include NPCs in these calculations! The pathfinding team did not intentionally touch any code that relates to dwell time calculations.

By the way, pathfinding actually uses Havok 2012.1. The previous servers used Havok 2010.2. Havok 7 has not been on the grid in over a year (possibly more).

 

Falcon

[Falcon Linden]

---

BlackMagi Darkwatch wrote:

So, I created a simple character using the new LL pathfinding functions and watched it merrily wander around - I won't be leaving the object out though as my simple 1 prim cube had a LE of 15!  Not sure if the LE values are going to tweaked - I do hope so as I feel that is way to costly.

 

---

All pathfinding characters have a fixed physics weight of 15. Adding prims will not cause this weight to go up and the weight is not affected by the physics shape type parameter. Streaming and script costs are computed are for any other object and the total land impact remains the max of all weights. So yes, a simple 1 prim cube has a LI of 15, but a much more complex character may still only have a LI of 15 (unless dominated by streaming or scripts, in which case the physics weight is irrelevant).

Falcon

[Falcon Linden]

---

Innula Zenovka wrote:

As to vehicles, one issue I know some people have had is that when stuff that was using llVolumeDetect automatically converts to physics shape none (llVolumeDetect prims) and convex hull (everything else), this obviously puts you into the mesh accounting system, and it doesn't necessarily do it the most economical way, in that (for example) tortured prims -- especially toruses -- can artificially inflate the LI unless at least some of them converted to physics shape none, too (the number of prims with a high rendering cost makes a difference).   But the automatic conversion process can't know what to convert and what not.

 

---

Incorrect. The sim will never automatically opt you into the new accounting system. We only change the child to shape none if it was already using mesh accounting. Otherwise, we change the prim to behave like a normal phantom object. You can see this by noticing that in a non-mesh-accounting linkset with llVolumeDetect hacks those hacked prims now collide with the terrain (like a phantom object) whereas they did not before (and do not if using new accounting a shape type none). This is a minor problem and we felt it was the best way to handle the issue without introducing significant additional complexity.

 

Falcon

[Innula Zenovka]

Clearly I'd misunderstood, then.  Thanks for clearing it up.   I'd been mislead by  the complaints from some people about problems they were having with their vehicles, but whatever the cause is, that can't be it.

 

[Nalates Urriah]

The post on the Phoenix/Firestorm blog seems to have done more harm than good.

Once all the regions were up on Pathfinding the performance was checked and watched. By Thursday's PF user group meeting enough stats had been collected for the Lindens to announce there had been no performance degradation. The regions are perfroming as well or even a tiny bit better than before pathfinding was rolled out.

Whether one optimizes for Pathfinding or not, the region should preform well. What many have missed is that the optimization is for the benefit of Pathfinding, not general performance. So, far that seems to be proving out. Look at you Viewer Stats to see how the regions are performaing.

------

Second Life is a huge system. No one completely understands it. Just because a developer is making viewers, doesn't mean they know what is happening with the servers or any specific feature. Oz Linden has on more than one occassion answered questions with an, "I don't know. Not my area." I have also see Lindens in one meeting give and answer about something that according to other Lindens was wrong. They don't know it all and they make mistakes every now and then.

When I asked Andrew Linden how big the asset database was he made an educated guess, but he was way off. He too was curious and checked with the people working to clean up the database. Pre-cleanup the db had grown to 1.2 petabytes. Post cleanup the db was 192 terabytes.

So, it is not surprising to see a third party developer miss the mark and not understand what the Lindens were saying. While many blame the Lindens for not communicating better, the real problem is simply ambigious human communcations and the loss of frame. In the US during this election season you can watch politicians misspeak and people take what they said in ways never intended. It happens to all of us.

[BlackMagi Darkwatch]

It would appear (to me at least) that suggestion 3 is the only viable option as most of my scripts rely on getting the prim name / description within the linkset - this was done to keep the script count down.

So now a question about the exclusion volume (EV).  Do they need to be bigger than the original prims in the build or can they be the same size?  Do av's interact with the EV in other words making the EV 100% transparent, will av's bump into it or by setting the pathfinding properties to EV will they then become phantom and then there will be no interaction with av's or does this need to be done manually?

Black

[Darrius Gothly]

---

Falcon Linden wrote:

We are investigating the dwell time issue ... [snip]

---

Falcon, this is the first in a long time that we've "heard" the Parcel Traffic/Dwell Time issue being addressed by ANY Linden. Would you please drop a sticky note on your co-workers that the JIRA covering this issue is in DIRE NEED of some Linden involvement. Silence .. especially in this case .. is decidely NOT the best course of action.

[Nalates Urriah]

I was at that meeting. Vladi was not called a liar. That is his inference. 

If you want more information on the meeting see: Pathfinding Meeting Week 32.

The Lindens refused to use up time in the meeting discussing problems they can't reproduce and the speaker couldn't reproduce. Another person did describe a problem well and got the Lindens attention. So, they did listen. It just depended on the quality of what one was saying. 

Vladi is apparently frustrated and I guess from his wording doesn't understand some of the information provided. That seems to have strongly colored his statements about the Lindens. That is understandable, but it lacks civility.

On some points he is very much entitled to his frustration. Where does one discuss problems with the viewer or a specific LSL function? The Lindens tend not to answer those types of questions because they don't know. There is no place to discuss viewer problems. Oz Linden's UG is as close as it gets to being viewer related, but that is for open source issues and Oz allows little else to enter the discussion.

Scripting issues used to be discussed at Kelly Linden's UG. That closed from lack of interest. Though Kelly does attend the Server/Scripting UG held by Andrew and Simon Linden.

Software development is an at times frustrating task. Falcon was trying to understand why some problems that would have been apparent in Beta testing never showed up until after the main roll out. Residents are trying to figure out how the problems made it to roll out also. I think the answer is simple enough. Human advancement is via a series of mistakes, catastrophes, and disasters not from well thought out plans. Read up on the discovery of metal fatigue in aircraft in the 1930's. People died while that was being figured out. Simply put in this case the problems did not occure to the Lindens so those tests were never added to the QA testing. The residents working in with PF in beta never needed to perform those tasks. This is all just human nature. Some get that and some don't.

[Ayesha Askham]

Nalates

Your post brings some points into focus for me, and once again thanks for providing a digest of UG meetings that even a clod like me can understand!

It seems clear from your post and the blog article you refer us to that there is a serious dislocation between SecondLife as perceived by a Linden and SecondLife as perceived by a resident.  That is a matter that needs urgent addressing by Linden Lab, or it will result in more "inconveniences" occurring.  What these Lindens, and I think, you also, do not see is the impact that such issues have on the viability of SecondLife as a competitive and successful virtual world in 2013 onwards.  To those who would say simply that this represents SNAFU, I would simply say that while I tend to agree, I can hope.

That such glaring oversights occur may well be "human nature", but they can and must be overcome.  That the Challenger loss could occur probably renders my hopes unlikely, and SecondLife would be another casualty.  It would be a shame and a great loss to those of us for whom SecondLife is not simply a diverting game, but a way of regaining some of the "life" that, for various reasons of infirmity, we have lost.

Despite all the bluster and paranoia, it remains clear, to me at least, that Linden Lab did not excercise sufficient care in testing their new code, and opinions will remain divided as to the wisdom or otherwise in their choices of Magnum RC test sims.  The exit to a busy Yacht Club sim does not strike me as being a wise choice for the placing of a test sim under any circumstances, and to say that Lindens could not have known this is pure rubbish.

There is a lot of mess to clear up and it is very refreshing to see a Linden such as Falcon posting to a thread such as this on a weekend.  Now, if a similar diligence could be applied to several recently generated JIRAs a lot of us would be very grateful!

[Perrie Juran]

---

Ayesha Askham wrote:

It seems clear from your post and the blog article you refer us to that there is a serious dislocation between SecondLife as perceived by a Linden and SecondLife as perceived by a resident. 

 

---

This is such a huge issue.

All I really know is that on three consecutive days after the roll outs, etc, I have lost four no copy objects after rezzing them.  And the only common denominator that I know was the roll outs.

 

First time I had found myself logged out of SL with NO warning messages about five minutes after rezzing the object.  The Viewer never crashed.  I was simply no longer connected to the Grid.

The next day, about five minutes after rezzing two no copy objects  I got the "You have been logged out ....the SIM you are on may be experiencing troubles, etc." message.

I logged right back in and the two objects were gone.  Forever. 

The third day I had rezzed the replacement of one (my thanks to the Merchant).  An hour later I logged out.  On logging back in, once again the object was gone.

This is all on a Blue Steel region.

How am I supposed to decipher this?  I only know this is the first time I have ever lost inventory like this in over five years here and I only know one common denominator.

Maybe my experience is a fluke, but regardless, I am not a happy camper about this.

In practice, It seems the Lab continually misses the boat on what is important to the Residents.  Does it not occur to them that peripherally speaking, that these issues also are impacting the New Users that they seem to have so much trouble retaining.  That maybe there is a cause and effect relationship that they are missing.

I know that this is off topic to this sub fora but where else can I be saying these things.

What would concurrency be today if the Lab had not lost so many Residents who left out of sheer frustration?

 

 

[Innula Zenovka]

Have you raised a ticket to find out why the sim kept on crashing, Perrie?   The last time this happened to me, it turned out to be because someone with a sim-crasher had a grievance against one of my tenants.

[Perrie Juran]

---

Innula Zenovka wrote:

Have you raised a ticket to find out why the sim kept on crashing, Perrie?   The last time this happened to me, it turned out to be because someone with a sim-crasher had a grievance against one of my tenants.

---

I am not Premium and I don't see that option available to me in the Support Portal.  If it happens again I will ask the SIM owner to file it.  There are only 5 of us including the owner on the SIM.

As I stated, the third time it happened, I logged out.  Whether the SIM crashed again while I was out I don't know.  I have a vague idea why no copy objects get lost in a restart or crash...kind of an assumption on my part.  That is, that somehow the 'save' to the server was not completed before the restart occurred.

Maybe the solution to losing no copy items is a massive undertaking.  It would sure be wonderful if it could be addressed. 

In the meanwhile I grabbed some no copy freebies and am putting out periodically to see if the problem has abated.  What other test could I possibly run?  I have some irreplaceable objects (Merchant no longer in SL) to put out and some expensive objects.  My Intan alone, with everything in it must be worth five or six thousand $L.

Thanks for the suggestion.

Perrie

[Innula Zenovka]

No copy items get lost if they were rezzed between the last time the sim took a backup snapshot of what's rezzed there and the sim restarting.   It's not supposed to happen any more if there's a scheduled restart, though in my experience it does sometimes, but I'd think that if the sim actually crashes, all bets are off.   So if the no-copy item was rezzed in the last five minutes or so before the sim goes offline unexpectedly, it's a risk of not being there when the sim comes back, because the sim doesn't know to replace it.

 

[Falcon Linden]

---

BlackMagi Darkwatch wrote:

So now a question about the exclusion volume (EV).  Do they need to be bigger than the original prims in the build or can they be the same size?  Do av's interact with the EV in other words making the EV 100% transparent, will av's bump into it or by setting the pathfinding properties to EV will they then become phantom and then there will be no interaction with av's or does this need to be done manually?

Black

---

An EV is phantom and therefore won't interact with avatars. The EV must encompass all areas of the navmesh that you wish to be removed (have a look at the View/Test... preview to see the navmesh, but remember that if an EV was already placed when the navmesh was last baked, no navmesh will appear inside the EV). So the basic workflow would be:

 

1) Make appropriate objects into walkables and, where possible, static obstacles

2) View the navmesh

3) Add EVs that encompass any unnecessary areas of the navmesh or any areas that look "weird" or generate bad paths (test that with the View/Test... floater)

4) Rebake the navmesh and preview again to verify your changes

 

Falcon

[BlackMagi Darkwatch]

Ok, that sounds reasonable - I will have a look at that tomorrow

Thank you Falcon

Black

[Perrie Juran]

---

Innula Zenovka wrote:

No copy items get lost if they were rezzed between the last time the sim took a backup snapshot of what's rezzed there and the sim restarting.   It's not supposed to happen any more if there's a scheduled restart, though in my experience it does sometimes, but I'd think that if the sim actually crashes, all bets are off.   So if the no-copy item was rezzed in the last five minutes or so before the sim goes offline unexpectedly, it's a risk of not being there when the sim comes back, because the sim doesn't know to replace it.

 

---

Let me see if I can phrase this question clearly and make sure my understanding is clear.

I know before a scheduled restart a snapshot of the SIM is taken, what you have referred to as a back up.  I might be slightly confused by the term "back up" because generally speaking I think of a "back up" as a duplicate copy of a file.  Regardless of that, the Server uses that snapshot when the SIM restarts.

Now I am guessing here that periodically in the normal course of events snapshots are being taken of the SIM for back up purposes?  And if the SIM crashes, the last known back up is used to restore it?  So how often are these back ups taken?

How long actually then does a no copy object hang in Limbo before it is actually safe?  An hour?  A day?  A week?  My last lost object was out for at least an hour before it 'disappeared.'

 

 

[Innula Zenovka]

I honestly don't know.   According to this transcript from 2007, back then the sim took a snapshot once an hour, but that was five years ago.   Things might have changed since.   

Certainly this process doesn't seem completely reliable, though.  I remember reading some months ago that the system had been changed so that, during normal restarts, the sim takes a final snapshot after all avatars have been booted from the sim, which should mean everything's safe, but, nevertheless, since then I've certainly had to help tenants who've lost no copy items that they'd rezzed in the ten or fifteen minutes before the restart.

And Intans in particular (I think, in fact, dance balls in general, if they contain no-copy animations) are notorious for disappearing for no apparent reason.   Mine one day just vanished for good, having been sitting in the same place for months.   Why that should be, I have no idea.   All the animations it contained had been bought direct from the creators' shops rather than on the black, or even grey, market.