Jump to content

Is LL putting more sims on fewer servers?


animats
 Share

You are about to reply to a thread that has been inactive for 1756 days.

Please take a moment to consider if this thread is worth bumping.

Recommended Posts

 

55 minutes ago, Aishagain said:

No one has accused LL of being underhand, just a bit less than transparent about a change.

That's literally the subject of this thread.

55 minutes ago, Aishagain said:

My guess that someone WILL produce (or maybe has indeed already produced) such a script and if my experience is valid LL will simply say that it is not measuring parameters correctly.

No. They have been too busy posting anecdotal information here... and that is why an open script would be needed.

55 minutes ago, Aishagain said:

As to the comment that our comments are "unsubstantiated moaning and speculation" I suggest you re read some of the previous posts in this thread.

Something HAS changed and really all most of us are asking is WHAT?  @Sharie Criss has a very valid case to be answered.

*facepalms* If you can't put your issue into a specific context that a developer can investigate, then you are just citing unconnected examples of weirdness and lumping them all together as having the same cause. Correlation is not causation. If you suspect something is broken or wrong, write a specific test for it. Gather actual data.

54 minutes ago, Profaitchikenz Haiku said:

(replying to Coffee Dejour's post)

Or, judge from something in the parcel that has observable behaviour.

Not good enough for a developer to reproduce and investigate.

Keep in mind that your issue will be investigated by a Linden staff member who probably never uses SL in a recreational manner, or spends all their days looking at the stats floater to make sure things are "right".

I did X.

I got Y, but I expected Z.

 for this ..

"I wrote a benchmark that counts the number of floating point math operations per second. I ran this benchmark on this region X times and got these results. I ran them again a week later and got these lower results. Here is the script I used. Here are the locations I tested."

That gives a developer a solid case that can be investigated.

54 minutes ago, Profaitchikenz Haiku said:

I have a scripted steam train that is puppeteered to get the wheels, connecting rods and synhcronised steam puffs all working.

 

That's nice. What do you propose, should a Linden come hang out and watch a black box object going through the motions for a few days?

That's not how Linden development works. That's not how any development works.

Make a test case. Test it. Give it them as part of a JIRA.

  • Like 2
Link to comment
Share on other sites

I did try to get some answers relevant to this topic at the Server User Group yesterday. The chat log is long and more comprehensively summarized in Inara Pey's blog post, but here's some specifically relevant text extracted from the session:

Quote

[2019/04/30 12:24]  Qie Niangao: That reminds me, this keeps coming up: There hasn't been a change in how sims are distributed (yet), right? Still one-per-core for full sims, 4 per core for homesteads?
[... other parallel conversations elided ...]
[2019/04/30 12:25]  Simon Linden: There hasn't been a change in that, Qie
[2019/04/30 12:26]  Qie Niangao: Thanks Simon

[... much later...]
[2019/04/30 12:59]  Qie Niangao: I wonder: Do sims ever run out of memory for running scripts? ss they did, back in the early days of Mono, causing the sims to swap to disk, etc., maybe(?) affecting other sims on the same server? (and, if scripts are blocked by an unavailable resource, like memory, do they count Script Time as "Sleep Time" in the Details?)
[...]
[2019/04/30 13:00]  Simon Linden: Qie - that's pretty rare these days, and usually due to some griefing or other unusual overload
 

I now suspect that the case from a few weeks ago of the Vallone sim having Sleep Time counted as Script Time is rare and hence maybe not as useful as we thought in diagnosing the broader problem of increasing script lag. For one thing, I think the London City sims were actually victims of more typical lag (the one with massive sleep time, London City West, is a Homestead, so should always have a minimum of 16ms sleep time). To be clear, I'm not doubting that London City sims, full and homestead, are suffering from increased script lag; I'm just not seeing a Sleep Time anomaly in the statistics posted to the jira. In fact, it appears to me that the full sim is busy doing other, non-script stuff, thus lagging scripts -- and if that's also happening in other script-lagged sims the focus should shift to what's causing non-script times to rise.

It's clear to me that the "more sims per server" theory is wrong. Sure, if there were more sims per core we might see what we're seeing, but many (unfortunately too many) other explanations could account for the problems, so there's no reason to doubt Simon's claim that there's been no change there. I won't be wasting any more time on that, so anybody wanting to further pursue that theory are on their own.

 

  • Like 1
  • Thanks 3
Link to comment
Share on other sites

3 hours ago, CoffeeDujour said:

None of what you're saying is actionable. 

The only numbers being thrown about are from the statistics floater, which could be wrong. A bug in stat reporting is a much simpler possibility than underhanded LL are doing the sneaky on the sneak because evil.

Instead of pages and pages of unsubstantiated moaning and speculation, you could just write a an open benchmark that tests multiple aspects of the scripting system.

Today my script did X things in Y seconds, yesterday it was much faster.

Here is the script. Here is the output. Here is the SLM page so others can try it. Here is the JIRA page to report your experiences.

 

So are you suggesting that we travel back in time to 18 months ago when this started so we can get a pre-change benchmark documented? That's what it would take to get what you are looking for. Since that's impossible, all we have is what we can see NOW and how things appear different.

As I've stated in other posts, non-logged occasional benchmarking was performed using scripts that are available on the SL Wiki. Go look.

Your post is essentially saying "Prove it." That's just not helpful. Can you prove the opposite, that LL did nothing to cause this problem? Do you have test results and statistics to back it up? Absolutely all the anecdotal info says otherwise.

  • Like 1
  • Haha 1
Link to comment
Share on other sites

42 minutes ago, Sharie Criss said:

So are you suggesting that we travel back in time to 18 months ago when this started so we can get a pre-change benchmark documented? That's what it would take to get what you are looking for. Since that's impossible, all we have is what we can see NOW and how things appear different.

As I've stated in other posts, non-logged occasional benchmarking was performed using scripts that are available on the SL Wiki. Go look.

Your post is essentially saying "Prove it." That's just not helpful. Can you prove the opposite, that LL did nothing to cause this problem? Do you have test results and statistics to back it up? Absolutely all the anecdotal info says otherwise.

That's why I'm agnostic.
(sorry, I couldn't resist putting a bit of humor into this thread)

[EDIT] Now seriously: I don't care about what background monitoring says, as user I'm only interested in functionality — and that is severely affected no matter what "geeks" say or think. Also, it's none of my business the reason why it is happening — that's up to LL.

Edited by MBeatrix
Link to comment
Share on other sites

52 minutes ago, Sharie Criss said:

So are you suggesting that we travel back in time to 18 months ago when this started so we can get a pre-change benchmark documented? That's what it would take to get what you are looking for. Since that's impossible, all we have is what we can see NOW and how things appear different.

As I've stated in other posts, non-logged occasional benchmarking was performed using scripts that are available on the SL Wiki. Go look.

Your post is essentially saying "Prove it." That's just not helpful. Can you prove the opposite, that LL did nothing to cause this problem? Do you have test results and statistics to back it up? Absolutely all the anecdotal info says otherwise.

If you can't show it happening then you can't expect anyone to do anything about it.

  • Like 1
Link to comment
Share on other sites

19 minutes ago, CoffeeDujour said:

If you can't show it happening then you can't expect anyone to do anything about it.

Yes you can. I guess that's the difference between you and me. When I am paying for a service and the service goes to crap, I'm going to say something and expect at least SOME response. LL has chosen to not even acknowledge the issue that MOST of us are seeing. To me, that speaks volumes. People generally clam up when they have something to hide. It's human nature.

Me: There is a pothole in the road. It needs to be fixed.

You: Can you prove that there wasn't always a pothole in the road?

Me: No, sorry, I don't keep photographic evidence of ever section of every road. I just know that there is a pothole here now that wasn't here last month.

You: How can you expect anyone to believe that the pothole wasn't there by design if you can't prove that it wasn't always there?

 

Really - it's that ridiculous.

Edited by Sharie Criss
  • Like 1
  • Haha 1
Link to comment
Share on other sites

19 minutes ago, Sharie Criss said:

Yes you can. I guess that's the difference between you and me. When I am paying for a service and the service goes to crap, I'm going to say something and expect at least SOME response. LL has chosen to not even acknowledge the issue that MOST of us are seeing. To me, that speaks volumes. People generally clam up when they have something to hide. It's human nature.

I believe that LL has acknowledged the issue, but internally. They didn't set regions restarts on a 10 days cycle just for fun. But if this is to continue, at some point many mainland regions (not all) will need to be restarted daily. Most private regions already are, so what about those? Will some need to be restarted every couple hours? From the users perspective can we call that functionality? I know, I know, a dozen years ago it was much worse. Seriously? Oh well, too bad I only joined SL in 2011 and missed all that fun...

Edited by MBeatrix
Link to comment
Share on other sites

9 minutes ago, Sharie Criss said:

Yes you can. I guess that's the difference between you and me. When I am paying for a service and the service goes to crap, I'm going to say something and expect at least SOME response. 

It's very rare for a commercial business to allow developers to work as close to actual customers as LL do with SL. You have the opportunity to present issues directly to the people responsible for fixing it. But being able to do that comes with some caveats, one of which being the way you have to approach things.

14 minutes ago, Sharie Criss said:

LL has chosen to not even acknowledge the issue that MOST of us are seeing. To me, that speaks volumes. People generally clam up when they have something to hide. It's human nature.

LL is not one person. There is no 'reading between the lines'. Your interactions with staff are governed by policy, which believe it or not, are intended to facilitate a professional and beneficial relationship. No one at LL is out to trick you.

13 minutes ago, MBeatrix said:

I believe that LL has acknowledged the issue, but in the background. They didn't set regions restarts on a 10 days cycle just for fun.

Regions are generally happier if they are rebooted every now and then, same as pretty much everything. If it ever got to the point that a weekly reset wasn't sufficient then you could be very sure that will be investigated and corrected long before it got to needing a daily reset (and probably long before we noticed, LL do love their statistics).

  • Like 1
Link to comment
Share on other sites

11 minutes ago, CoffeeDujour said:

[...] If it ever got to the point that a weekly reset wasn't sufficient then you could be very sure that will be investigated and corrected long before it got to needing a daily reset (and probably long before we noticed, LL do love their statistics).

Or that, yes. They already knew regions would need to be restarted more often due to whatever they have been doing (the new features?) Not long ago they set a 14 days cycle, and then it changed to 10 days. Gotta wonder why...
Anyway, just for the most skeptical here's a practical example of what is happening: a few days ago, a rental business moved all their tenants to other regions because at the one where they were most stuff wasn't working anymore, and apparently LL was unable to fix it. The region is Warrumbungles and is now for sale at L$1/sqm.

Edited by MBeatrix
Link to comment
Share on other sites

3 minutes ago, CoffeeDujour said:

It's very rare for a commercial business to allow developers to work as close to actual customers as LL do with SL. You have the opportunity to present issues directly to the people responsible for fixing it. But being able to do that comes with some caveats, one of which being the way you have to approach things.

LL is not one person. There is no 'reading between the lines'. Your interactions with staff are governed by policy, which believe it or not, are intended to facilitate a professional and beneficial relationship. No one at LL is out to trick you.

Regions are generally happier if they are rebooted every now and then, same as pretty much everything. If it ever got to the point that a weekly reset wasn't sufficient then you could be very sure that will be investigated and corrected long before it got to needing a daily reset (and probably long before we noticed, LL do love their statistics).

It seems that you are a bit confused. Developers and systems staff are two completely different teams. It's not the developers that as we suspect are putting more regions on a server, they have NOTHING to do with that. Most likely it's not even the systems staff, but rather the bean counters, attempting to figure out how to save costs on SL to continue development efforts on Sansar. We don't know what LL has done since they don't ever talk about this level of detail,  and all we have is guessing, which is not very productive. But - let's just say - hypothetically - that the bean counters decided that the budget per sim server needed to be cut by $500. With 26,000 active regions, and assuming 10 core processors, there may be on the order of 1500 physical servers (as homesteads use less.) That savings would be something like $750,000 - not chump change at all. It would be easy to save $500 / server when the CPU costs can vary for a 10 core xeon from about $600 - $1500 depending on the cache and speed rating. Obviously choosing a more budget processor would have a serious impact on region performance. But again, this is all hypothetical. A guess. Suspicion. Nothing more. Whatever the real cause, it's painfully - PAINFULLY clear to everyone (except the LL apologists on this forum) I've talked to in recent history that "something" changed.

FYI, Region restarts can help with the memory leaks, they can not and do not have any bearing on the performance problems we are now seeing.

By the way, thank you for the info that LL is not one person. I would have never guessed that *rolls eyes*.

  • Haha 1
Link to comment
Share on other sites

As no current details of SL infrastructure are known, your blanket refusal to do any kind of empirical measurement, you're just here to have a moan and throw armchair conjecture around?

Blah blah blah mean evil LL are ____________ .. What do you want, sympathy? soothing words? an angry mob to agree with you? This thread is not "everyone" and not agreeing (as you're not prepared to actually do anything) doesn't make anyone an "apologist". It makes you yet another jaded armchair critic. "Everyone" I know is having fun, and yes, by providing actual information to the people responsible for developing LL you can affect change.

I'd suggest you switch game/company, have you considered being a PC Diablo player without a phone?

  • Like 1
Link to comment
Share on other sites

This is getting ridiculous.  Sharie is complaining and yes, beligerently, since this problem is seriously impacting her regions.  It is NOT the work of a "jaded Armchair critic" but of a determinedly loyal (to SL) resident and creator.

As far as "empirical measurement" is concerned, if we cannot rely on the tools specifically created by LL to measure SL performance, what hope do we have of convincing LL that something we create is any more reliable?

As for not being an Apologist, you have disagreed strenuously with every point that Sharie, myself or Beatrix has made recently, yet you have offered no true path to a potential proof or disproof of the issue.  If that doesn't make you an apologist I don't know what would.  I had enough carping criticism from Theresa to last me a lifetime.  This issue exists, it is real and it is being ignored by LL systems people.

I am sick and tired of trying to convince folk like you, Coffee, Solar and Theresa, it is not possible.  There are none so blind as WILL NOT see.

For the last time: I am NOT saying categorically that LL are dissembling about putting more regions on servers, I am saying that something WAS done and I would like to know WHAT IT WAS, because for some folk in some locations it has hamstrung their operations.

Edited by Aishagain
spelling. I don't type well when annoyed!
  • Like 1
  • Haha 1
Link to comment
Share on other sites

1 minute ago, Aishagain said:

As far as "empirical measurement" is concerned, if we cannot rely on the tools specifically created by LL to measure SL performance, what hope do we have of convincing LL that something we create is any more reliable?

Write a script. Run it. Write down the results. Publish them and the script !!

No on here is talking about region performance, you're all talking about script performance. Scripts run on MONO which is not written by LL.

Script performance can be tested and measured by scripts! Put the stats floater down and do some actual tests.

 

The OP's proposition is false, in that it neglects to appreciate regions are made up of many interdependent elements. If LL were indeed stacking more regions per sim cpu, then there would be more symptoms in more areas of the service. Everything would suffer, not just the script time number on the stats floater.

 

Degraded script performance could be totally real, but unless those who feel affected by it actually take the time to put some effort in, nothing will be done or can be done. No one at LL is going assign tech/dev time so they can pull everything apart based on a handful of people on the forum having a moan. But if you give them numbers and a reproduction to investigate, you will get some actual attention.

  • Like 1
Link to comment
Share on other sites

@Aishagain

I hate conspiracy theories, which doesn't mean they don't happen, both conspiracies and theories. But I truly believe LL hasn't done what some people in this thread seem to think they did, as that would be too stupid, if there weren't other reasons. And I have to agree with @CoffeeDujour that if it were true we'd feel and see it in more areas like, for instance, building two cubes and link them.

Now, as far as I know, the scripts "thing" has been happening at least for about one year in some regions. I followed closely the case of Warrumbungles because my partner rented there, and I know that despite many attempts from the management to get it resolved, all LL did was restarting the region on demand, so very recently the business owners decided they couldn't cope with it anymore. But maybe LL didn't fix it because there was nothing to fix, because it's the way things are supposed to work now, and also that's why the JIRA was closed and people are being directed to Support to have their region(s) restarted.

All this makes me ask a sort of an opposite question to the OP's one: considering that it's going to escalate, as there will be more and more scripts to do whatever, and more complex code on the server side, shouldn't each hardware server have less simulators running? Maybe not immediately, but in the near future? Yes, that means a considerable investment but you can't make omelets without eggs, and if it comes to the point that cases like Warrumbungles become frequent...

Edited by MBeatrix
Link to comment
Share on other sites

1 hour ago, CoffeeDujour said:

No on here is talking about region performance, you're all talking about script performance. Scripts run on MONO which is not written by LL.

Script performance can be tested and measured by scripts! Put the stats floater down and do some actual tests.

Scripts can't see more than the most summarized data about how a sim is performing; they can poll the sim for dilation (and, basically, its inverse) and that's about it. They can measure some detail about the impact of individual scripted items (and slowly sum up collected measurements), and quite a bit more detail about their own resource consumption.

There is no way for a script to know, for example, that spare time has gone to hell because scripts are all blocked and the sim is sleeping in script time. The stats floater is essential to making any sense of what we're experiencing.

Imagine we had a decade of sim performance data. Imagine we could retroactively measure the contribution of script time to those data. And even conjure-up the script counts for every timeslice. But... were those all the same scripts back then as they are now? At some point, haggling over what data we don't have is not making use of what we do have: a host of recent anecdotal evidence that script lag is worsening for some unknown reason. And it's not just what's in this thread: we've been talking about it for months now, and the Linden developers have high on their priority list a study of what can be done to alleviate script scheduler overhead.

(In passing, Mono certainly didn't originate in the Lab, but it also didn't come with a language binding for LSL nor an integration with the sim script scheduler. Babbage and later Kelly Linden did a lot of work gluing Mono into Second Life, and I'm sure neither of them would have asserted that improvement was theoretically impossible.)

Now, if I had a dedicated sim to play with (as the Lindens do), I'd set up some performance experiments, loading up a bunch of scripted items, some physics, maybe some Pathfinding, etc., etc. That kind of experiment could use the crude tools available to scripts to do some rudimentary data collection. But I don't happen have such an experimental apparatus to hand, so y'all are welcome to do it yourself. (See how that sounds?)

3 hours ago, Sharie Criss said:

It seems that you are a bit confused. Developers and systems staff are two completely different teams. It's not the developers that as we suspect are putting more regions on a server, they have NOTHING to do with that.

Maybe in some future, cloud-based world, developers wouldn't need to be involved in deciding how many sims run on how many cores. That possible future world "has NOTHING to do with" the current state of Second Life software. The "more sims per server" claim is just ridiculous now. Think of all the other ways performance could be and has historically been affected by changes in operating system, licensed third party software like Havok, increased diagnostic data collection, feature creep... the hardware hypothesis was never parsimonious, and now with developers assuring us it's not the case, well, it's just not the case. Let's please get past it and instead consider theories that might have some chance of validity.

  • Like 2
Link to comment
Share on other sites

1 minute ago, Qie Niangao said:

Scripts can't see more than the most summarized data about how a sim is performing; they can poll the sim for dilation (and, basically, its inverse) and that's about it. They can measure some detail about the impact of individual scripted items (and slowly sum up collected measurements), and quite a bit more detail about their own resource consumption.

I'm not saying write a script that logs sim stats, I'm saying write a benchmark. A script that does X, Y times. Something that gives an actual in world performance metric.

It's not ideal, it's not the be all and end all, but it would be a solid reference that in moments of doubt that are independent of the numbers in the stats floater, which I will say again, can easily be wrong or fudged or irrelevant.

The only thing that matters is actual observable performance. Does your script do as many things in the same time as it did at other times or in other locations.

1 minute ago, Qie Niangao said:

There is no way for a script to know, for example, that spare time has gone to hell because scripts are all blocked and the sim is sleeping in script time. The stats floater is essential to making any sense of what we're experiencing.

The stats floater is also incomplete in that it's a window into what is essentially a single vm on a shared host .. you can have no spare script time and no idea why because there are other regions and processes on the simulator that could be impacting performance. Linden logging is known to be a bit of a pig depending on what their looking for, even if your specific region isn't involved at all.

The question isn't why are these stat floater numbers different (from memory in most cases), it's, am I able to identify any change in actual observable performance.

If you are, then you have a pretty solid stick to poke LL with and get a little love, even if that's just a region reboot and punt to a different sim.

1 minute ago, Qie Niangao said:

Imagine we had a decade of sim performance data. Imagine we could retroactively measure the contribution of script time to those data. And even conjure-up the script counts for every timeslice. But... were those all the same scripts back then as they are now? At some point, haggling over what data we don't have is not making use of what we do have: a host of recent anecdotal evidence that script lag is worsening for some unknown reason. And it's not just what's in this thread: we've been talking about it for months now, and the Linden developers have high on their priority list a study of what can be done to alleviate script scheduler overhead.

and it's going nowhere because scheduling developers to go down the rabbit hole on a wild goose chase isn't ever very high on the list. There will always be more important and pressing projects. EEP, BoM, TP attachments (that took a lot of convincing as all we had was anecdotal information), OS updates and so on.

 

1 minute ago, Qie Niangao said:

Now, if I had a dedicated sim to play with (as the Lindens do), I'd set up some performance experiments, loading up a bunch of scripted items, some physics, maybe some Pathfinding, etc., etc. That kind of experiment could use the crude tools available to scripts to do some rudimentary data collection. But I don't happen have such an experimental apparatus to hand, so y'all are welcome to do it yourself. (See how that sounds?)

I'm not the one making up conspiracy theories about how were all being defrauded on the sneak by LL .. I bet the conversations around the water-cooler are closer to sarcastically stating " damn .. I wish we had the time to randomly mess with the customers "

If you want to make that, or any other case, you're going to have to open a script editor at some point. It would almost certainly be shorter than some of the posts in this thread. 

Finding an empty or mostly empty regions to test stuff on isn't that hard, there are a number of them on both main and beta grids dedicated to that end, not to mention plenty of straight up empty regions.

 

  • Like 1
Link to comment
Share on other sites

19 hours ago, CoffeeDujour said:

As no current details of SL infrastructure are known, your blanket refusal to do any kind of empirical measurement, you're just here to have a moan and throw armchair conjecture around?

Blah blah blah mean evil LL are ____________ .. What do you want, sympathy? soothing words? an angry mob to agree with you? This thread is not "everyone" and not agreeing (as you're not prepared to actually do anything) doesn't make anyone an "apologist". It makes you yet another jaded armchair critic. "Everyone" I know is having fun, and yes, by providing actual information to the people responsible for developing LL you can affect change.

I'd suggest you switch game/company, have you considered being a PC Diablo player without a phone?

Wow - okay, clearly you are a simple troll here, with a combative attitude and putting words in my mouth that I did not say and mis-characterizing what I said in a very negative light for no reason. It's a baseless attack on me. When you stoop to personal attacks like this, it shows your lack of maturity. 

Since you completely and deliberately ignored the FACT that pre-change statistics would be needed shows that you have zero interest in solving the problem or that you lack the capacity to understand the process. Based on your personal attacks, I have to go with the latter. I suggest bowing out of this conversation before you make yourself look even more foolish than you already have.

My goal here is to attempt to convince LL to address these performance issues and make SL a better place. What is your goal? Tear down SL and move to Sansar? Please, enlighten us all with your pearls of wisdom on how you are going to make SL a better place to build, play, explore when performance gets so bad you can't move or use any of the wonderful creations people have made....

  • Haha 1
Link to comment
Share on other sites

28 minutes ago, Sharie Criss said:

Since you completely and deliberately ignored the FACT that pre-change statistics would be needed shows that you have zero interest in solving the problem or that you lack the capacity to understand the process. Based on your personal attacks, I have to go with the latter. I suggest bowing out of this conversation before you make yourself look even more foolish than you already have.

 

You said you had them.

 

On 9/6/2018 at 4:31 PM, Sharie Criss said:

Performance benchmarks were from my own mono torture test script that takes about 10 -20 minutes to run. The experienced / connected people in SL who have been here a long time can see the changes - feel the changes. No, there's not a lot of hard data especially when LL is so tight lipped about the infrastructure, but when you spend a lot of time day in and day out over many many years, it's painfully obvious that systems are no longer performing well.

  • Like 2
Link to comment
Share on other sites

I'm also puzzled that we are supposed to not be able to use the built-in viewer metrics such as the stats bar to identify that there *is* a problem and report it based on that. Instead of reporting that "My parcel abc in region xyz has scripts time of 45%" we are supposed to create a benchmark test script to demonstrate the script time deterioration independently of the viewer metrics, with the ludicrous problem of the test script itself being subject to the same performance issue that is causing the script problem we would like to have addressed? Is this LL's stated position, that they won't accept the results of their own supplied metrics?

 

Let's take the suggestion to the logical outcome. If I have script deterioration on my parcel, I have to create a bench mark script to demonstrate it. But then in order to identify where it is, I have to find other parcels in the same region where I can rezz the object and run the same benchmark to show it's not confined to my parcel but is happening elsewhere in the region. And THEN, because of the subject of this thread, I have to somehow find other regions on the same server as the one I am on, find parcels in those regions where I can perform the same tests, and finally put a case to LL that "all the regions on a particular machine are showing serious degradation in scripts" ? What is the effect of all this extra script-work going to have on the servers?  

 

I'm sticking with the simplest option, using the LL-supplied performance metrics tool

Edited by Profaitchikenz Haiku
damn mouse click posted it before I was halfway through
  • Thanks 1
  • Haha 1
Link to comment
Share on other sites

An apology in advance...

In case it should appear I am picking a fight with Coffee, let me state that is most definitely not the case. Her fundamental point is sound, you need to be able to demonstrate a problem exists by providing something that can be shown and measured.

My point is that for a person owning a scripted object that is predictable (eg goes from point A to Point B and back again at a defined speed such that it can keep to a timetable), the fact of it not behaving as expected with no other changes to it should be enough of a test case to illustrate the problem.

I have watched two other threads (TPs and sim-crossings) cause some intense inter-poster arguments to the point where a Linden has posted to the threads stating that a) they are aware that there is a problem and b) "there won't be a solution found by people beating each other up so cease and desist". It may be that the only way to get an acknowledgement from the Lab is for us to create a rumpus here in the forums, but I can see it is a dangerous course to follow.

If there is indeed a problem arising where scripts are being suffocated, perhaps we should be making the point to LL without adding any hypotheses about fewer servers? That would also perhaps assist all of us who are capable of observing, measuring and reporting on what we in the parcels are experiencing objectively.

As scripters, we ought to be taken as seriously as mesh-creators, our input to the world brings it to life, so we ought to be able to get LL to take our concerns just as seriously as they have those who couldn't get from A to B without logouts recently. People who have bought items with our scripts in them will not know enough to create test cases to report throttled scripts on their parcel to the Lab, and I doubt that half of them would even know to look at the stats bar to see of their region has a problem. It's up to us to report what we see as scattered but similar problems in such a way that they can be investigated, and if it means a Linden has to come in-world to a certain parcel to see for themselves, then so be it.

Edited by Profaitchikenz Haiku
Link to comment
Share on other sites

Re-reading that thread referencing benchmarks that Theresa cited, I'm reminded that different posts scattered through these various threads have been discussing a wide variety of problems all related to sim performance. I'm probably clumsily re-inventing the wheel here, but it would be handy to have a taxonomy of these problems, in hopes of finding ways residents and Lindens might work on refining and solving them.

To start, we might decide we're only interested in low Scripts Run percentage. There are lots of other problems we might look at, but let's boil one ocean at a time.

About problem onset:

  • Was it gradual or sudden? (I've been most focused on gradual decline, but some folks are saying their problems came on abruptly one day)
  • When did it start? (Some say two years ago, some say it coincided with the EEP or OS upgrade changes a month or so ago)

How have non-script times changed? Spare Time must (always?) fall to zero if there's a low Scripts Run percentage, but:

  • Has Script Time actually decreased because other metrics (Agent Time, Simulation Time, etc) increased, consuming frame time that used to be available to scripts?
  • Are frames extending so time dilation is less than 1? (Maybe don't look at these cases because scripts can't improve until the other times reduce)
  • Does "Detailed Times" : "Sleep Time" track Spare Time, or are scripts blocked while the sim is in Sleep Time? (the latter being the Vallone incident of a few weeks ago)

How does a sim restart affect the problem?

  • If it gets better and then gradually decays again, that may be some resource like memory or http connections that can "leak" over time
  • If it gets better and stays that way the problem may have been due to other regions hosted on the same server due to competition for shared resources, whatever those can even be
  • If it doesn't improve at all, then the cause is static and local (but still could be anything from region contents to OS changes to the dreaded "improved internal monitoring")

How have changes in script load affected the problem?

  • Are there more scripts on the region than before? or busier scripts, perhaps responding to more events?
  • Besides removing all new scripts, how many old scripts need to be removed to get back to normal performance?
  • Is there anything similar about those scripts, or is it simply the sheer volume of scripts (or events) that matter? (If there were a consistent answer to this question, we'd really have something to investigate.)

Honestly, I can't tell if this is going anywhere. I just sense we've been talking about different but related problems, only some of which maybe have the same root cause, and to formulate hypotheses that are testable (by residents and/or Lindens) it's pretty essential to define the specific phenomena those hypotheses are supposed to explain.

  • Like 2
  • Thanks 3
Link to comment
Share on other sites

You are about to reply to a thread that has been inactive for 1756 days.

Please take a moment to consider if this thread is worth bumping.

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
 Share

×
×
  • Create New...