Sharie Criss

It's the same illness that griefers have, they get enjoyment out of attempting to make other people miserable.

About 18 months ago, something changed overnight. Regions (not just mine, it was all over SL) that used to operate in the 90's for script run time dropped to the 40's and 50's. That's what happened. Lag spiked everywhere, avatars TPing in and out had a magnified impact on region performance. I filed quite a few tickets at the time for my own regions with absolutely ZERO resolution. LL wasn't talking about it then, and they aren't talking about it now. Over the years, I had occasionally run some of the benchmarks available at http://wiki.secondlife.com/wiki/Mono#Testing and from memory noticed that the benchmark was running at half speed. Some people here have suggested that since I didn't keep a record, that I should just go away, and that LL can't take any action without "my" benchmarks. Really? Does LL pay me to do take benchmarks and log them? No. Does the fact that I don't maintain benchmark logs mean that as a PAYING customer, I have no right to speak up when things go to heck? No! Do you really truly believe that LL runs THOUSANDS of servers and doesn't keep their own benchmarks or monitor server performance? SL is not run on some server in someone's basement, it's a large intricate network - and without benchmarks and performance data, it would be IMPOSSIBLE for capacity planning to happen, or even to know when things go wrong! They can get performance data on every part of server and network operations at a much deeper level than a silly Mono benchmark I run. Do you really think that they are not completely aware what happened? LL employs some very talented people who keep things running. Trust me, they have the performance data and LL staff has stated that they have it. In fact, it would be grossly incompetent for the infrastructure team NOT to maintain performance data (I'd fire them.) In short, LL is obviously fully aware of what changes they made that caused region performance to drop because the talented people they employ to run their server operations are just not that incompetent to NOT know! LL would not have survived this long if staff were that incompetent. So lets give them the credit they deserve and realize that they are talented and care about the company they work for and the product under their sphere of influence. As end users, all we REALLY have is perception and the silly region stats floater that only gives us a pinhole view into how things are running. Even benchmarks we run are suspect as the physical server is shared - performance is going to be affected by other regions on the same server, or someone TPing in and out, during the test etc. Even people in neighboring regions can affect performance based on draw distances and such. Only when changes are large like what happened ~18 months ago does it make it obvious that something changed for the worse. But again - claims that LL can't do anything without MY benchmarks are just - ignorant. It shows a total lack of understanding on how large services operate. Use your common sense folks! I responded to this thread because during the recent TP crashing issue, LL rolled a release that was so incredibly bad that my script run time couldn't get past 10 and menu popups that should be nearly instant were taking nearly 10 seconds. Fortunately they did correct this with a subsequent rolling restart. That release clearly should have failed QA, but I suspect that it performed poorly because of additional logging they needed to deal with the TP issue. But let's get back to the main topic. Why are some people so incredibly resistant to call for LL to resolve server performance issues? What do you have to gain? Why are you refusing to believe that LL did something (which they clearly did) which caused these performance issues that were not here 18 months ago? Do you hate SL that much that you want it to die as people get so frustrated they dump their land again like the mass exodus that happened when the Adult changes were rolled out? Maybe you were focused on other things (sansar?) and didn't notice the performance drop. Why is it impossible to believe that someone else did? How does increasing server side lag and other negative problems that are happening benefit you? Why is it so hard to believe that it's possible LL management was trying to save money and ended up cutting corners in a bad way? Have you ever met a CFO that wasn't constantly looking for cost savings? As a private company that hasn't received an influx of (investment) cash in years (according to the financial data I've been able to find, which is slim for private companies) I'm sure there have been cost saving measures all over the place. This is where I suspect the issue resides. It's my opinion based on the evidence I've found. I'm calling for LL to open up and address this issue. Enough of the silent treatment.

I said I RAN them. I did not keep them. Details matter folks.

Wow - okay, clearly you are a simple troll here, with a combative attitude and putting words in my mouth that I did not say and mis-characterizing what I said in a very negative light for no reason. It's a baseless attack on me. When you stoop to personal attacks like this, it shows your lack of maturity. Since you completely and deliberately ignored the FACT that pre-change statistics would be needed shows that you have zero interest in solving the problem or that you lack the capacity to understand the process. Based on your personal attacks, I have to go with the latter. I suggest bowing out of this conversation before you make yourself look even more foolish than you already have. My goal here is to attempt to convince LL to address these performance issues and make SL a better place. What is your goal? Tear down SL and move to Sansar? Please, enlighten us all with your pearls of wisdom on how you are going to make SL a better place to build, play, explore when performance gets so bad you can't move or use any of the wonderful creations people have made....

It seems that you are a bit confused. Developers and systems staff are two completely different teams. It's not the developers that as we suspect are putting more regions on a server, they have NOTHING to do with that. Most likely it's not even the systems staff, but rather the bean counters, attempting to figure out how to save costs on SL to continue development efforts on Sansar. We don't know what LL has done since they don't ever talk about this level of detail, and all we have is guessing, which is not very productive. But - let's just say - hypothetically - that the bean counters decided that the budget per sim server needed to be cut by $500. With 26,000 active regions, and assuming 10 core processors, there may be on the order of 1500 physical servers (as homesteads use less.) That savings would be something like $750,000 - not chump change at all. It would be easy to save $500 / server when the CPU costs can vary for a 10 core xeon from about $600 - $1500 depending on the cache and speed rating. Obviously choosing a more budget processor would have a serious impact on region performance. But again, this is all hypothetical. A guess. Suspicion. Nothing more. Whatever the real cause, it's painfully - PAINFULLY clear to everyone (except the LL apologists on this forum) I've talked to in recent history that "something" changed. FYI, Region restarts can help with the memory leaks, they can not and do not have any bearing on the performance problems we are now seeing. By the way, thank you for the info that LL is not one person. I would have never guessed that *rolls eyes*.

Yes you can. I guess that's the difference between you and me. When I am paying for a service and the service goes to crap, I'm going to say something and expect at least SOME response. LL has chosen to not even acknowledge the issue that MOST of us are seeing. To me, that speaks volumes. People generally clam up when they have something to hide. It's human nature. Me: There is a pothole in the road. It needs to be fixed. You: Can you prove that there wasn't always a pothole in the road? Me: No, sorry, I don't keep photographic evidence of ever section of every road. I just know that there is a pothole here now that wasn't here last month. You: How can you expect anyone to believe that the pothole wasn't there by design if you can't prove that it wasn't always there? Really - it's that ridiculous.

So are you suggesting that we travel back in time to 18 months ago when this started so we can get a pre-change benchmark documented? That's what it would take to get what you are looking for. Since that's impossible, all we have is what we can see NOW and how things appear different. As I've stated in other posts, non-logged occasional benchmarking was performed using scripts that are available on the SL Wiki. Go look. Your post is essentially saying "Prove it." That's just not helpful. Can you prove the opposite, that LL did nothing to cause this problem? Do you have test results and statistics to back it up? Absolutely all the anecdotal info says otherwise.

Wow. Just wow. When I don't get the service I believe I am paying for, yes, I say something. Apparently in your mind, that makes me just a complainer rather than - oh - someone who actually gives a crap rather than being a schill for LL making excuse after excuse for them without actually truly knowing the facts. One of the things I've bee asking for - oh - wait - COMPLAINING about, is better region tools to analyze script performance as I've described in the past due to the fact that the existing Top Scripts doesn't give us any valid info at all as it's only which object was using the most CPU at an instantaneous point in time. I also ask for - err - WHINED about, tools to limit a visiting avatar's script impact on a region as that useless top script tool was showing huge numbers for these avatars when they first arrive, with numbers that regularly show 15+ms PER avatar in script time. When things were really bad, I've seen avatars soaking 100+ms of script time for 15 seconds or so according to the Top Scripts tool. During this time, that Time Dialation number drops like a rock too indicating that the ENTIRE sim is suffering, not simply script performance. This manifests itself in ways like - you can't move at all or as if your were stuck in taffy, your vehicle careens off into a high speed lagfest rubber-banding 3 regions away before snapping back, etc. I must be imagining these things happening, because some people are convinced that LL fixed all that. I must be hallucinating now. We understand what 50% means. We also know what it implies. There are only a few metrics that we have access to that actually give us any good indication of a regions performance, such as Time Dialation, which when less than 1.0 means that EVERYTHING is suffering (this happens a lot more now than it used to, watching it drop to 0.01 is always good entertainment,) Ping Time, which tells us if a region is able to keep up network wise. Script time however is one of the most sensitive indicators of the sim server's overall capability. As we are told, scripts run in spare time meaning after everything else is handled, communications, physics, etc. etc., THEN scripts use whatever is left. LOGIC says that if there are less CPU resources available then the spare time pool will suffer the most - that's Script Time for those paying attention. About 18 months ago, there was a GRID WIDE drop of script time to where it was rare to see a region perform with better than 50% unless it was devoid of scripts such as empty G rated mainland regions (of which there is a lot of.) Let's toss my region aside for the moment. What about THOSE regions? Lastly, I take exception to your premise. "They're region simulators, not script running systems." This is wrong. The region simulators have an embedded script running system. Two actually, the old LSL and the slightly newer Mono. LL is advertising on the main SL web site: "Become a Creator" "Express yourself & create anything you can imagine." LL also created the scripting system to actually - you know - be USED. They keep adding to it, with new features and functions. They added additional script using systems such as Experience Keys, Pathfinding, Animesh, but - wait - are you saying that our creations shouldn't actually use these features????? I guess SL is only for those people creating no-mod mesh clothes and charging for each color separately. You heard it here first folks. Go wild, create anything you want, just don't use scripts and expect them to perform well because the LL apologists will cut you down. (Edit) LL markets SL as a social platform where you can enjoy venues and products created by residents. ANYONE can create a venue. Creating a venue that remains popular for over 10 years with non-stop traffic numbers is not so easy. That's what I have. It's a place for people to gather and have fun. It's what SL was CREATED for. To sit outside in your little ivory tower and dis my region because it has what people want shows a total lack of understanding for what SL is all about. When LL makes changes - whatever they are - that damage the ability to have these venues - they are doing it wrong. I am using SL and my regions resources AS LL INTENDED, and provided the tools for!

It used to be over 14000 running at 99% script run time without the sim being empty. So yeah - it's much better than it used to be script count wise. What's your point? Is your point that we should only expect good performance with an empty sim with no scripts? Geeze, your kinda sim seems like super fun to me. Cornfield anyone?

My TP's are causing logouts about once every 3 attempts. Sometimes right after logging in the first TP causes a logout. It's pretty serious. But even MORE concerning is that today, without getting new server software, my region is performing nearly 4 - 10 times slower than normal (it's hard to quantify.) With the region empty except for me, getting a simple menu from some item like furniture that should be nearly instant is taking nearly 10 seconds. Restarts haven't helped. I'm really close to cancelling my regions and just giving up. I filed a ticket yesterday but apparently there's nobody left at LL to look at it.

But this doesn't make sense. As I mentioned in a previous message, servers today are MUCH faster than the old systems used to be. SL is NOT running off 12 year old hardware. LL has moved subsystems off the servers (pretty much any of the old UDP delivery of data such as inventory, assets, textures is gone and it's now cloud delivered, or comes from other servers) so theoretically, with faster hardware and less to do the regions should perform better, not twice as bad as they used to. When you move stuff off, use faster hardware, and end up running slower, you are doing something very VERY wrong. The fact that LL REFUSES to acknowledge server performance issues AT ALL is Highly suspect. If they aren't sticking more regions on a server, then they screwed up royally. I can't believe LL is THAT incompetent.

FYI, the estate tools are USELESS at finding out what is burning time. Why? Because they show you an instantaneous point of time view, not how much resources (RAM / CPU) a script REALLY uses over time. Just hit refresh a few times and you can see what I mean. I've been asking for better tools but it's like a zero level priority for LL, which is sad, because without good performance analysis tools it's impossible to know which scripts are really causing an issue.

After the restarts, my region is now performing at 1/3rd the speed it used to. Restarted a second time, no better. Seriously LL, you owe us some answers. I will NOT pay for another month's service if the performance stays this bad. I'm done. Whatever you are doing, you are doing it wrong. TP's still broken too, which sucks the joy out of even logging in. It's painfully clear based on recent events that the executive team no longer cares about SL and it's paying user base. The Second Life product is in a terrible state - the worst it's been in many years. Not only will I no longer pay for a shoddy product, I'm due a refund. I've been paying for service but am not receiving it. -

I've been running a full region venue for over 10 years now. The region has certainly gone through changes over the years, but I've been monitoring script / prim usage closely, using estate tools to find poorly behaving objects and replace them as needed. From the peak, script count is down 30% yet performance is down even further, at best running at 50%. I occasionally run Mono benchmarks (see the LL wiki for them) and what is PAINFULLY clear to me is that my region is running at half the speed it used to. I've filed quite a few tickets regarding performance issues, but Larry Mo and Curly show up and claim there is no issue. Frankly, I've completely given up any hope whatsoever that we will ever get good region performance ever again. Wandering around SL vising some of the same places I've gone to for years, everyone is suffering with these same performance issues. What everyone knows (but a few here have trouble grasping) is that computers now are far far more capable than systems were 12+ years ago, and all that extra performance is available at a fraction of the price. Data center costs have also dropped due to massively increased competition and other economic factors, and bandwidth is cheaper too. Taking the same money we were paying for regions 12 years ago and applying it to today's systems, we should have zero server side lag. Zero. Also, keep in mind that LL has been doing a lot of work removing work from sims by moving things like asset downloads / inventory / etc. to the cloud (content servers) and away from the sim itself. Why aren't we actually seeing a benefit to the region performance from these efforts? No, I don't have any proof, LL won't talk about it, and everyone is just guessing, but based on all the data it's clear something has changed that has sucked the life out of region performance. I strongly suspect that LL is saving money by cutting the infrastructure available for regions during the last technology refresh. What's incredibly disappointing is that we finally have some really neat technology such as Animesh, pathfinding, Experience Keys etc. that could allow creators to build fantastic new venues, games, etc. BUT, now region performance is so freaking horrible that we can do none of that, and in fact, some venues that did take some advantage of these new features are unplayable. Vehicle usage is a joke with lag spikes that send you into a lag freeze shooting you 10 sims away before snapping back every time someone TPs in or out. Region crossing is a crapshoot. Clearly LL has lost any pride they had in SL. People show up to collect a paycheck. They go through the motions, throw us a bone now and then with some new feature that's pretty much useless due to server performance issues. Bleh.

the 256m limit is more of an LL configuration setting, just like the prim limits. That said, everything has been built with that size in mind, so allowing larger sims would have implications for scripts, the world map, and pretty much everything else. The simulator software itself would probably have no issues with larger regions, it's just everything ELSE that would.

FYI, you are replying to a 5 year old post....

FYI, with the latest rolling restart, TP's between regions that were JUST restarted are still failing.

https://www.google.com/search?q=linden+labs+layoffs There are other rather simple searches you can do too. While LL is a private company and most of the data is not public, you can find enough searching various data sources and news outlets to get a pretty clear picture of what's happening internally. Performance benchmarks were from my own mono torture test script that takes about 10 -20 minutes to run. The experienced / connected people in SL who have been here a long time can see the changes - feel the changes. No, there's not a lot of hard data especially when LL is so tight lipped about the infrastructure, but when you spend a lot of time day in and day out over many many years, it's painfully obvious that systems are no longer performing well. Why am I writing this and responding? Because I care and it bothers me that this is happening. I want LL to succeed really, but I would not be making the same choices they are. For example, I would look at server balancing strategies that put consistently busy regions on servers that only have very lightly loaded regions. or giving the option of doubling RAM / CPU for busy venues. I would investigate the possibility of putting empty regions into a "light sleep" mode where they only use a fraction of regular resources until someone enters the region. SL is a social platform. You can't size your systems for empty regions. You have to accommodate the fact that maybe, just maybe, people in a social platform may - gee, I don't know - be together in the same place. There's lot of room for performance improvements but nothing will get better unless you are willing to admit that a problem exists and are willing to actually address existing problems. Or you can be a defeatist - "SL is dead. Move on."

I realize it's a difficult concept to grasp, but when I'm paying for a region, and LL does "something" which causes the region I'm paying for to now perform horribly basically destroying the usability of my venue, I think I have a right to be upset that they are taking the operating revenue from SL and using it for something else. These aren't profits. SL doesn't have any.

Okay, I'm just surprised anyone would recommend going through a personal conversion project in light of the enormous effort it takes. I've done several, and found that it's just not worth the time as I can't "resell" the effort. It's worth it to just spend another few thousand lindens to just buy something that was made right to begin with. Creators on the other hand will find it worth the time and effort as it increases the marketability of their older products. I've seen quite a few refresh old products with the same animations, but updated mesh primwork and new scripts. Some (few) offer the update for free but most charge for it (which is fair - it's a lot of effort!)

The difference now is that sim performance is horrible all the time rather that only when people TP in and out. The TPing issue simply makes things worse, now instead of the sim dipping to a time dilation of 0.5 with script run at 30%, the script run times now are NEVER better than 50% and dip to 1%. That's what's different. Before you could at least do things most of the time, now - it's just horrible and laggy all the time. And it's not just "my" region, it's every region. Perhaps you were unaware of this.

While some people disagree, the anecdotal evidence makes it painfully clear that LL is "saving money" on SL so they can spend it on Sansar by cutting their operating expenses - such as technical support, and data center costs. I very very strongly suspect based on the significant reduction in server performance that they have increased the number of regions they place on a physical server. Based on LL's behavior, I REALLY doubt that any of these performance problems will ever be addressed. Some people point to continued development as "proof" LL is not doing what I claim, but if I were LL and wanted to keep my cash cow alive, I'd be attempting to retain gullible customers with the same type of lures. The reality though is that this development is painfully slow, and very very few existing bugs (some many years old) ever get fixed. Animesh? Bakes-on-mesh? Where are they? Will they go the way of Experience Keys where the concept of Grid-wide keys was dropped? You know - the version that would make Experience Keys actually - useful - outside of a very limited environment and use case.

Do you have ANY idea just how much work that is? It's not for the faint of heart. While there is a converter script, it does not work with all poseball based systems and only does about 1/2 the job when it does. For a bed with hundreds of animations it can take many many hours to do this conversion. Days. If anything, I'd go back to the creator and ask THEM to convert. Many people won't even BUY poseball-based furniture anymore (poseballs have been obsolete for over 5 years....) Recommendation while shopping - if you come across something you like, but it uses poseballs, IM the creator. Tell them "your product looks great but I see that it still uses the antiquated poseball system. Have you considered releasing an updated version that doesn't? Due to a variety of issues, I no longer buy items that still use poseballs."

Question: What is Linden Lab's plan for increasing region performance? We have seen a DRASTIC reduction of performance in the past 12 months resulting in a massive increase in server side lag which appears to be caused by a reduction of CPU / Memory / or something on the server side (maybe putting more regions on a physical server.) The net result is that all the new SL features for scripted content (Experiences, etc) are for naught if the region is performing so slowly that scripts can't run. Existing venues are now almost unusable. In a similar vein, there is a long standing bug which pretty much freezes a region for 15-20 seconds when avatars TP in and out which is DEADLY to busy regions. The bottom line with region performance is that the current situation makes popular destinations nearly impossible to enjoy. Popular destinations are what MAKE SL as it's all about being a social platform. You can't be a social platform if you can't handle the server load caused by a lot of people getting together!!!

Please exp Please explain the cause of the dramatic overnight increase in lag where Mono benchmark performance dropped by 50%. Please explain the draining of development resources from SL that got transferred to Sansar (this was publicly announced.) Please explain the severe reduction of support to the point where we can't even get a crashed region restarted during non-business hours (which is really fun when it happens on a Friday night.) I could go on and on here. You can't rewrite history, these are facts. The bottom line is that Linden Labs has reduced their expenditures on SL dramatically. The new development being done is done at a snails pace in an attempt to not cause a mass exodus ("We aren't abandoning SL, really!!!! Trust us!") This is how business works. Linden Labs is seeing SL as an eventual dead end and and wants to be in VR badly - so much so that they are willing to sacrifice SL in the process, but is doing it in a way that makes it not so obvious until you look under the covers. Seriously, you don't expect them to flat out say "SL is dead and we are moving on" do you? Their income would dry up to nothing and they would be out of business. But sure, keep kidding yourself thinking that the VR fairy dumped truckloads of cash on Linden Labs to do Sansar. Sansar is an ENORMOUSLY expensive undertaking. I'm not faulting LL for wanting to continue to live by investing in a new platform that they see is their future, but this comes at a cost. That cost is our ability to enjoy SL because of the sacrifices made. Spending the kind of cash that I do on SL (which has been a LOT over the years, it's my core hobby) it does bother me that LL is refusing to address the white elephant in the room that is increased lag. When Linden Labs does not have another source of income, it's naive in the extreme to refuse to believe that the money to fund Sansar is not at the expense of SL infrastructure, support, and development. I have a bridge I'd like to sell if you are interested.... Regarding Blizzard, since you wanted to bring something totally not relevant into the discussion, Blizzard was making a TON of money on Warcraft and had enough left over to do Overwatch. This is NOT the same case with LL - LL had to make a choice - SL or Sansar because they did NOT have a heaping big pile of profit lying around. They were barely making ends meet and laid off 30% of their staff to cut costs. The remaining staff had to be split - the Sansar Future and the small team to just keep SL alive long enough to get them there.