Lydia Craig

Oskar, from a closer examination of the sim I am satisfied a restart did occur today, but was it accompanied by a rollback, and what is the meaning of the new code numbers which seem to indicate a rollback to last week. Most important of all, given there was a restart and allocated memory should have reset to a normal 61mbs, why after about 12 hours has it climbed to 136 mbs and apparently is still climbing. As I reported previously the last time this happened it continued to climb to about 224 mbs then underwent a series or crashes which finally resulted in Elijah Linden intervening and fixing this mess. Unfortunately his fix was not incorperated into the next restart and the leak returned and now appears worse than ever.

Oskar can you clear this question up? It certainly seems a valid one and seems to have impacted a lot of people and sims. Also if there was a restart please explain why the allocated memory for the server in Zerango is growing by leaps and bounds and what if anything are your plans to deal with this mess. The last time this happened the server had a series of serious crashes by the weekend and Elijah Linden restarted it and did something that seemed to plug the leak until the next restart reopened it and it has continued to be a problem every since.

Oskar I am getting the same information on the update version and that is the code you installed last week. Would you please explain what is going on and when we will get the new code and if the restart was aborted or rolled back due to problems. This is at this point most confusing and hardly what we need to raise our confidence in LL or its reliability. Indeed, I would go so far as to venture Zerango was not restarted at all, since allocated memory due to a long term leak caused by a restart a few weeks ago, is about double what it normally should be, and if we had had a restart it would be in the low 60s, so I ask again with is going on and when will be be getting the restart we were designated to get today.

Your problem most likely is server hardware, take a look at allocated memory and you will likely find it in the three didgit area, look at it again Wednesday after the restarts and it will be much lower but will creap back up over the next few days, this seems to be a memory leak that appeared about three weeks ago and which LL has either not or been unable to fix. Blaming these problems on connections, routers, third party or even sl viewers is just a smoke screen that conveniently overlooks the fact that all of these things send information and commands to the server core which apparently is not sending the proper responses out to the servers. This in turn is leading to lag and sudden crashes. Adding new programs will not help and will almost certainly make things worse, and repairing server code on restarts apparently is doing little good either, this is a hardware problem likely related to to timing issues that LL has not resolved since it introduced MONO several years ago and which has just gotten progressively worse this year. Until someone takes the time to fix the problem, and it can be fixed as we saw it fixed in Zerango several week ago and then recreated in the next restart which did not take into consideration the modification that had been made in the server over the previous weekend when it nearly failed completely. These problems need to be given priority, however, they rather seem to be ignored in the hope that further code will undo the mess that previous code has made. Unfortunately that is like putting a bandaid on a gapping wound and things only get worse.

Oskar, this rezz time problem seems general, but it is worse in server which allocated memories that have grown significantly since the restar. Two come readily to memory Turnip which appears to have a significant memory leak that was not fixed in the last restart, and Echo Park which has a similar problem. Ironically Turnip in on the Blue Steel system and Echo Park the main server group. Significantly, Echo park also shows huge lag in general while turnip is slow at time to rezz but except after teleports and login otherwise show no severe lag. Ironically, Osbourne Walk which has not growing allocated memory problems and Zerango which has in the past had a severe problem which apparently was fixed again in this last restart are performing fairly well. I hope this helps in tracking down this problem because even though FPS and other traditional signs of lag has improved lag persists and the only probably explanations for it I can see are a leak that is causing massive increases in allocated memory and scripts timing out and corrupting our cpu caches. Both these problems see to be happening and seem to be relieved by relogging our entire systems and clearing cpu cache. Hopefully this can lead to some resolution for in some areas this is a crippling problem.

Would you please fix the crash followed by region is logging you out bug that apparently was introduced again in last weeks restarts. Also try to do something with the debilitating memory leaks and lag in the servers which are still clogging our cpu cache and causing problems, as well as using current saves rather than ones from previous weeks for the restarts, since those reintorduce problems that have already been fixed by LL or the players.

Here is a new bug we found. Several days ago my business partner and myself were both hit with sim rebake request repeatedly. Each time the rebake failed and this continued til we both crashed. When we reuturned to sl the bug continued but we ignored it and reported it to live chat and agreed to watch for a recurrance and report our finding. Live chat then restarted the sim and thing returned to normal, but we noticed that over the last few days allocated memory escalate dratically from a normal 80 after the restart to 255 tonight. Also I noted that after the rebake spam begain it jumped to 355 and then went down to 251 after we crashed. As agreed I contacted live chat gave the information to JonothanM and asked for another rebake and was told we could not have one since we had had one recently. Obviously this person did not listen to the information I gave him or, if he did did not understand it, and certainly did not look at the case notes on this issue which I assume were taken after the first incident. This issue needs serious investigation since obvious a serious problem is going on with this server and is not threatening to cripple it completely. Unfortunately, no one seems interested in dealing with it so I am brining it to your attention in hopes someone may encourage engineering to fix this mess before it srpeads to other servers.

Thank you Toy you have conducted yourself very well in this matter, now if only the Lindens would follow your example and show an equal amount of concern and responsibility. There is, however, two factors you seem to have overlooked. First, this problem occurrs much less often in the RC servers which allegedly have the same code at the moment as the main servers, so I am starting to doubt this is a soft ware issue, second it is often accompanied by huge amounts of sim ping which I take it to signify an over worked server. Indeed in some cases I have seen ping go above 30k in some main channel servers, and again this problem goes away when there is a reset, so if software is involved maybe we should look at the core servers and not the regional servers since they saw some serious attention at the start of this "improvement cycle" and some changes made to them then may now be bearing poisoned fruit. Whatever it is it is certainly a major problem and seems to be a carry over from the spam problem corrected with such fanfare earlier. My guess is the servers have been and are overworked and overloaded and are beging to show signs of failing and need to be dealt with at once or things will only get worse.

Not obvious at all actually, and when this "lag" can be cleared simply by teleporting to another sim and back it ovbviously is not a viewer issue, especially when the viewer was working well before the restarts. Truth is LL fixed the code problems involved with the bandwidth bug, but had yet to deal with the hardware issues that accompanied it, and probably have not fixed all the hardware setting involved in the issue. Also the servers were vastly overstressed by this problem and some may have been weakend to the point where they need repair or replacing. One thing that seems to come with this lag is bandwidth stays low but sim ping and allocated memory increase rapidly. These are not signs of a happy server expecially when people with different viewers and machinces are getting virtually the same numbers. LL is aware of these issues and in typical Linden fashion doing nothing about them, and it looks as though the main servers will get no restart again this week, but in LL eyes that is undoubtable fine for to them the issue is fixed. The reality though is it is far from fixed and on the very in some areas of becoming critical again.

Things work really well as long as you dont try to use camara controls open inventory, rezz items and do any of the numerous things you expect to do normally in sl. In other words sit still and do not move and it works fine and you may be able to even hold a conversation if you are lucky. Eventually though lag will catch up with you memory use will rise and ping sim will run out of sight, earlier tonight in Echo Park I saw it reach over 20,000. Now the good news, Oskar this is not your doing, the RCs which allegedly have the same code as installed today work fine have very little lag and are models of stability at least at the moment. This suggest that either the main server hardware is wearing out badly or someone over the weekend changed the settings on the servers from what was working on the RC channels. Whatever, it was that caused this does not seem to be on the RC servers and probably was inadequately tested by someone other than the restart team who innocently walked into a nightmare. This is not good procedure and needs to be corrected as do the problems that apparently still remain with the main servers including accounts going to negative figure when mone is spent. Whether these are repeatable bugs is hard to say, but what is certain is this mess is far from fixed and pretending otherwise only makes things worse.

Oskar we appreciate what you have done, but you should not be the one apologizing, it is the people who created this code and did not test it enough who need to be speaking not you, and we do not blame you for their neglect. That said, as of today's restart for some reason the the lag and use of memory has accellerated greatly all over sl, and is at the point where it is beginning to effect log in with the DNS error, which is coming not from our isp not sending the DNS but the SL program not picking it up and relaying it through the log in process. This is a major issue and if it continues will effect the recovery we have seen in online number over the last week and likely drive more away from sl. This needs to be looked at and addressed quickly before it either gets worse or becomes the accepted norm for SL service.

But intent and reality are two different things and if the reality is the problem increased then whether the problem was intended to be increased or not is a moot point, and in this case the problem has increased.

At least in Oakley the restarts have been far from unnoticed. Here lag has marked increased and bandwidth usage is up ten fold over what it was prior to the restart and sim ping is running at the three to four digit level constantly. I believe this needs to be seriously investigated because as of now the sim is all but unplayable. It might well be wise to run a comparison of this and other sims in the LeTigre system, because i have noticed that with the other RC groups performance can vary widely from sim to sim and is usually somewhat improved by a restart, which would seem to suggest that the problems are occurring at the server level, though what the source of the server problems are remains open to conjecture. Still, it raises the possibility that this is not the software problem that people suspect, it may be a hardware problem or a hardware involved problem. Either way it bears investigation and needs to be dealt with soon.

Regrettably this is too little and too late. LL does not have weeks to fix the mess they have created much less months, players patience and trust has gone and the exodus will continue till have a world full of little more than bots. Come to think of it this may be both what LL wants and deserves. By the way Oskar I do not blame you for this decision you are only the messager, but I do find little to respect in the LL management which will be charging people money for service they are no longer providing and difficulties they should have avoided if they had wanted.

For once be clear Pathfinding is not the chief issue, it seems to be the new Havok Physics engine, which is causing most of the trouble. Maybe you did not realize that though called the Pathfinding Project last weeks restart contained two components and it is the second Havok and the preparations leading up to its installation that seem to be the cause of these problems. This probably is of no concern to you, but is of huge concern to business owners and players in SL who trusted LL not to make this kind of mess as this critical time in SL history. LL, however, did not listen to the warnings of those who tested this physics engine and went on with the installation with little if no warning or consideration for the consequences. Those consequences have indeed been severe and are getting worse and if not quickly correcte may well lead to the demise of Second Life. If this seems bitter it actually is quite mild compared to what is being said by those who do not appear in these forums and are quietly preparing to exit SL talking with them a considerable part of its economy.

Given the chaos and problems caused by last weeks restart and the fact that this has litter made business all but impossible since last Wednesday, the sim on which our main business is located has crashed, been down or reset at least 14 and likely 15 times and the traffic count is totally inaccurate, one wonders if LL should not consider not charging tier this month since they are obviously not delivering the services they should be providing. I don't expect this to happen, but without a dramatic gesture like that I also do not see SL surviving much longer since when you take away the bots that are artifically propping up concurrency you get a live concurrency number that closely resembles what I saw when I first came to SL over six years ago.

Then when we cancel our accounts, the answer will be simple--LL and its arrogance. At the end of this month LL will expect Iam and myself to give them over 300 usd for services they have not given us. Normally we would make enough to cover most of those fees from the business that inhabits that and the other sims we own, but LL through its decisions has made it impossible for us or any other honest business to earn a living at the moment, and I for one see little chance of things improving in the near future. Some of you may call this a rant, well if a rant is defined as a statement made in anger it certainly is and the anger is more than justified. Over the last six months we have watched as LL has destroyed the player base which sl businesses depend on with one horrible decision after another, and as of last Wednesday took this destruction to a level where it is totally intolerable and we no longer see any hope of sl recovering. Quite frankly it would be foolish to continue to feed money to LL for the poor service we are getting and they deserve to fail. So if this is a rant lets make the most of it.

Now from what I have seen this weekend thing have gotten a lot worse, and if it continues through this week I am not sure SL can be saved or even should be. The point i was making and will make again is this situation is out of control and the player base has lost most of its trust in the hierarchy of LL to either fix or even care if these problems exist. As for the problems, part of it is bandwidth, part is a massive amount of lag between commands in their execution which seems to be causing massive amounts of crashes. Also there is a serious problem with traffic counting, which may again stem from command lag. The core of the problem is likely in the physics engine which apparently is not able to balance the demands of a mesh and a prim based building system, if in fact this can be reconciled or balanced at all. Some of this may eventally be solved by Project Shinings hardware upgrades but by then I suspect few players will be around to enjoy it. What is needed not is first a clear statement of purpose from the SL leadership acknowledging that these problems exists, two a promise and a plan to fix them rapidly, and three a promise of accountability for those who through either ignorance or wrongheadedness perpetuate it. None of this will probably take place and this week I expect to see a series of restarts that perpetuate the problem or in fact make it worse, at which time even more people will loose patience with LL close their businesses, leave and not come back.

Fact last time I looked Phil was Chairman of the Board.

Let us hope that this weeks restarts represent an honest attempt on LLs part to rectify the problems caused by last weeks fiasco, as anything less, or another mad adventure that only makes these problems worse may well drive massive numbrs of people from sl and leave it without the support it needs to survive, if in fact that support is there now. While wishing for for impossible of improbably things, it would also be good if the leaders of LL (Phil Linen, the CEO, and Head of Engineering would speak to this problem and assure us that LL is in fact committed to fixing it and not continuing with more innovations that will only make things worse. There is a crying need for openess, leadership, and transparency on LLs part at this time and we have so far seen little more than silences, which makes people wonder if the leadership of LL is aware of this problem or if they are care.

Giveyou two quick reads on three sims Turnip, zerango and Osbourne Walk. Turnip went form a memory allocation of about 24 to 25 meg to one of about 125, zerango jumpped from around 17 to 70 and Osbourne Walk went form 15 to about 50. Thus it appears that eitehr Pathfinding is taking extra memory resources, or creating an even larger memory leak. What is certain though is that it seems to be using more memory resources, and this may account for the increasing number of crashes we are seeing. What remains to be see is if this additional memory usuage lead so any improvement in overall service quality, but given what we have seen so far I am not optimistic. Also please note that the viewer that measured this does not use mesh and that you can count on mesh placing an even larger demand on memory. .

I just encountered an interesting discovery that needs to be addressed by someone in an Official capacity. It seems it is not taking around five time the memory to run an avi that it took prior to the restarts. Does this mean LL is going to be upping the base requirments for or computers or will something be done to bring this more under control. This though is solid proof that Pathfinding, contray to LL's previous statement is indeed using far more resources and that at least some of these are but passed off to our computers. This raises some interesting questions about what else is going on with Pathfinding that LL is not telling us.

Oskar when will the restarts be over it seems like these are taking much longer than usual?

They have been contacted, this post is to make people aware of what is going on so they do not think they are the only victim of this problem.

Or as happened with boats when Havok was introduced, make new ones that work and force people to pay to replace them. Personally, I hope builder take the honable path and create a mechanism to correct this problem, but given LL's history on these type matters don't expect much help from them or even an acknowlegment that there is a problem.