JoyofRLC Acker

Wetware = not a bot?

Is it my imagination or have the restarts been starting later the last few Tuesdays (and certainly finishing later)? While restarts have obvious benefits, the process is quite disruptive for any activity that traverses multiple sims on set routes (as opposed to just lurking in one sim contemplating one's navel, or , better, someone else's * cough). Sailing, flying, driving etc etc etc. We have had a long standing event in Blake Sea at 11 am on Tuesdays (for over a decade) - on normal schedule restarts are finished and all is good, but when they are late, its almost impossible. Effectively Blake Sea is 'down' for several hours.

This is def going slow. And the few sims at a time just prolongs the agony. Out of curiosity - is the the best time in terms of least number of users online to have these extended outages?

This is really interesting - just discovered the thread. Do keep up the pressure on LL. Im not technical so cant contribute to the analysis. But I would like to add my observations about SL over the last two or three weeks, form the perspective of sailing (racing). Primarily in Blake Sea. So we are talking about vehicles with quite "large" and slightly complex scripts. First the TP / sim border issue - that definitely got better; but seems to have deteriorated again somewhat but not to the depths of the worst days a few months ago. I cant speak for TP much as I dont use it a lot, other than under script control, but the rate of 'falling off' the boat at sim borders is definitely up again. Any one know why that might be? Second, we have been seeing increased 'lag' generally, as evidenced by sluggish response to controls. I also have a sense that the sim border processing delay has increased but I dont know how to measure that objectively (I just see it as my viewer is set to 'stop' rather than 'predict' for sim borders).

I'm no expert, but have been around a while. I have always thought that there was something pretty wonky about chat text handling. One of my first realisations this place had a pretty broken* architecture was being told "hands off the keyboard" when sailing across a sim border (this was almost a decade ago). * broken isnt quite the right word as it implies a prior state of un-brokenness. cough.

That reminds me .... do we have any data on the extent to which the number of vehicles attempting a particular sim border crossing at the same time has an impact? Either for disconnect or for agent detach problems?

Theresa, I sort of follow where you are coming from, except that ... I think a lot of people have given up complaining as they consider it normal service. I know that we certainly don't file tickets every time some one falls off a boat. If we did, they'd have to buy new servers to manage the JIRA queues! We tend to file tickets or JIRA bug for sims that are obviously broken, or maybe if we think things are especially bad, eg a couple of months ago. There's no doubt that individual set up (location, ISP, W/LAN, computer, SL settings) all play a part. I'm certainly of the view that things are better (as of Friday, I haven't been in-world over the weekend) but certainly not fixed, which I would take to be as things were two months ago. Specifically prior to that, getting logged out was, for me a rare event. Falling off the boat, maybe once every third or fourth race. Then during the bad spell it was Logout about every 10 - 30 sims of sailing, or just standing on my dock editing scripts whatever. Now its about 50% of races for logout, and at least once per race for falling off so not as good as it used to be. NO changes in my ISP (Broadband 11ms ping and 50+Mbps local; variable to Pgoenix of course) or network. A few weeks before the trouble started I upgraded PC (Dell 8930 w 16 GB, NV GTX 1060). So the any adverse changes from say 3 months (ignoring the peak disconnect time period) ago are down to SL. Sorry for so many words but that is what I mean when I say things aren't fixed (but are better than during the peak recent problem).

I think a definition of "fix" also has to include clear description, with evidence or code analysis, of the root cause and the changes made to resolve it. I'm reminded of the first lectures I attended on structured programming (ok, Im dating myself) and the comment that most "testing" can only prove that a program is incorrect, and can never prove that it is correct. They have built something beyond their comprehension. My suspicion is that they have tinkered with the 'freeze' at sim borders. For me its def a few seconds longer, but also its now a genuine freeze with motion actually stopped, ie the boats position at the end of the freeze is right where it was on the border, not skipped ahead into the next sim. Friday & Monday I had more or less clear sailing. Wednesday & Thursday I was getting logged out again. In fairness one session was where I was inadvertently racing with a DD of about 850m (I did wonder why the fan was on but didnt make the correct deduction alas). So, Id say better thanks to the band-aid, whatever it is exactly, but def NOT fixed.

Yesterday and Friday Id say that sailing in Blake was a lot better, if not perfect. However there is one thing that I think has changed on sim crossings. (I have the viewer prediction / extrapolation turned off). Previously if you froze on the border for more than say a couple of seconds, your vehicle would jump forward once visible motion resumed. In other words you would would be roughly where you would have been without any extended freeze. Now the freeze is consistently longer - 6 - 10 seconds Id say (vs 2 - 8 ) and the vehicle does NOT jump forward to where it should have been. In other words the vehicle "really" stops. Is anyone else observing this sort of thing?

Ok, now Im confused. Could someone please classify what the various problems discussed here are. Also please conform or correct my understanding: - vehicle sim crossings have always had a number of problems and the frequency has varied over time. these are include (but theres probably more Im overlooking): submarining / flying falling off, agent and vehicle appear to be in different sims (not always adjacent) - this sometimes follows apparently normal travel through 1 or 2 sims but the vehicle is non-responsive cam angle /focus goes wonky getting logged out (rare until the last few weeks) - TP can 'stall' for a number of reasons; and that may follow with a disconnect - this was not too common until the last few weeks My understanding is that a disconnect during a vehicle crossing is thought to be strongly related to the TP disconnect problem. For me the vehicle crossing problem was far more critical as I wasnt having any unusual problems with TP. I have also had it said to me, and not just recently, that the other region crossing problems are caused by 'bad server hand offs' which sounds an awful like what is being said specifically about the TP problem. LL has done a server release that will "REDUCE incidence of TP disconnects" (my emphasis). They do not say what they have done, nor that the fix will ELIMINATE the disconnects. They don't even say how much of a reduction there should be. The big concern obviously is that if they haven't gotten to the root cause, the band-aid will fall off at some point. Hopefully the next Server User Group will shed more light on the situation. Personally I'm still nervous.

One or two per hour is a big improvement on last week. Many of the people I race (and myself) with would get a disconnect after 15 - 30 sims. Another data point ... we had two races today around noon SLT Blake (and China thru Baltic). 4 boats completed both course. This is a first in many weeks. Regarding the 10 second pause, this is interesting. Today I was racing a boat that is not the best at sim crossings generally (it has other virtues and I was in the mind for a stress test). It always pauses for a variable 4 - 8 "seconds" (using 'one missippi' type counting on good sample of crossings ) on sim borders - today the delays were less variable. And heres the odd thing, today I think the boat did actually stop - there was 'leap ahead' when the freeze ended (I have the viewer extrapolation turned off). My sense is (anecdotal, not hard data) is that something is making things better, maybe by masking the problem, but that its not totally fixed. One other thing that have noticed all along, and its probably just a symptom that when the avi and vehicle get separated (part of the warm up for a disconnect) there is no change event for the detach triggered in the scripts.

As of yesterday, in Blake Sea (all Main Channel) there were still significant issues. Not clear if the situation was any better or not, but clearly not solved.

Indeed. My point really was that LL uses what I call "testing in production", as opposed to a separate test system/environment. One of the many things about this situation that appal me is that LL was apparently unaware of the problem until some JIRAs were posted - and job 1 had to be to update the system to actually get log data on the problem.

RC channels ... to the extent they are seamlessly attached to the main grid and are scattered as it were they are 'production' sims for all practical purposes. In most places you cant sail on "just" RC channel or "just" main channel (Blake Sea aside). Code released to production should be tested.

Sorry I mis-interpreted - thought you were making a distinction but you weren't really. For the sake of clarity, my scripted TP are just about always into different sims (and as I say they are nearly always successful even now). The connection (haha) between regions crossings and TP is that, according to what I was told, a region crossing in fact involves a TP, the agent and the vehicel cross separately and then reattach. But no idea of the credibility of my sources. Until now, region crossing disconnects were VERY rare for me, unlike falling off and cam dislocations which were more common. I wonder if somehow the latter problems have turned into the former. ie the underlying cause was always there, but with some new code somewhere its being handled differently by the servers / viewers, and so a different symptom for what is in fact the same cause. Finally - Heisenberg is alive and well!

Surely they have data on sim crossing failures as well as the TP failures - its the same underlying issue I thought. They dont have to rely on "reports" (but see below) for sim crossings they would have the same server log data. I do not see why you say (well, imply) that a disconnect on a sim crossing is not a solid all or nothing event. It is no less catastrophic than a disconnect on a TP. To be clear, I am talking specifically about the horrendous number of DISCONNECTS on sim crossings that started about a month ago - not the other sim crossing problems that, as you say, have been around in varying degrees for a long time (but are also much increased currently) - like detachment, cam going kaflooey (sorry for the technical term) etc etc - and which may or may not be related. Regarding your reference to "reports" - I infer that you are implying they are less useful that the hard data. I think both may be helpful, the reports may contain hints as to that else is going on, and may yield hints on where to look. An example of the contextual stuff is the apparent fact that TP triggered by scripts are less likely to fail than ones initiated by user action. (Some one else put me on to this which explains something that had puzzled me ... for myself Ive been having very few TP problems at all the last month, but 90% of TPS are initiated by my Nav HUD) whereas I have been getting very frequent disconnects when sailing; and even just standing on my dock chatting or editing scripts.

Can people please confirm that the viewer is not a factor. Almost everything Ive read says this is strictly server side, and the LL notice on this says all viewers are affected. However a sailing friend seems pretty convinced that the SL viewer is more robust in this regard (sailing / sim crossing specifically) than Firestorm. Other than possibly different thresholds for time-out disconnects, I cant see the viewer making a difference if it IS strictly server side. I'm probably being naive but I find it a bit worrying that it is taking the 'crack team' so long to diagnose the problem. (Yes, the fix can be lengthy depending on the problem, but Im surprised that diagnosis is proving so difficult). It suggests to me that there may not be a single cause (other than the basic architectural design and the inherent byzantine nature of region crossings). It does also raise the question of how they test system releases.

Any info on that they released today (Mon 15th). It seemed like it was all channels but the notice was not specific, and nothing on the purpose. There was not any obvious dramatic improvent at our race today. PS: DO we need two categeories, "Regularly Scheduled Maintenance" and "Unscheduled Scheduled Maintenance" or "Suddenly Scheduled Maintenance" sort of like "unknown unknowns "which is what the TP problem seems to be lol.

Its not fundamentally flawed if you are building a shopping mall / nightclub, which is what I suspect the original idea was. This architecture is bonkers for anything like flying or sailing etc. 256m "regions" - come on!

Which does rather beg the question! (Think about it - erorrs are normal....) I do wonder about the resources involved in all those errors. Not so much the writing of the error log (but that builds up - my session legs get to about 3 MB which is a LOT oftext) ... as the processing involved in detecting the error and taking whatever exception code etc.

Speaking of Firestorm logs ... Yes I see a TON of warning messages (that look more like errors to me!). It's absolutely staggering the volume of them. An hour of sailing generates a log file over 1.5 MB (last night , 3.7 MB) which is a LOT of text msgs. An awful lot seem to relate to http issues of one sort or another. Packets our of sync, 404 errors, bad asset type, yadd yadda yadda. cef_log also has stuff like [0412/013705.830:ERROR:cache_util.cc(134)] Unable to move cache folder C:\Users\...\AppData\Local\Firestorm_x64\cef_cache\Cache to C:\Users\...\AppData\Local\Firestorm_x64\cef_cache\old_Cache_000 [0412/013705.830:ERROR:disk_cache.cc(169)] Unable to create cache Is there any sort of info anywhere on which messages actually matter?

More anecdotal stuff. This relates to sailing (sim borders have same issues as TPs). My sense is (based on my own experience) that its worse for race fleets than solo sailing. That is not inconsistent with a latency issue. Also that its much worse in the central part of Blake than either the 'Aegean Channel' (China * heading east) or round Nautilus. Would be good to see the stats on that. Also, while this is strirctly speaking a different issue, Iv noticed an increased number of "falling off" problems (Avi "separates" from boat at sim crossing). This is an old problem but until last couple of weeks I wasnt much affected by it. Interestingly the detachment does NOT trigger a changed event in the boat.

Thanks for the clarification, I had read somewhere that the move to cloud servers was under way. I find the log files almost impossible to read but the blizzard of timeout & not found warnings cant be good. Its at the point where just writing all the error messages must take a fair bit of resource!!

I would not bet against the move to cloud servers being part of the problem. Its hard to see how it could improve latency in server handoffs. (sorry I cant find the irony font). iirc at one point they got some improvement in Blake Sea by co-locating the Blake servers. I guess thats out the window now.

Remember that what they built was a basically a shopping mall. 256 sq m sims and badly kludged logic for moving vehicles across was clearly never meant to be a full world simulation. Its an architecture issue as far as I can see.