Jump to content

Is LL putting more sims on fewer servers?


animats
 Share

You are about to reply to a thread that has been inactive for 1746 days.

Please take a moment to consider if this thread is worth bumping.

Recommended Posts

I think that there are some confused minds around here, certainly more confused than my own.

When I go to the doctor, I tell them what I feel, what symptoms I experience. It's up to the doctor to find out what is wrong with me and eventually prescribe a cure.
The very same applies to SL. I don't have to demonstrate anything, it's not my job as a paying user trying to go above my abilities to find out what the problem is — that is the service provider's job. All I have to do is let them know about the symptoms, and that's it.

What happened with the rental at Warrumbungles was something like going to the doctor and being told it would pass if I''d return home peacefully, while in fact I am suffering from a serious disease. And that went for about a whole year, so it means the simulator had been hosted in several different machines during that period, supposedly.

[EDIT] Now, how many more cases like Warrumbungles are there that we don't know of? Yes, I may be speculating, but the fact is that there are other regions with identical issues... Unless it's a new feature and it's now how things are supposed to work.

Edited by MBeatrix
note added
  • Like 1
Link to comment
Share on other sites

@Qie Niangao

With regard to the apparent changes at Woods of Heaven:

1)  The onset was sudden.  It was more than a year ago but I cannot be more precise.  The situation does not get significantly worse in the days following restarts, nor does it improve immediately after a restart.

2) The script number had not changed, the performance drop was after a restart.   Nothing else had changed, and we were still on Main Server channel.

3) So far as I can tell sleep time tracked and still does track spare time.  Script time has not noticeably increased since.

4) Script numbers have only increased slightly since then, as have script events.

5) I have not attempted to restore function by removing old scripts - since the change in performance of scripts was sudden with no other changes, I see no value in doing that.

and 6) There are no other significant performance reductions in the region.  The few scripted objects that we have seem unaffected.  There have been and remain significant changes to scripted event operations concomitant wit the deterioration in % script run, though these are not massive.

I raised the matter with Live Chat at the time it first occurred and I was told a restart would cure it.  When it did not I returned to Live chat who offered no further remedies other than I should reduce my script number (which was not particularly high anyway.  I did not nor could I do this.

Upon raising a JIRA, it was summarily closed with the comment that I should contact support (?).  Frankly I could not see the value in doing that.

Now I reiterate that all this was gleaned from the in-viewer Statistics panel and like other residents I think it is perfectly valid to rely on this metric to describe performance changes.  It was designed by LL developers for precisely this function and is the best tool for the purpose.

Edited by Aishagain
clarification
  • Like 2
Link to comment
Share on other sites

4 hours ago, Qie Niangao said:

To start, we might decide we're only interested in low Scripts Run percentage. There are lots of other problems we might look at, but let's boil one ocean at a time.

The problem I experienced came out of the blue, I logged in to find an issue. It's so long since I last used the stats bar I was confused by the new arrangement, I previously saw a time in seconds being taken up by scripts plus some other figures for events, and I didn't know how to interpret the scripts run time percentage until after I had replied to a post in one of these forums. The problem persisted through what I think was a rolling restart (I wasn't online during that time), but then two days later it was gone. I had been discussing with the other person I share the parcel with the possibility we might need to look for another parcel because I assumed the land owner had done something to their whole region. The problem vanished just as abruptly as it had manifested itself and hasn't re-occurred.

I can't reproduce it, or even report it unless it happens again. What I am considering is whether this is in the same category of fault that has lead to TP-fails and sim-border crossing problems, happening too rarely to be easily investigated?

What would be useful to know is what is the optimal figure for scripts run percentage, and what variation can be expected to it in normal conditions, ie 100% +- 5% (and yes, last time  I looked I had 102.5%)

Edited by Profaitchikenz Haiku
  • Like 3
Link to comment
Share on other sites

2 hours ago, Aishagain said:

@Qie Niangao

With regard to the apparent changes at Woods of Heaven:

1)  The onset was sudden.  It was more than a year ago but I cannot be more precise.  The situation does not get significantly worse in the days following restarts, nor does it improve immediately after a restart.

Scripts tend to have a hard cut off point when viewed from a region wide perspective. Everything is just fine and you have a little spare overhead, right up to it suddenly not being fine and you having none. Once you pass the point of performance degradation kicking, then it gradually shift downwards as more and more is added. Once you are in this part of the curve, finding problem scripts is impossible as their high usage / spikes get smeared into the 100% average.

Its like trying to pick out the one loud sound in an audio file but your volume is way to high and the outputs are already clipping badly.

The only option is to rip stuff up till you get back to running stably with some clearance and then evaluating everything.

2 hours ago, Aishagain said:

5) I have not attempted to restore function by removing old scripts - since the change in performance of scripts was sudden with no other changes, I see no value in doing that.

You have a very limited window onto the regions performance, yes you get a lot of numbers, but you are still fairly blind. So either:

  • Linden Lab is done the dirty on you and us all for the evil/lulz/cash/cake
  • The core service has shifted slightly, as is to be expected will with server and OS changes.
  • Your region usage has not been as static as you believe it has.

My money is on a combination of the latter two. Your usage has crept up slowly over time and incidental core changes have moved the ceiling a little. Given that the service provided by LL is what it is on the day, and there is nothing you can do to influence that, they are not being evil. OS, server, security changes have to happen and that can move the goalposts.

I'm not sure what you hope interactivity will accomplish.

2 hours ago, Aishagain said:

I raised the matter with Live Chat at the time it first occurred and I was told a restart would cure it.  When it did not I returned to Live chat who offered no further remedies other than I should reduce my script number (which was not particularly high anyway.  I did not nor could I do this.

<humorous light sarcasm / gentle ribbing>

: Lets verify your supply is fine, ok we switched it off and on again.

It still hurts when I lick the plugs sockets.

: Do that less?

: No.

2 hours ago, Aishagain said:

Upon raising a JIRA, it was summarily closed with the comment that I should contact support (?).  Frankly I could not see the value in doing that.

<humorous light sarcasm / gentle ribbing intensifies>

I spoke to postal service support about why my packages are all different shapes, they didn't seem to take me seriously.

: I asked the milk man, he said he couldn't help me, suggested I talk to postal service support.

: It's a conspiracy

2 hours ago, Aishagain said:

Now I reiterate that all this was gleaned from the in-viewer Statistics panel and like other residents I think it is perfectly valid to rely on this metric to describe performance changes.  It was designed by LL developers for precisely this function and is the best tool for the purpose.

The stats floater is a bit like sticking your finger in the wind to judge the direction. Your region is a stack of interdependent complex systems that can not be adequately communicated in a few numbers. This is made worse by the fact you are not recording and graphing the numbers you do have, you're watching then change in real time, with your eyes and hoping to see a pattern (science is only science when your write the numbers down). The best you can hope for is It being enough information top tell you something needs attention, sometimes. Don't expect it to hang a lantern on the problem or that you only have one problem. You might just have 3 things that on there own are fine, but all together on the same region equals a 3 .. which is obviously bad.

  • Like 1
Link to comment
Share on other sites

4 hours ago, MBeatrix said:

When I go to the doctor, I tell them what I feel, what symptoms I experience. It's up to the doctor to find out what is wrong with me and eventually prescribe a cure.

That's rarely how doctors work.

You go and list the 4 things you are suffering with. The doctor will either symptomatically attempt to relieve those things one by one, match the pattern of symptom occurrence to a known single cause, or (most likely) when the previous 2 fail send you for more specific testing and to see someone else.

This procedure will be repeated by every doctor and specialist you see, till eventually someone will identify the actual root cause and do something about it, by which point it's almost trial and error ordered by assessment cost.

This is why if you google any medical symptom, it's always cancer.

  • Like 1
Link to comment
Share on other sites

Thing is, there's damned near nothing useful a script can measure about sim script performance. As I mentioned before, there are two functions, one measuring sim FPS and the other time dilation, which should be simply the inverse of each other (unless they use different sampling intervals or something). Admittedly, I have never actually used either of these functions for anything beyond a hovertext readout of sim health.

Critically, a script has no direct way of measuring the percentage of scripts run per frame. I seem to recall that somebody (maybe @animats ?) had a script using llGetEnv("frame_number") called in rapid succession to estimate a likely range for that percentage. That's one wheel I really don't want to reinvent, especially if somebody already has a version that's less dreadful than any that come to mind in terms of contaminating the very metric it's trying to collect.

Or is there something else more useful we could actually measure?

And then there's the question of where to measure it. This isn't the first time I've wished that ages ago, Elanthius Flagstaff had simply ignored the rabid anti-landbaron criticism that eventually led him to abandon his project of owning one 16sq.m. microparcel in each Mainland region; those would be ideal for collecting the kind of data that could actually help now -- although about a half dozen more script-collectable metrics would sure come in handy, too. (Presumably Lindens can collect such script performance data from every sim; I wonder if they already do.)

Link to comment
Share on other sites

Just now, Qie Niangao said:

Thing is, there's damned near nothing useful a script can measure about sim script performance. As I mentioned before, there are two functions, one measuring sim FPS and the other time dilation, which should be simply the inverse of each other (unless they use different sampling intervals or something).

Which is why I keep saying, skip these numbers and all similar numbers.

Benchmark stuff that matters.

How much time does it take to do a pre defined amount of math, rezzing, physics, object changes, object to object inventory transfers, object inventory indexes, call backs etc etc.

Design some tests that measure stuff we care about, and record those numbers. Share this script and your resulting numbers widely.

Is my region doing badly today? How does this region release compare to last weeks / some other channel? etc etc etc

Test it.

 

Link to comment
Share on other sites

@CoffeeDujour

You clearly read my posts but you seem to show little or no understanding of what I write.  You display the same wilful misunderstanding of Sharie, Qie and Beatrix so I am led to the conclusion that while you DO understand them you are purposefully adopting a combative and opposing viewpoint.  To some that might constitute trolling.

Y'all have a nice day now.

Edited by Aishagain
additional sarcasm
  • Thanks 1
  • Haha 1
Link to comment
Share on other sites

About 18 months ago, something changed overnight. Regions (not just mine, it was all over SL) that used to operate in the 90's for script run time dropped to the 40's and 50's. That's what happened. Lag spiked everywhere, avatars TPing in and out had a magnified impact on region performance. I filed quite a few tickets at the time for my own regions with absolutely ZERO resolution. LL wasn't talking about it then, and they aren't talking about it now. Over the years, I had occasionally run some of the benchmarks available at http://wiki.secondlife.com/wiki/Mono#Testing and from memory noticed that the benchmark was running at half speed.

Some people here have suggested that since I didn't keep a record, that I should just go away, and that LL can't take any action without "my" benchmarks. Really? Does LL pay me to do take benchmarks and log them? No. Does the fact that I don't maintain benchmark logs mean that as a PAYING customer, I have no right to speak up when things go to heck? No! Do you really truly believe that LL runs THOUSANDS of servers and doesn't keep their own benchmarks or monitor server performance? SL is not run on some server in someone's basement, it's a large intricate network - and without benchmarks and performance data, it would be IMPOSSIBLE for capacity planning to happen, or even to know when things go wrong! They can get performance data on every part of server and network operations at a much deeper level than a silly Mono benchmark I run. Do you really think that they are not completely aware what happened? LL employs some very talented people who keep things running. Trust me, they have the performance data and LL staff has stated that they have it. In fact, it would be grossly incompetent for the infrastructure team NOT to maintain performance data (I'd fire them.) In short, LL is obviously fully aware of what changes they made that caused region performance to drop because the talented people they employ to run their server operations are just not that incompetent to NOT know! LL would not have survived this long if staff were that incompetent. So lets give them the credit they deserve and realize that they are talented and care about the company they work for and the product under their sphere of influence.

As end users, all we REALLY have is perception and the silly region stats floater that only gives us a pinhole view into how things are running. Even benchmarks we run are suspect as the physical server is shared - performance is going to be affected by other regions on the same server, or someone TPing in and out, during the test etc. Even people in neighboring regions can affect performance based on draw distances and such. Only when changes are large like what happened ~18 months ago does it make it obvious that something changed for the worse. But again - claims that LL can't do anything without MY benchmarks are just - ignorant. It shows a total lack of understanding on how large services operate. Use your common sense folks!

I responded to this thread because during the recent TP crashing issue, LL rolled a release that was so incredibly bad that my script run time couldn't get past 10 and menu popups that should be nearly instant were taking nearly 10 seconds. Fortunately they did correct this with a subsequent rolling restart. That release clearly should have failed QA, but I suspect that it performed poorly because of additional logging they needed to deal with the TP issue.

But let's get back to the main topic. Why are some people so incredibly resistant to call for LL to resolve server performance issues? What do you have to gain? Why are you refusing to believe that LL did something (which they clearly did) which caused these performance issues that were not here 18 months ago? Do you hate SL that much that you want it to die as people get so frustrated they dump their land again like the mass exodus that happened when the Adult changes were rolled out? Maybe you were focused on other things (sansar?) and didn't notice the performance drop. Why is it impossible to believe that someone else did? How does increasing server side lag and other negative problems that are happening benefit you? Why is it so hard to believe that it's possible LL management was trying to save money and ended up cutting corners in a bad way? Have you ever met a CFO that wasn't constantly looking for cost savings? As a private company that hasn't received an influx of (investment) cash in years (according to the financial data I've been able to find, which is slim for private companies) I'm sure there have been cost saving measures all over the place. This is where I suspect the issue resides. It's my opinion based on the evidence I've found.

I'm calling for LL to open up and address this issue. Enough of the silent treatment.

  • Thanks 1
  • Haha 1
Link to comment
Share on other sites

38 minutes ago, Qie Niangao said:

Critically, a script has no direct way of measuring the percentage of scripts run per frame. I seem to recall that somebody (maybe @animats ?) had a script using llGetEnv("frame_number") called in rapid succession to estimate a likely range for that percentage.

No, that wasn't me. I have a little "sim lag reporter" on Marketplace which checks how long it takes for a script event to get serviced after a timer event. That's to catch intermittent overload, which causes vehicle problems - suddenly, you can't steer or brake.

I do use llGetSimStats(SIM_STAT_PCT_CHARS_STEPPED) to find out if pathfinding is running slow. It returns a value in percent (0..100). Below 25, it's time to set the pathfinding option [CHARACTER_ACCOUNT_FOR_SKIPPED_FRAMES, FALSE], which makes pathfinding slower but more accurate. Otherwise, pathfinding characters tend to go out of bounds when the sim is overloaded. They're not getting the course corrections they need.

llGetSimStats, strangely, has only that one value you can request. More values would be useful.

Kelly Shergood does some sim load monitoring in her helicopters. If the scripts are running too slow, the pilot gets an autopilot disconnect due to "turbulence", and has to take over manually.

Not many scripts need this. It's mostly needed by scripts controlling motion in real-time. Those scripts need to know that the control system isn't working right.

  • Like 1
  • Thanks 1
Link to comment
Share on other sites

 

27 minutes ago, Aishagain said:

@CoffeeDujour

You clearly read my posts but you seem to show little or no understanding of what I write.  You display the same wilful misunderstanding of Qie and Beatrix so I am led to the conclusion that while you DO understand them you are purposefully adopting a combative and opposing viewpoint.  To some that might constitute trolling.

Y'all have a nice day now.

It's the same illness that griefers have, they get enjoyment out of attempting to make other people miserable.

  • Haha 2
Link to comment
Share on other sites

1 hour ago, CoffeeDujour said:

Which is why I keep saying, skip these numbers and all similar numbers.

Benchmark stuff that matters.

How much time does it take to do a pre defined amount of math, rezzing, physics, object changes, object to object inventory transfers, object inventory indexes, call backs etc etc.

Design some tests that measure stuff we care about, and record those numbers. Share this script and your resulting numbers widely.

Is my region doing badly today? How does this region release compare to last weeks / some other channel? etc etc etc

Test it.

 

Oh. That's not even remotely what I thought you were saying. Anyway, yeah, with enough volume, that would be nice data to have, historically, and it could be useful to start collect it now in hopes that over time it might reveal a pattern in what particular operations are slowing down, and when. 

I mention "with enough volume" because the performance of individual scripts is (and always has been) notoriously variable even on an otherwise empty sim. Try collecting a repeatable measure of memory use, even, which should be completely predictable but bounces wildly even using profiling; execution times are even noisier. But that's what statistics are for, so yeah: given enough samples, this could reveal something useful.

 

  • Like 1
Link to comment
Share on other sites

1 hour ago, Sharie Criss said:

I responded to this thread because during the recent TP crashing issue, LL rolled a release that was so incredibly bad that my script run time couldn't get past 10 and menu popups that should be nearly instant were taking nearly 10 seconds. Fortunately they did correct this with a subsequent rolling restart. That release clearly should have failed QA, but I suspect that it performed poorly because of additional logging they needed to deal with the TP issue.

Your region is still running the exact same "incredibly bad" server version that was rolled last week.

  • Like 1
Link to comment
Share on other sites

Menu dialog delays are one of the tricky things to diagnose, the temptation is to say that it is script lag, but because the dialogs use chat channels, chat-lag in the region or parcel could be the reason. I see menu delays or sometimes even refusal to show the pop up at odd intervals, but the problem is almost always cured by my logging out and relogging. The delays or malfunction could be caused by two possible events not functioning correctly, the touch event when the dialog is first requested, and the listen event when the button is chatted over the channel.

One thought has occurred to me, about events: there was apparently a change to LL's server code a few weeks back that did something about polling for events? It broke Singularity, and I had to go for a recent nightly build after Whirly pointed the way, but I realise now the episode I had in my parcel was before the Singularity release, but I donlt know if it was after the LL change to the server code that prompted this. Does anybody have a date so I can look back and check if the train episode was close to this?

  • Like 2
Link to comment
Share on other sites

 

On 5/1/2019 at 8:19 PM, MBeatrix said:

as user I'm only interested in functionality — and that is severely affected

When things ran pretty well ok, apart from the odd glitch here and there, I never took the slightest interest in the technicalities of how this all works. But now that it clearly isn't running so well, I'm getting curious. I've been following this thread very carefully, and I think I'm getting the drift of the arguments being put forward. Many of you guys are way over my head though, and thus it is is likely to remain!

But, like Shari, I never bothered to look at any stats in the past because there wasn't any need to - I don't photograph normal roads either!

But also, I'm a user too, and when it takes 20 seconds for my alpha hud to appear, I can't help thinking that something is seriously wrong with the service I should be getting.

This is what the region I rent a parcel on looked like last night:-

SimStats.thumb.png.73c82a547055cc2039812d7acd273acd.png

 

I can now realise that there are some very nasty figures in there. It would take me days to try to find out where all those scripts were, using Area Search.

Do I nudge the region owner into re-starting, raise a support ticket, or head off to college for a while to learn what the hell I'm talking about?

(If anyone could explain what the important figures should look like, I'd be most grateful!

  • Like 1
Link to comment
Share on other sites

4 minutes ago, Odaks said:

... when it takes 20 seconds for my alpha hud to appear, I can't help thinking that something is seriously wrong with the service I should be getting.

... It would take me days to try to find out where all those scripts were, using Area Search.

Do I nudge the region owner into re-starting, raise a support ticket, or head off to college for a while to learn what the hell I'm talking about?

(If anyone could explain what the important figures should look like, I'd be most grateful!

The mesh avatar alpha HUD tends to be the "canary in the coal mine":  it makes very evident the problem of low "Scripts Run" percentage. That number, 13.684 % in that Statistics bar, really needs to be as close to 100 % as possible.

One may wonder why such a small percentage of scripts are getting a chance to run each "frame" of the simulation. The other numbers can help answer some questions about that.

  1. Are scripts overall getting a fair shot of running, or are higher priority, non-script tasks consuming more than the normal amount of time? In this case, it's pretty normal. Both Physics Time and Simulation Time are a little higher than average, but still scripts are getting about two-thirds of the frame (14.629 of 22.246 milliseconds) so the non-script load probably isn't a big problem.
  2. If scripts are getting their fair share but they're still lagging, maybe there are simply too many Active Scripts (about 15,569), and maybe they're trying to process too many events (about 1400 per second). Both of those numbers are very high on this sim. There are simply way, way too many scripts.
    (And notice that there are only two avatars, aka "agents", in the region, so it's likely that most of those way, way too many scripts are free-standing on the region, not attached to avatars.)
    (Also, while we're idly noticing things, there are a tremendous number of Objects rezzed in the region -- over 27,000. That wasn't even possible just a couple years ago, and if a region is using that many objects, they better be mostly unscripted. In this sim, actually, they mostly are unscripted -- there are only about 2381 "active" objects -- but apparently some of the scripted ones have a lot of scripts.)
  3. Sometimes it's not this easy to figure out why a region is having such a script lag problem. Sometimes the region is misbehaving in some obscure way that might be fixed with a restart. For example, we've seen a sim or two that seem to have perfectly normal script counts and events per second, and little competition from any non-script load, but still only a handful of scripts are able to run each frame; upon closer inspection, the "Time Details" / "Sleep Time" is sometimes accounting for most of the Script Time. That's certainly not the case here, and even when it is the case, I don't think we know what causes it or what will reliably fix it, but when there's weird, inexplicable happenings is when a restart is worth a try. Unfortunately, that's not the case for your sim.

And unfortunately, there's not a lot you can do as a parcel renter. If your landlord owns the entire sim (maybe it's an Estate sim), then they might be able to track down which tenant has so many scripts on their parcel and get that tenant to reduce their load. That, however, assumes a landlord that cares about the quality of what they're renting, not merely motivated to keep vacancies low. I mean, you may leave, creating a vacancy, but if somebody has like a quarter-sim packed full of heavily scripted breedables, their rent may outweigh your vacancy.

  • Like 1
  • Thanks 1
Link to comment
Share on other sites

21 minutes ago, Qie Niangao said:

(Also, while we're idly noticing things, there are a tremendous number of Objects rezzed in the region -- over 27,000. That wasn't even possible just a couple years ago, and if a region is using that many objects, they better be mostly unscripted. In this sim, actually, they mostly are unscripted -- there are only about 2381 "active" objects -- but apparently some of the scripted ones have a lot of scripts.)

So is that "region" a single 256 square sim? Only, even with the object bonus multiplier, I can't see how you could get up to that figure.

 

ETA OK, just found some private sims can upgrade to a 30,000 limit.

Edited by Profaitchikenz Haiku
research
Link to comment
Share on other sites

Ok, by coincidence a friend just asked me to go see why something I wrote for them was taking a long time. When I got there and looked at the stats I saw scripts run at 21%, total objects 13018, active objects 1495, active scripts 8124, script events 1132.76, frame time 23.7, script time 16.046.

So the number of objects isn't likely to be the cause. But I see Qie's point about the number of scripts per object, there is an average of around 6.

ETA After spending 20 minutes there, and seeing figures that were shown in the LL V3 viewer, Singularity Test 7509, Catznip and Firestorm, the figures generally agreed, although there seemed to be a fluctuation of up to 10% in the scripts run time figure such that by the time one person said "22.5%" the figure had changed to perhaps 29%. There were only five avatars max present in the region, most of the time just the two of us, and that didn't seem to affect the figures.  

The figures for time dilation, Sim and Physics FPS that I have traditionally looked at when seeing issues were all good (0.959, 45, 45). 

Edited by Profaitchikenz Haiku
Fresh information
Link to comment
Share on other sites

Just in passing, the Statistics bar reports roughly the same performance characteristic using several different numbers to reflect the length of a simulation frame. Ideally, the sim should run 45 frames per second, so the duration of a frame -- the "Frame Time" -- should be 1 sec / 45 frames = 22.222... milliseconds. The ratio of that ideal duration to the measured frame time is (approximately) the "Time Dilation", the degree to which the sim overall is keeping up with real time. When any of those numbers go bad, they'll all go bad, and that means the sim has a worse problem than script lag such as Physics or Network lag, which are pretty rare these days and indicate a sim in big trouble, almost never related to scripts.

Link to comment
Share on other sites

3 hours ago, Odaks said:

When things ran pretty well ok, apart from the odd glitch here and there, I never took the slightest interest in the technicalities of how this all works. But now that it clearly isn't running so well, I'm getting curious. [...]

I am curious too, but not enough to spend my time in-world doing what Lindens are supposed to do. Also, I'm sure that they gather more and better data than any people from forums.
Knowing technical details about how SL works wasn't the reason I joined it in the first place. Learning a bit of it is just a consequence of being a user.
If I wanted to know all the deep stuff, I'd rather spend my money on a programming course or something, and certainly wouldn't be in SL. This is what I mean when I write that "I'm only interested in functionality" from the basic user's point of view.
I have far more interesting things to do in SL than spending the time I have available for it "getting my hands dirty".
If it will ever gets buggered to the point that the things I do aren't working, I'll simply stop paying it.

Edited by MBeatrix
correction
  • Like 1
  • Thanks 1
Link to comment
Share on other sites

5 hours ago, Qie Niangao said:

The mesh avatar alpha HUD tends to be the "canary in the coal mine":

I just love that description! I'll take great care with ignition sources next time I use it..

But many thanks for the very helpful explanation. Bit by bit, I'm beginning to understand things  better.

Link to comment
Share on other sites

You are about to reply to a thread that has been inactive for 1746 days.

Please take a moment to consider if this thread is worth bumping.

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
 Share

×
×
  • Create New...