Technology-Based Damage to a Legend Sim - and lack of sincere care by "the system"

[ArikTheRed]

 

Starting last Sunday night Innsmouth sim started showing signs of stress... lag, extremely slow rezzing times, etc.  On logging in Monday morning I received notifications from Innsmout visitors, and restarted the sim in the hopes of clearing the issues.  The problems persisted and, come Tuesday, buildings were vanishing and returning, parcel names and details were showing blank, and it was clear that something bigger was wrong.  Again I initiated a sim restart, and received word that rolling restarts were in progress and that LindenLabs was working on issues covering many places across the grid, so I let things carry on.  

 

Come Wednesday morning the lag and spontaneous chaos seemed to have subsided, but many of the buildings on the Innsmouth waterfront were partially or completely missing.  Thus, Wednesday night, when things remained incomplete, I filed a ticket requesting assistance.

 

Thursday was spent in chat and repeated communication with LindenLabs begging, pleading, and yes, irately pressing for Lindens to repair/restore the sim to its pre-disaster state.  Ultimately, LindenLabs has stated that they could not restore anything prior to Monday evening, when the disaster had begun to take on full damage-causing and instability.

 

Now, I am left with a significant portion of this much loved, highly acclaimed sim in ruin, with irreplaceable buildings gone.

 

That LindenLabs cannot restore damaged sims to "pre-disaster" state seems extremely suspicious.  Many people have put time, money and effort in many ways to keep Innsmouth covered financially, highly visible and in place for all SL to enjoy for the work of art that it is.

 

It is very disturbing to think that LindenLabs cannot/will not take responsibility and provide some sort of recourse, and not force the restoration efforts to be done by "yours truly," when the damage was done by LL.

 

Is there no reasonable, customer-considered option available on behalf of the people that receive nearly $300 US a month for, for the most part, doing little more than housing a piece of hardware and running occasional routine software maintenance routines?

 

A little background about Innsmouth... 

 

Innsmouth sim was created August 20, 2009, H.P. Lovecraft's Birthday, as a tribute to the late, great "Granddaddy of Horror and Weird Fiction."  For years it has been a popular sim for numerous reasons - as a work of artistic expression and originality, as a tribute to a famous author, as a place to roleplay, host various events and activities, and for many just somewhere to go and sit and absorb the sights and sounds of a classic "New England fishing village gone to rot."

In September, 2014, the creator/owner of Innsmouth announced a very hard, painful decision - the need to give up the sim for personal reasons.  As a longtime dweller and overseer of Innsmouth, I took it on myself to hold a campaign to save Innsmouth, with hundreds of Innsmouth fans coming out of nowhere to donate a phenomenal sum of money to purchase and preserve this highly loved SecondLife Legend.  Since then Innsmouth has been almost completely funded by donors, renters, pledge keepers, and even an Etsy store.

[wherorangi]

am sorry to hear that your sim has been disastered. That would be pretty awful

if LL have said they cant restore it back beyond the last restore point saved then I dunno what else they can do

for sure could say that LL should have more than one restore point saved for every sim on the grid, but if they havent then am not sure there is anything to be done other than rebuild it 

[Theresa Tennyson]

---

wherorangi wrote:

am sorry to hear that your sim has been disastered. That would be pretty awful

if LL have said they cant restore it back beyond the last restore point saved then I dunno what else they can do

for sure could say that LL should have more than one restore point saved for every sim on the grid, but if they havent then am not sure there is anything to be done other than rebuild it 

---

The Lab takes and keeps multiple restore points per day for every region, but they can only keep them a certain amount of days. The intention is that if someone sees a problem they should request a rollback as soon as possible after noticing it. It sounds like your region was griefed - one of the reasons for the Tuesday server roll was for a "security" issue. The restart may have neutralized whatever was causing active problems.

[avalonalpha]

To Ms. Tennyson, I understand that LL may have taken steps to portect against griefers.  It is summer, and a lot of people from teenagers to grayhairs that never matured past teenage are out doing all sorts of mischief.   However, as much as Linden Labs is focusing on it's tech side, it lives and dies on the fact people come here to make COMMUNITIES.  People stay here because they have built somethign here, because they have kept something going that they can care about.  I realize Innsmouth is but one among a seas of sims here, but if you were to talk to sim owners in every corner of the globe, from every culture, the one thign you will hear in common is that they would not want LInden to be callous when a bunch of griefers destory years of THEIR HARD WORK.  

Look at the so-called "AAA" Videogame industry.  For every game that makes the news for making tens of Millions, there are many that are flops.  Every Video gamer can think of at least one game that took years of time, lots of money, teams of trained professionals, especially artists, and for all the hype and care and genius, it flopped.   The same goes for Movies and TV, even more so for them, as everybody knows that one show that they cannot believe got cancelled.  Why am I takign you along this route?, it is because LL is the benficary of literally thousands of creative minds and hard workers, people like Arik that rescued the Innsmouth sim from near ruin, and as a result kept an asset that not only made LL's customers happy, but also made sure LL kept getting cash.  If a AAA company tired to do a Lovecraft inspired game, there is a chance they would flop.  Yet all LL has to do is keept the clockwork ticking, and it can let a bunch of volunteers worry about getting people into the carnival tent.

 I will say this hot and clear: LL needs us, because we put the work in that keeps the money coming in.  Now, we realize in an age where hacking makes the news, that no security is perfect.  If Washington, Moscow and Beijing have to nurse wounds from hackers, anyone will.  However, that does not mean you must not show care to your customers.  Most people realize that there will be moments that you have to say "there's nothing I can do."  But if someone has tried to address a problem several times, and if there is a chance, even a perception that something YOU did may have been to blame you had at the very least better avoid beign callous.  When you go ahead and say that "rolling maintenance" message, you cannot be so naive as to avoid the idea that hackers are going "Gee, we can do mischief because everyone will be expecting stuff not to work, and LL just gave that message."  You have a vested interest to make sure that when the smoke clears, people have a reason to trust that you will try to fix things at the very least

Yes, this is an odd time, where there is talk of SL 2.0, Oculus Rift, and all sorts of sexy stuff that the users of LL will probably want to spend more money on.  However, if you make it hard for people to invest in their communties, then, unlike RL, we can choose how much to invest in LL.  The one thing LL will not want is to hype the next generation of bells and whistles, only for people to go "I can blow all my disposabel income on this and then get told by LL "sorry, too bad, so sad."

Do not mistake harsh words for hate LL.  The reasons some of us are angry is because WE WANT THIS PLACE TO WORK.  By helping us, you help yourself, because we are not the tpyical user that is always chasing after the next shiny widget that will soon be outdated by someone else's shiny lure.

[Callum Meriman]

It's a pity the sim was eaten, but this stuff happens, and it happens with computers and applications that cause millions of dollar of loss and in some cases even destroy or worst case, cost lives.

The fact is simple, the lab can't create a restore point out of thin air. The sim owner should have notified them immediately so that they could archive the restore point they had at that time.

Instead, what I hope is some talented builders see and help in restoration and rebuilding, to create something even better.

---

avalonalpha wrote:

 

 I will say this hot and clear: LL needs us, because we put the work in that keeps the money coming in. 

---

With Linden Lab trying to rake up more money with $600 buydowns and with Linden Lab banning estate owners, I think the impression you should have is they don't need us. Running the grid costs money. Datacentres cost money. Bandwidth costs money. Technicians, programmers, helpdesk, clerical staff all cost money. And, we, as a source of money are drying up.

 

---

Yes, this is an odd time, where there is talk of SL 2.0, Oculus Rift, and all sorts of sexy stuff that the users of LL will probably want to spend more money on. 

---

The results of the Sansar project are not for us. The very fact that VR induced nausia can only be tackled by professionals should be a hint. There may be some *games* that use the platform, but it will never be capable of being a sandbox virtual world like SL, technology will prevent that.

[Flenser Juergens]

I really wish the people running Linden Labs these days would make some detailed and long term plans for what they want to do.  SecondLife is a success inspite of them;  it needs long term nurturing now and proper care of the infrastructure, both servers and virtual stuff that many hundreds of people have ceated and worked hard on.

**bleep**s going to happen from time to time.  Like that comment regarding learning from history - if you don't try and plan ahead for troubles your going to take a hard fall eventually.  That damages your revenue and makes the users wary of anything you say.

Secondlife is a unique place;  we need help with doing things from time to time.  Linden Labs has been content to collect their usage charges and wax all flowery about all their great plans they will emplement "real soon now".

We need them down in the trenches with us when we have serious troubles.

[Monti Messmer]

Hello, very sorry to hear your drama.

I´ve lost some things about a sim crash and LL refused to do a roll-back.

Anyway, if they (LL) would give the owner a chance to backup a complete sim with ALL builds then this behaviour would be ok.

If LL only can manage backups then - one backup a day kept for 30 days - should be the usual term.

Monti

[Chic Aeon]

I haven't been to Innsmouth for awhile but I used to spend a lot of time there. Lovely place. Lots of atmosphere.

 

That being said along with "sorry for your loss and hassles" AND the fact that I have lost huge sections of my inventory over the years (not rezzed items, but in inventory) I suggest folks take precautions. The TOS clearly states that this can happen and it does. The reasons why really don't matter. Preparation is important.

Just last week I took coalesced copies of all my major builds (about a quarter sim), put them in a folder called "coalesced backups June 2016" and crossed my fingers that I never have to use them. That being said, previous coalesced backups have saved inventory in the past (those database failures) so I suspect that somewhere along the way they may come in handy.

 

Whether you own your own sim or rent like I do, things DO go wrong. We can kick and scream and beat our hands against the wall -- and sometimes that helps a bit. But having an easy way to repair the damage is a much better plan in my book.

 

After my first giant inventory loss (many thousands of home and garden items) I was devastated. Things were just gone.  Last month I lost more items somehow (things that had been there the week before completely disappeared from the database -- no copies anywhere for anyone and it wasn't a DMCA thing). We all need to keep that in mind and do what we can to protect ourselves.

 

And when things are really gone for good, as someone said,  we can rebuild better than before. Photos are sometimes our only memories.



 

[Pamela Galli]

This is alarming. Did LL try resetting your inventory?

[Amethyst Jetaime]

I can't imagine that the builder of the buildings didn't make and retain copies.. If they didn't that is gross negligence as this type of thing happens all the time.  It can happen days after a build is down or years later.  They copies should be kept by someone as long as the buildings exist in world.

See if you can track them down.  If the builder is no longer around, ask the former owner.

[ArikTheRed]

Chic, that's a great image - I'd know that scene anywhere ;-)

 

I've lost loads of inventory off and on over time as well, and can feel for you there.  As you, I fell in love with Innsmouth early on (my 2nd day in SL) and at that time had no idea I'd be the founder/coordinator of a fairly significant annual event (Lovecraft Festival of SL), or even imagine I would be overseeing what I feel to be one of the finest oldtime sims of SL.

 

In any event, thank you to all for your comments, support and constructive criticisms.  My only purpose as such is to work to preserve what I feel strongly to be a classic "legend sim" of SL.  Thus, I get much more worked up over that, and Lindens' mostly cold approach, to the degree where they implied in wording in my support request that the issue was "user-originated."  My repeated inspections of the sim, monitoring and trying to make sense of it all, showed no sign of the nature of the damage and quirky meltdown of functions to result from griefers or overstressing of the sim by any present visitors in some manner.

 

In any case, it's over and done at this point, with the signature style of Linden customer care that I've experienced a couple of times since taking over the sim.

 

I give a huge thanks to Innsmouth's founder and creator, who did come and see the damage, and repaired/replaced the damaged area of the waterfront.  Thank you, Darmy!

[MicSteel Newman]

Yes, all that stuff costs money...but not NEARLY as much as the $300/month/sim that LL charges. They are raking in the bucks, and spending them SOMEWHERE, but not enough of it is going into basics. SL's core code is...a nightmare that needs to be started over, with different basic assumptions: The users have been showing LL for a decade that the initial architecture is inadequate, but LL has been adding patch after patch, rather than applying intelligent oversight. One is led to believe that they don't HAVE anyone with the requisite intelligence AND the desire to manage such a project AND put up with corporate leaders with inadequate vision or empathy or sensitivity to users' needs and desires. I wouldn't volunteer for that job!

I understand that LL runs four to six sims on each server. That means they have no hardware challenges: $1,200 to 1,800 a month will buy a hell of a piece of hardware. But that's not much of a problem with SL.It would also pay the salaries of the average IT tech, if the tech were responsible for as few as a dozen servers. I'm assuming HUGE inefficiencies, here, and monstrous greed and financial siphoning to other projects; something on the order of %1,000 or more.

The challenges are in the software: It is apparently not open-ended nor scalable enough for the uses to which user put it. UUIDs? Pah! Random-number generation is idiotic, when you consider the average user has upwards of 50,000 standing items in their personal inventory. (I'd recommend a UUID based on creator, date-time, and sim--including grid--where the creator was logged in, and adding serial numbers using a similar skeleton for copies. No chance of conflicting IDs, then.) Then, your assets could be DISTRIBUTED amongst mirrored asset-servers based on a variety of factors, such as how many items each creator has made, and how many copies of these items are in other avatars' inventories or rezzed in-world.

Another challenge is networking. Servers are too centralized, apparently, and there are terrible bottlenecks. The technology exists to cure that: P2P systems provide a clue. I'm not saying the users' PCs should be turned into servers, although that's something that LL should consider--and even consider giving certain high-powered users discounts for--but distributing copies of textures, for example, would most certainly ease one obvious bottleneck. I designed a hierarchical, user-balanced, shared data structure for a MMO that has never seen the light of the Internet, that would be capable of eliminating almost all of the server-side lag we see today. (Yes, I'm quite aware that most lag is client-side, and I have solutions for much of that, as well.)

In essence, I'm saying that corporate bloat has blinded project management--or disabled it--to the extent that they are only allowing themselves to put out fires. With this kind of management, SL2 will not manifest; its market will be taken by some game--accidentally, most likely--such as Elite: Dangerous or Star Citizen, when they allow users to build their own space ports, or ArcheAge or some similar game that allows users to have their own homes and farms in the game world. Enhancing such a product to create "rooms" that are non-combat, and thus isolated from the active gaming aspects so as to prevent lagging them, is trivial, and would only require someone in the corporate chain to say, "make it so." LL could lose its shirt in months, and those of us who LIVE in SL would find ourselves in a collapsing post-apocalyptic environment with NO support and NO reliability.

[Ayesha Askham]

Mic Steel

This is a tired, old argument and while it IS perfectly reasonable on one level, it is utterly impractical on many others.  LL is a business, like any other and it needs to feed its greedy shareholders with profit.  That it does it with excessive zeal is reprehensible but hardly a novel offense.

Sansara is Linden Lab's answer to the need to "remake" SL.  Sansara is rationally built and much better controlled, but as you might expect, being better optimized it is also far less open.  Those other "games" you mention may well allow others to build, but I'd surmize that all they really can do is choose from pre-optimized substructures, so that the overall function and user-load is more predictable.

To use a hackneyed phrase SL "is what it is".  It is badly built, badly patched and by and large badly run.  There is nothing that any of us can do about that, complain bitterly as we might.

It is also something that has liberated many from a RL highly restricted either by disability or ill-health, something we all too often forget.

[Flenser Juergens]

There are a lot of things I could say, in a followup to all the other messages but I'll just stick to some simple items.

SecondLife has grown a great deal since the simple little xml based system(that might have had some seed money from the Nez).  For a long long time it's just been growing like crazy and when ever a problem occurs more often then not it's just patched.

1.  It's going to be a slow and agonising business but the server code has got to be redone in something other then C++.

     Misuse of C++ has been responsible for more f*cked up systems worldwide then anything else around.  A proper    programming team needs to be given access to the code and do something in another language.  I appreciate what the "Firemen" have done in patching and other stuff but the server codes need a rewrite from the bottom up.

2.  There are dependacies in the viewer code and server code that should be removed - most of the music and other media streams go thru the SL servers at some point in their journeys;  that should be left to the users' systems once the server gives them the link to a stream.

3.  Absolutely no new features added to the SecondLife system for at least 6 months to a year;  mindlessly piling on new features again and again has damaged the system in many ways.  The physics system is very impressive but it took a long time to get it working right with SL, same with mesh and other features.  Linden Labs has a sick code base:  that needs to be fixed first before throwing more things on to the system.

4.  Work on Sansar should stop and the team working on that help with the main systems and make notes as they go.  So they don't make all the same errors that original programmers of SL did - not planning for expansion, nesting many calls for resources so deep that DB machines often timeout on requests if the network is lagging even a little(i.e.  Saturday nights).  There are no doubt many other problems but if a good foundation is laid, the code should keep going for many more years.

Making SL and Sansar sisters instead of vaguly related systems would go a long ways to helping the stability of these virtual systems.