Explanation is due

[Josh Susanto]

"What? I'm confused. Yeah, I've heard parts of your theories, I just don't get why it has to be an animator. Plus, you seem to know very little about the animation market. I mean, I could see that if some1 like myself was making such claims, it would, at least, have some weight, only because I'm an animator. Why a sculpty maker has theories about the motivations of animators, I'm somewhat dumbfounded."

That part of the theory goes back to August/September when several things happened which seemed to be directed primarily at merchants who were selling animations or items that included animations. 

It's not at all certain that he would have to be an animator, or even a merchant who continues to specialize in animations or animation-scripted items. But that would have been a good place to start looking back in August/September, if anyone had bothered.

That I am primarily a sculpty maker would seem to explain why, when there's a bork that affects me, it practically also affects plenty of people who make totally different stuff. Just because I'm a more vocal critic here doesn't mean I can't think animations merchants are only being more shamelessly abused than I am.

How many complaints do you see on this forum, for example, from people who are primarily texturers?

 

[Rene Erlanger]

Silence is golden at the Lab!

[Sera Lok]

---

Rage Riptide wrote:

Since when has the

Product

always been right?

---

LOL nice one.  and I agree with the rest. Whenever I get too emotionally invested I must remind myself that it's "Linden Research, Inc".... emphasis on the research.

[Madeliefste Oh]

Explanation about what hurlyburly we became victimes of on Valentines day is the least the lab could give us.

For me the most important thing is that they show that they care. Just a stupid standard 'we apologies for all inconvienence' is not satisfying at all. I would be so simple to make a gesture to show that they do care.

But maybe it is just idle hope to think that there is anybody around at the lab who really does care. Maybe they just do not think about the consequences for users when something goes wrong in the coding, but only about the coding itself.

[JeanneAnne]

---

Ela Talaj wrote:

By the way there is a rather simple and widespread failures management technique called "watchdog". That is if a number of queued events exceeds a certain value (individually set depending on the system parameters and throughput) the lights start blinking, the sirens sound

and an alert emails are sent to the maintenance personnel. A malfunctioning device may or may not be auto-shut down but a full memory image is recorded in either case. Would be interesting to know if such is implemented in the Marketplace and if not, why not.

---

Apparently, this is how the Japanese run their nuclear power plants, too...

TEPCO & LL = FAIL !!

Jeanne

[Nalates Urriah]

2012-02-14 04:34 PM - Re: MP Down??

2012-02-15 05:02 PM - [update] - Valentine's Day Downtime

Plus Rod was tweeting during the event. Reference - this link will age out as the tweets scroll off.

2 of the 3 were up before your opening post. The follow up about 3 hours after.

[Anaiya Arnold]

A belated one paragraph description of what we already know on the forums, even if posted twice, and someone's messages directed at individual users on some third party site many of do not belong to is not an adequate explanation to a problem of this scale.  It's just not.

[Toysoldier Thor]

I think Nalates was kidding since I am sure she knows LL's response so far to the disaster they created on the Valentines Disaster is an utter SHAMEFUL JOKE.

she is pointing out just disgusting LL's response has been to this poor avoidable incidence they caused.

Any responsible customer focused company would have been out front on this issue and trnasparent on what happened and had the leader of the company providing clear messages to the customer base that he is sorry on behalf of the company to assure his customers that he will be seeking answers to assure this can be avoided in the future.

I had a lot of respect for Rodvik at the beginning but it has become very clear over the past year that he is just a Gamer Geek and really doesnt have a good grasp of the critical skills on Customer Service and responsibilities of leading a company that badly needed leadership on improved customer service and matured business processes.

Its sad... and how he is dealing with this incident says so much.

[Toysoldier Thor]

To date we have not got the REAL TRUTH from LL on what they really did that screwed up all their systems.  Evidence shows it was related to their scheduled HW maintenance work at 4:51am they announced on grid status.  They have not giving a complete status if all have been cleaned up.  Most importantly they did not provide their customers any answers and assurances of how they will prevent this from happening again - Like implementing CHANGE FREEZES on critical events in the year.  And finally they owe their customers some sincere apology that all can see (not Rodvik making a lil Tweet because he was embarassed into it from some of us).

Until LL steps up and takes responsibility and ownership on this problem they caused.... THEY HAVE NOT RESPONDED RESPONSIBLY.

[sirhc Weatherwax]

From the response : "We anticipate that orders in this state should be processed by end of day Thursday". Well thats OK then, so nobody needs to worry about being ready for V Day next year....

Lovely. A certain lack of understanding of time criticality there

[Mikki Miles]

from merchant's view a Valentine's Week is better, anyway...

[Josh Susanto]

So what would be incorrect about the following summary of how they have explained it?

"Multiple hardware failures incidentally converge to incidentally rob merchants, again."

 

[Legend Hye]

Well,  glad to see LL did the right thing and gave everyone their L$ sale back to them, so thanks.

 

I have to admit, I was not expecting them too, I am sure they would have made quite a bit of money if they had not.

 

From what I hear most people have already got all their sales money back already.

[Luana Juliesse]

They didn't rob merchants. Merchants have been paid. Yes, it's unfortunate that the incident happened in such a busy day, but they are trying to fix it. So let's not exaggerate. We all had our losses, but you can hardly call it robbery.

[Sera Lok]

Also it should be noted that most of the items actually were delivered on Valentine's Day - it was just the payment and email confirmations to merchants that was delayed.

Doesn't make it any less horrific... tons of potential sales were lost on Tuesday.

[Medhue Simoni]

---

Luana Juliesse wrote:

They didn't rob merchants. Merchants have been paid. Yes, it's unfortunate that the incident happened in such a busy day, but they are trying to fix it. So 

let's not

 

exaggerate

.

 

We all

 

had

 our 

losses

,

 

but

 you 

can

 

hardly

 call it 

robbery.

---

Maybe not robbery, but when a system is set up so that a customer can get a product and the merchant never get paid, then what would you call it? The simple fact is that this does happen all the time. If any1 can dispute this, I'd love to here it. The worst part about it, is that the merchant will never know, unless that customer actually contacts the merchant. In my mind, the MP's only job is to take the money and they can't do that correctly. Whether the product gets delivered properly or not is on me. The MP should not be giving any refunds for bad deliveries, especially when this is a common issue.

Heck, even a shorter wait time would fix most of the lost sales issues. A customer waiting hours to get their refund is not likely going to result in the customer trying the purchase again. I mean, WTF are people thinking when they create these systems.

[Toysoldier Thor]

So what bothers me is why LL seems to be trying to cover up the root cause of LL Valentines Massacre instead of being transparent and honest and taking ownership to what likely was a problem they triggered by a scheduled DC HW Maintenance problem.

Here is what i mean....

Following is a posting from from the LL Commerce Team within the MP DOWN?? thread that they posted late afternoon on the 14th....

------------------------------------------------------------------------------------
MP DOWN?? posting # 128

CommerceTeam Linden wrote:

Starting at 11pm SLT, February 13, the Second Life Marketplace experienced two separate hardware failures which resulted in delayed deliveries and Marketplace downtime. While the site has been back up since 1pm SLT today, some merchants may see delays in order completion and merchant payout. We continue to investigate the cause of these issues to prevent this kind of failure from happening in the future. We apologize for any inconvenience.

---

 

So take careful note of the date/time that LL Commerce states that they had two unscheduled hardware failures that were the root cause to MP delivery coming to a griding halt.  They say the problem happened at 11pm SLT on the 13th.

Yet the first time that ANY Merchant of MP first noticed and publically reported something was going seriously wrong was not until shortly after 5am SLT.  Following are the details to prove this first detection...

Teagan Parnas opened up the "MP DOWN??" Merchant forum thread just after 5am SLT on Feb 14th.  I dont have the exact time as I cant find it in my email logs but very shortly after Teagan's thread opening post indicating the first Merchant detection of the problem,  Shiva India made the 2nd post to the thread and I have the timestamp to this 02-14-2012 05:25 AM PST/SLT.

Now as you all know very well, when a system failure like MP of this magnitude happens, it does not take long AT ALL for the MP Merchants to notice it and start posting about it.... specially on the busiest shopping day of the year on MP.  So one can safely assume that if it was first detected just after 5am SLT by Teagan, the root cause of the failure / the trigger to the failure happened shortly before that.

So... if it looks very clear that MP Deliveries went WONKY just after 5am, how could two hardware failure at 11pm SLT (6 hours earlier) not impact MP operations for 6 hours ???

So lets look at what other event prior to 5am SLT might have been the REAL root cause to the MP Valentines Day Massacre.....

Here is an alert from the SL Grid Status....

[Updated 4:51am PST, 14 February 2012] We will be starting our hardware maintenance shortly. Please refrain from rezzing any no-copy objects or making any L$ transactions during this time.

 

Notice the time when LL announced that they would be performing a "HARDWARE MAINTENANCE".  At 4:51am SLT....  Only minutes before the first merchants noticed the MP Deliveries coming to a grinding halt.

So LL Commerce Team, do you want to explain a bit more clearly how your explanation that two unplanned hardware failures 6 hours prior to the actual start of the MP failures were the cause to the MP outage and that it was not the scheduled HW maintenance that LL executed only minutes before the failure?

 

This is why Rodvik or some member from the LL Executive Team needs to come clean and be honest to their customer base.  Because the LL Commerce Team seems you be making excuses and stories up on what the cause was.

[Toysoldier Thor]

BTW.... Knowing how LL operates, you all better read my las posting fast unless you get email alerts on all forum postings.

LL doesnt like having egg on their face and it wont surprise me if my posting of facts that the LL Commerce Teams explanation of the problem was wrong will be removed by the moderator.  Not because it violated an posting tos rules.... just because...

Stay tuned

[Luana Juliesse]

I have noticed that something is wrong with MP when i came online at about 5.30 AM on 14th february. At that time i already had about 3 pages of queued deliveries. And my last sale on MP was at 02/14/2012 00:58:54. This is turning into a witch hunt.

[Toysoldier Thor]

You call it a witch hunt... I call it trying to perform forensics on the limited information available with little communications from LL on what happened and what they will do to prevent it from happening.

[Sera Lok]

I just hope that moving forward these things will be avoided on big shopping days.  And that the Commerce Team starts being more communicative with Merchants.. this is what is sorely missing right now.  I feel they are placing the importance on avoiding negative feedback (hence the silence) instead of placing priority on keeping merchants informed. Most of us are not negative LL haters, we just want the information so we can be aware of what's going on, so we don't sit for hours waiting for a post that gets buried in a forum or a tweet.

I would also like to add that it would be very beneficial to have User Group Meetings, as was promised us before.. Even if not all can attend, it would be a huge step in the right direction.

[Toysoldier Thor]

LL could have and still can stop a lot of the criticism against their handling of this entire terrible event by taking full public ownership of the problem, being transparent and honest with its customers that they badly impacted, and show their strong desire to want to improve and provide confidence to the Merchants and Customers.

IF while the event was happening someone even a low in the management foodchain like Brooke were to make a BLOG POSTING provide open and clear statements of what has happened, where is LL on detecting and resolving the problem, providing instructions to all affected, and just as importantly... showing some compassion with a simple apology for the size and scale of the disruption LL created....  this would have INSTANTLY reduced the anger and frustration by 10 fold.

But nope... as has been proven by Brooke and the Commerce Team, NO COMMUNICATIONS or at best bare basic cold termed brief forum posted statements have become the norm.

A Blog posting from Rodvik on the Merchant forums and a link on the MP site would have had even more impact to show that LL is on top of it and that they take it seriously and to assure all that LL will be performing a post-mortem to figure out exactly how this happened and what they can do to prevent this from happening again. 

OK and here is a bonus that some Customer Driven companies would have seriously considered.... offering the impacted community a "we are sorry" credit offer for all the pain they created. 

Can you image the community response if LL would have handled this major incident like I suggested above AND then offering a "WEEK of NO COMMISSION SALES on MP" as a small token of LL's appreciate for the Merchants patience during this unexpected outage?

 

See - this is how LL could have come out of this event smelling like roses and shutting down the magnitude of frustration and anger and accusations that they are still now enduring from their customers.

 

[Josh Susanto]

>They didn't rob merchants

The SLM, as a whole, was down, although the listing enhancements people had bought specifically to promote items during that downtime were nonetheless charged, and the money charged will not be returned.

Moreover, LL had specifically and transparently encouraged merchants to produce, list and promote V-Day items by showing a V-Day related ad for some time on the front page of the SLM; items which cost users to produce, cost users to list (by land rental) and cost users to promote; all these wasted costs to be construed as constructively solicited by LL, all  paid to LL and all not returned.

In what way is that not robbery?

 

[Toysoldier Thor]

Well I guess the LL Staff does actually read these threads and ocassionally listens....

"OK and here is a bonus that some Customer Driven companies would have seriously considered.... offering the impacted community a "we are sorry" credit offer for all the pain they created.

Can you image the community response if LL would have handled this major incident like I suggested above AND then offering a "WEEK of NO COMMISSION SALES on MP" as a small token of LL's appreciate for the Merchants patience during this unexpected outage?"

LL Comerce Team has just announced 1 week on no commissions for sales of items less than 10,000L ! 

Nice Gesture

[Marcus Hancroft]

Howdy, Toy.  It's not for a whole week.  But for the weekend 2-24 to 2-27. 

<snip>

"In appreciation of your patience, we will be offering free commission on all listings less than L$10,000 starting NEXT Friday for three days starting at noon PT Friday, February 24, 2012 (through noon Monday, February 27, 2012).*"

But, I guess they DID listen to you this time!