LL explain recent log-in problems

[Innula Zenovka]

People may have missed the lengthy and detailed explanation by Landon Linden of what he calls The Recent Unpleasantness.

This openness and honesty from the Lab about the causes of disruptions and technical difficulties is very refreshing and welcome, or at least I find it so.

Thank you Landon and LL both.

[Miro Collas]

Well said - and seconded.

[Zi Ree]

This is a very welcome statement and an interesting read about the architecture behind the system! We didn't have so much openness from the Lab in quite a while. Please keep it up, people want to know what's wrong when stuff breaks

 

[TankMaster Finesmith]

Couldn't agree more!  They gave lots of good info on whats going on, and why we are seeing issues with the service, all without spilling any coperate secrets. This just goes to show it is possible to tell people whats going wrong and what they are doing about it without harming the company in any way.  This is a nice change from the old way of "dont talk to anyone about anything" stance that previously griped the Lab.  Keep up the good communic communication lines!

[Drew Bhalti]

I read it, I'm impressed. the full transparency is a bit scary, but it's MORE then welcome.

[MBeatrix]

Yes, I do appreciate Landon's honesty and I hope that for the future the Lab contnues being as transparent as it did today. We needed to know what is going on. Not only to keep us from demanding what may not be possible at a given moment, but also to kill all speculation.

[Pamela Galli]

I am so grateful for these signs that LL gives a #%^* that we have been seeing since Ebbe's ascent to the throne. I had given up on LL.

[Willow Wilder]

Thank you Landon and LL. Your blog post is just awesome on so many levels.  

[KarenMichelle Lane]

I'm loving the new openness of linden Lab - KUDOS to Linden Lab & Landon Linden

Having worked  in the past as a Subject Area Expert [Accounting] on a Fortune 100 company JAD Team, that status report was just the right level of detail to understand the magnitude of the problem LL is facing. Here's to a successful database support infrastructure reorganization.

[Jenni Darkwatch]

Legacy errors. Lovely. They happen, and they always stink. Thanks for the post.

[Qie Niangao]

Thanks, Innula, for posting about that explanation. It's interesting and informative and some very welcome transparency. 

[Hitomi Tiponi]

An excellent, open explanation from LL.  This is a good sign from the new regime.

[Amethyst Jetaime]

Wow, what a breath of fresh air!! Thanks so much for the explanation, Landon and thanks for the initiative to analyze the entire log in process to fix any other legacy issues.  Kudos on the communication with residents.  Hopefully this will result in more residents being patient when things go wrong.

[Madelaine McMasters]

That's just remarkable, Innula. Thanks for the heads up.

It's fantastic to see LL putting bricks back into the customer relationship, rather than taking them out.

 

 

... offers root-beer to Landon, Ebbe and the entire LL crew.

Welcome back!

[Emma Krokus]

Thanks for highlighting that Innula.

And I too appreciate the openness about the issues that affect Linden Lab, Thank you Landon and Linden Lab.

Emma

[Dresden Ceriano]

Fifteen posted already and not one substantially negative comment... I'm impressed.  If only LL were able to continue understanding what will make their users happy.  They need not be perfect.  As long as they make the effort to be honest and open about the difficulties which they face and explicit in how they intend to deal with such issues, they'll be able to make a world of difference in the way they're perceived.

Of course, this assessment is based on only one, rather strict, criteria.  There are certainly many more issues which need to be addressed before things will actually improve.

...Dres

[sirhc DeSantis]

'A few weeks ago there was a massive distributed denial of service attack on one of our upstream service providers '

 

there you go a negative - who and on what? Else its the sort of fluff I used to peddle to middle management eedjuts

 

eta ach i forgot how to spell manglement

[Phil Deakins]

---

Madelaine McMasters wrote:

That's just remarkable, Innula. Thanks for the heads up.

It's fantastic to see LL putting bricks back into the customer relationship, rather than taking them out.

 

 

... offers root-beer to Landon, Ebbe and the entire LL crew.

Welcome back!

---

I agree with everyone else that it's an illuminating, and very welcome, blog post. And I agree with you that it's a brick in the right direction, and a big brick too. But I'm not holding my breath expecting general openness from LL. One brick doth not a house make. I'll believe the openness of the new regime when I see it - when it becomes the norm. This is just one sighting in the right direction. It's a cause for optimism, but not yet anything more than that.

[Innula Zenovka]

---

sirhc DeSantis wrote:

'

A few weeks ago there was a massive distributed denial of service attack on one of our upstream service providers

'

 

there you go a negative - who and on what? Else its the sort of fluff I used to peddle to middle management eedjuts

 

eta ach i forgot how to spell manglement

---

You should have read to the end of the sentence.

---

A few weeks ago there was a massive distributed denial of service attack

on one of our upstream service providers that affected most of their customers, including us,

and inhibited the ability of some to use our services. We have since mitigated future potential impact from such an attack by adding an additional provider.

---

That, it seems to me, tells us on what: on one of LL's upstream service providers.

As to who the culprits were, presumably LL only knows what the target of the attack, the upstream service provider, tells thsm.

[SaraCarena]

Apology accepted Landon and LL :smileyhappy:. 

Stability of sl has improved a lot since I`ve been here, well I think so.

[Crim Mip]

The explanation of the login problem shows the human side of the coders who work there. I'm sure the original coder never considered that that there would be enough people using SL that it would only take 10 years to run out of identifiers. The number of those possible identifiers was probably considered 'more than we'll ever need' at the time.

Also, I'd say it showed some darn good coding. It was doing it's job so well that there had never been a reason to look at it and so nobody did look at it. It went on quietly doing it's job for for 10 years. Sure, in hindsight,  it's easy to say "It should have warned that it was going to run out of identifiers." Well to do that check would have been one more bit of code it would have had to run each time anyone logged in. Nobody could predict the growth spurt of users several years after that code was written.

I bet there will be a code audit to check any code that hasn't actually been changed or updated in x number of years just to make sure there aren't any more inadvertent time bombs like this quietly doing their job until one day they simply can't.

Thank you for the explanation.

[Phil Deakins]

It reminds me of not too long after Google started. They got the number of pages indexed up to 4 billion and stopped increasing. 4 billions was a lot for a search engine at that time, and the stop was probably due to them running out of numbers - not enough bits.

[Theresa Tennyson]

---

sirhc DeSantis wrote:

'

A few weeks ago there was a massive distributed denial of service attack on one of our upstream service providers

'

 

there you go a negative - who and on what? Else its the sort of fluff I used to peddle to middle management eedjuts

 

eta ach i forgot how to spell manglement

---

How do we know that you're not asking for this information to put on your own DDOS attack? In the interest of true transparency, shouldn't the Lab also give us your personal information as well, "Mr. DeSantis"?

[Zi Ree]

The DDoS attack is a well known case, targeting UltraDNS, the service provider Linden Lab uses for DNS resolutions:

http://threatpost.com/ultradns-dealing-with-ddos-attack/105806

Nothing secret about it, just no information that was important for the blog post.

 

[MBeatrix]

I'm sure the original coder never considered that that there would be enough people using SL that it would only take 10 years to run out of identifiers.

Maybe if the Lab had been more concerned about the way things work instead of cheating numbers, that might had never happened. Just look below:

Total Residents:38,187,152 | Residents online:34,267 | Last update:2014-05-25 04:25:05 SLT

Data from http://www.gridsurvey.com/index.php

 

So, I hope that this last "unpleasantness" has been a good lesson for everyone, starting on the LL directors board and ending at us all, users. As I wrote before, in another thread, after more than a decade Second Life still is an experimental thing and we probably should look at it under that perspective.