Jump to content

Tools and Technology

  • entries
    72
  • comments
    3,233
  • views
    56,233

Contributors to this blog

About this blog

Entries in this blog

Linden Lab

Hi! I’m a member of the Second Life operations team, and I was the primary on-call systems engineer this past weekend. We had a very difficult weekend, so I wanted to take a few minutes to share what happened.

We had a series of independent failures happen that produced the rough waters Residents experienced inworld.

Shortly after midnight Pacific time on January 9th (Saturday) we had the master node of one of the central databases crash. The central database that happened to go down was one the most  used databases in Second Life. Without it Residents are unable to log in, or do, well, a lot of important things.

This sort of failure is something my team is good at handling, but it takes time for us to promote a replica up the chain to ultimately become the new master node. While we’re doing this we block logins and close other inworld services to help take the pressure off the newly promoted master node when it starts taking queries. (We reopen the grid slowly, turning on services one at a time, as the database is able to handle it.) The promotion process took about an hour and a half, and the grid returned to normal by 1:30am.

After this promotion took place the grid was stable the rest of the day on Saturday, and that evening.

That brings us to Sunday morning.

Around 8:00am Pacific on January 10th (Sunday), one of our providers start experiencing issues, which resulted in very poor performance in loading assets inworld. I very quickly got on the phone with them as they tracked down the source of the issue. With my team and the remote team working together we were able to spot the problem, and get it resolved by early afternoon. All of our metrics looked good, and I and my colleagues were able to rez assets inworld just fine. It was at this point that we posted the first “All Clear” on the blog, because it appeared that things were back to normal.

It didn’t take us long to realize that things were about to get interesting again, however.

Shortly after we declared all clear, Residents rushed to return to the grid. (Sunday afternoon is a very busy time inworld, even under normal circumstances!) The rush of Residents returning to Second Life (a lot of whom now had empty caches that needed to be re-filled) at a time when our concurrency is the highest put many other subsystems under several times their normal load.

Rezzing assets was now fine, but we had other issues to figure out. It took us a few more hours after the first all clear for us to be able to stabilize our other services. As some folks noticed, the system that was under the highest load was the one that does what we call “baking” - it’s what makes the texture you see on your avatar - thus we had a large number of Residents that either appeared gray, or as clouds. (It was still trying to get caught up from the asset loading outage earlier!) By Sunday evening we were able to re-stabilize the grid, and Second Life returned to normal for real.

One of the things I like about my job is that Second Life is a totally unique and fun environment! (The infrastructure of a virtual world is amazing to me!) This is both good and bad. It’s good because we’re often challenged to come up with a solution to a problem that’s new and unique, but the flip side of this is that sometimes things can break in unexpected ways because we’re doing things that no one else does.

I’m really sorry for how rough things were inworld this weekend. My team takes the stability of the grid extremely seriously, and no one dislikes downtime more than us. Either one of these failures happening independently is bad enough, but having them occur in a series like that is fairly miserable.

See you inworld (after I get some sleep!),

April Linden

Linden Lab

Hi! I wanted to take a moment to share why we had to do a full grid roll on a Friday. We know that Friday grid rolls are super disruptive, and we felt it was important to explain why this one was timed the way it was.

Second Life is run on a collection of thousands of Linux servers, which we call the “grid.” This week there was a critical security warning issued for one of the core system libraries (glibc), that we use on our version of Linux. This security vulnerability is known as CVE-2015-7547.

Since then we’ve been working around-the-clock to make sure Second Life is secure.

The issue came to light on Tuesday morning, and the various Linux distributions made patches for the issue available shortly afterwards. Our security team quickly took a look at it, and assessed the impact it might have on the grid. They were able to determine that under certain situations this might impact Second Life, so we sprang into action to get the grid fully patched. They were able to make this determination shortly after lunch time on Tuesday.

The security team then handed the issue over to the Operations team, who worked to make the updates needed to the machine images we use. They finished in the middle of the night on Tuesday (which was actually early Wednesday morning).

Once the updates were available, the development and release teams sprung into action, and pulled the updates into the Second Life Server machine image. This took until Wednesday afternoon to get the Second Life Server code built, tested, and the security team confirmed that any potential risk had been taken care of.

After this, the updates were sent to the Quality Assurance (QA) team to make sure that Second Life still functioned as it should, and they finished up in the middle of the night on Wednesday.

At this point we had a decision to make - do we want to roll the code to the full grid at once? We decided that since the updates were to one of the most core libraries, we should be extra careful, and decided to roll the updates to the Release Candidate (RC) channels first. That happened on Thursday morning.

We took Thursday to watch the RC channels and make sure they were still performing well, and then went ahead and rolled the security update to the rest of the grid on Friday.

Just to make it clear, we saw no evidence that there was any attempt to use this security issue against Second Life. It was our mission to make sure it stayed that way!

The reason there was little notice for the roll on Thursday is two fold. First, we were moving very quickly, and second because the roll was to mitigate a security issue, we didn’t want to tip our hand and show what was going on until after the issue had been fully resolved.

We know how disruptive full grid rolls are, and we know how busy Friday is for Residents inworld. The timing was terrible, but we felt it was important to get the security update on the full grid as quickly as we could.

Thank you for your patience, and we’re sorry for the bumpy ride on a Friday.

April Linden

Oz Linden

As part of launching our new more open viewer development process, we changed the open source license for the new viewer code base from the GPL with an exception to allow use with certain other open source licenses (the "FLOSS Exception") to the LGPL.  Some people have been asking about what this change means, and what our intentions are in making it.  In particular, the LGPL requires that the code first be compiled into a library and then may be used in an application.  Since the open source we make available does not directly provide for building this way, some people have speculated that the new license may actually be a barrier to other viewer developers.  I'll try to answer this as clearly as I can.

At Linden Lab, our intent is to enable the development of third party viewers and the improvement of our own viewer through mutual sharing of the requisite common technology.  We think that LGPL is a better license to accomplish that sharing. Even though it was originally written to apply only to libraries, there is significant room for interpretation when using it.

Our interpretation is this: modifying the build procedures provided in the source code to produce a library containing a subset of the licensed code is a minor programming exercise.  The result of linking that library (which may be either a static library or a shared one, and either is allowed) with whatever other code is used to build a viewer is again a minor problem. The results are essentially indistinguishable from those in which the developer just modified the build procedures in place to produce an executable application.  Essentially, the details of the linkage procedures used are a matter of packaging and have no end user effect.  In practical programming terms, a library is just a convenient package - the requirement in the LGPL is like requiring that a grocery store sell a product only if it is first placed into a shopping cart while inside the store.  Essentially a library is just that - a shopping cart full of individually compiled modules, so there's no real difference between using a library and just using the modules separately (one could build a library containing exactly one module, or build an application that had only one module and one library).

It is our intent that developers be able to use the components that we have licensed under the LGPL to produce viewer applications, regardless of the linkage procedures used, so long as they make available any changes to those components as specified in the license.

Q Linden

Today we launched Viewer 2.5, now out of Beta. The most significant update is a new, web-based profile system, which allows profiles to be viewed and edited both on the web and in the Viewer. For example, here's mine. Please note this is just a starting point for the web-based profiles; we’ll be doing a lot of work to refine the usability and make them richer over time.

In response to your feedback from the early beta versions of Viewer 2.5, we've added some privacy settings that will allow you to control just how public your profile is. Once you’ve logged in, click on “Privacy Settings” in the upper right corner of your profile. Group settings set in the Viewer will be overridden by these group privacy settings.

  • "Everyone" means that the information is available to the whole Internet and can be picked up by search engines.
  • "Second Life" means that the information is available to all Second Life Residents who are logged into the website or inworld. This is the default for all existing Residents using Viewer 2.5.
  • "Friends" means that only your Second Life friends can see the information on the web and inworld.

This is why we have a beta process--to address concerns and improve your user experience. We will continue to iterate as we get more feedback. Thank you for all your help and comments. Please attend the Viewer 2 User Group meetings if you would like to share your thoughts and feedback directly with me and the Snowstorm team.

Viewer 2.5 also has some other new features. The one I like best is that you can now have your Favorite landmarks also appear on the login screen, so that you can log directly into your favorite locations. Torley made a video about this, so check it out! We've also improved some texturing performance and fixed another batch of bugs. Watching the internal data, we've already seen a noticeable improvement in stability and performance--on par with Viewer 1.23.

Download Viewer 2.5, try it out, and keep the feedback coming! And, if you Twitter, please use the hashtag #slviewer2.

Helpful Links

Q Linden

In addition to many grid enhancements that FJ shared last week in his technology update blog post, the Viewer 2.5 beta cycle begins today. Viewer 2.5 Beta 1 is now available and includes several new features, usability improvements, and bug fixes.

Improved Web and Viewer Resident Profiles
The biggest change in Viewer 2.5 Beta 1 is that we've replaced the Viewer-based profiles with web-based profiles. You can access the redesigned profiles on the web using your Second Life user id and in the Viewer itself. For example, here’s mine. In the Viewer, profiles now open in a web browser and you can open as many profiles as you like at the same time. And, if you wish, you can even connect other social identities from Twitter, Facebook, and LinkedIn to your profile.

Logging In to your Favorite Locations
A long-standing feature request has been the ability to access some of  your favorite Landmarks from the login screen, so you can quickly teleport to places inworld. In the Viewer 2.5 Beta, we've added a preference that gives you access to the Landmarks on your Favorites Bar from the Login Screen. To use this feature, go to Preferences > Privacy and select the check box labeled, "Show my Favorite Landmarks at Login." If you log off and then restart your Viewer, then you'll see a list of your Favorite Landmarks in the "Start At" drop-down box on the login screen. Also, when this feature is enabled and you share a computer account (login) with other people, they will see your list of Favorites if they run the Viewer 2.5 Beta.

Decompression Performance Improvements
For a long time, we’ve used an older version of the Kakadu library known as KDU to load textures and images in the Viewer. The KDU Library has been updated to version 6.4 in the Viewer 2.5 Beta, which allows the Viewer to take advantage of significant decompression performance gains; our preliminary measurements gauge the performance gain at 30% for decoding images, which should translate to a visible improvement in rezzing time for complex scenes.

Please note that per our support policy, we’re also deprecating Viewer 2.1.1 today. If you’ve been running Viewer 2.1.1, you’ll be prompted to download the latest Viewer release. To stay up to date on future deprecation announcements, please subscribe to @secondlife on Twitter.

As  always, we want to know what you think of the latest beta so please keep that feedback coming. Share your impressions with us on Twitter and use the hashtag #SLViewer2.


Helpful Links

Esbee Linden

Viewer 2.4 Released!

As of today, SL Viewer 2.4 is the default Second Life Viewer download for new Residents! As was mentioned in the Viewer 2.4 Beta blog post, this is largely a maintenance release focused on improving user experience, stability, and performance. This release does, however, have a few important changes and additions, including the following:

A Cleaner User Experience and More Customization Options in Preferences

Throughout 2010, we’ve added many new Preferences to the Viewer and it was time to not only reorganize and clean up the layout, but also add popular customization options, such as:

  • Color and transparency options that allow you to change the look of Viewer windows
  • Options to easily enable the Advanced and Developer menus
  • A new preference that you can use to turn group and IM chat pop-ups on or off whenever you like
Support for External Text Editors when Working with LSL

Here’s a good one for content creators and developers. Now, you can use your favorite script editor to edit LSL scripts outside of the Viewer. In  Viewer 2.4, enabling the new ‘ExternalEditor’ debug setting will allow you to specify the path to your own text editor (for example: /Applications/TextMate.app/Contents/MacOS/TextMate). When the setting is enabled, just open the Script Editor in the Viewer and click the “Edit” button, and the text editor you specified in debug settings will then open with the script you have open in the Viewer. Any changes you make and save in the text editor will automatically appear in the Script window in the Viewer.

Graphics Improvements

We continue to make numerous improvements to Viewer graphics as we move closer to integrating Mesh Import into Viewer 2. You can see more detail in the Release Notes, but we’ve improved antialiasing, anisotropic filtering, snapshots, and we’ll see even more in upcoming releases.

Performance Improvements

Viewer 2 performance is improving, as shown by steadily decreasing crash rates that are now close to those for Viewer 1.23, but we know there’s still more to do. So if you do crash, please be sure to send us a crash  report, as they are essential to helping us understand where issues occur, which allows us to better prioritize our work.

An Auto-Updater Tool

We now have an auto-updater for Viewer 2! This means that whenever a new Viewer is released, the next one being Viewer 2.5, it automatically  downloads the newest software in the background and offers you the  ability to upgrade when it’s ready. You can still decide whether you want to install the new version or not, but the auto-updater will help ensure that you’re always using the most up-to-date version of the SL Viewer. Keep in mind, the auto-updater will only work with optional updates. Mandatory updates will continue to work as they have in the past.

Finally, to those of you who gave us feedback on previous versions of Viewer 2, we offer a heartfelt thank you for all your bug reports, forum and blog comments, and Tweets using the #slviewer2 hashtag. Your input is invaluable as we continue to prioritize our 2011 Viewer 2 roadmap.

So download Viewer 2.4 -- and keep that feedback coming!

Helpful Links

Jack.Linden

Today, we are excited to launch the latest version of Viewer 2 -- Viewer 2.3 Beta -- that makes it even easier to connect to friends and relevant experiences in Second Life. We’ve come a long way since we first launched the first Viewer 2 Beta in February. We’ve dramatically increased performance and stability while adding cool new features like Shared Media. We’ve also made many usability improvements, based on your feedback, that make it easier for content creators and developers to build in Second Life.

With the Version 2.3 Beta, Viewer 2 takes another big leap forward. This Beta not only includes many performance enhancements and bug fixes, but it also allows you to more freely express your inworld identity with Display Names, introduces helpful pop-up hints to make it easier to learn Viewer 2, and gives group owners more control of SL Events.

Display Names Arrives on the Main Grid

Display Names, as you know, gives all Residents two names: a unique username and an optional Display Name that you can change on a weekly basis. Display Names gives you the ability to use Unicode characters and multiple words so you can use your current SL Name, your real name, a pseudonym, or a gamer tag. It’s your choice!

Since we originally blogged about Display Names, and subsequently launched the Display Names Project Viewer on the test grid, we’ve made a range of changes to the feature that address the Resident concerns and feedback we received around impersonation and griefing. If you want to learn more, there’s a video below showcasing the enhancements and a full list detailing each change. We have also compiled a Frequently Asked Questions wiki page and two Knowledge Base articles, one on Display Names and another on usernames, complete with video tutorials from Torley that show you how to use this new feature.

Check out this video to see Display Names in action:


 

Because we need to ensure that Display Names operates correctly, we will not enable Display Names for the entire grid immediately. Today, it will work in a few thousand regions; once we see how it performs,  we will dial that up over the next few weeks until the whole grid is enabled. If you want to try using Display Names, then head to Blake Sea and Bay City, as those areas will be enabled first.

Group Owners Can Now Control Event Scheduling

The Viewer 2.3 Beta now allows group owners to control events scheduling on group-owned land. Please read more and see our step-by-step instructions on the SL Events wiki pages.

Viewer Hints Make the Switch to Viewer 2 Easy

We understand that there’s a bit of a learning curve when you make the switch from Viewer 1.23 to Viewer 2. So, we have added helpful pop-up hints for newer users around commonly used features, such as walk and chat. These disappear when the user takes the appropriate action, or dismisses the hint, and then you’ll never see them again.

Download the Viewer 2.3 Beta today, try it out, and Twitter your thoughts using the #slviewer2 hashtag.

Helpful Links

Esbee Linden

Today, we’re releasing Viewer 2 (Version 2.2.0 Beta) on our Beta release channel. This is an exciting release for us for two reasons. First, this release marks the first Viewer download since we announced the Snowstorm Team at SLCC in August. The Snowstorm Team has been working in the open to share our design and development process and directly engage with open source developers. Second, this beta release also marks the beginning of a new approach to our release process. Since we kicked off Project Snowstorm, we’ve made the latest Development Viewer available to Residents nightly so they can see our day to day progress and try out new things as soon as they’re in a build.

In our Beta and Release channels, we will make Viewers available to you more frequently. Our plan is to update Betas weekly with new official releases happening every 3-4 weeks. This lets us get new ideas and features into your hands faster and it gives us the ability to react to your feedback more quickly.

This latest beta release of Viewer 2 (Version 2.2.0 Beta), is mostly focused on bug fixes, but also includes some great user interface customization options. Here’s a quick run-down of what you’ll find in this release:

  • Sidebar Tabs can now be undocked and turned into regular floating windows.
  • The buttons on the Bottom Bar can be reordered.
  • Right-click on your avatar and select “Sit Down” to sit anywhere.
  • Parcel property icons (found in the Location Bar) are now on by default.
  • Selection beams are working again.
  • Language translations in local chat (via Google Translate API).
  • Group permissions now work via the Snapshot tool.
  • Turn off scripted particles and lights.
  • Builders can more easily align textures across linked prims using planar mapping.
  • View profile inspectors via the Mini-map.
  • Pan the mini-map with shift-drag.

Many of these features came to the official Viewer through resident contributions, both directly by way of the Snowstorm team, as well as indirectly through past contributions to the Snowglobe project. Thank you to all of you who have contributed code, testing, and commentary. Please keep it up!

To view a full list of changes, see the Release Notes.

Download the Beta Viewer : Windows | Mac | Linux

Please note that, as always on the beta channel, we only support one beta viewer at a time. If you’re using an older Beta Viewer, then you will be asked to upgrade on your next login.

Thank you!

Esbee Linden

Viewer 2 (Version 2.2.0) has been released today as the main Second Life Viewer! This is an important release for us because it marks the first big release using the Snowstorm process. A few months ago, we formed the Snowstorm Team as a dedicated Viewer team who works with the open source community and entirely in the open to make user experience improvements to the Viewer. We've been holding almost all our meetings in public, working with open source developers to integrate their work, sharing our backlog with the community, and releasing daily development Viewers which are available for anyone to download.

Over the last few months, we've improved the velocity of our teams and shortened the release cycle. The Version 2.2.0 release today comes after a three-week beta cycle during which we released Betas weekly and focused on stabilization and crash fixes on the Beta Viewer. Your feedback, crash, and bug reports during our beta cycles are crucial - so thank you to all who participated in that process.

Version 2.2.0 introduces some new features, usability enhancements, and crash fixes. Here are a few highlights:

  • Sidebar Panels can now be undocked and turned into regular floating windows: Simply click on a Sidebar Tab and drag it away from the Sidebar to undock it. You can also click the undock icon in the top right of a panel. To redock a sidebar panel, click the redock icon in the top right of the panel. Once sidebar panels are undocked, they behave like any other floating window in the Viewer.

  • Buttons on the Bottom Bar can now be reordered: In the last Viewer release, we added a few optional buttons to the Bottom Bar. In the 2.2.0 release, you can now reorder buttons on the Bottom Bar, so they’re arranged just the way you like them.

  • Right-click on your avatar and select “Sit Down” to sit anywhere: A great feature that came to the Viewer, by way of Snowglobe, allows you to right-click on yourself and select “Sit Down” from the context menu to make your avatar sit anywhere. To stand back up, just click the “Stand Up” button or right-click on yourself again and select “Stand Up” from the context menu.

  • Parcel Property Icons are now on by default: Parcel property icons indicate what users can, or can’t, do on a particular parcel. The icons can be found in the Location Bar and are now on by default. If you’d like to turn them off, you can do so by right-clicking the Location Bar and unchecking “Show Parcel Properties” from the context menu.

  • Selection Beams are working again: In Viewer 2, there was a bug that hid selection beams. This made it hard for people to see when you were building or the object you were currently editing. This has been corrected!

  • Language translations in local chat (via Google Translate API): Another great feature that came to the Viewer via Snowglobe is local chat translation. Simply click the “Translate chat” check box in the upper left hand corner of the local chat window. Language preferences can be set at the bottom of the Chat tab in the Preference window.

  • Group permissions now work in Snapshots: This corrected a bug in Snapshots which prevented you from sharing a Snapshot texture with a Group.

  • Turn off scripted particles and lights: This feature allows photographers and machinemists to turn off scripted lights by accessing the Debug Setting, “EffectScriptChatParticles” and setting it to FALSE. To turn scripted particles and lights back on, simply go to the Debug Setting window again and set “EffectScriptChatParticles” to TRUE.

  • Builders can more easily align textures across linked prims using planar mapping: This feature allows builders to align textures across faces so that several prims can look like a single prim. Simply select the faces of a set of linked prims, then open the Textures tab of the Build tool, make sure your Mapping setting is set to “Planar”, then click the checkbox labeled “Align planar faces.”

  • View profile inspectors via the Mini Map: When you hover over the little dots (people indicators) on the Mini-Map, you’ll see the familiar “i” icon pop open with a user’s name. Clicking the inspector allows you to view that user’s Mini Profile.

  • Pan the Mini-Map with shift-drag: If you hold down the <SHIFT> key and then click and drag on the Mini-map, you can now pan the map to view other areas.

As I mentioned in the blog post announcing the Version 2.2 beta, many of the features included in this release come directly from Resident contributions - via the Snowstorm process, as well as through past contributions to the Snowglobe project. Thank you to all of you who have contributed code, testing, and commentary. Please keep it up!

The Viewer 2 (Version 2.2.0) release is not a mandatory upgrade at this time, but will be the default download for the Second Life Viewer on the downloads page.

You can view the full release notes for Viewer 2 (Version 2.2.0) here.

Download Viewer 2 (Version 2.2.0)
Win | Mac | Linux

Linden Lab

UPDATE: 04.24.2015 - For an update on how the Tool Viewer Release affects these systems, visit this blog.


 


We have made some changes to the Second Life System Requirements to bring them more up to date, and are making some related changes to the Viewer:




  • We have removed Windows XP and Mac OSX 10.6 from the list of supported operating systems. Microsoft has announced the end of support for XP, and it has been some time since Apple has released updates for 10.6. For some time now, the Viewer has been significantly less stable on these older systems, and the lack of security updates to them make them more hazardous to use.
    We have no plans to actually block those systems, but problems reported on them that cannot be reproduced on supported systems will not likely be fixed.




  • The Windows installer has been modified to verify that the system has been updated with the most recent Service Packs from Microsoft. While we will not block installation on Windows 8 at this time, we strongly recommend upgrading to 8.1 for greater stability. Our data shows that the Viewer is significantly less stable on systems that have not been kept up to date, so the installer will now block installation until the updates have been applied. This change will be effective in a Viewer version to be released in the next few weeks, so it would be a good idea to get your system up to date before then. You can find information on how to install the latest updates at the Microsoft Windows Update page.



Linden Lab

Screenshot 2018-10-31 10.08.28.png

Hello amazing Residents of Second Life!

A few days ago (on Sunday, October 28th, 2018) we had a really rough day on the grid. For a few hours it was nearly impossible to be connected to Second Life at all, and this repeated several times during the day.

The reason this happened is that Second Life was being DDoSed.

Attacks of this type are pretty common. We’re able to handle nearly all of them without any Resident-visible impact to the grid, but the attacks on Sunday were particularly severe. The folks whothat were on call this weekend did their best to keep the grid stable during this time, and I’m grateful they did.

Sunday is our busiest day in Second Life each week, and we know there’s lot of events folks plan during it. We’re sorry those plans got interrupted. Like most of y’all, I too have an active life as a Resident, and my group had to work around the downtime as well. It was super frustrating.

As always, the place to stay informed of what’s going on is the Second Life Grid Status Blog. We do our best to keep it updated during periods of trouble on the grid.

Thanks for listening. I’ll see you inworld!

April Linden
Second Life Operations Team Lead

Linden Lab

Hi everyone.

As I’m sure most of y’all have noticed, Second Life has had a rough 24 hours. We’re experiencing outages unlike any in recent history, and I wanted to take a moment and explain what’s going on.

The grid is currently undergoing a large DDoS (Distributed Denial of Service) attack. Second Life being hit with a DDoS attack is pretty routine. It happens quite a bit, and we’re good at handling it without a large number of Residents noticing. However, the current DDoS attacks are at a level that we rarely see, and are impacting the entire grid at once.

My team (the Second Life Operations Team) is working as hard as we can to mitigate these attacks. We’ve had people working round-the-clock since they started, and will continue to do so until they settle down. (I had a very late night, myself!)

Second Life is not the only Internet service that’s been targeted today. My sister and brother opsen at other companies across the country are fighting the same battle we are. It’s been a rough few days on much of the Internet.

We’re really sorry that access to Second Life has been so sporadic over the last day. Trying to combat these attacks has the full attention of my team, and we’re working as hard as we can on it. We’ll keep posting on the Second Life Status Blog as we have new updates.

See you inworld!

April Linden
Second Life Operations Team Lead

Linden Lab

As we continue to work on improving the Second Life experience, one challenge we’ve been tackling is the length of the Viewer installation process. No one likes waiting, and now with Project Zipper, you don’t have to!

With the project Viewer available today, there’s really only one thing different - the installation is super fast. Rather than waiting for install to complete, you’ll quickly be in Second Life doing what you love.

Try out Project Zipper with the project Viewer here.

This is still a project Viewer, and if you find bugs while testing it out, please let us know by filing them in BUG project in JIRA.

Jack.Linden

Today we’re making available the first beta release of Viewer 2.4. We’ve done a great deal of maintenance work for this release, with significant help from the Second Life open-source community, including fixing a number of interface problems, hunting down and eliminating some key crash bugs, and making several performance improvements.

A number of features and changes are worth noting with this release. For the most part, we consider the work we're releasing today to be intermediate steps on the way to a better future:

  • A new auto-updater: Whenever there’s a new viewer release, the viewer will now download the new installer automatically in the background, and offer to install an upgrade when you next log in. You still control whether you want the new version, of course. We’ll continue to improve the auto-updater experience in future releases.
  • Clearer preferences: We’ve reorganised several of the preferences tabs to make them more understandable (and to make it easier for us to create new preferences, should the need arise). We've added some colour and transparency options, and fixed some bugs too. There will be more work to come in this area as we strive to make the viewer more customisable.
  • External text editor support: This is one we’re extremely happy about, as it's something Residents have wanted for a long time. If you’re a scripter, then you can now use your own text editor for scripting. The feature still needs a lot of user interface work, but as we work more on build tools in coming months, it will keep getting better and easier to use.
  • Graphics improvements: We’ve also integrated some graphics improvements as we work towards integrating mesh import into Viewer 2. Our graphics systems are undergoing major renovation, and there have been improvements to anti-aliasing, anisotropic filtering, and snapshots, with more to come in future releases.

So, although Viewer 2.4 is fundamentally a maintenance release, it has a lot of great stuff in it. Please give it a try and give us your feedback!

Download the Viewer 2.4 Beta today, try it out, and Twitter your thoughts using the #slviewer2 hashtag.

Helpful Links
Guest

Last week, we made a new page available as a replacement for the old Transaction History page. Due to your feedback, we rolled back the changes to this page to allow us to gather more feedback, and we are now providing this new page for review, without removing the old Transaction History page.

We have not yet made any changes to the new page, because we would like time to collect your feedback and review it. We have created a wiki page giving background on why changes were made to this page, where the new page is, and how to provide feedback. We will be closing feedback on April 30, 2014, so please take a look before then.

 

Linden Lab

A new set of Materials work has become a Release Candidate Viewer. This Viewer includes support for Materials underwater, render support for impostors, and fixes that will address issues with transparency and alpha layers. See the Release Notes for full details.

 

UnderwaterCoral.pngA quick coral study with Normal and Specular maps shown underwater. Land Impact is 14.

As of today, approximately one fifth of the regions in Second Life have at least one object using Materials, and this is increasing at a rate of about 2% per week. Users with high-end graphics cards are seeing a significant increase in frames per second when Advanced Lighting Model (ALM) is turned on -- we highly recommend that those users turn ALM on. In addition, we will be updating the GPU tables to include newer graphics cards, allowing more users to take advantage of advanced graphics features like Materials.

 

original.png

 

Thanks to those who have already tried out Materials and submitted a bug report to the MATBUG JIRA project! Once this release candidate has completed the release process, we will be moving any active MATBUGs to the BUG project and track any future issues from the BUG project.


If you’d like to experiment with Materials, download the Materials Release Candidate Viewer today. And don’t forget to check the Good Building Practices wiki for more tips and tricks and the Knowledge Base for updated user documentation.

Linden Lab

Yesterday, with much rejoicing, we promoted Viewer 3.7.28.300918 (Tools Update) to release. While this Viewer doesn’t have a shiny new featureset on the surface (other than reverting to a single-button login), it’s what’s inside that really matters - we’ve updated the numerous tools used to build the Viewer. The immediate expected effect of this is performance stability and a decreased crash rate.

We go to great lengths to maintain backwards compatibility in Second Life, both to never break users’ creations and to support the wide range of systems our Residents use to log in. However, sometimes we have to make the hard decisions: a year ago we announced that we were dropping support for Windows XP and Mac OSX 10.5 & 10.6 (a complete list of current system requirements is available here). Today, with the Tools Update release, the Viewer will no longer run on those systems. You will still be able to log in with an older Viewer until it is aged out based on our deprecation policy, however we strongly recommend updating your system.

It's unfortunate that we have to stop supporting some older systems, but upgrading the tools we use to build the viewer will help us to bring you other improvements to your Second Life experience more quickly and reliably.

Linden Lab

At one time or another, many of us have suffered a Viewer crash that interrupted our time in Second Life. At Linden Lab, we collect crash data to help us in our ongoing efforts to identify and fix issues in the Viewer that can cause crashes. Thanks to that information, nearly every release we make includes fixes of this sort.

The data we collect also reveals that many users can greatly reduce their risk of Viewer crashes by taking a few steps to update their software outside of Second Life.

The nature of Second Life as a platform for user creativity means that the Viewer faces different challenges than client software for an online game, for example, which would just need to handle the limited and carefully optimized content created by the game’s developer. This can make Second Life a demanding application for your computer and can mean that if your operating system is out of date, your Viewer is more likely to crash.

The good news is you can take steps today to help this! Here are a couple of tips:

Upgrade your Operating System

There is a very clear pattern in our statistics - the more up to date your operating system is, the less likely your Viewer is to crash. This applies on both Windows and Macintosh (Linux is a little harder to judge, since "up to date" has a more fluid meaning there, and the sample sizes are small). Some examples:

  • Windows 8.1 reports crashes only half as often as Windows 8.0

    Those of you who stuck with Windows 7 (roughly 40% of users of our Viewer right now) rather than upgrade to 8.0 made a good choice at the time; version 7 still has a much better crash rate than 8.0, but not quite as good as 8.1 (now about 15% of users), so waiting is no longer the best approach.

  • Mac OSX 10.9.3 reports crashes a third less than 10.7.5

    OSX rates do not have as much variation as Windows versions do, but newer is still better, and there are other non-crash reasons to be on the up to date version, including rendering improvements.

 Upgrading will probably also better protect you from security problems, so it's a good idea even aside from allowing you to spend more time in Second Life.

Use the 64 bit version of Windows if you can

For each version of Windows for the last several years, you have had a choice between 32 bit and 64 bit variants; if your system can run the 64 bit variant, then you will probably crash much less frequently by changing to it. While we don't have a fully 64 bit version of the Viewer yet, you can run it on 64 bit Windows, and statistically you'll be much better off if you do.

  • Generally speaking the 64 bit Windows versions report crashes half as often as the 32 bit versions.

    According to the data we collect, a little more than 20% of users are running 32 bit Windows versions; most of you can probably upgrade and would benefit by it.

If you bought your computer any time in the last 5 years, chances are very good that it can run the 64 bit version of Windows (as will some systems that are even older). Microsoft has a FAQ page on this topic; go there and read the answer to the question "How do I tell if my computer can run a 64-bit version of Windows?". That page also explains how to do the upgrade and other useful information.

We'll of course continue working hard to find and fix things that lead to Viewer crashes. Even as we do that, though, you can decrease your chances of crashing today by taking the steps above.

Best Regards,

Oz Linden

 

Linden Lab

Hi! I’m a member of the Second Life Operations team. On Friday afternoon, major parts of Second Life had some unplanned downtime, and I want to take a few minutes to explain what happened.

Shortly before 4:15pm PDT/SLT last Friday (May 6th, 2016), the primary node for one of the central databases that drive Second Life crashed. The database node that crashed holds some of the most core data to Second Life, and a whole lot of things stop working when it’s inaccessible, as a lot of Residents saw.

When the primary node in this database is offline we turn off a bunch of services, so that we can bring the grid back up in a controlled manner by turning them back on one at a time.

My team quickly sprung into action, and we were able to promote one of the replica nodes up the chain to replace the primary node that had crashed. All services were fully restored and turned back on in just under an hour.

One additional (and totally unexpected) problem that came up is that for the first part of the outage, our status blog was inaccessible. Our support team uses our status blog to inform Residents of what’s going on when there are problems, and the amount of traffic it receives during an outage is pretty impressive!

A few weeks ago we moved our status blog to new servers. It can be really hard to tune a system for something like a status blog, because the traffic will go from its normal amount to many, many times that very suddenly. We see we now have some additional tuning we need to do with the status blog now that it’s in its new home. (Don’t forget that you can also follow us on Twitter at @SLGridStatus. It’s really handy when the status blog in inaccessible!)

As Landon Linden wrote a year ago, being around my team during an outage is like watching “a ballet in a war zone.” We work hard to restore Second Life services the moment they break, and this outage was no exception. It can be pretty crazy at times!

We’re really sorry for the unexpected downtime late last week. There’s a lot of fun things that happen inworld on Friday night, and the last thing we want is for technical issues to get in the way.


April Linden

Linden Lab

HTTP Project Recap

Earlier this year we blogged about the HTTP project and how, step-by-step, the project is overcoming various limitations. Viewer release 3.4.3 introduced a new HTTP library that made better use of network resources. Texture fetches were the first operations to take advantage of this library, which improved throughput while using fewer connections. But the viewer was still constrained by a one-request-per-connection model.

Changing that model required back-end modifications. Those shipped early in 2013 in the DRTSIM-203 simulator release. For the first time, texture fetches could re-use existing HTTP connections. And for most users, this doubled the theoretical maximum texture request rate.

But all the world is not textures, and mesh fetches were next to receive attention. Meshes required quite a bit more work. Both back-end and viewer engineering was needed, culminating in viewer release 3.7.2. This release brought mesh fetching into behavioral parity with textures.

req.png

These releases have brought the viewer up to the request rate limits of region 'C'. Here, the limits are dictated by serialization, distance, and the speed of light. We are preparing to move beyond this region with changes to concurrency and locality. HTTP request concurrency will be vastly increased by the introduction of HTTP pipelining. Locality will be changed by the use of a Content Delivery Network (CDN) to move texture and mesh data nearer to most users.

 

Pipelining

Common HTTP communication is a simple back-and-forth exchange of requests and responses. A request is issued, a response is returned, and only then is another request issued. As distance increases between endpoints, the time to perform this ping-pong increases, which lowers the effective request rate. Pipelining attacks this distance-induced loss by issuing multiple requests at once without waiting for responses.

The pipelining viewer will use this more aggressive request model for both texture and mesh fetches. This viewer is currently in QA and is expected to go to RC soon after the 3.7.16 release. It has also had trials outside of North America and the results have been very good. The effective request rate has approached the limits imposed by our servers and has exceeded the download rate of UDP texture fetching.

 

CDN

As for those server limits, the operations team at Linden has been making rapid progress on the CDN project (DRTSIM-258). This project will replicate the Linden services that supply meshes and textures to a CDN's PoPs (Points-of-Presence). With PoPs on multiple continents, request service time will be reduced for most users.

Combined with pipelining, CDN experiments are producing results that have only been dreamed about. How fast?  Well, the engineer's universal response of "It depends" applies. But several 100's of fetches per second have been seen far from the USA. This takes us into never before encountered performance territory.

 

And one more thing

The HTTP Project has focused on textures and meshes. But the inventory system, which maintains item ownership, is often described as... sluggish. So as an exercise in expanding the use of the new HTTP library, the pipelining viewer was modified to use it for inventory fetches. As with textures and meshes before, inventory is now fetching in the 'C' region of its specific performance graph. The difference can be surprising.

For several years, HTTP has figured prominently in Linden's plans for Second Life. "HTTP will give you speed and throughput, consistency and robustness."  A promised land, but never quite realized. These next steps are payment on that promise. HTTP done well can support an amazing experience. And you'll have no reason to look back to the V1 world of UDP textures and inventory.

Nous sommes embarqués,

Monty (Linden)

 

Linden Lab

 

Hi everyone! Mazidox here. I’d like to give you an overview of what happened on Wednesday (09/06) that ended up with some Residents’ objects being mass returned.

Two weeks ago, we had several problems crop up all at once - starting with a DNS server outage (a server that helps route requests between different parts of Second Life). Unfortunately, when the dust settled, we started seeing a disturbing trend: mass-returns of objects.

We diagnosed an issue where a region starts up with incorrect mesh Land Impact calculations, which could lead to a lot of objects getting returned at once, as we had encountered several months ago. At that time we applied what we call a speculative fix. A speculative fix means that while we can’t recreate the circumstances that led to a problem, we are still fairly confident that we can stop it from happening again. Unfortunately, in this case we were mistaken; because the fix we applied was speculative, the problem wasn’t fixed as completely as it could have been, and we found out how incomplete the fix was in a dramatic fashion that Wednesday night.

When a problem like this occurs with Second Life we have three priorities:

  1. Stop the problem from getting worse

  2. Fix the damage that has been done

  3. Keep the problem from happening again

We had the first priority taken care of by the end of the initial outage; we could be certain at that point that our servers could talk to each other and there weren’t going to be any more mass-returns of objects that day. At that point, we started assessing the damage and figuring out how to fix as much as we could. In this case it turned out that restarted affected regions where no objects had been returned fixed the problem of some meshes showing the wrong Land Impact.

For regions where a mass-return had happened, there wasn’t a quick fix. Our Ops team managed to figure out a partial list of what regions were affected by a mass object return, which kept our Support team very busy with clean up. Once we helped everyone we knew, who had experienced mass object returns our focus shifted once more, this time to keeping the problem from happening again.

In order to recreate all the various factors that caused this object return we needed to first identify each contributing factor, and then put those pieces together in a test environment. Running tests and finding strange problems is the Server QA team’s specialty so we’ve been at it since the morning after this all happened. I have personally been working to reproduce this, along with help from our Engineering and Ops teams. We’re all focused on trying to put each of the pieces together to ensure that no one has to deal with a mass-return again.

Your local bug-hunting spraycan,

Mazidox Linden

Linden Lab

When I came to Linden Lab over five years ago, Second Life had gone through a period of the coveted hockey-stick growth, and we had just not kept up with the technical demands such growth creates. One or more major outages a week were common.

In my first few months at the Lab, we removed more than a hundred major single points of failure in our service, but several major ones still loomed large, the granddaddy of them all being the core MySQL database server. By late Winter 2009 we were suffering from a core database outage a few times each week.

With a lot of hard work and countless long nights we stabilized the service and started making major improvements to the overall stability and performance of Second Life. However, despite our continued improvements, and the relative tranquility they have created, the spectres of technical debt and single points of failure still loom over our operations. In recent weeks some of them have struck and disrupted Second Life. So much so that I want to explain the outages that have occurred, how we addressed them, and what we are doing going forward.

First, that core MySQL database cluster still exists. It is still the core of many of our central functions. When the write server fails it takes a minimum of thirty minutes to promote a new server into position. The promotion itself is actually relatively quick, but its numerous dependent services must all be taken down and brought back up carefully to ensure that they are all functioning properly.

In the last two months the core MySQL write database has hit two different fatal hardware faults, driving us to temporarily halt most Second Life operations. In some sense, two major write database failures close together is bad luck, but we cannot depend on luck to ensure the reliability of Second Life. In the very near future, we are moving the core MySQL write server to a new hardware class, on which production read servers are already running. Moving the write server will further improve overall database performance and make failures less frequent. It does not of course solve the root single point of failure problem so in the coming days, weeks, and months we will be reducing the impact of database failures even more. This includes continued improvement to the rotation process, extracting more functions out of the core database cluster, and further reducing the number of features that depend on the single write server.

The core MySQL database, however, has not been our only problem recently. A few weeks ago there was a massive distributed denial of service attack on one of our upstream service providers that affected most of their customers, including us, and inhibited the ability of some to use our services. We have since mitigated future potential impact from such an attack by adding an additional provider. There have also been hardware failures in the Marketplace search infrastructure that have impacted that site, a problem that we are continuing to work through. Most seriously though was this week’s four and a half hour long login outage.

On Tuesday morning, users stopped being able to get into Second Life. The root cause was created over ten years ago in a system designed to assign a unique identifier to the hand-off of sessions from login to users’ initial regions. At 7:40AM Pacific Time, that system quietly ran out of possible numbers to assign. It took us four hours to isolate the problem, test a fix, and deploy the change. Users could immediately log in at that point, but it took an additional two hours for systems to settle out. When tens of thousands of users rush back into Second Life following an outage, we have to deliberately throttle some services to prevent further breakage.

Having such a hidden fault in a core service  is unacceptable, so we are doing a thorough review of the login process to determine if there are any more problems like this lurking. Our intent at this point also is to remove the identifier assignment service altogether. It not only was the ultimate source of this outage, but is also one more single point of failure that should have been dispatched long ago.

We want to apologize for all of the recent problems and the frustration they have caused. We too are frustrated and are intent on making our service better. Few things give me more pleasure than every day helping to make Second Life a happy and fun place. Thank you for your patience and support. We simply could not have a more devoted user base and for that we owe you better.

Most sincerely,

Landon (Linden)

 

Linden Lab

Keeping the systems running the Second Life infrastructure operating smoothly is no mean feat. Our monitoring infrastructure keeps an eye on our machines every second, and a team of people work around the clock to ensure that Second Life runs smoothly. We do our best to replace failing systems proactively and invisibly to Residents. Unfortunately, sometimes unexpected problems arise.

In late July, a hardware failure took down four of our latest-generation of simulator hosts. Initially, this was attributed to be a random failure, and the machine was sent off to our vendor for repair. In early October, a second failure took down another four machines. Two weeks later, another failure on another four hosts.

Each host lives inside a chassis along with three other hosts. These four hosts all share a common backplane that provides the hosts with power, networking and storage. The failures were traced to an overheating and subsequent failure of a component on these backplanes.

After exhaustive investigation with our vendor, the root cause of the failures turned out to be a hardware defect in a backplane component. We arranged an on-site visit by our vendor to locate, identify, and replace the affected backplanes. Members of our operations team have been working this week with our vendor in our datacentre to inspect every potentially affected system and replace the defective component to prevent any more failures.

The region restarts that some of you have experienced this week were an unfortunate side-effect of this critical maintenance work. We have done our best to keep these restarts to a minimum as we understand just how disruptive a region restart can be. The affected machines have been repaired, and returned to service and we are confident that no more failures of this type will occur in the future. Thank you all for your patience and understanding as we have proceeded through the extended maintenance window this week.

 

Linden Lab

 

Hello everyone,

The software which we use for keeping track of the bugs found in Second Life is long overdue for an upgrade.  If you've never interacted with the system before, you don't need to start now; however if you are one of the dedicated Residents who spends time helping us improve Second Life via https://jira.secondlife.com then this message is for you.

We are planning the upgrade on Wednesday, August 29, 2018, starting at 8:30 pm PDT.  We're allowing a 6-hour window to give us time to chase down any problems, though hopefully we will be done much more quickly.

If you are interested in what this upgrade actually means, the good news is that most of the changes should be improvements behind the scenes, or cosmetic upgrades.  The look and feel of the user interface will be changing, which isn't surprising since we are going from Jira version 5 to version 7. Importantly for some of you, the new login system actually updates the email address that jira uses for you, every time you log in, instead of only the very first time as the current system does.

The very first change you will likely see is a new login page:

Screen Shot 2018-08-27 at 1.43.09 PM.png

 

If you want to see more about the upgrade, there are a few links on the Atlassian (maker of Jira) website: here and here.

Mention this blog to me inworld and get a free Linden Bear!

Ekim Linden

FJ Linden

As we begin 2011, I want to share the progress that we’re making on several important technology enhancements that I discussed in my last post.  As I mentioned, we are focused on improving the overall performance of Second Life while addressing some long standing limitations such as  raising group limits, improving the chat system, and reducing lag.

Group Limits Raised to 42 Today

In October, we committed to increase group limits from the current 25 up to 40 in the first quarter of 2011. As of today, group limits have been raised to 42! To add groups beyond the previous limit of 25, you must be using Viewer  2.4 (or a more recent version). And if you’re still using Viewer 1.23, or a third-party viewer based on Viewer 1.23 code, then you can add more groups in Viewer 2.4 and they will still be accessible when you switch back to Viewer 1.23.

That said, if there is an unexpected load, then we may need to lower the group limitation to maintain acceptable performance levels across the grid. If we decide to do that, then any Residents who have up to 42  groups will not lose their memberships. But, other Residents will not be able to exceed the new limit.

Group Chat System Will Launch Gridwide By March 31st

We were set to deploy a prototype of the new group chat system in December, but last minute licensing issues were found with our chosen open source library. Now that a solution is in place, we expect to have the prototype available by the end of this month and an industry standard and high performing group chat system available by the end of this quarter.

Performance Improvements When Teleporting and Crossing Regions

As you teleport, or cross regions, all of your avatar data (often a very  large amount of data) needs to be processed by both source and destination regions. In order to streamline this process, we are now compressing avatar information, making your teleports and region crossings faster and more reliable. In fact, we’ve found that teleport failures, due to avatar complexity, have dropped 40%. See the graph below for a more detailed view showing how much we’ve shortened region crossing time.

region_crossing_compression_2011_01_11.jpg

Viewer 2.5 Beta Releasing Soon

We will soon launch the latest version of the Second Life Viewer -- Viewer 2.5 Beta. In addition to more performance and stability improvements, we’ve added enhanced web-based profiles, accessible both on the web and in the Viewer. And, if you wish, you can even connect your Second Life profile to other social identities including Twitter, Facebook, and LinkedIn! You will also have the option in Preferences to choose your first inworld destination from the saved Landmarks in your Favorites Bar. This is very handy when you need to get to a specific destination quickly.

Also, one of the most important new features added to Viewer 2.4 was the auto-updater capability. If you're using Viewer 2.4 or higher, then you won’t be inconvenienced by the notification and download process when we release a new Viewer.

Planning to Implement Significant Grid Infrastructure Enhancements in 2011

We’re planning significant grid infrastructure enhancements throughout the year including technologies to speed server-side rendering (SSR) and server virtualization (web and simulator services). We are also exploring new storage and asset delivery systems. Some of the benefits will not always be noticeable, but they are foundational platform changes that set the stage for rapid performance and scalability improvements. We will continue to keep you updated as we roll out these systems.

I’m pleased with the progress that we made across the platform last year and I'm looking ahead to newer technologies that we will deploy in 2011 to enhance your Second Life experience. As always, I'll be watching for your feedback and thank you for making Second Life such an amazing place.

×