Jump to content

Tools and Technology

  • entries
  • comments
  • views

Contributors to this blog

Back to Basics Technology Improvements

FJ Linden


As  we head into the fourth quarter, I wanted to share some of the progress  that we’ve made since we realigned engineering and created our Platform  team. The team is comprised of simulator, services, and viewer  developers and testers (Platform Development), as well as our  operations, IT, data warehouse and customer support teams (Platform  Operations). The focus of the teams will be to drive “back to basics”  improvements across the grid and evolve our current architecture into a  scalable set of services and APIs. Let’s take a look at our progress  over the last few months, delivering against the goals that Philip set  at the Second Life Community Convention (SLCC), in August.

I’m  happy to report that we have already delivered on a number of those,  and are making good progress toward the rest. A few of them may take  longer than we’d originally anticipated, but we plan on completing all  of them.

Below  you’ll find a list of projects and initiatives we’ve delivered on in  the two months since SLCC, as well as some back-up performance data. A  number of these projects will have a major impact on the Second Life  experience, as we currently know it. Expect to see further posts from  some of the Platform team, focused on how we are improving Second Life  in terms of faster performance, increased stability, and a generally  smoother experience for all.

Reduced Lag
Here’s  an interesting factoid: there are about two million teleports in Second  Life every day. Previous to our recent release of Server 1.42, when an avatar teleported or crossed into a new region, everyone on the  destination sim would experience a “lag” event as the simulator stalled  while processing the incoming avatar. This was often experienced as  “jitter” on the sim, especially evident when many avatars arrived at the  same time, such as for a live event. In the new simulation code, this  slow point has been moved to a separate thread. Our simulator  performance profiling tools show that this lag pain point is almost  entirely gone, greatly improving performance for highly trafficked  regions.

Faster Texture Loading
In August, we made some changes that improve the speed with which textures are loaded in a scene. Our release of HTTP Textures changed the way textures are delivered to, and loaded in, Viewer 2,  resulting in less waiting around for the scene in front of you to come  into focus. This change will also serve as the foundation for a series  of bigger improvements, including the deployment of a new asset system,  which will improve texture loading and object rez times even more. For  the real techies, here’s the back story: Last month, we moved away from  the Isilon storage clusters, as our primary storage for assets. Most  assets are now stored on Amazon’s S3 platform, and we have deployed a  “middle-layer” caching service, which is where most asset calls will be  served from. The time to complete asset “puts” and “gets” (the times  that the viewer requests assets and times those assets are delivered)  has improved significantly. We reduced the average PUT latency by over  60%, from 1.3 seconds to 0.6 seconds, and our GET latency by 20%, from  0.25 seconds to 0.20 seconds. What does this really mean? Well, when you  add up the cumulative time saved in asset latency across the millions  of requests a day, there is a net savings of over 4,000 hours of time  shaved off of asset latency per day! That translates into less load on  the simulator processes and better rez times in Viewer 2.

The next step here is to get the simulator out of this process,  and that will happen in early 2011. At that point, we’ll have extended  our asset services and can begin enabling a content delivery network  (CDN), bringing latency-sensitive assets (objects and textures) closer  to Residents for even faster rez times.

New Chat Service Coming Soon
We’re  finally going to tackle the group chat problem that has been a Resident  complaint for a long time. Group chat can often be confusing, with  “chat lag” causing responses to appear late, or sometimes not at all.  You should start to see real improvements in this early next year. Over  the last few months, we’ve completed a number of development sprints to  prototype an XMPP service and have decided to move forward with an ejabberd deployment.  We’re targeting to have a test deployment of the new group chat service  by the end of this year, and full production deployment in early 2011.

Group Limits Will be Raised to 40
Another  big pain point for Residents is the current limit that lets you be a  member of only 25 groups. We plan to raise this limit to 40 by the end  of this year. In the past, our biggest concern about raising group  limits was potential performance degradation, with additional stress  placed on our central database. After completing some internal analysis,  we now feel comfortable enough to extend group limits up to 40. I’ll  put a qualifier on the group limits increase, however, and state that if  we see a decrease in performance (i.e., more lag), then we may decide  to roll back to the 25 limit again.

Snowstorm Driving Viewer 2 Improvements
At SLCC, we announced Project Snowstorm,  our new open development program for Second Life’s Viewer 2 client  software. Snowstorm is already incorporating Resident input,  particularly around a more customizable user interface, has produced a  number of beta releases, and is deploying daily open-source code  releases. You’ll see the first fruits of their efforts in production  very shortly, with the release of Viewer 2.2.

New Main Grid Deployment Process
It  was pretty clear that if we were going to make velocity improvements,  then we needed to be able to update our server software more than three  to four times per year.  As Lil Linden discussed last month,  we have significantly changed our deploy process and are now deploying  code on a weekly basis to the main grid (Agni). This means that bug  fixes will be deployed faster and new features and functionality don’t  have to pile up and wait for the next major “release train.”  It also  means that if we do encounter problems, then we are able to quickly back  out the changes, without affecting other code that is working properly.  This process also works much better with the agile development  practices that we have formalized internally over the last three months.

Display Names and Mesh Public Betas Available Today
While  we’re focusing our resources for the moment on platform improvements  designed to make Second Life faster, easier to use, and more fun for all  Residents, we are also testing two new features that had been in  development for some time before SLCC. The new Display Names feature is in public beta, and will enable more freedom of expression in Second Life. We have also just launched the public beta of Mesh Import,  which will revolutionize content creation in Second Life. So I  encourage you to download the Project Viewers and share your feedback.

I’ll  share more with you on our “back to basics” technology projects in  coming weeks and months. In the meantime, I’m looking forward to hearing  your thoughts on our recent progress in the comments below. And, as  always, thanks for all of your passion and participation.


Recommended Comments

Nice to hear that the group limit will be raised. It was long overdue.

However the whole section about "Reduced Lag" is not very accurate. The sim freezes that you call "lag events" are still there and there was no visible improvements in server 1.42. Perhaps that is something in the pipeline that hasn't made it into the release server yet?

Link to comment

Horray on the group limit increase.  This will be a major help for me - and should save me some frustration. I also look forward to seeing what other changes may be down the pipeline for group functions.

Link to comment

Thanks for the imteresting update

I'm sure the group increase will make many people very happy

Related to this I have a couple of comments for your consideration:

1) Under the recent announcement about Avatar's United closing it was stated that parts of its technology would be ported over to SL. Has it been considered how the 'Groups' functionality could be used. It would be fantastic if you guys come somehow develop 'offline' groups functionality in a similar manner to what some third-parties offer that would allow for management of groups both 'offline' and from within the viewer. Perhaps this may even give opportunities for removal of group limits altogether?

2) I have notced some interesting 'lag' issues recently when opening groups with a large number of members. As the viewer struggles to populate the large members list my frame rate plummets to near zero and SL becomes temporarily unusable until the members list has populated. As soon as the list has fully populated my FPS climbs right back up to its normal 60+FPS and all is well again. Is this related to the group chat lag issue, or is it something else that needs to be looked at?

Link to comment

Finally ! Some news from LL thats actually positive and genuinely what users have been asking for. 40 groups and fixing chat. Forget the shiny, getting the nuts and bolts to work right is EXACTLY what the platform needs.

Thank you Linden Lab, this single blog post has done more to restore my personal confidence in the platform as provided by LL than ANYTHING else posted this year.

Link to comment

Absolutely excited about ALL of this! Except for Display Names, about which you people have been told AD NAUSEUM no one but a very small minority wants, for a lot of very good reasons ... the rest of the stuff has been a very long time in coming and I'm personally stoked to see it on the list. Excellent!

Link to comment

Increasing groups inworld is just increasing issues inworld. Groups should be for object and land management.

Notices, subscriptions, chat and announcements need to leverage the dashboard on the web site to deliver such content, giving residents the option to enable or disable any or all of the above at their discretion.

The dashboard was heralded as providing widgets at some point that would allow this to take place and once again, a 180 degree turn is taken and something started is something kicked into the dustbin.

It appears increasingly obvious that the only thing adhered to vehemently are the projects that residents have disapproved of resoundingly and that's not a good thing. Not good at all.

The war against lag is a step in the right direction. Everyone is in agreement on that, but why get a handle on the issue only to nullify it with additional groups?

Region Owners have asked for an addition to the dashboard that will allow them to manage banlists and change estate settings from the web site for years now. No real mention is made of the Lab doing anything about that here other than a possibility of APIs in the future. With projects and plans taking 180 degree turns with great consistency (improved default walk, speech gesture, pushes toward education and inworld music resources - all of which were heralded and announced on the blog with great fanfare) where are the assurances that these current plans aren't simply another cry of "Wolf!"?

Link to comment

Very interesting developments, specially the part about CDN.

One thing, if you are ready, go public and grab the new ones. I know they must be out there. Just grabbem, tie-em and signem so they never leave.

My message to Torley about the Marketplace dev team is still standing strong.

Link to comment

I'm running the beta of the new 2.2 viewer and it's improved dramatically in the short couple weeks it has been out.  It has gone from a crash festival to completely stable.  And yes, I have noticed less lag in my sim.  Getting more groups is a welcome change as well.

Keep up the good work.  We have been wanting a number of tech improvements for a long time.  While it doesn't address the dreadful leadership of the organization, it does give us hope that the tech side of the system will finally get some much needed repairs.  And I, for one, am glad to finally see these repairs and more like them in the future.

Link to comment

It appears increasingly obvious that the only thing adhered to vehemently are the projects that residents have disapproved of resoundingly and that's not a good thing. Not good at all.

We never asked for a dashboard, we have asked for more groups forever.

The use of groups right now is limited by the lag inherent in the chat system, make group chat work and you might find that the use of groups will adapt to be more then just announcements.

Link to comment

This is all in good, but will any of this improve the teleport as a whole? Most of the times, even with the lightest of scripted items it is impossible to teleport anywhere, even back home. I have to mainly log out and relog just to get back home or log out and type in the region name just to get there. Even when I am the only one on the Sim and the script timing on the Sim is moderate. The only scripted items I am wearing at most is my hair, my shoes are off, my HUDS are detached and have on something simple as a bikini!

Link to comment

Great News! I hope we get more web-based widgets for our Dasboards that incorporate some of the AU technology.  Would love to Blip and Photoshare to it.

Link to comment

I'm not sure who came up with the name Snowstorm, but being a Canadian I can tell you that when a snowstorm hits, you can't see anything except white.  o.O

Link to comment

Ohh that looks like awesome news!

Group problem solved looks like the best of all to me. That would be cool if i do not need to make separate groups for every role and can actually moderate our too big group again to chatter with all our fans.

Link to comment

Thank you for the update. I wish I could be as positve about lag --and sim crossings.

I experience lag in sims with .99-1.00 Time dilation and 0.00 images cache --where the frame rate is dropping down to single digit and the sim is restarted almost if not daily.

In last week --lag so bad --again in a sim with great numbers--was "tossed" half-way into sim next door. All I was doing was flying.

Driving a well scripted vehicle between sims--get tossed under the terrain --then flipped in the air and then finally land . Makes me feel like a figure skater.

I was only avatar in either sim --no child avatar issues.

Also, get the sim walk --uncontrolled movement-- in many sims.

Again, this is all from * this *week .

Hope your monitoring of performance can pick up and fix these very very annoying not "fun fast easy"   --- "features."

And 40 for group membership --great -- since many groups now charge for membership (stores) --please keep that in mind if you decide to rollback groups from 40 to 25. People will then have lost money on the increase .

Link to comment

Yay for increasing group limits!

As for the sim lag when folks teleport in...i dunno what sims youve been testing it on, but that lag spike is still happening..

Link to comment

RE: "I’ll  put a qualifier on the group limits increase, however, and state that if we see a decrease in performance (i.e., more lag), then we may decide to roll back to the 25 limit again."

FJ, you puzzle me there. Is that a reflection of the American culture of all or nothing?

How about just giving us 30 for now, and not having the risk of roll back? I can't imagine for you to be able to roll back like that, and ask people to kill groups once they have more than 25. Unless I am missing something in your reasoning, the possibility of roll back seems messy and a nightmare to handle on both sides.


Link to comment
  • Create New...