Is Group Chat lag fix EVER coming??

[Sarrah Windlow]

Over 6 months ago (somewhere around September of last year if I remember right actually) we were told there was a fix for the Group Chat lag problem being worked on.  However Group Chat lag is still just as bad as ever and with Mesh going gridwide sometime in the coming months I think, wouldn't it make sense to have the Group Chat lag problem fixed first??  I mean it's only going to piss off people if they can't use Group Chat to get help for their mesh projects because their posts keep getting eaten by the server instead of actually making it through to people who might be able to help them while they're inworld.  Prettying up the 'Join' process is all well and good but if the tools residents rely on to make their SL experience enjoyable aren't working what's the use?  People are only going to join, then find out that all sorts of things are broken and then just end up leaving to go to another virtual world that doesn't have these problems (or just giving up on virtual worlds altogether maybe).  Prioritizing things to do to improve SL still appears to be a VERY MAJOR problem for the SL staff, you need to get the underpinnings fixed before you go adding bling, this is just common sense, fix the basics then worry about making improvements, any decent programmer will tell you the same thing.  Why is this concept so hard for the SL programming staff to grasp?  Companies like Adobe and Autodesk and Microsoft all understand this, why not Linden Labs and Second Life?

[Imagin Illyar]

Sarrah, I rarely complain but I have to agree.  It seems like this problem has been going on an embarrassingly long amount of time.  You'd think that it would be given a higher priority by now but it's like their pet bug.  There are plenty of important things to be worked on but group chat is a pretty basic element that you'd think they'd have perfected in a virtual world that has been running as long as SL has.

[Minerva Dover]

Yes, the group chat is basically non-functional, leading to my getting more and more mass TPs, which, as you know, can be aggravating (especially when all your friends are DJs). Please get this fixed tout-de-suite!

[Nalates Urriah]

Chat has problems and group chat is especially problematic. I too am so tired of getting chats bounced back because they can't connect or arrive out of order when they do arrive.

Linden Lab knows about it. A whole new chat system based on XMPP was built. The system was rolled out for testing on the ADITI grid. Support for it has even been rolled onto the main grid. Some TPV's even have support for it (Kirsten's S21(7a). 

The problem is the more it was tested the more it was obvious it was not going to fix SL's chat problems. So... the whole thing is a huge disappointment... to us and to the Lindens. Rather than give us something that doesn't work, they went back to the drawing board.

However, it is not a total loss. The Lindens learned a lot about the chat bottle necks and accumulated lots of stats. A new plan is in process. The Lindens aren't saying much about chat, which in my experience means they are not confident they have a solid solution. Everyone knows it is a problem. Linden Lab sees it as a priority. 

I suspect that since chat is such a common computer task the Linden's may have underestimated the problem. Or may be not. Whatever the case is, SL chat is a more complex creature than any of the other social net chats I use. I suspect they have a number of people figuring out a system based on the new knowledge.

[Void Singer]

the main problem seems to be user joins and user exits... the more there are, the worse it lags... you'll notice often that after a new chat opens, there seems to be tremendous lag at first that tapers off to just spotty lag as the conversation continues... unfortunately when the chat goes idle for a bit it all happens again.

XMPP was thought to be a cure, because it does handle high volume of traffic (lots of groups)... unfortunately it doesn't seem to be up to large groups which it was previously untested for, and large groups are the major problem area. According to Oskar, the project was killed when that was discovered, although alternatives are being looked into. According to Andrew today, the lead position either isn't filled, or has changed hands again (which per his comment seemed frequent)

Oskars Beta User group meeting on aditi seems like the best place to get the current status of the group chat project (Andrew was unaware of XMPP being a dead end) which is held on thursdays in Morris, on the beta grid.

[Sarrah Windlow]

The fact that none of this information about the XMPP attempt ever made it to any of the official status updates is another indication of the communication problems that have been plaguing the SL staff for the last couple years now, if they really want to improve the lines of communication with residents then this sort of information should be provided as well.  If they had bothered to release a notice stating that they had hit a minor roadblock but that other options were being pursued then people would at least know SOMETHING was being done.  As things stand right now though with no (or extremely sparse) official updates on any of the problems with SL people are left to wonder if the programming staff is even bothering to do ANYTHING at all about the problems or if they're just choosing to ignore them hoping they'll go away while they work on adding even more features that nobody asked for or wanted.

[Sarrah Windlow]

The really messed up part about all of this is that until about a year and a half ago (maybe 2 years tops) group chat worked just fine, I never had posts just disappear, they always made it quickly into the chat window where others were just as quickly able to reply to them.  And while I know that the number of residents online at any given time has increased a bit since then I don't think that's what's responsible for the current state of group chat, instead it's like someone flipped a switch on the server one day labeled 'group chat fu** ups on/off' to 'on', it just pretty much started acting up suddenly one day leaving me to think that something was changed in the code and now needs to be 'unchanged' in order to fix the problem.

[Void Singer]

and if they had announced they were testing XMPP? you'd see people complaining about how nothing ever get done when it failed, or how they must be inept for it to fail, or for not knowing in advance that it would fail. and of course the very loud "But you said it was going to be fixed"... instead they tested an option quietly, and information did trickle through that they were testing options (and I assume will be testing others if they aren't already) so that people would know that yes, it is a problem they are aware of and searching for a solution to.

But they don't want to make a big announcement about something they have no clue on whether it will work.... no one had tested the that for the unique kind of use that is SL groups..... ever notice that you don't see much in the way of large groups actively chatting away in a single live space.... anywhere on the web? and by large I mean hundreds to potentially thousands at one time? then multiply that by hundreds of those groups at once, ranging from large to small.... then do it over an international natwork from mutliple data centers.... and make it easy to increase or decrease in size, and not waste gobs of resources or be glacially slow... that's what they're up against.

[Sarrah Windlow]

You know I for one (of probably thousands of SL residents) would still like to have known that they WERE actually trying to do something about it, EVEN if it didn't work out, knowing something is better than knowing nothing in my book!  I shouldn't have to go to office hours that are often scheduled at times when I am unable to be inworld just to find out that yes they are actually doing something instead of ignoring the problem.  Regardless of whether it worked or not keeping residents 'in the loop' is what I'm talking about here, and SL has ALWAYS had a problem with doing that, and yet every so often they claim they're 'going to improve communication so that there is more transparency', and yet invariably they never truly do.  I've been an SL resident for around 4 years now (counting my time on another AV before this one) and up until a couple years ago I was very happy with the way things were being handled, then what's his face took over and SL went to sh*t as far as I'm concerned, they shifted focus to trying to get businesses to use SL and essentially turned their backs on the average user for quite a while.  Then Philip came back for a while and everyone thought it was like the return of the messiah and expected everything to go back to the way it was or better (even I wasn't that gullible), and now we get all these messages saying things are going to be addressed per user concerns instead of just big business and yet we still don't see the sort of transparency that has been promised to us time and time again since 'what's his face' left, so EXCUSE me for wanting them to actually live up to all the promises they've made over the last so many years, unlike most of them I'm actually inworld on a pretty regular basis and actually see the state of things in many locations including how group chat has gone down the toilet not slowly but as if someone pulled the flush lever one day and apparently never looked back.  I see things breaking daily as they roll out new server code that only puts small patches on the underlying problems instead of actually rewriting the server code to eliminate the source of the problem.

 

But regardless of that, I still say that 'Knowing is better than NOT knowing' EVEN if it does fail!!  At least we'll know they are actually trying then!!

[Void Singer]

give an inch, they want a mile....

have you considered that making the information available at all, whether it's convenient for you or not is 100 times better than the standard in the gaming industry? wher you're lucky if they even acknowledge that there is a problem, and when they do it's usually after they've fixed it?

I'm not saying they can't do it better or shouldn't try, there are suggestions about ways to make the information more convenient to get to, for instance.

have you ever seen an instance where LL (or any company really) just ignored a known problem, instead of working to do something about it, or at least stating plainly that they can't or won't?

have you noticed that yelling and complaining doesn't get us very far, but when people step up and suggest methods to get what they need or want, suddenly things start to get done? vague complaints such as "I don't like this" or "it's not convenient enough don't give them a direction to go in... saying things like "We need meeting summaries posted to forums" or "This that and the other are important issues that we need regular updates on" do, and tell them how we want them.

and seriously, if you think rewiting the server code is the solution to all our problems, then may I remeind your of how well that worked out for the viewer code? V2 was initially promised shortly after your entry into secondlife, and took ~2.5 years to finish, and when unleashed on the populous was so bug ridden that it was a bad joke... because there was no way to test widespread use in such a random and dynamic environment as SL... now imagine the same for the server code, and try not laugh too hard.

[Sarrah Windlow]

And have you considered that Viewer 2 wasn't conceived as simply a rewrite of the viewer with the intent of fixing bugs, it was a marketing ploy aimed at drawing in the 'browser idiots' who can't do anything on their computer unless it's in their web browser.  So the comparison isn't really valid in this instance.  I will however grant you that they do a better job of telling users about certain things than many of the MMO's, but by and large those same MMO's don't take years to fix the problems that exist in their game products, they know that if they did it would cost them subscribers.  Let's also not forget that regardless of some similarities SL is NOT a game platform, it is a virtual world that is constantly evolving and is shaped by it's residents, and those residents have a right to expect quality service, but is it 'quality service' when there are bugs in the jira that have been there for many years now, or when they roll out new server code and it breaks several basic things just because they didn't bother to do adequate testing, or when they change search such that it drops listings for 1/3rd of it's resident businesses and take 9 months fixing it when they could have rolled it back to the old method until they got it fixed on the beta grid?  Short answer, NO.  You don't find those sort of problems afflicting MMO's like World of Warcraft and others, I know this from experience because I was a WoW player for even longer than I've been in SL.  Is it so wrong to ask that they try to get their act together and start coordinating things better AND keep residents informed?  I don't think so and I'll almost guarantee that most of the residents that have noticed the existing bugs and have sat patiently by waiting for them to be fixed would like to know that something is actually being done to fix the bugs.  And what do they do now, they remove voting from the jira making it next to impossible for them to prioritize which bugs resident's want fixed first!  Again you don't see this sort of behavior from the major MMO's, they actually embrace the input from their users for the most part.  So yes I DO want to have my cake and eat it too!  I do want to know if the service I spend money on is actually doing something to help improve my experience!  I do want them to tell me if something they tried recently didn't work so that at least I KNOW they are at least doing SOMETHING!!  It took them YEARS to 'fix' most of the TP failures and now what happens most of the time when a TP fails?  YOU GET LOGGED OUT, instead of just sent back to where you started to try to TP from, do I consider this a 'fix'?  NO, and most of the people I talk to in SL don't either, but yet we're stuck with it all because instead of rewriting the sections of code that deal with TP's they apply patch after patch after patch such that it makes the TP process take many times longer than it used to and as a result things 'timeout' and the server gives up on the TP!  Once again I say that you don't see this sort of behavior from MMO's though much of the same sort of transfer occurs in the background in games like WoW when you are 'flying' to other places in the gaming world, again they know that if this sort of thing kept happening they'd lose subscribers, and I'll guarantee that SL has lost a great many residents because of the problems they either have not addressed yet or some of the ones that took them much longer than it should have taken them, like the whole search debaucle, or the problems with getting the marketplace up and running, do you have any idea how many content creators and business owners pulled out of SL because of these 2 MAJOR problems?  I've heard tell it was close to 20% of all the businesses and content creators in SL!  All because they couldn't be bothered to roll things back until they actually managed to fix things properly!  So yeah I don't have much confidence in the programming teams abilities at this point which is one of the reasons I want to be kept informed about the progress (or even lack of) on fixing the numerous bugs that they introduce with each new viewer feature that nobody wanted in the first place!  You may not like that fact but it's not up to you whether I want to be informed or not, nor is it up to you if anyone else wants to.  People have made suggestion after suggestion after suggestion after suggestion after suggestion until they're blue in the face on some things that the SL programming team needs to do (and that LL as a whole should change in regards to SL) but in the last couple years all I've seen is things continue much the way they've always been done paying no attention whatsoever to those suggestions.  So I for one am pretty much done offering' suggestions' I've done so in the past (I even had a ticket rejected that was started to help them to realize that there was a problem with the SL website itself!!).  I'm done trying to coddle them and hold their hand.  It's time they actually started doing the many things they've been promising to do over the last few years, and if you don't like that then tough luck, it's my opinion, one I'm free to express if I choose.

[Herman Gears]

Damn, wall of text aside, damn. Yep, mhm, yep, yes, alright, and okey dokie. Damn.

[Void Singer]

tl;dr

next time please use paragraph breaks

[Jenni Darkwatch]

XMPP has its own set of problems, many of which are reasonably well known by anyone running a large XMPP server. To me, it would have brought interop to the table, though I never saw it as a straight solution to group chat.

Historically, there's never been any group chat solution able to handle truly large groups. IRC didn't, instant messengers didn't, XMPP doesn't. Even MMO group chat suffers similar problems - lag being the prime problem.

Without knowing how LL handles chat server-side I won't bother to even come up with any suggestions myself. Right now I suspect(!) things might be especially sucky simply because of futzing with chat code. I do wholeheartedly agree that group chat is for all practical purposes unuseable right now. I'm guessing it'll improve in a year or two, simply because that particular issue does not seem to be easy to solve.

[Oskar Linden]

Allow me to jump in here. I will apologize first off for not reading every comment in this thread. Let me summarize where we are with all of this. Fixing group chat is a very high priority for Linden Lab. It has never been off our top 5 list. We thought we had something going with the XMPP project and prematurely gave a fix-by date. After we attempted to implement the fixes as we had architected them it failed to provide the improvements that we needed to see. We couldn't move forward with it as it was. 

We have continued to investigate solutions to the group chat problems. We're not even ruling out the possibility that XMPP may have a new role in a future solution. We have put in a lot of time this last month coming up with some new ideas. Hopefully very soon we will be able to communicate our findings and ultimately have something for you to try out on ADITI.

We appreciate your patience.

__Oskar 

[Jenni Darkwatch]

Thank you for the update.

I'm wondering if testing on Aditi will be sufficient, considering that groups on Agni are probably much, much larger. Of course, I might be talking out of my ass as usual, since I haven't got the foggiest what testing procedures you have in place.

Large group chat is a tough nut to crack, though I'd guess if you'd lay out where your specific bottlenecks are there'll probably be a few constructive and useful ideas from the community. There do seem to be a few knowledgeable and experienced geeks in the audience.

[leliel Mirihi]

---

Jenni Darkwatch wrote:

Historically, there's never been any group chat solution able to handle truly large groups. IRC didn't, instant messengers didn't, XMPP doesn't. Even MMO group chat suffers similar problems - lag being the prime problem.

---

 

IRC could handle the vast majority of groups in SL just fine, it's only truly large groups that would still have problems (as you point out). But in those cases you'd be bumping into the limits of the hardware so there's really not much you can do about it. However as anyone that has been in a large IRC channel can tell you, it's not exactly easy for a human to keep up either. So it's debatable as to whether we really want to fix this problem, especially since fixing it could very well require a fundamental change in how CPUs work.

I think a better solution would be to expand the group communication tools so users don't need to chat with 1M people in real time as often.

[Void Singer]

someone reminded me of it recently in another thread... I believe IRC has been tried, and netsplits caused it to be unworkable...

whatever they are using now seems to have a heck of a time with user joins and disconnects, as that's where I see the worst of the lag.... once into the initial few minutes and those have mostly stopped, chat seems to be fine... at least until the connection goes idle and then it happens again.

allowing users (or even group managers) to specify "only connect on user demand", "connect at login/first instance", and "always connect" might help cut down on some traffic and spread the joins out a bit, and is certainly better from a user perspective than the currently hacked in behavior to only reconnect on demand after closing.... there are groups I'd love to set on-demand for at all times, and others I don't dare close, because I might miss something important after they go idle, despite the clutter.

those suggestion are obviously only a potential interim patch, that MIGHT quell some of the lag, but it's a thought.

[Jenni Darkwatch]

IRC can handle millions of groups - just about any chat medium can. Groups with as few as a hundred people in it on the other hand can be problematic.

Not sure what the biggest groups are in SL - considering peak concurrency is hardly above 50k users, I doubt any group has at any given time 1000 of its members online. I might be wrong.

Voids solution might be the most sensible. There's plenty of groups where I couldn't care less about the inane blubber on group chat, where I only need the notices. Yet there's no setting to disable group chat only for this group (TPV hacks aside). Especially since he hit the nail on the head... the obscene amount of info involved when someone joins or leaves a group chat session is one of the bigger causes of lag.

OTOH, sending a list of users to every user doesn't seem like it could cause problems, but it definitely can and does cause problems. Try looking at any group membership list and you can see it for yourself. Another way to alleviate the issue therefore might be to simply not keep that list updated or even send it at all unless a client/user specifically requests it.

Caveat emptor: I know nothing about how SL group chat works internally, hence all I wrote was hot air. Or maybe not even that.

[Void Singer]

---

Jenni Darkwatch wrote:

[...] Not sure what the biggest groups are in SL - considering peak concurrency is hardly above 50k users, I doubt any group has at any given time 1000 of its members online. I might be wrong. [...]

---

I know that Beta Server group has something like 3k.... and I don't even want to imagine what group like XCite Premier have (don't look at me like that, I'm a scripter, I need to be able to get answers on products that people want me to make work with theirs, which incidently I'd love to ignor the chat on while retaining the ability to ask as needed)

to be fair, my proposal would require changes to the viewer and server code that are probably non-trivial

[Jenni Darkwatch]

---

Void Singer wrote:

to be fair, my proposal would require changes to the viewer and server code that are probably non-trivial

---

I'd imagine there's already a mechanism to exclude offline users, there almost has to be. Might be possible to expand on that pre-existing code. Without knowing the server code tho... dunno.

[Void Singer]

---

Jenni Darkwatch wrote:

I'd imagine there's already a mechanism to exclude offline users, there almost has to be. Might be possible to expand on that pre-existing code. Without knowing the server code tho... dunno.

---

 one can hope (heavens know I do), but if there isn't it might explain a few things =X

[TriJin Bade]

Group chat lag is now getting to a point where it's killing the Community.

How to obliterate a resident community and drop users so they go elsewhere? Simply give them LAGGY GROUP CHAT

Oh come on Linden Lab... It's not hard to do. I even had to create a website chat for our RP community in Legacies cos the chat lag is SOOOO bad in groups. It took me 2 days to make it. It takes the Lab 8 years to not even come close to a fix.

So, if Linden Lab want residents to come into Second Life and be part of a community that embelishes ease of communication and creativity - FIX THE DAMN GROUP CHAT.

Users NEED to communicate. You've made this forum, you've made Marketplace, you've placed in-world chat features. Surely the basics is... to work. If I had you on my Software development payroll.... [in Trump Voice] YOU'RE FIRED!

But seriously.... Group Chat NEEDS to be fixed NOW!!! If you're working on it, then TELL US!

Give us a day by day progress on what you are doing. Forget Mesh, we can wait (although the money grabbing merchants won't LOL). Forget shiney Windlight regions. Forget bloatware. Fix the basic thing that residents NEED...

... Breaths

Okay.... there's my feedback. Enjoy 

^V^

 

 

[TriJin Bade]

I was actually PLEASED to finally see some step forward to group chat lag fix. Now let's see how long it takes after they've measured the shortest distance between two points.

They are asking for helpers on the beta grid, so if anyone is wanting to help out Linden Lab fix the damn thing, then see here:

http://wiki.secondlife.com/wiki/GroupChatTest

One step forward...  even though it's a baby step.

^V^

[Eio Tuqiri]

Concerning the July Update for 2011 and the section on "Group Chat Lag", a quote from the post "we don’t see that dude around much these days".  I’d like to know where you’re getting your information, because there is severe lag every time I’m in group chat and that’s pretty much any time I’m logged in. 

The Group Chats I use are User Groups and New Citizens.  Groups that function to help users with problems, and it is very frustrating to have things show up out of context or not show at all. Multiply that times how many users happen to be in the chat and it's difficult to get answers to questions or know who is saying what about what to whom.

Come on Linden Labs, work on these things that are broken, have been broken for a long time, that are important to users, and get them fixed before you start bringing on new features like V2 web based search that are severely broken at launch, creating more things that need to be fixed.