Group Chat Ongoing Conversation

[Dottie Linden]

Group chat is a pivotal and important part of helping Second Life communities stay connected, and we've been working on improving its reliability and function for some time now. The challenging part of that is primarily due to how large and complex the system is. It may not seem too complicated from the outside, but inside the group chat apparatus it's busier than the mailroom at the North Pole at Christmas -- every message sent to a group chat has to be relayed as quickly as possible to every online member of that group. For a smaller group, this isn't too difficult, but for some larger and more active group chats this can be hundreds or even thousands of people.

We have made some teaks to improve reliability and deliverability and to more efficiently balance the load placed on the group chat systems. However, we are aware that we are still experiencing intermittent issues with group chat. This is a feature we are continuing to work on in the long term and we would like to hear Resident's ongoing feedback to help us to not only maintain but improve group chat as a feature. As such, we feel that this subject is more suited to a forum communication than to an outage-like alert on the blog.

Whenever you may experience an issue with group chats, please feel free to also reach out to support via a support ticket at https://lindenlab.freshdesk.com/support/home!

[Mister Acacia]

I haven't seen many issues and have not personally experienced any for several days. I had been reporting on https://lindenlab.freshdesk.com/support/tickets/1683919

[Miro Collas]

I had a post drop 2 days ago. Otherwise, as far as chat goes, things have been better.  [crosses fingers and toes]

[BryanR Rickena]

It's been a while, but I on rare occasions have opened a group chat and it errors out with unable to connect, so I let it close and start again and normally does great. It is even rarer to have it not connect twice.

[Mister Acacia]

[13:22:27] [Second Life: The message sent to Firestorm Support English is still being processed.
If the message does not appear in the next few minutes, it may have been dropped by the server.]

I got a dropped message error multiple times, then tried to close and re-open the group and got the upable to connect/sync error

[CarlaWetter]

And I haven't seen a lifesign from Firestorm Support English chat at all the last 13 days. Peaceful days, but not what I'd expect from that group chat.  Did mute notices around that time in FS, but chat still shows as unmuted.

[Mire Iwish]

all my land groups still like this Join Group Chat failed.
Cannot send group IM.
Join Group Chat failed. more then 2 days now

did opened a ticket and no answer it is happening alot

 

Edited May 22, 2021 by Mire Iwish

[Flyer Beaumont]

group chat isn't working for me at all!

[Skell Dagger]

As of today, the same issue that I experienced between 14th and 19th April has started again. I blogged about it here, including some of the testing that I carried out. Group chat was fine last night, but this evening my avatar is not subscribing to group chat, as before. Relogging doesn't help, and I didn't expect it to, as it didn't help the last time this happened either.

This is what the participants list looks like when I initiate a chat with the (normally very busy) Catwa Head Friends group, of which I'm a moderator:

There should be several hundred names in that list, not just mine.

This is what the Members list shows me:

What that 'Everyone' role should show:

Edited May 23, 2021 by Skell Dagger

[WereBeast Alpha]

On 5/20/2021 at 1:57 PM, Miro Collas said:

I had a post drop 2 days ago. Otherwise, as far as chat goes, things have been better.  [crosses fingers and toes]

Same had a few of my posts drop a couple days ago and I lost the whole paragraph for an RP entrance in a group. I also tried teleporting last week and ended up log crashing my region was open to the place I was going it was a bit annoying had to wait 10 mins to log back in because it said attempting to log out of region but it ended up sending me back TP home to the hub landing area.

[Aishagain]

Just to add my experiences over the past 2 or 3 days:

Many group chats work until I teleport after which I receive no further messages and the moderators for that/those groups have vanished from the lists. In my case a relog usually (though not always) restores the function.

My chat appears to me whether those groups are working or not, but no-one else sees it.

One group, of which I am a moderator, seems to work normally most times but TP always kills it.

[Mister Acacia]

[10:40:53] [Second Life: The message sent to Firestorm Support English is still being processed.
If the message does not appear in the next few minutes, it may have been dropped by the server.]

[Dottie Linden]

Hi everyone!

We are still collecting information on groups that still seem to have trouble more than others, so please go ahead and submit a support ticket with your group's name! 

Thank you!!

[Aishagain]

If my experience is in any way typical, Dottie, you must be knee deep in tickets by now.

[AlettaMondragon]

It didn't get any better yet, to be honest. It is still as unreliable as it has been since last August. No, not "as it always have been", because the massive fail of group chat is definitely rooted in some simulator server and agent presence communication issues, which began in August 2020. Fortunately in the last month, for me all group conversations work mostly fine, once they start to get delivered. The problem is the circumstances of getting the messages delivered. (Supposedly the viewer and the group chat servers doing the necessary communication to subscribe to group chat delivery, and supposedly through the agent presence service, right?) It isn't a new occurrence for me that after logging in, regardless whether the last login or logout was within a few minutes or several hours ago, I don't get any group chat messages delivered for a certain (but random amount of) time. Sometimes it is about 30 minutes before group chats start to pop up, and then they do one after another. Sometimes, fortunately more rarely at this time, it takes up to 2 hours before they start to work, like today. This especially happens on days with rolling restarts, and occasionally when the overall online count is above 50000. The latter quite makes sense, actually, since the more people online, the more people the messages have to be routed to and delivered, and not only that, but monitoring their online status in the meantime, while they move from one region to another by crossing or teleporting. And as I experienced a lot of times, changing region is the most common source for the agent presence service to fail to recognize you're online, which usually lasts for only a few seconds, but it is enough to miss the delivery of personal or group IMs. However, I still can't see why rolling restarts should correlate with failing group chat, and I can't even say it is in a direct correlation. This is when we, who don't know how the entire infrastructure works, can't figure out what's happening. I just suspect without any proof that the "thing" that triggers the rolling restarts, can affect the agent presence service, so people that log in during rolling restarts might experience these issues. Yes, this is just my suspicion, and I can't support it with anything yet.

 

On 5/24/2021 at 3:56 AM, Aishagain said:

Many group chats work until I teleport after which I receive no further messages and the moderators for that/those groups have vanished from the lists. In my case a relog usually (though not always) restores the function.

I can confirm this. This is part of what I am referring to above. After teleporting, or while travelling and crossing many regions, at a certain point, all group chats "stop". I can even add that since that participant list blocking feature was implemented, usually it simply stops updating the list of participants at some point. In large group chats where I'm not a moderator, the moderators in the list stop updating, meaning that moderators joining the chat sometime after the conversation was opened for me, don't show in the list, but also, moderators that were in the chat when the conversation was opened on my end, and now even went offline, still show in the list. The same goes with the list in a group chat where I'm a moderator, but with everyone.

 

On 5/23/2021 at 11:48 PM, Skell Dagger said:

As of today, the same issue that I experienced between 14th and 19th April has started again. I blogged about it here, including some of the testing that I carried out. Group chat was fine last night, but this evening my avatar is not subscribing to group chat, as before. Relogging doesn't help, and I didn't expect it to, as it didn't help the last time this happened either.

I can confirm this too. For me, this happens randomly, not as severely as in April, but for example last night, when the overall online user count was above 51000, I didn't get any group chat for several hours (and then none until today before starting to write this post, because I went to sleep, obviously. I know, it's surprising I ever sleep.) So I'm in the Catwa Head Friends group too, as a normal member (everyone role). It is one of the 4 largest groups I'm in, the other 3 are over 70k members if I'm correct, but less active in their group chat, normally. When things are working fine, whenever I log in, I should have the Catwa Head Friends group chat popping up within a minute. Maybe 2 if there really isn't anything being said at the very moment. Since this has been the case every single day, in any hour I might have logged in the first time for the day or I relogged, we can say it is how it works. That group is really active 24/7. (By the way it's amazing the Catwa crew can manage sorting out all those questions all the time!) If I log in and no Catwa Head Friends chat is popping up within 2 minutes, it already means something isn't working properly. In these cases, like today, usually no other group chat pops up either, until that "certain" time (today 110 minutes) after logging in, then the Catwa chat pops up, and then the others (this is because it is more active than any of my other groups). Relogging indeed doesn't help, it is totally random whether group chats start getting delivered right when they should, or after 3 minutes, 30, or 110.

 

It might be also an important part of this issue, that an old bug is still present, where you don't get the first message of a group conversation, only the second or third one. It is easy to recognize, since you get a group chat popping up, but only with an answer as the first message. You ask what was the question, and they'll repost it with the timestamp, and you can see it was at the same minute, or a minute earlier than the answer. It might be related to the failed delivery of the chat of a certain group, if after the first message didn't get sent to certain members, the subsequent ones get sent to the same members again. If I get something that can serve as a proof, I guess I'll go and file a JIRA again, for things that haven't been addressed before, but until then, since this topic is here (and it was a good idea, it's good to see when the Lab gets a proactive idea), maybe others can relate to these findings as well.

Edited May 26, 2021 by AlettaMondragon
typo...

[Extrude Ragu]

I have a very small group of ~25 members that sometimes I would like to talk to.

I'm the group owner, however opening the group chat, only a few of the people who were online in the group showed up in the sidebar. I typed a simple hello into the chat and nobody replied. Asked a couple of friends who are online and in the group if they had gotten the message - Nope.

I mean here's the thing. I don't think this is good enough and I ask Linden Lab - What are you going to do to make up for it? You know, I'm here paying for premium, running a community and me nor the people in it are able to rely on basic SecondLife functionality. I expect these features to work. Right now LL are failing to meet my expectations and I am not convinced that this issue is going to get solved any time soon.

If it is not financially viable for Linden Lab to put the resources forward to fix the problem in a reasonable amount of time - I have to ask myself, is it financially viable to keep paying Premium and paying for my full sim?

[Lyndka Cochrane]

I have to say that group chat has always been frustrating since Day 1 but I'm surprised over the last 12 months that Linden Labs haven't been talking to the likes of Discord  to see if there was a proven technology they could leverage.

It probably wouldn't work now to replace group chat - heck there's so much legacy code, it's too much of a job but including more than just a Discord button that launches an external server that folks have to join (and in a community where folks don't know about Outfits, Picks etc. it's asking too much of them to log in to Discord - set up a user ID there - link it back to SL).

I'm sure it's not beyond Linden Labs or Discord to provide a "seamless" experience as an alternative to group chat - which might then even reduce the overall load on the group servers?

[CarlaWetter]

On 5/21/2021 at 11:05 PM, CarlaWetter said:

And I haven't seen a lifesign from Firestorm Support English chat at all the last 13 days. Peaceful days, but not what I'd expect from that group chat.  Did mute notices around that time in FS, but chat still shows as unmuted.

Some more testing seems to support at least my intial suspicion that on Firestorm 6.4.13 (63251), when I mute group notices I won't receive any group chat in the Firestorm Support English group. With notices enabled again, FS chat tends to pop up within minutes after login.

Obviously this is anecdotal evidence for not much at all, but may at least be some further vector to look in one day.

[bigmoe Whitfield]

On 6/2/2021 at 1:34 PM, Lyndka Cochrane said:

I have to say that group chat has always been frustrating since Day 1 but I'm surprised over the last 12 months that Linden Labs haven't been talking to the likes of Discord  to see if there was a proven technology they could leverage.

It probably wouldn't work now to replace group chat - heck there's so much legacy code, it's too much of a job but including more than just a Discord button that launches an external server that folks have to join (and in a community where folks don't know about Outfits, Picks etc. it's asking too much of them to log in to Discord - set up a user ID there - link it back to SL).

I'm sure it's not beyond Linden Labs or Discord to provide a "seamless" experience as an alternative to group chat - which might then even reduce the overall load on the group servers?

I can agree,  discord would be a stepping stone to bring over to replace xmpp or what ever protocol they moved to years ago.

[MoiraKathleen]

14 hours ago, CarlaWetter said:

Some more testing seems to support at least my intial suspicion that on Firestorm 6.4.13 (63251), when I mute group notices I won't receive any group chat in the Firestorm Support English group. With notices enabled again, FS chat tends to pop up within minutes after login.

Obviously this is anecdotal evidence for not much at all, but may at least be some further vector to look in one day.

You might want to check in the Firestorm preferences under "Chat" -> "Chat Windows" - there is a preference option which will disable group chat when group notices are disabled.  Perhaps you have that checked.

 

[CarlaWetter]

Oh dear, never remembered that one and of course FS Support is the only group where I'd want the chat and not the notices. Usually it's the other way around if at all. Yes, it was set and probably carried along through my settings backup routine. Never mind me, I'll be out now.

[Stranger Nightfire]

I have suddenly stopped getting messages from any of the groups that I belong to.  What setting change could have cuased this. 

[Sharie Criss]

On 6/2/2021 at 3:34 PM, Lyndka Cochrane said:

I have to say that group chat has always been frustrating since Day 1 but I'm surprised over the last 12 months that Linden Labs haven't been talking to the likes of Discord  to see if there was a proven technology they could leverage.
 

The Discord idea has been brought up before (years ago,) and the underlying problems of group chat have been well known by the Lab for over a decade. Small incremental fixes have been rolled out now and then, but the underlying design of Groups (not just chat) is fundamentally broken and the lab has been struggling to make it scale. We SHOULD be able to belong to Hundreds of groups, but as the permission system has to check all our groups every time we do a Land Permission thing (even as simple as the ability to run scripts in a parcel) the reality is that there is a finite limit to the number of groups that can be checked before the sim is spending all it's time doing permission checks and nothing else. [Side note: it's not just land, but that's the easiest to describe.] The inefficiencies in groups account for the limits to the number of roles in a group, the Lab shutting off the group list for large groups, and a plethora of other limitations - group chat is smack in the middle of that mess.

Based on what the Lab has shared with us over the years here and there in forums, in-world meetings, etc., it's a monumental problem where the solution will affect EVERYTHING. I'm not shy of finding fault with LL, but in this case I believe they honestly have maxed out what they can do with the existing system and are probably trying to figure out how to move forward with a replacement that can be done by the small dev team they have without sacrificing all their other projects / plans.

I know it sucks because it's not integrated into the viewer, but just putting that Discord link in your group description and telling everyone to use that instead really is the only near-term workaround that will probably be needed for a couple years. Discord isn't a panacea - even Discord has a limit of 100 "servers" (groups) you can belong to! If you belong to a lot of active "servers" it can be overwhelming to manage (IMHO significantly worse than SL groups in that regard.) Honestly, some of Discords features can be seen as a huge negative with the same people that do those big gestures in group chat now posting old meme pictures over and over and over and over under the misguided belief they are being clever....

Discord has shown ZERO interest in offering their technology as an integration so there would be a seriously steep uphill battle for LL to convince them to do so. The $$$ aren't there. I mean, Discord just turned down $12 Billion from Microsoft.....

[Anylyn Hax]

On 6/2/2021 at 8:59 PM, Extrude Ragu said:

I have a very small group of ~25 members that sometimes I would like to talk to.

XMPP is a fererating Chat system, SL started to die when Mesh came out and it stopped his open policy. If we want to revive SL, we need to urgently re-establish an open chat system. XMPP is the solution. It is open free and federating.

[bigmoe Whitfield]

8 hours ago, Anylyn Hax said:

XMPP is a fererating Chat system, SL started to die when Mesh came out and it stopped his open policy. If we want to revive SL, we need to urgently re-establish an open chat system. XMPP is the solution. It is open free and federating.

I posted in the other topic about this,  xmpp is not going to happen.