Recent updates to the group chat system

[Qie Niangao]

At the most recent Simulator User Group meeting, somebody mentioned the idea of replacing the group messaging system backend with XMPP. I mentioned that this had been attempted already, about ten years ago, when the Lab invested a chunk of development calendar to the project which they then abandoned in the spring of 2011. I don't know why it was abandoned, but it may be recorded; you can see some of the record by searching the wiki for XMPP. (Focus on results from the spring of 2011. Also, ignore anything from "AW Groupies" meetings—a generally useful rule, not limited to this topic.)

That approach would have separated the group chat service from any land management functions, with viewers connecting directly to XMPP servers rather than routing all chat through the sims, although presumably groups themselves would still be associated both with the chat function and with the land management functions. (Thus, existing groups would have still been used for both functionalities, instead of needing to define all new, separately administered chat-only groups, as with something like Discord—if it were even available.)

Anyway, at some point it was decided to shelve the XMPP effort and instead tweak the performance of the existing group chat implementation. Point is, as@Solar Legionrightly points out, we've been down this path before, and group chat has been a sore point for over a decade, not really starting with Uplift.

[CaithLynnSayes]

How is that a loaded question? It's really not. I'd really like to see your train of thought here. that's all really.

 

In your earlier post you say that LL failed to properly disclose the intended use of the group chat system. I actually agree with you on that. My point... (3th time now today, and then you say i don't listen, how ironic) is that LL had literally years to come up with a solution for what the group chat system clearly has been used INSTEAD of its intended use. As a developer it is up to you to then take the hint, this is what your users want and try to use what you already provide them with. So make it capable of handling that. This is what LL had to do years ago. You just saying "that ship has sailed" and just live with it, just isn't going to cut it. Group chat is a massive part of LL's social aspect. I will keep emphasizing the fact that it is shameful that they just let it come to this. to completely let the group chat system be "misused" and inevitably grind to a near halt. You just can not deny the fact that the group chat system is and has been used like this for years, so why hasn't it been maintained or reworked years ago. this is the actual issue at hand here.

 

To clarify once again. I fully understand your argument on what the group chat system is and how it was originally meant to work, but the reality here is that it isn't being used that way. For years. A lot of users don't even know how group chat was intended to work.

 

 

[Jenna Huntsman]

2 minutes ago, CaithLynnSayes said:

How is that a loaded question? It's really not. I'd really like to see your train of thought here. that's all really.

 

In your earlier post you say that LL failed to properly disclose the intended use of the group chat system. I actually agree with you on that. My point... (3th time now today, and then you say i don't listen, how ironic) is that LL had literally years to come up with a solution for what the group chat system clearly has been used INSTEAD of its intended use. As a developer it is up to you to then take the hint, this is what your users want and try to use what you already provide them with. So make it capable of handling that. This is what LL had to do years ago. You just saying "that ship has sailed" and just live with it, just isn't going to cut it. Group chat is a massive part of LL's social aspect. I will keep emphasizing the fact that it is shameful that they just let it come to this. to completely let the group chat system be "misused" and inevitably grind to a near halt. You just can not deny the fact that the group chat system is and has been used like this for years, so why hasn't it been maintained or reworked years ago. this is the actual issue at hand here.

 

To clarify once again. I fully understand your argument on what the group chat system is and how it was originally meant to work, but the reality here is that it isn't being used that way. For years. A lot of users don't even know how group chat was intended to work.

 

 

The reason why it didn't happen sooner, like many things, boils down to one word:

Sansar.

A lot of major work on SL slowed down while LL was working on Sansar (as it, to them at least, was SL's successor), and as a company they only had so many resources to go around. Unfortunately too many of those resources were spent on Sansar.

Now Sansar is gone, LL have got a mountain of work to catch up on, in unforeseen circumstances. Cut them some slack, at least we're beginning to move in the right direction.

[Solar Legion]

1 minute ago, CaithLynnSayes said:

How is that a loaded question? It's really not. I'd really like to see your train of thought here. that's all really.

 

In your earlier post you say that LL failed to properly disclose the intended use of the group chat system. I actually agree with you on that. My point... (3th time now today, and then you say i don't listen, how ironic) is that LL had literally years to come up with a solution for what the group chat system clearly has been used INSTEAD of its intended use. As a developer it is up to you to then take the hint, this is what your users want and try to use what you already provide them with. So make it capable of handling that. This is what LL had to do years ago. You just saying "that ship has sailed" and just live with it, just isn't going to cut it. Group chat is a massive part of LL's social aspect. I will keep emphasizing the fact that it is shameful that they just let it come to this. to completely let the group chat system be "misused" and inevitably grind to a near halt. You just can not deny the fact that the group chat system is and has been used like this for years, so why hasn't it been maintained or reworked years ago. this is the actual issue at hand here.

 

To clarify once again. I fully understand your argument on what the group chat system is and how it was originally meant to work, but the reality here is that it isn't being used that way. For years. A lot of users don't even know how group chat was intended to work.

 

 

Yet you seem to deny that what we have now is something they could not have seen coming until it was well too late.

An assumption on your art is that they just sat back and did nothing whatsoever - an assumption that it is frankly not my job to dissuade or correct beyond an incredibly basic refutation/rebuttal. Most especially since it serves no purpose whatsoever in as far as suggestions/spitballing on future actions.

You are free to feel how you like as far as the history goes. It does not change that history or the realities of it.

Focus on where to go from here.

[Solar Legion]

2 minutes ago, Jenna Huntsman said:

The reason why it didn't happen sooner, like many things, boils down to one word:

Sansar.

A lot of major work on SL slowed down while LL was working on Sansar (as it, to them at least, was SL's successor), and as a company they only had so many resources to go around. Unfortunately too many of those resources were spent on Sansar.

Now Sansar is gone, LL have got a mountain of work to catch up on, in unforeseen circumstances. Cut them some slack, at least we're beginning to move in the right direction.

Sansar is - sadly - only a small part of the greater problem and came late to the party.

[CaithLynnSayes]

I totally agree, as soon as LL announced Sansar and told us they would shift of a bunch of LL minions to the Sansar project, the only difference between LL and the Titanic was a band and deck chairs.

[CaithLynnSayes]

2 minutes ago, Solar Legion said:

Yet you seem to deny that what we have now is something they could not have seen coming until it was well too late.

An assumption on your art is that they just sat back and did nothing whatsoever - an assumption that it is frankly not my job to dissuade or correct beyond an incredibly basic refutation/rebuttal. Most especially since it serves no purpose whatsoever in as far as suggestions/spitballing on future actions.

You are free to feel how you like as far as the history goes. It does not change that history or the realities of it.

Focus on where to go from here.

My assumption is obviously based on what i and a hell of a lot of other users experienced. Ongoing group chat issues with no permanent or improved solution. This leads me and of course others to conclude not much has been done. Maybe lots has been done behind the scene, but where is the result i ask?

[Jenna Huntsman]

7 minutes ago, CaithLynnSayes said:

My assumption is obviously based on what i and a hell of a lot of other users experienced. Ongoing group chat issues with no permanent or improved solution. This leads me and of course others to conclude not much has been done. Maybe lots has been done behind the scene, but where is the result i ask?

My best guess is that right now *not much* has been done towards fixing group chats. With Oz retiring, there isn't any one person to guide where development resources are being spent right now, so that's a big problem.

Another thing is that group chats aren't the only thing broken right now. My best guess is that a lot of development time is being spent on the map tile problems as they're a lot quicker to make (proper) fixes to as opposed to group chats which require a lot more time and effort to reengineer for something they're getting just as much fuss over. Not to mention the issues with the beta grid not syncing right now (probably another quick fix, but again map tiles).

To put the cherry on top of this however, is that trying to make changes to the existing system to increase stability (most likely) has just been met with an incredible amount of blowback over a feature which seemingly wouldn't be that useful to regular group members. It's a situation where LL can't win right now.

[CaithLynnSayes]

26 minutes ago, Jenna Huntsman said:

It's a situation where LL can't win right now.

Allow me to call out the obvious... Who let it come to this?

[Qie Niangao]

5 minutes ago, CaithLynnSayes said:

Allow me to call out the obvious... Who let it come to this?

I don't suppose it's satisfying, but FWIW this approach of not updating every group member's online status in every chat window was requested, week after week for the past few months, by a resident attending the Server User Group. (I gather that at some point some Linden observed that it might help some.)

[Aishagain]

For what this is worth I am sick and tired of this "Dialogue of the Deaf" between @CaithLynnSayesand @Solar Legion.  It gets us precisely nowhere.  LL know what needs doing.  The real question is whether or not they have the means to do it.

[CaithLynnSayes]

2 minutes ago, Aishagain said:

For what this is worth I am sick and tired of this "Dialogue of the Deaf" between @CaithLynnSayesand @Solar Legion.  It gets us precisely nowhere.  LL know what needs doing.  The real question is whether or not they have the means to do it.

i agree

[AlettaMondragon]

1 hour ago, Qie Niangao said:

I don't suppose it's satisfying, but FWIW this approach of not updating every group member's online status in every chat window was requested, week after week for the past few months, by a resident attending the Server User Group. (I gather that at some point some Linden observed that it might help some.)

If only someone kept going there to remind them about fixing the map as well... I'm really disappointed the time that user group is at isn't good for me, but oh well. To stay on topic, yes, it might help some, and if it really did, then for example I wouldn't complain about it at all. It would be just another recent change in the line that might or might not have some real benefit, but isn't really important anyway.

The problem with the change is that it didn't work, and it seems to correlate with the group chat delivery issues that arose right after the changes were made. A few hours ago it started again for me, not getting any group conversations after relogging at some point. I logged in with an alt as well, like yesterday, that I only keep nowadays to test group chat when it is failing on my main. Now my alt didn't receive any conversations either, not even some that I initiated with my main. The common pattern in the failing group chat delivery was that, on both my main and alt, the full participant lists got sent to the viewer (despite not being a moderator in those groups, and group members well over 10, which is the intended limit - they were over 1000, 3000, or 70000, actually). In group conversations where the full list got sent to me, I wasn't able to see or send chat with either account, initially, while those that showed the participant list "as intended", worked properly. Since we already determined with @Maestro Linden yesterday that this issue is about the viewer not subscribing for group chat delivery on login, I started to do the same tests that we have done yesterday, and then I also created different scenarios with my main and alt to further test how the group chat behaves. The overall result is that this participant list blocking feature is entirely inconsistent now at best, and in the worst case scenario, it might be the cause of the failed group chat subscription on login. If not the blocking of the full list from being sent, then something else that was accidentally affected while implementing this feature, which is even more likely. I've run different scenarios, in which either my main or alt initiated a group chat session, where either or both of them got the entire participant list of the conversation, or the intended cut list, then also refreshed group profile, closed and reopened the group conversation, and started the test again, and then also relogged with both accounts and did the same tests again.

Eventually, I became able to read and send messages in all the groups I've been running these tests in (until proceeding to point 4 below, where it got entirely broken again for both accounts), but there are several things that still aren't working properly:

1) It is random whether a group chat conversation is getting sent to the account after logging in. After relogging with both accounts, my alt got a group conversation popping up, where my main is also a member, and didn't get the conversation until opening the conversation manually. So the group chat subscription on login randomly fails.

2) There is one group, where my main got the full chat participant list during all of these tests while not being a moderator. After relogging, it still shows the full list. My alt has received the full list as well, initially, and persistently after refreshing group profile and reopening the conversation, however, after relogging, unlike my main, it only gets the cut list with only moderators showing, as intended. Another person in that group chat confirmed that she also received the full participant list despite not being a moderator, and it persisted for her over closing chat, refreshing group profile, and reopening chat as well. This means the feature that is supposed to block the participant list from being sent to those that shouldn't receive them, is unreliable and only randomly works.

3) While initially it seemed to be a consistent pattern, and a correlation that if the account got the full participant list while it shouldn't have, it would mean it is unable to receive or send messages in that group chat, it turned out within 10 minutes of testing that it isn't the case, and there is no direct correlation between these two aspects. You can get the full participant lists while you shouldn't, and be able to receive and send group chat messages at the same time.

4) During the tests, I stayed with both accounts in the same region they were in (a different region for either of them) for the entire time. While writing point 3 above, I had to teleport out to another region with my main to check on something. Right after the teleport, my main stopped receiving group chat messages (in opened conversations as well), that my alt (still in the region it has been in the whole time) kept getting. Then I teleported to another region with my alt too, and this caused my alt from receiving chat as well, and even in my own group with only my main and alt, my main now cannot send messages in that group chat, my alt can send messages that my main gets, though. After closing the conversation with my main, refreshing the group profile, reopening the conversation, my main can send messages in the group which my alt can see but my main can't, and my alt can still send messages that only my alt can see this time, so my main cannot see anything in the conversation. This means changing simulator does correlate with the failure of group chat delivery, and after changing regions, the simulator doesn't recognize the agent being present there. It is not news, it has been happening since last August here and there. Maestro, please, take this seriously, when we keep saying the root of this issue is in the simulator servers, we aren't joking or guessing. It is proven.

 

I'm writing this here instead of filing a new bug report (yet) because more people read the forums than the Jira. People, if you feel like you want to experiment with these things, and collect data and see if it correlates with my findings, especially if you are having issues with group chat recently, please do tests similar to what I described, and share your findings here, so we can learn more about the issue together, and I can add those results to the bug report I'm going to file along these lines.

[Qie Niangao]

19 minutes ago, AlettaMondragon said:

If only someone kept going there to remind them about fixing the map as well... I'm really disappointed the time that user group is at isn't good for me, but oh well.

Perhaps this is irony, but just in case: Of course somebody raises this every week, and we've been doing so for over four months. You can assess for yourself the effect.

[Lucia Nightfire]

10 hours ago, Qie Niangao said:

I don't suppose it's satisfying, but FWIW this approach of not updating every group member's online status in every chat window was requested, week after week for the past few months, by a resident attending the Server User Group. (I gather that at some point some Linden observed that it might help some.)

Oz said in a TPV meeting almost two years ago that the majority of the group chat traffic was the user online/offline messages being sent to everyone and proposed culling those to make group chat better.

 

Edited April 16, 2021 by Lucia Nightfire

[Lucia Nightfire]

7 hours ago, AlettaMondragon said:

If only someone kept going there to remind them about fixing the map as well.

It's been brought up in the Server meetings weekly.

7 hours ago, AlettaMondragon said:

I'm writing this here instead of filing a new bug report (yet) because more people read the forums than the Jira.

Most of the developers don't read the forums. They read the jiras.

[AlettaMondragon]

7 hours ago, Lucia Nightfire said:

Most of the developers don't read the forums. They read the jiras.

You misunderstood the point, then. I wrote it here first for people like you and me, so you can do the same tests if you want, if you notice failing group chat delivery, and then it won't be only me poking them on the Jira with this all the time, while others just silently accept the fact things are so broken, or even try to tell people off of seeking a fix when we bring it up here. It is nice you were able to read the first sentence of a consistent paragraph, but it doesn't take too much effort to quote myself here, so here you go, reading it again can't hurt.

13 hours ago, AlettaMondragon said:

I'm writing this here instead of filing a new bug report (yet) because more people read the forums than the Jira. People, if you feel like you want to experiment with these things, and collect data and see if it correlates with my findings, especially if you are having issues with group chat recently, please do tests similar to what I described, and share your findings here, so we can learn more about the issue together, and I can add those results to the bug report I'm going to file along these lines.

Edited April 16, 2021 by AlettaMondragon
added quote of myself, for reading comprehension improvement purposes

[Phoebe Avro]

Group chat is totally borked I have a staff group of 54 members and group chat does not work for most members if any we now have to use discord, when is LL going to fix this bs

 

Edited April 16, 2021 by Phoebe Avro

[Phoebe Avro]

Ok i have been doing some experiments, my alt account cant use our staff chat, so i deleted the cache folder uninstalled FireStorm and re installed it same fault, installed singularity viewer still no chat in our staff group so ive come to the conclusion this is account based fault that some accounts have. I can chat in some other groups ok.

 

Edited April 17, 2021 by Phoebe Avro

[Aishagain]

Whatever the issue, and whatever the history of group chat, the fact is that it USED to work...not well but at least partially.  Now it is utterly useless and recent attempts by LL to reduce the server-load have proven less than useful.

The current situation is a woeful indictment of LL's present lack of engineering and coding abilities.

ETA: It cannot have escaped LL's attention that the recent issues wih group chat have apparently resulted in a +5% reduction in online numbers.  This issue is hitting you in the wallet LL and the fees hike is not going to help.  Fix your product, LL or the atrophy will become a torrent.

Edited April 17, 2021 by Aishagain

[LittleMe Jewell]

1 hour ago, Aishagain said:

Now it is utterly useless

Not totally. 

I've still got chat working in multiple groups.  Sometimes posts don't end up going through and sometimes they get out of order --  both of which has happened since my first day in SL over a decade ago -- but many of my groups are still chatting.

[Aishagain]

38 minutes ago, LittleMe Jewell said:

Not totally. 

I've still got chat working in multiple groups.  Sometimes posts don't end up going through and sometimes they get out of order --  both of which has happened since my first day in SL over a decade ago -- but many of my groups are still chatting.

Well, your definition of utterly is not mine.  As far as I am concerned if I cannot see moderator chat or be sure that all that is sent to a group is actually being seen by either the mods or other users, it is useless.  That applies to all my groups without exception.  Indeed some do appear to work better than others and it does not seem to be dependant on the number in each group, rather whether or not that particular group is having its resources sucked away by another more numerous group.

In short if I cannot be sure of a group working, for me, that group is useless. 

While chat-lag and order scrambling did occasionally occur back in 2010, it was never remotely as bad as it is now.

Edited April 17, 2021 by Aishagain

[LittleMe Jewell]

7 minutes ago, Aishagain said:

While chat-lag and order scrambling did occasionally occur back in 2010, it was never remotely as bad as it is now.

Chat lag, posts totally not showing, and posts out of order has happened since at least as early as 2007 -- so, by your definition of "utterly useless" it has always been that way.   

In some ways chat is actually better for me now than it was last Oct-Nov - though the issues do happen more frequently than 2007.

I actually have more problems getting group notices actually delivered to me than the issues that I have with group chat.

Then again, I don't spend my entire time in SL chatting in groups or even paying close attention to the things that other people say in groups.

[MarissaOrloff]

69 out of 70 groups are completely dead right now for me

[Rowan Amore]

3 hours ago, Aishagain said:

group chat have apparently resulted in a +5% reduction in online numbers

Doubtful this is the main cause unless you have some proof.  Warmer weather in the northern hemisphere, people being vaccinated and back doing some normal things...those would have more impact.  iMHO