Why Can't the Groups Be Fixed and Do They Really Need to Be?

[Prokofy Neva]

I'd love to hear a concise even if techie answer to this question. Please, no "valence electrons," just the straight dope.

What feature of the software is "hard" or "too buggy to ever fix" or "impossible to swap out while the plane is flying" causing this?

Is it even in "the viewer"? Or is it in fact "in the server code"? If so, move this post over there?

Holocluck suggested that the problem with groups is the fault of users because they "don't clear out no-show members". This is meant both to silence criticism of group failure and to imply that your groups are inactive if people don't log into them. 

But...I have even very new groups with only 30 people in them, all of whom log on nearly every day as they are paying rentals customers, and they don't work. So it's not about old no-shows.

Clearing out, say, 1000 dead members from a group of 2000 might make the chat more speedy? I don't know. I personally don't want to boot people who paid to join a group as they will complain. If that really, truly, is proven to be "the main reason" for "why groups don't work," why, I'll work at it. But I need an official opinion on this.

So, I think groups with land in them are especially problematic and that's a much smaller subset. Most people are on islands; most people are in groups only to chat and get shopping discount notices.

One fix I hear often is: "Discord".

I think the code hasn't been updated since about 2008 or so when they had the "group tool reforms" after various torturous town hall and small group discussion with "the community" and various torturous meetings and work sessions of their own, where they had to "scram" or "scrum" or what have you.

It's a fact of life that I accept that you cannot look up a person's name in a group by pasting their name into the group menu. 

The way to get their name is to type one-two letters and slowly wait for it to produce a list, which you then pick out.

At one time you could click on any parcel in the group and TP there, but that has been gone for awhile, and is tied to map failure of course, too.

Groups still function more or less to give a set of powers to a set of people with certain criteria. So that's worth keeping I suppose.

I think what most people mean when they say "groups are broken" is that they can't chat in them. Even when chat is enabled, it lags, doesn't work for some people, posts some of their comments and only intermittently. But if you don't encourage chat in groups and go inworld for it or on Discord, you've solved that problem.

The group accounts sometimes don't update on schedule, or at all, but I think that isn't really chronic.

So at what point should you say, hey, groups don't work, we can't fix them, but they work well enough and if you need to chat at length and fast, go to Discord, like you go to Firestorm for area search in shopping.

I'm fine with that, I just would like to understand it better.

[NiranV Dean]

The issue is in a lot of places but generally almost exclusively on the server.

Its debatable whether cleaning up dead members from a group would really speed it up because offline members do not enter the chat, they do not participate and thus have no active impact in theory. While the server has to iterate through all members at some point (at least to generate a currently online member list) its not the cause for chat lag. Generally a few things could be done about group chats to lessen the overall impact and improve user experience as a whole.

A) Add a server side option to opt-out of group chats, this should lessen the impact on the server as it wouldn't have to check against you since you will never ever participate in said chat and if you do so you do it manually anyway. This would also allow us to mute server chats, which in turn means that less inactive members will have to receive chats, this alone probably drastically lessens the amount of data required to send.

B) Clean out dead members to reduce the amount of people needed to be iterated through.

C) Improve the chat server infrastructure. This is the biggest one and basically the only that really is the problem, the server side infrastructure for group chats is simply bad. Compared to absolutely everything else in SL it just downright doesn't work and when it does, it does so just barely. Not sure what's so hard sending a couple thousand chat messages per second around. Everything ever everywhere does it all the time. The servers have more to do in general at any given day than the group chat servers should ever in their entire lifetime but for some reason they simply seem to be made out of potatoes.

[Chic Aeon]

Back at one of the beginning of an SL Birthday bash (not sure what year but fairly recently) Patch made a comment that large groups were the most detrimental part of "LAG" -- contributing greatly.   I even went through and took out some large groups (free ones) -- and thinking about that now I may do it again. So there IS apparently an important reason to keep a maximum on groups. 

[LittleMe Jewell]

It isn't just the chat function of groups that is hosed.  Getting notices is hit or miss in some groups and the size of the group does not seem to matter.  I have some groups that I never get the notices from, even though I do have the box checked to get them.  Other groups, I'll sometimes get notices and sometimes not.  Yet other groups, I never miss a notice from, and some of those are very large groups.

[Profaitchikenz Haiku]

57 minutes ago, NiranV Dean said:

While the server has to iterate through all members at some point (at least to generate a currently online member list)

You might be able to answer a question that's been puzzling me on this, given you see the source.

When group chat opens, I see lots of little pictures of the online members in the group chat floater, the profile pictures of the online members, and clicking on one gives me a full-size (profile-size) picture. Are these profile pictures already in my cache or is the server having to squirt loads of profile pictures to my client together with the two dozen or so letters somebody has just typed? Two dozen letters isn't a lot to send across the wires to my client, but two dozen profile pictures in one go is going to take a wee bit of time to squeeze through the copper. Is that one of the sources of lag in group chat, showing me pictures of who's online in the groups, regardless of whether I want to see what they look like or not?

(Even if they are already there, it means there's going to be quite a flurry of cache-searching for each UUID to see if it can put up the picture or have to request it, all just to let me look at a few lines of text.)

So when group chat opens, my client has to handle a list of who's there and what they look like, and all the time group chat is open, I assume there's also this constant monitoring of who's just joined and who's just logged off, and it seems to me a lot of overhead when just the lines of text is the point of the exercise.

Edited March 17, 2021 by Profaitchikenz Haiku

[Aishagain]

@Profaitchikenz HaikuI have noticed that if I open a group IM window and then look at the online members list, as it populates I see a massive increase in udp data flow, so I guess that answers one of your points, the data is held server-side and not on your PC.  This, I suggest, is one of the reasons that the group IM servers are being taxed, having to send out great wodges of data.  To redesign group chat so this data is not sent would be a pretty major reworking job and would, I assume, involve changes both server- and client-side.

[animats]

Beq Janus explained part of the problem at Server User Group. The trouble is that group messages are not "reliable" at the SL network level, which means they do not get re-transmitted if lost due to viewer overload or network lossage.  So, to compensate, after the viewer receives a group message, it asks for all the group messages available. This gets it the recent messages plus duplicates of old ones. This in turn creates a huge load on both server and viewer with a group with a lot of message history sends a message.

That's why. It probably made sense in the early days of SL. They were thinking of a few people organizing a raid, not 10,000 users of a popular body getting an product promotion. Worse, this goes through the sim servers, which means there's back-end traffic between sims and some message server to get messages to where you are in world.

Group messages really ought to be handled by a server independent of the sim servers, because group messages have no locality. They have nothing to do with the 3D world. (Local chat does; who can hear you depends on where you are.) Gradually, SL has moved non-world-oriented things off the sim servers - first assets, then inventory. It's time to see that done for group chat.

SL is a social network. The messaging has to work.

[Profaitchikenz Haiku]

1 hour ago, Aishagain said:

To redesign group chat so this data is not sent would be a pretty major reworking job and would, I assume, involve changes both server- and client-side.

At least V1-style viewers don't show the pictures (but might still be downloading them for all I know).

[NiranV Dean]

6 hours ago, Profaitchikenz Haiku said:

You might be able to answer a question that's been puzzling me on this, given you see the source.

When group chat opens, I see lots of little pictures of the online members in the group chat floater, the profile pictures of the online members, and clicking on one gives me a full-size (profile-size) picture. Are these profile pictures already in my cache or is the server having to squirt loads of profile pictures to my client together with the two dozen or so letters somebody has just typed? Two dozen letters isn't a lot to send across the wires to my client, but two dozen profile pictures in one go is going to take a wee bit of time to squeeze through the copper. Is that one of the sources of lag in group chat, showing me pictures of who's online in the groups, regardless of whether I want to see what they look like or not?

(Even if they are already there, it means there's going to be quite a flurry of cache-searching for each UUID to see if it can put up the picture or have to request it, all just to let me look at a few lines of text.)

So when group chat opens, my client has to handle a list of who's there and what they look like, and all the time group chat is open, I assume there's also this constant monitoring of who's just joined and who's just logged off, and it seems to me a lot of overhead when just the lines of text is the point of the exercise.

The profile pictures are handled by the asset server which is independent of everything else now. Its like loading a texture inworld or your inventory.

[Gabriele Graves]

Perhaps a good strategy would be to first of all split the title/land/object functions from the chat related things.  Develop a best of breed replacement group IM chat feature that can accommodate notices and attachments completely independently of the existing groups functionality.  When that is in place alongside existing groups, turn off group chat related functionality in existing groups so they can only be used for titles, land and objects that need group related things.  Inevitably there will some gnarly details to work out but I think as a goal it is a good place to aim.

Edited March 18, 2021 by Gabriele Graves

[Beq Janus]

Just to clarify something regarding thr group "inefficiencies" to be polite about it. 

The specific cases I referred to in the meeting that @animatsmentioned were to do with the manner in which group notices are handled. This was especially bad during December and January of this year due to an error caused by the uplift project that meant that old notices were not being cleared down. while group chat is related to this it is not the same set of problems. As@NiranV Deansays the issue is largely server side with a helping of poor protocol design and (I don't doubt) a bunch of places where the viewer could improve. The server-side is the biggest part of this though.

One of the issues ties very well into what @Gabriele Gravessays. Groups have been used and abused for too many reasons. We have groups that are used predominantly for messaging (consider Firestorm Support English, or Builders Brewery, for example) where we have 10s of 1000s of users. Most of which are silent most of the time. You have massive merchant groups such as Blueberry where people don't chat, they are there to get notices and updates. Then we have land groups, that allow us access to the places we live. all of this overloading of the group system is a massive problem. The group chat problems are one side of this, consider your TP failures... How much of the time it takes to complete a TP is caused by the receiving server having to check through your list of groups and roles to determine whether you can enter the land you are moving to?

The group system has been abused all of its life and it is very hard to roll back from that abuse I fear. 

 

[animats]

22 minutes ago, Beq Janus said:

The group system has been abused all of its life and it is very hard to roll back from that abuse I fear. 

Some years ago, I went to a talk at Stanford by one of the early architects of Facebook. Facebook, remember, started as a system for college students. Their original infrastructure had servers dedicated to individual campuses and regions, on the assumption that most traffic would be local. The servers could talk to other distant servers, but were not intended to do that much.

As Facebook grew beyond college campuses, the traffic pattern changed. Traffic wasn't as local, and the inter-server links became clogged. Fixing that would require a complete redesign of their infrastructure.

They did it.

[Lucia Nightfire]

I'm still waiting on them to remove what they believed to be a major contributor to the group chat failures if not the primary contributor, the online/offline tracking/udpates sent to everyone in every chat session.

Here is Oz talking about it way back in June of 2019:

 

[Ati Thei]

The chat needs to be rebuild from ground up so it's a future-proof and easy to scale.

If they can come up with something like Socket.io + AWS Fargate + Redis that would be actually nice.

More groups and no missed notices means potential revenue.

Edited April 4, 2021 by Ati Thei

[Lewis Luminos]

I've had ongoing problems with group chat for my club group for more than a year. It's not even a big group (considering the size of some that I get constant chat-spam for). 

Everything else works fine, except for chat; no-one can post it at all.  I have been in contact with support for an extended time and all they can do is "re-set" the group (whatever that means). Once they've done that, chat will work exactly once, then it breaks again. Now I can't just call Support every time I need to announce an event so I have given up with the chat altogether and I just send only notices. 

I would like to make a new group and start again but it is so much work to invite everyone and too many people will get annoyed at being booted off the original because they have notices turned off and won't see the reason why.

[animats]

2 hours ago, Ati Thei said:

The (group) chat needs to be rebuild from ground up so it's a future-proof and easy to scale.

Yes, it does. The trouble is that group chat is handled as an in-world feature, going through the sim servers. That's a legacy of early SL, where the sim servers did everything. But it has nothing to do with where you are in world.  It should be something where the viewer connects directly to some chat server, which handles IMs and group chats independently of where you are in world. The current system needs to coordinate between the chat server, the sims, and the presence server, so that messages can be passed to the correct sim for delivery. This adds much unnecessary complexity and traffic, plus problems such as losing messages during teleports.

Open source, scaleable general purpose chat servers are widely available. The viewers need to be modified to talk to them. They need to authenticate through the SL authentication system so you can't spoof user names. Messages sent from new viewers could be sent on both the old and new systems and un-duplicated on receipt during the transition period. The legacy sim code doesn't need to be modified at all, until it's obsolete and can be removed.

Firestorm could probably do this on their own, if LL didn't object.

[Lucia Nightfire]

A recent change, as of a few days ago, may have helped things.

[Gabriele Graves]

Sadly not, a few of the groups that I belong to were today experiencing exactly the same issues as they have been for ages now.  No discernable change I can detect.

Edited April 5, 2021 by Gabriele Graves
cannot form sentences for some reason.

[dayz Short]

Speaking as a moderator of a very large group, the current issue with group performance is not lag at all. It is that more than 50% of our member base cant post in, see their own posts or see what others post. same for announcements IT MATTERS . we are a very active and very large community and our chat room is where we live. our chat room is also the official room of our long standing business and many many landowners depend on SL being able to produce chat. 

This IS a social network and our business is taking a big hit.  Now today it started again with after tp-ing one has to close/reopen the room to post or see posts again. its not just a few folks its more than 50% of our member base .  This issue has been ongoing for the last 1 to 2 months .

[MajikVixen Lorefield]

All I know is:

That I have enjoyed SL groups very much for a very long time.  If your friends aren't online, you'll never feel lonely by chatting in groups.  I feel like this is needed more now than ever because of the chaos the world is currently in.  Stability is one thing I'm sure a lot more ppl other than just myself would like to very much cling to right now.  And if SL is supposedly offering socialization when the RL pandemic does not, then this feature should be a priority.

When I brought this issue to Premium chat several months ago (when this first started happening), they explained that THEIR viewer (the regular SL one) has no group chat communication issues.  I could not disprove it because I unfortunately loathe the regular SL viewer and would rather "suffer" on FS.  I could only stand about an hour on the regular SL viewer.  It did not feel like home or comfy to me, and SL is an escape I would definitely like to feel relaxed in.  (This is also why I never took helping newcomers seriously either, something I really would've like to have done, but as those newcomer friendly places insist you use the SL viewer as a priority, nevermind.)  Thus, I could not push the issue, and LL did not take me seriously because they very effectively passed the buck.

But I do know for a fact that the SL group chat hasn't had any major communication issues in the last several years until SL moved their servers to Amazon, fondly referred to as "the cloud" by them and "the LL fart" by me.  Now that all regions have been moved to Amazon, I'm not sure what the issue is except that the holiday happened and everything went to hades (and not even in a fun way).  It would be nice if this could be resolved when business resumes tomorrow (hopefully), or maybe by reset day.  ...Until it predictably breaks again.  Band-Aids are acceptable until surgery can remove the problem once and for all.

Meanwhile, Discord is an app that goes on my antiquated phone for the purpose of keeping in contact when I'm offline and in RL ...Plus it's just too far for me on the desk to even care about while I'm on the gaming rig that I'd rather use to experience SL.  Seriously.  And all those notifications that I cannot keep up with on my phone from all those groups... I give up and could care less.  Maybe I'm not alone in that, IDK.

But I'm sure that if LL had the power, they would fix this, along with the no new regions that server issues + pandemic have also caused ...somewhere between EEP and some other dumb thing I could really care less about other than stability.  ...Although, the BoM was nice.  Thanks for that at least.

Oh, well.  Good luck, Pepsi.

[rasterscan]

I'll fetch the spanker ....

[Theresa Tennyson]

On 4/6/2021 at 6:22 AM, MajikVixen Lorefield said:

But I do know for a fact that the SL group chat hasn't had any major communication issues in the last several years until SL moved their servers to Amazon, fondly referred to as "the cloud" by them and "the LL fart" by me. 

Theresa Tennyson nods and smiles.

Of course there haven't been any major group communications issues before the cloud migration.

https://www.google.com/search?q=second+life+group+chat+not+working+site:community.secondlife.com&client=safari&rls=en&sa=X&ved=2ahUKEwiF3cny0-7vAhWTKM0KHa9tB9EQrQIoBHoECAQQBQ&biw=1086&bih=711

[LittleMe Jewell]

On 4/6/2021 at 4:22 AM, MajikVixen Lorefield said:

But I do know for a fact that the SL group chat hasn't had any major communication issues in the last several years until SL moved their servers to Amazon

If this is your perception, you are really super lucky.  The fact is that folks have been complaining about chat issues for about as long as I've been in SL -- and I got here in early 2007.

[MajikVixen Lorefield]

My mistake.  I meant for days and days on end.  I forgot that maybe because I have to work and cannot be on SL all the time, I might've missed a day or so.  Sorry I thought I was on your side.

[Kyrah Abattoir]

My very limited understanding is that groups are distributed along multiple servers. Which server handles which group being sort of a fixed assignment, (most likely based on the group key, which never changes and is picked "randomly" at creation)

The problem with this is that it is possible for a few extremely large groups to live in the same server, negatively affecting any and all groups that share that server with them.