Why does the Inventory use so much memory?

[Ananda Sandgrain]

I've been experiencing out-of-memory related crashes with viewers ever since the advent of the mesh versions.  Eventually I got around to discovering something odd - by far the largest chunk of my computer's memory is being used up by the Inventory when I first load up a viewer.  If I clear cache first, it typically uses about 350,000 K of memory.  But if my whole inventory has loaded, it will start off with 950,000 K.  Now, having been around for 8 years, my inventory has grown to be a huge mess, I "only" have 77,000 items in it.  But what I can't understand is why SL feels the need to keep over half a Gigabyte of something-or-other in active memory, just so I can do an inventory search now and then!

Who's got an explanation for what's going on here?

[Dora Gustafson]

In your Preference/Advanced tab you can set the size of your cache
In my case it is set to default = 512 MByte
Reducing the size may solve your problem or it may not:)

[Tamara Artis]

Why do you have the need to keep 77,000 items? Box everything you haven't used in a while and leave only important stuff. Maybe it helps you:) 

[Madelaine McMasters]

I watched memory usage while launching V3 on Snugs' account, which has no inventory beyond the library stuff. It started at 150MByte and grew to 700MBytes within seconds, far too fast to have been filled by anything obtained from SL servers. My internet connection tops out at about 1MByte/sec, so it would take nearly ten minutes to fill that much memory.

I cleared cache and relaunched. Memory once again started at 150M and quickly climed to 350M before slowly climbing to 550M over the next minute. Even so, memory usage climbed far faster than could be explained by downloads from the SL servers, as my bandwidth meter settled to 15KBytes/sec long before memory usage plateaued. During the last half minute before memory use plateaued at 550M, memory usage grew 100Mbytes while the bandwidth meter sat at 10-15KBytes/sec (that's less than 1MByte/minute). So memory use was growing more than 100x faster than download speed.

So, I don't think inventory is using memory, a good portion of it is probably textures and maps, but a good portion is simply unexplained.

[Ananda Sandgrain]

Interesting.  I haven't actually tested it out with a low-inventory account.  In that case what would be nice to find out is what exactly *is* SL loading into memory during those initial minutes that clog things up so badly.  It has something to do with inventory loading, because what I find is that I will consistently crash every 20-30 minutes if it is loaded, but if for some reason my inventory fails to load again after a cache clear, it will run fine for hours.  I kind of need that memory space back and it is clearly nonessential to actually running the program, so it would be nice to find out how to get it.  It'd be nice to have a better understanding of what is going on *before* I embark on an epic inventory reduction project.   I'm not sure I have it in me just to continue using SL once or twice a week.

The cache size makes for another question:  if the inventory is this large (larger than 512mb) would things improve if I increased rather than decreased the cache size?  Or would it not matter.  Mine is also currently at 512mb.

[Madelaine McMasters]

---

Ananda Sandgrain wrote:

Interesting.  I haven't actually tested it out with a low-inventory account.  In that case what would be nice to find out is what exactly *is* SL loading into memory during those initial minutes that clog things up so badly.  It has something to do with inventory loading, because what I find is that I will consistently crash every 20-30 minutes if it is loaded, but if for some reason my inventory fails to load again after a cache clear, it will run fine for hours.  I kind of need that memory space back and it is clearly nonessential to actually running the program, so it would be nice to find out how to get it.  It'd be nice to have a better understanding of what is going on *before* I embark on an epic inventory reduction project.   I'm not sure I have it in me just to continue using SL once or twice a week.

The cache size makes for another question:  if the inventory is this large (larger than 512mb) would things improve if I increased rather than decreased the cache size?  Or would it not matter.  Mine is also currently at 512mb.

---

I can't imagine that a 70K inventory would be more than a few megabytes. Each inventory item is (I think) nothing more than the text name of the thing and its UUID. The longest names in my inventory are prolly less than 64 characters long, so even 70K of those would take only 5Mbytes or so. The correlation between inventory loading and crashing might be just a coincidence.

As to what is actually taking up all the space, that's a mystery. It can't be all textures because I've watched the bandwidth meter and it just never goes high enough to explain the memory growth rate. I suppose there's just some magic stuff the viewer does, like building scene lists or precomputing stuff to speed up the frame rate. I run my cache at maximum (nearly 10G) and that seems to help. I can flip around the various sims I visit and they all load PDQ.

[WhiteStar Magic]

I saw this posting earlier today and for curiosity sake have been playing with different viewers on SL & OpenSim and monitoring the ram usage with the various viewers.

V1 based viewers are behaving as expected, V3 viewers up to 3.2 codebase seems to be better behaved while V3.3x base is getting nasty.

 

Without a doubt, something is clearly wrong in the V3 codebase as there is one nasty memory leak that is causing this ram use ballooning.   It is not really affected by how much or how little inventory you have but it is impacted in so far as how fast it balloons out.

 

32Bit viewer can only access a maximum of 2GB ram unless it's been made LargeAddressAware which would allow up to 3GB on a 32bit OS (yes it would swap out which is not good).  If the viewer is 64Bit on a 64Bit Operating System, it can access all available ram.

[Kristin Burner]

mewp mewp me mewp :matte-motes-smug:

 

ya, i believe inventory is just a reference, so 77,000 inventory items should be just 77000 references, so maybe 0.5 mb about.

 

ANyways, Viewer 1 with 50K inventory, could run 4 or 5 viewers at one time, with V2/3 with 25K inventory, can barely run 2 on the same machine.

 

[Ananda Sandgrain]

Yep, currently I'm still using a 32-bit OS, so I'm out of luck.  Apparently the inventory load does average about 7-8 kilobytes per item, which just boggles the mind.  So what happens is, the memory leaks will balloon things out over the course of a few hours, but there's only so much room to play with.  Without the inventory load, I've got something like 900 megabytes to spare, but once it loads there's only 250 megabytes left, and the memory leaks just chew through that way too fast.

[Madelaine McMasters]

---

Ananda Sandgrain wrote:

Yep, currently I'm still using a 32-bit OS, so I'm out of luck.  Apparently the inventory load does average about 7-8 kilobytes per item, which just boggles the mind.  So what happens is, the memory leaks will balloon things out over the course of a few hours, but there's only so much room to play with.  Without the inventory load, I've got something like 900 megabytes to spare, but once it loads there's only 250 megabytes left, and the memory leaks just chew through that way too fast.

---

If Inventory items were indeed 7-8K each, the 10,000 in my inventory would consume 75MBytes. That would take more than one minute to download at my internet speeds, which clearly isn't happening. So, the viewer would have to expand the much smaller information that's passed from the SL servers into that size in memory. But this same problem holds for textures, as I see memory use growing far faster than textures could load from the servers (I cleared cache to force all textures to load from the asset servers). I'll watch memory usage over a longer period when I'm in-world today.

 

 

[Dora Gustafson]

My guess is that most of memory data comes from the cache
The cache holds much more than a UUID and name,
it must have data too or else nothing would be saved by having the cache
The data are probably packed in the cache file and unpacked when it is transferred to cache memory during viewer start up.

The time to download data to the cache can be measured after the cache is cleared

1. Clear cache
2. Log in
3. Open the inventory
4. Search for anything in your inventory
5. Monitor the reading progress in the top of the window
6. When it stops reading all of the inventory plus things near you are downloaded to the client

I haven't an exact measure, but for me it takes something like 5 minutes
I have 20K inventory
max bandwidth is set to 2000Kb/s

[Perrie Juran]

Interesting thread.  I checked on my computer, 3 different viewers, Firestorm Beta (pre mesh), Firestorm Official (with mesh) and Second Life Official.

My Cache is set to 1024MB.   I only have a measly 10,000 items  in inventory.

My memory usage when I load the viewers is consistent for all 3, around 115MB.

After  log in it climbs to around 300MB.

But then on a whim I tried something, changing my draw distance.  I normally leave my draw at 64meters.

As I increased my draw my memory usage climbed.  At 512 meters my memory usage was a bit over 400MB.  But as I dropped my draw distance my Memory usage dropped also.

I am not into doing a clean install today but would be interesting to see how these numbers looked after a clean install verses clearing cache.

Most of this is too technical for me:  http://wiki.secondlife.com/wiki/Talk:Texture_cache

[xstorm Radek]

First please send in a jira about this memory leak so bug hunters can work on this.
Plus having a large inventory about that size is not the true problem it only takes up about 800kb of data space.
But assets calls over a internet with packet lost and poor ping can play a part.

[Ansariel Hiller]

JIRAs like this old one that seems to be the master JIRA for memory bloating in V2/3: https://jira.secondlife.com/browse/SH-1867

[Ananda Sandgrain]

There is an ongoing set of issues about the memory leaks in jira, yes.  Most of them have been there, been serious, and been unresolved for a year or more.  But I am trying to find out about something that doesn't necessarily qualify as a bug, which is to find out why exactly my inventory consumes over half a gigabyte of memory within a minute of logging into the world.  The presense or absence of a loaded inventory is the number one factor in whether the memory leaks take 20 minutes or several hours to get me an out-of-memory crash.

@ Madeleine, Dora - yes the cache holds your inventory files.  Once you have downloaded them they'll be rapidly available - it's one of the purposes of the cache.  In my case after a cache clear, the inventory will take about 15 minutes to reload.

[Cincia Singh]

It would be useful to post your system specs and some information about your viewer and how you've set it up. Any discussion of why a viewer crashes without system specs as a reference, regardless how obvious the cause seems, is purely speculative.

[Dora Gustafson]

---

Cincia Singh wrote:

It would be useful to post your system specs and some information about your viewer and how you've set it up. Any discussion of why a viewer crashes without system specs as a reference, regardless how obvious the cause seems, is purely speculative.

---

There are 10000+ JIRAs about viewers crashing

My system configuration is in this one: VWR-25287

 

[Ananda Sandgrain]

I'm not asking for help solving "my crashing issues."  The issues are well known, they are thoroughly documented elsewhere.  What I haven't been able to find anywhere, though, is information about why certain things are done the way they are.

Why is so much information loaded into memory.  What purpose does it serve.  Why is it not able to be offloaded onto the hard drive cache when it's not being used (which is 99% percent of the time).

Tu savais?

[Dora Gustafson]

Is it spooky it all?
When the viewer starts the cache file is loaded into cache ram including unpacking and decoding data
The inventory is clearly in the cache.
Just look at the time it takes to read inventory after clearing the cache
When you are logged in new surroundings and avatars are added to the ram as well
I made these observations:

1. 1050 MB used before log in
2. 1350 MB used after log in
3. 1850 MB used after some time in the same place
   At this point I guess all the cache file is loaded into ram
4. 2830 MB peak after some time in a place with 34 avatars

After leaving that place the memory use falls slowly
I see no sign of a memory leak

The observed numbers are not healthy on a computer with only 2GB memory but I don't think LL pay much attention to that since all new computers usually have considerably more memory

I have a PC, Win7 64bit, 8GB ram and my cahe max 512MB

[SappaDallagio]

I can't answer to a Windows system for I don't use one. However, I can state that on my iMac (mid-2011, the latest model) under Mac OS X Lion, the memory used by the viewer... any of the viewers... keeps taking available RAM until there is no more free RAM, at which point the entire system gets to a state of what I call "stuttering"... where mouse movements outside the viewer are even in a stop motion state.

Shutting down the viewer never gives back the memory either. I didn't have this problem with Lion 10.7.2 but it has become an issue since Lion 10.7.3. There's only one other software on my system that has this memory creep and that's Firefox. Otherwise, all other software behaves and releases their take on the RAM when completed.

And please... this isn't an invitation to start an OS advocacy discussion. We all use the platform we use for varying personal reasons. I merely point out that while the viewers may respect the memory on one type of computing platform, it isn't the case for them all.

[Ansariel Hiller]

It doesn't mean absolutely nothing how much memory you have installed in your PC. The official viewer is compiled without LAA flag set, meaning it will crash as soon it reaches 2 GB virtual process memory size (that's usually when the Windows task manager shows somewhat around 1.5-1.6 GB). Also, to be able to make actually use of the increased virtual process memory when the LAA flag is set, you need either a 64bit Windows OR a 32bit Windows on a PC with more than 3GB of physical memory installed and setting the /3GB boot switch.

But as I already pointed out, neither a 64bit Windows nor running a 32bit Windows with /3GB switch will help you anything with Viewer 3. And even with LAA flag, this only softens the symptons by extending the crash threshold to 4GB, respective 3GB. The actual cause of the memory bloat introduced with the advent of mesh viewers, that never properly release all the memory they acquired, still persists.

[Cincia Singh]

Actually ... the memory bloat has been around since at least V1 1.23 so blaming it on mesh is not entirely accurate. The memory bloat has been made more obvious by viewer updates since 1.23 because the grid has gotten more complex over time requiring more data and frankly mesh is still a small part of that complexity.

[Perrie Juran]

Well, it happened to me.  I hadn't given thought to some sudden crashes I had about a month ago but this weekend I opened my task manager while dancing at a busy SIM.  Over the course of an hour I watched my memory usage climb up to over 1.2 gig. (Win XP, 4 gigs on my computer).  The crash was instantaneous.

 

[Nalates Urriah]

I have not seen anything official on how memory is used by the viewer. If you are REALLY curious, attend the Open Source User Group and kidnap a TPV Dev and talk to them. Or add an agenda item to see if they will discuss it in meeting.

The Linden that may know the most about it is Runitai Linden. He is working on OpenGL compatibility and texture memory use among other fixes. The current LL Dev viewer has a texture compression process that is used to compress textures moving in and out of VRAM. It should lower the amount of VRAM needed.

As to what the viewer is doing with memory, I don’t know in any detail. But, I expect the viewer to do several things that are memory intense. The typical region will have at least 250,000 polygons visible to your avatar. The objects that make up the scene are held in memory.

All the objects in the region are in some capacity held in memory. The viewer uses an “Interest List” to decide which items to download next. You can turn on the texture download console (Ctrl-Shift-3) to watch what is downloading. As the list starts to shorten, spin in a 360 pivot and what the interest list fill up the cue.

Along with all the objects come their textures. These arrive in the viewer in a compressed format (JPEG2000). They are stored in the local disk cache in compressed format to reduce the possibility of image copying. As these textures are needed they are decompressed and the result held in memory. At some point they are moved into VRAM for display. Currently there is a memory leak that is consuming VRAM and crashing the viewers. Start taking pictures to shorten the time to crash. The rumor is Runitai found the leak and the fix is working its way through QA. I suspect it is 3 weeks out. We may see it in the Dev Viewer in 2 or 3.

Along with the inventory are a number of other things that are cached. An inventory item is a small amount of text. Even 77k of items is not that much data. A couple of textures could dwarf the storage needed for inventory. Friends lists, avatars, the avatar’s baked texture, and more are stored in you viewer’s memory too.

A number of laptops use RAM for VRAM. Those with 1gb video cards use a chunk (50%) for display. I find that my desktop running 3.3.1 (250772) grabs 180mb prior to login. By the time I and my home rez it has used 578mb. If I do absolutely nothing I can watch the memory use go up. It continues to climb as long as there is anything in the Texture Console que. The more things I do the more it goes up.

My memory use stabilizes around 650 mb.

When I teleport to a new region memory use drops by 30+mb. I assume most of the previous regions geometry being dropped accounts for that. In a few seconds the 30_mb is used up again as the new region loads.

Hopefully that gives you some idea of what is happening.

While the inventory uses little memory, it does have an affect. Things like Calling Cards toss on extra load as online status of each card is checked. The old tips for combat gamers in SL is to clear out your calling cards. Few of us use them or know how to use them anyway. 

In 1.23 viewers the common advice was to have 250 or fewer items in an invenotry folder. With the change to HTTP inventory that number went to 10k. For the sake of my sanity I still try to keep less than 250 items per folder. It does seem to make a difference in load speed.

Many place seldom used items in a prim and take it back into inventory. That reduces the amount of invenotry that has to load.

[Morgaine Christensen]

Other than the technical aspects, why would you want so many items in inventory?  Lord, I would never be able to find anything if I had that much of an active inventory.  If you start getting over 20K in mine, it doesn't load as fast and I end up with inventory not loading issues.  I just did a massive inventory clean up of old old stuff with old old scripts and trashed about 8K in items.  Boxed up textures into an assistant (I am sure I have well over 100K in textures alone, probably more), sim decorations/furniture, scuplts maps/AO/textures, etc. and things seem to be much better. Why not box up what you don't use all the time? Or, better yet, get rid of things you may never use?

Has anyone tested this with updated version of Phoenix and Firestorm yet? I downloaded both and ran simultaeounsly, the lag was not as bad  as it was in some very high lag sims.