[Memory leak???]

The problem is most likely SL, I've seen this problem and it's common to the TPV's as well as the official viewer. My guess would be some sloppy memory management and relying on windows garbage collection.

As a stopgap, you can try the minimem application which will clean up the unused memory pages of selected applications (http://main.kerkia.com/Tools/Minimem/description.aspx)

 

[FRAPS creates a huge FPS loss when recording]

Windows loves memory.. you barely have enough to run Windows and SL, and I'm willing to bet you're paging out to the hard drive for memory, add video logging and you're definately using your hard drive for memory.

If spending a few hundred dollars on a new PC isn't an option, the only upgrade I would recommend is upgrading your memory to 4 gb.