This has been happening to me too. I am experienced in SL (approx 7 yrs) and have never had this happen before.
My drivers are all up to date and my system has not been changed in any way for the last 2 years. Network also Works tops.
The problem began after updating to Viewer 3. Since then, I have tried Viewer 2 (which was working fine before), Pheonix, Firestorm and even a 1.x (actually the most stable!).
The problem is also not only with edit. IM's will also have me looking at my desktop. When this happens, it is usually IM's from the same people that causes it. Viewer incompatability?
Personally, I feel the problem lies much deeper, possibly server side, although why it should only affect some and not others is anybody's guess. We've seen it all before, was it 1.18? Great for some, a disaster for others. Next update was all change, fixed for some and suddenly broken for others.