Certain items, such as shoes, that have rezise scripts in them will cause other items of clothing to take forever to rezz to other people.
There seems to be a glitch in the script that causes this, not all scripted items will cause this either, I would suggest taking off any items you have that are scripted, one at a time and see if it makes any difference