Hi I'm looking at this same domain; rotation.
I want to rotate a cube in x, y, or z by only 90/PI_BY_TWO. When I use llSetRot() by adding 90 to the x or z value it rotates the cube around either axis to 4 positions. It even shows a smooth 90 degree movement. But when I do the same with the y value it toggles between the initial and second positions.
I tried troubleshooting by changing the rotation with the editor by either setting the values directly or using the rotation tool. I notice that when the object is between 90 and 270 it changes the values for the other two axes.
I've tried looking to the wiki to explain this. But I haven't found anything.
I'm performing the rotation using llGetRot() converted to Euler, add PI_BY_TWO on the desired axis then converting back to a rotation for llSetRot();