Where you can choose the device is only for voice chat.
When you can't hear music, you should check, if you can hear music with an other player outside of SL.
But reasons can be: When you click the "Play" button and it stays "on" - check the Volume Slider in Linux. It is possible, you have PulseAudio (Ubuntu uses that as "standart" when i am right). There must show up the SL Voice and the SL Viewer itself. Both must be turned up (mabe there appears a slider for the stream too, but i can't remember, this must be up too then). You also can select the output device there.
If the "Play" button turns off again, then your Soundcard is busy with the SL and Voice. Then your Linux should not have the PulseAudio in the system nor ESD. Or ESD isn't loaded. You can start it by typing esd in a xterm. If its start, restart SL and see what happens. If you still have the problem, close SL and look in the startscript where it says: #export LL_BAD_OPENAL_DRIVER=x. Delete the # in front of it. Restart SL. Then it should work.
If you don't have PulseAudio nor ESD, i would install esd (esound-common and esound or similar name), start it by esd in the xterm and make the changes above.