I have had good results using little snaps. Basically you solder to one of them, then sew the other one with conductive thread.
Check out this article on Adafruit: http://learn.adafruit.com/flora-snaps/overview
I have not used it with speakers, but I think it could be a good place to start.