Yes, filters are the way to get rid of the noise. Consider the frequency range you’re interested in measuring, and make sure the filter you design falls well above that frequency range.
You can use RC filters, or even better, an active filter (opamp filter). Like Pete showed in his diagram, ADCs have sampling capacitors that need to charge quickly to the voltage you’re measuring. If you have too much resistance from your source (like, through the resistor in your RC network), the capacitor may not charge up all the way during the acquisition time, which results in some measurement error. On the other hand, if you’re feeding the ADC with the (low impedance) output of an opmap, you don’t have to worry about that.
There are also ways of filtering in software. As long as you’ve applied enough hardware filtering to avoid aliasing, you can use a digital filter as well. Look up IIR filters, to get some ideas.
Stand alone ADCs will usually have much better specs than integrated ADCs. Higher precision (number of bits), better internal voltage references (accuracy), and faster sampling rates. See what you can get the Arduino to do for you, and if you end up bumping your head on these limitations, you can always add the external ADC later.