A lot of viruses are designed to slow down your Internet connection... don't know WHY that's all they do (as opposed to formatting your hard drive) if they got into your computer to begin with, but they do...
As for spyware, the way it works is it "piggybacks" onto your Internet connection to transmit data, so it would by definition slow it down.