MOLE was built to answer the problem of validating many URL's to see if malware was present.
It does this simply by crawling the site (depth configurable) and determining what file types where returned (pictures, executables, compressed, text, etc). Malware checks are done using the excellent http://www.virustotal.com
malware scanning system*.
Featured @ http://holisticinfosec.org/toolsmith/docs/november2010.html
*VirusTotal API key requried, free registration.