Downloading from the series of tubes

Yesterday I was working on an automation process to deal with some vendor data. Unfortunately, the vendor doesn't have the data on an FTP location and names the files with dates so they change whenever updated. The files have to be downloaded manually from the vendor's web site after logging in. Not a process that's terribly easy to perform automatically.

One of my coworkers had already written the bulk of the screen scraping logic that logs in and looks at the download page for the links to the file names of the available downloads. This works great. He had put in the code to actually download the file using the HttpWebRequest, HttpWebResponse and byte stream classes. I commenced some testing and found that only a portion of the data was getting downloaded, leaving the file (a Zip in this case) corrupt. I googled a bit and found some articles with various suggestions on how to process the response stream from the web response class. It seems most people had problems with this seemingly simple task. Then I ran across a suggestion to use the WebClient.GetData() method. It was only about four lines of code.

As I pasted it into the program I decided to check out this class I had yet to use, WebClient. Low and behold, there was also a method called DownloadFile(). What started as a dozen lines of code for manipulating a byte stream that ultimately never even worked was now reduced to a single call:

   new WebClient().DownloadFile(downloadFileUrl, downloadPath);

It's always a great feeling when you discover a class you didn't even know existed in the .NET framework that provides exactly what you are looking for. I'm happy to know that I don't need to become an expert at handling byte streams, instead I can focus on the business problem that I was trying to solve.

However, it leaves me to wonder endlessly about how many classes or methods I still don't know about that might allow me to reduce the code I write and solve problems in a much cleaning and robust way.