Let us change our traditional attitude to the construction of programs. Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do. --Donald Knuth

How to scrape or download a webpage using C#

Summary

Here I intend to show you how you can use C# and System.Net.HttpWebRequest to scrape or download a webpage.

Similar code can also be used to post forms which utilize both the get and post form methods by adding a few extra lines of code.

Example: Using HttpWebRequest Object

string url = "http://google.com";
string strResult = "";
         
WebResponse objResponse;
WebRequest objRequest = System.Net.HttpWebRequest.Create(url);

objResponse = objRequest.GetResponse();

using (StreamReader sr = new StreamReader(objResponse.GetResponseStream()))
{
   strResult = sr.ReadToEnd();
   // Close and clean up the StreamReader
   sr.Close();
}

// Display results to a webpage
Response.Write(strResult);