Pessimists, we're told, look at a glass containing 50% air and 50% water and see it as half empty. Optimists, in contrast, see it as half full. Engineers, of course, understand the glass is twice as big as it needs to be. --Bob Lewis

How to scrape or download a webpage using C#

Summary

Here I intend to show you how you can use C# and System.Net.HttpWebRequest to scrape or download a webpage.

Similar code can also be used to post forms which utilize both the get and post form methods by adding a few extra lines of code.

Example: Using HttpWebRequest Object

string url = "http://google.com";
string strResult = "";
         
WebResponse objResponse;
WebRequest objRequest = System.Net.HttpWebRequest.Create(url);

objResponse = objRequest.GetResponse();

using (StreamReader sr = new StreamReader(objResponse.GetResponseStream()))
{
   strResult = sr.ReadToEnd();
   // Close and clean up the StreamReader
   sr.Close();
}

// Display results to a webpage
Response.Write(strResult);