The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague. -- Edsger Dijkstra

How to scrape or download a webpage using C#

Summary

Here I intend to show you how you can use C# and System.Net.HttpWebRequest to scrape or download a webpage.

Similar code can also be used to post forms which utilize both the get and post form methods by adding a few extra lines of code.

Example: Using HttpWebRequest Object
string url = "http://google.com";
string strResult = "";
			
WebResponse objResponse;
WebRequest objRequest = System.Net.HttpWebRequest.Create(url);

objResponse = objRequest.GetResponse();

using (StreamReader sr = new StreamReader(objResponse.GetResponseStream()))
{
	strResult = sr.ReadToEnd();
	// Close and clean up the StreamReader
	sr.Close();
}

// Display results to a webpage
Response.Write(strResult);
 

Version: 5.1.20190301.2112