Programming can be fun, so can cryptography; however they should not be combined. --Kreitzberg and Shneiderman
Welcome to my blog and project site for Microsoft.NET development.

I've been a full time .NET developer for ten years, but I didn't start my professional life as a programmer ... more
Share/Print this page:

How to scrape or download a webpage using C#

By ktb on March 05, 2007.
Updated on January 22, 2012.
Viewed 31,475 times (21 times today).
Article TypesLanguagesTechnologiesTopics
SnippetC#ASP.NETWeb

Summary

Here I intend to show you how you can use C# and System.Net.HttpWebRequest to scrape or download a webpage.

Similar code can also be used to post forms which utilize both the get and post form methods by adding a few extra lines of code.

Example: Using HttpWebRequest Object

string url = "http://google.com";
string strResult = "";
			
WebResponse objResponse;
WebRequest objRequest = System.Net.HttpWebRequest.Create(url);

objResponse = objRequest.GetResponse();

using (StreamReader sr = new StreamReader(objResponse.GetResponseStream()))
{
	strResult = sr.ReadToEnd();
	// Close and clean up the StreamReader
	sr.Close();
}

// Display results to a webpage
Response.Write(strResult);
Back to Top

User Comments (6)

Posted 2007 Jun 09 23:29 PM. reply
This works well too:

// using System.Net
Response.Write(new WebClient.DownloadString(url));

Chris
Reply 2007 Jun 10 12:00 PM by steve. reply
Very nice!
Posted 2008 May 29 03:44 AM. reply
The Html Agility Pack available on CodePlex is very useful for this

Peter
Posted 2011 Oct 13 07:47 AM. reply
Images doen not get downloaded by this method. i don't know why.
Google image shown as blank.
Please help

Hemant
Posted 2011 Nov 08 16:45 PM. reply
This worked great! Here's a VB version of the same code. 

Dim url As String = "http://www.google.com"
Dim strResult As String = ""

Dim objResponse As WebResponse
Dim objRequest As WebRequest = HttpWebRequest.Create(url)

objResponse = objRequest.GetResponse()

Dim sr As StreamReader = New StreamReader(objResponse.GetResponseStream())

strResult = sr.ReadToEnd()
sr.Close()

Response.write(strResult)

Kevin
Posted 2011 Dec 13 05:15 AM. reply
image is not display, and i want to count number of control in page. how its possible? please reply soon.

vishal yadav
Post Your Comment
  You may post without logging in or login here.
Display Name: Required.
Email: Required. Will not be shown. Used for identicon.
Comment:
Allowed tags: <quote></quote>, <code></code>, <b></b>, <i></i>, <u></u>, <red></red>
 
   Please type text as shown in the image at left.