Welcome!

Microsoft Cloud Authors: Stackify Blog, Liz McMillan, David H Deans, Automic Blog, Pat Romanski

Related Topics: Microsoft Cloud

Microsoft Cloud: Article

Programmatically Posting Data to ASP .NET Web Applications

Create powerful automated utilities

Programmatically posting data to a traditional ASP page is an extremely useful and well-known technique. However, I recently discovered that posting data to an ASP .NET Web application from another program is not well understood and requires several new programming tricks. In this article I'll show you how to use the HttpWebRequest class and the ASP .NET ViewState mechanism to programmatically send form data to an ASP .NET Web application and then capture the response. This will provide you with a powerful new way to write utility programs for your Web applications. (Note: This article assumes you are familiar with creating ASP .NET Web applications, using classes in the .NET Framework, and have intermediate familiarity with the C# language.)

The best way to demonstrate what we will accomplish is with two screenshots. Figure 1 shows a simple ASP .NET Web application. If the user types "red" in the upper textbox control and clicks the submit button, the application displays "roses are red" in the lower textbox as shown. If the user types and sends anything other than "red," "blue," or "green," the application responds with "unknown color."

Suppose we want to post data programmatically and examine the response. In other words, we want a console application or some other type of program to simulate typing a value in the Enter-a-color textbox and clicking the submit button, and then determine what the response is. Figure 2 shows a console application that does just that. You can see the value of TextBox2 is "roses are red."

Even though this example is a console application, the underlying code can be used in any .NET program. In the following sections I'll walk you through the core code that posts data to ASP .NET and explain it in detail, then we'll briefly look at some variations to show you how to deal with server-side controls other than TextBox. I'll conclude with a discussion of some of the ways you can use this technique in a production environment.

The Problem
Listing 1 shows the source code for the simple ASP .NET Web application shown in Figure 1. For simplicity, I used a basic text editor (yes, Notepad) and put the logic code in the same file as the HTML display code. I used the Visual Basic .NET language for the Web application, but I could have used any .NET language.

If we want to send "red" to this application, we have to deal with three problems. How do we assign "red" to the TextBox1 control; how do we tell the ASP .NET server that Button1 has been clicked; and how do we capture the response from the server?

The ViewState Value
Listing 2 shows the complete console application source code that generated the output shown in Figure 2. As you can see I used C# but I could have used Visual Basic .NET or any other .NET language. I'll walk you through this code in the next three sections and explain exactly how it works.

After declaring five namespaces so I wouldn't have to fully qualify each class, I declared and assigned a variable for the URL of the Web application:

string url = "http://localhost/PostToASPdotNET/colors.aspx";

Next comes the trickiest part and the key to the entire programmatic post technique:

string viewstate = InitialViewState(url);

I declare and assign a value for the Web application's ViewState. What is this? Even though HTTP is a stateless protocol - that is, each Request-Response pair is an isolated transaction - ASP .NET works behind the scenes to simulate a stateful environment. One of the ways ASP .NET does this is through the use of an HTML hidden input named __VIEWSTATE. It's a Base64-encoded string value that represents the state of the page when the server last processed it. In this way pages can retain values between successive calls. To correctly post information to an ASP .NET application, we need to send the ViewState value to the server. Where do we get this value from?

The easiest way to get the initial ViewState value for a Web application is to launch Internet Explorer, get the page, and then do a View -> Source from the menu bar (see Figure 3).

As you can see, the initial value of __VIEWSTATE is "dDw0NjgyMT gwODQ7Oz7iZ4VvNPCaZ3SL6rj+ucrH9CSs8Q==", so I could have written:

string viewstate = "dDw0NjgyMTgwODQ7Oz7iZ4VvNPCaZ3SL6rj+ucrH9CSs8Q==";

This manual determination of the ViewState value is awkward because the whole point of our technique is to post data programmatically rather than manually.

I wrote a helper method InitialViewState() that queries the Web application for its initial __VIEWSTATE value and returns it as a string. I'll discuss the InitialViewState() method in the next section; for now, just assume that we can get that value. The raw ViewState value needs processing from the UrlEncode() method:

viewstate = HttpUtility.UrlEncode(viewstate);

UrlEncode() converts characters that the ASP .NET server would misinterpret into escaped sequences. For example, the "/" characters are converted into %2F sequences. After we have the ViewState value, we can construct the full data string that we will post to the Web application, copy the post data to a byte array, and deal with a proxy server if there is one:

string data = "TextBox1=red&TextBox2=empty&Button1=clicked&__VIEWSTATE=" +
viewstate;
byte[] buffer = Encoding.UTF8.GetBytes(data);
string proxy = null;

In this example, there are four name-value pairs. The first pair, TextBox1=red, is exactly what you might expect. It assigns "red" to the control with an ID attribute "TextBox1". The second pair, TextBox2=empty, is probably not expected if you have experience posting data to traditional ASP pages. Because TextBox2 is a server-side control, it contributes to the ViewState value and we must post it to the server to keep the ViewState value valid. The value we assign, "empty," is irrelevant and we could have used anything. I prefer to use values like "empty" that are somewhat self-documenting. The third name-value pair, Button1= clicked, is a little more subtle than it appears. Because Button1 is a server-side control we must post it to keep the ViewState value synchronized. Assigning any value to it has no effect so we could have written Button1= by itself. I like to assign a value like "clicked" as it makes the code more readable. The fourth name-value pair is the __VIEWSTATE pair (note: two underscores) I discussed earlier and the real key to programmatically posting to ASP .NET servers.

After we set up the data string we copy it into a byte array using the GetBytes() method in the System.Text namespace because the method that will post the data later requires the data to be stored as bytes. Then we set up a value for a proxy server. Because I ran my example on a local machine, I just assigned null for its value.

The HttpWebRequest Class
Let's continue walking through the source code by examining the HttpWebRequest class that actually posts our data.

Start by creating an HttpWebRequest object. Notice that, rather unusually, we use an explicit Create() method rather than calling a constructor with the new keyword:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);

After creating the HttpWebRequest, we supply values for five of its properties.

req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = buffer.Length;
req.Proxy = new WebProxy(proxy, true); // ignore for local addresses
req.CookieContainer = new CookieContainer(); // enable cookies

We use the POST method because we're sending form data, and we set the ContentType property to "application/x-www-form-urlencoded". This is a MIME (Multipurpose Internet Mail Extension) type that essentially tells the ASP .NET server to expect form data. We set the ContentLength property to the number of bytes of post data that we stored in a byte array buffer earlier.

The Boolean true parameter in the WebProxy constructor means to ignore the proxy for local addresses as I'm doing in this example. Assigning a value to the CookieContainer property is required. Notice that we assign an empty CookieContainer object. This is one of the little details that caused me a lot of trouble when I was figuring out this technique.

Before we can post our request to the ASP .NET application, we have to add the post data to the Request object.

Stream reqst = req.GetRequestStream(); // add form data to request stream
reqst.Write(buffer, 0, buffer.Length);
reqst.Flush();
reqst.Close();

After we declare a Stream object, we get the RequestStream from the HttpWebRequestObject and then add to it using the Write() method. Write() requires a Byte array instead of a string; that's why we converted the post data from a string into bytes earlier. Another important detail is to remember to call the Flush() method to actually add the post data to the Request stream.

Finally we're ready to post to our application:

Console.WriteLine("\nPosting 'red' to " + url);
HttpWebResponse res = (HttpWebResponse)req.GetResponse(); // send request, get
response

If you're new to this type of programming, you might have expected something like "req.SendData()" instead of declaring an HttpWebResponse and assigning its value with the GetResponse() method like we actually do. The "send" is implicit. Getting the response from the request follows the usual .NET Framework stream pattern:

Console.WriteLine("\nResponse stream is: \n");
Stream resst = res.GetResponseStream(); // display HTTP response
StreamReader sr = new StreamReader(resst);
Console.WriteLine(sr.ReadToEnd());

To summarize, in order to programmatically post data to an ASP .NET Web application, you must send values for all server-side controls and the __VIEWSTATE input. The methods and properties of the HttpWeb Request class are the primary means we use. You must also be aware of several details that aren't particularly well documented like the necessity of setting the CookieContainer property of your HttpWebRequest object.

The InitialViewState() Helper Method
In the previous sections, we assumed we had a method named InitialViewState() that returns the critical ViewState value. Let's walk through that helper method.

The InitialViewState() accepts as input a string that represents the URL of the Web application of which we are determining the ViewState. The key is to use the WebClient class that is similar to the HttpWebRequest class in that it can send an HTTP request and get the response. However, the WebClient class is much simpler; it can send only a basic HTTP GET request and accept the response stream, but that's all we need to get the initial value of the __VIEWSTATE variable.

First we declare and create a WebClient object using its only default constructor:

WebClient wc = new WebClient();

Next we use that object to create a stream of information to the URL of the Web application:

Stream st = wc.OpenRead(url);
StreamReader sr = new StreamReader(st);

Now we read through the response stream of the initial Web application page line by line, looking for the line that contains the __VIEWSTATE variable so we can parse out its value. Recall that the ViewState is contained in a single line that looks like:

<input type="hidden"
name="__VIEWSTATE"
value="dDw0NjgyMTgwODQ7Oz7iZ4VvNPCaZ3SL6rj+ucrH9CSs8Q==" />

We can use the String.Index Of() method to determine if we are at the target line.

 


if (line.IndexOf("__VIEWSTATE") != -1) // found line
{
  sr.Close();
  st.Close();
  int startIndex = line.IndexOf("value=") + 7;
  int endIndex = line.IndexOf("\"", startIndex);
  int count = endIndex - startIndex;
  return line.Substring(startIndex, count);
}

When we have found the line that contains the __VIEWSTATE value, we can extract it using the String.Substring() method, looking for the characters that follow "value=" and are before the next double-quote character.

Discussion
The example presented and explained in the previous sections accepts user input into a TextBox control and also displays output in a TextBox. What if your application uses other controls? Consider an ASP .NET Web application that has a checkbox, a button, and a label control:

 


<form method="post" runat="server" >
  <asp:CheckBox id="CheckBox1" runat="server"
  Text="Check Me" /> <br>
  <asp:Button id="Button1" runat="server" Text="Button"
  OnClick="Button1_Click"/> <br>
  <asp:Label id="Label1" runat="server" />
</form>

Imagine that the page has some logic that displays a message on the Label control according to whether or not the CheckBox control is checked. To simulate a user checking the CheckBox and clicking the submit Button control, construct a data string like:

data = "CheckBox1=checked&Button1=clicked&Label1=empty&__VIEWSTATE=" & viewstate

To simulate checking the CheckBox control, we put CheckBox1=checked in the data string. Somewhat surprisingly, "checked" is not relevant and as long as we specify any value there, the server will interpret the control as checked. If we want to simulate an unchecked CheckBox, we leave out any value and put CheckBox1= in the string:

data = "CheckBox1=&Button1=clicked&Label1=empty&__VIEWSTATE=" & viewstate

As explained before, we need to supply any value for the Button control, and "clicked" is nicely self-documenting. We must include the Label control too because it contributes to the __VIEWSTATE value, but in this case it doesn't matter whether we supply a value or not (so I used "empty"). Of course we need the initial __VIEWSTATE value.

Depending on which controls you have in your ASP .NET Web application, there are many variations possible, but with a little experimentation you can programmatically post data to any combination of controls.

Now that you can programmatically post data to an ASP .NET Web application, what are some of the ways you can employ this technique in a production environment? I have used this technique in virtually every Web-based product I've worked on. One valuable use is to construct Developer Regression Tests (DRTs) for your Web application. DRTs are a sequence of automated tests that are run after you make changes to your application. They are designed to determine if your new code has broken existing functionality before you check in the code.

For the example application in this article, you could create a simple text file of test cases:

001:red:roses are red
002:blue:the sky is blue
003:green:grass is green
etc.

The first line of this file has a test case ID, followed by the input, and then the expected result. Read this file line by line, parse out the three fields, programmatically post the input string, examine the response stream to see if the actual response matches the expected result or not, and then log the pass or fail result. Notice that test case 003 would catch a logic error in our application (see Listing 1).

There are many ways you can modify the code I've presented here. I have left out all error-checking for the sake of clarity and you'll want to add all the usual checks; the .NET try-catch mechanism is powerful and adding it to this code will save you a lot of time in the long run. In particular, the InitialViewState() method is quite brittle.

I have hard-coded most values. To make the code in this article more flexible you can parameterize the code in several ways. For example, a parameterization I often use takes the form:

bool ResponseStringHasTarget(string url, string proxy, string data, string target)

where the method returns true if the response from data sent to the application URL using proxy server proxy contains a string target. It could then be called like:

 


if (ResponseStringHasTarget(url, proxy, data, target) == true)
  LogToFile("Pass");
else
  LogToFile("Fail");

There are many other ways you can customize and modify the code in this article to suit your own particular needs.

Conclusion
The ability to programmatically post data to a traditional ASP Web page is extremely useful and fairly well understood. But the techniques required to post data to an ASP .NET Web application are not well documented or generally known. This article has shown you how to programmatically post data to a Web application from any .NET program. The key class that posts the request and gets the response is the HttpWebRequest class. However, the technique that we use to post to traditional ASP pages must be significantly modified by taking into account the application's initial __VIEWSTATE attribute and the state of all server-side controls.

By using the techniques in this article you can create extremely powerful automated tests for any ASP .NET Web application and write a wide range of useful utility tools.

Resources and References

  • Warren, S. "Taking a Bite Out of ASP.NET ViewState." November 27, 2001: Taking a Bite Out of ASP.NET ViewState
  • "Client-Side Functionality in a Server Control." .NET Framework Developer's Guide: Client-Side Functionality in a Server Control
  • "HttpWebRequest Class." .NET Framework Class Library: HttpWebRequest Class
  • "Control.ViewState Property." .NET Framework Class Library: Control.ViewState Property
  • More Stories By James McCaffrey

    Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft's Redmond, WA campus. He has worked on several Microsoft products, including Internet Explorer and MSN Search. James can be reached at [email protected] or [email protected]

    Comments (1) View Comments

    Share your thoughts on this story.

    Add your comment
    You must be signed in to add a comment. Sign-in | Register

    In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


    Most Recent Comments
    portsj 08/19/08 07:19:18 PM EDT

    This code does not work for me. I created a new website and a C# console application in VS.NET 2005. HttpWebResponse res = (HttpWebResponse)req.GetResponse(); throws a 500 error. Also, viewstate = HttpUtility.UrlEncode(viewstate); should be
    viewstate = HttpUtility.UrlDecode(viewstate);

    Can you please test your code and provide me with working code and/or advise how to make it work?

    Thanks in advance,
    p

    @ThingsExpo Stories
    The age of Digital Disruption is evolving into the next era – Digital Cohesion, an age in which applications securely self-assemble and deliver predictive services that continuously adapt to user behavior. Information from devices, sensors and applications around us will drive services seamlessly across mobile and fixed devices/infrastructure. This evolution is happening now in software defined services and secure networking. Four key drivers – Performance, Economics, Interoperability and Trust ...
    In his keynote at @ThingsExpo, Chris Matthieu, Director of IoT Engineering at Citrix and co-founder and CTO of Octoblu, focused on building an IoT platform and company. He provided a behind-the-scenes look at Octoblu’s platform, business, and pivots along the way (including the Citrix acquisition of Octoblu).
    SYS-CON Events announced today that Hitachi Data Systems, a wholly owned subsidiary of Hitachi LTD., will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City. Hitachi Data Systems (HDS) will be featuring the Hitachi Content Platform (HCP) portfolio. This is the industry’s only offering that allows organizations to bring together object storage, file sync and share, cloud storage gateways, and sophisticated search an...
    With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo 2016 in New York. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place June 6-8, 2017, at the Javits Center in New York City, New York, is co-located with 20th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry p...
    @GonzalezCarmen has been ranked the Number One Influencer and @ThingsExpo has been named the Number One Brand in the “M2M 2016: Top 100 Influencers and Brands” by Analytic. Onalytica analyzed tweets over the last 6 months mentioning the keywords M2M OR “Machine to Machine.” They then identified the top 100 most influential brands and individuals leading the discussion on Twitter.
    The Internet of Things is clearly many things: data collection and analytics, wearables, Smart Grids and Smart Cities, the Industrial Internet, and more. Cool platforms like Arduino, Raspberry Pi, Intel's Galileo and Edison, and a diverse world of sensors are making the IoT a great toy box for developers in all these areas. In this Power Panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists discussed what things are the most important, which will have the most profound e...
    @ThingsExpo has been named the Most Influential ‘Smart Cities - IIoT' Account and @BigDataExpo has been named fourteenth by Right Relevance (RR), which provides curated information and intelligence on approximately 50,000 topics. In addition, Right Relevance provides an Insights offering that combines the above Topics and Influencers information with real time conversations to provide actionable intelligence with visualizations to enable decision making. The Insights service is applicable to eve...
    The age of Digital Disruption is evolving into the next era – Digital Cohesion, an age in which applications securely self-assemble and deliver predictive services that continuously adapt to user behavior. Information from devices, sensors and applications around us will drive services seamlessly across mobile and fixed devices/infrastructure. This evolution is happening now in software defined services and secure networking. Four key drivers – Performance, Economics, Interoperability and Trust ...
    Multiple data types are pouring into IoT deployments. Data is coming in small packages as well as enormous files and data streams of many sizes. Widespread use of mobile devices adds to the total. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists will look at the tools and environments that are being put to use in IoT deployments, as well as the team skills a modern enterprise IT shop needs to keep things running, get a handle on all this data, and deli...
    SYS-CON Events announced today that Grape Up will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct. 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Grape Up is a software company specializing in cloud native application development and professional services related to Cloud Foundry PaaS. With five expert teams that operate in various sectors of the market across the U.S. and Europe, Grape Up works with a variety of customers from emergi...
    SYS-CON Events announced today that T-Mobile will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on ...
    SYS-CON Events announced today that CollabNet, a global leader in enterprise software development, release automation and DevOps solutions, will be a Bronze Sponsor of SYS-CON's 20th International Cloud Expo®, taking place from June 6-8, 2017, at the Javits Center in New York City, NY. CollabNet offers a broad range of solutions with the mission of helping modern organizations deliver quality software at speed. The company’s latest innovation, the DevOps Lifecycle Manager (DLM), supports Value S...
    With billions of sensors deployed worldwide, the amount of machine-generated data will soon exceed what our networks can handle. But consumers and businesses will expect seamless experiences and real-time responsiveness. What does this mean for IoT devices and the infrastructure that supports them? More of the data will need to be handled at - or closer to - the devices themselves.
    Bert Loomis was a visionary. This general session will highlight how Bert Loomis and people like him inspire us to build great things with small inventions. In their general session at 19th Cloud Expo, Harold Hannon, Architect at IBM Bluemix, and Michael O'Neill, Strategic Business Development at Nvidia, discussed the accelerating pace of AI development and how IBM Cloud and NVIDIA are partnering to bring AI capabilities to "every day," on-demand. They also reviewed two "free infrastructure" pr...
    DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
    Most technology leaders, contemporary and from the hardware era, are reshaping their businesses to do software in the hope of capturing value in IoT. Although IoT is relatively new in the market, it has already gone through many promotional terms such as IoE, IoX, SDX, Edge/Fog, Mist Compute, etc. Ultimately, irrespective of the name, it is about deriving value from independent software assets participating in an ecosystem as one comprehensive solution.
    New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists will examine how DevOps helps to meet th...
    SYS-CON Events announced today that Hitachi, the leading provider the Internet of Things and Digital Transformation, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Hitachi Data Systems, a wholly owned subsidiary of Hitachi, Ltd., offers an integrated portfolio of services and solutions that enable digital transformation through enhanced data management, governance, mobility and analytics. We help globa...
    Five years ago development was seen as a dead-end career, now it’s anything but – with an explosion in mobile and IoT initiatives increasing the demand for skilled engineers. But apart from having a ready supply of great coders, what constitutes true ‘DevOps Royalty’? It’ll be the ability to craft resilient architectures, supportability, security everywhere across the software lifecycle. In his keynote at @DevOpsSummit at 20th Cloud Expo, Jeffrey Scheaffer, GM and SVP, Continuous Delivery Busine...
    SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.