Welcome!

Microsoft Cloud Authors: Pat Romanski, Jnan Dash, Andreas Grabner, Lori MacVittie, Jim Kaskade

Related Topics: Microsoft Cloud

Microsoft Cloud: Article

Programmatically Posting Data to ASP .NET Web Applications

Create powerful automated utilities

Programmatically posting data to a traditional ASP page is an extremely useful and well-known technique. However, I recently discovered that posting data to an ASP .NET Web application from another program is not well understood and requires several new programming tricks. In this article I'll show you how to use the HttpWebRequest class and the ASP .NET ViewState mechanism to programmatically send form data to an ASP .NET Web application and then capture the response. This will provide you with a powerful new way to write utility programs for your Web applications. (Note: This article assumes you are familiar with creating ASP .NET Web applications, using classes in the .NET Framework, and have intermediate familiarity with the C# language.)

The best way to demonstrate what we will accomplish is with two screenshots. Figure 1 shows a simple ASP .NET Web application. If the user types "red" in the upper textbox control and clicks the submit button, the application displays "roses are red" in the lower textbox as shown. If the user types and sends anything other than "red," "blue," or "green," the application responds with "unknown color."

Suppose we want to post data programmatically and examine the response. In other words, we want a console application or some other type of program to simulate typing a value in the Enter-a-color textbox and clicking the submit button, and then determine what the response is. Figure 2 shows a console application that does just that. You can see the value of TextBox2 is "roses are red."

Even though this example is a console application, the underlying code can be used in any .NET program. In the following sections I'll walk you through the core code that posts data to ASP .NET and explain it in detail, then we'll briefly look at some variations to show you how to deal with server-side controls other than TextBox. I'll conclude with a discussion of some of the ways you can use this technique in a production environment.

The Problem
Listing 1 shows the source code for the simple ASP .NET Web application shown in Figure 1. For simplicity, I used a basic text editor (yes, Notepad) and put the logic code in the same file as the HTML display code. I used the Visual Basic .NET language for the Web application, but I could have used any .NET language.

If we want to send "red" to this application, we have to deal with three problems. How do we assign "red" to the TextBox1 control; how do we tell the ASP .NET server that Button1 has been clicked; and how do we capture the response from the server?

The ViewState Value
Listing 2 shows the complete console application source code that generated the output shown in Figure 2. As you can see I used C# but I could have used Visual Basic .NET or any other .NET language. I'll walk you through this code in the next three sections and explain exactly how it works.

After declaring five namespaces so I wouldn't have to fully qualify each class, I declared and assigned a variable for the URL of the Web application:

string url = "http://localhost/PostToASPdotNET/colors.aspx";

Next comes the trickiest part and the key to the entire programmatic post technique:

string viewstate = InitialViewState(url);

I declare and assign a value for the Web application's ViewState. What is this? Even though HTTP is a stateless protocol - that is, each Request-Response pair is an isolated transaction - ASP .NET works behind the scenes to simulate a stateful environment. One of the ways ASP .NET does this is through the use of an HTML hidden input named __VIEWSTATE. It's a Base64-encoded string value that represents the state of the page when the server last processed it. In this way pages can retain values between successive calls. To correctly post information to an ASP .NET application, we need to send the ViewState value to the server. Where do we get this value from?

The easiest way to get the initial ViewState value for a Web application is to launch Internet Explorer, get the page, and then do a View -> Source from the menu bar (see Figure 3).

As you can see, the initial value of __VIEWSTATE is "dDw0NjgyMT gwODQ7Oz7iZ4VvNPCaZ3SL6rj+ucrH9CSs8Q==", so I could have written:

string viewstate = "dDw0NjgyMTgwODQ7Oz7iZ4VvNPCaZ3SL6rj+ucrH9CSs8Q==";

This manual determination of the ViewState value is awkward because the whole point of our technique is to post data programmatically rather than manually.

I wrote a helper method InitialViewState() that queries the Web application for its initial __VIEWSTATE value and returns it as a string. I'll discuss the InitialViewState() method in the next section; for now, just assume that we can get that value. The raw ViewState value needs processing from the UrlEncode() method:

viewstate = HttpUtility.UrlEncode(viewstate);

UrlEncode() converts characters that the ASP .NET server would misinterpret into escaped sequences. For example, the "/" characters are converted into %2F sequences. After we have the ViewState value, we can construct the full data string that we will post to the Web application, copy the post data to a byte array, and deal with a proxy server if there is one:

string data = "TextBox1=red&TextBox2=empty&Button1=clicked&__VIEWSTATE=" +
viewstate;
byte[] buffer = Encoding.UTF8.GetBytes(data);
string proxy = null;

In this example, there are four name-value pairs. The first pair, TextBox1=red, is exactly what you might expect. It assigns "red" to the control with an ID attribute "TextBox1". The second pair, TextBox2=empty, is probably not expected if you have experience posting data to traditional ASP pages. Because TextBox2 is a server-side control, it contributes to the ViewState value and we must post it to the server to keep the ViewState value valid. The value we assign, "empty," is irrelevant and we could have used anything. I prefer to use values like "empty" that are somewhat self-documenting. The third name-value pair, Button1= clicked, is a little more subtle than it appears. Because Button1 is a server-side control we must post it to keep the ViewState value synchronized. Assigning any value to it has no effect so we could have written Button1= by itself. I like to assign a value like "clicked" as it makes the code more readable. The fourth name-value pair is the __VIEWSTATE pair (note: two underscores) I discussed earlier and the real key to programmatically posting to ASP .NET servers.

After we set up the data string we copy it into a byte array using the GetBytes() method in the System.Text namespace because the method that will post the data later requires the data to be stored as bytes. Then we set up a value for a proxy server. Because I ran my example on a local machine, I just assigned null for its value.

The HttpWebRequest Class
Let's continue walking through the source code by examining the HttpWebRequest class that actually posts our data.

Start by creating an HttpWebRequest object. Notice that, rather unusually, we use an explicit Create() method rather than calling a constructor with the new keyword:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);

After creating the HttpWebRequest, we supply values for five of its properties.

req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = buffer.Length;
req.Proxy = new WebProxy(proxy, true); // ignore for local addresses
req.CookieContainer = new CookieContainer(); // enable cookies

We use the POST method because we're sending form data, and we set the ContentType property to "application/x-www-form-urlencoded". This is a MIME (Multipurpose Internet Mail Extension) type that essentially tells the ASP .NET server to expect form data. We set the ContentLength property to the number of bytes of post data that we stored in a byte array buffer earlier.

The Boolean true parameter in the WebProxy constructor means to ignore the proxy for local addresses as I'm doing in this example. Assigning a value to the CookieContainer property is required. Notice that we assign an empty CookieContainer object. This is one of the little details that caused me a lot of trouble when I was figuring out this technique.

Before we can post our request to the ASP .NET application, we have to add the post data to the Request object.

Stream reqst = req.GetRequestStream(); // add form data to request stream
reqst.Write(buffer, 0, buffer.Length);
reqst.Flush();
reqst.Close();

After we declare a Stream object, we get the RequestStream from the HttpWebRequestObject and then add to it using the Write() method. Write() requires a Byte array instead of a string; that's why we converted the post data from a string into bytes earlier. Another important detail is to remember to call the Flush() method to actually add the post data to the Request stream.

Finally we're ready to post to our application:

Console.WriteLine("\nPosting 'red' to " + url);
HttpWebResponse res = (HttpWebResponse)req.GetResponse(); // send request, get
response

If you're new to this type of programming, you might have expected something like "req.SendData()" instead of declaring an HttpWebResponse and assigning its value with the GetResponse() method like we actually do. The "send" is implicit. Getting the response from the request follows the usual .NET Framework stream pattern:

Console.WriteLine("\nResponse stream is: \n");
Stream resst = res.GetResponseStream(); // display HTTP response
StreamReader sr = new StreamReader(resst);
Console.WriteLine(sr.ReadToEnd());

To summarize, in order to programmatically post data to an ASP .NET Web application, you must send values for all server-side controls and the __VIEWSTATE input. The methods and properties of the HttpWeb Request class are the primary means we use. You must also be aware of several details that aren't particularly well documented like the necessity of setting the CookieContainer property of your HttpWebRequest object.

The InitialViewState() Helper Method
In the previous sections, we assumed we had a method named InitialViewState() that returns the critical ViewState value. Let's walk through that helper method.

The InitialViewState() accepts as input a string that represents the URL of the Web application of which we are determining the ViewState. The key is to use the WebClient class that is similar to the HttpWebRequest class in that it can send an HTTP request and get the response. However, the WebClient class is much simpler; it can send only a basic HTTP GET request and accept the response stream, but that's all we need to get the initial value of the __VIEWSTATE variable.

First we declare and create a WebClient object using its only default constructor:

WebClient wc = new WebClient();

Next we use that object to create a stream of information to the URL of the Web application:

Stream st = wc.OpenRead(url);
StreamReader sr = new StreamReader(st);

Now we read through the response stream of the initial Web application page line by line, looking for the line that contains the __VIEWSTATE variable so we can parse out its value. Recall that the ViewState is contained in a single line that looks like:

<input type="hidden"
name="__VIEWSTATE"
value="dDw0NjgyMTgwODQ7Oz7iZ4VvNPCaZ3SL6rj+ucrH9CSs8Q==" />

We can use the String.Index Of() method to determine if we are at the target line.

 


if (line.IndexOf("__VIEWSTATE") != -1) // found line
{
  sr.Close();
  st.Close();
  int startIndex = line.IndexOf("value=") + 7;
  int endIndex = line.IndexOf("\"", startIndex);
  int count = endIndex - startIndex;
  return line.Substring(startIndex, count);
}

When we have found the line that contains the __VIEWSTATE value, we can extract it using the String.Substring() method, looking for the characters that follow "value=" and are before the next double-quote character.

Discussion
The example presented and explained in the previous sections accepts user input into a TextBox control and also displays output in a TextBox. What if your application uses other controls? Consider an ASP .NET Web application that has a checkbox, a button, and a label control:

 


<form method="post" runat="server" >
  <asp:CheckBox id="CheckBox1" runat="server"
  Text="Check Me" /> <br>
  <asp:Button id="Button1" runat="server" Text="Button"
  OnClick="Button1_Click"/> <br>
  <asp:Label id="Label1" runat="server" />
</form>

Imagine that the page has some logic that displays a message on the Label control according to whether or not the CheckBox control is checked. To simulate a user checking the CheckBox and clicking the submit Button control, construct a data string like:

data = "CheckBox1=checked&Button1=clicked&Label1=empty&__VIEWSTATE=" & viewstate

To simulate checking the CheckBox control, we put CheckBox1=checked in the data string. Somewhat surprisingly, "checked" is not relevant and as long as we specify any value there, the server will interpret the control as checked. If we want to simulate an unchecked CheckBox, we leave out any value and put CheckBox1= in the string:

data = "CheckBox1=&Button1=clicked&Label1=empty&__VIEWSTATE=" & viewstate

As explained before, we need to supply any value for the Button control, and "clicked" is nicely self-documenting. We must include the Label control too because it contributes to the __VIEWSTATE value, but in this case it doesn't matter whether we supply a value or not (so I used "empty"). Of course we need the initial __VIEWSTATE value.

Depending on which controls you have in your ASP .NET Web application, there are many variations possible, but with a little experimentation you can programmatically post data to any combination of controls.

Now that you can programmatically post data to an ASP .NET Web application, what are some of the ways you can employ this technique in a production environment? I have used this technique in virtually every Web-based product I've worked on. One valuable use is to construct Developer Regression Tests (DRTs) for your Web application. DRTs are a sequence of automated tests that are run after you make changes to your application. They are designed to determine if your new code has broken existing functionality before you check in the code.

For the example application in this article, you could create a simple text file of test cases:

001:red:roses are red
002:blue:the sky is blue
003:green:grass is green
etc.

The first line of this file has a test case ID, followed by the input, and then the expected result. Read this file line by line, parse out the three fields, programmatically post the input string, examine the response stream to see if the actual response matches the expected result or not, and then log the pass or fail result. Notice that test case 003 would catch a logic error in our application (see Listing 1).

There are many ways you can modify the code I've presented here. I have left out all error-checking for the sake of clarity and you'll want to add all the usual checks; the .NET try-catch mechanism is powerful and adding it to this code will save you a lot of time in the long run. In particular, the InitialViewState() method is quite brittle.

I have hard-coded most values. To make the code in this article more flexible you can parameterize the code in several ways. For example, a parameterization I often use takes the form:

bool ResponseStringHasTarget(string url, string proxy, string data, string target)

where the method returns true if the response from data sent to the application URL using proxy server proxy contains a string target. It could then be called like:

 


if (ResponseStringHasTarget(url, proxy, data, target) == true)
  LogToFile("Pass");
else
  LogToFile("Fail");

There are many other ways you can customize and modify the code in this article to suit your own particular needs.

Conclusion
The ability to programmatically post data to a traditional ASP Web page is extremely useful and fairly well understood. But the techniques required to post data to an ASP .NET Web application are not well documented or generally known. This article has shown you how to programmatically post data to a Web application from any .NET program. The key class that posts the request and gets the response is the HttpWebRequest class. However, the technique that we use to post to traditional ASP pages must be significantly modified by taking into account the application's initial __VIEWSTATE attribute and the state of all server-side controls.

By using the techniques in this article you can create extremely powerful automated tests for any ASP .NET Web application and write a wide range of useful utility tools.

Resources and References

  • Warren, S. "Taking a Bite Out of ASP.NET ViewState." November 27, 2001: Taking a Bite Out of ASP.NET ViewState
  • "Client-Side Functionality in a Server Control." .NET Framework Developer's Guide: Client-Side Functionality in a Server Control
  • "HttpWebRequest Class." .NET Framework Class Library: HttpWebRequest Class
  • "Control.ViewState Property." .NET Framework Class Library: Control.ViewState Property
  • More Stories By James McCaffrey

    Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft's Redmond, WA campus. He has worked on several Microsoft products, including Internet Explorer and MSN Search. James can be reached at [email protected] or [email protected]

    Comments (1) View Comments

    Share your thoughts on this story.

    Add your comment
    You must be signed in to add a comment. Sign-in | Register

    In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


    Most Recent Comments
    portsj 08/19/08 07:19:18 PM EDT

    This code does not work for me. I created a new website and a C# console application in VS.NET 2005. HttpWebResponse res = (HttpWebResponse)req.GetResponse(); throws a 500 error. Also, viewstate = HttpUtility.UrlEncode(viewstate); should be
    viewstate = HttpUtility.UrlDecode(viewstate);

    Can you please test your code and provide me with working code and/or advise how to make it work?

    Thanks in advance,
    p

    @ThingsExpo Stories
    Web Real-Time Communication APIs have quickly revolutionized what browsers are capable of. In addition to video and audio streams, we can now bi-directionally send arbitrary data over WebRTC's PeerConnection Data Channels. With the advent of Progressive Web Apps and new hardware APIs such as WebBluetooh and WebUSB, we can finally enable users to stitch together the Internet of Things directly from their browsers while communicating privately and securely in a decentralized way.
    The cloud market growth today is largely in public clouds. While there is a lot of spend in IT departments in virtualization, these aren’t yet translating into a true “cloud” experience within the enterprise. What is stopping the growth of the “private cloud” market? In his general session at 18th Cloud Expo, Nara Rajagopalan, CEO of Accelerite, explored the challenges in deploying, managing, and getting adoption for a private cloud within an enterprise. What are the key differences between wh...
    Internet of @ThingsExpo, taking place June 6-8, 2017 at the Javits Center in New York City, New York, is co-located with the 20th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. @ThingsExpo New York Call for Papers is now open.
    WebRTC has had a real tough three or four years, and so have those working with it. Only a few short years ago, the development world were excited about WebRTC and proclaiming how awesome it was. You might have played with the technology a couple of years ago, only to find the extra infrastructure requirements were painful to implement and poorly documented. This probably left a bitter taste in your mouth, especially when things went wrong.
    IoT is at the core or many Digital Transformation initiatives with the goal of re-inventing a company's business model. We all agree that collecting relevant IoT data will result in massive amounts of data needing to be stored. However, with the rapid development of IoT devices and ongoing business model transformation, we are not able to predict the volume and growth of IoT data. And with the lack of IoT history, traditional methods of IT and infrastructure planning based on the past do not app...
    "LinearHub provides smart video conferencing, which is the Roundee service, and we archive all the video conferences and we also provide the transcript," stated Sunghyuk Kim, CEO of LinearHub, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
    With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo 2016 in New York. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place June 6-8, 2017, at the Javits Center in New York City, New York, is co-located with 20th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry p...
    The 20th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held June 6-8, 2017, at the Javits Center in New York City, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Containers, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal ...
    "There's a growing demand from users for things to be faster. When you think about all the transactions or interactions users will have with your product and everything that is between those transactions and interactions - what drives us at Catchpoint Systems is the idea to measure that and to analyze it," explained Leo Vasiliou, Director of Web Performance Engineering at Catchpoint Systems, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York Ci...
    Discover top technologies and tools all under one roof at April 24–28, 2017, at the Westin San Diego in San Diego, CA. Explore the Mobile Dev + Test and IoT Dev + Test Expo and enjoy all of these unique opportunities: The latest solutions, technologies, and tools in mobile or IoT software development and testing. Meet one-on-one with representatives from some of today's most innovative organizations
    20th Cloud Expo, taking place June 6-8, 2017, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy.
    SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in Embedded and IoT solutions, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 7-9, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/Big Data, HPC and E...
    SYS-CON Events announced today that Linux Academy, the foremost online Linux and cloud training platform and community, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Linux Academy was founded on the belief that providing high-quality, in-depth training should be available at an affordable price. Industry leaders in quality training, provided services, and student certification passes, its goal is to c...
    WebRTC is the future of browser-to-browser communications, and continues to make inroads into the traditional, difficult, plug-in web communications world. The 6th WebRTC Summit continues our tradition of delivering the latest and greatest presentations within the world of WebRTC. Topics include voice calling, video chat, P2P file sharing, and use cases that have already leveraged the power and convenience of WebRTC.
    "A lot of times people will come to us and have a very diverse set of requirements or very customized need and we'll help them to implement it in a fashion that you can't just buy off of the shelf," explained Nick Rose, CTO of Enzu, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
    WebRTC sits at the intersection between VoIP and the Web. As such, it poses some interesting challenges for those developing services on top of it, but also for those who need to test and monitor these services. In his session at WebRTC Summit, Tsahi Levent-Levi, co-founder of testRTC, reviewed the various challenges posed by WebRTC when it comes to testing and monitoring and on ways to overcome them.
    Every successful software product evolves from an idea to an enterprise system. Notably, the same way is passed by the product owner's company. In his session at 20th Cloud Expo, Oleg Lola, CEO of MobiDev, will provide a generalized overview of the evolution of a software product, the product owner, the needs that arise at various stages of this process, and the value brought by a software development partner to the product owner as a response to these needs.
    WebRTC services have already permeated corporate communications in the form of videoconferencing solutions. However, WebRTC has the potential of going beyond and catalyzing a new class of services providing more than calls with capabilities such as mass-scale real-time media broadcasting, enriched and augmented video, person-to-machine and machine-to-machine communications. In his session at @ThingsExpo, Luis Lopez, CEO of Kurento, introduced the technologies required for implementing these idea...
    The WebRTC Summit New York, to be held June 6-8, 2017, at the Javits Center in New York City, NY, announces that its Call for Papers is now open. Topics include all aspects of improving IT delivery by eliminating waste through automated business models leveraging cloud technologies. WebRTC Summit is co-located with 20th International Cloud Expo and @ThingsExpo. WebRTC is the future of browser-to-browser communications, and continues to make inroads into the traditional, difficult, plug-in web co...
    The Internet of Things will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, demonstrated how to move beyond today's coding paradigm and shared the must-have mindsets for removing complexity from the develop...