When dealing with problems, it's also important to prioritize the individual issues and start with the most critical ones. IT administrators should sort performance data to identify the individual applications, servers, and even performance metrics that represent the biggest hot spots. In a utility computing or "server farm" environment, it's also important to identify which server (or virtual server) is involved in running a particular application (or even a particular URL) and whether this server is generating the most threshold violations. Certainly IT should be dealing with the single worst offender first.

Once performance issues are identified and prioritized, IT should have a way to determine whether the problem lies outside or inside the data center. By tracing ill-performing transactions from browser to database, IT can link together all of the different views of data (including end-user response time, network latency, and Web, app, and database server performance) for particular transactions. For example, if outside-the-data center metrics such as Network Latency and Parsing Time represent the bulk of the end-user response time, we know that the problem is on the outside. If inside-the-data center metrics such as Base Page Response Time, as measured at the Web server predominate, then the problem likely lies inside the data center. In some cases, there are problems both inside and outside the data center.

If the problem lies outside the data center, the next question is whether the issue stems from the client or the network. By comparing the figures for Network Latency and Parsing Time, an administrator can quickly determine the source of the issue. For example, a long Network Latency Time might be attributable to the end user accessing the application via a dial-up link.

Once the problem has been isolated to the client or network level, the identity of the individual user and/or that user's IP address can be determined. This lets IT dive deeper using desktop or network management tools to resolve the problem.

On the other hand, if the problem lies inside the data center, IT should take the infrastructure response time associated with a particular page object and break it down into front-end Web server, and back-end application server and database server response times.

If the problem originates in the front-end Web servers, IT should determine which Web pages, objects, and servers are involved. IT can assess whether the issue comes from page design or Web server resource constraints. If the problem originates in the back-end application or database servers, IT should determine which objects and servers are involved, and whether the slowdown stems from the application server or database server tiers.

Once a problem has been isolated to the back-end infrastructure, IT should inventory the various method calls and SQL queries involved in generating a particular page object, along with the response time of each.

Using the Method Stack Trace the troubleshooter can then identify the exact method calls and SQL queries that are causing the problem, along with the particular server responding to the request.

Certainly, the only way to implement the described browser-to-database approach in real-time on a 24/7 basis is to use an integrated tool that approaches the challenge in a systematic manner.

Metrics for Real-Time Problem Identification, Diagnosis, and Isolation
Real-time performance metrics from browser to database are needed to effectively perform this workflow. Lacking any key metrics would hinder the timely resolution of the identified problem. On the other hand, it is also important to avoid collecting extraneous data since it will hamper the effective diagnosis of performance problems. For example, an attempt to isolate the cause(s) of user complaint by searching through a gigabyte's worth of server and network log files would most likely be extremely time-consuming.

Table 1 shows a set of common metrics required to identify, diagnose, and isolate performance problems.

To avoid apples-to-oranges comparisons, the metrics collected should correlate to a specific user running a specific transaction at a particular time. It will be hard to diagnose performance problems experienced by User A through analyzing the server or method call response time of transactions related to User B. Furthermore, all data collection and correlation must be achieved in a non-intrusive, scalable, and repeatable manner. The practical matter is that any data collection and correlation will consume a certain amount of infrastructure bandwidth. However, it makes no sense if the management function takes up more than, say, 5% of the CPU or network bandwidth as that will slow down the application.

This set of important metrics spans the informational needs of operations, networking, server, application, and database administrators, allowing all relevant functions to effectively diagnose and isolate identified performance problems instead of wasting time reproducing and debating the issue while the application remains slow and unavailable.

Conclusion
Tools and problem resolution workflow from the client/server era cannot adequately address performance issues encountered in a more complex Web application environment. The solution is to monitor application performance from the perspective of the real user running real transactions in real-time. In this way, problematic transactions can be traced through the application infrastructure from browser to database to pinpoint root cause in the infrastructure or application.