|By Steve Weisfeldt||
|January 7, 2013 08:00 AM EST||
Mobile applications and mobile websites have become a major channel for conducting business, improving employee efficiency, communicating, and reaching consumers. In Part I of this article we discussed the differences between testing traditional and mobile applications, specific challenges associated with mobile load testing, mobile testing basics and best practices for recording mobile load test scenarios. In this Part 2 of the article, we will look at how to conduct realistic tests and how to best analyze the results.
How to Run Realistic Load Tests
Once you've recorded a mobile test scenario, you need to be parameterize it so that it can emulate users with different identities and behaviors as it is played back to produce a realistic load on the server. This step is required for traditional and mobile web applications, and the tools used to complete it are the same. When playing back the test scenarios, however, there are several challenges specific to mobile load testing.
Simulating Network Conditions
Today's mobile devices generally access the server over networks that are slower than those used by desktop computers. Network conditions have a significant effect on the user experience, and the effect may be more or less pronounced depending on the application. Network characteristics including bandwidth, latency and packet loss have a huge impact on client response times and on the way the server is loaded. By simulating different network conditions in a test lab environment you can forecast the effects of changes in the network infrastructure on the application's performance. Doing so also allows you to discover application issues in the development cycle, therefore reducing costs.
Bandwidth, Latency and Packet Loss
For example, low bandwidth increases the time it takes to download a resource, which then results in higher page load times. If the customer is connected longer, front-end servers hold sockets longer, load balancers have more active TCP sessions and application servers use more threads.
Mobile networks have limited bandwidth and high latency compared to Wi-Fi and broadband. Since the latency is a time added to each request and webpages are composed of many sub-requests, the time required to load a webpage on a mobile device greatly depends on the latency.
Limiting bandwidth and simulating latency and packet loss during a load test allows you to check that all of your users, including mobile users, will get the best user experience and acceptable response times while ensuring your servers won't have problems under load.
Bandwidth and Response Times
The bandwidth is directly correlated with how long it takes to download data from the server. The lower the bandwidth, the higher the response time. A server that provides acceptable response times for desktop users using DSL or another high-speed broadband service may deliver a poor end-user experience to mobile users with lower bandwidth.
It is important to validate your service-level agreements (SLAs) and performance objectives with tests that use the same bandwidth limitations as your users to avoid making decisions based on misleading test results. Such tests must incorporate bandwidth simulation, which is the process of artificially slowing down the traffic during a test to simulate a slower connection.
Bandwidth and Server Load
Clients using lower bandwidth connections also affect the server. The lower the bandwidth, the longer the connections. Longer connections, in turn, lead to more simultaneous connections on your web server and your application server. Thus, mobile users tend to consume more connections than their wired counterparts. Most servers have settings that limit the number of simultaneous connections that they can handle. Without a testing tool that realistically simulates bandwidth, these settings cannot be properly validated.
Simulating Bandwidth Limitations for Individual Virtual Users
When load testing, effective bandwidth simulation requires the ability to individually limit the bandwidth for each user or groups of users, independent of the others.
Consider a situation in which you need to verify performance when 100 mobile users are accessing the server. In this scenario, you'd want to simulate 100 virtual users, with each user limited to a 1Mbps 3G connection. In this case, the total bandwidth for all users is 100Mbps (100 users * 1Mbps/user). Though it is possible to use WAN emulation software or a network appliance to globally limit the bandwidth for the load generation machine to 100 Mbps (or any other arbitrary limit), in practice this does not provide a realistic test because it does not impose a strict 1Mbps constraint on each user. Bandwidth simulation support must be integrated in the load testing tool to enable bandwidth limits to be applied to individual virtual users.
To conduct an even more realistic test, you'll want to simulate a mixed population of users accessing your application with a variety of bandwidths. With a tool capable of bandwidth simulation on a per virtual user basis, you can determine the response times for users at each bandwidth across a range of bandwidths in a single test. This saves times when you need to compare the response times of web applications and business transactions for clients who have different bandwidth limits.
Simulating Browsers and Browser Capabilities
When a browser requests a resource from a web server, it identifies itself via the user-agent header sent with each request. This header contains information about the browser and the platform on which it is running. Servers use this information to deliver different versions of the content based on the client system. As noted earlier, many web applications deliver different content to mobile users and desktop users. Some further differentiate mobile users into subgroups based on information in the user-agent header, delivering less text and smaller images to devices with small screens. This can lead to bandwidth consumption and loading times that vary widely with the browser and platform being used.
As a result, the ability to manipulate the user-agent header is essential not only for recording test scenarios, but also for playing them back. Tools that lack this capability will fail to retrieve the appropriate content from the server.
Simulating Parallel Connections
Mobile browsers, like desktop browsers, can generate the HTTP requests needed to retrieve the static resources of a web page in parallel. Rather than waiting for each image to finish loading before requesting the next, this approach requests multiple images at once to shorten the overall page load time. To measure response times accurately, load testing tools must replicate this behavior by generating multiple requests in parallel. Moreover, they must simulate the appropriate number of parallel connections as this number may differ from one mobile browser to another. Again, tools that lack this capability are not performing realistic tests, placing the results they deliver into question.
Identifying the Most Appropriate Settings for Realistic Tests
Finding the appropriate values for key test settings - such as the user-agent, bandwidth, and number of simultaneous connections - can be a challenge. More advanced load testing tools can help testers set these values. For example, test scenario playback is greatly simplified by tools that can automatically inform the tester of which user-agent string and number of parallel connections to use based on the browser name, version, and platform. The process is further streamlined when the tools can suggest the most appropriate upload and download bandwidth settings based on the technology used (for example, Wi-Fi, 3G, 3G+, and so on) and the quality of the signal (for example, poor, average, or good).
Using the Cloud
You can use load testing with the cloud after (or in conjunction with) on-premise testing in the lab to improve the realism of your tests by generating high loads and testing from different locations, while saving time and lowering costs.
Generating a High Load
For consumer-facing apps and websites, it's often difficult to predict the number of users your applications will have to handle. Traffic spikes that results from a promotion, marketing campaign, new product release, or even unexpected social network buzz can be substantial. To generate a similar load in-house, you would need a significant investment in hardware. Using the cloud, you can generate the same high load using on-demand resources at a much lower cost.
Testing from Different Geographies
Your web application's real users likely access the server from many different geographical locations and use different networks. To properly validate the application and the server infrastructure, your virtual users should operate under similar real world conditions.
Testing the Entire Application Delivery Chain
When your real users are located outside the firewall, you should run your virtual users from the cloud to validate the parts of the application delivery chain that are not tested when testing from the lab, including the firewall, load balancers, and other network equipment.
Tools for Testing with the Cloud
While the cloud represents an opportunity to rapidly increase the scale and improve the realism of load testing at low costs, cloud testing is most effective when it's used to complement internal load testing. Note that the primary factor in the success of load testing with the cloud is not the move to the cloud, rather it's the tool you select and how well it uses cloud technology. In particular, it's best to select a solution that is integrated with multiple cloud platforms, enables in-house test assets to be reused in the cloud, and supports realistic, large-scale tests across multiple geographical regions.
The default results of a load test are frequently delivered as averages. For example, load testing tools will typically show what errors occurred and the average response times for a request, web page, or business transaction regardless of the type of users being simulated or the bandwidth available to them.
Because bandwidth may vary widely for the different kinds of users simulated, the errors and response times can also vary widely. Taking an average of results with significant variation does not provide an accurate picture of what is really happening. To gain meaningful insights and to validate your SLAs and performance requirements for each network condition, it is important to go beyond the default results and analyze the results for each kind of user.
In many ways, mobile load testing is similar to load testing classic web applications. As a result, testers can leverage much of their existing knowledge and reuse existing techniques - like using the cloud for realistic, large-scale tests. However, there are specific requirements for testing mobile applications that are not addressed by traditional load testing techniques. Recording mobile test scenarios, conducting realistic tests that simulate real-world bandwidth and browser characteristics, and properly analyzing the results are some of the key areas that require special attention for mobile applications. Addressing challenges in these areas is essential to ensuring mobile web applications are sufficiently tested prior to release and that they will perform well under load in production.
24Notion is full-service global creative digital marketing, technology and lifestyle agency that combines strategic ideas with customized tactical execution. With a broad understand of the art of traditional marketing, new media, communications and social influence, 24Notion uniquely understands how to connect your brand strategy with the right consumer. 24Notion ranked #12 on Corporate Social Responsibility - Book of List.
Sep. 30, 2016 10:45 PM EDT Reads: 481
SYS-CON Events announced today that Roundee / LinearHub will exhibit at the WebRTC Summit at @ThingsExpo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. LinearHub provides Roundee Service, a smart platform for enterprise video conferencing with enhanced features such as automatic recording and transcription service. Slack users can integrate Roundee to their team via Slack’s App Directory, and '/roundee' command lets your video conference ...
Sep. 30, 2016 10:45 PM EDT Reads: 1,533
Web Real-Time Communication APIs have quickly revolutionized what browsers are capable of. In addition to video and audio streams, we can now bi-directionally send arbitrary data over WebRTC's PeerConnection Data Channels. With the advent of Progressive Web Apps and new hardware APIs such as WebBluetooh and WebUSB, we can finally enable users to stitch together the Internet of Things directly from their browsers while communicating privately and securely in a decentralized way.
Sep. 30, 2016 10:00 PM EDT Reads: 1,255
"My role is working with customers, helping them go through this digital transformation. I spend a lot of time talking to banks, big industries, manufacturers working through how they are integrating and transforming their IT platforms and moving them forward," explained William Morrish, General Manager Product Sales at Interoute, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
Sep. 30, 2016 09:30 PM EDT Reads: 4,117
Just over a week ago I received a long and loud sustained applause for a presentation I delivered at this year’s Cloud Expo in Santa Clara. I was extremely pleased with the turnout and had some very good conversations with many of the attendees. Over the next few days I had many more meaningful conversations and was not only happy with the results but also learned a few new things. Here is everything I learned in those three days distilled into three short points.
Sep. 30, 2016 09:30 PM EDT Reads: 5,427
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, wh...
Sep. 30, 2016 09:00 PM EDT Reads: 4,047
Adobe is changing the world though digital experiences. Adobe helps customers develop and deliver high-impact experiences that differentiate brands, build loyalty, and drive revenue across every screen, including smartphones, computers, tablets and TVs. Adobe content solutions are used daily by millions of companies worldwide-from publishers and broadcasters, to enterprises, marketing agencies and household-name brands. Building on its established design leadership, Adobe enables customers not o...
Sep. 30, 2016 08:45 PM EDT Reads: 486
Why do your mobile transformations need to happen today? Mobile is the strategy that enterprise transformation centers on to drive customer engagement. In his general session at @ThingsExpo, Roger Woods, Director, Mobile Product & Strategy – Adobe Marketing Cloud, covered key IoT and mobile trends that are forcing mobile transformation, key components of a solid mobile strategy and explored how brands are effectively driving mobile change throughout the enterprise.
Sep. 30, 2016 08:30 PM EDT Reads: 472
In this strange new world where more and more power is drawn from business technology, companies are effectively straddling two paths on the road to innovation and transformation into digital enterprises. The first path is the heritage trail – with “legacy” technology forming the background. Here, extant technologies are transformed by core IT teams to provide more API-driven approaches. Legacy systems can restrict companies that are transitioning into digital enterprises. To truly become a lea...
Sep. 30, 2016 08:15 PM EDT Reads: 747
What are the new priorities for the connected business? First: businesses need to think differently about the types of connections they will need to make – these span well beyond the traditional app to app into more modern forms of integration including SaaS integrations, mobile integrations, APIs, device integration and Big Data integration. It’s important these are unified together vs. doing them all piecemeal. Second, these types of connections need to be simple to design, adapt and configure...
Sep. 30, 2016 07:30 PM EDT Reads: 566
What happens when the different parts of a vehicle become smarter than the vehicle itself? As we move toward the era of smart everything, hundreds of entities in a vehicle that communicate with each other, the vehicle and external systems create a need for identity orchestration so that all entities work as a conglomerate. Much like an orchestra without a conductor, without the ability to secure, control, and connect the link between a vehicle’s head unit, devices, and systems and to manage the ...
Sep. 30, 2016 07:15 PM EDT Reads: 482
The Jevons Paradox suggests that when technological advances increase efficiency of a resource, it results in an overall increase in consumption. Writing on the increased use of coal as a result of technological improvements, 19th-century economist William Stanley Jevons found that these improvements led to the development of new ways to utilize coal. In his session at 19th Cloud Expo, Mark Thiele, Chief Strategy Officer for Apcera, will compare the Jevons Paradox to modern-day enterprise IT, e...
Sep. 30, 2016 07:00 PM EDT Reads: 2,457
Major trends and emerging technologies – from virtual reality and IoT, to Big Data and algorithms – are helping organizations innovate in the digital era. However, to create real business value, IT must think beyond the ‘what’ of digital transformation to the ‘how’ to harness emerging trends, innovation and disruption. Architecture is the key that underpins and ties all these efforts together. In the digital age, it’s important to invest in architecture, extend the enterprise footprint to the cl...
Sep. 30, 2016 06:45 PM EDT Reads: 749
SYS-CON Events announced today that Commvault, a global leader in enterprise data protection and information management, has been named “Bronze Sponsor” of SYS-CON's 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Commvault is a leading provider of data protection and information management solutions, helping companies worldwide activate their data to drive more value and business insight and to transform moder...
Sep. 30, 2016 06:30 PM EDT Reads: 2,914
What does it look like when you have access to cloud infrastructure and platform under the same roof? Let’s talk about the different layers of Technology as a Service: who cares, what runs where, and how does it all fit together. In his session at 18th Cloud Expo, Phil Jackson, Lead Technology Evangelist at SoftLayer, an IBM company, spoke about the picture being painted by IBM Cloud and how the tools being crafted can help fill the gaps in your IT infrastructure.
Sep. 30, 2016 06:15 PM EDT Reads: 3,165
SYS-CON Events has announced today that Roger Strukhoff has been named conference chair of Cloud Expo and @ThingsExpo 2016 Silicon Valley. The 19th Cloud Expo and 6th @ThingsExpo will take place on November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. "The Internet of Things brings trillions of dollars of opportunity to developers and enterprise IT, no matter how you measure it," stated Roger Strukhoff. "More importantly, it leverages the power of devices and the Interne...
Sep. 30, 2016 06:15 PM EDT Reads: 3,547
Digital innovation is the next big wave of business transformation based on digital technologies of which IoT and Big Data are key components, For example: Business boundary innovation is a challenge to excavate third-party business value using IoT and BigData, like Nest Business structure innovation may propose re-building business structure from scratch, as Uber does in the taxicab industry The social model innovation is also a big challenge to the new social architecture with the design fr...
Sep. 30, 2016 05:45 PM EDT Reads: 1,380
Data is an unusual currency; it is not restricted by the same transactional limitations as money or people. In fact, the more that you leverage your data across multiple business use cases, the more valuable it becomes to the organization. And the same can be said about the organization’s analytics. In his session at 19th Cloud Expo, Bill Schmarzo, CTO for the Big Data Practice at EMC, will introduce a methodology for capturing, enriching and sharing data (and analytics) across the organizati...
Sep. 30, 2016 04:30 PM EDT Reads: 1,836
DevOps at Cloud Expo, taking place Nov 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 19th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long dev...
Sep. 30, 2016 04:00 PM EDT Reads: 3,598
IoT offers a value of almost $4 trillion to the manufacturing industry through platforms that can improve margins, optimize operations & drive high performance work teams. By using IoT technologies as a foundation, manufacturing customers are integrating worker safety with manufacturing systems, driving deep collaboration and utilizing analytics to exponentially increased per-unit margins. However, as Benoit Lheureux, the VP for Research at Gartner points out, “IoT project implementers often ...
Sep. 30, 2016 03:45 PM EDT Reads: 3,674