On and off you come across the battle of performance for your application and most of us don’t take load testing seriously until it has raised some red flags. Yeah, that’s when we all start to talk about performance and testing applications, monitoring all the shit which is happening, looking at the speed at which data is loaded and RPS of API is critically looked to make judgement. We all know about Load testing but yet very few of us really invest in it and mostly just live on the unit tests.
I am also sure that most of us would have come across a group which believes in testing things only once and then forgetting about it has really damaged our reputation and invited you to a meeting with the CTO or CEO of the company. I mean as a dev we change so much on a monthly basis in any application and its UX for users that at times we don’t seem to remember all of it and the business reasons behind it.
This blog post is not out of frustration but more about understanding the importance of Load Testing and why its done, the basics, what you need measure and do’s / don’ts, etc.
What is Load Testing:
Load testing is the process of putting simulated demand on software, an application, or website in a way that demonstrates its behavior under various conditions.
It used to be a simple pre deployment exercise to ensure a web application could handle the load of multiple users and is slowly becoming an integral part of software development and continuous performance improvement. As websites and web applications become more innovative and complex, load testing poses a significant challenge for teams to
- Execute properly and
- Cover all the bases.
As a result, load testing needs to be done more frequently, more effectively, and more efficiently. It also created a need to simply train more people in the basics of load testing. Given that our lives increasingly rest on apps functioning properly — whether its in medical devices, transportation, communications, defense, or entertainment, app performance has never been more important.
We’ll cover:
- Load testing basics
- How to prepare for load testing
- Emulating real life conditions in your load tests
- Load testing dos and don’ts
Basics of Load Testing
Load testing is a type of performance testing. Performance testing is a series of testing methods employed to understand how a system performs in terms of responsiveness and stability under a specific set of strains. Oftentimes, teams begin performance testing in conjunction with or after functional testing within the test phase of the software development lifecycle. Each type of performance testing method helps answer a specific question about your application’s behavior that helps carve a path for issue identification & iterative performance improvement.
For instance, load testing helps you answer, how will my application behave in production with the typical traffic that we see? Load testing is used to verify your applications behavior under normal and peak load conditions. This allows you to verify that your application can meet the desired performance objectives; which are often specified in a service level agreement (SLA). Load testing also enables you to measure response times, throughput rates, resource utilization levels, and to identify your applications breaking point, assuming that breaking point occurs below the peak load condition. Load testing helps you check your web servers performance under a massive load, determine its robustness, and estimate its scalability.
Stress Testing enables you to evaluate your applications behavior when it is pushed beyond the normal or peak load conditions, and helps you understand what is my applications breaking point? The purpose of web server stress testing is to find the target applications crash point. The crash point is not always an error message or access violation. It can be a perceptible slowdown in the request processing. The goal of stress testing is to unearth application bugs that surface only under high load conditions. Stress testing helps you find your applications weak points, and how it behaves under extreme load conditions.
Capacity Testing is complementary to load testing and determines your servers ultimate failure point, whereas load testing monitors results at various levels of load and traffic patterns. For example, lets say you’re developing a new online voting platform, and you’d like it to be able to handle potentially up to 10,000 user submissions per minute during peak load times. While developing the software, you may have performed unit tests as the code was being written, plus periodic regression tests to make sure you didn’t break existing functionality with each new modification as development progressed, but at what point did you begin testing for multiple users?
At what point did you begin testing the program to accept hundreds or even thousands of overlapping field entries, form submissions, and other commands? This is when load testing comes into the picture.
Goals of Load Testing
- Identify bottlenecks and their causes
- Optimize the application configuration (both the hardware and software) for maximum performance.
- Verify the reliability of your application under stress
Load/stress testing helps in identifying the following characteristics:
- Response time
- Throughput
- Maximum concurrent users supported
- Resource utilization in terms of the amount of CPU, RAM, network I/O, and disk I/O resources your application consumes during the test
- Behavior under various workload patterns including normal load conditions, excessive load conditions, and conditions in between
- Application breaking point
- Symptoms and causes of application failure under stress conditions
- Weak points in your application
- What is required to support a projected increase in load
We’ll approach load testing with a four-step process:
- Prepare: to prepare your web application for load testing
- Record: How to record load testing scenarios
- Test: Creating tests that match real-life circumstances
- Analyze: Understanding and using load testing data
Preparing your Load Test
With DevOps, development teams are pressured to test more and test often. Testing should be performed at each step in the development cycle and should continue after the application is live. While it can be frustrating that a testers job is never done, its important to take into consideration that with each testing and remediation cycle, the application improves. Applications go down under load for two reasons:
— either developers didn’t load test or,
— worse, they took the time to load test but didn’t prepare properly. Without adequate prep work, a load test can’t find all the issues that it should.
So, how do you best prepare for a load test?
- What do you really need to know?
Determine what you want to learn about your application or system. Each type of test is run differently, and looks at your application in a different light. So, you’ll need to run different types of tests based on what you hope to find out.
For example:
– If you hope to discover how your application performs with little or no load in order to get a baseline, you will run a single user test.
– If you hope to determine how your system will perform under normal expected load, you will run a load test.
– If you hope to determine the breaking point, the point where your application either stops responding or responds so slowly that it is unusable, you need to run a stress test.
– If you want to know if your application has memory leaks, you will want to run endurance or soak test.
- Decide on a number of users
If you are going to load test, how many virtual users do you want to simulate? In order to answer this, you will want to approximate how many concurrent users may visit your site, depending on the time of day. Don’t guess. Instead, leverage some of the data you already have. Talk to your marketing team and take real traffic patterns from tools like Google Analytics, or engage your Operations team and use data from their Real User Monitoring (RUM) tools, which can provide insight into realistic scenarios to test. If you want to know some concrete statistics from a historical perspective, go directly to your analytics reports. You may even want to ask your engineers how many concurrent users they designed the application for, and your product owner for projected numbers, based on promotional activities. Plan to test that number and some percentage above it.
There are numerous ways you can find the number of virtual users needed to run a load test. We recommend the following formula to find the number of virtual users needed to run a load test.
Concurrent users = (peak hourly visits * visit duration in seconds) 3600
You can find peak hourly visits and visit duration from an analytics tool, such as Google Analytics. Note that this formula should be used to estimate the number of concurrent users required to achieve a specific page-view rate at fully ramped load. It should not be used to estimate the number of pages that will load within a specific time interval.
- Study your analytics
Don’t pretend to know how your customers use your application. The only way to truly understand your users is to study history (i..e. analytics). By studying your analytics, you will be able to create tests that are representative of your actual users, as opposed to tests that you think are representative of your users. In this regard, analytics are a tester’s best friend. - Performance is a team goal
Gather your team You need to involve a number of people in the testing effort, including: developer, network engineer, DBA, and business owner. All of these individuals have a vested interest in making the application successful, and each will approach the problem from a different angle.
The correct solution will not fall directly into one of these buckets, but will be a combination of two or more. Make sure each is aligned with the performance expectations and is available during testing to:
- Monitor their area of expertise
- Provide balanced feedback
- Gain a sense of ownership for the health and performance of the application
- Prepare your browsers
Use testing software that brings you as close to your actual users experience as possible. You should be able to record your scenario in the browsers of your choice, but you also need to anticipate the browsers your users will most likely use. Consider the countries and regions where you anticipate high usage, and research the most used browsers.
You’ll need to have these installed on your machine to begin testing. Then you need to make sure your load testing software emulates as closely as possible actual user behavior.
This includes:
- Parallel thread processing
- Think time
- Multiple concurrent scenarios
- Complex scenarios
- Parameterization
- Generating load from multiple agents (network/cloud)
- Be prepared to test your production application
While it is valuable to test your application when it is in a staging environment, this can leave some holes in your testing. There are a few reasons for this, including:
- Staging environments are not often exact duplicates of production.
- Staging environments are often accessible from only inside the firewall.
- There is something to be said for testing the same system that you are gathering information about.
- Set aside time to analyze results
You should be prepared to spend some time analyzing test results as a group (remember all of those people that were present during testing?). Results need to be looked at to ensure bottlenecks/ errors/weaknesses are understood and remediation is effective. Reach out to everyone involved and schedule adequate time. - Different remediation will have differing costs in terms of time. Remediation such as implementing a caching strategy, refactoring code, database optimization and hardware upgrades have a wide range of costs to implement in terms of both time and money. As an example, adding more hardware will require time to order the hardware, receive the shipment, test the new hardware, install software and data, test, install into the network, and test some more. This can be weeks or months of work. This is less of a problem if you are in the cloud. In which case, it takes less than a day. Many companies are opting to move to cloud infrastructures, offering an unlimited number of environments without the need for additional hardware costs and time constraints.
However, its always advisable to load test applications that are accessible within your firewall as well.
- Plan an Agile testing methodology
Once you remediate, it is time to test again. The saying, “testing is a process, not a destination†is very true. Each time a bottleneck is uncovered and corrected, another one rises to take its place. It is important to plan an Agile testing methodology, whereby performance testing is baked into each step of the development cycle. Additional testing should be performed:
- When code is modified or updated
- When environment/infrastructure changes are introduced
- When changes are made to the application server or DB server
- When traffic spikes are anticipated
- How to record scenarios
The first thing to do is to determine the roles you will define for use in your test. A role is equivalent to a certain type of user that will visit the tested web- site, and the steps they will take while visiting.
If the tested site is a retail site, for example, you might have the following roles:
- Browse and leave
- Browse, add something to the cart, and check out
- If your tested site is a restaurant site, your roles might look something like this:
— Browse menu and find directions
— Look at hours of operation and make a reservation
Its best to choose at least three of the most common pathways through your site, and add a few uncommon routes as well. Next, you need to break these roles down by percentage of traffic. A typical retail site may have 95% of users browsing and leaving, and 5% (or less!) actually making a purchase. The combination of these two roles, or scenarios, will represent actual site traffic.
After each scenario is recorded, you need to verify it individually. This involves running a single virtual user for a single pass through the scenario. This step should never be forgotten.
Now you’re ready to start testing. By recording scenarios that imitate actual user traffic, you’re setting the stage to greatly improve customer experience and, if you’re into e-commerce, get that percentage of purchases above 5%.
| Percentage | Role |
| 95% | Browse and Leave |
| 5% | Browse, add something to the cart and checkout |
Making it Real
Development teams load test should be able to emulate real-life conditions. Otherwise, whats the point of testing?
More often than not, developers and QA managers come away from a trial of load testing software with little more than the number of users that will crash their system. Unless they have a professional load tester on staff, most development teams don’t have the resources, or knowledge to garner all they could from their load tests.
That data is wasted because the person running the tests is unable to apply it to an applications performance. Luckily, improvements in graphics and UIs have made interpreting data much easier — if you know what to look for.
Here’s a list of the most important results in load testing and how you should be working with them
Page Load Time
You need to know the average page load time for each page in your scenario. You might have a strict Service Level Agreement (SLA) that mandates how quickly pages must load, or may just want to know what this number is. It is also important to know if one page takes longer than others to load this indicates a bottleneck in your application.
Response Load Time
Just knowing page load time is not enough. If a page is slow, you need to know why. Being able to look at average response times for each response gives you a detailed look into where the time is being spent.
Errors and Warnings
You need to know which errors and warnings were generated and at what level of load. This is especially important information to see in chart format. It is important to see which errors and warnings are generated and be able to see how that changes as load increases. A common error at high levels of load is Server Error 500
Navigation Timings
Understanding what your end user experience and why is it key in identifying performance issues. Some of these metrics include redirect time, connect time, first byte time, response time, DOM load time, and event time. Its important to understand what your performance benchmarks (or competitive benchmarks are) for these metrics to make sure that the experience you’re delivering through the browser meets your standards.
Request and Response Throughput
Its important to see the amount of data going to and coming from the tested system. This is especially important in a case where load is increasing, but bandwidth reaches and maintains a plateau. In this case, it becomes apparent that bandwidth is being throttled at some point in the process, possibly at the firewall.
Hosts
Because so many of today’s websites call out to a plethora of additional hosts for things like content delivery networks, ad servers, analytics servers, social media and syndicated content, it is important for these sites to be enumerated in your reports. It’s equally important to be able to view all of the calls to a particular host. If a host is called from your pages, the response time for those requests will add to the time it takes your pages to render.
The Do’s and Don’ts of Load Testing
As you get ready to implement load testing to your performance strategy, there are a few dos and don’ts you’ll want to keep in mind.
Dos:
Record Tests with an end-user in mind. Map out user journeys that are realistic, comprehensive, and critical to business functions. Understanding that your application can handle what a user would realistically do is key in ensuring it will stand up in production.
Generate Load from different servers than those that host your application. Serve them from different environments so you can get the best view of performance without accidentally altering results.
Start Recording a new scenario from the web browsers start. If you start recording a scenario after you connect to the tested web server and open a few web pages, the playback of the scenario will fail. This will happen because the recorded traffic will not reproduce the authentication procedure, and the tested web server will ignore the simulated requests.
Parameterize Scenarios to simulate more realistic load on the server. Parameterizing scenarios involves replacing recorded parameters in the requests with variable values. This way you can make virtual users send user-specific data to the server. These data-driven tests will give you a more realistic view of how your application will perform when a unique group of users interact with it.
Verify User Scenarios. Before creating tests on the basis of a recorded scenario, make sure that the scenario is executed successfully for one virtual user. This can help you identify bottlenecks of the scenario and eliminate problems which are not related to the number of virtual users and additional testing conditions.
Arrange User Scenarios in your tests so that critical functionality is tested first.
Don’ts
Do not run tests in real environments. A real environment can have other network traffic, and this may affect test results. To avoid excess data transfer, use a test environment that behaves in the same way as the real environment except that there is no other traffic usage.
Do not try to crash the tested server. The goal of web server performance testing is not to break the server, but to identify web application performance bottlenecks under various loads.
Do not overly stress the client test systems.
Do not use zero think time. Make sure that think time in your test is based on real-life conditions. Using zero think time does not provide realistic user simulation and puts an abnormal load on the tested server. However, omitting think time can help you determine bottleneck issues.
I hope the above helps in planning, implementing and running your Load Testing Strategy and also maintain it for future releases.