A colleague from recently asked me about when should we use Load and Performance Tests? Well my answer would be: “All the time during the product development”, but there are specific stages where this is critical.
Some agile teams have a verification week before moving to the next iteration. During verification week, they spend first 2-3 days of the week verifying the product in their test environments. If everything is okay, by Thursday they deploy to preproduction environment. Once they have green light in pre-production, they deploy to production, which is targeted over the weekend or Monday morning.
So the tests that they run during this verification week are the same they run continuously in their working branch.
Load testing is a critical part of our software development process. We find many serious issues in load testing, everything from performance regressions, deadlocks and other timing-related bugs, to memory leaks that can’t be effectively found with functional tests. Load testing is critical to being able to confidently deploy.
Load Test Script Architecture
TFS has web service end points, rest end points and web front ends.
In order to generate the massive loads we need, we use Web tests to drive load to our web site and web pages, and unit tests to drive load to our web services.
Here a representation of the Load Tests Script Architecture.
All tests scripts use common configuration to determine which TFS instance to target in the test, and scripts can be configured to target an on-premises server or Visual Studio Online.
Types of Load Testing we do
Performance Regression Testing: executed every sprint to ensure that no performance or scale regressions are introduced.
Increasing the number of accounts and users to find new bottlenecks in scale. Some key counters to consider are:
- RPS (requests per second)
- % CPU AppTiers
- Average Response Time
- Current Server Requests
- Active Team Project Collection Service Hosts
- Private Bytes
We run a load tests while the upgrade is happening, so, as activity is constantly going against the service, we can detect outages through all the stages.
- Stage 1: Deploy new binaries into web front ends and job agents.
- Stage 2: Run jobs to upgrade the databases.
- Stage 3: Run jobs to do necessary modifications of the data stored in the databases.
Directed Load Testing
Isolate a particular component or service for performance, stress and scale testing.
Testing in Production
These test are fundamentally focused on analysing the following data to look for regressions:
Activity log analysis.
- Increases in failed command counts
- Increases in response times, which indicates a perf regression
- Increase in call counts, which indicates a client chattiness regression
CPU and memory usage.
- Increase in CPU usage after deployment
- Memory leaks
- PerfView analysis: mainly driven by a tool called PerfView. Useful for finding memory and CPU regressions in production.
As you see, running performance and load tests is something that could be done anytime, but depends on the time where you do it, it will involve different techniques and data to capture.
I encourage you to create your first Web Performance and Load Tests following some of the next links and find yourself the answer of… When and Why to run Web Performance and Load Tests.
Generate and run coded web performance test
Step by Steps tutorial- Run performance tests on your app
Distributing Load Test Runs Across Multiple Test Machines Using Test Controllers and Test Agents
Analysing Load Tests Results Using the Load Test Analyser