When and why to run Web Performance and Load Tests

A colleague from recently asked me about when should we use Load and Performance Tests? Well my answer would be: “All the time during the product development”, but there are specific stages where this is critical.

Some agile teams have a verification week before moving to the next iteration. During verification week, they spend first 2-3 days of the week verifying the product in their test environments. If everything is okay, by Thursday they deploy to preproduction environment. Once they have green light in pre-production, they deploy to production, which is targeted over the weekend or Monday morning.

So the tests that they run during this verification week are the same they run continuously in their working branch.

Load testing is a critical part of our software development process. We find many serious issues in load testing, everything from performance regressions, deadlocks and other timing-related bugs, to memory leaks that can’t be effectively found with functional tests. Load testing is critical to being able to confidently deploy.

Load Test Script Architecture

TFS has web service end points, rest end points and web front ends.

In order to generate the massive loads we need, we use Web tests to drive load to our web site and web pages, and unit tests to drive load to our web services.

Here a representation of the Load Tests Script Architecture.

All tests scripts use common configuration to determine which TFS instance to target in the test, and scripts can be configured to target an on-premises server or Visual Studio Online.

Types of Load Testing we do

Performance Regression Testing: executed every sprint to ensure that no performance or scale regressions are introduced.
Scale Testing:

Increasing the number of accounts and users to find new bottlenecks in scale. Some key counters to consider are:

  • RPS (requests per second)
  • % CPU AppTiers
  • Average Response Time
  • Current Server Requests
  • Active Team Project Collection Service Hosts
  • Private Bytes
Deployment Testing

We run a load tests while the upgrade is happening, so, as activity is constantly going against the service, we can detect outages through all the stages.

  • Stage 1: Deploy new binaries into web front ends and job agents.
  • Stage 2: Run jobs to upgrade the databases.
  • Stage 3: Run jobs to do necessary modifications of the data stored in the databases.
Directed Load Testing

Isolate a particular component or service for performance, stress and scale testing.

Testing in Production

These test are fundamentally focused on analysing the following data to look for regressions:

  • Activity log analysis.

    • Increases in failed command counts
    • Increases in response times, which indicates a perf regression
    • Increase in call counts, which indicates a client chattiness regression
  • CPU and memory usage.

    • Increase in CPU usage after deployment
    • Memory leaks
  • PerfView analysis: mainly driven by a tool called PerfView. Useful for finding memory and CPU regressions in production.

As you see, running performance and load tests is something that could be done anytime, but depends on the time where you do it, it will involve different techniques and data to capture.

I encourage you to create your first Web Performance and Load Tests following some of the next links and find yourself the answer of… When and Why to run Web Performance and Load Tests.

Generate and run coded web performance test

Step by Steps tutorial- Run performance tests on your app

Distributing Load Test Runs Across Multiple Test Machines Using Test Controllers and Test Agents

Analysing Load Tests Results Using the Load Test Analyser

Load Tests in the Cloud

Happy testing!

Eduardo Ortega

Posted in P&L Tests, Testing, Visual Studio | Leave a comment

Reporting on TFS

Today during the Deep dive in Visual Studio and MTM course we were covering the types of reports that can be produced in TFS. Either for tracking your tasks or your tests progress, these reports are produced from the same place and with the same mechanism. Let’s take a look to the next picture:

As we all know, the SQL Server database that is in TFS is in charge of storing and producing the right KPIs for our reports. Most of the information we query is directly extracted directly from the database where our project is hosted (our project collection DB). But most of the reports are produced directly not from the project DB but from the OLAP data cube that is managed by SQL Server Analysis Services.

So said that, we have two extra Data Bases, one is the OLAP Cube, TfsAnalysis data base, that is mainly use to generate reports in Excel or to export reports to SharePoint wherever you have an integration with it.

The other database, TfsWarehouse, is being used to generate reports through SQL Server Reporting Services, which give us the chance to generate also our own reports using the Report Designer tool. This reports are easily accessible through the Web Access of our team portal or from our Team Explorer in Visual Studio.

Either for TfsAnalysis or TfsWarehouse databases, you should have permissions to them both in order to create this reports.

For how to GRANT PERMISSIONS TO VIEW OR CREATE REPORTS IN TFS follow the next link.

SQL Server Reporting Services

When you finally get access to SSRS, you would be able to access to the folder above represented, and, depending on the project template that your team is using for the project, you will see less or more reports. But remember, the report builder is there!

Some of the reports are the next:

  • Test and bug reports

    • Test Case Readiness
    • Test Plan Progress
    • Bug Status
    • Bug Trends
    • Reactivations
  • Project Management Reports:

    • Backlog Overview
    • Release Burndown
    • Sprint Burndown
    • Velocity

All these reports and more are explained here

Probably the test case readiness is one of the most detailed and spectacular, showing us the next information:


This is an extended version of the simple chart that we usually take from the Microsoft Test Manager (Plan tab, Results menu), where we can analyse the Test Result Summary of our tests executions by tester, by suite, by configuration and answer to questions such as:

How much testing has the team completed?

How many tests are left to be run?

How many tests are passing?

How many tests are failing?

How many tests are blocked?

Why tests are failing?

Are new issues going into production?

Is there any regression on the failing tests?

Last but not least, another way to create reports in TFS is using Microsoft Excel.

These awesome pivot tables can be generated from either the Team tab in Microsoft Excel or from Visual Studio (straight from a query).

If you are creating the report from Visual Studio Team Explorer, you basically have to look for the query you want to use to generate the report, and then right click and create report in Excel.

If you decide to create the Report from the OLAP Cube, in Microsoft Excel there is a tab called “Data”. There you will find an action called “From other sources” where you can select “From Analysis services”

After you chose this option, a wizard will be introduce to you:

  1. Connect to the DataBase server
  2. Select DataBase and Table (Tfs_Analysis will be ours)
  3. Give it a name and point it to our file
  4. Report will be generated

I hope this blog post has thrown some light on what reports are available in TFS and do not forget… In terms of having these reports, you have to generate the data first! So make sure you create your work items such as tasks, user stories, test cases, and others properly or reports will be useless J

Enjoy!

Eduardo Ortega

Posted in ALM, TFS | Leave a comment

C# Programming resources

While running the Certified Scrum Developer Training at London, we went through an amazing journey across TDD and BDD techniques using C# as a main programming languages.

For some of us, it was a good practice that helped us to refresh our rusty programming skills, for others was a way to move a Tester role to a Software Developer Engineer in Testing role.

Let me share with all of you some useful resources about C# programming that will make your journey on programming even more interesting.

C# Programming Guide: https://msdn.microsoft.com/en-us/library/67ef8sbd.aspx

[Video Training]C# Fundamentals for Absolute Beginners:http://www.microsoftvirtualacademy.com/training-courses/c-fundamentals-for-absolute-beginners

[Video Training]Programming in C# Jump Start:http://www.microsoftvirtualacademy.com/training-courses/developer-training-with-programming-in-c

[Video Training]Twenty C# Questions Answered:http://www.microsoftvirtualacademy.com/training-courses/twenty-c-questions-explained

[Video Training]What’s New in C# 6: http://www.microsoftvirtualacademy.com/training-courses/developer-productivity-what-s-new-in-c-6

[Videos]How Do I? Videos for Visual C#: https://msdn.microsoft.com/en-us/vstudio/bb798022

[Samples]Visual Studio Samples: https://code.msdn.microsoft.com/vstudio

[Free Book]C# Language Specification 5.0 https://www.microsoft.com/en-us/download/details.aspx?id=7029

[Books]Visual Studio Books: https://msdn.microsoft.com/en-us/vstudio/dd285474

More Visual C# Resources:
Get started with Visual C#
Asynchronous programming with Visual C#
Getting started with .NET and Visual Studio
Programming concepts

Last but not least, you can download the different Visual Studio versions from herehttps://www.visualstudio.com/en-us/downloads
I recommend to download the Community Edition as it’s totally free and fulfil most of the needs you have about Coding.

I hope this info is useful and enjoy coding!

  • May the code be with you –

Eduardo Ortega

Posted in c#, Visual Studio | Leave a comment

Exporting Fiddler2 sessions to Visual Studio Web Tests

While finishing the revisited and new “Web Performance Testing with Microsoft Visual Studio 2013” training, I found a very interesting topic to cover, idea of my colleague Carl Bricknell, that is to migrate Fiddler2 sessions to Visual Studio generating new Web Tests.

As you know Visual Studio 2013 Ultimate and Visual Studio 2015 Enterprise have included the templates for Load and Performance Tests.
It’s as easy as go into VS and create a new “Web Performance and Load Test project”.

Usually, once the project is done, we have to record our session with the Web Test Recorder as follows:

And right after our session, we will be able to edit the recorded test steps on the WebTest view. This will allow us to create our validation rules that would be used to let the test pass or fail; also create, modify and delete web requests, their headers, parameters (dynamic and statics) and, the most fancy feature, to add data parameters in order to replay the tests several times with different data straight from an Excel Sheet, CSV file or SQL Database.

Many testers and developers use Fiddler2 in a daily basis, and this is because is one of the most powerful web traffic capture tools.

Fiddler2 includes the ability to capture web traffic (including AJAX requests) for later playback with the Visual Studio Web Performance Test feature. It can also be used to help debug your Web performance tests. Comparing your Fiddler2 recording and your Web performance test recording can help identify key missing headers that your Web performance test may not record, amongst other things.

We can summarize as:

Fiddler is at HTTP debugging proxy server application that captures HTTP and HTTPS traffic and logs it for the user to review.

Fiddler can also be used to modify HTTP traffic for troubleshooting purposes.

In order to export Fiddler2 recording sessions to Visual Studio Web Test Project we need to follow the next steps:

  1. Create a Visual Studio empty Web Test Project
  2. Record your session using Fiddler2
  3. Export the Fiddler 2 session to the Visual Studio Web Test Project

The exporting is very straight forward:

  1. Go to File à Export Sessions à All sessions

  1. Select Visual Studio Web Test as Export Format

  1. Select all the plugins installed by default:

  1. Find the generated web test in your hard drive and attach it to your Visual Studio Web Test Project:

Done!

From here, everything that happens is result of your creativity, it’s ready for testing!

  • Happy testing! –

Eduardo Ortega Bermejo

Posted in P&L Tests, Testing, Visual Studio | 1 Comment

FREE Upgrade to Visual Studio Enterprise with MSDN!

There is no better time to get ready for the release of Visual Studio 2015. The new Visual Studio is coming in May 2015 and the prices have been reduced dramatically!
When you upgrade from Visual Studio Professional or Visual Studio Test Professional to Visual Studio Premium, the price is cut to 50%! And also get a free upgrade to Visual Studio Enterprise with MSDN automatically when Visual Studio 2015 gets released.

The offer is only valid for 2 months, expiring on June 30th so hurry up.
But there is much more. Visual Studio 2015 license system has changed. These are the versions that we will have now:

  • Visual Studio Community

    • Integrated Development Environment for building Web, Windows Desktop and cross-platform iOS, Android, and Windows apps
    • Ecosystem with thousands of extensions to choose form the Visual Studio
    • Free for open source projects, academic research, training, education and small professional teams
  • Visual Studio Professional with MSDN

    • Professional developer tool for building any application type
    • Powerful features to improve your team’s productivity such as CodeLens
    • Improve team collaboration with Agile project planning tools, Team Rooms, charts and more
    • MSDN subscription benefits including access to core software for dev/test, Team Foundation Server, Visual Studio Online Basic, $50/month in Azure credits, training and support.
  • Visual Studio Test Professional with MSDN
  • Visual Studio Enterprise with MSDN

    • End-to-end solution for your development teams, including the most feature-rich Visual Studio IDE for working on any type of project
    • Build quality applications at scale with advanced features such as Load Testing, automated and manual testing and new IntelliTest capabilities
    • Manage complexity and resolve issues quickly with features such as Code Map and IntelliTrace
    • Enhanced MSDN subscription benefits including comprehensive access to software for dev/test, Team Foundation Server, Visual Studio Online Advanced, $150/month in Azure credits, training and support

Customer price has changed as well:

If you have You will get New customer price
Visual Studio Community Visual Studio Community 2015 Free
Visual Studio Professional with MSDN Visual Studio Professional 2015
Your Visual Studio Professional with MSDN
subscription will continue without change
$1,199
Visual Studio Premium with MSDN Visual Studio Enterprise 2015
Your subscription will be upgraded to
Visual Studio Enterprise with MSDN
$5,999

 

Here you can find a comparison between the different versions of Visual Studio, including the Visual Studio Test Professional (that is the most relevant product for many of us).]

References:

Posted in ALM, Visual Studio | Leave a comment

Supported configurations and platforms for VS Coded UI Tests

Coded UI automated tests! Sounds good, uh? Well, yes, it’s very exciting to see how a machine is simulating that is playing with your app while input some predefine values in your textboxes and performing some actions. Said that, it’s good to know where you can create and apply these Coded UI Tests.

First, supported Operating Systems:

  • Windows 7
  • Windows Server 2008 RD
  • Windows 8

Supported Architectures:

  • .NET 2.0, 3.0, 3.5, 4 and 4.5

Second, Platform Support:

  • Windows Phone Apps: Supported
  • Windows Store Apps: Fully supported including HTML5 in IE9 and IE10
  • Internet Explorer 8 and 9.
  • Internet Explorer 10 and 11: is only supported on the desktop, not Modern UI
  • Windows Forms and WPF: Fully supported
  • Internet Explorer 6 and 7: Not supported
  • Chrome: Recording of action steps is not supported. Coded UI Test can be played back on Chrome and Firefox.
  • Opera: Not supported.
  • Safari: Not supported
  • Silverlight Not supported
  • Flash/Java: Not supported
  • Windows Win32 and MFC: Partially supported
  • SharePoint: Fully supported
  • Office Client Applications: Not supported
  • Dynamics CRM web client: Fully supported
  • Dynamics AX 2012: Action recording and playback are partially supported
  • SAP: Not supported
  • Citrix/Terminal Services: Partially supported
  • PowerBuilder: Partially supported

Of course, when the talk about ASP.NET, HTML5 pages, these are included in the fully supported list of apps.

If you want to know how to Create, Edit and Maintain a Coded UI Test, check this out: https://msdn.microsoft.com/en-us/vstudio/ff977233

Happy support!

Eduardo Ortega​

Posted in Coded UI, Testing, Visual Studio | Leave a comment

How to update to TFS 2013.4

​Step 1: Download the Visual Studio Team Foundation Server 2013 Update 4 from your MSDN subscription or Microsoft downloads

Step 2: Start the installer:

<Image not available>

Step 3: Let the installation be. It will copy the needed components for the update

<Image not available>

Step 4: Follow the Team Foundation Server Upgrade starting with clicking “Next”
<Image not available>

Step 5: Before doing any upgrade (as happens for example in the TFS2010 to TFS2013 migration), you should do a DataBase backup of the main databases. So click where you read “Click here to launch the Database Backup Tool”.
<Image not available>

Step 6: Once finished. Go back to the previous screen and check “By checking this box, I confirm that I have a current backup”.

Step 7: Provide the settings for the application Tier. This means, which system account you want to use for the Application Tier services. By default it will use the Network Service and NTLM authentication.

<Image not available>

Step 8: Here is your moment to set up the Reporting Services. Even if you don’t have reporting services set up previously, you can do it now. If they are already in place just populate properly the fields. Example below:
<Image not available>

In the next step, the database will show up automatically in the databases list. Just make sure you Test the connection to it.

Step 9: Proceed on the same way with the Analysis Services. In case the Analysis Services is stopped, make sure it’s running.

Step 10: Report Reader Account. In this step we should provide a user account as System or Network Services accounts cannot be used here.
<Image not available>

So that’s it! We have skipped the SharePoint configuration as for this instance we don’t have such services set up. Just go to review and click on Configure.

<Image not available>

<Image not available>

Now you are ready to enjoy the new features on TFS 2013 Update 4. I recommend you to see what’s new on TFS2013.4.

But just a sneak peak, check the new access levels from the configuration console in the Team Web Access:

<Image not available>

Happy upgrade!

Eduardo Ortega

Posted in TFS | Leave a comment