Old times when Microsoft and Linux lived in parallel universes are over. Nowadays with the introduction of Cloud Services and Containers, the matter of the chosen OS is meaningless. While practising DevOps, you will find out that the technologies and tools are not the important ones, we should focus on delivering fast and with quality in a full fledged agile environment.

Today’s recipe is about how to add quality gates to our CI builds, using tools such as SonarQube, leveraging on the Microsoft Azure Container Instances.

Recipe: CI Builds with SQ and Azure Containers

Ingredients:
- 1 SonarQube Server
- 1 Azure Container Registry
- 1 Azure Container Instance
- 1 VSTS CI Build Pipeline
- 1 Ubuntu host OS
- 1 Azure CLI for Linux

Cooking time: 6 minutes

1) Creating the SonarQube server

Quickest way to create a SonarQube server instance is deploying  it from a Docker  pre-baked image, like the ones you can find in DockerHub or like in our recipe, Azure Container  Registry.

The tool chosen to deploy download the image from the ACR and create the container as an Azure Container Instance is the Azure CLI, which works under several OS platforms. For this example we are going to install the Azure CLI for Ubuntu.

Installing Azure CLI for Ubuntu:

Step 1: Copy and paste this into your Linux shell

AZ_REPO=$(lsb_release -cs)
echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" | \
sudo tee /etc/apt/sources.list.d/azure-cli.list

sudo apt-key adv --keyserver packages.microsoft.com --recv-keys 52E16F86FEE04B979B07E28DB02C46DF417A0893
sudo apt-get install apt-transport-https
sudo apt-get update && sudo apt-get install azure-cli

Now that we have the Azure CLI installed, let’s proceed with the container into Azure Container Instances, for that we need first to create the Azure resource group in the location we are going to host the Docker Container Instance.

Step 2: Create Azure Resource Group

az group create --name myResourceGroup1 --location westeurope

Output should be something like:

{
 "id": "/subscriptions/xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/resourceGroups/myResourceGroup1",
 "location": "westeurope",
 "managedBy": null,
 "name": "myResourceGroup1",
 "properties": {
 "provisioningState": "Succeeded"
 },
 "tags": null
 }

Then, create the container, providing the name of the image in the container registry, the port to open and the public DNS name for the container.

Step 3: Create the SonarQube container in the Azure Container Instances

az container create --resource-group myResourceGroup1 --name sqcontainer --image sonarqube --dns-name-label sqmagentysdemo --ports 9000

Output should be something like:

{
 "additionalProperties": {},
 "containers": [
 {
 "additionalProperties": {},
 "command": null,
 "environmentVariables": [],
 "image": "sonarqube",
 "instanceView": null,
 "name": "sqcontainer",
 "ports": [
 {
 "additionalProperties": {},
 "port": 9000,
 "protocol": null
 }
 ],
 "resources": {
 "additionalProperties": {},
 "limits": null,
 "requests": {
 "additionalProperties": {},
 "cpu": 1.0,
 "memoryInGb": 1.5
 }
 },
 "volumeMounts": null
 }
 ],
 "id": "/subscriptions/xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/resourceGroups/myResourceGroup1/providers/Microsoft.ContainerInstance/containerGroups/sonarqubecontainer",
 "imageRegistryCredentials": null,
 "instanceView": {
 "additionalProperties": {},
 "events": [],
 "state": "Pending"
 },
 "ipAddress": {
 "additionalProperties": {},
 "dnsNameLabel": "sqmagentysdemo",
 "fqdn": "sqmagentysdemo.westeurope.azurecontainer.io",
 "ip": "52.178.112.203",
 "ports": [
 {
 "additionalProperties": {},
 "port": 9000,
 "protocol": "TCP"
 }
 ]
 },
 "location": "westeurope",
 "name": "sqcontainer",
 "osType": "Linux",
 "provisioningState": "Creating",
 "resourceGroup": "myResourceGroup1",
 "restartPolicy": "Always",
 "tags": null,
 "type": "Microsoft.ContainerInstance/containerGroups",
 "volumes": null
 }

Step 4: Check if the container is up.

az container show --resource-group myResourceGroup1 --name sqcontainer

We also can check it in the Azure portal:

2) Integrate the SQ Container Instance as part of your  CI pipeline

Now that we have SonarQube up and running on Azure, it’s time to use it in our Builds. For this example I’m using one of my Build Definitions in VSTS (similarly works for Jenkins, TeamCity, Bamboo and others) to include 2 steps one for start the SonarQube analysis and another one to generate the summary report.

You can find the SonarQube plugin here: https://marketplace.visualstudio.com/items?itemName=SonarSource.sonarqube#overview

All you need to provide to these plugins is the URL and port of the SonarQube server and the Token generated when you finish the first time configuration.

You can see how the integration works in VSTS with our container:

And how the results are displayed in SonarQube

If you want to check the logs of the SonarQube server you can access to them from the Azure CLI:

az container logs --resource-group myResourceGroup1 --name sqcontainer

Deleting your SQ container

Eventually, if you don’t want to leave your container living forever, you can delete your container as easy as:

az container delete --resource-group myResourceGroup1 --name sonarqubecontainer

Other possible recipes:

• Create your own Docker Container Images with SonarQube and deploy it into ACI using the Azure CLI.
• Create the Docker Container Images from the bash or PowerShell as part of your Build steps, deploy it into ACI, use it in your pipeline and then destroy it after.

Summary

I found this solution particularly simple, as avoids you to setup your own SQ server from scratch. SQ is not going to run 24/7, you can take the container up and down any time, you don’t need a server for it, even more, the virtual machine where it is running the SQ server, can be used for multiple purposes, containers will help us to draw this isolation boundaries between this and other containers, having a better utilisation of the VM resources.

Integrating with your CI pipelines can be done easily through the plugins available in the market, either for VSTS/TFS, Bamboo, Jenkins, etc. In case you want to do your own plugin, is as easy as using the REST APIs that SQ is offering to use its quality gates even from your scripts.

Azure CLI allows you to do quick and easy operations into Azure besides you use Windows, MacOs or Linux. No need to go fancy with PowerShell or other scripting.

I hope you find your perfect recipe!

Happy baking!

References:

• Install Azure CLI for ubuntu: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-apt?view=azure-cli-latest
• Install Visual Studio Code: https://code.visualstudio.com/
• Creating Azure Container Instances Quickstart: https://docs.microsoft.com/en-us/azure/container-instances/container-instances-quickstart

How do you analyse the behaviour of your application or services during development or when moving the code to production?

This is one of the most challenging things to control when we deploy software into an environment. Yes, the deployment is successful, but is the application really working as expected?

There are a number of ways to check if it is working as expected. One way is to analyse the behaviour of your application by extracting the component and transaction logs generated internally and somehow analyse them through queries and dashboards, This should help us to understand what’s going on.

SPLUNK

I’m a big fan of Splunk, you just need to create your log files and send them to Splunk, and in a matter of minutes, you can create shiny cute dashboards to query and monitor your log events.

My only issue with Splunk is the cost. Using it for a few solutions is okay, but when you’re having to process a large amount of data it then becomes very expensive. As the offering we might need is based on a set of data per day. Even so, I can say it’s extremely easy to parse your data, create data models, create panes and dashboards and also provide alerts.

Some teams may rather opt for other (cheaper) solutions. Remember, open source doesn’t always mean free. The time your dev team is going to spend implementing the solution is not free!

ELK

A cheaper (sometimes) alternative is to use Elastic Search and Kibana (for extracting logs and analysis in that order).

Kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster, which is also open source.

Both can be hosted on your own servers, deployed on AWS or Azure or another cloud provider and you even have the option to use a hosted Elastic Cloud if you don’t care too much about the infrastructure.

How does this work?

1st) log your operations with a proper log management process (unique log code, log message, severity, etc).

2nd) Ingest the log files into an elastic search index and extract from your events the fields that you want to use for your charts representation and searches.

3rd) Create searches and dashboards according to the needs of the team. E.g. All logs, error logs and transactions for Dev and Test, error logs per component per system, no. Of HTTP requests, and HTTP error codes for Business Analysis, Operations and Support, etc.

4th) Give to the team the access and tools they really need. Yes, you can provide access to Kibana to the whole team and everybody’s happy, but why not use the full potential of Elastic Search? If I would be doing the testing, I would use the Elastic Search REST API to query the events logged by the application from my tests.

At MagenTys we have done ELK implementations from Zero to Hero for a wide range of projects, and not only for software development. It can also be used to ingest and represent the data of sources, such as Jenkins, Jira, Confluence, SonarQube and more!

Don’t like ELK? There are other options for application logging, that can be also extended to your infrastructure, like Azure Monitor.

Azure Monitor

Microsoft has recently changed the names to some of their products and has also grouped them all together. For example, Log Analytics and Application Insights have been consolidated into Azure Monitor to provide a single integrated experience.

Azure Monitor can collect data from a variety of sources. You can think of monitoring data for your applications in tiers ranging from your application, any operating and services it relies on, down to the platform itself

OMS (Operations Management Suite) as such is being retired, moving all its services into Azure Monitor. For those that are currently using it, you should know that by Jan 2019, the transition will be complete and you might have to move into Azure Monitor

Saying that, the new Azure Monitor experience looks like this:

Azure Monitor collects data from each of the following tiers:

• Application monitoring data
• Guest OS monitoring data
• Azure resource monitoring data
• Azure subscription monitoring data
• Azure tenant monitoring data

To compare it with Splunk and ELK, we can leave the operations and resources monitoring aside for a moment and focus on Log Analytics and Application Insights.

Log data collected by Azure Monitor is stored in Log Analytics which collects telemetry and other data from a variety of sources and provides a query language for advanced analytics.

Common sources of data usually are .NET and .NET Core applications, Node.js applications, Java applications and Mobile Apps. But we can import and analyse custom logs too.

There are different ways to use Log Analytics, but mostly is being done through Log Queries:

Remember that with Log Analytics and Log Queries, we are extracting the events created in our log files, organising and parsing them, filtering and then creating our Dashboards, Reports and Alert from them, similar to the Splunk model. With the advantage that we can cross-reference this logs with the information extracted from Application Insights:

Application Insights (which used to be separated from OMS and Log Analytics), is better used for analysing the traffic and actions around your applications. For example, on a web page, it’s straightforward with Application Insights to see the number of web requests, the pageViews, the HTTP Error codes or even analyse the stack trace of the errors captured and link this to our source code.

On the visualisation side,

• Azure dashboards are able to put together all these metrics:

It still has some limitations in terms of customisation of visualisations, but it’s extensible as we can link it to wonderful tools such as PowerBi or Grafana.

Azure Monitor views allow you to create custom visualizations with log data stored in Log Analytics.

Application Insight workbooks, provides deep insights into your data, to help your development team to focus on the most important.

Last but not least, you can use Log Analytics in conjunction with PowerBi or Grafana, which are “nice to have”. The problem of Grafana is that you can monitor and build metrics but not analyse logs:

The bright side is that Grafana is Open Source, free and can be used with many many data sources, Elastic Search included.

The last thing to mention, Azure Monitor it’s not free but it’s quite affordable!

In Summary

We have briefly discussed Splunk, ELK and Azure Monitor. What type of data we can extract and analyse, different visualisations, and cost.

Most development teams use ELK as they are used to it or either come from a Java background.

I’m seeing more and more teams using Splunk, which I really recommend but it is still an expensive tool to have.

Azure Monitor, traditionally has been used extensively in Operations (a legacy from System Center family, moved to the cloud, now integrated with other analytic tools) and Performance Testing. Now they bring together the other missing piece, Log Analytics and Application Insights, for application logs analysis, and offers a very good combo of metrics and logs tools for a very good price.

Not to go into deep details about any of those, just mentioning the most common scenarios I’m finding out there.

I hope this information is useful for you!