Let’s start with a typical scenario…We have multiple teams requiring multiple services on different clouds. One team comes to you and asks about deploying a new Kafka Rest Proxy server to support an integration microservices solution. Another team asks you to set up a Jenkins and Nexus server in order to build your Build & Release pipelines. And, Developers want to analyse the application logs, so they request the creation of an ELK setup for monitoring transactions and creating alerts.
Does this ring a bell? It happens every day in most software development teams. We try to be “smart” and make some short-term decisions about how and where to deploy these services. In some cases, we just want to move away from the complexity of creating the service from scratch so we decide to go from Virtual Machines to SaaS, for example moving to Elastic Cloud instead of deploying an AWS ELK.
Without realising, you’ll have created a problem, which is that managing, monitoring, optimising and maintaining all these systems separately becomes unmanageable.
Another discussion point – cost. How can we optimise the cost of all these resources, ensure that the performance is adequate, able to scale up & down when needed and check our overall processing and data usage etc?
Most people try to optimise in the following ways:
- Automatic Scaling
- Moving from Physical Servers to VMs
- Moving from VMs to Containers
- Moving from VMs or Containers to Serverless architectures
- Shutting down on idle times
- Constant monitoring and tweaking the performance
At MagenTys we have been transforming companies moving them from VMs to containers and Serverless architectures so, why not going one step ahead?
All of these points led me to explore a different approach and to look into solutions such as DC/OS.
Let me briefly tell you about DC/OS first…
DC/OS (Data Center Operating System) is a cluster manager and system to run data services. It allows customers to run services and containers into their own clouds or infrastructure. As a data centre operating system, DC/OS is itself a distributed system, a cluster manager, a container platform, and an operating system.
It belongs to a company called Mesosphere and it’s being built on the top of Apache Mesos.
To add more details into DC/Os, we could say that it’s a modern data-driven application model for containerised microservices, data services and AI/ML tools that is supported on the top of Hybrid Clouds.
Just one slide courtesy of Mesosphere that explains this a bit better:
One of the great advantages of DC/OS is that as a cluster manager, it runs in every infrastructure. This could be Physical Servers, Virtual Servers, Private Clouds or Public Cloud Providers such as Google, AWS and Azure.
This means that you can move DC/OS from one cloud to another, which is as easy as creating new nodes (VMs) in another cloud and move just the machines that are conforming DC/OS, gradually. This means you’re not tied into a specific cloud provider anymore.
DC/OS also includes a unique section where you can administer the security, compliance of your infrastructure & services and where you can schedule when, and how your applications are deployed and for how long.
As part of DC/OS we can find:
- Application-Aware Automation
- Intelligent Resourcing
- Security
- Service Catalog
- Hybrid & Edge Cloud
- Operations & Troubleshooting
- Multitenancy
- Networking & Storage
Don’t forget that DC/OS is built on the top of Apache Mesos, a distributed systems Kernel that provides applications (e.g., Hadoop, Spark, Kafka, Elasticsearch) with APIs for resource management and scheduling across entire datacenter and cloud environments.
If you want to compare DC/OS with Linux, we could go as far as to say that DC/OS is like a distribution of Apache Mesos Cluster Management.
As a container platform, DC/OS includes two built-in task schedulers (Marathon and DC/OS Jobs (Metronome)) and two container runtimes (Docker and Mesos). All tasks on DC/OS are containerised, which allows the system to optimise and distribute the hardware resources in a more efficient way, adding also resiliency to these containers.
Adding to this abstractions beast, we have an offer of 100+ frameworks (resources) available on the catalogue, which can be deployed into DC/OS with a couple of clicks.
Why Industry Leaders choose Mesosphere?:
- Build and deploy time from weeks to minutes.
- Save 50% on AWS or Azure bills.
- Don’t want to get locked to a cloud provider.
- Fast data + Microservices: With Kubernetes on DC/OS we can use the container orchestrator alongside big data services with a common set of security, maintenance, and management tools”.
One picture I saw during my workshop with DC/OS which I really liked (see below) shows a typical technology backend that we find in most of our projects, with the difference being that all those are hosted in DC/OS instead on multiple clouds as PaaS or SaaS.
To summarise, it was really worth it to explore this solution as I found out that instead of having all these services and applications such as Jenkins, Artifactory, Kafka, ElasticSearch, Kibana, Hadoop, Bitbucket, Kubernetes etc. in different providers/clouds/offerings, I have seen that it makes more sense to have them all together under the same umbrella. With DC/OS you basically provide the hardware power you want based on the number of nodes (VMs) that you want to deploy and use those resources as a pool for all the services that you will deploy inside, making a better use of your resources.
If you want to know more just explore it here
References:
http://mesos.apache.org/documentation/latest/architecture/
A Microsoftee in IBM 