30 Days of DevOps: Elastic cloud vs AWS PaaS ELK

Some time ago I read an interesting article titled: “Is It the Same as Amazon’s Elasticsearch Service?

It was quite a good article, to be honest, it compared perfectly 2 great elastic implementations: Elastic cloud from Elastic.co and Elasticsearch services from AWS.
Nevertheless, I thought the article was not fully objective, as it was mostly saying that AWS implementation was an Elastic Search fork of the Elastic Search mainstream and that was lacking all the capabilities that now Elastic cloud offers in the X-Pack package, This is amazing but costs a pretty penny!

In the end, both should be the same right? As Elasticsearch is a search engine based on Lucene and is developed in Java and is released as open source under the terms of the Apache License.

So both are offering the same product but with small differences, one is offering a ton of plugins provided by X-Pack, the other is relying on the current AWS services to match his rival.

At MagenTys I have worked with both and also with the On-Premises version of ELK, but I want to give you my opinion. Let’s take a closer look at both and also analyse a vital part of it, which is the cost.

Elastic Cloud

It’s the company behind the Elastic stack, that means Elasticsearch, Kibana, Beats, and Logstash.

They officially support the Elasticsearch open source project, and at the same time offers a nice top layer of services around it, this is formerly known as X-Pack.

X-Pack is made of enterprise-grade security and developer-friendly APIs to machine learning, and graph analytics.This includes security, alerts, reporting, graph, machinelearning, Elasticsearch SQL and others.

It has a very nice cost calculator: https://cloud.elastic.co/pricing

Which we will be using for this article in order to compare it with AWS offering. For such purpose we will be comparing a t2.medium AWS instance.

Elasticsearch service AWS 
Instance type Two instances:
– aws.data.highcpu.m5
– aws.kibana.r4
Instance count 2
 Dedicated master  No
 Zone awareness  No
 ES data memory  4 GB
 ES data storage  120 GB
Kibana memory  1 GB
Estimated price: $78.55
As we can see, Kibana and Elastic are deployed in separate instances and the total storage is 120 GB, which is quite good in comparison with what comes by default with AWS (35GB).
Thanks to X-Pack we will enjoy of a few new features from either Kibana, ES or Logstash. Main plugins are:
– Graph
– Machine Learning
– Monitoring
– Reporting
– Security
– Watcher
More information here
Blog-Kibana1

Elasticsearch service AWS

Another alternative is Amazon Elasticsearch service, which is a fully managed service by AWS. This means it’s fully deployed, secured and ready to scale Elasticsearch.

It also allows us to ingest, search, analyse and visualise data in real-time. It offers Kibana access as well, and LogStash integration, but it lacks of the X-Pack, this means that some of the previous features we’ve seen such us users and group management and alerts are missing. This could be tackled with a different approach, letting AWS to manage the access to ES and Kibana using the “access policy” where we can whitelist ip addresses and apply access templates to IAM users. Also offers integration with Amazon Cognito for SSO and Amazon CloudWatch for monitoring and alerts.

Another advantage is that can be integrated in your VPCs.

Let’s take a look to the pricing:

Elasticsearch service AWS 
Instance type t2.medium.elasticsearch (2vCPU, 4GB)
Instance count 1
Dedicated master No
Zone awareness No
Storage type EBS
EBS volume type General Purpose (SSD)
EBS volume size 35 GB
Estimated price: $59.37

$0 per GB-month of general purpose provisioned storage – EUW2 under monthly free tier 10 GB-Mo – $0.00

$0.077 per t2.medium.elasticsearch instance hour (or partial hour) – EUW2 -720 Hrs – $55.44

$0.157 per GB-month of general purpose provisioned storage – EUW2 – 25.000 GB-Mo – $3.93

You need to pay standard AWS data transfer charges for the data transferred in and out of Amazon Elasticsearch Service. You will not be charged for the data transfer between nodes within your Amazon Elasticsearch Service domain.
Amazon Elasticsearch Service allows you to add data durability through automated and manual snapshots of your cluster. The service provides storage space for automated snapshots free of charge for each Amazon Elasticsearch domain and retains these snapshots for a period of 14 days. Manual snapshots are stored in Amazon S3 and incur standard Amazon S3 usage charges. Data transfer for using the snapshots is free of charge.
Data transfer costs in AWS are quite small but also we have to take them into consideration.
Data Transfer OUT From Amazon EC2 To Internet
Up to 1 GB / Month $0.00 per GB
Next 9.999 TB / Month $0.09 per GB
Next 40 TB / Month $0.085 per GB
Next 100 TB / Month $0.07 per GB
Greater than 150 TB / Month $0.05 per GB
And last but not least, as X-Pack is not available, the plugins we discussed about before are not present.
Blog-Kibana3

Summarising

If you compare the costs, there is really not much difference between one and the other, but some extra work to setup properly the AWS implementation needs to be taken in consideration. In Elasticcloud some stuff comes out of the box, and despite requires some tricky configuration (such alerts), in AWS we have to build this from scratch using CloudWatch, events and alerts, so we will spend the money on a consultant that can take of it.

Snapshots is another big point of discussion, as in Elasticcloud snapshots are taking daily 48 times per day every 30 minutes and get stored for 48 hours, while in AWS snapshots are being taken once a day and retained for 14 days with no cost too.

I hope this article helps you to decide which one is your best fit, and do not forget that you can also go for another path, which is create your own ELK stack on premise or in your Cloud, from scratch, deploying it straight into your EC2 instances or Containers hosts and manage fully the infrastructure, services and applications.
Happy searching!
Advertisements

About Eduardo Ortega Bermejo

Technical Evangelist and good guy
This entry was posted in DevOps, ELK, Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s