Setting up an AWS Environment for a Personal Elasticstack

This is Part 1 of a series of guides about setting up a personal Elasticstack.

  1. Building Your Environment in AWS
  2. Setting up and Installing Elasticsearch
  3. Setting up Kibana
  4. Using a Proxy for Kibana with HAProxy
  5. Enabling Security and Using Password Authentication
  6. Making Kibana Internet Accessible with Cloudfront
  7. Securing Cloudfront with Security Groups
  8. Inserting Data into Elasticsearch with Logstash

In order to actually build a personal Elasticstack, you are going to want to setup a bit of security and configuration first. I will be using AWS for this as my cloud provider of choice. This could easily be applied to your own servers, or to Azure.

Here is a brief rundown of what instances I will be using throughout this guide:

Kibanat3a.nanoRoot 8GBCentos 7
Proxyt3a.nanoRoot 8GBCentos 7
Elastic x2t3a.microRoot 8GB, Extra 10GBCentos 7
Logstasht3a.microRoot 8GBCentos 7

I use the region us-east-1 (N. Virginia) in order to keep the costs lower, as where my cluster is doesn’t really matter.

Step 1: Security Groups

First we need to configure the Security Groups to allow the servers to talk to each other. However, we don’t want anything from outside our environment to talk to them. We can use Security Groups to ensure that only wanted communication occurs between instances. (Similar principles apply if you are using Azure instead)

Start by navigating to the Security Groups tab under Network & Security.

Hit the Create security group button on the right.

Create security group

In here, give it a name and description first (I have called mine ElasticComms). Next leave the Inbound rules blank for now and keep the Outbound Rules as the default All traffic.

Add any tags if you wish and hit Create security group. It should look something like this:

Initial empty config

Now click Edit inbound rules and add the following inbound rules:

TypePort RangeSourceDescription
Custom TCP9200ElasticCommsElasticsearch
Custom TCP9300ElasticCommsElastic Communication
Custom TCP9600ElasticCommsLogstash
Custom TCP5601ElasticCommsKibana
SSH22My IPRemote Access

In the source, you are going to select the Security Group you are currently editing, in my case ElasticComms. This makes it so only instances with this Security Group can talk to each other over these ports. The SSH is so that you, and only you, can connect to the servers. You can remove this at a later date if you want added security.

It should look something like this:

Inbound rules configuration

Now that we have Security Groups, its time to launch our servers.

Step 2: Launch Instances

You are going to probably have to repeat this for each instance you are deploying.

In the Instances tab, hit the Launch instance button.

I am using Centos 7 AMI for this guide, but feel free to use RHEL if you want to pay extra, or Amazon Linux AMI instead.

Centos 7

I am using t3a. instances as they are one of the cheapest instances to run.

GP t3a.micro

I have left everything default, the only change I made was to turn off Unlimited Credits, to save a bit of costs.

Turn off Credit specification Unlimited

For storage, I leave it default, except the Elasticsearch nodes, where I add an extra drive to it. Feel free to make it whatever size you want, but know you can increase the size at any time, but you cannot reduce it. Keep it small if you want to keep it cheap.

Storage Settings

In the security groups section, you are going to want to add the security group you made earlier. Hit the Select and existing security group option and ,in my case, select ElasticComms

Add your security group

Once you are happy with everything, launch the instance.

Step 3: Verify Access

The last step is a simple one, try to remote onto your server. If you need help or are unable to do so, checkout the official documentation on how to connect.

Now that are are done setting up and you can connect, lets move onto Part 2: Setting up and Installing Elasticsearch.

Any thoughts, concerns, mistakes? Let me know in the comments or via the Contact page.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s