Setting up Security and Password Authentication for Small Elasticsearch Clusters

Now that you have a cluster, you are going to want to add some security to it. This is part 5 of a series on creating a personal Elasticstack. You can check the others out here:

1. Building Your Environment in AWS
2. Setting up and Installing Elasticsearch
3. Setting up Kibana
4. Using a Proxy for Kibana with HAProxy
5. Enabling Security and Using Password Authentication
6. Making Kibana Internet Accessible with Cloudfront
7. Securing Cloudfront with Security Groups
8. Inserting Data into Elasticsearch with Logstash

This guide only follows the basic way to use password authentication. This is only secure if your servers are not accessible from the outside world and only communicate between each other. This guide assumes you have implemented something similar to my previous guides in the series and have a firewall and proxy server in front.

You will not want to use this security if you are connecting directly to the Kibana or they are not secure from the outside world.

Step 1: Generate a Certificate File

Logon to your server, and then sudo to the root account. You only need to do step 1 on a single elasticsearch node.

Go to the Elasticsearch directory in the /usr/share directory.

cd /usr/share/elasticsearch

Use the certificate generation tool to create a Certificate Authority file:

bin/elasticsearch-certutil ca

Leave the output file as default, and set a password for the CA. You will need the password later so store it securely somewhere.

This will generate a elastic-stack-ca.p12 file

Now use this file to create a certificate:

bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

You will need that password you just created for the CA. You will also need to make a new password for this key. Store that somewhere securely as well.

You should now have the elastic-certificates.p12 file as well.

Step 2: Send Certificate to Cluster Nodes

Now that you have a certificate, you need to put it on each node in the cluster. WinSCP is a good tool if you are on Windows. Alternatively you could use the SCP command to send it directly to your other nodes. However you do this is up to you. You may have to change the permissions on the file in order to do so.

Move the file into the /etc/elasticsearch/ directory on each node.

Step 3: Add the Certificate to Elasticsearch

First go to the same directory as before /usr/share/elasticsearch as root.

cd /usr/share/elasticsearch

Using the two commands below, add the password for the elastic-certificates.p12 certificate to the keystore and truststore passwords. The p12 file acts as both the keystore and truststore, so we only need one file for both components.

bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password

bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

Next move your elastic-certificates.p12 certificate in the /etc/elasticsearch/ directory if you have not already done so. Then navigate to that directory as a standard user.

sudo mv elastic-certificates.p12 /etc/elasticsearch/

sudo cd /etc/elasticsearch/

You may have to change a few permissions on the certificate file first.

sudo chmod 660 /etc/elasticsearch/elastic-certificates.p12

sudo chown root:elasticsearch /etc/elasticsearch/elastic-certificates.p12

Now that the certificate file is ready to go, we can edit our Elasticsearch config.

sudo vim /etc/elasticsearch/elasticsearch.yml

At the bottom here we have to add some configuration to enable the certificate and node encrypted communication.

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12 
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

Repeat step 3 for each node in your cluster.

Step 4: Restart Elasticsearch Cluster

Once you have edited the config on each node, you need to do a full cluster restart.

Stop all nodes, then start them back up again.

sudo systemctl stop elasticsearch

sudo systemctl start elasticsearch

Remember that you can check if there are any issues with startup using:

sudo journalctl -xef --unit elasticsearch

Once started up, if you now run a curl command on the node, you will get a slightly different response: (Remember to change the IP for your own node)

curl -X GET "172.30.90.13:9200/_cluster/health?timeout=50s&pretty"

It will instead return a security authentication error, HTTP 401 Unauthorized. This is what we want.

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "missing authentication credentials for REST request [/_cluster/health?timeout=50s&pretty]",
        "header" : {
          "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "missing authentication credentials for REST request [/_cluster/health?timeout=50s&pretty]",
    "header" : {
      "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
    }
  },
  "status" : 401
}

This means that the nodes now need authentication to to be connected to.

Step 5: Setup System Passwords

If the cluster is running, that’s great, but we can’t use Elasticsearch anymore. We now need to setup the passwords to continue.

On any one of the nodes, navigate to the /usr/share/elasticsearch/ directory as root.

cd /usr/share/elasticsearch

Run the password setup tool. SAVE the passwords.

(BE CAREFUL, you can only run this command once. If you lose the passwords you set, you may not have any way of recovering them.)

bin/elasticsearch-setup-passwords interactive

You will be asked to set the password for 7 accounts. You should receive something like below once you are done:

Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

Step 6: Setup the Kibana Password

Now that you have your passwords setup and saved, you need to use the kibana_system user for kibana.

On the Kibana node(s) open the config file for each one:

sudo vim /etc/kibana/kibana.yml

Add the following two lines to the bottom of the file: (be sure to use the password you set)

elasticsearch.username: "kibana_system"
elasticsearch.password: "kibanapassword"

NOTE: This is going to be stored in plain text in the config file. This is why it is not recommended you use this method if there is any outside connectivity for the Kibana nodes.

Step 7: Restart Kibana

One you have configured Kibana, all that is left is to restart it.

sudo systemctl stop kibana
sudo systemctl start kibana

sudo journalctl -xef --unit kibana

If everything starts successfully you can now navigate to your Kibana dashboard (via your proxy) and it will prompt you for a password.

You can use the elastic account to login. From here you can setup your own account to login with in the future.

All this now has the issue that we are sending the passwords unencrypted over the network. We currently are using HTTP not HTTPS, so our passwords are transferred in plain text.

Check out Part 6: Creating an Internet Accessible Kibana With Cloudfront where we use Cloudfront to make our front end use HTTPS and do the offloading to HTTP for us.

Any thoughts, concerns, mistakes? Let me know in the comments or via the Contact page.