In this tutorial, I will describe Elasticsearch, Logstash and Kibana 4 setup with index mapping template. Every field can be mapped to desired core type that you can specify explicitly. Typing the data properly will give a significant boost to your ELK stack performance which is important for production-grade systems.
This article will not be describing a hypothetical situation such as a spherical cow. The following examples are taken from a production website with 100,000 page views a day.
After performing the ELK setup for the client they quickly found and fixed several major issues thanks to the transparency and ease of real-time monitoring ELK adoption provides.
Pre-existing conditions: dedicated server on CentOS 6.5 with LAMP stack (Apache 2.4, PHP 5.5, MariaDB [MySQL] 10), 32Gb of RAM and 1Tb 15K RPM HDD.
Ok, it happens, let's deal with it.
Part One: Install Elasticsearch
ELK stack consists of three main components:
- Elasticsearch - distributed RESTful search
- Logstash - logs and event management system
- Kibana - visualisation and analytics platform
The latest version of Elasticsearch (1.4.4) can be installed by APT or YUM as well as from zip, rpm, deb or tar archive. But first I need some Java here!
sudo yum install -y java-1.8.0-openjdk
If you want to use Oracle Java 8 (as recommended by Elasticsearch official docs), you can get it from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
I need to set up Elasticsearch repo. Elasticsearch provides repositories for YUM and APT distributions. I used YUM since the server is on CentOS.
Install public signing key of Elasticsearch repository:
sudo rpm --import https://packages.elasticsearch.org/GPG-KEY-elasticsearch
I want to enable Elasticsearch repository. I created a new file /etc/yum.repos.d/elasticsearch.repo with the following content:
[elasticsearch-1.4] name=Elasticsearch repository for 1.4.x packages baseurl=http://packages.elasticsearch.org/elasticsearch/1.4/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1
Repository is ready to go. Now I can install Elasticsearch with:
sudo yum install -y elasticsearch
To start Elasticsearch automatically upon reboot:
sudo chkconfig --add elasticsearch
I want to have my Elasticsearch installation secured with access control. I installed Sonian plugin elasticsearch-jetty (https://github.com/sonian/elasticsearch-jetty):
cd /usr/share/elasticsearch/ sudo bin/plugin -url https://oss-es-plugins.s3.amazonaws.com/elasticsearch-jetty/elasticsearch-jetty-1.2.1.zip -install elasticsearch-jetty-1.2.1
Jetty plugin configuration files can be found at /usr/share/elasticsearch/plugins/jetty-1.2.1/config
ls /usr/share/elasticsearch/plugins/jetty-1.2.1/config jetty-hash-auth.xml jetty-restrict-all.xml jetty.xml
File jetty.xml – main Elasticsearch config file. Should be listed before all others config files.
File jetty-hash-auth.xml - basic file-based auth login service.
File jetty-restrict-all.xml - security credentials that require password for all.
I configured security realms to provide access control and authentication for Elasticsearch. Default configuration remains untouched, and I just changed directory to store my realm.properties file to /etc in jetty-hash-auth.xml file:
<?xml version="1.0"?> <!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd"> <Configure id="ESServer" class="org.eclipse.jetty.server.Server"> <!-- Create new file based auth service that uses realm.properties file in elasticsearch config directory --> <Call name="addBean"> <Arg> <New class="org.eclipse.jetty.security.HashLoginService" id="DefaultLoginService"> <Set name="name">Default Realm</Set> <Set name="config"><SystemProperty name="jetty.home" default="."/>/etc/realm.properties</Set> <Set name="refreshInterval">5</Set> </New> </Arg> </Call> </Configure>
I created a file /etc/realm.properties with the following content (I need only one administrator login):
Where 827ccb0eea8a706c4c34a16891f84e7b is an MD5 encrypted password "12345" which can be created by either of two ways:
Using the java with jetty plugin "util" jar:
cd usr/share/elasticsearch/plugins/jetty-1.2.1 sudo java -cp ./jetty-util-8.1.14.v20131031.jar org.eclipse.jetty.util.security.Password admin 12345 OBF:19bv19bx19bz19c119c3 MD5:827ccb0eea8a706c4c34a16891f84e7b CRYPT:adpliAB3dA.06
Or using md5sum Linux command that will produce the same MD5 encrypted password:
echo -n "12345" | md5sum 827ccb0eea8a706c4c34a16891f84e7b -
Detailed explanation of Jetty realm authentication can be found on: https://wiki.eclipse.org/Jetty/Tutorial/Realms http://www.eclipse.org/jetty/documentation/9.1.1.v20140108/configuring-security-authentication.html
In this part of the tutorial, I described Elasticsearch installation on CentOS 6.5 with Sonian elasticsearch-jetty plugin; I generated credentials and configured security realm.
Please continue to the next chapter and don't hesitate to leave your comments or suggestions below.
Part One: Install Elasticsearch (you are here)