PipeBug: Monitoring Using Graphite, Logstash, Sensu, and Tessera

The dark side of the ELK stack. Unleash the Logstash mapping.

In memory of the beloved Kibana 3. We will never forget.

In this tutorial, I will describe Elasticsearch, Logstash and Kibana 4 setup with index mapping template. Every field can be mapped to desired core type that you can specify explicitly. Typing the data properly will give a significant boost to your ELK stack performance which is important for production-grade systems.

This article will not be describing a hypothetical situation such as a spherical cow. The following examples are taken from a production website with 100,000 page views a day.

After performing the ELK setup for the client they quickly found and fixed several major issues thanks to the transparency and ease of real-time monitoring ELK adoption provides.

Pre-existing conditions: dedicated server on CentOS 6.5 with LAMP stack (Apache 2.4, PHP 5.5, MariaDB [MySQL] 10), 32Gb of RAM and 1Tb 15K RPM HDD.

Ok, it happens, let's deal with it.

Part One: Install Elasticsearch

ELK stack consists of three main components:

The latest version of Elasticsearch (1.4.4) can be installed by APT or YUM as well as from zip, rpm, deb or tar archive. But first I need some Java here!

sudo yum install -y java-1.8.0-openjdk

If you want to use Oracle Java 8 (as recommended by Elasticsearch official docs), you can get it from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

I need to set up Elasticsearch repo. Elasticsearch provides repositories for YUM and APT distributions. I used YUM since the server is on CentOS.

Install public signing key of Elasticsearch repository:

sudo rpm --import https://packages.elasticsearch.org/GPG-KEY-elasticsearch

I want to enable Elasticsearch repository. I created a new file /etc/yum.repos.d/elasticsearch.repo with the following content:

[elasticsearch-1.4]
name=Elasticsearch repository for 1.4.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

Repository is ready to go. Now I can install Elasticsearch with:

sudo yum install -y elasticsearch

To start Elasticsearch automatically upon reboot:

sudo chkconfig --add elasticsearch

I want to have my Elasticsearch installation secured with access control. I installed Sonian plugin elasticsearch-jetty (https://github.com/sonian/elasticsearch-jetty):

cd /usr/share/elasticsearch/
sudo bin/plugin -url https://oss-es-plugins.s3.amazonaws.com/elasticsearch-jetty/elasticsearch-jetty-1.2.1.zip -install elasticsearch-jetty-1.2.1

Jetty plugin configuration files can be found at /usr/share/elasticsearch/plugins/jetty-1.2.1/config

ls /usr/share/elasticsearch/plugins/jetty-1.2.1/config
jetty-hash-auth.xml  jetty-restrict-all.xml  jetty.xml

File jetty.xml – main Elasticsearch config file. Should be listed before all others config files.

File jetty-hash-auth.xml - basic file-based auth login service.

File jetty-restrict-all.xml - security credentials that require password for all.

I configured security realms to provide access control and authentication for Elasticsearch. Default configuration remains untouched, and I just changed directory to store my realm.properties file to /etc in jetty-hash-auth.xml file:

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd">

<Configure id="ESServer" class="org.eclipse.jetty.server.Server">

	<!-- Create new file based auth service that uses realm.properties file in elasticsearch config directory --> 
	<Call name="addBean">
		<Arg>
			<New class="org.eclipse.jetty.security.HashLoginService" id="DefaultLoginService">
				<Set name="name">Default Realm</Set>
				<Set name="config"><SystemProperty name="jetty.home" default="."/>/etc/realm.properties</Set>
				<Set name="refreshInterval">5</Set>
			</New>
		</Arg>
	</Call>

</Configure>

I created a file /etc/realm.properties with the following content (I need only one administrator login):

admin: CRYPT:827ccb0eea8a706c4c34a16891f84e7b,server-administrator,content-administrator,admin,readwrite

Where 827ccb0eea8a706c4c34a16891f84e7b is an MD5 encrypted password "12345" which can be created by either of two ways:

Using the java with jetty plugin "util" jar:

cd usr/share/elasticsearch/plugins/jetty-1.2.1
sudo java -cp ./jetty-util-8.1.14.v20131031.jar org.eclipse.jetty.util.security.Password admin 12345

OBF:19bv19bx19bz19c119c3
MD5:827ccb0eea8a706c4c34a16891f84e7b
CRYPT:adpliAB3dA.06

Or using md5sum Linux command that will produce the same MD5 encrypted password:

echo -n "12345" | md5sum
827ccb0eea8a706c4c34a16891f84e7b  -

Detailed explanation of Jetty realm authentication can be found on: https://wiki.eclipse.org/Jetty/Tutorial/Realms http://www.eclipse.org/jetty/documentation/9.1.1.v20140108/configuring-security-authentication.html

In this part of the tutorial, I described Elasticsearch installation on CentOS 6.5 with Sonian elasticsearch-jetty plugin; I generated credentials and configured security realm.

Please continue to the next chapter and don't hesitate to leave your comments or suggestions below.

Part One: Install Elasticsearch (you are here)

Part Two: Elasticsearch tuning

Part Three: Install Logstash

Part Four: Logstash mapping

Part Five: Install Kibana 4 and create dashboard

Andrey Kanevsky, DevOps engineer @ DevOps Ltd.

Elasticsearch, Kibana, Logstash and Grafana are trademarks of the Elasticsearch BV.
Nagios is a trademark of the Nagios Enterprises.
Sensu is a trademark of the Heavy Water Operations.
Pagerduty is a trademark of the PagerDuty Inc.