PipeBug: Monitoring Using Graphite, Logstash, Sensu, and Tessera

The dark side of the ELK stack. Unleash the Logstash mapping.

In memory of the beloved Kibana 3. We will never forget.

Part Two: Elasticsearch tuning

Elasticsearch configuration YAML file can be found at /etc/elasticsearch/elasticsearch.yml

Configure Jetty:

http.type: com.sonian.elasticsearch.http.jetty.JettyHttpServerTransport 
sonian.elasticsearch.http.jetty: 
	config: jetty.xml,jetty-hash-auth.xml,jetty-restrict-all.xml 
	bind_host: example.com 
	publish_host: example.com

By default, cross-origin resource in Elasticsearch is disabled. With following two lines I enabled cross-origin resource sharing in Elasticsearch and allow CORS requests from any origin:

http.cors.allow-origin: "/.*/" 
http.cors.enabled: true 
http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, Access-Control-Allow-Credentials, Access-Control-Allow-Origin"

Some main definitions:

http.host: "example.com" 
cluster.name: elasticsearch 
node.name: "example.com"

Turn on the credentials:

http.cors.allow-credentials: true

Make sure JVM will not swap. Not even a little bit! Poor Elasticsearch performance is boring:

bootstrap.mlockall: true

After Elasticsearch had started, I checked if this option applied by curl http://admin:12345@example.com:9200/_nodes/process?pretty

If I can see "mlockall" : true in the output that means that Java process will try to lock the process address space into memory. Preventing any memory from being swapped to disk.

But if I see "mlockall" : false there are few possible scenarios:

a) Elasticsearch doesn’t have enough permission to lock memory (try to run sudo ulimit -l unlimited before start);

b) The temporary directory is mounted without noexec option (you can specify another temporary directory by passing -Djna.tmpdir=SOME_TMP_DIR option to Elasticsearch during startup)

Be aware, if mlockall tries to allocate more memory than available it can cause Java Virtual Machine to exit abnormally.

Finally, I defined a real hostname:

network.bind_host: example.com 
network.publish_host: example.com 
network.host: example.com

When I finished with elasticsearch.yml config file, I checked the file /etc/sysconfig/elasticsearch.

The most important environment variable here is ES_HEAP_SIZE, which allows allocating memory to elasticsearch java process. It will set up the same value to both min and max values. If you want, you can set explicit values to ES_MIN_MEM and ES_MAX_MEM variables, but official Elasticsearch tutorial does not recommend it.

Also, I increased max number of open files descriptors up to 64K and set max number of virtual memory areas to 262144.

You can can check max_map_count in your system by running sysctl vm.max_map_count and set a new value with sysctl -w vm.max_map_count=262144. To make this setting permanent add vm.max_map_count variable in /etc/sysctl.conf

# Directory where the Elasticsearch binary distribution resides 
ES_HOME=/usr/share/elasticsearch 
 
# Heap Size 
ES_HEAP_SIZE=14g 
 
# Maximum number of open files 
MAX_OPEN_FILES=65535 
 
# Maximum number of VMA (Virtual Memory Areas) a process can own 
MAX_MAP_COUNT=262144 
 
# Elasticsearch log directory 
LOG_DIR=/var/log/elasticsearch 
 
# Elasticsearch data directory 
DATA_DIR=/var/lib/elasticsearch 
 
# Elasticsearch work directory 
WORK_DIR=/tmp/elasticsearch 
 
# Elasticsearch conf directory 
CONF_DIR=/etc/elasticsearch 
 
# Elasticsearch configuration file (elasticsearch.yml) 
CONF_FILE=/etc/elasticsearch/elasticsearch.yml 
 
# User to run as, change this to a specific elasticsearch user if possible 
# Also make sure, this user can write into the log directories in case you change them 
# This setting only works for the init script, but has to be configured separately for systemd startup 
ES_USER=elasticsearch 
 
# Configure restart on package upgrade (true, every other setting will lead to not restarting) 
RESTART_ON_UPGRADE=true

Now I want to run Elasticsearch service:

service elasticsearch start

And I believe I did it all right:

curl 'http://admin:12345@example.com:9200/'

What a relief, I did!

{ 
	"status" : 200, 
	"name" : "example.com", 
	"cluster_name" : "elasticsearch", 
	"version" : { 
		"number" : "1.4.4", 
		"build_hash" : "c88f37ffc92739dfa9dfd81ca2232f09588bd512", 
		"build_timestamp" : "2015-02-19T13:05:36Z", 
		"build_snapshot" : false, 
		"lucene_version" : "4.10.3" 
	}, 
	"tagline" : "You Know, for Search" 
}

In this part of the tutorial, I described Elasticsearch secure configuration and fine tuning. Please continue to the next chapter and don't hesitate to leave your comments or suggestions below.

Part One: Install Elasticsearch

Part Two: Elasticsearch tuning (you are here)

Part Three: Install Logstash

Part Four: Logstash mapping

Part Five: Install Kibana 4 and create dashboard

Andrey Kanevsky, DevOps engineer @ DevOps Ltd.

Elasticsearch, Kibana, Logstash and Grafana are trademarks of the Elasticsearch BV.
Nagios is a trademark of the Nagios Enterprises.
Sensu is a trademark of the Heavy Water Operations.
Pagerduty is a trademark of the PagerDuty Inc.