PipeBug: Monitoring Using Graphite, Logstash, Sensu, and Tessera

The dark side of the ELK stack. Unleash the Logstash mapping.

In memory of the beloved Kibana 3. We will never forget.

Part Five: Install Kibana 4 and create dashboard

Kibana 4 release is out, and I want to give it a try:

cd /opt
wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz
tar zxvf kibana-4.0.1-linux-x64.tar.gz
ln -sf kibana-4.0.1-linux-x64 kibana

I changed the config file /opt/kibana/config/kibana.yml a little bit:

# Kibana is served by a backend server. This controls which port to use.
port: 5601

# The host to bind the server to.
host: "example.com"

# The Elasticsearch instance to use for all your queries.
elasticsearch_url: "http://example.com:9200"
username: "admin"
password: "12345"

# preserve_elasticsearch_host true will send the hostname specified in `elasticsearch`. If you set it to false,
# then the host you use to connect to *this* Kibana instance will be sent.
elasticsearch_preserve_host: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations
# and dashboards. It will create a new index if it doesn't already exist.
kibana_index: ".kibana"

# If your Elasticsearch is protected with basic auth, this is the user credentials
# used by the Kibana server to perform maintence on the kibana_index at statup. Your Kibana
# users will still need to authenticate with Elasticsearch (which is proxied through
# the Kibana server)
kibana_elasticsearch_username: "admin"
kibana_elasticsearch_password: "12345"

# The default application to load.
default_app_id: "discover"

# Time in milliseconds to wait for responses from the backend or elasticsearch.
# This must be > 0
request_timeout: 300000

# Time in milliseconds for Elasticsearch to wait for responses from shards.
# Set to 0 to disable.
shard_timeout: 0

# Plugins that are included in the build, and no longer found in the plugins/ folder
 - plugins/dashboard/index
 - plugins/discover/index
 - plugins/doc/index
 - plugins/kibana/index
 - plugins/markdown_vis/index
 - plugins/metric_vis/index
 - plugins/settings/index
 - plugins/table_vis/index
 - plugins/vis_types/index
 - plugins/visualize/index

To run Kibana, execute the following command:

cd /opt/kibana
bash -c "bin/kibana > /var/log/kibana.log &"

Kibana 4 web UI setup

I opened http://example.com:5601 and after entering username and password I can see Kibana settings page: Kibana welcome screen

Kibana time field name Entering the index pattern - "example-*" allows me to choose time-field name "@timestamp" and then click the green "Create" button.

It's time to discover Kibana 4.

Kibana discover screen

By default, we can see logs for the last 15 minutes. You can change the interval with the click on the small clock symbol in the top right corner:

Kibana time filter

There are three types of time filters:

Also, there is useful tab of "Refresh interval" from 5 seconds to 1 day, which is off by default.

The left column is a list of available fields. Click on the field title will open five most popular metrics. The small magnified glass can be used to add output filters in one click:

Kibana selected field

And also, you have the possibility to add fields to form the desired output:

Kibana selected fields

Kibana selected fields view

When the desired filters are selected, it is a good time to save the search by clicking on a small floppy disk icon on the right side of search field:

Kibana save search

Next step is to create a visualization from saved search. Click on Visualize tab will open a "Create a new visualization" window:

Kibana create visualization

Here I selected the visualization I prefer, picked a search source from saved search:

Kibana select search source

Important thing to know is a response code dynamic. "Vertical bar" chart metrics: Y-Axis: "Count"; bucket type X-Axis, aggregation "Date Histogram" by "@timestamp" field with "Auto" interval; Sub Aggregation with bucket type "Split Bars" by "Terms" "response" field:

Kibana response code metrics

From which countries visitors come to us? "Pie chart" metrics: Slice Size: "Count"; bucket type "Split slices", aggregation by "Terms", Field "geoip.country_name", option "Donut":

Kibana countries donut metrics

Great looking map of visitors based on MapQuest tiles can be created by "Tile map" visualization. Metrics value: "Count"; bucket type "Geo coordinates"; aggregation by "Geohash", field "geoip.location", option "Shaded Circle Markers", Precision - 4:

Kibana tile map shaded

For DDos detection the same map, but with Scaled Circle Markers can be very useful:

Kibana tile map scaled

The nesting implemented in Kibana 4 is very handy! I want to see which devices were used in which country. "Pie chart" metrics: Slice size "Count"; bucket type "Split chart" by columns; aggregation "Terms"; Field "geoip.country_name"; Sub aggregation "Split slices" by terms "agent.device"; option: "Donut":

Kibana nesting country device donut

In this part of the tutorial, I described the visualization and dashboard creation for production Apache server monitoring.

Part One: Install Elasticsearch

Part Two: Elasticsearch tuning

Part Three: Install Logstash

Part Four: Logstash mapping

Part Five: Install Kibana 4 and create dashboard (you are here)

Andrey Kanevsky, DevOps engineer @ DevOps Ltd.

Elasticsearch, Kibana, Logstash and Grafana are trademarks of the Elasticsearch BV.
Nagios is a trademark of the Nagios Enterprises.
Sensu is a trademark of the Heavy Water Operations.
Pagerduty is a trademark of the PagerDuty Inc.