PipeBug: Monitoring Using Graphite, Logstash, Sensu, and Tessera

The dark side of the ELK stack. Unleash the Logstash mapping.

In memory of the beloved Kibana 3. We will never forget.

Part Four: Logstash mapping

Using mapping template you can easily achieve a number of benefits, such as:

Kibana mapping Logstash mapping is an important moment, I want to have a control over the default settings. I created mapping template which is a JSON file in the file /etc/logstash/http-log-logstash.json.

I got rid of all garbage in config file /etc/logstash/conf.d/logstash-httpd.conf, I removed all the unnecessary fields by "remove_field".

In http-log-logstash.json I defined proper field types and told Logstash not to analyze them and not break field down into searchable terms. Also, I disabled _all field, and I limited the number of shards to 1:

{ 
    "template" : "http-log-logstash", 
    "settings" : { "index.refresh_interval" : "60s" }, 
    "mappings" : { 
        "_default_" : { 
            "_all" : { "enabled" : false }, 
            "dynamic_templates" : [{ 
              "message_field" : { 
                "match" : "message", 
                "match_mapping_type" : "string", 
                "mapping" : { "type" : "string", "index" : "not_analyzed" } 
              } 
            }, { 
              "string_fields" : { 
                "match" : "*", 
                "match_mapping_type" : "string", 
                "mapping" : { "type" : "string", "index" : "not_analyzed" } 
              } 
            }], 
            "properties" : { 
                "@timestamp" : { "type" : "date", "format" : "dateOptionalTime" }, 
                "@version" : { "type" : "integer", "index" : "not_analyzed" }, 
                "agent" : { "type" : "string", "index" : "not_analyzed" }, 
                "bytes" : { "type" : "long", "norms" : { "enabled" : false } }, 
				"host" : { "type" : "string", "index" : "not_analyzed" }, 
                "clientip" : { "type" : "ip", "norms" : { "enabled" : false } }, 
                "httpversion" : { "type" : "float" }, 
                "referrer" : { "type" : "string", "index" : "not_analyzed" }, 
                "request" : { "type" : "string", "index" : "not_analyzed", "include_in_all": false }, 
                "response" : { "type" : "integer", "index" : "not_analyzed" }, 
                "geoip" : { "type" : "object", "dynamic" : true, "path" : "full", "properties" : { "location" : { "type" : "geo_point" } } }, 
                "verb" : { "type" : "string", "norms" : { "enabled" : false } } 
            } 
        } 
    } 
}

The default template has a lower order, so I override it with my definitions. The new index will be created with my custom mapping.

I want to check it out with:

curl -XGET http://admin:12345@example.com:9200/_template/http-log-logstash?pretty

Amazing! The default structure is overridden by my template:

{
    "http-log-logstash": {
	    "order": 0,
	    "template": "example-*",
	    "settings": {
		    "index.refresh_interval": "60s",
		    "index.number_of_shards": "1"
		},
		"mappings": {
    		"_default_": {
    			"dynamic_templates": [
					{
    					"message_field": {
						    "mapping": {
						            "index": "not_analyzed",
						            "type": "string"
						        },
						        "match_mapping_type": "string",
						        "match": "message"
						    }
					},
					{
    					"string_fields": {
						    "mapping": {
						                "index": "not_analyzed",
						                "type": "string"
						            },
						            "match_mapping_type": "string",
						            "match": "*"
						        }
						    }
				],
				"properties": {
				    "response": {
					    "index": "not_analyzed",
					    "type": "integer"
					},
				"httpversion": {
				    "type": "float"
				},
				"@timestamp": {
				    "format": "dateOptionalTime",
				    "type": "date"
				},
				"referrer": {
				    "index": "not_analyzed",
				    "type": "string"
				},
				"host": {
				    "index": "not_analyzed",
				    "type": "string"
				},
				"bytes": {
				    "norms": {
				        "enabled": false
				    },
				    "type": "long"
				},
				"verb": {
				    "norms": {
				        "enabled": false
				    },
				    "type": "string"
				},
				"clientip": {
				    "norms": {
				        "enabled": false
				    },
				    "type": "ip"
				},
				"request": {
				    "include_in_all": false,
				    "index": "not_analyzed",
				    "type": "string"
				},
				"geoip": {
				    "dynamic": true,
				    "path": "full",
				    "properties": {
					    "location": {
					            "type": "geo_point"
					        }
				    },
				    "type": "object"
				},
				"agent": {
				    "index": "not_analyzed",
				    "type": "string"
				},
				"@version": {
				        "index": "not_analyzed",
				        "type": "integer"
				}
			},
			"_all": {
				"enabled": false
			}
		}
	},
	"aliases": { }
	}
}

If you are testing your template and want to upload a new one, I should mention important thing - templates are cached in Elasticsearch!

If you make changes, delete an old template first. To see a templates list from Elasticsearch:

curl -XGET http://admin:12345@example.com:9200/_template?pretty

To delete a template from Elasticsearch (we previously defined a template "http-log-logstash"):

curl -XDELETE http://admin:12345@example.com:9200/_template/http-log-logstash

It's an extremely good idea to check your config file before you start a service:

/opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/logstash-httpd.conf 
 
Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.5.0.beta1/plugin-milestones {:level=>:warn} 
Using milestone 1 filter plugin 'fingerprint'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.5.0.beta1/plugin-milestones {:level=>:warn} 
Configuration OK

Well, "Configuration OK" sounds good! I want to have a Logstash up and running:

service logstash start

Check the logs:

ls -s /var/log/logstash/logstash.err 
0 /var/log/logstash/logstash.err

Great news, no errors!

What do I have in logstash.log?

cat /var/log/logstash/logstash.log 
{:timestamp=>"2015-03-03T15:17:35.328000+0200", :message=>"Using milestone 1 filter plugin 'fingerprint'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.5.0.beta1/plugin-milestones", :level=>:warn} 
{:timestamp=>"2015-03-03T15:17:35.677000+0200", :message=>"Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.5.0.beta1/plugin-milestones", :level=>:warn}

Logstash is working, no error lines is always a good sign.

Last check - are there any logs collected in Elasticsearch?

curl -XGET 'http://admin:12345@example.com:9200/_cat/indices?v' 
health status index              pri rep docs.count docs.deleted store.size pri.store.size 
yellow open   example-2015.03.03   5   1      32533           15     14.9mb         14.9mb

Perfect, I have an index defined, and 32K docs already collected. It's Kibana time!

Part One: Install Elasticsearch

Part Two: Elasticsearch tuning

Part Three: Install Logstash

Part Four: Logstash mapping (you are here)

Part Five: Install Kibana 4 and create dashboard

Andrey Kanevsky, DevOps engineer @ DevOps Ltd.

Elasticsearch, Kibana, Logstash and Grafana are trademarks of the Elasticsearch BV.
Nagios is a trademark of the Nagios Enterprises.
Sensu is a trademark of the Heavy Water Operations.
Pagerduty is a trademark of the PagerDuty Inc.