cowrie/docs/elk

How to process Cowrie output in an ELK stack
#############################################

(Note: work in progress, instructions are not verified)


Prerequisites
================

* Working Cowrie installation
* Cowrie JSON log file (enable database json in cowrie.cfg)
* Java 8

Installation
================


We'll examine simple installation, when we install ELK stack on the same machine that used for cowrie.

Add Elastic's repository and key::

    wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
    echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
    apt-get update

Install logstash, elasticsearch and kibana::

     sudo apt-get install elasticsearch logstash kibana

Set them to autostart::

    sudo update-rc.d elasticsearch defaults 95 10
    sudo update-rc.d kibana defaults 95 10

ElasticSearch Configuration
=============================

TBD

Kibana Configuration
=============================

Make a folder for logs::

    sudo mkdir /var/log/kibana
    sudo chown kibana:kibana /var/log/kibana

Change the following parameters in `/etc/kibana/kibana.yml` to reflect your server setup::

    "server.host"  - set it to "localhost" if you use nginx for basic authentication or external interface if you use XPack (see below)
    "server.name" - name of the server
    "elasticsearch.url" - address of the elasticsearch
    "elasticsearch.username", "elasticsearch.password" - needed only if you use XPack (see below)
    "logging.dest" - set path to logs (/var/log/kibana/kibana.log)

Make sure the file `/etc/kibana/kibana.yml` contains a line like::

    tilemap.url: https://tiles.elastic.co/v2/default/{z}/{x}/{y}.png?elastic_tile_service_tos=agree&my_app_name=kibana

or your map visualizations won't have any background. When this file is created during the installation
of Kibana, it does _not_ contain such a line, not even in commented out form.

Logstash Configuration
=============================

Download GeoIP data::

    wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz

Place these somewhere in your filesystem and make sure that "logstash" user can read it::

    sudo mkdir -p /var/opt/logstash/vendor/geoip/
    sudo mv GeoLite2-City.mmdb /var/opt/logstash/vendor/geoip

Configure logstash::

    sudo cp logstash-cowrie.conf /etc/logstash/conf.d

Make sure the configuration file is correct. Check the input section (path), filter (geoip databases) and output (elasticsearch hostname)::

    sudo service logstash restart

By default the logstash is creating debug logs in /tmp.

To test whether logstash is working correctly, check the file in /tmp::

    tail /tmp/cowrie-logstash.log

To test whether data is loaded into ElasticSearch, run the following query::

    curl 'http://<hostname>:9200/_search?q=cowrie&size=5'

(Replace `<hostname>` with the name or IP address of the machine on which ElasticSearch is running, e.g., `localhost`.)

If this gives output, your data is correctly loaded into ElasticSearch

When you successfully configured logstash, remove "file" and "stdout" blocks from output section of logstash configuration.

Distributed setup of sensors or multiple sensors on the same host
================================================================================

If you have multiple sensors, you will need to setup up FileBeat to feed logstash with logs from all sensors
 
On the logstash server:
 
Change "input" section of the logstash to the following::
 
    input {
       beats {
           port => 5044
       }
    }

On the sensor servers:
 
Install filebeat::

    wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
    echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
    sudo apt-get update
    sudo apt-get install filebeat
 
Enable autorun for it::

    sudo update-rc.d filebeat defaults 95 10

Configure filebeat::
 
    sudo cp filebeat-cowrie.conf /etc/filebeat/filebeat.yml

Check the following parameters::

    paths - path to cowrie's json logs
    logstash - check ip of the logstash host
 
Start filebeat::

    sudo service filebeat start

Tuning ELK stack
==================

Refer to elastic's documentation about proper configuration of the system for the best elasticsearch's performance

You may avoid installing nginx for restricting access to kibana by installing official elastic's plugin called "X-Pack" (https://www.elastic.co/products/stack)