Rooba
/
cowrie
mirror of https://github.com/cowrie/cowrie.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
cowrie/doc/elk/README.md

4.3 KiB
Raw Blame History

How to process Cowrie output in an ELK stack

(Note: work in progress, instructions are not verified)

Prerequisites

  • Working Cowrie installation
  • Cowrie JSON log file (enable database json in cowrie.cfg)
  • Java 8

Installation

We'll examine simple installation, when we install ELK stack on the same machine that used for cowrie.

  • Add Elastic's repository and key
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
apt-get update
  • Install logstash, elasticsearch and kibana
sudo apt-get install elasticsearch logstash kibana
  • Set them to autostart
sudo update-rc.d elasticsearch defaults 95 10
sudo update-rc.d kibana defaults 95 10

ElasticSearch Configuration

TBD

Kibana Configuration

  • Make a folder for logs
sudo mkdir /var/log/kibana
sudo chown kibana:kibana /var/log/kibana
  • Change the following parameters in /etc/kibana/kibana.yml to reflect your server setup:
"server.host"  - set it to "localhost" if you use nginx for basic authentication or external interface if you use XPack (see below)
"server.name" - name of the server
"elasticsearch.url" - address of the elasticsearch
"elasticsearch.username", "elasticsearch.password" - needed only if you use XPack (see below)
"logging.dest" - set path to logs (/var/log/kibana/kibana.log)
  • Make sure the file /etc/kibana/kibana.yml contains a line like
tilemap.url: https://tiles.elastic.co/v2/default/{z}/{x}/{y}.png?elastic_tile_service_tos=agree&my_app_name=kibana

or your map visualizations won't have any background. When this file is created during the installation of Kibana, it does not contain such a line, not even in commented out form.

Logstash Configuration

  • Download GeoIP data
wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
  • Place these somewhere in your filesystem and make sure that "logstash" user can read it
sudo mkdir -p /var/opt/logstash/vendor/geoip/
sudo mv GeoLite2-City.mmdb /var/opt/logstash/vendor/geoip
  • Configure logstash
sudo cp logstash-cowrie.conf /etc/logstash/conf.d
  • Make sure the configuration file is correct. Check the input section (path), filter (geoip databases) and output (elasticsearch hostname)
sudo service logstash restart

  • By default the logstash is creating debug logs in /tmp.

  • To test whether logstash is working correctly, check the file in /tmp

tail /tmp/cowrie-logstash.log
  • To test whether data is loaded into ElasticSearch, run the following query:
curl 'http://<hostname>:9200/_search?q=cowrie&size=5'

(Replace <hostname> with the name or IP address of the machine on which ElasticSearch is running, e.g., localhost.)

  • If this gives output, your data is correctly loaded into ElasticSearch

  • When you successfully configured logstash, remove "file" and "stdout" blocks from output section of logstash configuration.

Distributed setup of sensors or multiple sensors on the same host

If you have multiple sensors, you will need to setup up FileBeat to feed logstash with logs from all sensors

On the logstash server:

  • Change "input" section of the logstash to the following:
input {
   beats {
       port => 5044
   }
}

On the sensor servers:

  • Install filebeat
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
sudo apt-get update
sudo apt-get install filebeat
  • Enable autorun for it
sudo update-rc.d filebeat defaults 95 10
  • Configure filebeat
sudo cp filebeat-cowrie.conf /etc/filebeat/filebeat.yml
  • Check the following parameters
paths - path to cowrie's json logs
logstash - check ip of the logstash host
  • Start filebeat
sudo service filebeat start

Tuning ELK stack

  • Refer to elastic's documentation about proper configuration of the system for the best elasticsearch's performance

  • You may avoid installing nginx for restricting access to kibana by installing official elastic's plugin called "X-Pack" (https://www.elastic.co/products/x-pack)