mirror of https://github.com/cowrie/cowrie.git
146 lines
4.5 KiB
ReStructuredText
146 lines
4.5 KiB
ReStructuredText
How to process Cowrie output in an ELK stack
|
|
#############################################
|
|
|
|
(Note: work in progress, instructions are not verified)
|
|
|
|
|
|
Prerequisites
|
|
================
|
|
|
|
* Working Cowrie installation
|
|
* Cowrie JSON log file (enable database json in cowrie.cfg)
|
|
* Java 8
|
|
|
|
Installation
|
|
================
|
|
|
|
|
|
We'll examine simple installation, when we install ELK stack on the same machine that used for cowrie.
|
|
|
|
Add Elastic's repository and key::
|
|
|
|
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
|
|
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
|
|
apt-get update
|
|
|
|
Install logstash, elasticsearch and kibana::
|
|
|
|
sudo apt-get install elasticsearch logstash kibana
|
|
|
|
Set them to autostart::
|
|
|
|
sudo update-rc.d elasticsearch defaults 95 10
|
|
sudo update-rc.d kibana defaults 95 10
|
|
|
|
ElasticSearch Configuration
|
|
=============================
|
|
|
|
TBD
|
|
|
|
Kibana Configuration
|
|
=============================
|
|
|
|
Make a folder for logs::
|
|
|
|
sudo mkdir /var/log/kibana
|
|
sudo chown kibana:kibana /var/log/kibana
|
|
|
|
Change the following parameters in `/etc/kibana/kibana.yml` to reflect your server setup::
|
|
|
|
"server.host" - set it to "localhost" if you use nginx for basic authentication or external interface if you use XPack (see below)
|
|
"server.name" - name of the server
|
|
"elasticsearch.url" - address of the elasticsearch
|
|
"elasticsearch.username", "elasticsearch.password" - needed only if you use XPack (see below)
|
|
"logging.dest" - set path to logs (/var/log/kibana/kibana.log)
|
|
|
|
Make sure the file `/etc/kibana/kibana.yml` contains a line like::
|
|
|
|
tilemap.url: https://tiles.elastic.co/v2/default/{z}/{x}/{y}.png?elastic_tile_service_tos=agree&my_app_name=kibana
|
|
|
|
or your map visualizations won't have any background. When this file is created during the installation
|
|
of Kibana, it does _not_ contain such a line, not even in commented out form.
|
|
|
|
Logstash Configuration
|
|
=============================
|
|
|
|
Download GeoIP data::
|
|
|
|
wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
|
|
|
|
Place these somewhere in your filesystem and make sure that "logstash" user can read it::
|
|
|
|
sudo mkdir -p /var/opt/logstash/vendor/geoip/
|
|
sudo mv GeoLite2-City.mmdb /var/opt/logstash/vendor/geoip
|
|
|
|
Configure logstash::
|
|
|
|
sudo cp logstash-cowrie.conf /etc/logstash/conf.d
|
|
|
|
Make sure the configuration file is correct. Check the input section (path), filter (geoip databases) and output (elasticsearch hostname)::
|
|
|
|
sudo service logstash restart
|
|
|
|
By default the logstash is creating debug logs in /tmp.
|
|
|
|
To test whether logstash is working correctly, check the file in /tmp::
|
|
|
|
tail /tmp/cowrie-logstash.log
|
|
|
|
To test whether data is loaded into ElasticSearch, run the following query::
|
|
|
|
curl 'http://<hostname>:9200/_search?q=cowrie&size=5'
|
|
|
|
(Replace `<hostname>` with the name or IP address of the machine on which ElasticSearch is running, e.g., `localhost`.)
|
|
|
|
If this gives output, your data is correctly loaded into ElasticSearch
|
|
|
|
When you successfully configured logstash, remove "file" and "stdout" blocks from output section of logstash configuration.
|
|
|
|
Distributed setup of sensors or multiple sensors on the same host
|
|
================================================================================
|
|
|
|
If you have multiple sensors, you will need to setup up FileBeat to feed logstash with logs from all sensors
|
|
|
|
On the logstash server:
|
|
|
|
Change "input" section of the logstash to the following::
|
|
|
|
input {
|
|
beats {
|
|
port => 5044
|
|
}
|
|
}
|
|
|
|
On the sensor servers:
|
|
|
|
Install filebeat::
|
|
|
|
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
|
|
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
|
|
sudo apt-get update
|
|
sudo apt-get install filebeat
|
|
|
|
Enable autorun for it::
|
|
|
|
sudo update-rc.d filebeat defaults 95 10
|
|
|
|
Configure filebeat::
|
|
|
|
sudo cp filebeat-cowrie.conf /etc/filebeat/filebeat.yml
|
|
|
|
Check the following parameters::
|
|
|
|
paths - path to cowrie's json logs
|
|
logstash - check ip of the logstash host
|
|
|
|
Start filebeat::
|
|
|
|
sudo service filebeat start
|
|
|
|
Tuning ELK stack
|
|
==================
|
|
|
|
Refer to elastic's documentation about proper configuration of the system for the best elasticsearch's performance
|
|
|
|
You may avoid installing nginx for restricting access to kibana by installing official elastic's plugin called "X-Pack" (https://www.elastic.co/products/stack)
|