mirror of https://github.com/cowrie/cowrie.git
175 lines
5.8 KiB
ReStructuredText
175 lines
5.8 KiB
ReStructuredText
How to process Cowrie output in an ELK stack
|
|
#############################################
|
|
|
|
(Note: work in progress, instructions are not verified)
|
|
|
|
|
|
Prerequisites
|
|
================
|
|
|
|
* Working Cowrie installation
|
|
* Cowrie JSON log file (enable database json in cowrie.cfg)
|
|
* Java 8
|
|
|
|
Installation
|
|
================
|
|
|
|
This is a simple setup for ELK stack, to be done on the same machine that is used for cowrie. We use *Filebeat* to send logs to *Logstash*, and we use *Nginx* as a reverse proxy to access *Kibana*. Note there are many other possible configurations!
|
|
|
|
Add Elastic's repository and key::
|
|
|
|
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
|
|
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list
|
|
apt-get update
|
|
|
|
Install logstash, elasticsearch, kibana and filebeat::
|
|
|
|
sudo apt -y install apt-transport-https wget default-jre
|
|
sudo apt install elasticsearch logstash kibana
|
|
sudo apt install filebeat
|
|
sudo apt install nginx apache2-utils
|
|
|
|
Enable the services::
|
|
|
|
sudo systemctl enable elasticsearch logstash kibana filebeat nginx
|
|
|
|
|
|
ElasticSearch Configuration
|
|
=============================
|
|
|
|
ElasticSearch configuration file is located in `/etc/elasticsearch/elasticsearch.yml`. The default settings need not be changed.
|
|
|
|
By default, ElasticSearch listens on port 9200. Test it::
|
|
|
|
curl http://localhost:9200
|
|
|
|
You should get a JSON object in return.
|
|
|
|
|
|
Kibana Configuration
|
|
=============================
|
|
|
|
Make a folder for logs::
|
|
|
|
sudo mkdir /var/log/kibana
|
|
sudo chown kibana:kibana /var/log/kibana
|
|
|
|
Change the following parameters in `/etc/kibana/kibana.yml` to reflect your server setup::
|
|
|
|
"server.host" - set it to "localhost" if you use nginx for basic authentication or external interface if you use XPack (see below)
|
|
"server.name" - name of the server
|
|
"elasticsearch.url" - address of the elasticsearch: ["http://localhost:9200"]
|
|
"elasticsearch.username", "elasticsearch.password" - needed only if you use XPack (see below)
|
|
"logging.dest" - set path to logs (/var/log/kibana/kibana.log)
|
|
|
|
Logstash Configuration
|
|
=============================
|
|
|
|
Get GeoIP data from www.maxmind.com (free but requires registration): download the GeoLite2 City GZIP. Unzip it and locate the mmdb file.
|
|
Place it somewhere in your filesystem and make sure that "logstash" user can read it::
|
|
|
|
sudo mkdir -p /opt/logstash/vendor/geoip/
|
|
sudo mv GeoLite2-City.mmdb /opt/logstash/vendor/geoip
|
|
|
|
Configure logstash::
|
|
|
|
sudo cp logstash-cowrie.conf /etc/logstash/conf.d
|
|
|
|
Make sure the configuration file is correct. Check the input section (path), filter (geoip databases) and output (elasticsearch hostname)::
|
|
|
|
sudo systemctl restart logstash
|
|
|
|
|
|
Filebeat Configuration
|
|
==============================
|
|
|
|
Filebeat is not mandatory (it is possible to directly read cowrie logs from Logstash) but nice to have, because if Logstash is under pressure, it automatically knows to slow down + it is possible to deal with multiple sensor inputs.
|
|
|
|
Configure filebeat::
|
|
|
|
sudo cp filebeat-cowrie.conf /etc/filebeat/filebeat.yml
|
|
|
|
Check the following parameters::
|
|
|
|
filebeat.inputs: the path must point to cowrie's json logs
|
|
output.elasticsearch: must be false because we want Filebeat to send to Logstash, not directly to ElasticSearch
|
|
output.logstash: must be true. The default port for Logstash is 5044, so hosts should be ["localhost:5044"]
|
|
|
|
|
|
Start filebeat::
|
|
|
|
sudo systemctl start filebeat
|
|
|
|
Nginx
|
|
==================
|
|
|
|
ELK has been configured on localhost. If you wish to access it remotely, you can setup a reverse proxy to Kibana's backend server, which runs on port 5601 by default.
|
|
|
|
Install Nginx::
|
|
|
|
sudo apt install nginx apache2-utils
|
|
|
|
Create an administrative Kibana user and password::
|
|
|
|
sudo htpasswd -c /etc/nginx/htpasswd.users admin_kibana
|
|
|
|
Edit Nginx configuration /etc/nginx/sites-available/default. Customize port to what you like, and specify your server's name (or IP address)::
|
|
|
|
server {
|
|
listen YOURPORT;
|
|
|
|
server_name YOURIPADDRESS;
|
|
|
|
auth_basic "Restricted Access";
|
|
auth_basic_user_file /etc/nginx/htpasswd.users;
|
|
|
|
location / {
|
|
proxy_pass http://localhost:5601;
|
|
proxy_http_version 1.1;
|
|
proxy_set_header Upgrade $http_upgrade;
|
|
proxy_set_header Connection 'upgrade';
|
|
proxy_set_header Host $host;
|
|
proxy_cache_bypass $http_upgrade;
|
|
}
|
|
}
|
|
|
|
Start the service::
|
|
|
|
sudo systemctl start nginx
|
|
|
|
|
|
Using Kibana
|
|
==================
|
|
|
|
You can list indexes with::
|
|
|
|
curl 'http://localhost:9200/_cat/indices?v'
|
|
|
|
You should see a cowrie index cowrie-logstash-DATE... Its health is yellow because the number of replicas should be set to 0 (unless you want another configuration)::
|
|
|
|
curl -XPUT 'localhost:9200/cowrie-logstash-REPLACEHERE/_settings' -H "Content-Type: application/json" -d '{ "index" : {"number_of_replicas" : 0 } }'
|
|
|
|
It should answer {"acknowledged":true}
|
|
|
|
In Kibana's GUI, create an index pattern (Management / Index Patterns) for ::
|
|
|
|
cowrie-logstash-*
|
|
|
|
Use default settings and timestamp.
|
|
|
|
|
|
Tuning ELK stack
|
|
==================
|
|
|
|
Refer to elastic's documentation about proper configuration of the system for the best elasticsearch's performance
|
|
|
|
You may avoid installing nginx for restricting access to kibana by installing official elastic's plugin called "X-Pack" (https://www.elastic.co/products/stack)
|
|
|
|
ELK log files get big: ensure you have enough space in /var, consider setting up LVM or ZFS partitions.
|
|
|
|
Troubleshooting
|
|
==================
|
|
|
|
- View service logs with: sudo journalctl -u service
|
|
- If the date in Kibana is incorrect, check (Advanced Settings / dateFormat)
|