Guide for configuring Graylog log rotation.
Graylog offers a number of settings to manage the log files stored in the server. Logs are stored in indicies. To view the current index set sizes and rotation settings, log into your Graylog web interface and navigate to System -> Indices
The Default index set is the location where log files from your clients are stored. Depending on your needs, you may find that the default settings are not appropriate for your server's storage capacity or your data retention requirements. For example, a Graylog server running with default settings for a few weeks might have a Default index set that looks like this:
There are a number of settings that determine how long the logs are kept for:
Index rotation strategy - This can be set to either Index Message Count where the log will continue to grow until it contains the set amount of messages, Index Size where the log will continue to grow to a specified file size or Index Time where the log will be rotated after a certain amount of time, regardless of it's message count or size.
Index retention strategy - Your options here are Archive (only available for Graylog Enterprise customers), Delete to remove a rotated log from disk, Close to prevent new messages being written to the log file or Do Nothing.
Max number of indices / Rotation Period - This is the number of indexes to keep before deleting the oldest index when using the Delete retention strategy.
The default settings for the Graylog Default index set is rotate to the next index when the current index gets to 20 million messages and to keep 20 indexes before deleting old indexes. The size on disk of 20 million messages is highly variable, but from my experience you'd want to have at least 500GB to store that amount of logs.
You may find it makes more sense to, for example, only keep 30 days of logs and rotate logs each day. To do this, change the Index rotation strategy to Index Time and adjust the rotation period to P1D. Change the retention strategy to Delete Index and set the Max number of indices to 30.
Manually delete Graylog Log files
If you find that Graylog wont remove an old log file, you can manually delete them using the ElasitcSearch API.
Connect to your Graylog server using an SSH client such as PuTTY, and run the following command:
curl http://localhost:9200/_cat/indices
This will output all of the indices (log files) that Graylog is currently using. Most likely, it's the Default index set that you are interested in deleting, as these are the log files that your clients logs are saved to. To filter the output to just show the Default index set log files, run the following command:
curl http://localhost:9200/_cat/indices | grep graylog_
If you are looking to see the creation date of each index file to work out which are the oldest, you can use this command:
curl http://localhost:9200/_cat/indices?h=index,creation.date.string | grep graylog_
Once you've determined which log file(s) you want to remove, delete them with this command:
curl -XDELETE http://localhost:9200/<index name>
I'd recommend rebooting your Graylog server once you've deleted log files so the server can rebuild any indexes that it needs to function correctly:
sudo reboot
References
https://linuxhint.com/elasticsearch-show-index-creation-time/
https://community.graylog.org/t/graylog-filled-my-disk-how-do-i-delete-some-data/6662/2