How To Manage Log Files

Written by Nick Otter.

Contents

Introduction

Big disk space woes come from mismanagement of log files generated by applications. Most Linux distributions (RHEL 8.1 in this case) ship with logrotate which can be configured to compress, delete, move and rename log files over a specific period of time. If log files are not managed correctly the servers will become unresponsive due to disk space filling up (trust me!) and the majority of applications I have ever managed generate a large volume of log files.

Without logrotate it is common practise to create a separate partition or logical volume for a /var/log mount point to isolate any space issues to a single purpose partition.

Requirements

Updated 05/2020
Linux Kernel 5.4 RHEL 8 4.18

Logrotate

logrotate runs via cron. Take a look at the logrotate script on GitHub here if you fancy.

Logrotate configuration files

/etc/logrotate.conf Global configuration file.
/etc/logrotate.d/* Specific configuration files for services.
/var/lib/logrotate/logrotate.status Status logs. Find last date and time of a logs rotation here.
/etc/cron.daily/logrotate Logrotate crontab.
man logrotate See directives that can be used for specific configurations.

Restarting logrotate

Logrotate isn’t wrapped as a daemon or service. It’s just a script. So, no restart is required after any configuration has been updated. The script is run via cron or you can run the application manually.

Running logrotate on command

This can be done so by forcing it in verbose mode…

$ logrotate -vf /etc/logrotate.d/foo

N.B. logrotate -vf /etc/logrotate.conf will run all configurations as opposed to a specific configuration.

Force is required to override the status check the script executes on /var/lib/logrotate/logrotate.status to ensure logs are not rotated outside of configured time period.

Configuration file format

Each custom configuration should be created in separate files at snap in path /etc/logrotate.d/.

/etc/logrotate.d/foo

# global directives for all configurations in this file go here
compress

# local directives that will be executed by logrotate on files at path /var/log/foo.log go within braces
/var/log/foo.log {
}

Configuration options

Directives table. For full list of directives that can be used in configurations see the man page.

compress Logs are compressed with gzip after they are rotated.
copy Make a copy of the log file with no change to the original.
create Create duplicate log file at path immediately after rotation. This is set by default.
nocreate New log files are not created (this overrides the create option).
size Log files are rotated only when they grow bigger than a determined size (e.g. 100K 10M 1G extensions accepted). Exclusive to time intervals.
nomissingok Issues an error if log file doesn’t exist.
daily weekly monthly Time speicifcations for rotation. So, monthly rotate 3 would mean rotate the last 3 months of logs, archive the rest.
pretotate postrotate endscript Shell commands that will be executed before (prerotate) or after (postrotate) rotation.
rotate N Amount of log files specified at that path that are kept. So with the daily flag set and rotate 7 that would mean keep the last 7 days worth of log files on this rotation and any others outisde of that time period will be removed.
maxage N Archive log files that are older than N days.

N.B.

It’s important to note that the rotate directive is not a lookup on log files system access or creation date. But it is a lookup on when logrotate has last been run.

Examples

So, we have an application named foo which appends to a log file all day long at path /var/log/foo.log.

[root@aws-ec2-1 log]# ls -l --block-size=M | grep foo
-rw-r--r--. 1 root   root   2M Sep 16 11:40 foo.log

This file is constantly growing, so let’s create a custom configuration to manage this.

/etc/logrotate.d/foo

/var/log/foo.log {
   compress
   rotate 1
   create
   dateext 
}

So.. this configuration should create an identical new file, compress the current log and append today’s date to the compressed file name at path /var/log/foo.log. The descriptor rotate will ensure no data will be overwritten - a new compression (‘rotation’) will occur of the original log file everytime.

Let’s run it and find out.

$ logrotate -vf /etc/logrotate.d/foo

[root@aws-ec2-1 log]# ls -l --block-size=M | grep foo
-rw-r--r--. 1 root   root   2M Sep 16 11:50 foo.log
[root@aws-ec2-1 log]# ls -l --block-size=M | grep foo
-rw-r--r--. 1 root   root   0M Sep 16 11:50 foo.log
-rw-r--r--. 1 root   root   1M Sep 16 11:50 foo.log-20200916.gz

Great. And how about only compressing if the log file’s size is greater than 3 Megabytes? A bit hacky and not much rotating going on but this will work for that.

/etc/logrotate.d/foo

/var/log/foo.log {
   compress
   create
   dateext 
   size 3M
}

And if we want to compress but not create a new log file everytime, we can just use nocreate. Neat.

Thanks. This was written by Nick Otter.