GrabDuck

Configuring centralized logging from text log files in Unix (remote_syslog2) · ...

:

Use remote_syslog2 to aggregate logs from any text file, like app log files, in Unix and BSD.

Why remote_syslog2

Papertrail reads log messages from applications and server daemons which do not support - or even know about - syslog.

To send log files from those applications to Papertrail, run Papertrail’s tiny standalone remote_syslog2 daemon. It tracks one or more log files and sends new entries to Papertrail in realtime.

remote_syslog2 works for any text log files, has no impact to the daemon or its logging configuration, and is easy to setup. It forwards logs directly to Papertrail, without relying on the system syslog daemon. No adjustments to syslog.conf, rsyslog.conf, or syslog-ng.conf are required.

Note: Some apps, such as MySQL, Apache, and Tomcat (log4j), have internal support for logging directly to syslog. For apps like these, you may use its internal syslog support or use remote_syslog2 as described below.

Download the current release. To extract it and copy the binary into a system path, run:

$ tar xzf ./remote_syslog*.tar.gz
$ cd remote_syslog
$ sudo cp ./remote_syslog /usr/local/bin

RPM and Debian packages also available.

2. Configure

Paths to log file(s) can be specified on the command-line, or save log_files.yml.example as /etc/log_files.yml. Edit it to define:

  • path to this app’s log file, and any other log file(s) to watch.
  • destination host and port provided under log destinations. If no destination port was provided, set host to logs.papertrailapp.com and remove the port config line to use the default (514).

3. Start

Start the daemon:

Logs should appear in Papertrail within a few seconds of being written to the on-disk log file. Problem? See Troubleshooting.

remote_syslog requires read permission on the log files it is monitoring.

4. Auto-start

remote_syslog2 can be automated to start at boot using init scripts (examples) or your preferred daemon invocation method, such as monit or god. See remote_syslog --help or the full README on GitHub.


Note: remote_syslog2 is daemonized as remote_syslog. When you’re looking for the process, look for remote_syslog (not remote_syslog2).

Operations

The system rebooted and remote_syslog didn’t start

Install an init file.

Logs not appearing?

First, please feel free to drop a mail into support@papertrailapp.com, either instead of following these instructions or while trying them. We enjoy helping, and these steps are only here if they let you save time by troubleshooting independently.

Second, there’s a few reasons remote_syslog might not be sending logs. In order:

  • Is it running? Check with ps auxww | grep [r]emote_syslog. Exactly 1 process should be shown, like this:

    root 24501 0.0 0.4 13952 8864 ? S Mar01 2:45 remote_syslog

  • In the example above, the process is running as user root. If a user other than root is shown, does that user running remote_syslog have permission to write to /var/run/? /var/run/ is the default location for PID files (background)?

If remote_syslog isn’t already running as root, try running it as root to test (such as with sudo), or specify an alternate location for the PID file (such as with remote_syslog --pid-file=/tmp/some.pid).

  • If you can start remote_syslog but it subsequently stops running, try leaving it attached to the terminal (rather than daemonizing): remote_syslog -D

  • Was it working for a day or a week, then stopped overnight (perhaps across many systems at the same time)? This could be a firewall policy change or log rotation. However, remote_syslog plays well with all common log rotation systems (with no changes to the configuration of either), so we’d like to hear about this problem.

  • Is remote_syslog monitoring log files that are stored on an NFS share? If so, enable polling via the --poll switch.

  • Detailed troubleshooting.

The next step is to see what remote_syslog is actually doing. Typically that’s with strace, such as:

strace -tt -s 500 -fp 12345

.. where 12345 is the process ID of remote_syslog, obtained from the second column of ps auxww. This will output every call that it makes. We suggest sending the strace output to a file:

strace -tt -s 500 -fp 12345 -o strace.log

Feel free to send the strace.log file to us via support@papertrailapp.com. We’ll look for a few things. Did the OS notify remote_syslog of writes to the log files? Did remote_syslog call sendto() (UDP) or connect() and send() (TCP) to try sending the message? What happened?

See other options with:

See also Troubleshooting reachability

Can we help? Just ask.