Log Courier is a lightweight tool created to ship log files speedily and securely, with low resource usage, to remote Logstash instances. The project is an enhanced fork of Logstash Forwarder 0.3.1 with many fixes and behavioural improvements.
Table of Contents generated with DocToc
- Main Features
- Differences to Logstash Forwarder
- Public Repositories
- Building from Source
- Logstash Integration
- Generating Certificates
- Documentation
- Read events from a file or over a Unix pipeline
- Follow log file rotations and movements
- Close files after inactivity, reopening if they change
- Add extra fields to events prior to shipping
- Reload configuration without restarting
- Ship events securely using TLS with server (and optionally client) certificate verification
- Ship events securely to multiple Logstash instances using ZeroMQ with Curve security (requires ZeroMQ 4+)
- Ship events in plaintext using TCP
- Ship events in plaintext using ZeroMQ (requires ZeroMQ 3+)
- Monitor shipping speed and status with the Administration utility
- Pre-process events using codecs (e.g. Multiline, Filter)
- Logstash Integration with an input and output plugin
- Very low resource usage
Log Courier is an enhanced fork of Logstash Forwarder 0.3.1 with many fixes and behavioural improvements. The primary changes are:
- The publisher protocol is rewritten to avoid many causes of "i/o timeout" which would result in duplicate events sent to Logstash
- The prospector and registrar are heavily revamped to handle log rotations and movements far more reliably, and to report errors cleanly
- The harvester is improved to retry if an error occurred rather than stop
- The configuration can be reloaded without restarting
- An administration tool is available which can display the shipping speed and status of all watched log files
- Fields configurations can contain arrays and dictionaries, not just strings
- Codec support is available which allows multiline processing at the sender side
- A TCP transport is available which removes the requirement for SSL certificates
- There is support for client SSL certificate verification
- Peer IP address and certificate DN can be added to received events in Logstash to distinguish events send from different instances
- Windows: Log files are not locked allowing log rotation to occur
- Windows: Log rotation is detected correctly
The Log Courier repository depends on the EPEL repository which can be
installed automatically on CentOS distributions by running
yum install epel-release. For other distributions, please follow the
installation instructions on the
EPEL homepage.
To install the Log Courier repository, download the corresponding .repo
configuration file below, and place it in /etc/yum.repos.d. Log Courier may
then be installed using yum install log-courier.
- CentOS/RedHat 6.x: driskell-log-courier-epel-6.repo
- CentOS/RedHat 7.x: driskell-log-courier-epel-7.repo
Once installed, modify the configuration file at
/etc/log-courier/log-courier.conf to suit your needs, then start the Log
Courier service to begin shipping.
service log-courier start
A Debian/Ubuntu compatible PPA repository is under consideration. At the moment, no such repository exists.
Requirements:
Linux/Unix: Most requirements can usually be installed by your favourite
package manager.
OS X: Git and GNU make are provided automatically by XCode.
Windows: Git and GNU make are available as Cygwin packages.
To build the binaries, simply run make as follows.
git clone https://github.com/driskell/log-courier
cd log-courier
make
The log-courier program can then be found in the 'bin' folder. Service scripts for various platforms can be found in the contrib/initscripts folder, or it can be run on the command line:
bin/log-courier -config /path/to/config.conf
Note: If you receive errors whilst running make, try gmake instead.
Log Courier communicates with Logstash via an input plugin called "courier".
You may install the plugin using the Logstash 1.5 Plugin manager. Run the following as the user Logstash was installed with.
cd /path/to/logstash
bin/plugin install logstash-input-log-courier
Detailed instructions, including integration with Logstash 1.4.x, can be found on the Logstash Integration page.
Note: If you receive a Plugin Conflict error, try updating the zeromq output
plugin first using bin/plugin update logstash-output-zeromq
Log Courier provides a commands to help generate SSL certificates: lc-tlscert.
This utility is also bundled with the packaged versions of Log Courier, and
should be immediately available at the command-line.
When building from source, running make selfsigned will automatically build
and run the lc-tlscert utility that can quickly and easily generate a
self-signed certificate, along with the corresponding configuration snippets,
for the 'tls' transport.