mtail - extract internal monitoring data from application logs for collection into a timeseries database

mtail is a tool for extracting metrics from application logs to be exported into a timeseries database or timeseries calculator for alerting and dashboarding.

It fills a monitoring niche by being the glue between applications that do not export their own internal state (other than via logs) and existing monitoring systems, such that system operators do not need to patch those applications to instrument them or writing custom extraction code for every such application.

The extraction is controlled by mtail programs which define patterns and actions:

# simple line counter
counter lines_total
/$/ {
  lines_total++
}

Metrics are exported for scraping by a collector as JSON or Prometheus format over HTTP, or can be periodically sent to a collectd, StatsD, or Graphite collector socket.

Read the programming guide if you want to learn how to write mtail programs.

Ask general questions on the users mailing list: https://groups.google.com/g/mtail-users

Installation

There are various ways of installing mtail.

Precompiled binaries

Precompiled binaries for released versions are available in the Releases page on Github. Using the latest production release binary is the recommended way of installing mtail.

Windows, OSX and Linux binaries are available.

Building from source

mtail uses bazel for its speed, hermeticity, and support for cross-language compilation. Install bazelisk to manage Bazel for you.

Then:

bazel build //cmd/mtail

See the Build instructions for more details.

The build system can also emit an OCI container image for you.

bazel build //:oci_image

will create it, but to run it the best option is to load it into your local container runtime

bazel run //:load_image

Deployment

mtail works best when paired with a timeseries-based calculator and alerting tool, like Prometheus.

So what you do is you take the metrics from the log files and you bring them down to the monitoring system?

It deals with the instrumentation so the engineers don't have to! It has the extraction skills! It is good at dealing with log files!!

Learn more about interoperability with other tools

Read More

Full documentation at https://jaqx0r.github.io/mtail/

Read more about writing mtail programs:

• Programming Guide
• Language Reference
• Metrics
• Managing internal state
• Testing your programs

Read more about hacking on mtail

• Building from source
• Contributing
• Style

Read more about deploying mtail and your programs in a monitoring environment

• Deploying
• Interoperability with other systems
• Troubleshooting
• FAQ

Getting more help and reporting defects

If you have any questions, please use the GitHub Discussions Q&A.

We also have an email list : https://groups.google.com/forum/#!forum/mtail-users

For any defects please file a new issue.