diff --git a/README.md b/README.md index 629cb8d1..6f62f91c 100644 --- a/README.md +++ b/README.md @@ -46,19 +46,28 @@ audience are still missing. This application is intended to be used as a HTTP endpoint that runs alongside an EZproxy instance. This endpoint receives webhook requests from a monitoring system (Splunk as of this writing), disables the user account identified by -the monitoring system rules and generates one or more notifications listing -the action taken. fail2ban is used to ban the offending IP for `MaxLifetime` -minutes (EZproxy setting) + a small buffer to force active sessions associated -with the disabled user account to timeout and terminate. +the rules enabled on the monitoring system and generates one or more +notifications listing the action taken. At this point, the associated user +sessions can be optionally (and automatically) terminated using two +approaches: -The combination of stopping new logins for the disabled user account and -timing out existing sessions works around the lack of native support for this -behavior in EZproxy itself. +1. using (not officially documented) EZproxy binary subcommand +1. using the provided fail2ban config files + +If using native termination support, all active user sessions associated with +the reported username will be terminated using the kill subcommand provided by +the official ezproxy binary. The sysadmin will need to provide the path to the +ezproxy and the associated Active Users and Hosts "state" file where sessions +are tracked. + +If installed and configured appropriately, fail2ban can be used to to monitor +the reported users log file and ban the associated IP address for +`MaxLifetime` minutes (EZproxy setting) + a small buffer to force active +sessions associated with the disabled user account to timeout and terminate. The net effect is that reported user accounts are immediately disabled and -existing sessions forced to timeout, at which point compromised accounts can -no longer be used on EZproxy until manually removed from the disabled users -file. +compromised accounts can no longer be used with EZproxy until manually removed +from the disabled users file. **NOTE:** This application has not been designed to identify user accounts directly, but rather relies on other systems (currently limited to Splunk) to @@ -75,6 +84,16 @@ See also: - Highly configurable (with more configuration choices to be exposed in the future) +- Optional automatic (but not officially documented) termination of user + sessions via official EZproxy binary + +- `es` CLI application + - small CLI app to list and optionally terminate user sessions for a + specific username + - intended for quick troubleshooting or as an optional replacement for + logging into the admin UI to terminate user sessions for a specific + username + - Supports configuration settings from multiple sources - command-line flags - environment variables @@ -106,19 +125,27 @@ See also: - due to IP Address inclusion in ignore file for IP Addresses - username disabled -- `contrib` files/content provided to allow for spinning up a [demo - environment](docs/demo.md) in order to provide a hands-on sense of what this - project can do - - `fail2ban` - - `postfix` - - `docker` - - `Maildev` container - - `brick` - - `rsyslog` - - `systemd` - - sample JSON payloads for use with `curl` or other http/API clients - - [demo environment](docs/demo.md) doc - - slides from group presentation/demo +- `contrib` files/content + - intended solely for demo purposes + - `postfix` + - `docker` + - `Maildev` container + - sample JSON payloads for use with `curl` or other http/API clients + - [demo environment](docs/demo.md) doc + - slides from group presentation/demo + - shell scripts to setup test/demo environment + - intended for demo *and* as a template for production use + - `fail2ban` + - `brick` + - `rsyslog` + - `systemd` + +The `contrib` content is provided both to allow for spinning up a [demo +environment](docs/demo.md) in order to provide a hands-on sense of what this +project can do and (at least some of the files) to use as a template for a +production installation (e.g., the `fail2ban` config files). At some point we +hope to provide one or more Ansible playbooks (GH-29) to replace the shell +scripts currently used by this project for setting up a test/demo environment. ### Missing @@ -143,7 +170,7 @@ Known issues: See the [`CHANGELOG.md`](CHANGELOG.md) file for the changes associated with each release of this application. Changes that have been merged to `master`, -but not yet an official release may also be noted in the file under the +but not yet in an official release may also be noted in the file under the `Unreleased` section. A helpful link to the Git commit history since the last official release is also provided for further review. diff --git a/doc.go b/doc.go index 53323be9..53645fde 100644 --- a/doc.go +++ b/doc.go @@ -15,19 +15,28 @@ PURPOSE This application is intended to be used as a HTTP endpoint that runs alongside an EZproxy instance. This endpoint receives webhook requests from a monitoring system (Splunk as of this writing), disables the user account identified by -the monitoring system rules and generates one or more notifications listing -the action taken. fail2ban is used to ban the offending IP for `MaxLifetime` -minutes (EZproxy setting) + a small buffer to force active sessions associated -with the disabled user account to timeout and terminate. +the rules enabled on the monitoring system and generates one or more +notifications listing the action taken. At this point, the associated user +sessions can be optionally (and automatically) terminated using two +approaches: -The combination of stopping new logins for the disabled user account and -timing out existing sessions works around the lack of native support for this -behavior in EZproxy itself. +(1) using (not officially documented) EZproxy binary subcommand +(2) using the provided fail2ban config files + +If using native termination support, all active user sessions associated with +the reported username will be terminated using the kill subcommand provided by +the official ezproxy binary. The sysadmin will need to provide the path to the +ezproxy and the associated Active Users and Hosts "state" file where sessions +are tracked. + +If installed and configured appropriately, fail2ban can be used to to monitor +the reported users log file and ban the associated IP address for +`MaxLifetime` minutes (EZproxy setting) + a small buffer to force active +sessions associated with the disabled user account to timeout and terminate. The net effect is that reported user accounts are immediately disabled and -existing sessions forced to timeout, at which point compromised accounts can -no longer be used on EZproxy until manually removed from the disabled users -file. +compromised accounts can no longer be used with EZproxy until manually removed +from the disabled users file. NOTE: This application has not been designed to identify user accounts directly, but rather relies on other systems (currently limited to Splunk) to @@ -49,58 +58,12 @@ FEATURES • Logging of all events (e.g., payload receipt, action taken due to payload) - +• Optional automatic (but not officially documented) termination of user sessions via official EZproxy binary USAGE -Help output is below. See the README for examples. - -$ ./brick -h - -Automatically disable EZproxy users via webhook requests - -brick x.y.z -https://github.com/atc0005/brick - - -Usage: brick [--port PORT] [--ip-address IP-ADDRESS] [--log-level LOG-LEVEL] [--log-output LOG-OUTPUT] [--log-format LOG-FORMAT] [--disabled-users-file DISABLED-USERS-FILE] [--disabled-users-entry-suffix DISABLED-USERS-ENTRY-SUFFIX] [--disabled-users-file-perms DISABLED-USERS-FILE-PERMS] [--reported-users-log-file REPORTED-USERS-LOG-FILE] [--reported-users-log-file-perms REPORTED-USERS-LOG-FILE-PERMS] [--ignored-users-file IGNORED-USERS-FILE] [--ignored-ips-file IGNORED-IPS-FILE] [--teams-webhook-url TEAMS-WEBHOOK-URL] [--teams-notify-delay TEAMS-NOTIFY-DELAY] [--teams-notify-retries TEAMS-NOTIFY-RETRIES] [--ignore-lookup-errors] [--config-file CONFIG-FILE] - -Options: - --port PORT TCP port that this application should listen on for incoming HTTP requests. - --ip-address IP-ADDRESS - Local IP Address that this application should listen on for incoming HTTP requests. - --log-level LOG-LEVEL - Log message priority filter. Log messages with a lower level are ignored. - --log-output LOG-OUTPUT - Log messages are written to this output target. - --log-format LOG-FORMAT - Log messages are written in this format. - --disabled-users-file DISABLED-USERS-FILE - fully-qualified path to the EZproxy include file where this application should write disabled user accounts. - --disabled-users-entry-suffix DISABLED-USERS-ENTRY-SUFFIX - The string that is appended after every username added to the disabled users file in order to deny login access. - --disabled-users-file-perms DISABLED-USERS-FILE-PERMS - Desired file permissions when this file is created. Note: The ezproxy daemon will need to be able to read this file. - --reported-users-log-file REPORTED-USERS-LOG-FILE - Fully-qualified path to the log file where this application should log user disable request events for fail2ban to ingest. - --reported-users-log-file-perms REPORTED-USERS-LOG-FILE-PERMS - Desired file permissions when this file is created. Note: fail2ban will need to be able to read this file. - --ignored-users-file IGNORED-USERS-FILE - Fully-qualified path to the file containing a list of user accounts which should not be disabled and whose IP Address reported in the same alert should not be disabled by this application. Leading and trailing whitespace per line is ignored. - --ignored-ips-file IGNORED-IPS-FILE - Fully-qualified path to the file containing a list of individual IP Addresses which should not be disabled and which user account reported in the same alert should not be disabled by this application. Leading and trailing whitespace per line is ignored. - --teams-webhook-url TEAMS-WEBHOOK-URL - The Webhook URL provided by a preconfigured Connector. If specified, this application will attempt to send client request details to the Microsoft Teams channel associated with the webhook URL. - --teams-notify-delay TEAMS-NOTIFY-DELAY - The number of seconds to wait between Microsoft Teams message delivery attempts. - --teams-notify-retries TEAMS-NOTIFY-RETRIES - The number of attempts that this application will make to deliver Microsoft Teams messages before giving up. - --ignore-lookup-errors - Whether application should continue if attempts to lookup existing disabled or ignored status for a username or IP Address fail. - --config-file CONFIG-FILE - Full path to optional TOML-formatted configuration file. See contrib/brick/config.example.toml for a starter template. - --help, -h display this help and exit - --version display version and exit +See the README for examples. + */ package main diff --git a/docs/deploy.md b/docs/deploy.md index c555e183..c85a0654 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -144,7 +144,8 @@ docs did). Depending on your target environment, you'll either need the 32-bit or 64-bit version of the binary that you generated earlier by following the [build -instructions](build.md). +instructions](build.md). Replace the `v0.1.0-0-g721e6d2` pattern below with +the latest available stable version. | If you see this `uname -m` output | Use the filename with this pattern | Your EZProxy server has this architecture | | --------------------------------- | ------------------------------------- | ----------------------------------------- | @@ -153,8 +154,7 @@ instructions](build.md). For example, if you run `uname -m` on your EZproxy server and get `x86_64` as the output, you will want to deploy the `brick-v0.1.0-0-g721e6d2-linux-amd64` -binary. Replace the `v0.1.0-0-g721e6d2` pattern with the latest available -stable version. +binary. #### Deploying the binary @@ -177,6 +177,9 @@ version. 1. Copy the starter/template configuration file from `contrib/brick/config.example.toml` and modify accordingly using the [configuration](configure.md) guide. +1. Decide whether you will enable automatic sessions termination or use + `fail2ban`. See the [fail2ban](fail2ban.md) doc and the + [configuration](configure.md) guide for more information. 1. Set a Microsoft Teams webhook URL to enable Teams channel notifications. - Skip this step if you don't use Microsoft Teams. 1. Copy the starter/template "ignore" files and modify accordingly diff --git a/docs/ezproxy.md b/docs/ezproxy.md index abd1cadb..ae92a78d 100644 --- a/docs/ezproxy.md +++ b/docs/ezproxy.md @@ -22,9 +22,19 @@ user accounts using an authentication method different from the one users are normally authenticated with. This allows user accounts to authenticate using LDAP, but be explicitly disabled using flat-files. -As of April 2020 (v7.0.16), the OCLC Support team has confirmed that EZproxy -does not support automatic/programmatic termination of active sessions. This -means two things: +When asked, the OCLC Support team confirmed that EZproxy (April 2020, v7.0.16) +did not officially support automatic/programmatic termination of active +sessions. I later learned otherwise (GH-13, GH-31) and was told (paraphrasing) +that while it may work now, the feature I discovered was not officially +supported. I took this to mean that the feature could be pulled (or simply +stop working) in the future. + +`brick` v0.2.0 added optional support for automatic user sessions termination. +See the [configure](configure.md) doc and the main [README](../README.md) file +for additional information for this feature. If enabled, `fail2ban` becomes an +optional layer in abuse control. + +If automatic/native user sessions termination is not enabled: - abusive user login sessions continue to be active until they are manually terminated or timeout @@ -37,12 +47,12 @@ currently supports. This is accomplished by adding user accounts to a specific file, one per line, with an explicit `::deny` suffix per user account entry. When those user accounts attempt to login again they will be denied access. -To handle terminating active sessions, this application relies upon a local +To handle timing out active sessions, this application relies upon a local installation of `fail2ban` which runs alongside an EZproxy instance and this -application. +application. See the [fail2ban](fail2ban.md) document for settings specific to +`fail2ban`. -This document covers specific settings used by EZproxy. See the -[fail2ban](fail2ban.md) document for settings specific to `fail2ban`. +This document covers specific settings used by EZproxy. ## Settings @@ -82,7 +92,7 @@ EZproxy honors it and denies login access to the user account when the next login attempt occurs. As already noted, active sessions must be manually terminated or timed out in order to interrupt access. -By default, this application is configured to write entries to +By default, this application is [configured](configure.md) to write entries to `/var/cache/brick/users.brick-disabled.txt`. This can be overridden by command-line flag (e.g., systemd unit file) or configuration file (usually located at `/usr/local/etc/brick/config.toml`). To configure EZproxy to use diff --git a/docs/fail2ban.md b/docs/fail2ban.md index c9ceda93..20347bfe 100644 --- a/docs/fail2ban.md +++ b/docs/fail2ban.md @@ -26,9 +26,13 @@ terminate. New Splunk alerts should trigger causing further IP Addresses to be logged (and banned) until all (or nearly all) sessions for IP Addresses associated with the disabled user account are terminated. -`fail2ban` is in a lot of ways the "missing piece" for this application. -Without `fail2ban`, `brick` serves the purpose by helping control new abuse, -but is otherwise (at this time) unable to reliably stop *active* abuse. +**NOTE**: As of v0.2.0, `brick` has the ability to perform session termination +using the official `ezproxy` binary's `kill` subcommand. This support can be +used alongside `fail2ban` or in place of it. + +The directions in this doc assume that you wish to either use both methods to +control user sessions *or* that you will use `fail2ban` exclusively for this +purpose. ## Directions @@ -38,8 +42,7 @@ but is otherwise (at this time) unable to reliably stop *active* abuse. - substitute with the appropriate package manager for your Linux distribution (e.g., replace `apt-get` with `yum`). - your EZproxy server is *not* a Windows system. - - Note: Due to `brick` relying on `fail2ban`, Windows is not a supported - platform at this time. + - Note: Windows is not a supported platform at this time. - your EZproxy server does not already have `fail2ban` installed - you are going to test these (and all other setup instructions) in a test environment *first* before deploying to production diff --git a/docs/rsyslog.md b/docs/rsyslog.md index 3eb1a549..b9109351 100644 --- a/docs/rsyslog.md +++ b/docs/rsyslog.md @@ -29,12 +29,13 @@ messages destined for our remote log server(s). You may need to disable this directive if you encounter problems with messages not forwarding as expected in your environment. -Tangent: If you're not already forwarding/centralizing your server log -messages, you should consider doing so. Having log messages centralized in one -searchable location (e.g., Graylog, Splunk, ...) makes -troubleshooting/monitoring much more effective and with lower cost -alternatives such as Graylog, centralized log management becomes approachable -for even lightly staffed IT teams. +Tangent: + +If you're not already forwarding/centralizing your server log messages, you +should consider doing so. Having log messages centralized in one searchable +location (e.g., Graylog, Splunk, ...) makes troubleshooting much more +effective. With lower cost alternatives such as Graylog, centralized log +management becomes approachable for even lightly staffed IT teams. ## Log files diff --git a/docs/splunk.md b/docs/splunk.md index b82d367a..80b8e726 100644 --- a/docs/splunk.md +++ b/docs/splunk.md @@ -18,10 +18,10 @@ If drawn on a whiteboard, Splunk is likely somewhere off to the side. EZproxy is in the center, `fail2ban` and `brick` are nearby (or in the same sphere) and other resources (such as maybe Microsoft Teams and email) are off -somewhere off to another side. This quick sketch for illustration wouldn't +somewhere to another side. This quick sketch for illustration wouldn't tell the whole picture however, and would greatly undervalue the role Splunk plays in resource abuse "management" (it's hard to say with a straight face -that it can ever be completely prevent). +that it can ever be completely prevented). By taking the time to implement and refine alerts, Splunk will become an invaluable tool to monitor and report abusive activity to your sysadmin team @@ -37,7 +37,8 @@ Once you have created, tested and refined one or more email-based alerts, you are ready to begin using webhook payloads to report problematic user accounts to `brick`. -1. Review the [official documentation](references.md) for setting up a "webhook alert action" +1. Review the [official documentation](references.md) for setting up a + "webhook alert action" 1. Follow those instructions and set the target webhook URL 1. we'll assume that your EZproxy server has a FQDN of ezproxy.example.com and is normally accessible at @@ -47,10 +48,11 @@ to `brick`. 1. As noted in the [deploy](deploy.md) doc, make sure you have a firewall rule in place to limit payload delivery to the `disable` endpoint to only your Splunk server and any trusted SysAdmin / IT Support team members -1. If you haven't already done so, [build](build.md) and [deploy](deploy.md) - the `brick` application -1. The same goes for `fail2ban`, if you haven't yet, install and configure - [fail2ban](fail2ban.md) +1. If you haven't already done so, [build](build.md), [deploy](deploy.md) and + [configure](configure.md) the `brick` application +1. The same goes for `fail2ban`, if you haven't yet, install and + [configure](configure.md) [fail2ban](fail2ban.md) *or* exclusively use + EZproxy's native (unofficial) support for terminating user sessions. 1. Test! ## Payload schema / format @@ -158,8 +160,8 @@ application: ``` `brick` parses and uses select fields from that payload for its work. As time -permits we will refine further to exclude unwanted fields and bring in new -ones. +permits, we hope to further refine the delivered payload to exclude unwanted +fields and bring in new ones. Files: diff --git a/docs/start-here.md b/docs/start-here.md index e51c351a..0d85c52a 100644 --- a/docs/start-here.md +++ b/docs/start-here.md @@ -10,6 +10,8 @@ - [Requirements / Dependencies](#requirements--dependencies) - [Configuration](#configuration) - [Behavior](#behavior) + - [If using EZproxy's native (unofficial) session termination support](#if-using-ezproxys-native-unofficial-session-termination-support) + - [If using fail2ban to timeout sessions](#if-using-fail2ban-to-timeout-sessions) - [Notifications](#notifications) ## Problem Scenario @@ -29,18 +31,18 @@ around to manually do so. `brick` is one answer to that problem. ## Requirements / Dependencies -| Application | brick's role | Application's role | (Potential) Alternatives | -| ----------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------ | -| Splunk | brick accepts Splunk payloads, processes them (including generating notifications) | Splunk provides payloads that match specified alert criteria | Graylog | -| fail2ban | brick logs alert receipt, logs actions taken (including disable and ignore actions) | fail2ban uses brick's disable action log messages to ban/unban IP Addresses associated with a disable request | | -| EZproxy | brick maintains disabled users flat-file | EZproxy monitors brick's generated disabled users flat-file to deny login access to specified user accounts | | +| Application | brick's role | Application's role | (Potential) Alternatives | +| ------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | +| Splunk | brick accepts Splunk payloads, processes them (including generating notifications) | Splunk provides payloads that match specified alert criteria | Graylog | +| fail2ban (optional) | brick logs alert receipt, logs actions taken (including disable and ignore actions) | fail2ban uses brick's disable action log messages to ban/unban IP Addresses associated with a disable request | Built-in/native session termination support | +| EZproxy | brick maintains disabled users flat-file | EZproxy monitors brick's generated disabled users flat-file to deny login access to specified user accounts | | ## Configuration 1. The `brick` application is installed on our EZproxy server and runs as a separate service -1. The `fail2ban` application is installed on our EZproxy server and runs as a - separate service +1. The `fail2ban` application is (optionally) installed on our EZproxy server + and runs as a separate service 1. `EZproxy` is configured to look at a new, automatically-maintained disabled users file 1. Splunk alerts are configured & enabled @@ -49,6 +51,37 @@ around to manually do so. `brick` is one answer to that problem. ## Behavior +As of v0.2.0, `brick` supports using one or both of EZproxy's native +(unofficial) session termination support and `fail2ban` to force sessions to +timeout. The sysadmin deploying `brick` can choose one or both of the options. +The behavior for these choices is noted below. + +### If using EZproxy's native (unofficial) session termination support + +In this scenario the native session termination support is enabled and +`fail2ban` is either not installed or not configured to monitor the report +users log file. + +1. Splunk sends alert based on time-tested thresholds +1. `brick` receives the alert, logs it, evaluates it (based on ignore + lists, etc), takes an action, sends notifications + - If not asked to ignore a username or IP Address, `brick` + 1. adds the username to a local "disabled users" flat-file that EZproxy + is configured to monitor + 1. logs that action + - If ignoring a username or IP Address + - logs that the username or IP Address was ignored +1. EZproxy sees that the user account is disabled and denies future logins, + but refrains from terminating existing sessions +1. `brick` calls `/fully/qualified/path/to/ezproxy kill SESSION_ID` for each + active user session for the reported username +1. EZproxy sessions for the disabled user account are terminated + +### If using fail2ban to timeout sessions + +In this scenario the native session termination support is *not* enabled and +`fail2ban` is used exclusively to halt abusive user accounts. + 1. Splunk sends alert based on time-tested thresholds 1. `brick` receives the alert, logs it, evaluates it (based on ignore lists, etc), takes an action, sends notifications