Skip to content

Commit

Permalink
Merge pull request #17 from danielme85/dev
Browse files Browse the repository at this point in the history
Add log-cleaner-upper
  • Loading branch information
danielme85 authored Feb 5, 2020
2 parents cc2b3ae + a0f49a6 commit 431d320
Show file tree
Hide file tree
Showing 9 changed files with 291 additions and 81 deletions.
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ services:
- mysql
- mongodb
before_script:
- pecl install mongodb
- pecl install -f mongodb
- mysql -e 'CREATE DATABASE testing;'
- composer self-update
- composer install --no-interaction
Expand Down
71 changes: 44 additions & 27 deletions readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,15 @@
Custom Laravel 6 and >=5.6 Log channel handler that can store log events to SQL or MongoDB databases.
Uses Laravel native logging functionality.


* [Installation](#installation)
* [Configuration](#configuration)
* [Usage](#usage)
* [Fetching Logs](#fetching-logs)
* [Log Cleanup](#log-cleanup)
* [Processors](#processors)


## Installation
Use the composer require or add to composer.json.
```
Expand All @@ -29,7 +38,7 @@ You will need to add an array under 'channels' for Log-to-DB here like so:
'stack' => [
'name' => 'Log Stack',
'driver' => 'stack',
'channels' => ['database', 'mongodb'],
'channels' => ['database', 'file'],
],

'database' => [
Expand All @@ -43,6 +52,8 @@ You will need to add an array under 'channels' for Log-to-DB here like so:
'queue' => false,
'queue_name' => '',
'queue_connection' => '',
'max_records' => false,
'max_hours' => false,
'processors' => [
//Monolog\Processor\HostnameProcessor::class
]
Expand All @@ -65,39 +76,31 @@ More info about some of these options: https://laravel.com/docs/5.6/logging#cust
There are some default settings and more information about configuring the logger in the 'logtodb.php' config file.
This could be copied to your project if you would like edit it with the vendor publish command.
```
php artisan vendor:publish
php artisan vendor:publish --provider="danielme85\LaravelLogToDB\ServiceProvider"
```
You can also set default log-to-db config settings in your .env file, for ex:
You can also change these settings in your env file.
```
LOG_DB_CONNECTION='default'
LOG_DB_DETAILED=false
LOG_DB_MAX=100
LOG_DB_QUEUE=false
LOG_DB_QUEUE_NAME='logToDBQueue'
LOG_DB_QUEUE_CONNECTION='default'
## Usage
Use the default Laravel Facade "Log"
```php
Log::channel()->info("This thing just happened");
Log::channel()->warning("This kind of bad thing happened...");
```
You can give the logging channels whatever name you want instead of: 'database', as well as the log levels.
The naming can be used later if you want to send a Log event to a specific channel:
```php
Log::channel('database')->info("This thing just happened");
Log::channel('mongodb')->info("This thing just happened");
LOG_DB_MAX_COUNT=false
LOG_DB_MAX_HOURS=false
```
This logger works the same as any other across Laravel, for example you can add it to a stack.
You can log multiple levels to multiple DB connections... the possibilities are ENDLESS! 😎

#### Config priority order
Lowest number has highest priority (overrides the one below);
1. Log Channel config array in config/logging.php overrides logtodb.php
2. .env File overrides config/logtodb.php file
3. config/logtodb.php is the default config.
There are three places you can change different options when using log-to-db:
1. The config file: config/logtodb.php (after doing vendor:publish).
2. Your .env file will override settings in the logtodb.php config file.
3. The Laravel logging config file: config/logging.php. You need to add a custom array here as mentioned above,
in this same array you can specify/override config settings specifically for that log channel.

### Log Worker Queue
Config values set in point 1 & 2 would work as default for all new log channels you add in the "channels" array for the
Laravel logging configuration (config/logging.php).

#### Log Worker Queue
It might be a good idea to save the log events with a Queue Worker. This way your server does not have to wait for
the save process to finish. You would have to configure the Laravel Queue settings and run the Queue listener.
https://laravel.com/docs/5.6/queues#running-the-queue-worker
Expand All @@ -124,7 +127,7 @@ You can also log to specific log channels:
Log::channel('database')debug("This is an test DEBUG log event");


### Fetching Logs
## Fetching Logs
The logging by this channel is done trough the Eloquent Model builder.
LogToDB::model($channel, $connection, $collection);
You can skip all function variables and the default settings from the config/logtodb.php will be used.
Expand Down Expand Up @@ -211,7 +214,19 @@ No migrations needed for MongoDB.

No indexes are added per default, so if you fetch a lot of log results based on specific time ranges or types: it might be a good idea to add some indexes.

#### Log Cleanup
## Log Cleanup
There are config values that you can set to specify the max number of log records to keep, or the max record age in hours.

* logging.php channel array -> (max_records, max_hours).
* .env file -> (LOG_DB_MAX_COUNT, LOG_DB_MAX_HOURS).

<b>These option is set to *false* per default, these have to be set to desired integers before you can run the "log:delete" artisan command.</b>
```
php artisan log:delete
```
This command will delete records based on settings described above. Add this command to your Console/kernel.php, or run manually in cron etc to enable automatic cleanup.

#### Manual Cleanup
There is a helper function to remove the oldest log events and keep a specified number
```php
LogToDB::removeOldestIfMoreThan(100);
Expand All @@ -224,7 +239,7 @@ LogToDB::model()->removeOlderThan('2019-01-01');
LogToDB::model()->removeOlderThan('2019-01-01 23:00:00');
```

#### Processors
## Processors
Monolog ships with a set of [processors](https://github.com/Seldaek/monolog/tree/master/src/Monolog/Processor), these will generate additional data and populate the 'extra' field.

You could also create your own custom processor, make sure they implement [Monolog\Processor\ProcessorInterface](https://github.com/Seldaek/monolog/blob/master/src/Monolog/Processor/ProcessorInterface.php).
Expand All @@ -250,7 +265,7 @@ class PhpVersionProcessor implements ProcessorInterface {

```

#### Advanced /config/logging.php example
## More logging.php config examples
```php
'default' => env('LOG_CHANNEL', 'stack'),

Expand All @@ -269,7 +284,9 @@ class PhpVersionProcessor implements ProcessorInterface {
'detailed' => true,
'queue' => true
'queue_name' => 'logQueue'
'queue_connection' => 'redis'
'queue_connection' => 'redis',
'max_records' => 1000,
'max_hours' => 24,
],

'mongodb' => [
Expand Down
93 changes: 93 additions & 0 deletions src/Commands/LogCleanerUpper.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
<?php
/**
* Created by PhpStorm.
* User: dmellum
* Date: 2/2/20
* Time: 10:38 PM
*/

namespace danielme85\LaravelLogToDB\Commands;

use Carbon\Carbon;
use danielme85\LaravelLogToDB\LogToDbHandler;
use danielme85\LaravelLogToDB\LogToDB;
use Illuminate\Console\Command;

class LogCleanerUpper extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'log:delete';

/**
* The console command description.
*
* @var string
*/
protected $description = 'Cleanup/delete/prune/trim log records.';

/**
* Create a new command instance.
*
* @return void
*/
public function __construct()
{
parent::__construct();
}

/**
* Execute the console command.
*
*/
public function handle()
{
$config = config('logtodb');
$channels = $this->getLogToDbChannels();

if (!empty($channels)) {
foreach ($channels as $channel) {
$maxRecords = $channel['max_records'] ?? $config['purge_log_when_max_records'] ?? false;
$maxHours = $channel['max_hours'] ?? $config['purge_log_when_max_records'] ?? false;
$connection = $channel['connection'] ?? 'default';
$collection = $channel['collection'] ?? 'log';

//delete based on numbers of records
if (!empty($maxRecords) && $maxRecords > 0) {
if (LogToDB::model($connection, $collection)->removeOldestIfMoreThan($maxRecords)) {
$this->warn("Deleted oldest log records on channel: {$connection}->{$collection}, keep max number of records: {$maxRecords}");
}
}

//delete based on age
if (!empty($maxHours) && $maxHours > 0) {
$time = Carbon::now()->subHours($maxHours)->toDateTimeString();
if (LogToDB::model($connection, $collection)->removeOlderThan($time)) {
$this->warn("Deleted log records on channel: {$connection}->{$collection}, older than: {$time}");
}
}
}
}
}

/**
* @return array
*/
private function getLogToDbChannels()
{
$list = [];
$logging = config('logging');
if (!empty($logging) && isset($logging['channels']) && !empty($logging['channels'])) {
foreach ($logging['channels'] as $log) {
if (isset($log['via']) && $log['via'] === LogToDbHandler::class) {
$list[] = $log;
}
}
}

return $list;
}
}
61 changes: 44 additions & 17 deletions src/LogToDB.php
Original file line number Diff line number Diff line change
Expand Up @@ -36,64 +36,85 @@ class LogToDB
public $collection;

/**
* @var mixed
* Use queue driver to save record.
* @var bool
*/
public $saveWithQueue;

/**
* @var mixed
* Name of the queue to use.
* @var string
*/
public $saveWithQueueName;

/**
* @var mixed
* Queue connection to use
* @var string
*/
public $saveWithQueueConnection;

/**
* Purge after number of records
* @var int
*/
private $purgeAfterNumber;

/**
* Purge after number of hours old
* @var int
*/
private $purgeAfterNumberOfHours;

/**
* The DB config details
* @var null
*/
public $database;


/**
* LogToDB constructor.
*
* @param string|null $channelConnection
* @param string|null $collection
* @param bool|null $detailed
* @param bool|null $queue
* @param string|null $queueName
* @param string|null $queueConnection
* @param string $connection The DB connection name to use.
* @param string $collection The name of the collection/table to use.
* @param array $config Array of combined config values.
*/
function __construct(string $channelConnection = null,
string $collection = null,
bool $detailed = null,
bool $queue = null,
string $queueName = null,
string $queueConnection = null)
string $queueConnection = null,
int $purgeAfterNumber = null,
int $purgeAfterNumberOfHours = null)
{
//Log default config if present
$config = config('logtodb');
if (!empty($config)) {
if (isset($config['connection'])) {
if (isset($config['connection']) && !empty($config['connection'])) {
$this->connection = $config['connection'];
}
if (isset($config['collection'])) {
if (isset($config['collection']) && !empty($config['collection'])) {
$this->collection = $config['collection'];
}
if (isset($config['detailed'])) {
if (isset($config['detailed']) && !empty($config['detailed'])) {
$this->detailed = $config['detailed'];
}
if (isset($config['queue_db_saves'])) {
if (isset($config['queue_db_saves']) && !empty($config['queue_db_saves'])) {
$this->saveWithQueue = $config['queue_db_saves'];
}
if (isset($config['queue_db_name'])) {
if (isset($config['queue_db_name']) && !empty($config['queue_db_name'])) {
$this->saveWithQueueName = $config['queue_db_name'];
}
if (isset($config['queue_db_connection'])) {
if (isset($config['queue_db_connection']) && !empty($config['connection'])) {
$this->saveWithQueueConnection = $config['queue_db_connection'];
}
if (isset($config['purge_log_when_max_records']) && !empty($config['purge_log_when_max_records'])) {
$this->purgeAfterNumber = $config['purge_log_when_max_records'];
}
if (isset($config['purge_log_when_max_hours']) && !empty($config['purge_log_when_max_hours'])) {
$this->purgeAfterNumberOfHours = $config['purge_log_when_max_hours'];
}
}

//Set config based on specified config from the Log handler
Expand All @@ -115,6 +136,12 @@ function __construct(string $channelConnection = null,
if (!empty($queueConnection)) {
$this->saveWithQueueConnection = $queueConnection;
}
if (!empty($purgeAfterNumber)) {
$this->purgeAfterNumber = $purgeAfterNumber;
}
if (!empty($purgeAfterNumberOfHours)) {
$this->purgeAfterNumberOfHours = $purgeAfterNumberOfHours;
}

//Get the DB connections
$dbconfig = config('database.connections');
Expand Down Expand Up @@ -232,7 +259,7 @@ public function newFromMonolog(array $record)
dispatch(new SaveNewLogEvent($this, $record))
->onQueue($this->saveWithQueueName);
}
} else {
} else {
$log = CreateLogFromRecord::generate(
$this->connection,
$this->collection,
Expand Down
Loading

0 comments on commit 431d320

Please sign in to comment.