Create test scenarios with pgbackrest local and remote, using vagrant

.gitignore

@@ -1,2 +1,4 @@
 pgbackrest.conf
-/test/.vagrant/
+/test/.vagrant/
+/packaging/.vagrant/
+*.rpm

CHANGELOG.md

@@ -3,7 +3,8 @@ Changelog
 
 2019-xx-xx v1.6:
 
-  - 
+  - Avoid chdir when scanning a directory to avoid some problems with 
+  `sudo -u` (Christophe Courtois).
 
 2019-03-18 v1.5:
 

INSTALL.md

@@ -1,3 +1,19 @@
+## RPM
+
+To install check_pgbackrest with the provided rpm file:
+
+```
+yum install -y epel-release
+yum install -y nagios-plugins-pgbackrest-*.noarch.rpm
+```
+
+The rpm will also require nagios-plugins to be installed and put the 
+`check_pgbackrest` script there. 
+That's why the epel-release package is needed too.
+
+
+## Manual installation
+
 To handle the json output format of the pgBackRest info command, you need to
 install the following module:
 
@@ -11,13 +27,7 @@ To list archived WALs using SFTP, you need to install the following module:
 - On Debian-like: `libnet-sftp-foreign-perl`
 
 
-To install check_pgbackrest with the provided rpm file:
-
-```
-yum install -y epel-release
-yum install -y nagios-plugins-pgbackrest-*.noarch.rpm
-```
+The Data::Dump perl module is also needed:
 
-The rpm will also require nagios-plugins to be installed and put the 
-`check_pgbackrest` script there. 
-That's why the epel-release package is needed too.
+- On RedHat-like: `perl-Data-Dumper`
+- On Debian-like: `libdata-dump-perl`

RELEASING.md

@@ -12,18 +12,22 @@ In `check_pgbackrest.spec`:
   * update the version in `Version:`
   * edit the changelog
 
+Update the `CHANGELOG.md` file too.
+
+Update the tests results `test/regress/expected/version.out`.
+
 ## Documentation
 
 Generate updated documentation:
 
-```
+```bash
 pod2text check_pgbackrest > README
 podselect check_pgbackrest > README.pod
 ```
 
 ## Tagging and building tar file
 
-```
+```bash
 TAG=REL1_6
 git tag -a $TAG -m "Release $TAG"
 git tag
@@ -52,15 +56,15 @@ git archive --prefix=check_pgbackrest-$TAG/ -o /tmp/check_pgbackrest-1.6.tar.gz
 
 ### Installation
 
-```
+```bash
 yum group install "Development Tools"
 yum install rpmdevtools
 useradd makerpm
 ```
 
 ### Building the package
 
-```
+```bash
 su - makerpm
 rpmdev-setuptree
 git clone https://github.com/dalibo/check_pgbackrest.git
@@ -72,9 +76,18 @@ The RPM is generated into `rpmbuild/RPMS/noarch`.
 
 Don't forget to upload the package on GitHub release page.
 
-### Community
+### Vagrant
+
+A specific Vagrant script has been created in the `packaging` directory to 
+build the rpm. To launch it, go the the `packaging` directory and execute:
+
+```bash
+make all
+```
+
+## Community
 
-## pgsql-announce
+### pgsql-announce
 
 Send a mail to the pgsql-announce mailing list. Eg.:
 

check_pgbackrest

@@ -385,6 +385,7 @@ sub pgbackrest_info {
         $infocmd = $args{'prefix'}." $infocmd";
     }
 
+    dprint("pgBackRest info command was : '$infocmd'\n");
     my $json_output = `$infocmd 2>&1 |grep -v WARN |grep -v ERROR`;
 
     die("Can't get pgBackRest info.\nCommand was '$infocmd'.\n") unless ($? eq 0);
@@ -581,18 +582,18 @@ sub check_wal_archives {
                                     });
         }else{
             find ({ wanted => sub {
-                return unless -f;
-		return unless /$filename_re_full/;
-
-                if ( $args{'ignore-archived-since'} ) {
-                    my $diff_epoch = time() - (stat($File::Find::name))[9];
-                    dprint ("file ".$_." as interval since epoch : ".to_interval($diff_epoch)."\n");
-                    return if ( $diff_epoch <= get_time($args{'ignore-archived-since'}) );
-                }
-                push @filelist, [basename($_), (stat($File::Find::name))[9,7], $File::Find::name];
-                push @filelist_simplified, substr(basename($_), 0, 24);
+                    return unless -f;
+                    return unless /$filename_re_full/;
+
+                    if ( $args{'ignore-archived-since'} ) {
+                        my $diff_epoch = time() - (stat($File::Find::name))[9];
+                        dprint ("file ".$_." as interval since epoch : ".to_interval($diff_epoch)."\n");
+                        return if ( $diff_epoch <= get_time($args{'ignore-archived-since'}) );
+                    }
+                    push @filelist, [basename($_), (stat($File::Find::name))[9,7], $File::Find::name];
+                    push @filelist_simplified, substr(basename($_), 0, 24);
                 }, no_chdir=> 1
-               }, $archives_dir );
+            }, $archives_dir );
         }
 
         return unknown $me, ['no archived WAL found'] unless @filelist;

check_pgbackrest.spec

@@ -14,6 +14,7 @@ BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
 Requires: nagios-plugins
 Requires: perl-JSON
 Requires: perl-Net-SFTP-Foreign
+Requires: perl-Data-Dumper
 Provides: check_pgbackrest = %{version}
 
 %description

packaging/Makefile

@@ -0,0 +1,10 @@
+all: clean_before create_vm clean_after
+
+clean_before:
+	vagrant destroy -f
+
+create_vm:
+	vagrant up
+
+clean_after:
+	vagrant destroy -f

packaging/Vagrantfile

@@ -0,0 +1,29 @@
+vm_prefix = 'c_pgbr_c7'
+
+Vagrant.configure(2) do |config|
+    config.vm.box = 'centos/7'
+
+    # hardware and host settings
+    config.vm.provider 'libvirt' do |lv|
+        lv.cpus = 1
+        lv.memory = 512
+        lv.default_prefix = vm_prefix
+	end
+
+	# don't mind about insecure ssh key
+    config.ssh.insert_key = false
+
+    # don't share the default vagrant folder
+    config.vm.synced_folder ".", "/vagrant", disabled: true
+
+    # mount check_pgbackrest path for testing
+    config.vm.synced_folder "..", "/check_pgbackrest", nfs_udp: false
+
+    # set node name
+    config.vm.define "build-srv" do |node|
+        node.vm.hostname = 'build-rpm-check-pgbackrest'
+    end
+
+    # build check_pgbackrest rpm
+    config.vm.provision 'check_pgbackrest', type: 'shell', path: 'provision/check_pgbackrest.bash'
+end

packaging/provision/check_pgbackrest.bash

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+# build rpm and install check_pgbackrest
+yum install --nogpgcheck --quiet -y -e 0 epel-release
+yum install --nogpgcheck --quiet -y -e 0 rpmdevtools
+rpmdev-setuptree
+spectool -R -g /check_pgbackrest/check_pgbackrest.spec
+rpmbuild -ba /check_pgbackrest/check_pgbackrest.spec
+ls -lr ~/rpmbuild/RPMS/noarch/*
+yum install --nogpgcheck --quiet -y -e 0 ~/rpmbuild/RPMS/noarch/*.rpm
+cp -rf ~/rpmbuild/RPMS/noarch/*.rpm /check_pgbackrest/

test/Makefile

@@ -1,13 +1,17 @@
-all: create_vm pgbackrest_local check_pgbackrest
+s1: clean create_vm pgbackrest_local
+s2: clean create_vm pgbackrest_remote
 
 create_vm:
 	vagrant up
 
 pgbackrest_local:
 	vagrant up --provision-with=pgbackrest_local
+	vagrant up --provision-with=icinga2_local
 
-check_pgbackrest:
-	vagrant up --provision-with=check_pgbackrest
+pgbackrest_remote:
+	vagrant up --provision-with=pgbackrest_remote_primary
+	vagrant up --provision-with=pgbackrest_remote_standby
+	vagrant up --provision-with=icinga2_remote
 
 clean:
 	vagrant destroy -f

test/README.md

@@ -2,22 +2,111 @@
 
 ## Introduction
 
-This `Vagrantfile` is bootstrapping a fresh install with:
-  * pgsql-srv hosting a pgsql cluster
-  * pgBackRest installed and configured to backup and archive locally
+This `Vagrantfile` is bootstrapping 3 possible test cases:
+
+### 1. pgBackRest configured to backup and archive locally
+
+  * `icinga-srv` executes check_pgbackrest by ssh with Icinga 2;
+  * `pgsql-srv` hosting a pgsql cluster with pgBackRest installed;
+  * `backup-srv` not used, clean pgsql cluster running on it.
+
+Backups and archiving are done locally on `pgsql-srv`.
+
+### 2. pgBackRest configured to backup and archive remotely
+
+  * `icinga-srv` executes check_pgbackrest by ssh with Icinga 2;
+  * `pgsql-srv` hosting a pgsql cluster with pgBackRest installed;
+  * `backup-srv` hosting the pgBackRest backups and archives.
+
+Backups of `pgsql-srv` are taken from `backup-srv`. 
+Archives are pushed from `pgsql-srv` to `backup-srv`.
+Checks (retention and archives) are done both locally (on `backup-srv`) and 
+remotely (on `pgsql-srv`). Checks are performed from `icinga-srv` by ssh.
+pgBackRest backups are use to build a Streaming Replication with `backup-srv` 
+as standby server.
 
 ## Testing
 
 The easiest way to start testing is with the included `Makefile`.
 
-_Build and Logon_:
+### Test case 1
+
+_Build_:
 
 ```bash
 cd test
-make all
-vagrant ssh -c "sudo /check_pgbackrest/test/regress/tests.bash"
+make s1
 ```
 
+Expected make time: 6 min.
+
+_Check the results of a manual execution of check_pgbackrest_:
+
+```bash
+vagrant ssh pgsql-srv -c "sudo /check_pgbackrest/test/regress/test-s1.bash"
+```
+
+Expected run time: 15 sec.
+
+_To simulate some activity with pgBackRest_:
+
+```bash
+vagrant ssh pgsql-srv -c "sudo /check_pgbackrest/test/regress/simulate-activity-local.bash"
+```
+
+Modify `SCALE` factor or `ACTIVITY_TIME` in this script to simulate more activity.
+
+### Test case 2
+
+_Build_:
+
+```bash
+cd test
+make s2
+```
+
+Expected make time: 7 min.
+
+_Check the results of a manual execution of check_pgbackrest_:
+
+```bash
+vagrant ssh pgsql-srv -c "sudo /check_pgbackrest/test/regress/test-s2-from-primary.bash"
+```
+
+Expected run time: 30 sec.
+
+### Icinga 2 connectivity
+
+_Check the results of _check_pgbackrest_ launched by Icinga 2:
+
+```bash
+vagrant ssh icinga-srv -c "sudo icingacli monitoring list services --service=pgbackrest* --verbose"
+```
+
+_Navigate to Icinga Web 2_:
+
+Get the IP address of `icinga-srv` with:
+
+```bash
+vagrant ssh icinga-srv -c "ip addr show eth0"
+```
+
+And then go to `http://IP/icingaweb2`. Credentials are `icingaweb / icingaweb`.
+
+### Clean up
+
+Don't forget to clean the VM's after the tests:
+
+```bash
+make clean
+```
+
+### Generate expected regress test results
+
+Edit the `test-*.bash` scripts and set `GENERATE_EXPECTED` to true. Run it.
+
+Currently, expected tests results are the same for test cases 1 and 2.
+
 ## Tips
 
 Find all existing VM created by vagrant on your system:
@@ -32,7 +121,7 @@ Shutdown all VM:
 vagrant halt
 ```
 
-Restart cluster:
+Restart the halted cluster:
 
 ```bash
 vagrant up