Merge pull request #794 from 18F/develop

Staging deployment

README.md

@@ -2,19 +2,19 @@
 [![Snyk](https://snyk.io/test/github/18F/analytics-reporter/badge.svg)](https://snyk.io/test/github/18F/analytics-reporter)
 [![Code Climate](https://codeclimate.com/github/18F/analytics-reporter/badges/gpa.svg)](https://codeclimate.com/github/18F/analytics-reporter)
 
-## Analytics Reporter
+# Analytics Reporter
 
-A lightweight system for publishing analytics data from Google Analytics profiles.
-Uses the [Google Analytics Core Reporting API v3](https://developers.google.com/analytics/devguides/reporting/core/v3/)
-and the [Google Analytics Real Time API v3](https://developers.google.com/analytics/devguides/reporting/realtime/v3/).
+A lightweight system for publishing analytics data from the Digital Analytics Program (DAP) Google Analytics 4 government-wide property.
+This project uses the [Google Analytics Data API v1](https://developers.google.com/analytics/devguides/reporting/data/v1/rest) to acquire analytics data and then processes it into a flat data structure.
 
-This is used in combination with [analytics-reporter](https://github.com/18F/analytics-reporter-api) and [analytics.usa.gov](https://github.com/18F/analytics.usa.gov) to power the government analytics website, [analytics.usa.gov](https://analytics.usa.gov).
+The project previously used the [Google Analytics Core Reporting API v3](https://developers.google.com/analytics/devguides/reporting/core/v3/)
+and the [Google Analytics Real Time API v3](https://developers.google.com/analytics/devguides/reporting/realtime/v3/), also known as Universal Analytics,  which has slightly different data points. See [Upgrading from Universal Analytics](#upgrading-from-universal-analytics) for more details. The Google Analytics v3 API will be deprecated on July 1, 2024.
 
-Available reports are named and described in [`reports.json`](reports/reports.json). For now, they're hardcoded into the repository.
+This is used in combination with [analytics-reporter-api](https://github.com/18F/analytics-reporter-api) to power the government analytics website, [analytics.usa.gov](https://analytics.usa.gov).
 
-### Installation
+Available reports are named and described in [`reports.json`](reports/reports.json). For now, they're hardcoded into the repository.
 
-### Docker
+## Docker Setup
 
 * To build the docker image on your computer, run:
 
@@ -30,25 +30,38 @@ Then you can create an alias in order to have the analytics command available:
 alias analytics="docker run -t -v ${HOME}:${HOME} -e ANALYTICS_REPORT_EMAIL -e ANALYTICS_REPORT_IDS -e ANALYTICS_KEY analytics-reporter"
 ```
 
-To make this command working as expected you should export the env vars as follows:
+To make this command work as expected you should export the env vars as follows:
 
 ```bash
 export ANALYTICS_REPORT_EMAIL="your-report-email"
 export ANALYTICS_REPORT_IDS="your-report-ids"
 export ANALYTICS_KEY="your-key"
 ```
 
-### NPM
+## Local development setup
+
+### Prerequistites
+
+* NodeJS > v20.x
+* A postgres DB running
+
+### Running the application as a npm package
 
 * To run the utility on your computer, install it through npm:
 
 ```bash
 npm install -g analytics-reporter
 ```
 
-If you're developing locally inside the repo, `npm install` is sufficient.
+### Running the application locally
+
+#### Install dependencies
+
+```bash
+npm install
+```
 
-### Setup
+## Configuration and Google Analytics Setup
 
 * Enable [Google Analytics API](https://console.cloud.google.com/apis/library/analytics.googleapis.com) for your project in the Google developer dashboard.
 
@@ -64,7 +77,7 @@ If you're developing locally inside the repo, `npm install` is sufficient.
 
 ```bash
 export ANALYTICS_REPORT_EMAIL="YYYYYYY@developer.gserviceaccount.com"
-export ANALYTICS_REPORT_IDS="ga:XXXXXX"
+export ANALYTICS_REPORT_IDS="XXXXXX"
 ```
 
 You may wish to manage these using [`autoenv`](https://github.com/kennethreitz/autoenv). If you do, there is an `example.env` file you can copy to `.env` to get started.
@@ -141,8 +154,7 @@ There are cases where you want to use a custom  object storage server compatible
 export AWS_S3_ENDPOINT=http://your-storage-server:port
 ```
 
-
-### Other configuration
+## Other configuration
 
 If you use a **single domain** for all of your analytics data, then your profile is likely set to return relative paths (e.g. `/faq`) and not absolute paths when accessing real-time reports.
 
@@ -163,7 +175,7 @@ This will produce points similar to the following:
 }
 ```
 
-### Use
+## Use
 
 Reports are created and published using the `analytics` command.
 
@@ -178,6 +190,8 @@ A report might look something like this:
 ```javascript
 {
   "name": "devices",
+  "frequency": "daily",
+  "slim": true,
   "query": {
     "dimensions": [
       {
@@ -194,7 +208,7 @@ A report might look something like this:
     ],
     "dateRanges": [
       {
-        "startDate": "90daysAgo",
+        "startDate": "30daysAgo",
         "endDate": "yesterday"
       }
     ],
@@ -205,15 +219,12 @@ A report might look something like this:
         },
         "desc": true
       }
-    ],
-    "samplingLevel": "HIGHER_PRECISION",
-    "limit": "10000",
-    "property": "properties/393249053"
+    ]
   },
   "meta": {
     "name": "Devices",
-    "description": "90 days of desktop/mobile/tablet visits for all sites."
-  },
+    "description": "30 days of desktop/mobile/tablet visits for all sites."
+  }
   "data": [
     {
       "date": "2023-12-25",
@@ -245,7 +256,7 @@ A report might look something like this:
 }
 ```
 
-#### Options
+### Options
 
 * `--output` - Output to a directory.
 
@@ -294,7 +305,7 @@ analytics --frequency=realtime
 analytics --publish --debug
 ```
 
-### Saving data to postgres
+## Saving data to postgres
 
 The analytics reporter can write data is pulls from Google Analytics to a
 Postgres database. The postgres configuration can be set using environment
@@ -313,6 +324,48 @@ server](https://github.com/18f/analytics-reporter-api) that consumes and publish
 To write reports to a database, use the `--write-to-database` option when
 starting the reporter.
 
+## Upgrading from Universal Analytics
+
+### Background
+
+This project previously acquired data from Google Analytics V3, also known as Universal Analytics (UA).
+
+Google is retiring UA and is encouraging users to move to their new version Google Analytics V4 (GA4).
+UA will be deprecated on July 1st 2024.
+
+### Migration details
+
+Some data points have been removed or added by Google as part of the move to GA4.
+
+#### Deprecated fields
+
+- browser_version
+- has_social_referral
+- exits
+- exit_page
+
+#### New fields
+
+##### bounce_rate
+
+The percentage of sessions that were not engaged.  GA4 defines engaged as a
+session that lasts longer than 10 seconds or has multiple pageviews.
+
+##### file_name
+
+The page path of a downloaded file.
+
+##### language_code
+
+The ISO639 language setting of the user's device.  e.g. 'en-us'
+
+##### session_default_channel_group
+
+An enum which describes the session. Possible values:
+
+'Direct', 'Organic Search', 'Paid Social', 'Organic Social', 'Email',
+'Affiliates', 'Referral', 'Paid Search', 'Video', and 'Display'
+
 ### Deploying to Cloud.gov
 
 The analytics reporter runs on :cloud:.gov. Please refer to the `manifest.yml`
@@ -365,28 +418,30 @@ Compose:
 docker-compose up
 ```
 
-#### Running unit tests locally
+## Running the unit tests
 
-The unit tests require a postgres database to be running and accepting
-connections at 127.0.0.1:5432
+The unit tests for this repo require a local PostgreSQL database. You can run a
+local DB server or create a docker container using the provided test compose
+file. (Requires docker and docker-compose to be installed)
 
-To run the test database locally with docker:
+Starting a docker test DB:
 
 ```shell
 docker-compose -f docker-compose.test.yml up
 ```
 
-The test scripts run database migrations and then the tests themselves.  These
-require database connection details to be provided in the shell environment:
+Once you have a PostgreSQL DB running locally, you can run the tests. The test
+DB connection in knexfile.js has some default connection config which can be
+overridden with environment variables.  If using the provided docker-compose DB
+then you can avoid setting the connection details.
+
+Run the tests (pre-test hook runs DB migrations):
 
 ```shell
-POSTGRES_PASSWORD=123abc \
-POSTGRES_USER=analytics \
-POSTGRES_DATABASE=analytics_reporter_test \
 npm test
 ```
 
-### Public domain
+## Public domain
 
 This project is in the worldwide [public domain](LICENSE.md). As stated in [CONTRIBUTING](CONTRIBUTING.md):
 

knexfile.js

@@ -1,17 +1,21 @@
-const config = require("./src/config")
-
 module.exports = {
   development: {
     client: 'postgresql',
-    connection: config.postgres,
+    connection: {
+      database: process.env.POSTGRES_DATABASE,
+      host: process.env.POSTGRES_HOST,
+      user: process.env.POSTGRES_USER,
+      password: process.env.POSTGRES_PASSWORD,
+      port: 5432,
+    }
   },
   test: {
     client: 'postgresql',
     connection: {
-      host: '127.0.0.1',
-      database: "analytics_reporter_test",
-      user: process.env.CIRCLECI ? "postgres" : config.postgres.user,
-      password: process.env.POSTGRES_PASSWORD,
+      database: process.env.POSTGRES_DATABASE || "analytics_reporter_test",
+      host: process.env.POSTGRES_HOST || 'localhost',
+      user: process.env.POSTGRES_USER || 'analytics',
+      password: process.env.POSTGRES_PASSWORD || '123abc',
       port: 5432,
     },
     migrations: {
@@ -21,12 +25,11 @@ module.exports = {
   production: {
     client: 'postgresql',
     connection: {
+      database: process.env.POSTGRES_DATABASE,
       host: process.env.POSTGRES_HOST,
       user: process.env.POSTGRES_USER,
       password: process.env.POSTGRES_PASSWORD,
-      database: process.env.POSTGRES_DATABASE,
-      port: 5432,
-      ssl: true
+      ssl: true,
     }
   },
 }

package.json

@@ -12,7 +12,7 @@
     "migrate": "knex migrate:latest",
     "pretest": "NODE_ENV=test npm run migrate",
     "start": "node app.js",
-    "test": "mocha",
+    "test": "NODE_ENV=test mocha",
     "prepare": "npm run snyk-protect",
     "snyk-protect": "snyk-protect"
   },