Merge pull request #9 from qld-gov-au/QOL-7906-ckan-2.9

QOL-7906 Prepare for CKAN 2.9

.github/workflows/test.yml

@@ -3,7 +3,7 @@ name: Tests
 on: [push, pull_request]
 jobs:
   lint:
-    runs-on: ubuntu-18.04
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
@@ -32,7 +32,7 @@ jobs:
       fail-fast: false
 
     name: CKAN ${{ matrix.ckan-version }}
-    runs-on: ubuntu-18.04
+    runs-on: ubuntu-latest
     container:
       image: openknowledge/ckan-dev:${{ matrix.ckan-version }}
     services:
@@ -85,9 +85,8 @@ jobs:
         if: ${{ matrix.ckan-version == '2.7' || matrix.ckan-version == '2.8' }}
         run: |
           paster --plugin=ckan db init -c test.ini
-          paster report initdb -c test.ini
-          paster report generate tagless-datasets -c test.ini
+          paster --plugin=ckanext-report report initdb -c test.ini
+          paster --plugin=ckanext-report report generate tagless-datasets -c test.ini
 
-      - name: Run all tests
-        run: |
-          pytest --ckan-ini=test.ini --cov=ckanext.report ckanext/report/tests
+      - name: Run tests
+        run: pytest --ckan-ini=test.ini --cov=ckanext.report --disable-warnings ckanext/report/tests

README.md

@@ -14,34 +14,32 @@ Example report:
 
 ![Demo report image](report-demo.png)
 
-A number of extensions currently offer reports that rely on this extension, e.g. [ckanext-archiver](https://github.com/datagovuk/ckanext-archiver/blob/master/ckanext/archiver/reports.py), [ckanext-qa](https://github.com/datagovuk/ckanext-qa/blob/master/ckanext/qa/reports.py), [ckanext-dgu](https://github.com/datagovuk/ckanext-dgu/blob/master/ckanext/dgu/lib/reports.py).
+A number of extensions currently offer reports that rely on this extension, e.g. [ckanext-archiver](https://github.com/ckan/ckanext-archiver/blob/master/ckanext/archiver/reports.py), [ckanext-qa](https://github.com/ckan/ckanext-qa/blob/master/ckanext/qa/reports.py), [ckanext-dgu](https://github.com/datagovuk/ckanext-dgu/blob/master/ckanext/dgu/lib/reports.py).
 
 TODO:
 
 * Stop a report from being generated multiple times in parallel (unnecessary waste) - use a queue?
 * Stop more than one report being generated in parallel (high load for the server) - maybe use a queue.
 
-Compatibility: Requires CKAN version 2.1 or later (but can be easily adapted for older versions).
+## Compatibility:
 
+| CKAN version    | Compatibility       |
+| --------------- | ------------------- |
+| 2.6 and earlier | yes                 |
+| 2.7             | yes                 |
+| 2.8             | yes                 |
+| 2.9             | yes                 |
 
-| CKAN version    | Compatibility |
-| --------------- | ------------- |
-| 2.6 and earlier | yes           |
-| 2.7             | yes           |
-| 2.8             | yes           |
-| 2.9             | yes           |
+Status: was in production at data.gov.uk around 2014-2016, but since that uses its own CSS rather than core CKAN's, for others to use it CSS needs adding. For an example, see this branch: see https://github.com/GSA/ckanext-report/tree/geoversion
 
-
-Status: in production at data.gov.uk but since that uses its own CSS rather than core CKAN's, for others to use it CSS needs adding. For an example, see this branch: see https://github.com/yaditi/ckanext-report/tree/geoversion
-
-Author(s): David Read
+Author(s): David Read and contributors
 
 
 ## Install & setup
 
 Install ckanext-report into your CKAN virtual environment in the usual way:
 
-    (pyenv) $ pip install -e git+https://github.com/datagovuk/ckanext-report.git#egg=ckanext-report
+    (pyenv) $ pip install -e git+https://github.com/ckan/ckanext-report.git#egg=ckanext-report
 
 Initialize the database tables needed by ckanext-report:
 
@@ -54,7 +52,7 @@ Enable the plugin. In your config (e.g. development.ini or production.ini) add `
 
 ## Command-line interface
 
-The following operations can be run from the command line using the ``paster --plugin=ckanext-report report`` command:
+The following operations can be run from the command line using the ``paster --plugin=ckanext-report report`` or ``ckan report`` commands:
 
 ```
   report list
@@ -67,14 +65,17 @@ The following operations can be run from the command line using the ``paster --p
 Get the list of reports:
 
     (pyenv) $ paster --plugin=ckanext-report report list --config=mysite.ini
+    (pyenv) $ ckan --config=mysite.ini report list
 
 Generate all reports:
 
     (pyenv) $ paster --plugin=ckanext-report report generate --config=mysite.ini
+    (pyenv) $ ckan --config=mysite.ini report generate
 
 Generate a single report:
 
     (pyenv) $ paster --plugin=ckanext-report report generate <report name> --config=mysite.ini
+    (pyenv) $ ckan --config=mysite.ini report generate <report name>
 
 
 ## Demo report - Tagless Datasets
@@ -142,6 +143,9 @@ Report (snippet)
 table - main data, as a list of rows, each row is a dict
 data - other data values, as a dict
 #}
+
+{% set ckan_29_or_higher = h.ckan_version().split('.')[1] | int >= 9 %}
+{% set dataset_read_route = 'dataset.read' if ckan_29_or_higher else 'dataset_read' %}
 <ul>
     <li>Datasets without tags: {{ table|length }} / {{ data['num_packages'] }} ({{ data['packages_without_tags_percent'] }})</li>
     <li>Average tags per package: {{ data['average_tags_per_package'] }} tags</li>
@@ -160,7 +164,7 @@ data - other data values, as a dict
       {% for row in table %}
         <tr>
           <td>
-            <a href="{{ h.url_for(controller='package', action='view', id=row.name) }}">
+            <a href="{{ h.url_for(dataset_read_route, id=row.name) }}">
               {{ row.title }}
             </a>
           </td>

ckanext/report/controllers/__init__.py

@@ -38,6 +38,9 @@ def report_view(report_name, organization=None, refresh=False):
         t.abort(401)
     except t.ObjectNotFound:
         t.abort(404)
+    except Exception as e:
+        log.error("Failed to get report: %s", e)
+        raise
 
     # ensure correct url is being used
     if 'organization' in _get_routing_rule()\
@@ -70,7 +73,8 @@ def report_view(report_name, organization=None, refresh=False):
             log.warn('Not displaying report option HTML for param %s as option not recognized')
             continue
         option_display_params = {'value': options[option],
-                                 'default': report['option_defaults'][option]}
+                                 'default': report['option_defaults'][option],
+                                 'report_name': report_name}
         try:
             options_html[option] = \
                 t.render_snippet('report/option_%s.html' % option,
@@ -121,13 +125,14 @@ def report_view(report_name, organization=None, refresh=False):
             except t.NotAuthorized:
                 t.abort(401)
             filename = 'report_%s.csv' % key
-            t.response.headers['Content-Type'] = 'application/csv'
-            t.response.headers['Content-Disposition'] = six.text_type('attachment; filename=%s' % (filename))
-            return make_csv_from_dicts(data['table'])
+            response_headers = {
+                'Content-Type': 'application/csv',
+                'Content-Disposition': six.text_type('attachment; filename=%s' % (filename))
+            }
+            return make_csv_from_dicts(data['table']), response_headers
         elif format == 'json':
-            t.response.headers['Content-Type'] = 'application/json'
             data['generated_at'] = report_date
-            return json.dumps(data)
+            return json.dumps(data), {'Content-Type': 'application/json'}
         else:
             t.abort(400, 'Format not known - try html, json or csv')
 
@@ -141,4 +146,4 @@ def report_view(report_name, organization=None, refresh=False):
         'report_date': report_date, 'options': options,
         'options_html': options_html,
         'report_template': report['template'],
-        'are_some_results': are_some_results})
+        'are_some_results': are_some_results}), {}

ckanext/report/controllers/blueprints.py

@@ -1,9 +1,11 @@
 # encoding: utf-8
 
-from flask import Blueprint
+import six
+
+from flask import Blueprint, Response
 
-from ckanext.report.controllers import report_index, report_view
 from ckan.plugins import toolkit
+from . import report_index, report_view
 
 
 reporting = Blueprint(
@@ -16,17 +18,28 @@ def redirect_to_index():
     return toolkit.redirect_to('/report')
 
 
+def view(report_name, organization=None, refresh=False):
+    body, headers = report_view(report_name, organization, refresh)
+    if headers:
+        response = Response(body)
+        for key, value in six.iteritems(headers):
+            response.headers[key] = value
+        return response
+    else:
+        return body
+
+
 reporting.add_url_rule(
     u'/report', 'index', view_func=report_index
 )
 reporting.add_url_rule(
     u'/reports', 'reports', view_func=redirect_to_index
 )
 reporting.add_url_rule(
-    u'/report/<report_name>', 'view', view_func=report_view
+    u'/report/<report_name>', 'view', view_func=view, methods=('GET', 'POST',)
 )
 reporting.add_url_rule(
-    u'/report/<report_name>/<organization>', 'org', view_func=report_view
+    u'/report/<report_name>/<organization>', 'org', view_func=view, methods=('GET', 'POST',)
 )
 
 

ckanext/report/controllers/pylons_controllers.py

@@ -1,5 +1,7 @@
 # encoding: utf-8
 
+import six
+
 import ckan.plugins.toolkit as t
 from ckanext.report.controllers import report_index, report_view
 
@@ -10,4 +12,8 @@ def index(self):
         return report_index()
 
     def view(self, report_name, organization=None, refresh=False):
-        return report_view(report_name, organization, refresh)
+        body, headers = report_view(report_name, organization, refresh)
+        if headers:
+            for key, value in six.iteritems(headers):
+                t.response.headers[key] = value
+        return body

ckanext/report/helpers.py

@@ -35,11 +35,11 @@ def relative_url_for(**kwargs):
         args = dict(list(tk.request.environ['pylons.routes_dict'].items())
                     + user_specified_params
                     + list(kwargs.items()))
-    # remove blanks
-    for k, v in list(args.items()):
-        if not v:
-            del args[k]
-    return tk.url_for(**args)
+        # remove blanks
+        for k, v in list(args.items()):
+            if not v:
+                del args[k]
+        return tk.url_for(**args)
 
 
 def chunks(list_, size):

ckanext/report/templates/report/option_organization.html

@@ -9,7 +9,7 @@
     <label for="option-organization"> {{ _('Organization') }}: </label>
     {% set offer_organization_index = (default == None) %}
     {% if offer_organization_index and value != None %}
-        <a href="{{ h.report__relative_url_for(organization=None) }}">{% trans %}Index of all organizations{% endtrans %}</a>
+        <a href="{{ h.url_for('report.view', report_name=report_name) }}">{% trans %}Index of all organizations{% endtrans %}</a>
     {% endif %}
     <select id="option-organization" name="organization" class="inline js-auto-submit">
         {% if offer_organization_index %}

ckanext/report/templates/report/view.html

@@ -14,7 +14,7 @@ <h1>{{ report.title }}</h1>
           {{ _('Generated') }}: {{ h.report__render_datetime(report_date, '%d/%m/%Y %H:%M') }}
       </p>
       {% if c.userobj.sysadmin %}
-      <div class="panel panel-info" style="width=700px">
+      <div class="panel panel-info">
           <div class="panel-heading"><strong>{% trans %}Refresh report{% endtrans %}</strong></div>
           <div class="panel-body">
             <form action="" method="POST">
@@ -43,8 +43,8 @@ <h3>{{ _('Options') }}</h3>
       {% if are_some_results %}
         <div class="pull-right">
             {{ _('Download') }}:
-            <a class="btn btn-primary" href="{{ h.report__relative_url_for(format='csv') }}">CSV</a>
-            <a class="btn btn-primary" href="{{ h.report__relative_url_for(format='json') }}">JSON</a>
+            <a class="btn btn-primary" href="{{ h.url_for('report.view', format='csv', report_name=report_name) }}">CSV</a>
+            <a class="btn btn-primary" href="{{ h.url_for('report.view', format='json', report_name=report_name) }}">JSON</a>
         </div>
       {% endif %}
       <h3>{{ _('Results') }}</h3>