Version 0.9

- PyPI release! Can now install with ``pip install matador-db`` (unfortunately ``matador`` was taken, but they are sufficiently orthogonal that the package name ``matador`` is retained here.
- Much improved code structure and many additional classes that wrap the raw calculation dictionaries for e.g. :class:`matador.crystal.Crystal` and spectral classes.
- New module :mod:`matador.orm` containing useful models for data handling.

  - :class:`matador.orm.orm.DataContainer` as a base class that allows for easy
    access to underlying dictionaries.
  - :mod:`matador.orm.spectral` module that contains many useful classes for
    manipulating and plotting e.g. bandstructures, DOS and finite temperature
    properties.

- New features in :mod:`matador.hull` module:

  - Pseudo-ternary phase diagrams (building towards arbitrary n-dimensional phase diagrams).
  - :class:`matador.hull.EnsembleHull` class and submodule to support the Bayesian Error Estimate Functional (BEEF) and finite temperature phase diagrams.
  - Refactoring of hull calculation into light-weight :class:`matador.hull.PhaseDiagram` class.
  - Finite temperature hulls based on :class:`matador.hull.EnsembleHull` with
    :class:`matador.hull.TemperatureDependentHull`.

- Refactored old PDF `similarity` module into new module :mod:`matador.fingerprints`.

  - Added new fingerprint class, :class:`matador.fingerprints.PXRD`, with associated plots (thanks for James Darby for some initial code). Defaults calibrated with GSAS-II.
  - :class:`matador.fingerprints.PDF` sped up by an order of magnitude using `numba`.

- :class:`matador.workflows.castep.CastepSpectralWorkflow` extended to include latest projected dispersion curve developments from OptaDOS, with associated projected dispersion plots (see tutorial).

  - Updated dispersion script to automatically perform naive Gaussian smearing if OptaDOS output not detected.

- Abstracted and simplified :mod:`matador.compute` module to allow for extension to new codes via :mod:`matador.compute.calculators` submodule.

  - Should now be more robust and transferrable, with many HPC environments automatically detected.
  - Added ``--scratch_prefix`` to run3 to allow for temporary files to e.g. be written to faster filesystem with appropriate symlinks to work folder.

- All CASTEP 19 keywords supported, as well as `devel_code` blocks.
- Several new tests: coverage now around 75% when CASTEP is available.
- New tutorials:

  - :ref:`MongoDB setup<mongo>`
  - :ref:`Spectral calculations with run3<run3_spectral>`
  - Example notebooks

.coveragerc

@@ -1,27 +1,16 @@
-# .coveragerc to control coverage.py
-[run]
-branch = True
-concurrency = multiprocessing
-parallel = True
-
 [report]
 # Regexes for lines to exclude from consideration
 ignore_errors = True
-omit = 
-    */tests/*
-    setup.py
-    */__init__.py
-    *matador/plotting*
+omit =
     *matador/workflows*
     *matador/plugins*
-    *matador/cli/dispersion.py
     *matador/cli/run3.py
-exclude_lines = 
+    *matador/config/quickstart.py
+exclude_lines =
     pragma: no cover
     if verbosity
     if self.verbosity
     if debug
     if self.debug
-    def plot_
     if kwargs.get('verbosity')
     if kwargs.get('debug')

.flake8

@@ -0,0 +1,13 @@
+[flake8]
+# E226: whitespace around operators which can look ugly, IMO
+# W503 and W504 can be contradictory: linebreaks before/after binary operators
+ignore = E226, W503, W504
+# same width as GH editor
+max-line-length = 127
+# these files are either autogenerated or no longer maintained
+exclude = docs,plugins,castep_params 
+# unfortunately complexity is a harsh requirement on some of this project...
+max-complexity = -1
+# ignore star imports in chem utils that bring in all constants
+per-file-ignores =
+    matador/utils/chem_utils.py:F405

.github/workflows/ci.yml

@@ -0,0 +1,65 @@
+name: Run tests
+
+on:
+  pull_request:
+  push:
+    branches:
+      - develop
+      - master
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      max-parallel: 2
+      matrix:
+        python-version: [3.6]
+
+    steps:
+    - uses: actions/checkout@v1
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v1
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        cp tests/data/matador_pipelines_conf.yml $HOME/.matadorrc
+        python -m pip install --upgrade pip
+        pip install flake8
+    - name: Check for syntax errors and lint with flake8
+      run: |
+        flake8 . \
+            --count --select=E9,F63,F7,F82 --show-source --statistics
+        # check for bad code smell on all but the least important files...
+        flake8 . \
+            --count \
+            --statistics
+
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.6]
+
+    steps:
+    - uses: actions/checkout@v1
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v1
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        cp tests/data/matador_pipelines_conf.yml $HOME/.matadorrc
+        python -m pip install --upgrade pip
+        pip install -e .[test,plotting,crystal,optional]
+    - name: Test with pytest
+      run: |
+        py.test -rs --cov-config .coveragerc --cov=./matador/ --cov-report=xml
+    - name: Upload coverage to Codecov
+      uses: codecov/codecov-action@v1
+      with:
+        token: ${{ secrets.CODECOV_TOKEN }}
+        file: ./coverage.xml
+        flags: unittests
+        yml: ./codecov.yml

.pre-commit-config.yaml

@@ -0,0 +1,6 @@
+repos:
+    - repo: https://gitlab.com/PyCQA/flake8
+      rev: master 
+      hooks:
+      - id: flake8
+        name: Flake8

CHANGELOG.rst

@@ -1,41 +1,71 @@
 Changelog
 =========
 
-For planned developments, please see the `Boards tab <https://bitbucket.org/ml-evs/matador/addon/trello/trello-board>`_ on Bitbucket.
+New in release (0.9) [15/05/2020]
+---------------------------------
 
-New in development version (rolling) (0.9a)
--------------------------------------------
+- PyPI release! Can now install with ``pip install matador-db`` (unfortunately ``matador`` was taken, but they are sufficiently orthogonal that the package name ``matador`` is retained here.
+- Much improved code structure and many additional classes that wrap the raw calculation dictionaries for e.g. :class:`matador.crystal.Crystal` and spectral classes.
+- New module :mod:`matador.orm` containing useful models for data handling.
 
-- SpectralWorkflow extended to include latest projected dispersion curve developments from OptaDOS.
-- Updated dispersion script to automatically perform naive Gaussian smearing if OptaDOS output not detected.
+  - :class:`matador.orm.orm.DataContainer` as a base class that allows for easy
+    access to underlying dictionaries.
+  - :mod:`matador.orm.spectral` module that contains many useful classes for
+    manipulating and plotting e.g. bandstructures, DOS and finite temperature
+    properties.
+
+- New features in :mod:`matador.hull` module:
+
+  - Pseudo-ternary phase diagrams (building towards arbitrary n-dimensional phase diagrams).
+  - :class:`matador.hull.EnsembleHull` class and submodule to support the Bayesian Error Estimate Functional (BEEF) and finite temperature phase diagrams.
+  - Refactoring of hull calculation into light-weight :class:`matador.hull.PhaseDiagram` class.
+  - Finite temperature hulls based on :class:`matador.hull.EnsembleHull` with
+    :class:`matador.hull.TemperatureDependentHull`.
+
+- Refactored old PDF `similarity` module into new module :mod:`matador.fingerprints`.
+
+  - Added new fingerprint class, :class:`matador.fingerprints.PXRD`, with associated plots (thanks for James Darby for some initial code). Defaults calibrated with GSAS-II.
+  - :class:`matador.fingerprints.PDF` sped up by an order of magnitude using `numba`.
+
+- :class:`matador.workflows.castep.CastepSpectralWorkflow` extended to include latest projected dispersion curve developments from OptaDOS, with associated projected dispersion plots (see tutorial).
+
+  - Updated dispersion script to automatically perform naive Gaussian smearing if OptaDOS output not detected.
+
+- Abstracted and simplified :mod:`matador.compute` module to allow for extension to new codes via :mod:`matador.compute.calculators` submodule.
+
+  - Should now be more robust and transferrable, with many HPC environments automatically detected.
+  - Added ``--scratch_prefix`` to run3 to allow for temporary files to e.g. be written to faster filesystem with appropriate symlinks to work folder.
+
+- All CASTEP 19 keywords supported, as well as `devel_code` blocks.
+- Several new tests: coverage now around 75% when CASTEP is available.
 - New tutorials:
 
-   + :ref:`MongoDB setup<mongo>`
-   + :ref:`Spectral calculations with run3<run3_spectral>`
+  - :ref:`MongoDB setup<mongo>`
+  - :ref:`Spectral calculations with run3<run3_spectral>`
+  - Example notebooks
 
-- Added ``--scratch_prefix`` to run3 to allow for temporary files to e.g. be written to faster filesystem with appropriate symlinks to work folder.
 
-New in current release (0.8b) [03/08/2018]
+New in release (0.8b) [03/08/2018]
 ------------------------------------------
 
 - Wholesale changes, complete refactoring of most of the code.
 - Released open source under the MIT license!
 - Documentation now hosted on `readthedocs <matador-db.readthedocs.org>`_,
 - Workflows: chaining up job steps with run3:
 
-   + spectral and phonons (combined DOS, dispersion calculations) with automated kpoint paths.
-   + bulk modulus calculations and EOS fitting.
+  - spectral and phonons (combined DOS, dispersion calculations) with automated kpoint paths.
+  - bulk modulus calculations and EOS fitting.
 
 - New tutorials:
 
-   + :ref:`Geometry optimisations with run3<run3_geom>`
+  - :ref:`Geometry optimisations with run3<run3_geom>`
 
-- Temperature-dependent convex hulls (by Angela Harper).
+- Temperature-dependent convex hulls (thanks to Angela Harper).
 - New per-used configuration that allows changing of plotting styles, colourschemes, database names etc.
 - Improvements to compute module:
 
-   + automatically handle walltime constraints,
-   + estimate memory usage with CASTEP and skip if exceeds machine capacity,
+  - automatically handle walltime constraints for Slurm and PBS.
+  - estimate memory usage with CASTEP and skip if exceeds machine capacity,
 
 - All CASTEP 18 keywords supported.
 - Better support for electronic structure data, OptaDOS, NMR calculations, CIF files, partial occupancy.

INSTALL.rst

@@ -4,38 +4,34 @@ Installation
 ============
 
 
-Installing with conda/pip
--------------------------
+Simple installation with pip
+----------------------------
+
+The matador package can be found on PyPI under the name `matador-db <https://pypi.org/project/matador-db>`_ and installed with
+``pip install matador-db``, preferably in a fresh virtual environment (see conda instructions below). Extra dependencies may be installed with e.g. ``pip install matador-db[plotting]``.
+
+Development installing with conda/pip
+-------------------------------------
 
 The tl;dr way to install matador, on e.g. a computing cluster, is as follows:
 
-1. Clone the matador source onto your local machine ``git clone https://bitbucket.org/ml-evs/matador.git``.
+1. Clone the matador source onto your local machine ``git clone https://github.com/ml-evs/matador.git``.
 
 Optional (but recommended) steps:
 
-2. `Install conda <https://conda.io/miniconda.html>`_, if you have not already (choose the version suitable for you, the installers Python version does not matter that much, but it may as well be 3+!).
-3. Create a new conda environment to install matador into (``conda create -n matador python=3.6``)
+2. `Install conda <https://conda.io/miniconda.html>`_, if you have not already. There may be a package available already if you are using a supercomputer (e.g. `anaconda-compute/2.2.0-python3` on ARCHER 30/10/2017).
+3. Create a new conda environment to install matador into (``conda create -n matador python=3.6``) and activate it with (``conda activate matador``).
 4. Install some of the heavier requirements (e.g. NumPy and SciPy) through conda with ``conda install --yes --file requirements/requirements.txt``. You may also want to install some of the optional dependencies in this manner, e.g. ``conda install --yes --file requirements/plotting_requirements.txt``.
 
 Required steps:
 
-4. Run ``pip install .`` from inside the top-level matador directory, or ``pip install -e .`` for an editable developer install.
-5. You now have a basic matador API installation, if you wish to use all matador features, install extra dependencies from the other requirements files inside ``requirements/`` using either conda (preferably) or pip. e.g. for plotting, running your own database and Jupyter notebook visualisation functionality, use ``pip install .[plotting,db,viz]``. If you wish to just install everything use ``pip install .[all]``.
-6. To use matador, you will need to work inside the conda environment from step 2, by running ``source activate matador``.
-7. You can test your installation using ``python -m unittest discover`` or simply ``py.test`` (if you have it installed).
-
-
-More esoteric installation on ARCHER (30th October 2017)
---------------------------------------------------------
+5. Run ``pip install .`` from inside the top-level matador directory, or ``pip install -e .`` for an editable developer install.
+6. You now have a basic matador API installation, if you wish to use all matador features, install extra dependencies from the other requirements files inside ``requirements/`` using either conda (preferably) or pip. e.g. for plotting, running your own database and Jupyter notebook visualisation functionality, use ``pip install .[plotting,db,viz]``. If you wish to just install everything use ``pip install .[all]``.
+7. To use matador, you will need to activate the conda environment from step 2, by running ``conda activate matador``. You will also need this in e.g. any job scripts. You can test your installation using ``python -m unittest discover -v -b`` or simply ``py.test`` (if you have it installed). By default this will look for an executable called `castep` to run CASTEP tests, which are probably the most useful.
 
-These steps assume you have your ``work`` directory symlinked inside
-``$HOME``.
+Troubleshooting
+---------------
 
-1. ``module load anaconda-compute/2.2.0-python3``
-2. ``conda create -p $HOME/work/.conda/matador-env python=3.6 numpy scipy``
-3. ``source activate $HOME/work/.conda/matador-env``
-4. ``git clone git@bitbucket.org:ml-evs/matador.git``
-5. ``cd matador``
-6. Due to some weird permissions on ARCHER, we must move the git folder
-   so that ``pip`` works...
-7. ``mv .git $HOME/matador_git_stash; pip install . ; mv $HOME/matador_git_stash .git``
+1. (10/09/2019) When installing with ``conda``, if you receive the following error (or
+   similar): ``/home/#####/.local/conda/envs/matador/compiler_compat/ld: build/temp.linux-x86_64-3.6/psutil/_psutil_common.o: unable to initialize decompress status for section .debug_info``, then you are using a modern compiler that breaks ``conda``'s attempts to be backwards compatibile (in this case it was GCC 9). The simple fix is to rename/remove the copy of ``ld`` inside your conda environment (path in the message above) such that your system ``ld`` is used.
+2. (10/10/2017) On some machines (e.g. ARCHER/Thomas) you may receive permissions errors at step 5; if so, try moving matador's `.git` and install again (``mv .git $HOME/matador_git_stash; pip install . ; mv $HOME/matador_git_stash .git``).

MANIFEST.in

@@ -1,8 +1,7 @@
 recursive-include matador/config *
 recursive-include docs *
 recursive-include matador/tests/data *
-include matador/scrapers/words
-include matador/scrapers/nouns
+recursive-include matador/data *
 include README.rst
 include CHANGELOG.rst
 include INSTALL.rst

README.rst

@@ -2,66 +2,47 @@
 matador
 =======
 
-|Bitbucket Pipelines| |Develop Pipelines| |Coverage Status| |Documentation Status| |MIT License|
+|GH Actions| |Coverage Status| |Documentation Status| |MIT License|
 
-matador is an aggregator, manipulator and runner of first-principles
-calculations, written with a bent towards battery electrode materials. 
-The source can be found on `Bitbucket <https://bitbucket.org/ml-evs/matador>`_.
+matador is an aggregator, manipulator and runner of first-principles calculations, written with a bent towards battery electrode materials.
+The source can be found on `GitHub <https://github.com/ml-evs/matador>`_ and online documentation is hosted on `ReadTheDocs <https://docs.matador.science>`_.
 
-Written by `Matthew Evans <https://www.ml-evs.github.io>`_ (2016-). 
+Example Jupyter notebooks and tutorials can be found `online <https://docs.matador.science/en/latest/examples_index.html>`_ or in the ``examples/`` folder of the matador source code.
+
+Written & maintained by `Matthew Evans <https://ml-evs.science>`_ (2016-).
 
 
 .. image:: docs/src/img/lipzn.png
    :name: LiPZn
    :align: center
 
-Core functionality
--------------------
+Installation
+------------
 
-The core functionality can be summarised by the various sub-commands of the
-command-line interface. The API has many more features that can be explored
-in the examples and API documentation.
-
-
-1. The scraping of CASTEP/Quantum Espresso output files into flexible
-   Python dictionaries with a sensible schema via ``matador import``.
-2. The transferal of the scraped dictionaries into a MongoDB database.
-3. Powerful CLI querying of the database, with a focus on energy storage
-   applications using ``matador query``.
-4. Calculation and presentation of binary and ternary phase diagrams
-   with ``matador hull``.
-5. 0K voltage curves for binary and ternary systems, as well as arbitrary intercalation electrodes, using
-   ``matador voltage``.
-6. Atomic species swapping and polishing from previous calculations using 
-   ``matador swaps``.
-7. Automated high-throughput geometry optimisations, electronic and vibrational properties, 
-   plus convergence tests, using CASTEP or Quantum Espresso, with the ``run3`` entrypoint.
-   Tested on several supercomputers.
-8. Refinement of structural and chemical data, and symmetries powered by ``spglib``, via
-   ``matador refine``.
+In the simplest case, ``pip install matador-db`` is sufficient to get up and running, preferably in a fresh virtual environment. More detailed instructions can be found in the `Installation instructions <https://docs.matador.science/en/latest/install.html>`_.
 
 Usage
 ------
 
-For basic command-line usage, please explore the help system for each sub-command. Common workflows can be found under ``examples/`` and :ref:`tutorials_index`.
+For basic command-line usage, please explore the help system for each sub-command. Common workflows can be found inside ``examples/`` and in the `online docs <http://docs.matador.science/en/latest/examples_index.html>`_.
 
-The full Python API documentation can be found under :ref:`modules`.
+Please consult the full `Python API documentation <http://docs.matador.science/en/latest/modules.html>`_ for programmatic usage.
 
 .. code-block:: text
 
     usage: matador [-h] [--version]
                    {stats,query,import,hull,voltage,changes,hulldiff,swaps,refine}
                    ...
-    
+
     MATerial and Atomic Database Of Refined structures.
-    
+
     optional arguments:
       -h, --help            show this help message and exit
       --version             show program's version number and exit
-    
+
     subcommands:
       valid sub-commands
-    
+
       {stats,query,import,hull,voltage,changes,hulldiff,swaps,refine}
         stats               print some stats about the database.
         query               query and extract structures from the database
@@ -77,18 +58,30 @@ The full Python API documentation can be found under :ref:`modules`.
         refine              update structures in the database according to
                             specified --task
 
-License
---------
+Core functionality
+-------------------
+
+The core functionality can be summarised by the various sub-commands of the
+command-line interface above. The API has many more features that can be explored
+in the examples and API documentation.
+
+- Scraping of CASTEP (and Quantum Espresso) input/output files into flexible
+Python dictionaries/models.
+- The creation and curation of MongoDB collections of geometry optimisation,
+calculations, with a powerful querying CLI/API.
+- Customisable, publication-ready plots for all models, e.g. phase diagrams, PDF, PXRD,
+voltage profiles, electronic/vibrational bandstructures etc.
+- Automated high-throughput geometry optimisations, electronic and vibrational properties
+using CASTEP (and Quantum Espresso) with ``run3``. Tested on several supercomputers.
+- Creation of phase diagrams and electrochemical voltage profiles from the
+results of DFT calculations.
 
-matador is available under the `MIT License <https://bitbucket.org/ml-evs/matador/src/master/LICENSE>`_.
 
-.. |Bitbucket Pipelines| image:: https://img.shields.io/bitbucket/pipelines/ml-evs/matador/master.svg?label=master
-   :target: https://bitbucket.org/ml-evs/matador/addon/pipelines/home#!/results/branch/master/page/1
-.. |Develop Pipelines| image:: https://img.shields.io/bitbucket/pipelines/ml-evs/matador/develop.svg?label=develop
-   :target: https://bitbucket.org/ml-evs/matador/addon/pipelines/home#!/results/branch/develop/page/1
+.. |GH Actions| image:: https://img.shields.io/github/workflow/status/ml-evs/matador/Run%20tests/develop?label=develop&logo=github
+   :target: https://github.com/ml-evs/matador/actions?query=branch%3Adevelop
 .. |MIT License| image:: https://img.shields.io/badge/license-MIT-blue.svg
-   :target: https://bitbucket.org/ml-evs/matador/src/master/LICENSE
-.. |Coverage Status| image:: https://codecov.io/bb/ml-evs/matador/branch/master/graph/badge.svg
-  :target: https://codecov.io/bb/ml-evs/matador
+   :target: https://github.com/ml-evs/matador/blob/master/LICENSE
+.. |Coverage Status| image:: https://img.shields.io/codecov/c/gh/ml-evs/matador/develop?logo=codecov
+  :target: https://codecov.io/gh/ml-evs/matador
 .. |Documentation Status| image:: https://readthedocs.org/projects/matador-db/badge/?version=latest
    :target: https://matador-db.readthedocs.io/en/latest/?badge=latest