Add description of the sample based test system

README.md

@@ -1,5 +1,5 @@
 
-# d1_python
+## d1_python
 
 Python components for DataONE clients and servers.
 
@@ -9,7 +9,7 @@ See the [documentation on ReadTheDocs](http://dataone-python.readthedocs.io/en/l
 [![Coverage Status](https://coveralls.io/repos/github/DataONEorg/d1_python/badge.svg?branch=master)](https://coveralls.io/github/DataONEorg/d1_python?branch=master)
 [![PyPI version](https://badge.fury.io/py/dataone.common.svg)](https://badge.fury.io/py/dataone.common)
 
-#### v2 and v1 API
+### v2 and v1 API
 
 * DataONE Generic Member Node:
 [PyPI](https://pypi.python.org/pypi/dataone.gmn) –
@@ -24,7 +24,7 @@ See the [documentation on ReadTheDocs](http://dataone-python.readthedocs.io/en/l
 [PyPI](https://pypi.python.org/pypi/dataone.test_utilities) –
 [Docs](http://dataone-python.readthedocs.io/en/latest/test/index.html)
 
-#### v1 API
+### v1 API
 
 * DataONE Command Line Client (CLI):
 [PyPI](https://pypi.python.org/pypi/dataone.cli) –
@@ -41,7 +41,7 @@ See the [documentation on ReadTheDocs](http://dataone-python.readthedocs.io/en/l
 * Google Foresite Toolkit:
 [PyPI](https://pypi.python.org/pypi/google.foresite-toolkit)
 
-#### Contributing
+### Contributing
 
 Pull Requests (PRs) are welcome! Before you start coding, feel free to reach out to us and let us know what you plan to implement. We might be able to point you in the right direction.
 
@@ -82,11 +82,39 @@ Notes:
   * **Flake8 validation**: the same procedure as for `YAPF` can be used, as `Flake8` searches for its configuration file in the same way. In addition, IDEs can typically do code inspections and tag issues directly in the UI, where they can be handled before commit.
 
 
-#### Unit tests
+### Unit tests
 
-* Testing is based on the [pytest](https://docs.pytest.org/en/latest/) unit test framework.
+Testing is based on the [pytest](https://docs.pytest.org/en/latest/) unit test framework.
 
-* We have added some custom functionality to pytest which can be enabled to launching pytest with the following switches:
+#### Sample files
+
+Most of our tests work by serializing objects generated by the code being tested and comparing them with reference samples stored in files. This allows us to check all properties of generated objects without having to write asserts that checks individual properties, eliminating a time consuming and repetitive part of the test writing process.
+
+When writing comparisons manually, one will often select a few properties to check, and when those are determined to be valid, the remaining values are assumed to be correct as well. By comparing complete serialized versions of the objects, we avoid such assumptions.
+
+By storing the expected serialized objects in files instead of in the unit tests themselves, we avoid embedding hard coded documents inside the unit test modules and make it simple to automatically update the expected contents of objects as the code evolves.
+
+When unit tests are being run as part of CI or as a normal guard against regressions in a local development environment, any mismatches between actual and expected serialized versions of objects simply trigger test failures. However, when a test is initially created or the serialized version of an object is expected to change, tests can automatically write or update the sample files they use. This function is enabled by starting `pytest` with the `--update-samples` switch. When enabled, missing or mismatched sample files will not trigger test failures, instead starting an interactive process where differences are displayed together with yes/no prompts for writing or updating the samples. By default, differences are displayed in a GUI window using `kdiff3`, which provides a nice color coded view of the differences.
+
+The normal procedure for writing a sample based unit test is to just write the test as if the sample already exists, then running the test with `--update-samples` and viewing and approving the resulting sample, which is then automatically written to a file. The sample file name is displayed, making it easy to find the file in order to add it to tracking so that it can be committed along with the test module.
+
+Typically, it is not desirable to track generated files in Git. However, although the sample files are generated, they are an integral part of the units tests, and should be tracked just like the unit tests themselves.
+
+Also implemented is a simple process for cleaning out unused sample files. Sample files are often orphaned when their corresponding tests are removed or refactored. The default storage location for sample files is a directory called `test_docs`. To clean out unused files, move all the files from `test_docs` to `test_docs_tidy`, and run the tests. Any files that are accessed by the tests will automatically be moved back to `test_docs`, and any files remaining in `test_docs_tidy` after a complete test run can be untracked and deleted.
+
+#### DataONE Client to Django test adapter
+
+GMN tests are based on an adapter that enables using d1_client with the Django test framework. The adapter mocks Requests to issue requests through the Django test client.
+
+Django includes a test framework with a test client that provides an interface that's similar to that of an HTTP client, but calls Django internals directly. The client enables testing of most functionality of a Django app without actually starting the app as a network service.
+
+For testing GMN's D1 REST interfaces, we want to issue the test requests via the D1 MN client. Without going through the D1 MN client, we would have to reimplement much of what the client does, related to formatting and parsing D1 REST requests and responses.
+
+This module is typically used in tests running under django.test.TestCase and requires an active Django context, such as the one provided by `./manage.py test`.
+
+#### Command line switches
+
+We have added some custom functionality to pytest which can be enabled to launching pytest with the following switches:
 
   * `--update-samples`: Enable a mode that invokes `kdiff3` to display diffs and, after user confirmation, can automatically update or write new test sample documents on mismatches.
 
@@ -96,7 +124,6 @@ Notes:
 
   * See `./conftest.py` for implementation and notes.
 
-
 #### Debugging tests with PyCharm
 
 * By default, the PyCharm `Run context configuration (Ctrl+Shift+F10)` will generate test configurations and run the tests under the native unittest framework in Python's standard library. This will cause the tests to fail as they require pytest. To generate pytest configurations by default, set `Settings > Tools > Python Integrated Tools > Default test runner` to py.test. See the [documentation](https://www.jetbrains.com/help/pycharm/2017.1/testing-frameworks.html) for details.
@@ -112,17 +139,17 @@ Notes:
 * `pytest` by default captures `stdout` and `stderr` output for the tests and only shows the output for the tests that failed after all tests have been completed. Since a test that hits a breakpoint has not yet failed, this hides any output from tests being debugged and also hides output from the debug console prompt (where Python script can be evaluated in the current context). To see the output while debugging, go to `Run / Debug Configurations > Edit Configurations > Defaults > py.test > Additional Arguments` and add `--capture=no` (`-s`). Verbosity can also be increased by adding one or more `-v`.  
 
 
-##### Django
+### Django
 
 * Testing of the GMN Django web app is based on pytest and [pytest-django](https://pytest-django.readthedocs.io/en/latest/).
 
 * The tests use `settings_test.py` for GMN and Django configuration.
 
 * Pytest-django forces `settings.DEBUG` to `False` in `pytest_django/plugin.py`. To set `settings.DEBUG`, override it close to where it will be read, e.g., wit `@django.test.override_settings(DEBUG=True)`.
 
-### Create a new release
+### Creating a new release
 
-##### Update dependencies
+#### Update dependencies
 
 The stack should pass all the tests with the most recent available versions of all the dependencies.
 
@@ -131,11 +158,11 @@ Start by updating all the dependencies:
     $ cd d1_python
     $ sudo ./dev_tools/pip-update-all.py
 
-##### Make sure that the tests still pass
+#### Make sure that the tests still pass
 
     $ pytest
 
-##### Update the setup.py files
+#### Update the setup.py files
 
 The DataONE Python stack specifies fixed versions of all its dependencies. This ensures that a stack deployed to production matches one that passed the tests. As updating the versions in the `setup.py` files manually is time consuming and error prone, a script is included that automates the task. The script updates the version information for the dependencies in the `setup.py` files to match the versions of the currently installed dependencies. Run the script with:
 
@@ -144,12 +171,12 @@ The DataONE Python stack specifies fixed versions of all its dependencies. This
 
 The `<version>` argument specifies what the version will be for the release. E.g., `"2.3.1"`. We keep the version numbers in sync between all of the packages in the d1_python git repository, so only one version string needs to be specified.
 
-##### Build the new packages
+#### Build the new packages
 
     $ clean-tree.py
     $ setup-all.py sdist bdist_wheel
 
-##### Push the new packages to PyPI
+#### Push the new packages to PyPI
 
     TODO
 
@@ -192,19 +219,19 @@ Run the following commands (all sections), except, change the location for opens
 
 Run the tests and verify that they all pass:
 
-      $ pytest
+    $ pytest
 
 
 ### Building the documentation
 
 
-      TODO
+    TODO
 
 
 ### Troubleshooting
 
-* Clear out the installed libraries and reinstall:
+Clear out the installed libraries and reinstall:
 
-      $ sudo rm -rf /usr/local/lib/python2.7/dist-packages/d1_*
-      $ sudo nano /usr/local/lib/python2.7/dist-packages/easy-install.pth
-      Remove all lines that are: dataone.*.egg and that are paths to your d1_python.
+    $ sudo rm -rf /usr/local/lib/python2.7/dist-packages/d1_*
+    $ sudo nano /usr/local/lib/python2.7/dist-packages/easy-install.pth
+    Remove all lines that are: dataone.*.egg and that are paths to your d1_python.

test_utilities/src/d1_test/sample.py

@@ -55,7 +55,6 @@ def assert_equals(
   )
 
   if pytest.config.getoption('--update-samples'):
-
     _save_interactive(got_str, exp_path)
   else:
     raise AssertionError('Sample mismatch. filename="{}"'.format(filename))
@@ -106,15 +105,8 @@ def save_path(got_str, exp_path, mode_str='wb'):
 
 
 def _get_or_create_path(filename):
-  """Get the path to a sample file
-
-  Also provides a mechanism for cleaning out unused sample files. To clean, move
-  all files from `test_docs` to `test_docs_tidy`, and run the tests. Any files
-  that are used by the tests will be moved back to `test_docs`. Files that
-  remain in `test_docs_tidy` can be untracked and deleted.
-
-  This procedure moves files while pytest is running, which may confuse pytest.
-  Fix by clearing out pytest's cache with clean-tree.py.
+  """Get the path to a sample file and enable cleaning out unused sample files.
+  See the test docs for usage.
   """
   path = get_path(filename)
   if not os.path.isfile(path):