refactor: naming of modules

[maxdinkel]

Description and Context

Currently, the cluster pipeline is failing because with import pickle in queens.utils.remote_operations.py the queens.utils.pickle.py is imported instead of the standard library. This bug was introduced with #136 (my bad 👼 ).
This PR refactors the pickle.py utils into the io.py utils to solve this issue.

Related Issues and Pull Requests

• Closes
• Blocks
• Is blocked by
• Follows 119 naming standard for queens #136
• Precedes
• Related to
• Part of
• Composed of

How Has This Been Tested?

Cluster pipeline

Checklist

• My commit messages mention the appropriate issue numbers (advised but optional).
• I mention the appropriate issue numbers in this PR.
• I updated documentation where necessary.
• I have added tests to cover my changes.

Additional Information

Interested Parties

Possible reviewers:

Other interested parties:

[maxdinkel]

I guess we have the same problem with numpy.py utils (see pipeline).
tensorflow.py and gpflow.py could also be problematic in the future. Any suggestions how to resolve that?

[gilrrei]

I guess we have the same problem with numpy.py utils (see pipeline). tensorflow.py and gpflow.py could also be problematic in the future. Any suggestions how to resolve that?

What about adding something like _utils, _wrapper or _queens for these cases? I know it breaks the convention. However, I personally feel that it is more than justified, usability and clarity always comes first IMO

[leahaeusel]

I guess we have the same problem with numpy.py utils (see pipeline). tensorflow.py and gpflow.py could also be problematic in the future. Any suggestions how to resolve that?

What about adding something like _utils, _wrapper or _queens for these cases? I know it breaks the convention. However, I personally feel that it is more than justified, usability and clarity always comes first IMO

As ugly as it is, I agree that we need to break the convention in such cases. I would prefer to simply append the package name, here _utils. We should run into the same problem with our Numpy data processor, right?

[leahaeusel]

iterators/pymc.py, schedulers/dask.py, and utils/pymc.py are also problematic, although I would argue that the pymc iterator should be renamed anyway such that the new name includes the actual method, e.g. mcmc_pymc.py

[maxdinkel]

I am starting to think that we should name all modules with a leading underscore. The user should import the modules through the __init__.py anyway and will not notice those leading underscores.

[leahaeusel]

I am starting to think that we should name all modules with a leading underscore. The user should import the modules through the __init__.py anyway and will not notice those leading underscores.

At first glance, this doesn't sound like a bad idea to me. Are there any unwanted side effects we need to take into account? Nevertheless, this wouldn't work for the utils modules.

[maxdinkel]

I am starting to think that we should name all modules with a leading underscore. The user should import the modules through the __init__.py anyway and will not notice those leading underscores.

At first glance, this doesn't sound like a bad idea to me. Are there any unwanted side effects we need to take into account? Nevertheless, this wouldn't work for the utils modules.

Why would it not work for the utils modules?

[leahaeusel]

At first glance, this doesn't sound like a bad idea to me. Are there any unwanted side effects we need to take into account? Nevertheless, this wouldn't work for the utils modules.

Why would it not work for the utils modules?

Because they are not imported through the __init__.py. Of course, you could still import them via queens.utils._numpy.py but the leading underscore usually indicates that these files are only for internal use and shouldn't be imported by the user, which would be misleading in this case.

[maxdinkel]

At first glance, this doesn't sound like a bad idea to me. Are there any unwanted side effects we need to take into account? Nevertheless, this wouldn't work for the utils modules.

Why would it not work for the utils modules?

Because they are not imported through the __init__.py. Of course, you could still import them via queens.utils._numpy.py but the leading underscore usually indicates that these files are only for internal use and shouldn't be imported by the user, which would be misleading in this case.

The idea is to import the modules as import _numpy as numpy in the __init__.py. Then the user can import the util module as import queens.utils.numpy as numpy_utils. And still, when we import numpy, for example, in queens.utils.classifier.py, this will import the numpy package and not our util package.

[leahaeusel]

The idea is to import the modules as import _numpy as numpy in the __init__.py. Then the user can import the util module as import queens.utils.numpy as numpy_utils. And still, when we import numpy, for example, in queens.utils.classifier.py, this will import the numpy package and not our util package.

You're completely right, I didn't think of that. Sorry for the confusion.

Just one last concern: Will the documentation display the Module names without the underscore? From a usability perspective, the documentation should be sufficient to tell the user how to import each module without having to look at the __init__.py, in my opinion.

[gilrrei]

As decided upon in a personal meeting:

• numpy dp -> numpy_file, utils -> linalg, array
• csv dp -> csv_file
• tensorflow utils -> configure_tensorflow
• gpflow utils -> gp_flow
• dask schelduer -> _dask
• pymc iterator, utils -> _pymc, move utils to _pymc
• All base classes to _

[maxdinkel]

Cluster pipeline is fixed.

[gilrrei]

Should we rename this file to test_numpy_array.py?