Skip to content

Commit

Permalink
Merge branch 'master' into kma/ruff
Browse files Browse the repository at this point in the history
  • Loading branch information
kmagusiak committed Dec 4, 2023
2 parents c337264 + a93f0e1 commit 3d622c7
Show file tree
Hide file tree
Showing 32 changed files with 1,508 additions and 923 deletions.
12 changes: 6 additions & 6 deletions .github/workflows/lint-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,12 @@ jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
# use minimum version here from setup.py
python-version: "3.8"
python-version: "3.9"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
Expand All @@ -28,11 +28,11 @@ jobs:
strategy:
matrix:
# lowest, common (defaut ubuntu LTS), newest
python-version: ["3.8", "3.10", "3.11"]
python-version: ["3.9", "3.10", "3.11"]
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/python-publish.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
with:
# don't use shallow checkout to determine an intermediary version correctly
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install dependencies
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ __pycache__
# Local env
.env
.venv
/local*

# Other tools
dist
Expand Down
5 changes: 1 addition & 4 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -1,12 +1,9 @@
{
"python.formatting.blackArgs": [
"black-formatter.args": [
"--skip-string-normalization",
"--line-length",
"100"
],
"python.formatting.provider": "black",
"python.linting.flake8Enabled": false,
"python.linting.mypyEnabled": true,
"python.testing.pytestArgs": [
"tests"
],
Expand Down
104 changes: 71 additions & 33 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
A small library to ease writing parameterized scripts.
The goal is to execute a single script and be able to overwrite the parameters
easily.
The configuration is based on [OmegaConf](https://omegaconf.readthedocs.io/).
Optionally, loading from toml is possible.
The configuration is based on [OmegaConf].
Optionally, loading from toml or using [pydantic] is possible.

To run multiple related tasks, there is an integration with
[invoke](https://www.pyinvoke.org).
Expand All @@ -23,28 +23,27 @@ To run an application, you need...
import alphaconf
import logging
# define the default values and helpers
alphaconf.setup_configuration("""
server:
url: http://default
""", {
alphaconf.setup_configuration({
"server.url": "http://default",
}, {
"server.url": "The URL to show here",
})

def main():
log = logging.getLogger()
log.info('server.url:', alphaconf.get('server.url'))
log.info('has server.user:', alphaconf.get('server.user', bool))
log.info('has server.user:', alphaconf.get('server.user', bool, default=False))

if __name__ == '__main__':
alphaconf.run(main)
alphaconf.cli.run(main)
```

Invoking:
```bash
python myapp.py server.url=http://github.com
```

During an interactive session, you can set the application in the current
During an *interactive session*, you can set the application in the current
context.
```python
# import other modules
Expand All @@ -64,29 +63,29 @@ Then configuration is built from:

- default configurations defined using (`alphaconf.setup_configuration`)
- `application` key is generated
- PYTHON_ALPHACONF may contain a path to a configuration file
- `PYTHON_ALPHACONF` environment variable may contain a path to load
- configuration files from configuration directories (using application name)
- environment variables based on key prefixes,
except "BASE" and "PYTHON";
except "BASE" and "PYTHON"; \
if you have a configuration key "abc", all environment variables starting
with "ABC_" will be loaded where keys are converted to lower case and "_"
to ".": "ABC_HELLO=a" would set "abc.hello=a"
with "ABC_" will be loaded, for example "ABC_HELLO=a" would set "abc.hello=a"
- key-values from the program arguments

Finally, the configuration is fully resolved and logging is configured.

## Configuration templates and resolvers

Omegaconf's resolvers may be used as configuration values.
For example, `${oc.env:USER,me}` would resolve to the environment variable
USER with a default value "me".
Similarly, `${oc.select:path}` will resolve to another configuration value.

Additional resolvers are added to read file contents.
These are the same as type casts: read_text, read_strip, read_bytes.

The select is used to build multiple templates for configurations by providing
base configurations.
Configuration values are resolved by [OmegaConf].
Some of the resolvers (standard and custom):
- `${oc.env:USER,me}`: resolve the environment variable USER
with a default value "me"
- `${oc.select:config_path}`: resolve to another configuration value
- `${read_text:file_path}`: read text contents of a file as `str`
- `${read_bytes:file_path}`: read contents of a file as `bytes`
- `${read_strip:file_path}`: read text contents of a file as strip spaces

The *oc.select* is used to build multiple templates for configurations
by providing base configurations.
An argument `--select key=template` is a shortcut for
`key=${oc.select:base.key.template}`.
So, `logging: ${oc.select:base.logging.default}` resolves to the configuration
Expand All @@ -97,18 +96,54 @@ dict defined in base.logging.default and you can select it using

### Typed-configuration

You can use *omegaconf* with *dataclasses* to specify which values are
enforced in the configuration.
Alternatively, the *get* method can receive a data type or a function
which will parse the value.
By default, bool, str, Path, DateTime, etc. are supported.
You can use [OmegaConf] with [pydantic] to *get* typed values.
```python
class MyConf(pydantic.BaseModel):
value: int = 0

def build(self):
# use as a factory pattern to create more complex objects
# for example, a connection to the database
return self.value * 2

# setup the configuration
alphaconf.setup_configuration(MyConf, prefix='a')
# read the value
alphaconf.get('a', MyConf)
v = alphaconf.get(MyConf) # because it's registered as a type
```

### Secrets

When showing the configuration, by default configuration keys which are
secrets, keys or passwords will be masked.
Another good practice is to have a file containing the password which
you can retrieve using `alphaconf.get('secret_file', 'read_strip')`.
You can read values or passwords from files, by using the template
`${read_strip:/path_to_file}`
or, more securely, read the file in the code
`alphaconf.get('secret_file', Path).read_text().strip()`.

### Inject parameters

We can inject default values to functions from the configuration.
Either one by one, where we can map a factory function or a configuration key.
Or inject all automatically base on the parameter name.

```python
from alphaconf.inject import inject, inject_auto

@inject('name', 'application.name')
@inject_auto(ignore={'name'})
def main(name: str, example=None):
pass

# similar to
def main(name: str=None, example=None):
if name is None:
name = alphaconf.get('application.name', str)
if example is None:
example = alphaconf.get('example', default=example)
...
```

### Invoke integration

Expand All @@ -123,7 +158,10 @@ alphaconf.setup_configuration({'backup': 'all'})
alphaconf.invoke.run(__name__, ns)
```

### Interactive and manual usage
## Way to 1.0
- Run a specific function `alphaconf my.module.main`:
find functions and inject args
- Install completions for bash `alphaconf --install-autocompletion`

Use `alphaconf.interactive.mount()` or load manually create an
`alphaconf.Application`, configure it and set it.
[OmegaConf]: https://omegaconf.readthedocs.io/
[pydantic]: https://docs.pydantic.dev/latest/
Loading

0 comments on commit 3d622c7

Please sign in to comment.