Skip to content

Commit

Permalink
Merge branch '171-remove-cookieecutter-logic' into 'master'
Browse files Browse the repository at this point in the history
[RFC] Resolve "Remove cookieecutter logic"

Closes #171

See merge request mutt_data/soam!116
  • Loading branch information
fedemutt committed Aug 9, 2021
2 parents 1c7bdf3 + 6687d67 commit 03464ea
Show file tree
Hide file tree
Showing 44 changed files with 10 additions and 1,288 deletions.
2 changes: 1 addition & 1 deletion .bumpversion.cfg
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 0.8.1
current_version = 0.9.0
tag = False

[bumpversion:file:soam/__init__.py]
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.9.0 - 2021-08-09]

### Removed
- Cookiecutter logic.

## [0.8.1 - 2021-08-06]

### Added
Expand Down
8 changes: 1 addition & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,13 +109,7 @@ _Note_: The `pdf_report` extra might need to run the following command before in
[Here](quickstart.ipynb) is an example for a quick start into SoaM. In it a time series with AAPL stock prices is loaded, processed and forecasted. As well, there's [other example](soamflowrun.ipynb) with the same steps, but exploding the power of flows.

## Usage
For further info check our **[end to end](end2end.md)** example where we explained how SoaM will interact with Airflow and Cookiecutter on a generic project.

## SoaM CLI
After installing SoaM locally, there are some commands available:

* `soam init --output`, scaffolding for a project ready to use SoaM. `cookiecutter` will then ask for parameters via prompt.
[//comment]: # (TODO: Add further info about directories and their functionalities created by cookiecutter. Maybe in end2end.md?)
For further info check our [end to end](end2end.md) example where we explained how SoaM will interact with Airflow on a generic project.

## Database management
For database storing there are complementary tools:
Expand Down
3 changes: 0 additions & 3 deletions documentation/source/project_structure.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,6 @@ This directory contains the migrations for the database that stores the data and
### [alembic.ini](../../soam/alembic.ini)
Configures the alembic library to manage the migrations for the DBSaver.

### templates
Contains the directory structure that cookiecutter is going to generate.

### resources
Files that will be used in the pipeline, like a template for an email report in a postprocess step.

Expand Down
10 changes: 1 addition & 9 deletions documentation/source/references.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,7 @@ This is the priority order to understand the dependencies:
1. prefect
2. alembic
3. sqlalchemy
4. jinja?
5. cookiecuter
6. darts
4. jinja

<!--- # (TODO: Ask Pedro to improve the background documents
with the knowledge learned from delver)--->
Expand Down Expand Up @@ -85,12 +83,6 @@ Used in: console.py
[click docs](https://click.palletsprojects.com/en/7.x/)


## cookiecutter
Used in: console.py

[cookiecutter docs](https://cookiecutter.readthedocs.io/en/latest/)


## datetime
Used in: constants.py, helpers.py, plot_utils.py, runner.py

Expand Down
8 changes: 0 additions & 8 deletions documentation/source/soam.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,6 @@ soam.cfg module
:undoc-members:
:show-inheritance:

soam.console module
-------------------

.. automodule:: soam.console
:members:
:undoc-members:
:show-inheritance:

soam.constants module
---------------------

Expand Down
19 changes: 1 addition & 18 deletions end2end.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,12 @@
# End to End data product with SoaM:

The purpose of this document is to show how a generic end to end data product would look line using SoaM, [Cookiecutter](https://cookiecutter.readthedocs.io/en/) and [Airflow](http://airflow.apache.org/).
The purpose of this document is to show how a generic end to end data product would look line using SoaM and [Airflow](http://airflow.apache.org/).

Let's imagine you need to run a process everyday that consists on loading the daily ABT, querying your database, transforming some datapoints, forecasting on a desired timeframe, plotting the results and sharing it by slack with your workteam. After that, promote results to the production environment. The diagram bellow show this process:

```mermaid
flowchart LR;
soma_main_repo[/SoaM Main Repo/]--->|defines|cookiecutter[/Cookiecutter/];
raw_data[(Raw Data)]-->load_daily_abt
cookiecutter--generates-->soam_instance;
subgraph DAG
soam_instance-->promote_to_prod[Promote Results to Production];
load_daily_abt[Prepare new data in analytics DB]-->soam_instance[SoaM Instance];
Expand All @@ -27,21 +25,6 @@ In this case, SoaM and Airflow will be interacting elbow to elbow to get this ru

See the following sections where we double click on this so that it's clearer for you.


## Cookiecutter strucutre

[Cookiecutter](https://medium.com/worldsensing-techblog/project-templates-and-cookiecutter-6d8f99a06374) is a CLI tool (Command Line Interface) to create an application boilerplate from a template. It uses a templating system — [Jinja](https://jinja.palletsprojects.com/) — to replace or customize folder and file names, as well as file content.

We use Cookiecutter to save time constructing a new repository, to avoid forgetting mandatory files like Readme or Changelog; and to lower the entry level to new collaborators — new team members, freelancers, partners.

After installing SoaM locally, you can run the following command:

```bash
$ soam init --output
```

To generate a project structure ready to use SoaM based on Cookiecutter. After executing that command, cookiecutter will ask you for parameters via prompt.

## Airflow

We use **Apache Airflow** to create the DAG that schedules the following pipeline:
Expand Down
1 change: 0 additions & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,6 @@
"prefect==0.14.17",
"filelock",
"click",
"cookiecutter",
"wheel",
"muttlib>=1.1.2,<2",
"numpy>=1.19,<1.20",
Expand Down
2 changes: 1 addition & 1 deletion soam/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"""Version."""

__version__ = '0.8.1'
__version__ = '0.9.0'
55 changes: 0 additions & 55 deletions soam/console.py

This file was deleted.

7 changes: 0 additions & 7 deletions soam/templates/init/cookiecutter.json

This file was deleted.

13 changes: 0 additions & 13 deletions soam/templates/init/hooks/post_gen_project.py

This file was deleted.

2 changes: 0 additions & 2 deletions soam/templates/init/{{ cookiecutter.package_name }}/.bump

This file was deleted.

This file was deleted.

Empty file.
21 changes: 0 additions & 21 deletions soam/templates/init/{{ cookiecutter.package_name }}/.flake8

This file was deleted.

145 changes: 0 additions & 145 deletions soam/templates/init/{{ cookiecutter.package_name }}/.gitignore

This file was deleted.

Loading

0 comments on commit 03464ea

Please sign in to comment.