black, prune & precommits #234

tgrandje · 2025-01-20T20:21:56Z

I ran pytest and pyflakes on the whole code. I've also cleaned some comment (removed those which seemed unnecessary and kept those I had a doubt on), but that should be marginal.

There are also cases where I was completly sure of what to do, regarding some unused variables. Just to be sure, I flagged those in a different issue.

I suggest merging that PR if possible before merging anything else (I'll clean the current PR resolving github timeouts afterwards for instance).

tfardet

Thanks for the PR, I just have a few minor comments.
I also think it would be good to add a tutorial somewhere (readme.md) for contributors to install black and a pre-commit hook

tfardet · 2025-01-21T13:53:28Z

pynsee/geodata/_add_insee_dep.py

@@ -33,8 +33,7 @@ def _add_insee_dep_region(df):
                    )
                    df = df.merge(dep, on="insee_reg", how="left")

-    except Exception as e:
-        # print(e)
+    except Exception:
        pass


should this be logged?

I don't think so, the other exceptions in the same file are already silenced... What do you think?

pynsee/geodata/_get_full_list_wfs.py

pynsee/localdata/_get_insee_local_onegeo.py

pynsee/macrodata/_download_idbank_list.py

pynsee/macrodata/get_series_list.py

pynsee/sirene/search_sirene.py

tgrandje · 2025-01-22T10:57:16Z

I've just added pre-commits and a mention in the CONTRIBUTING.md. Please double-check this is working as expected, I'm in uncharted waters here 😉 !

tfardet

almost good, config is missing from the yaml file and I proposed a couple of changes to the contribution file.

tfardet · 2025-01-23T08:37:10Z

.flake8

+count = True
+exclude = .git,__pycache__,docs/*,*.yaml,*.zip,.env,*.svg,*.md,*.qmd,*.css,*.txt,*.rst,Dockerfile,.spyproject,.pytest_cache,*.lock,*.toml
+max-complexity = 10
+max-line-length = 127


do we want 127? I don't remember whether we discussed this already?
cc @hadrilec

That's from the existing CI :

pynsee/.github/workflows/pkgTests.yml

Lines 28 to 33 in 9dc111f

- name: Lint with flake8

run: |

# stop the build if there are Python syntax errors or undefined names

flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics

# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide

flake8 . --count --ignore=E722,C901 --exit-zero --max-complexity=10 --max-line-length=127 --statistics

OK, let's discuss it on Friday, then!

.pre-commit-config.yaml

CONTRIBUTING.md

Co-authored-by: tfardet <79037344+tfardet@users.noreply.github.com>

tgrandje · 2025-01-24T15:10:15Z

TODO : update CI

tfardet

@tgrandje there are files (.svg, .xml...) and docstrings that should not have been changed, could you check what happened (maybe you applied it during tests and did not undo it)?

pynsee/sirene/search_sirene.py

tgrandje · 2025-01-24T18:17:59Z

@tgrandje there are files (.svg, .xml...) and docstrings that should not have been changed, could you check what happened (maybe you applied it during tests and did not undo it)?

Well, that's just the first hooks, removing trailing whitespaces (speaking from memory). Nothing to worry (let's hope...)

tfardet · 2025-01-24T19:00:26Z

@tgrandje well, the .black file says that it should only touch .py files, so something is not consistent here...

tgrandje · 2025-01-24T19:19:23Z

.pre-commit-config.yaml

+    -   id: trailing-whitespace
+    -   id: end-of-file-fixer
+    -   id: check-yaml
+    -   id: check-added-large-files


@tfardet I think that's the part which impacted svgs and the like, nothing to worry too much

tgrandje · 2025-01-24T20:29:51Z

@tfardet ok, this time it might be good 😄 !

I just tested (and stopped) the CI, it passed and ran black and pyflakes without problems !

tfardet

looks good to me, I'll merge it right away, thanks @tgrandje !

tgrandje added 14 commits January 20, 2025 20:46

black & pyflakes formatting on download

05baae6

black & pyflakes formatting on geodata

b880c5a

black & pyflakes formatting on localdata

43b017e

black & pyflakes formatting for macrodata

58858a1

Black & pyflakes formatting on metadata

0b372d7

black & pyflakes formatting for sirene

499fe4c

black & pyflakes formatting for utils

5281968

less agressive black & pyflakes formatting for tests

b338091

revert mismatch in update pruning

4f80f58

prune unused variables/arguments

30bb229

remove unused _convert_polygon file

c782e58

remove ignored tweak argument

43f6217

remove ignored includeHistory arguments

78aaa4e

prune unused update argument (#236)

5db5ec1

tfardet requested changes Jan 21, 2025

View reviewed changes

tgrandje added 2 commits January 21, 2025 16:28

edits

aa316ed

test pre-commit

2f5c1bf

tfardet requested changes Jan 23, 2025

View reviewed changes

tgrandje and others added 7 commits January 23, 2025 09:57

Update .pre-commit-config.yaml

e1241ef

Co-authored-by: tfardet <79037344+tfardet@users.noreply.github.com>

Update CONTRIBUTING.md

5a7e289

Co-authored-by: tfardet <79037344+tfardet@users.noreply.github.com>

Update CONTRIBUTING.md

cbbc378

Co-authored-by: tfardet <79037344+tfardet@users.noreply.github.com>

Update download_file.py

d0551e2

finish pruning

2e67474

test merge

bfb2822

Update get_dataset.py

33c5b68

tgrandje changed the title ~~black formatting & pyflakes alerts resolution~~ black, prune & precommits Jan 23, 2025

tgrandje added 3 commits January 23, 2025 20:44

black config

62b27dc

Update CONTRIBUTING.md

2fbca09

update flake config

e2bb5ed

run precommit on all files

9d03502

tfardet requested changes Jan 24, 2025

View reviewed changes

pynsee/sirene/search_sirene.py Show resolved Hide resolved

tgrandje commented Jan 24, 2025

View reviewed changes

tgrandje added 7 commits January 24, 2025 20:41

reset docstring & fix type check

e0713bb

clean melodi merge

4919c71

align CI with pre-commits

03d22c7

force CI test

622db97

fix flake args

50c96db

fix black args

7999cc4

reset CI to original state

cebe326

tfardet approved these changes Jan 24, 2025

View reviewed changes

tfardet merged commit 91bdadb into master Jan 24, 2025
2 of 6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

black, prune & precommits #234

black, prune & precommits #234

tgrandje commented Jan 20, 2025

tfardet left a comment

tfardet Jan 21, 2025

tgrandje Jan 21, 2025

tgrandje commented Jan 22, 2025

tfardet left a comment

tfardet Jan 23, 2025

tgrandje Jan 23, 2025 •

edited

Loading

tfardet Jan 23, 2025

tgrandje commented Jan 24, 2025

tfardet left a comment

tgrandje commented Jan 24, 2025

tfardet commented Jan 24, 2025

tgrandje Jan 24, 2025

tgrandje commented Jan 24, 2025

tfardet left a comment

	- name: Lint with flake8
	run: \|
	# stop the build if there are Python syntax errors or undefined names
	flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
	# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
	flake8 . --count --ignore=E722,C901 --exit-zero --max-complexity=10 --max-line-length=127 --statistics

black, prune & precommits #234

black, prune & precommits #234

Conversation

tgrandje commented Jan 20, 2025

tfardet left a comment

Choose a reason for hiding this comment

tfardet Jan 21, 2025

Choose a reason for hiding this comment

tgrandje Jan 21, 2025

Choose a reason for hiding this comment

tgrandje commented Jan 22, 2025

tfardet left a comment

Choose a reason for hiding this comment

tfardet Jan 23, 2025

Choose a reason for hiding this comment

tgrandje Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

tfardet Jan 23, 2025

Choose a reason for hiding this comment

tgrandje commented Jan 24, 2025

tfardet left a comment

Choose a reason for hiding this comment

tgrandje commented Jan 24, 2025

tfardet commented Jan 24, 2025

tgrandje Jan 24, 2025

Choose a reason for hiding this comment

tgrandje commented Jan 24, 2025

tfardet left a comment

Choose a reason for hiding this comment

tgrandje Jan 23, 2025 •

edited

Loading