Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

407 create cloud space #415

Merged
merged 48 commits into from
Dec 22, 2023
Merged
Show file tree
Hide file tree
Changes from 44 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
1a28fdc
Adding secrets folder but git ingore contents
jwestw Apr 14, 2023
7d4742a
Add google-cloud-storage
jwestw Apr 14, 2023
5a07c5b
GCP class and methods
jwestw Apr 14, 2023
0f9afef
File lister
jwestw Apr 14, 2023
0c770cc
File name lister (not just blobs)
jwestw Apr 14, 2023
a46d35d
adding google auth
jwestw May 19, 2023
1f6609f
tidy docstring
jwestw May 19, 2023
5f3f902
tidying and removing any_to_pd
Aug 11, 2023
268f717
tidying geospatial mods
Aug 11, 2023
ee75a5e
new path-url function, json key generalised
Aug 11, 2023
efb8545
no cwd in import, updated path-url
Aug 11, 2023
f275578
fixed NI csv reads and import tidy up
Aug 11, 2023
5f9bc64
using path_to_url for more functions
Aug 11, 2023
b9c8a8a
changing more reading data to path url
Aug 11, 2023
9f5c2ba
fixing bugs with path_url function
Aug 11, 2023
3495ed8
added in stops_geo_df to scot script
Aug 18, 2023
c762779
added in download data func for geospatial
Aug 18, 2023
96d8ea5
Merge branch 'master' into 407-create-cloud-space
paigeh-fsa Aug 18, 2023
faff3dc
added in new SDG_NI file to resolve errors
Aug 18, 2023
49728b0
fixing mkdocs error
Aug 18, 2023
18d6a09
Auto download missing train files
abhsheksingh Oct 13, 2023
ebe8afb
Signed Url for get_oa_la_csv_abspath function
abhsheksingh Oct 13, 2023
5a78da7
Merge remote-tracking branch 'origin/master' into 407-create-cloud-space
jwestw Oct 20, 2023
d929ad1
Change data ingest function names
jwestw Oct 20, 2023
907dcf6
fixing train timetable error
Oct 20, 2023
7c42325
Fixing the read csv for oa la lookup
jwestw Oct 20, 2023
df34171
Nice changes
jwestw Oct 20, 2023
54717c6
Improved the docstring and func name
jwestw Oct 20, 2023
2133c14
Change name of called func. Provided extension arg
jwestw Oct 20, 2023
9e18a61
Fix path to read from cloud
jwestw Oct 20, 2023
2002a58
Random LA
jwestw Oct 20, 2023
0a0b744
Gitkeep for NI pop centroids
jwestw Nov 10, 2023
d772a07
rounded seconds
jwestw Nov 10, 2023
448056b
Missing gitkeeps
jwestw Nov 10, 2023
9cdca4f
Running pipelines in order
jwestw Nov 24, 2023
a5b09e2
Copes with missing folder structure.
jwestw Nov 24, 2023
ddec206
Making pipelines run sequentially.
jwestw Nov 24, 2023
d95809d
All working
jwestw Nov 24, 2023
bd21cbc
Works to run all pipelines
jwestw Nov 24, 2023
78e77a7
Making folders
jwestw Nov 24, 2023
6bdcedb
Tell user to run pre-process
jwestw Nov 24, 2023
ead7207
Better timing
jwestw Nov 24, 2023
ddbb1c7
Logging
jwestw Nov 24, 2023
36de65b
Mainly logging
jwestw Nov 24, 2023
2c8c8b8
Fixed module pickup issue
Dec 1, 2023
5991b1b
Rectifying timetable import issue
Dec 1, 2023
298de85
Trying to fix bugs
Dec 22, 2023
702b3ca
Merge branch master into 407 cloud space
Dec 22, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,10 @@ data/**
!data/**/.gitkeep
.SDG_11.2.1

# Secrets folder
secrets/*
!secrets/.gitkeep

# VS Code stuff
workspace.code-workspace
sdg-11-2-1.code-workspace
Expand Down
6 changes: 5 additions & 1 deletion config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ outfile_ni: "NI_results.csv"
outfile_sc: "SC_results.csv"
eng_wales_preprocessed_output: "./data/eng_wales_preprocessed"

# Switch
cloud_local: "cloud"

# Mapping - geospatial
default_crs: 'EPSG:27700'

Expand All @@ -35,6 +38,7 @@ NI_bus_stops_data: 'https://www.opendatani.gov.uk/dataset/495c6964-e8d2-4bf1-994
NI_train_stops_data: 'https://www.opendatani.gov.uk/dataset/5f27f171-b8aa-4511-983d-6df6e87bbf20/resource/967e32c3-1cc2-4aee-b485-92121a32eb4d/download/nir-rail-stations.csv'
eng_bus_timetable_data: 'https://data.bus-data.dft.gov.uk/timetable/download/gtfs-file/all/'
auto_download_bus: true
auto_download_train: true
early_timetable_hour: 06
late_timetable_hour: 20
high_cap_buffer: 1000
Expand All @@ -47,7 +51,7 @@ station_locations: 'station_locations.csv'
bus_in_dir : 'data/england_bus_timetable/'
train_in_dir : 'data/england_train_timetable/'

# Ages
# Ages
age_lst:
- '0'
- '1'
Expand Down
Empty file added data/LA_shp/.gitkeep
Empty file.
Empty file added data/oa_la_mapping/.gitkeep
Empty file.
Empty file.
Empty file.
Empty file.
4 changes: 2 additions & 2 deletions docs/SDG_NI.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
Technical documentation for the SDG_NI module. Any docstrings in this file are automatically copied to this page.
Technical documentation for the SDG_NI module. Any docstrings in this file are automatically copied to this page.

::: src.SDG_NI
::: src.SDG_northern_ireland
2 changes: 2 additions & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,6 @@ openpyxl
sphinx_rtd_theme
convertbng
pre-commit
google-cloud-storage
google-auth
mkdocs
Empty file added secrets/.gitkeep
Empty file.
334 changes: 0 additions & 334 deletions src/SDG_NI.py

This file was deleted.

Loading