Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

work adding import functionality using django-import-export #800

Merged
merged 26 commits into from
Jan 25, 2024
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
14b480d
person import work
quadrismegistus Jan 16, 2024
726ee48
first attribute working
quadrismegistus Jan 16, 2024
2b17de0
both working
quadrismegistus Jan 16, 2024
5e897f7
condensing code
quadrismegistus Jan 16, 2024
20d7e08
minor
quadrismegistus Jan 16, 2024
cc4b970
adding countryadmin back
quadrismegistus Jan 16, 2024
e38652f
turning off index signals while importing
quadrismegistus Jan 16, 2024
66bd9c6
working now with shared set of cols
quadrismegistus Jan 17, 2024
d6b0209
quick fixes
quadrismegistus Jan 17, 2024
099c027
should be working now
quadrismegistus Jan 18, 2024
a05b9a4
new settings; export cols separated
quadrismegistus Jan 19, 2024
feec6bd
adding index logic
quadrismegistus Jan 19, 2024
9e810e8
indexing
quadrismegistus Jan 19, 2024
c6e86c1
update
quadrismegistus Jan 19, 2024
c5b45a3
all working now
quadrismegistus Jan 19, 2024
ae06676
tests for import and export at requests level
quadrismegistus Jan 22, 2024
10ee53c
minor fix
quadrismegistus Jan 22, 2024
239d979
remove redundant code
quadrismegistus Jan 22, 2024
305ae29
reformatting
quadrismegistus Jan 24, 2024
26e767d
reformatting
quadrismegistus Jan 24, 2024
75e0e4f
cleanup
quadrismegistus Jan 24, 2024
bda61a6
added unit test
quadrismegistus Jan 24, 2024
1c27716
ensuring indexing disabled after person admin testing
quadrismegistus Jan 25, 2024
3dcba17
ensuring indexing disabled after person admin testing 2
quadrismegistus Jan 25, 2024
c2d7df8
Cleanup viaf person save test method for new skip lookup config
rlskoeser Jan 25, 2024
b2dd870
cleanup and making import model resource extensible for books
quadrismegistus Jan 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions dev-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ sphinx
wheel
pre-commit
wagtail-factories
django-import-export
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs to go in the main requirements (dev reqs aren't installed on the servers)

45 changes: 42 additions & 3 deletions mep/people/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
from django.utils.timezone import now
from tabular_export.admin import export_to_csv_response
from viapy.widgets import ViafWidget

from mep.accounts.admin import AddressInline
from mep.common.admin import (
CollapsedTabularInline,
Expand All @@ -27,6 +26,12 @@
Relationship,
RelationshipType,
)
from import_export.admin import (
ImportExportModelAdmin,
)
from import_export.resources import ModelResource
from import_export.widgets import ManyToManyWidget, Widget
from import_export.fields import Field


class InfoURLInline(CollapsibleTabularInline):
Expand Down Expand Up @@ -82,7 +87,6 @@ class Meta:
)
}


class CountryAdmin(admin.ModelAdmin):
form = CountryAdminForm
list_display = ("name", "geonames_id", "code")
Expand Down Expand Up @@ -436,8 +440,43 @@ class Media:
]




class PersonResource(ModelResource):
def before_import(self, dataset, *args, **kwargs):
# lower and camel_case headers
dataset.headers = [x.lower().replace(' ','_') for x in dataset.headers]

def before_import_row(self, row, **kwargs):
# gender to one char
def fmt_gender(x):
x = str(x).strip()
return x[0].upper() if x else ''
row['gender']=fmt_gender(row.get('gender'))

name = Field(column_name='name', attribute='name')
gender = Field(column_name='gender', attribute='gender')
nationalities = Field(
column_name='nationality',
attribute='nationalities',
widget=ManyToManyWidget(Country, field='name', separator=';')
)

class Meta:
model = Person
fields = ('name','gender','nationalities')
import_id_fields = ('name',)
skip_unchanged = True
report_skipped = True


class PersonAdminWithImport(PersonAdmin, ImportExportModelAdmin):
resource_class = PersonResource
change_list_template = "templates/admin/people/person/change_list.html"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this the default? do we need to customize for some reason?



# enable default admin to see imported data
admin.site.register(Person, PersonAdmin)
admin.site.register(Person, PersonAdminWithImport)
admin.site.register(Country, CountryAdmin)
admin.site.register(Location, LocationAdmin)
admin.site.register(Profession, NamedNotableAdmin)
Expand Down
1 change: 1 addition & 0 deletions mep/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@
"mep.books",
"mep.footnotes",
"mep.pages",
"import_export"
]

MIDDLEWARE = [
Expand Down