Updating DCML annotation workflow (v2.4)

What's Changed

• Updating DCML annotation workflow by @johentsch in #259

Full Changelog: v2.3...v2.4

Full extraction with ms3 v2.4.1; updated workflow to v4.3

v2.3

chore: files updated with tag: v2.3

Updated extracted data with ms3 v2.4.0

Apart from updating the metadata, this update also adds

• the chords facet that includes control events (such as dynamics) and thorough bass figures
• the annotation workflow v4.1

v2.1: Re-extracts all files using ms3 v2.2.1

Changes to the TSV facets

Updated filenames

The TSV files have been renamed and now come with a suffix specifying the data facet that they represent. The three facets are

• measures
• notes
• harmonies

so, for example, the file measures/op01n01a.tsv has been renamed to measures/op01n01a.measures.tsv.

Added resource descriptors

Each TSV file is now accompanied with a JSON resource descriptor following the frictionless specification for Tabular Data Resources. They can and have be used to validate the tabular data.

These files have the same names as the TSV files they describe but replacing the .tsv extension with .resource.json. For example, the file measures/op01n01a.tsv has the corresponding descriptor measures/op01n01a.resource.json.

These metadata files replace the previous csv-metadata.json files (which followed the CSVW standard) that had described the data inadequately.

Added column quarterbeats_all_endings

The quarterbeats column (also known as qstamp) that had already been available does not provide values for first or third endings, expressing the dimensions of a singular playthrough without taking into account any repeats. The newly added column quarterbeats_all_endings does not have any empty values, providing continuous positions as if each measure followed the previous one.

Renamed index column

The first column of metadata.tsv was renamed from fname to piece.

Changes to the reviewed folder

The _reviewed.mscx files compare the current set of annotation labels against those from the previous version (v2.1). They are identical, so no differences are displayed.

Wherever the MuseScore parser throws one or several warnings (e.g. because an
annotation label does not match the score very well), these warnings appear in a .warnings file. For example, the
warnings for op01n01a can be seen
in reviewed/op01n01a.warnings.
One goal for future versions of this dataset would be to reduce the number of warnings to the point
of addressing all of them.

Frictionless datapackage

This release is the first one that has a frictionless datapackage attached (below). It consists of

• corelli.zip, a ZIP file containing one TSV file per facet, that corresponds to a concatenation of the TSV files in the respective folder, that is
  • corelli.expanded.tsv
  • corelli.measures.tsv
  • corelli.metadata.tsv (concatenation of a single file)
  • corelli.notes.tsv
• corelli.datapackage.json, the package descriptor.

If one has the frictionless framework installed, one can use the descriptor to validate the package using the command

frictionless validate corelli.datapackage.json

Arcangelo Corelli - Trio Sonatas (A corpus of annotated scores) (v2.0)

This corpus of annotated MuseScore files has been created within
the DCML corpus initiative and employs
the DCML harmony annotation standard. It was relased together with and as part
of the "workflow paper"

Hentschel, J., Moss, F. C., Neuwirth, M., & Rohrmeier, M. A. (2021). A semi-automated workflow paradigm for the distributed creation and curation of expert annotations. Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR, 262–269. https://doi.org/10.5281/ZENODO.5624417

The corpus comprises 36 Sonate a tre, divided into 149 separate movements. Together they make up for
three of the four famous cycles of 12 trio sonatas each:

Opus	Cycle	Publication	Included
1	12 sonate da chiesa	Rome 1681	Yes
2	12 sonate da camera	Rome 1685	No
3	12 sonate da chiesa	Rome 1689	Yes
4	12 sonate da camera	Rome 1694	Yes

Versions

Version 2.0

• TSV files now come with the column quarterbeats, which measures in quarter notes each event's position as its
  distance from the beginning
• Extracted notes now come with the columns name and octave.
• Column volta (containing first and second endings) removed from pieces that don't have any.
• metadata.tsv has been enriched with further columns, in particular information about each movement's dimensions,
  including dimensions upon unfolding repeats (for instance, last_mn has the number of
  measures, last_mn_unfolded the number of measures when playing all repeats)
• The folder reviewed contains two files per movement:
  • A copy of the score where all out-of-label notes have been colored in red;
    are shown in these files in a diff-like manner (removed in red, added in green).
  • A copy of the harmonies TSV with six added columns that reflect the coloring of out-of-label notes ("coloring
    reports")
• As long as the ms3 review has any complaints, it stores them in the file warnings.log. Currently, it is
  showing
  those labels where over 60% of the notes in the segment have been colored in red and probably need revisiting (
  Pull Requests welcome)
• TSV files are automatically kept up to date using the new GitHub action
  dcml_corpus_workflow which is the successor of the implementation
  used in the creation of this dataset.

Version 1.1

This release marks the moment where all 149 movements include a reviewed set of annotations that adhere to version
2.3.0 of the DCML harmony annotation standard. The metadata have not been
completed yet and the data were extracted one last time with the now deprecated version 0.4.11 of the
MuseScore parser ms3 for matters of completeness and homogeneity. The purpose is
mainly to substantiate the claim that the "semi-annotated workflow paradigm", as it had been implemented at publication
time (see the ISMIR paper cited above), can indeed be put to effective use in the creation of a large dataset. This
version is, however, to be followed by a version with upgraded tabular data based on the more mature ms3 > 1.0.0.

Version 1.0

The first release reflects the state of the dataset when finalizing chapter 4 of the workflow paper cited above.

Arcangelo Corelli - Trio Sonatas (A corpus of annotated scores) (v1.1)

This corpus of annotated MuseScore files has been created within the DCML corpus initiative and employs the DCML harmony annotation standard. It was relased together with and as part of the "workflow paper"

Hentschel, J., Moss, F. C., Neuwirth, M., & Rohrmeier, M. A. (2021). A semi-automated workflow paradigm for the distributed creation and curation of expert annotations. Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR, 262–269. https://doi.org/10.5281/ZENODO.5624417

The corpus comprises 36 Sonate a tre, divided into 149 separate movements. Together they make up for three of the four famous cycles of 12 trio sonatas each:

Opus	Cycle	Publication	Included
1	12 sonate da chiesa	Rome 1681	Yes
2	12 sonate da camera	Rome 1685	No
3	12 sonate da chiesa	Rome 1689	Yes
4	12 sonate da camera	Rome 1694	Yes

Version 1.0 reflects the state of the dataset when finalizing chapter 4 of the workflow paper cited above.

Version 1.1 marks the moment where all 149 movements include a reviewed set of annotations that adhere to version 2.3.0 of the DCML harmony annotation standard. The metadata have not been completed yet and the data were extracted one last time with the now deprecated version 0.4.11 of the MuseScore parser ms3 for matters of completeness and homogeneity. The purpose is mainly to substantiate the claim that the "semi-annotated workflow paradigm", as it had been implemented at publication time (see the ISMIR paper cited above), can indeed be put to effective use in the creation of a large dataset. This version is, however, to be followed by a version with upgraded tabular data based on the more mature ms3 > 1.0.0.

This corpus of annotated [MuseScore](https://musescore.org/) files has been created within the [DCML corpus initiative](https://github.com/DCMLab/dcml_corpora) and employs the [DCML harmony annotation standard](https://github.com/DCMLab/standards). It was relased together with and as part of the "workflow paper"

Hentschel, J., Moss, F. C., Neuwirth, M., & Rohrmeier, M. A. (2021). A semi-automated workflow paradigm for the distributed creation and curation of expert annotations. Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR, 262–269. https://doi.org/10.5281/ZENODO.5624417

The corpus comprises 36 Sonate a tre, divided into 149 separate movements. Together they make up for three of the four famous cycles of 12 trio sonatas each:
Opus Cycle Publication Included
1 12 sonate da chiesa Rome 1681 Yes
2 12 sonate da camera Rome 1685 No
3 12 sonate da chiesa Rome 1689 Yes
4 12 sonate da camera Rome 1694 Yes

Version 1.0 reflects the state of the dataset when finalizing chapter 4 of the workflow paper cited above.

Version 1.1 marks the moment where all 149 movements include a reviewed set of annotations that adhere to version 2.3.0 of the DCML harmony annotation standard. The metadata have not been completed yet and the data were extracted one last time with the now deprecated version 0.4.11 of the MuseScore parser ms3 for matters of completeness and homogeneity. The purpose is mainly to substantiate the claim that the "semi-annotated workflow paradigm", as it had been implemented at publication time (see the ISMIR paper cited above), can indeed be put to effective use in the creation of a large dataset. This version is, however, to be followed by a version with upgraded tabular data based on the more mature ms3 > 1.0.0.

Arcangelo Corelli - Sonate a tre

This version represents the state of the dataset when finalizing chapter 4 of

Hentschel, J., Moss, F. C., Neuwirth, M., & Rohrmeier, M. A. (2021). A semi-automated workflow paradigm for the distributed creation and curation of expert annotations. Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR, 262–269. https://doi.org/10.5281/ZENODO.5624417