Add Danish NER corpus

[alanakbik]

This PR adds the DANSK corpus for NER in Danish.

[alanakbik]

@stefan-it there are some discrepancies in the dataset statistics, but I think this is probably explained by the removal of the OpenSubtitles part of the corpus, as noted here: https://huggingface.co/datasets/chcaa/dansk-ner#domain-distribution

[stefan-it]

Hey @alanakbik , thanks for this PR!

I compared it quickly to my draft implementation, and I spotted a few differences - right side is output of this PR, training split:

or:

[alanakbik]

@stefan-it thanks for noticing this! There was an issue with handling of newline characters that were previously simply excluded, causing the token offsets to not line up anymore. The first sentence in your example was prefaced by a single newline token, which was excluded, explaining the offsets.

With this fix, the statistics of the dataset now fully line up with the statistics posted on https://huggingface.co/datasets/chcaa/dansk-ner#domain-distribution, minus the OpenSubtitles part that was excluded.

[stefan-it]

I think lines 5659 - 5669 can be removed due to debugging-only :)

[stefan-it]

Thanks @alanakbik , current version looks really good, I am just wondering, if we really need empty tokens + O in the dataset:

I was able to avoid this by using this code fragment:

# Write tokens and tags
                            for token, tag in zip(tokens, tags):
                                token_text = text[token["start"] : token["end"]]  # Don't strip the token
                                if token_text:  # Still skip empty tokens
                                    # Replace newlines with space in output to maintain CoNLL format
                                    token_text = token_text.replace("\n", " ")

                                    if not token_text.strip():
                                        continue

                                    f_out.write(f"{token_text}\t{tag}\n")

🤔

[alanakbik]

@stefan-it good point. One issue is that some "sentences" actually only consist of a single non-token, so there would need to be logic to handle this as well, otherwise we might get multiple empty lines after another. I considered filtering out those sentences, but then the dataset statistics no longer line up with the statistics posted on the DANSK page. Do you think it still makes sense to remove empty tokens and sentences, or should we prioritize matching the dataset statistics?

[stefan-it]

So I think for reproducibility reasons, we should then stick to the current solution, to really match the numbers from their documentation ☺️