Normalize IPA strings to NFC for consistency

[lars76]

Hi, I noticed that the dataset contains a mixture of NFC and NFD Unicode forms for IPA strings. For example:

• Row 277, Col 7: ãː (NFD: a + COMBINING TILDE) vs. ãː (NFC: single precomposed ã).

Out of ~5.1 million cells, ~6,200 are not in NFC. This causes issues with string matching, e.g., "ã" != "ã" even though they look identical.

To fix this, I applied NFC normalization across the CSV like this:

import csv, unicodedata

with open("input.csv", "r", encoding="utf-8", newline="") as infile, \
     open("output.csv", "w", encoding="utf-8", newline="") as outfile:
    reader = csv.reader(infile)
    writer = csv.writer(outfile)
    for row in reader:
        writer.writerow([unicodedata.normalize("NFC", cell) for cell in row])

Example of a normalized cell:

Row 1747, Col 8
  Original   : o̞˞ o̞ õ̞ ɔ   [U+006F U+031E U+02DE U+0020 U+006F U+031E U+0020 U+006F U+031E U+0303 U+0020 U+0254]
  Normalized : o̞˞ o̞ õ̞ ɔ   [U+006F U+031E U+02DE U+0020 U+006F U+031E U+0020 U+00F5 U+031E U+0020 U+0254]

[drammock]

thanks for raising this issue. I think what you're seeing is expected/intentional behavior. We purposely chose not to use either full NFD or full NFC, because:

• NFC makes it harder to assign phonological features based on the glyphs, because it is sensitive to the order of diacritics/modifier letters. For example:
  • ã́ and á̃ (codepoints 0061+0301+0303) become either 00E1+0303 or 00E3+0301 depending on the order of the diacritics (and, for IPA purposes, that order shouldn't matter, but NFD doesn't re-order them)
• NFD wrongly decomposes ç which in IPA is not really a combination of c + cedilla, it's its own unitary glyph. This again makes it hard to assign features based on glyphs.

Our solution was to run NFD and then selectively re-compose ç. you can see that here:

dev/scripts/aggregation-helper-functions.R

Lines 148 to 159 in 68b8984

	denorm <- function(strings, fix_c_cedilla=TRUE) {
	## denormalize strings
	denormed <- stri_trans_general(strings, "Any-NFD")
	if (fix_c_cedilla) {
	# renormalize c-cedilla
	c_cedilla <- "\u00E7"
	c_cedilla_denormed <- "\u0063\u0327"
	denormed <- stri_replace_all_fixed(denormed, pattern=c_cedilla_denormed,
	replacement=c_cedilla)
	}
	denormed
	}

We then impose a consistent order on the diacritics and modifier letters, with this function:

dev/scripts/aggregation-helper-functions.R

Line 265 in 68b8984

order_ipa <- function(strings, keep_stars=FALSE, keep_brackets=TRUE) {

We then assign features based on lookup tables for each of the base glyphs, diacritics, and modifier letters.

That said, it's possible we're actually doing something wrong, or that there's a cleaner/faster way to do it! The code was written a long time ago and we're much better programmers now, but haven't revisited this code very recently. If after reviewing the code you still think there's a bug, or a case to be made for doing things differently, feel free to re-open this issue (or open a new one). For now I'll close as "intended behavior, not actually a bug".

[lars76]

After looking at the code, I see your point. But you could keep all your internal processing logic and just apply NFC normalization as a final step before writing the CSV. Or at minimum, document this in the README so users know what to expect.

I was matching phonemes from your dataset against other sources, and it took me a while to figure out why some identical-looking phonemes weren't matching. The mixed normalization was the culprit. Not sure how many other users run into this, but figured it was worth mentioning. Your call either way :)

[drammock]

But you could keep all your internal processing logic and just apply NFC normalization as a final step before writing the CSV. Or at minimum, document this in the README so users know what to expect.

good suggestions. I'll re-open, to remind us to revisit this question.