CLTS knows the "ultra-long" length, but pyclts only parses it for a closed list of vowels.

[XachaB]

Hi,

I saw that CLTS knows the "ultra-long" length:

https://github.com/cldf-clts/clts/blob/d52aa50ec524b590f7715334b58c89b839dc585b/pkg/transcriptionsystems/features.json#L65

However, pyclts only parses it for a closed list of vowels, which are coded extensively in the vowels file, e.g.:

https://github.com/cldf-clts/clts/blob/cccee296b1e54e653e1b4bea103bf0e870072765/pkg/transcriptionsystems/bipa/vowels.tsv#L52

I am working with Nuer right now, where morphological contrasts can combine tone, 3 levels of length, vowel quality, and breathyness; which leads to many attested combinations. When other diacritics than length are involved, pyclts incorrectly parses "ultra-long" as if it were "long":

import pyclts
clts = pyclts.CLTS()
o1 = clts.bipa["oːː"]
print(o1, o1.featuredict["duration"])
o2 = clts.bipa["o̤ːː"]
print(o2, o2.featuredict["duration"])
o3 = clts.bipa["ó̤ːː"]
print(o3, o3.featuredict["duration"])

outputs:

oːː ultra-long
o̤ː long
ó̤ː long

I saw that the ultra-long diacritic is not in the diacritics file:

https://github.com/cldf-clts/clts/blob/cccee296b1e54e653e1b4bea103bf0e870072765/pkg/transcriptionsystems/bipa/diacritics.tsv#L82

However, adding a row in that file with double "ː" is not enough, from which I would guess that the parser does not allow any combinations of sound + compatible diacritics.

Is this intended behavior ? I understand that for some applications, losing such fine sound resolution might not matter. For morphology, where I was hoping to use CLTS as a parser (in order to obtain featural definitions from grapheme sequences, and where I want to trust the data sources), the contrast between, for example, long and ultra-long, is sometimes crucial.

[XachaB]

Narrowing it down, it looks like the cause is the assumption that diacritics are a single character long here, when iterating on each remaining diacritic characters to check for a match (same thing with diacritics before the sound, a few lines above):

pyclts/src/pyclts/transcriptionsystem.py

Line 284 in c7420d9

for dia in [EMPTY + p for p in post]:

An alternative could be to segment diacritics using a regex based on the diacritics from the diacritics file.

[XachaB]

See PR #46 for an example of what parsing multiple character diacritics could look like.
If something like this is accepted, then adding a row for ultra-long to the bipa diacritics in CLTS would be the next step.

[XachaB]

@xrotwang, @LinguList : could someone have a look at this ?

[LinguList]

@XachaB, modifying one part of the parsing procedure always has larger consequences for other parts as well. I prefer to code the vowels with long diacritics extra, in our vowels.tsv file. This always works, and it has the advantage of not fiddling with the code.

[LinguList]

So one would only have to assemble all ultralong sounds (which should not be many) and add them to the vowel.tsv file in the clts package.

[XachaB]

I understand not wanting to touch the parser, for fear of breaking something else (though there are tests, and they do all pass).

Is there a more general reason to not allow combinations of diacritics which are more than one character long with their compatible sounds ? Currently the parser does allow the arbitrary combination of diacritics, as long as these diacritics are 1 character long. The distinction seems to me to be purely a detail of implementation, rather than a feature.

Arguing further in favor of allowing for all combinations, I would add that it could be possible for a language to display any combination of other vowel diacritics + this over-long length. Maintaining a closed list means that one needs to know in advance all of the possible graphemes which the users might want to use. For all cases but multi-char diacritics, CLTS is able to parse previously unseen, valid BIPA graphemes (that is combinations of C/V and compatible diacritics).

For my own purposes, for now, I only need vowels and consonants which exist in Nuer, Dinka or Estonian.

The vowels which need to be added for this are:

aːː
a̤ːː
e̤ːː
i̤ːː
o̤ːː
æ̤ːː
ɑːː
ɔːː
ɔ̤ːː
ə̤ːː
ɛ̤ːː
ɤːː
ṳːː

Estonian also has the overlong consonants:

fːː
hːː
kːː
lːː
mːː
nːː
pːː
rːː
sːː
ʃːː
tːː
vːː

But of course, these overlong sounds are not the only existing ones in the world. For example I think Wichita has ɪːː, and I imagine there are others which I do not know about.

[LinguList]

Sorry, did not see you made a PR. See my comment there. I can only get back to this next week, but will then do so and check, much likely I'll follow your proposal, but want to understand it in detail first.

[XachaB]

I see now that it doesn't pass all tests, I can check why and try to fix later on too. I'll check your comments there too

[XachaB]

Ok, the discussion on the PR concluded to avoid changing the parser. As a fix, I made two PRs above which introduce more ultra-long sounds, and breathy ultra-long sounds in the respective CLTS BIPA files.

Still, this does not completely solve the problem. For example, if a user used an alias for tone and specifies, e.g. "ó̤ːː" (Nuer has both tones and 3 levels of length, and breathyness), then this will still be mis-recognized as "ó̤ː".

I am hesitant, however, to propose all combinations of tones for all ultra-long Nuer vowels, breathy & non-breathy, as we are starting to get into a large number of sounds. I know for lingpy & lexibank you prefer noting tones after syllables using the number system; though writing tones using diacritic is also a widespread practice, and might make a lot of sense for some specific applications (I am again thinking of morphology, where it can be very useful to mark tone as a supra-segmental feature of the vowel).

How should we proceed ?

[xrotwang]

I'd say that a "widespread practice to write tones using diacritics" sounds a lot like the use case for shareable orthography profiles - i.e. transparent pre-processing of the data.