Morphological Analysis Libraries - Domain Explainer#

For: Business stakeholders, non-linguists, developers new to NLP Purpose: Understand what morphological analysis is and why it matters for language learning applications

What is Morphological Analysis?#

Morphological analysis breaks down words into their component parts and identifies grammatical features.

Simple Example (English)#

Word: “running”

• Lemma (base form): “run”
• Part of Speech: VERB
• Features: present participle, -ing form

Word: “cats”

• Lemma: “cat”
• Part of Speech: NOUN
• Features: plural, nominative case

Why It Matters for Language Learning#

Problem: You Can’t Learn Languages Word-by-Word#

When you see “Я читаю книгу” (Russian: “I’m reading a book”), you need to know:

• читаю is the verb “читать” (to read) in 1st person singular
• книгу is the noun “книга” (book) in accusative case (direct object)

A dictionary won’t help if you don’t know the grammatical forms.

Solution: Morphological Analysis#

Parse the sentence to identify:

1. Lemma (dictionary form) - so you can look it up
2. Part of speech - noun, verb, adjective, etc.
3. Grammatical features - case, tense, number, gender, etc.

This lets the trainer ask: “What case is книгу?” (Answer: accusative)

The Three Languages#

Japanese: 3 Scripts + Agglutinative Morphology#

Challenge: Multiple scripts in one sentence

私は本を読んでいます
(watashi wa hon wo yonde imasu)
"I am reading a book"

What we need:

• 私 (watashi) → pronoun, “I”
• は (wa) → particle (topic marker)
• 本 (hon) → noun, “book”
• を (wo) → particle (direct object marker)
• 読んでいます (yonde imasu) → verb “読む” (yomu, “to read”) + present progressive form

Key insight: Particles are critical for understanding Japanese grammar.

Russian: 6 Cases + Aspect System#

Challenge: Case inflection changes word endings

Я читаю книгу в библиотеке
"I'm reading a book in the library"

What we need:

• книгу → “книга” (book), accusative case (direct object)
• библиотеке → “библиотека” (library), prepositional case (location)
• читаю → “читать” (to read), imperfective aspect, present tense

Key insight: Same word has 12 forms (6 cases × 2 numbers). You must identify which form you’re seeing.

Czech: 7 Cases + Complex Declension#

Challenge: Even more complex than Russian

Čtu knihu v knihovně
"I'm reading a book in the library"

What we need:

• knihu → “kniha” (book), accusative case
• knihovně → “knihovna” (library), locative case (7th case!)
• Čtu → “číst” (to read), 1st person singular

Key insight: 7 cases (vs Russian’s 6) + complex declension patterns.

The Business Problem#

Without Morphological Analysis:#

❌ Can’t parse real text (only memorize vocabulary lists) ❌ Can’t identify grammar patterns in context ❌ Limited to artificial examples (not authentic texts) ❌ Slow learning (decontextualized grammar drills)

With Morphological Analysis:#

✅ Parse Caesar, Pushkin, Japanese manga ✅ Identify grammar in real sentences ✅ Progressive difficulty (i+1 principle) ✅ Context-based learning (proven more effective)

The Library Selection Problem#

Challenge: every analyzer is two products#

A morphological analyzer bundles two things that are usually built by different people on different schedules: a lexicon listing what a written form could be, and a disambiguator deciding which reading it is in this sentence. Some packages ship one, some ship the other, some ship both.

That split is what makes the shopping hard, because the two halves have different accuracy, different licenses, and different maintenance clocks — and the license and the release date you can look up on a package page describe the code, not the lexicon. The full comparison is in 01-discovery/S2-comprehensive/; the short version:

Japanese — the lexicon does the heavy lifting, because the language is written without spaces and the dictionary is what decides where words begin and end.

• MeCab (2003), reached from Python through fugashi or mecab-python3. The dictionary is chosen separately, and the default one most tutorials produce dates from 2013.
• SudachiPy — Rust-backed since version 0.6, with an Apache-2.0 dictionary that ships quarterly and three granularity settings.
• spaCy — a full pipeline that calls SudachiPy underneath; its Japanese models carry a share-alike license even though the library is MIT.

Russian — the string is already segmented, so the work is choosing among readings.

• pymorphy3 — a dictionary that lists every reading and ranks them without looking at the sentence. Excellent lemmas, weak on case for that reason.
• spaCy — a trained pipeline that commits to one reading using context.
• UDPipe, Stanza — the same idea with more accuracy and a licensing question attached.

Czech — the same shape as Russian, with a harder licensing picture.

• MorphoDiTa and UDPipe — the accuracy leaders, from Charles University, with models licensed for non-commercial use only.
• Stanza — commercially usable if you pin the right model package, which is not its default.
• spaCy — has no Czech pipeline at all, only a tokenizer and a lookup table.

Decision Framework:#

Expected Output Format#

Goal: JSONL#

{
  "sentence": "私は本を読んでいます",
  "words": [
    {"text": "私", "lemma": "私", "pos": "PRON", "reading": "わたし"},
    {"text": "は", "lemma": "は", "pos": "ADP", "type": "particle"},
    {"text": "本", "lemma": "本", "pos": "NOUN", "reading": "ほん"},
    {"text": "を", "lemma": "を", "pos": "ADP", "type": "particle"},
    {"text": "読んでいます", "lemma": "読む", "pos": "VERB", "tense": "present", "reading": "よんでいます"}
  ]
}

This feeds into:

• japanese-train: Interactive trainer
• japanese-analyze: Progress analysis

ROI: Why This Research Matters#

Time Investment:#

• Research: 8-12 hours (library selection)
• Implementation: 20-30 hours per language (parser + trainer)
• Total: ~100 hours for 3-language system

Value Created:#

• Reusable pattern across all future languages
• Context-based learning (proven effective)
• No ongoing costs (self-hosted, no API fees)
• Multi-language support (Japanese, Russian, Czech immediately)
• Extensible (add Korean, Arabic, etc. with same pattern)

Strategic Value:#

• Validates Path 1 (self-operated) for language learning
• Creates competitive moat (most language apps don’t parse real text)
• Enables progressive corpus training (i+1 principle)
• Foundation for polyglot learning system

Success Criteria#

For each language, we need:

1. ✅ Accurate morphological analysis (>90% lemma + POS accuracy)
2. ✅ pip installable (or minimal binary deps)
3. ✅ <100ms per sentence (real-time performance)
4. ✅ JSONL output format (consistent across languages)
5. ✅ Active maintenance (not abandoned projects)

Overall:

• Choose 1 library per language (Japanese, Russian, Czech)
• Prototype parser for each (japanese-parse, russian-parse, czech-parse)
• Validate with 10-20 real sentences per language
• Document trade-offs and limitations

Questions This Research Answers#

1. Japanese: MeCab vs SudachiPy vs spaCy?
2. Russian: pymorphy2 vs spaCy vs UDPipe?
3. Czech: UDPipe vs spaCy vs MorphoDiTa?
4. Unified API: Can we use spaCy for all three? Or per-language specialists?
5. Installation: Which libraries have minimal dependencies?
6. Performance: Which are fast enough for interactive training?
7. Accuracy: Which give correct morphological features?

What Happens After This Research#

Immediate (Research Complete):#

• Know which library to use for each language
• Understand installation + performance characteristics
• Clear decision: spaCy everywhere vs per-language specialists

Next Steps (Implementation):#

• Experiment 1.950: japanese-parse (MeCab/SudachiPy + JSONL)
• Experiment 1.951: russian-parse (pymorphy2/spaCy + JSONL)
• Experiment 1.952: czech-parse (UDPipe/spaCy + JSONL)

Long-term (Application):#

• japanese-train: Interactive Japanese grammar trainer
• russian-train: Interactive Russian grammar trainer
• czech-train: Interactive Czech grammar trainer
• Multi-language app: Unified language learning system

Glossary#

Morphological analysis: Breaking words into parts and identifying grammatical features

Lemma: Base form of word (dictionary entry) Example: “running” → “run”, “books” → “book”

Part of Speech (POS): Grammatical category (noun, verb, adjective, etc.)

Case: Grammatical role in sentence (nominative=subject, accusative=object, etc.) Russian/Czech have 6-7 cases; English has remnants (he/him, who/whom)

Aspect: Russian/Czech verbs have two forms:

• Imperfective: ongoing action (Я читаю - “I am reading”)
• Perfective: completed action (Я прочитал - “I have read”)

Particle: Japanese grammatical markers (は, が, を, に, etc.) Indicate grammatical relationships without inflecting words

Agglutinative (Japanese): Grammar added by stacking suffixes Example: 読む (read) → 読んでいます (reading now)

Fusional (Russian, Czech): Grammar encoded in word endings Example: книга (book) → книгу (book-ACC), книге (book-PREP)

JSONL: JSON Lines format - one JSON object per line, easy to stream

S1: Rapid Discovery - Approach#

Methodology: Rapid Library Search (speed-focused) Time Box: 90-120 minutes maximum Goal: Identify viable libraries for Japanese, Russian, Czech morphological analysis

Core Philosophy#

Quickly map the solution space for each language:

• What libraries exist?
• Which are popular/actively maintained?
• Can they produce JSONL output?
• Is there a unified solution (spaCy) or per-language specialists?

Discovery Process#

1. Japanese Libraries (25 min)#

• MeCab: Classic Japanese morphological analyzer (2003)
• fugashi: Modern Python wrapper for MeCab
• SudachiPy: Modern Japanese tokenizer (2017)
• spaCy: General NLP with Japanese model

Check:

• PyPI downloads, GitHub stars
• Installation complexity (system dependencies?)
• Output format (can we get lemma + POS + reading?)

2. Russian Libraries (25 min)#

• pymorphy2: Morphology specialist for Russian
• spaCy: General NLP with Russian model
• UDPipe: Universal Dependencies parser

Check:

• Accuracy for case + aspect identification
• Performance (<100ms per sentence?)
• API usability

3. Czech Libraries (25 min)#

• UDPipe: Universal Dependencies (best Czech support)
• spaCy: Experimental Czech model
• MorphoDiTa: Czech-specific morphological analyzer

Check:

• 7 cases correctly identified?
• Installation ease
• Active maintenance

4. Unified vs Specialist Decision (15 min)#

• Option A: spaCy for all three (unified API)
• Option B: Per-language specialists (MeCab, pymorphy2, UDPipe)
• Option C: Hybrid (spaCy where good, specialists where needed)

Trade-offs:

• API consistency vs accuracy
• Installation complexity vs performance
• Maintenance burden vs optimal results

5. Quick Recommendation (10 min)#

• Default library per language
• Confidence level
• When to reconsider (S2/S3 signals)

Evaluation Criteria#

Primary:

• Popularity (PyPI downloads, GitHub stars)
• Active maintenance (last commit, open issues)
• Installation ease (pip install vs binary deps)

Secondary:

• Accuracy (if easily testable)
• Performance (if documented)
• Documentation quality

Tertiary:

• API consistency across languages
• Community size

Output Files#

• approach.md (this file)
• japanese-libraries.md - MeCab vs SudachiPy vs spaCy
• russian-libraries.md - pymorphy2 vs spaCy vs UDPipe
• czech-libraries.md - UDPipe vs spaCy vs MorphoDiTa
• recommendation.md - Per-language recommendations + unified decision

Success Criteria#

• Found 2-3 viable options per language
• Clear popularity leader identified (or not)
• Can answer: “What should I use for each language?”
• Total time: <120 minutes

Note for S2/S3#

S1 identifies viable options. If no clear winner per language:

• S2: Deep-dive accuracy testing, API comparison
• S3: Validate against real Japanese/Russian/Czech texts

This research likely needs more than S1 (3 languages, complex trade-offs).

Czech Morphological Analysis Libraries#

Winner: UDPipe#

PyPI Package: ufal-udpipe Downloads: 52,308/month Latest Update: June 2024 (improved Czech support) Maintenance: Active (Charles University) Python Requirement: 3.x

Best Czech Support#

• Academic backing (Charles University Prague)
• June 2024 improvements: 50% error reduction in lemmatization, 58% in POS tagging
• Universal Dependencies format
• 7 cases correctly identified

Key Features#

• Morphological dictionary-supplemented deep learning
• Lemmatization, POS tagging, dependency parsing
• Trained on PDT-C 1.0 (Prague Dependency Treebank)
• Web service + Python client

Installation#

pip install ufal-udpipe

Alternative: spaCy Czech#

Models: Experimental Czech support Maintenance: Active (spaCy ecosystem)

Why Consider#

• ✅ Unified API (same as other languages)
• ✅ spaCy ecosystem familiarity

Why Not Winner#

• ⚠️ Experimental (not mature)
• ⚠️ UDPipe has better Czech accuracy (50-58% error reduction)
• ⚠️ UDPipe is Czech-specialist

Alternative: MorphoDiTa#

Source: Czech-specific morphological analyzer Maintenance: Academic

Why Consider#

• ✅ Czech-specific tool
• ✅ Academic backing

Why Not Winner#

• ⚠️ Harder to find Python bindings
• ⚠️ UDPipe supersedes it (June 2024 paper shows improvements)
• ⚠️ Less popular in Python ecosystem

Recommendation#

Use UDPipe for czech-parse implementation.

Rationale:

1. Best Czech support (June 2024 improvements)
2. 50% lemmatization error reduction vs alternatives
3. 7 cases correctly handled
4. Python client available
5. Universal Dependencies format (standard)

Caveat:

• Lower adoption (52K downloads/month vs 585K for Russian, 1.9M for Japanese)
• But this reflects Czech being smaller language community
• Still viable and actively maintained

Confidence: MEDIUM-HIGH (7/10)

• High confidence in quality (academic backing, recent improvements)
• Medium confidence in ecosystem (lower adoption than Japanese/Russian)

Sources#

• UDPipe PyPI
• UDPipe Web Service
• June 2024 Paper - Improved Czech support
• UDPipe 2
• spaCy Czech - Experimental

Japanese Morphological Analysis Libraries#

Clear Winner: SudachiPy#

PyPI Package: SudachiPy Downloads: 1,936,812/month Latest Version: 0.6.10 (Jan 2025) Maintenance: Active (Nov 2024 release) Python Requirement: 3.x

Popularity Winner#

• 2.7× more downloads than fugashi (1.9M vs 720K/month)
• Modern development (0.6+ uses Sudachi.rs)
• Multi-granular tokenization (A/B/C split modes)

Key Features#

• Morpheme information (dictionary forms, readings, POS)
• Built-in dictionaries (sudachidict_small/core/full)
• Active development by Works Applications

Installation#

pip install SudachiPy
# Dictionary included by default (sudachidict_core)

Alternative: fugashi (MeCab wrapper)#

PyPI Package: fugashi Downloads: 720,141/month Latest Version: 1.5.2 Maintenance: Active

Why Consider#

• ✅ Pythonic MeCab wrapper (Cython-based)
• ✅ Classic MeCab reliability (2003+)
• ✅ UniDic dictionary support

Why Not Winner#

• ⚠️ 2.7× fewer downloads than SudachiPy
• ⚠️ Requires dictionary installation (unidic-lite or unidic)
• ⚠️ MeCab is older technology (SudachiPy more modern)

Alternative: spaCy Japanese (ja_core)#

Models: ja_core_news_sm/md/lg Maintenance: Active (spaCy ecosystem)

Why Consider#

• ✅ Unified API (same as other languages)
• ✅ Full NLP pipeline (not just morphology)

Why Not Winner#

• ⚠️ Uses SudachiPy internally for tokenization
• ⚠️ Heavier (full NLP vs morphology specialist)
• ⚠️ Morphology = subset of what SudachiPy provides

Recommendation#

Use SudachiPy for japanese-parse implementation.

Rationale:

1. Clear popularity leader (1.9M downloads/month)
2. Modern, actively maintained (2024-2025 releases)
3. Multi-granular tokenization (flexible parsing)
4. spaCy uses it internally anyway

Confidence: HIGH (9/10)

Sources#

• SudachiPy PyPI
• SudachiPy GitHub
• fugashi PyPI
• spaCy Japanese Models

S1 Rapid Discovery - Recommendation#

Superseded 2026-08-25 by S2-S4. This pass concluded that S2/S3 were not needed. They were: four of the five claims below did not survive primary-source checking. spaCy has no Czech pipeline; UDPipe’s models are CC BY-NC-SA (non-commercial); pymorphy3’s dictionary was compiled once, in January 2022, and performs no contextual disambiguation; and fugashi is not the largest MeCab wrapper. The corrected recommendation is in ../S2-comprehensive/recommendation.md. This file is kept as the record of what was decided and when.

Time Spent: ~100 minutes Confidence Level: HIGH (Japanese, Russian), MEDIUM-HIGH (Czech)

Per-Language Winners#

Installation#

# Japanese
pip install SudachiPy

# Russian
pip install pymorphy3

# Czech
pip install ufal-udpipe

Key Findings#

Pattern 1: No Unified Solution#

Unlike 1.141 (FSRS) and 1.142 (genanki) where one library wins across the board, morphological analysis requires per-language specialists:

• SudachiPy: Japanese-specific
• pymorphy3: Russian-specific (+ Ukrainian)
• UDPipe: Multi-language but strongest for Czech

spaCy could provide unified API but uses these specialists internally anyway (e.g., spaCy ja_core uses SudachiPy).

Pattern 2: Popularity Varies by Language Community#

• Japanese: 1.9M downloads/month (large community, active NLP)
• Russian: 585K downloads/month (strong community)
• Czech: 52K downloads/month (smaller language community)

Lower Czech adoption reflects language community size, not tool quality.

Pattern 3: Morphology Specialists Win#

All three winners are morphology-focused (not general NLP):

• SudachiPy: Japanese morphology + tokenization
• pymorphy3: Russian morphology + inflection
• UDPipe: Multi-language morphology (Universal Dependencies)

Takeaway: Don’t use general NLP if you need deep morphological analysis.

Implementation Path#

Next Steps (Experiments)#

1.950-japanese-text-parser:

• Use SudachiPy for tokenization + morphology
• Output JSONL
• Extract: lemma, POS, reading, particles

1.951-russian-text-parser:

• Use pymorphy3 for morphology
• Output JSONL
• Extract: lemma, POS, case, aspect, gender

1.952-czech-text-parser:

• Use UDPipe for morphology
• Output JSONL
• Extract: lemma, POS, case (7 cases), number, gender

Application Implementation#

applications/language-learning/:

• src/japanese/japanese_train.py - Interactive trainer
• src/russian/russian_train.py - Interactive trainer
• src/czech/czech_train.py - Interactive trainer

Trade-offs Accepted#

Unified API vs Accuracy#

Decision: Choose accuracy (per-language specialists) Trade-off: 3 different APIs instead of unified spaCy API Rationale: Morphology quality critical for language learning

Installation Complexity#

Decision: 3 separate packages Trade-off: pip install × 3 vs pip install spacy + models Rationale: Simpler individual installs than managing spaCy models

Maintenance Burden#

Decision: Track 3 libraries instead of 1 Trade-off: Monitor 3 release cycles Rationale: All actively maintained (2024-2025 releases)

Confidence Assessment#

Japanese: HIGH (9/10)#

✅ Clear popularity leader (1.9M downloads/month) ✅ Active development (Jan 2025 release) ✅ Modern technology (Sudachi.rs) ✅ Multi-granular tokenization

Russian: HIGH (8/10)#

✅ Morphology specialist (not general NLP) ✅ Strong adoption (585K downloads/month) ✅ Actively maintained (successor to pymorphy2) ✅ Excellent case + aspect analysis

Only risk: pymorphy3 is newer (2.0.4) vs pymorphy2 (0.9.1 unmaintained)

Czech: MEDIUM-HIGH (7/10)#

✅ Best Czech accuracy (June 2024 improvements) ✅ Academic backing (Charles University) ✅ Universal Dependencies standard ⚠️ Lower adoption (52K downloads/month) ⚠️ Smaller language community

Risk: Lower community = fewer Stack Overflow answers, examples

When to Revisit This Decision#

Reconsider Japanese (SudachiPy):

• If SudachiPy development stalls (check GitHub activity)
• If spaCy ja_core significantly improves (check benchmarks)

Reconsider Russian (pymorphy3):

• If pymorphy3 becomes unmaintained (fork like pymorphy2?)
• If spaCy ru_core morphology matches pymorphy3 quality

Reconsider Czech (UDPipe):

• If spaCy Czech model matures (currently experimental)
• If Czech-specific library emerges with better adoption

General reconsideration signal:

• Unified spaCy API becomes compelling (if building 10+ languages)
• Per-language specialists become unmaintained

S2/S3 Not Required Because…#

S1 answered key questions:

1. ✅ Which libraries exist? (SudachiPy, pymorphy3, UDPipe)
2. ✅ Which are popular? (Clear download numbers)
3. ✅ Which are maintained? (All have 2024-2025 releases)
4. ✅ Is there unified solution? (No - per-language specialists win)

S2 would add (not needed):

• Detailed API comparison (all have morphology APIs)
• Accuracy benchmarks (pymorphy3/UDPipe papers already show this)
• Performance testing (<100ms/sentence likely for all)

S3 would add (not needed):

• Real text validation (popularity suggests they work)
• Integration prototypes (defer to experiments 1.950-1.952)

Decision: S1 sufficient - clear winners, high confidence

Hardware Store Philosophy#

“In Stock Now” (1.148 base):

• Japanese: SudachiPy ✅
• Russian: pymorphy3 ✅
• Czech: UDPipe ✅

“Catalog Entries” (1.148.X - LANGUAGE_FAMILIES_ROADMAP.md):

• Arabic, Chinese, Hebrew, ASL, Korean, Turkish, etc.
• Mapped but not researched
• Research when needed (user demand signal)

Pattern validated: Per-language specialists > unified general NLP for morphology-intensive tasks

Sources#

Japanese#

• SudachiPy PyPI Stats - 1.9M downloads/month
• fugashi PyPI Stats - 720K downloads/month
• spaCy Japanese Models

Russian#

• pymorphy3 PyPI Stats - 585K downloads/month
• pymorphy2 GitHub - unmaintained
• spaCy Russian Models

Czech#

• UDPipe PyPI Stats - 52K downloads/month
• June 2024 Paper - Czech improvements
• UDPipe Web Service

Russian Morphological Analysis Libraries#

Clear Winner: pymorphy3#

PyPI Package: pymorphy3 Downloads: 584,844/month Latest Version: 2.0.4 Maintenance: Active (successor to pymorphy2) Python Requirement: 3.9-3.14

Morphology Specialist#

• Dedicated Russian morphology (not general NLP)
• Successor to pymorphy2 (original is unmaintained)
• 585K downloads/month = strong adoption

Key Features#

• Case identification (6 cases: nominative, genitive, dative, accusative, instrumental, prepositional)
• Aspect analysis (perfective/imperfective)
• Gender, number, tense, person
• Inflection engine (generate forms from lemma)

Installation#

pip install pymorphy3

Alternative: spaCy Russian (ru_core)#

Models: ru_core_news_sm/md/lg Components: morphologizer, lemmatizer, parser Maintenance: Active

Why Consider#

• ✅ Unified API (same as other languages)
• ✅ Full NLP pipeline (NER, dependency parsing)
• ✅ Token.morph for morphological features

Why Not Winner#

• ⚠️ General NLP (not morphology specialist)
• ⚠️ pymorphy3 has deeper morphology analysis
• ⚠️ Trained on Nerus dataset (good but not specialized)

Alternative: UDPipe#

PyPI Package: ufal-udpipe Downloads: 52,308/month Maintenance: Active (v1 and v2)

Why Consider#

• ✅ Universal Dependencies format
• ✅ Multi-language support
• ✅ Academic backing (Charles University)

Why Not Winner#

• ⚠️ 11× fewer downloads than pymorphy3
• ⚠️ Czech is its strength, not Russian
• ⚠️ More complex setup than pymorphy3

Recommendation#

Use pymorphy3 for russian-parse implementation.

Rationale:

1. Morphology specialist (not general NLP)
2. Strong adoption (585K downloads/month)
3. Actively maintained (successor to pymorphy2)
4. Excellent case + aspect analysis (critical for Russian)
5. Simple API for morphological parsing

When to consider spaCy:

• If building unified multi-language parser with same API
• If need full NLP pipeline (NER, dependency parsing)

Confidence: HIGH (8/10)

Sources#

• pymorphy3 PyPI
• pymorphy2 GitHub - original (unmaintained)
• spaCy Russian Models
• UDPipe

S2: Comprehensive Analysis — Approach#

Methodology: Comprehensive Discovery (S2) — how these libraries are built, what they measure, and where the differences between them actually sit. All figures below were measured or read from a primary source on 2026-08-25.

Why S1’s organizing idea had to be replaced#

S1 filed this category by language: one file for Japanese, one for Russian, one for Czech, each ending in a winner. That is a filing system. It answers “what do people use for language X” and stops, which leaves a reader who arrives with a fourth language — or with a second requirement in the same language — nothing to reason with.

S2 tests three candidate spines against evidence and keeps the one the measurements support.

The spine that carries all three observations at once:

A morphological analyzer is two products in one package: a LEXICON that lists what a surface form could be, and a DISAMBIGUATOR that decides which reading it is. Libraries differ in whether they ship one half, the other, or both — and the two halves carry different accuracy, different licenses, different maintenance clocks, and different failure modes.

Everything measured below is a consequence of that split. The per-language filing that S1 used falls out of it: Japanese puts almost all of its difficulty in the lexicon half (where do words start and stop, and what is the dictionary form), while Czech and Russian put almost all of theirs in the disambiguator half (this string has four readings, choose).

What was measured, and how#

Gold data#

Universal Dependencies test splits, downloaded 2026-08-25 from the UD GitHub organization. Every treebank used here is CC BY-SA 4.0, so it can be quoted and redistributed:

UD_Czech-PDT and UD_Russian-SynTagRus were read for their license headers but not used as gold, because both are CC BY-NC-SA 4.0.

Protocol#

Gold tokenization was supplied wherever a tool accepts it. Feeding raw text would mix segmentation error into every score and make the tools incomparable. The one place segmentation is measured on purpose is the Japanese boundary experiment, where it is the subject.

Lemma comparison is case-insensitive, with ё folded to е for Russian. That folding matters: OpenCorpora writes ё where UD often writes е, and skipping the fold costs pymorphy3 about a point for no linguistic reason.

Where a tool’s tagset is not UD, an explicit mapping is stated in the file that uses it, and only for the one feature being scored:

• OpenCorpora → UD Case (pymorphy3): nomn→Nom, gent/gen1/gen2→Gen, datv→Dat, accs/acc2→Acc, ablt→Ins, loct/loc1/loc2→Loc, voct→Voc.
• PDT positional tag → UD Case (MorphoDiTa): tag position 5, digits 1..7 → Nom, Gen, Dat, Acc, Voc, Loc, Ins.

Both mappings are small, documented, and reversible. They are stated so that anyone disputing a number knows which step to dispute.

The measurement that matters most: oracle versus top-1#

For any tool that returns a list of readings, two numbers were taken:

• top-1 — the accuracy of whichever reading the tool puts first.
• oracle — whether the correct answer appears anywhere in the returned list.

The gap between them is the value of disambiguation, isolated from the value of the lexicon. A lexicon with a 99% oracle and a 50% top-1 is a fine lexicon attached to no decision procedure, and that turns out to describe several widely used packages.

Reproducibility check#

simplemma publishes its own per-language accuracy. Its README (read 2026-08-25) claims 0.95 for Czech measured on UD CS-FicTree. Measured here independently on the same treebank’s test split: 95.1%. An independent reproduction of a vendor’s own published figure is the cheapest available evidence that a measurement harness is sound, and it is the reason the other numbers in this pass are stated without hedging.

Versions under test#

Installed 2026-08-25 into a clean Python 3.12.3 environment:

Two services were called rather than installed, because their models are CC BY-NC-SA and downloading them locally would have added nothing the REST endpoint does not give:

• UDPipe 2 at lindat.mff.cuni.cz/services/udpipe — 961 models offered; the UD 2.17 set dated 251125 was used throughout.
• MorphoDiTa at lindat.mff.cuni.cz/services/morphodita — model czech-morfflex2.1-pdtc2.0-250909.

Both are shared academic services. Runs were capped at 200-250 sentences per model, which is enough to separate an eighteen-point difference from noise and small enough to be a polite guest.

What S2 does not claim#

• No cross-language accuracy comparison. “Czech scores 96% and Japanese scores 98%” would be meaningless: they are different tasks scored on different feature inventories. Comparisons here are always within one language, on one gold set.
• No end-to-end raw-text pipeline scores. Those exist on the vendors’ own pages and are reproduced where useful, but they conflate tokenizer, tagger, and lemmatizer into one number that hides which half failed.
• No claim that any figure here is state of the art. The small models were used throughout, because the question under test is where the differences between designs sit, and the small models make those differences legible without a GPU.

Output files#

Disambiguation-First Languages: Where Choosing Does the Work#

Czech and Russian arrive pre-segmented. Whitespace does the job that a lattice does in Japanese, and the lexicon lookup that follows is close to solved. What remains is that one written form usually corresponds to several grammatical readings, and nothing inside the word says which.

Czech ulici is dative, accusative, or locative singular of ulice. Russian стали is either the past plural of стать (to become) or five different cases of сталь (steel). Both are ordinary words in ordinary sentences.

The tools#

Downloads for the last 30 days, read 2026-08-25: pymorphy3 501,262; simplemma 217,440; pymorphy2 211,607; natasha 63,643; ufal-udpipe 24,984; ufal-morphodita 13,708.

S1 reported 584,844/month for pymorphy3 and 52,308 for ufal-udpipe. Both have fallen: pymorphy3 by 14%, ufal-udpipe by 52%. The UDPipe halving is the one to watch, and S4 takes it up.

Correction: spaCy has no Czech pipeline#

S1 states that spaCy has “experimental Czech support” and lists as a reconsideration trigger “if spaCy Czech model matures (currently experimental)”.

Checked against spaCy’s own compatibility.json on 2026-08-25, the languages with trained pipeline packages for spaCy 3.8 are:

ca, da, de, el, en, es, fi, fr, hr, it, ja, ko, lt, mk, nb, nl, pl, pt, ro, ru, sl, sv, uk, xx, zh

Twenty-five entries. There is no cs. What spaCy ships for Czech is spacy/lang/cs, containing four files — __init__.py, examples.py, lex_attrs.py, stop_words.py — which give a tokenizer and stop-word list and nothing else. A lemma lookup table exists separately in spacy-lookups-data (cs_lemma_lookup.json, 35,021 entries).

Measured on gold tokens, 2026-08-25:

A reader waiting for the Czech spaCy model to mature is waiting for something that does not exist as a project. The Czech options are UFAL’s tools, Stanza, UDPipe, or a lookup table — and the free lookup table beats spaCy’s by fifteen to eighteen points.

Measurement 1: what disambiguation buys, in one tool#

MorphoDiTa exposes both halves through the same model file: analyze returns every reading MorfFlex knows, tag returns the perceptron’s choice. That makes it the cleanest available experiment, because nothing changes between the two runs except whether a disambiguator was applied.

UD_Czech-CAC test, first 250 sentences (3,520 tokens), gold tokenization, czech-morfflex2.1-pdtc2.0-250909, measured 2026-08-25:

The lexicon is close to complete: for 999 tokens in 1,000 the correct case is sitting in the candidate list. The perceptron recovers 93.6 of the available 99.8. Picking arbitrarily recovers 48.5.

MorfFlex’s analyze output is in dictionary order rather than probability order, so 48.5% is a floor rather than a fair baseline for a well-ordered lexicon. The next measurement supplies the fair version.

Measurement 2: what a ranked lexicon leaves on the table#

pymorphy3 ranks its output using a P(t|w) prior estimated from a tagged corpus. It does not consult context — the documented design is that the analyzer returns all parses ordered by estimated probability and leaves homonymy to the caller.

Gold tokenization, restricted to tokens where pymorphy3 proposes at least one case, measured 2026-08-25:

Unrestricted, over every token carrying a gold Case feature: pymorphy3 top-1 scores 70.0% on GSD and 74.6% on Taiga, and the correct case is missing from every candidate on 11.4% of GSD tokens against 0.5% on Taiga.

That 11.4% is an annotation-convention artifact rather than a lexicon gap, and diagnosing it earns a paragraph, because the same artifact contaminates any naive cross-tool comparison in this field. The missing-candidate buckets on GSD are dominated by:

• Digits carrying a case in UD — 2013, 5, 15 annotated ADJ or NUM with Case=Gen. pymorphy3 returns a numeral analysis with no case. 248 tokens across the top three buckets.
• Short-form participles — приглашён-type forms that UD annotates VERB, Case=Nom and OpenCorpora tags PRTS with no case at all. 115 tokens.

Neither is an error by either party. They are two tagsets disagreeing about whether a number has a case, and the “accuracy” number moves eight points depending on which one you call correct. The restricted table above is the one to quote.

The mechanism behind the number#

Read from the loaded pymorphy3-dicts-ru metadata, 2026-08-25:

The ranking prior covers 45,471 words against a 5.1-million-form lexicon. For any token outside that set, the ordering carries no evidence about which reading is likelier, and the top-1 score is close to a coin flip weighted by paradigm shape. That is a 78.7% dressed as a ranked answer.

Measurement 3: lemma and features come apart#

UD_Russian-GSD test, all 11,385 tokens, gold tokenization, measured 2026-08-25:

pymorphy3 also emits 3.72 analyses per token overall, with 50.2% of tokens ambiguous.

The context-free lexicon wins the lemma column by two points and loses the case column by fifteen. Both facts are stable across two treebanks in two registers. So:

“Which Russian morphology library is more accurate” is not a question with an answer. It becomes one only after naming the output column, and the answer flips between columns.

Czech shows the same shape from a different codebase: disambiguation buys 6.3 points of lemma and 45.1 points of case. Lemmas are mostly decided by the form; features are mostly decided by the sentence.

An intuition that makes this stick: книги is unambiguously a form of книга — all three of its readings share the lemma — so a tool that never looks at context still gets the lemma right. Which of the three readings it is depends on the verb four tokens away.

What the cheap tools are worth#

simplemma is a 19 MB install with no per-language download, no model, and no context. Measured against tools hundreds of times its size:

Three points behind MorphoDiTa on Czech lemmas, two behind pymorphy3 on Russian lemmas, and ahead of spaCy’s Russian pipeline. At 49,174 tokens/sec on Czech and 12,527 on Russian, with 54 languages in one package.

The boundary is sharp and follows the spine exactly: for lemmas, the cheap tool is close; for features, there is no cheap tool. simplemma emits no case, no gender, no number, and no candidate list, so the 45-point Czech case gap is not something it can be pointed at.

simplemma’s README is candid about what its own benchmark measures: “The benchmark only incidentally captures what this library is most useful for, the lemmatization of less frequent words.”

The Russian options S1 did not reach#

natasha / slovnet (MIT) is a Russian-specific stack — razdel for tokenization, slovnet for tagging and NER, navec for embeddings — designed around compact models. natasha last released 2023-07-24 with 63,643 downloads/month; slovnet last released 2023-01-23 and its repository was last pushed 2023-07-24. The stall test on natasha reads 7 open PRs against 0 merged since 2026-02-01. Not archived, not dead, but not moving.

pymystem3 wraps Yandex’s Mystem 3.1, which performs POS disambiguation — the capability pymorphy3 lacks. Its README states the licensing position without ambiguity:

This wrapper is open sources under MIT license. However, please consider that the Yandex Mystem is not open source and licensed under conditions of the Yandex License.

The wrapper’s last release is 2018-07-04 and the wrapped binary is from June 2014. It appears in this survey as a documented instance of the pattern — MIT wrapper, non-open payload — rather than as a live recommendation.

RNNMorph (Apache-2.0, last release 2021-10-14) is a neural Russian morphology tagger, effectively frozen.

Repository health#

Read from the GitHub API on 2026-08-25.

pymorphy2 with six open PRs and nothing merged since February is the textbook stall, and it is why pymorphy3 exists — the fork carries the Python-version fixes the original stopped taking. pymorphy3’s own zero-and-zero is not a stall, because there is no queue.

simplemma and Stanza are the two healthiest projects in the Slavic column by this measure, and neither was in S1.

Summary of measured findings#

1. Enumeration is solved; choosing is not. MorfFlex holds the correct Czech case for 99.8% of tokens; the tagger recovers 93.6%; dictionary order recovers 48.5%.
2. Disambiguation is worth +45.1 points on Czech case and +6.3 on Czech lemma, from the same lexicon.
3. pymorphy3 leaves ~20 points of Russian case on the table (78.7% top-1 against a 98.4% oracle), because its ranking prior covers 45,471 words against a 5.1M-form lexicon.
4. On Russian lemmas, pymorphy3 beats spaCy 95.5% to 93.5%, and loses on case 93.5% to 78.7%. The library ranking inverts between output columns.
5. spaCy has no Czech pipeline. S1’s “experimental Czech model” does not exist; what exists is a 35k-entry lookup table scoring 75.6-80.3%.
6. simplemma lands within 2-3 points of the heavy tools on lemmas at 19 MB and ~49k tokens/sec, and offers nothing at all on features.
7. pymorphy3-dicts-ru was compiled on 2022-01-08 and has never been rebuilt. See lexicon-licensing.md and S4.

Footprint and Throughput#

The two halves of an analyzer also have wildly different costs, and the expensive half is usually the data. This file records what each option costs on disk, in process memory, and in tokens per second, so that a reader sizing a container or a mobile bundle has numbers rather than impressions.

All figures measured 2026-08-25 on one Linux machine (WSL2, Python 3.12.3), single process, single thread. Absolute throughput is machine-specific; the ratios are the useful part.

Disk#

Measured with du -sh on the installed package directories in a clean virtual environment.

Three observations a reader can act on:

The Japanese dictionary is the deployment. The smallest working Japanese setup in this survey is SudachiPy plus SudachiDict-small at roughly 120 MB, and the code half of that is a few megabytes. Choosing core over small costs 90 MB and buys compound coverage; choosing full costs another 136 MB and buys proper nouns. Those are the numbers behind the segmentation differences in segmentation-first.md.

Slavic is an order of magnitude cheaper. pymorphy3’s complete Russian lexicon is 16 MB — barely larger than spaCy’s smallest Russian pipeline and seven times smaller than the smallest Japanese dictionary. Inflectional paradigms compress; a segmentation lexicon has to list surface strings.

spaCy’s sm models are small because they carry no vectors. ru_core_news_sm at 14 MB has "vectors": 0; lg at 489 MB has 500,002. If the downstream use is morphology rather than similarity, the 35× size difference buys 0.08 points of UPOS and 0.19 points of morph accuracy, per the models’ own published metadata (UPOS 0.9876 → 0.9884; morph 0.9710 → 0.9728). The vectors are the product, and morphology is not what they are for.

Throughput#

Measured on the same text in one process, after warm-up, with gold tokenization where the tool accepts it.

Japanese — 543 sentences, 13,034 tokens, raw text in#

Sudachi’s mode B and C are faster than mode A because they emit fewer tokens and do less work materializing results, not because the lattice search is cheaper.

Janome’s position is the expected one for a pure-Python implementation and is its documented trade: no compiler, no system dependency, no wheel-platform matrix, at roughly a thirteenth of Sudachi’s mode-A speed and a thirty-fifth of its mode-B speed. For a script processing a few thousand sentences, that is invisible. For a batch job over a corpus, it is the difference between minutes and hours.

Russian and Czech — gold tokens in#

The two simplemma Czech figures differ by 15× on the same code because of cache behavior: the FicTree run followed the CAC run in the same process, and simplemma memoizes. Both numbers are reported rather than one, because the gap is a fair warning that lemmatizer throughput benchmarks are dominated by cache state and should be read as an order of magnitude rather than a measurement.

pymorphy3 returns every reading at 17,230 tokens/sec — 3.72 analyses per token, so roughly 64,000 analyses per second — while spaCy’s pipeline returns one reading at 2,167. An 8× throughput ratio for a tool doing more work, because one is a DAWG lookup and the other is a neural tagger plus parser plus NER.

Disabling spaCy components you do not need is the obvious lever and was not measured here; the ru_core_news_sm pipeline as loaded is tok2vec, morphologizer, parser, attribute_ruler, lemmatizer, ner, and only the first two and the lemmatizer are morphology.

Startup cost#

Not benchmarked systematically, but two effects are large enough to name:

• pymorphy3’s MorphAnalyzer() loads a 16 MB DAWG. Instantiating it per request in a web handler is a mistake that shows up as latency rather than as an error. One analyzer per process, reused, is the intended pattern.
• Sudachi’s dictionary loads memory-mapped. Dictionary(dict='full').create() does not read 344 MB into the heap, but the resident set grows as the lattice touches pages, and a container memory limit sized on startup RSS will be wrong under load.

Sizing rules of thumb#

The cost of the accuracy you are buying#

Putting the two dimensions together for Russian, which has the fullest data in this survey:

The first three rows are within 5 MB of each other and span fifteen points of case accuracy and eight times the throughput. Size is not the axis that separates these options; what they compute is.

The fourth row is the accuracy ceiling in this table and is CC BY-NC-SA. That is the shape of the whole category in four lines.

Where the Code License and the Data License Part Company#

Every license claim in this file was read from a primary source on 2026-08-25: a LICENSE, LEGAL, or COPYING file in the project’s own repository, the project’s own model page, the PyPI JSON metadata, or the treebank’s own README.md. Nothing here is inferred from a badge.

The finding is uniform enough to state as a rule:

In this category, the PyPI license field describes the wrapper. The thing you are actually installing — the lexicon or the model — is usually under different terms, and the wrapper is usually the more permissive of the two.

That matters because a morphological analyzer is not a library you call and forget. The lexicon is a database that ships inside your artifact, and several of the most recommended options in this field are non-commercial.

The full picture#

The three shapes this takes#

1. Permissive code, non-commercial data#

The two UFAL tools state it in their own READMEs, in nearly identical words.

UDPipe:

UDPipe is a free software distributed under the Mozilla Public License 2.0 and the linguistic models are free for non-commercial use and distributed under the CC BY-NC-SA license, although for some models the original data used to create the model may impose additional licensing conditions.

MorphoDiTa:

MorphoDiTa is a free software under Mozilla Public License 2.0 and the linguistic models are free for non-commercial use and distributed under CC BY-NC-SA license, although for some models the original data used to create the model may impose additional licensing conditions.

This is not a footnote on an obscure model. Reading the UDPipe 2 models page, every published model set carries it — UD 2.17, 2.15, 2.12, 2.10, 2.6, the Czech PDT-C 1.0 and 2.0 models, and the EvaLatin 2020 models. There is no permissive UDPipe model set.

S1 recommended ufal-udpipe as its Czech answer, with a rationale built on accuracy, academic backing, and Universal Dependencies conformance, and did not mention the license. Everything S1 said about UDPipe’s quality is defensible. The recommendation is unusable for a commercial product, and a reader following S1 into a shipping product would find that out later than they should have.

The nuance that keeps this fair: the code is MPL-2.0, which is a file-level copyleft compatible with proprietary distribution, and pip install ufal.udpipe installs only the code. The restriction attaches at the moment you download a model, which is a separate, manual step that no requirements.txt records.

2. Permissive library, copyleft model#

spaCy is MIT. Its Japanese pipelines are not.

From ja_core_news_lg-3.8.0.json and ja_core_news_sm-3.8.0.json:

• "license": "CC BY-SA 4.0"
• sources: UD Japanese GSD v2.8 (CC BY-SA 4.0), UD Japanese GSD v2.8 NER (CC BY-SA 4.0), and for lg, chiVe embeddings (Apache-2.0)

And ja_core_news_trf-3.8.0.json is "license": "CC BY-SA 3.0", inherited from cl-tohoku/bert-base-japanese-char-v2.

By contrast ru_core_news_sm and ru_core_news_lg are "license": "MIT", drawing on Nerus and Navec, both MIT. uk_core_news_lg is MIT with CC0 vectors.

So within one MIT library, from one vendor, in one release: the Russian model is MIT and the Japanese model is share-alike. A team that adopted spaCy on the strength of “spaCy is MIT” and then added Japanese has changed its obligations without changing a dependency policy.

CC BY-SA on a model is an unsettled question — whether a trained artifact is a derivative of its training data has no clean answer in most jurisdictions — but spaCy has answered it for its own artifacts by stamping the field, and a downstream consumer inherits that declaration.

3. The maintainers say they do not know#

Stanza’s code is Apache-2.0; the LICENSE file is the standard Apache text with a Stanford copyright line. Its models are a different matter, and Stanza’s own performance page says so:

License information for models built from the UD data is unclear, but users are encouraged to click on the git links below and check the license for the relevant data if they are unsure of the licenses for the data used to build these models. To the extent that The Trustees of Leland Stanford Junior University have ownership and rights over these language packs, all these Stanza language packs are made available under the Open Data Commons Attribution License v1.0.

That is a careful and creditable disclosure, and it puts the work on the user. Doing the work for the three languages in this survey, using each treebank’s own README:

Both of Stanza’s Slavic defaults trace to non-commercial data. Both have a CC BY-SA sibling one package string away.

stanza.Pipeline('cs')                    # -> pdt, CC BY-NC-SA lineage
stanza.Pipeline('cs', package='cac')     # -> CC BY-SA lineage

The previous file measured what that string does to output shape. It also decides the license, and it is the same string.

The instructive case: one build script, two licenses#

pymorphy3-dicts is a single repository holding a single update.py. Its README describes what that script does:

For Russian it downloads data from http://opencorpora.org, compiles the dictionary using build-dict.py script and creates pymorphy3-dicts-ru package using cookiecutter. For Ukrainian it downloads LanguageTool data from Google Drive, converts dictionary to OpenCorpora format using LT3OpenCorpora, then compiles it and creates pymorphy3-dicts-uk package.

The repository’s LICENSE is MIT, and its README says “License is MIT” — both describing the scripts.

The packages the scripts produce declare, on PyPI:

• pymorphy3-dicts-ru — MIT license
• pymorphy3-dicts-uk — GPLv3 License

Same tooling, same author, same release cycle; different upstream lexicon owner, different license. Nothing in pymorphy3’s own MIT declaration predicts either one. A team that reads the analyzer’s license and adds Ukrainian support has taken on GPLv3.

One caution against over-reading this, because the survey should be careful where the evidence is: pymorphy3-dicts-ru’s MIT declaration covers the packaging, and the OpenCorpora dictionary it was built from carries its own terms. The opencorpora.org download page returned HTTP 521 when queried on 2026-08-25, so the upstream terms could not be verified from the primary source and are recorded here as unverified rather than restated from memory. What is verified is that the Ukrainian sibling, built by the same script from a different upstream, declares GPLv3 — which is the point being made either way.

Japanese: the disclosure that reads like a license#

SudachiDict is the cleanest large lexicon in this survey. The repository ships LICENSE-2.0.txt and README states:

SudachiDict by Works Applications Co., Ltd. is licensed under the Apache License, Version2.0 … This project includes UniDic and a part of NEologd.

The LEGAL file then does something more useful than most projects manage: it enumerates the provenance of the incorporated material. UniDic content in small_lex.csv and matrix.def.zip under the UniDic Consortium’s BSD-3-Clause terms, and NEologd-derived content in core_lex.csv and notcore_lex.csv whose own upstream is traced through five sources — Hatena keyword listings, Japan Post postal code data, a station-name list, a personal-name dataset, and web-crawled readings — each with an attribution note in Japanese.

Works Applications has taken the position that the assembled result is Apache-2.0 and has published the chain that a reader would need to check it. That combination is rare and is a large part of why the Japanese column of this survey has an unencumbered answer while the Czech column does not.

unidic-lite is the counterexample within the same language. Its PyPI license field reads “MIT License”. Its repository README says “This code is licensed under the MIT or WTFPL license, as you prefer. Unidic 2.1.2 is copyright the UniDic Consortium and distributed under the terms of the BSD license.” But the COPYING file that ships inside the installed package says something broader:

unidic-mecab is copyrighted free software by the UniDic Consortium, and is released under any of the GPL (see the file GPL), the LGPL (see the file LGPL), or the BSD License (see the file BSD).

Three license files — GPL, LGPL, BSD — are installed alongside a 249 MB dicdir. A recipient may select BSD, so nothing here blocks commercial use. What it blocks is a license-scanning tool: pip-licenses reports “MIT” for a package whose data payload is 99% of its bytes and is triple-licensed by a third party.

ipadic is worse and admits it. PyPI records no license classifier and no license field. The shipped COPYING is a bespoke NAIST grant that incorporates ICOT Free Software conditions, with clauses requiring a “NO WARRANTY” section to “ALWAYS appear on, or be attached to, the Program”. It is permissive in intent and is not an OSI-approved license, so it fails any policy written as an allowlist. Janome bundles it in the wheel.

Practical readings#

If you are building a commercial product, the encumbered set is: UDPipe models (all of them), MorphoDiTa models, Stanza’s Czech and Russian defaults, and anything downstream of UD_Czech-PDT, UD_Czech-FicTree, or UD_Russian-SynTagRus. The unencumbered set is: SudachiPy + SudachiDict (Apache-2.0), fugashi + unidic (MIT + BSD), pymorphy3 + -dicts-ru (MIT), simplemma (MIT), spaCy’s Russian and Ukrainian pipelines (MIT), and Stanza with a CC BY-SA package string pinned.

If share-alike is acceptable but non-commercial is not — the common position for an open-source project — spaCy’s Japanese models, Stanza with cac/gsd/taiga, and anything from a CC BY-SA treebank are all available, and the entire UFAL model catalog still is not.

If you are inside a university or doing research, the CC BY-NC-SA restriction is close to invisible and UFAL’s tools are the strongest Czech option in this survey by a wide margin. The restriction bites at the transition to a product, which is exactly when it is most expensive to discover.

In every case, record the model identity, not the package name. A pip freeze records stanza==1.14.0 and ufal.udpipe==1.4.0.1 and tells a future auditor nothing about syntagrus or czech-pdtc-ud-2.17-251125. The license lives on the model, and the model is not in the lockfile.

What could not be verified#

• OpenCorpora’s own dictionary terms. opencorpora.org/?page=downloads returned HTTP 521 on 2026-08-25. pymorphy3-dicts-ru declares MIT; whether that declaration is downstream-complete is not established here.
• Whether a trained model is a derivative work of its training corpus in any given jurisdiction. This survey reports what each project declares and does not offer a legal conclusion.
• Per-model licenses for Vaporetto and nagisa, whose model files were not downloaded.

S2 Recommendation#

S2 replaced S1’s organizing idea, corrected four of its claims, and produced a decision procedure that does not depend on which language the reader arrived with.

The finding#

A morphological analyzer is two products in one package: a lexicon that says what a form could be, and a disambiguator that says which reading it is. Which half you are buying decides the accuracy you get, the license you inherit, the maintenance clock you are on, and the failure mode you will debug.

Three measurements carry it:

1. The lexicon is nearly complete and choosing is where accuracy lives. MorfFlex holds the correct Czech case for 99.8% of tokens. Attaching MorphoDiTa’s perceptron to that same lexicon moves case accuracy from 48.5% to 93.6% — and moves lemma accuracy only from 90.0% to 96.3%.

2. Lemma and features are different problems, and the library ranking inverts between them. On UD_Russian-GSD, pymorphy3 beats spaCy on lemmas 95.5% to 93.5% and loses on case 78.7% to 93.5%. “Which library is more accurate” has no answer until the question names an output column.

3. What the two halves even are differs by language. UD Japanese annotates a morphological feature on 0.8% of tokens using one key; UD Czech uses twenty-eight keys on 77% of tokens. Japanese puts its difficulty in the lexicon (segmentation, normalized_form); Czech and Russian put theirs in the choice. simplemma covers 54 languages and cannot cover Japanese, because a form-keyed table needs a form.

Does the 1.140 pattern hold?#

Yes, and larger than in 1.140 — but by a different mechanism.

Cross-evaluating UDPipe 2 models on fixed text with gold tokenization, the package chosen inside one library moves UFeats by 11.4 to 19.2 points across four language and register settings, against 0.6 to 1.4 points for lemma. 1.140 measured 25 points on Latin nouns; the effect is the same order and lands on the feature bundle rather than the lemma.

The mechanism differs in a way that matters more than the size. In 1.140 the wrong package produced a wrong answer — poetae parsed as the verb possum, visible to anyone reading the output. Here the mismatched model produces a differently shaped answer: agreement on the feature keys both models emit is 97.3-99.0% regardless of match, and for Russian on GSD gold the mismatched models are marginally more accurate on what they emit. The entire 18-point gap is which keys appear at all.

That failure is silent. token.morph.get("Animacy") returns empty, nothing raises, and accuracy metrics stay flat. Assert on the key set, not on accuracy.

Corrections to S1#

S1’s per-language download figures have also moved: SudachiPy 1,936,812 → 1,906,682 (flat), pymorphy3 584,844 → 501,262 (−14%), ufal-udpipe 52,308 → 24,984 (−52%).

The decision procedure#

Answer two questions before choosing anything.

1. Does your language need segmentation before analysis?#

Yes (Japanese, Chinese, Thai; also relevant for Arabic and Hebrew clitics): the analyzer and the dictionary are one purchase. Pick the dictionary edition and split granularity first; the library follows.

• SudachiPy + SudachiDict — Apache-2.0 throughout, quarterly lexicon releases, four times faster than the alternatives, and normalized_form for spelling and script variants. The default recommendation.
• fugashi + unidic — scores highest against UniDic-convention gold (99.0% boundary F1), which is the right choice when your downstream data or annotations also follow UniDic. Install unidic, not unidic-lite: the default is UniDic 2.1.2 from 2013.
• Janome — when a pure-Python dependency is worth 5.8 points of boundary F1 against SudachiPy mode A, and a 13× slowdown against it (35× against mode B). Rarely.

No (Czech, Russian, Polish, and most of Europe): go to question 2.

2. Do you need lemmas, or do you need features?#

Lemmas only — for a search index, a frequency list, a vocabulary tracker:

• simplemma (MIT, 19 MB, 54 languages) is within 2-3 points of tools a hundred times its size: 95.1% Czech, 93.8% Russian. Start here.
• pymorphy3 (MIT) is two points better on Russian lemmas than anything else measured, at 17,230 tokens/sec.

Features — case, gender, animacy, aspect:

• You need a disambiguator. Nothing cheap does this; the 45-point Czech gap is not something a lookup table can be pointed at.
• Commercially unencumbered: spaCy ru_core_news_* (MIT) for Russian; Stanza with an explicitly pinned CC BY-SA package (ru → gsd or taiga, cs → cac or cltt).
• Research or internal use: UDPipe 2 and MorphoDiTa are the accuracy leaders for Czech and are CC BY-NC-SA. UDPipe 2’s russian-gsd reached 97.1% case and 94.4% UFeats on encyclopedic prose.
• Both: run the lexicon and the tagger together. MorphoDiTa’s analyze gives you the candidate set for a correction UI; tag gives the answer.

3. Then pin the model, by register#

The model choice is worth more than the library choice and is invisible in a lockfile. russian-gsd scores 94.4% UFeats on encyclopedic prose and 81.0% on web text; russian-taiga inverts it. Neither is Stanza’s default.

Write the package string explicitly even when it matches the default, and record it next to the version number.

Confidence#

What S3 and S4 pick up#

S3 turns the two-halves finding into personas: who needs the lexicon half, who needs the disambiguator half, and who has a constraint (license, footprint, language coverage) that decides for them before accuracy enters.

S4 takes the durability question the licensing file opened. The lexicons have their own maintenance clocks — SudachiDict quarterly, pymorphy3-dicts-ru frozen since 2022, unidic-lite shipping a 2013 payload — and a lexicon that stops being rebuilt degrades silently as the language moves.

Segmentation-First Languages: Where the Lexicon Does the Work#

Japanese is written without spaces. Before anything can be lemmatized, something has to decide where the words are — and that decision is made by consulting a dictionary, which means the dictionary decides what the words are.

This file measures what follows from that.

The tools#

PyPI downloads for the last 30 days, read from pypistats.org on 2026-08-25: SudachiPy 1,906,682; SudachiDict-core 1,814,335; mecab-python3 1,063,206; fugashi 867,300; nagisa 646,642; unidic-lite 527,660; Janome 367,094.

That mecab-python3 line is the first correction S1 owes. S1 compared SudachiPy against fugashi and concluded SudachiPy wins on adoption by 2.7×. The larger MeCab wrapper is mecab-python3, which S1 did not count at all; taken together the two MeCab wrappers draw 1.93M downloads a month against SudachiPy’s 1.91M. Adoption does not separate these families.

Measurement: whose word boundaries?#

Gold: UD_Japanese-GSD test split, 543 sentences, 13,034 tokens. Each analyzer was given the raw sentence text reconstructed by concatenating the gold tokens, and scored on exact character-span agreement with the gold boundaries. Measured 2026-08-25.

Reading this table correctly#

fugashi’s 99.0% is not evidence that fugashi is a better analyzer. It is evidence that UD_Japanese-GSD is annotated in UniDic’s convention, and fugashi with unidic-lite emits UniDic’s convention natively. The treebank’s own README says so: the data is tokenized in “Short Unit Word (SUW)”, and “We use the slightly changed version of SUW as the UD word tokenization.”

Score the same tools against a Sudachi-convention gold and the ordering would invert. There is no analyzer-independent Japanese word, so there is no analyzer-independent segmentation score. What the table does establish, and what a reader can act on:

1. Convention agreement dominates everything else. The spread between the two modern, well-maintained analyzers is one point, and it is a convention difference rather than a quality difference.
2. The split mode is a bigger lever than the library. Within SudachiPy alone, mode A to mode C moves F1 by 3.4 points and token count by 3.3%. Choosing between SudachiPy and fugashi moves F1 by 1.0.
3. The dictionary generation is a bigger lever than either. Janome’s 92.2% — 5.8 points below SudachiPy mode A — is the cost of shipping IPAdic, whose last substantive update predates the current Japanese era name.

Split modes are not a quality setting#

Sudachi’s A/B/C modes are frequently described as “short, medium, long”. They are better understood as which layer of the Japanese word hierarchy you want, and the right answer depends entirely on the consumer.

Measured on SudachiDict-core, mode C, 2026-08-25:

Two independent knobs are visible here. Split mode decides how aggressively to merge units that the dictionary already knows. Dictionary edition decides whether the compound is in the dictionary at all. Requesting mode C on SudachiDict-small still splits スマートフォン, because small does not contain the compound to merge to.

The vendor’s own description of the three editions (SudachiDict README, read 2026-08-25):

• Small — “includes only the vocabulary of UniDic”
• Core — “includes basic vocabulary (default)”
• Full — “includes miscellaneous proper nouns”

For a search index, mode A on core gives the recall-friendly units. For entity extraction, mode C on full keeps 東京特許許可局 whole. For a boundary score against a UniDic-convention treebank, mode A on small was marginally the best of the Sudachi configurations tested — which is the opposite of the “bigger dictionary is better” intuition, and follows from small being the edition that tracks UniDic.

The dictionary has its own release clock, and it is not the code’s#

This is where the two-halves model earns its keep on maintenance rather than accuracy.

SudachiDict’s last eight releases: 2024-10-23, 2025-01-29, 2025-05-19, 2025-08-29, 2025-11-04, 2026-01-20, 2026-04-30, 2026-07-24. That is a quarterly cadence on a lexicon, sustained for years.

unidic-lite’s last eight releases all fall in 2020 and 2021, and the payload is UniDic 2.1.2, whose ChangeLog (shipped in the package) records its release on 2013-01-26. The package README is upfront about why: “it uses Unidic 2.1.2, from 2013, which is the most recent release of UniDic that’s small enough to be distributed via PyPI.”

The trap is that this is the default. Installing fugashi and calling fugashi.Tagger() with no arguments loads unidic-lite if it is present — confirmed here by reading Tagger.dictionary_info, which reported a sys.dic inside unidic_lite/dicdir holding 756,264 entries. A user who followed a tutorial in 2026 is running a 2013 lexicon and has been given no signal.

The escape hatch exists and is one command: pip install unidic then python -m unidic download gets UniDic 3.1.0. The cost is 770 MB on disk and a post-install network step, which is why so few installations take it.

IPAdic’s own maintainer states the position more bluntly than any survey should. The ipadic-py README carries a section headed “You Shouldn’t Use This”:

This is the version of IPAdic included with MeCab. It hasn’t been updated since at least 2007. The organization that created it no longer does this kind of work. The contact URLs listed in the source no longer resolve. It doesn’t contain important recent terms like 令和, the current era name.

Janome ships IPAdic in its wheel, which is the whole explanation for its 92.2%.

Repository health#

Read from the GitHub API on 2026-08-25. The stall test — open pull requests against pull requests merged since 2026-02-01 — is applied as a durability signal, not as a verdict on quality.

fugashi’s three open PRs against zero merged since February is a mild signal rather than a stall: the project has one maintainer and a release cadence measured in quarters, and it shipped four releases in October 2025. mecab-python3’s zero-and-zero is not a stall at all, because a stall requires a queue.

Upstream MeCab is the interesting case. It has no GitHub releases; the last widely distributed version is 0.996 from 2013, and both Python wrappers vendor their own build. The engine is frozen, which for a CRF over a fixed feature set is close to harmless — the lexicon is where the movement is, and the lexicon is separately packaged.

The neural alternatives#

nagisa (MIT, 646,642 downloads/month, released 2026-07-06) is a bi-LSTM word segmenter and POS tagger with no external dictionary. It sidesteps the whole dictionary-edition question, which also means it sidesteps normalized_form, dictionary_form, and the ability to add a domain lexicon.

Vaporetto (Apache-2.0 OR MIT, 2026-05-30) replaces the lattice with a pointwise linear classifier over character n-grams: for each boundary position, decide independently whether a word starts here. It advertises speed as its reason to exist, and its Python bindings are thin over Rust.

Neither was benchmarked here, because both require downloading their own model files and neither changes the argument: they are stage-one tools that fold stage two into a trained model, and they inherit the same “whose convention?” problem from whatever data they were trained on.

What is not available in this column#

No language-agnostic lemmatizer covers Japanese. simplemma supports 54 languages; enumerating its SUPPORTED_LANGUAGES on 2026-08-25 confirms ja is not among them. This is structural rather than an oversight: a form→lemma table is keyed on a form, and you cannot key on a form until something has told you where the form ends.

simplemma’s README makes the same point about the two segmenting languages it does cover, in its own accuracy table: Arabic 0.91 “on UD AR-PADT; real-world (unsegmented) input scores ≈0.85”, Hebrew 0.93 “real-world (unsegmented) input scores ≈0.82”. Eight to eleven points, paid for the segmentation the treebank did for it.

Summary of measured findings#

1. Against a UniDic-convention gold, fugashi + unidic-lite scores 99.0% boundary F1 and SudachiPy core mode A scores 98.0% — a convention effect, not a quality ranking.
2. Split mode moves F1 more than library choice does: 98.0% → 94.6% across Sudachi’s A → C on identical text.
3. Dictionary edition changes segmentation independently of split mode. Mode C on small still splits スマートフォン; on full it keeps 東京特許許可局 whole.
4. Sudachi is 4× faster than fugashi in mode A (243k vs 62k tokens/sec) and ~10× in mode B, on identical text in one process.
5. Janome’s bundled IPAdic costs 5.8 points of boundary F1 against SudachiPy mode A (92.2% vs 98.0%) and 6.8 against fugashi + unidic-lite.
6. The default MeCab dictionary in Python is thirteen years old. unidic-lite ships UniDic 2.1.2 (2013) and is what fugashi.Tagger() picks up with no arguments.
7. SudachiDict releases quarterly; its code sibling releases roughly annually. The data half moves faster than the code half, and the version you care about is the one on the data.

The Model Inside the Library#

Survey 1.140 (Classical Language Libraries) recorded a finding that reorganized its own recommendation: for Latin, the Stanza package loaded inside the library moved accuracy by 25 percentage points — 45% on ITTB, 70% on PROIEL — while the choice of library moved it far less. The default package was trained on medieval scholastic Latin and parsed poetae as a form of the verb possum.

This file tests whether that pattern holds for the modern inflecting languages, and finds that it does — with a different mechanism, which makes it more dangerous rather than less.

The packages nobody types#

Three libraries in this category load a per-treebank model, and each has a default the user never sees.

Stanza, read from stanza/resources/default_packages.py on 2026-08-25:

UDPipe 2 offers 961 models through its REST service, including four Czech, four Russian, and two Japanese in the UD 2.17 set alone.

spaCy ships one pipeline family per language in three or four sizes; the model choice is a size choice, not a corpus choice, and the corpus is fixed by Explosion.

Cross-evaluation: the same text, four models, one library#

Stanza and UDPipe both publish per-model accuracy tables. Those tables evaluate each model on its own treebank’s test set, which tells a reader how well the model fits its training distribution and nothing about how it will behave on their text. The experiment that answers the reader’s question is to hold the text fixed and vary the model.

All runs below use UDPipe 2, UD 2.17 models dated 251125, gold tokenization supplied, measured 2026-08-25.

Czech — gold is UD_Czech-CAC test (academic prose), 200 sentences, 2,628 tokens#

Czech — gold is UD_Czech-FicTree test (fiction), 200 sentences, 2,828 tokens#

Russian — gold is UD_Russian-GSD test (encyclopedic prose), 200 sentences, 3,707 tokens#

Russian — gold is UD_Russian-Taiga test (web and social), 200 sentences, 2,568 tokens#

Japanese — gold is UD_Japanese-GSD test, 200 sentences, 4,431 tokens#

The Japanese UFeats column reads 100% for both models because both emit nothing and the gold is empty — see the-two-halves.md for why UD Japanese has one feature key in its entire test split.

The pattern holds, and the spread is large#

Against those spreads, the lemma column moves by 0.6 to 1.4 points across all four Czech and Russian settings, and UPOS by 1.0 to 2.8.

So the 1.140 finding replicates: the package chosen inside the library moves the answer more than the choice of library does, and it does so on the column a morphology user cares about most. A reader who switches from spaCy to Stanza for Russian and leaves the default package alone has changed the library and inherited syntagrus, and the model choice they did not make is worth more than the one they did.

But the mechanism is different, and worse#

In 1.140, the wrong package produced a wrong answer: poetae came back as possum, a verb, which is visibly incorrect to anyone reading the output.

Here, the mismatched model mostly produces a differently shaped answer that is correct as far as it goes. Decomposing the UFeats gap, 2026-08-25:

Read the Russian rows. The matched model and the two mismatched models agree with gold on the features they all emit at 98.8%, 98.9%, and 99.0% — statistically indistinguishable, and the mismatched models are marginally higher. The entire 18-point “accuracy” gap is the key set: which feature names appear at all.

Enumerating the mismatches directly confirms it. russian-syntagrus on GSD gold:

Of the earlier 11.1-point Case drop, 242 tokens are the model declining to emit a Case at all and 27 are an actual disagreement about which case it is. Roughly nine parts convention to one part error.

Czech shows the same decomposition with one extra wrinkle. czech-pdtc on FicTree gold emits Aspect on 410 tokens the gold leaves bare and omits Animacy on 196 tokens the gold marks. Where both emit Gender, they disagree 132 times — and the example rows show why: gold Fem, predicted Fem,Neut. PDT-C annotates underspecified gender as a value set; FicTree commits to one value. Same key, same language, different value vocabulary.

Why the softer failure is the more dangerous one#

A wrong lemma announces itself. poetae → possum is caught by the first person to read the output.

A model that silently declines to emit Animacy does not announce itself at all. Downstream code written as

if token.morph.get("Animacy") == ["Anim"]:

returns False for every token, forever, and the pipeline reports 98.9% agreement on everything it does emit. Nothing errors. Nothing looks wrong. A rule that was supposed to fire on animate nouns fires on none of them, and a test suite built from the same pipeline’s output agrees with itself.

Three consequences follow, and they are the practical content of this file:

1. Pin the package string explicitly, even when it matches the default. stanza.Pipeline('ru', package='syntagrus') documents a decision; stanza.Pipeline('ru') inherits one.
2. Assert on the key set, not on accuracy. A pipeline change that keeps accuracy flat while dropping a feature key is the failure mode this category actually produces. A test that reads assert "Animacy" in doc[3].morph catches it; a test that reads assert accuracy > 0.95 does not.
3. Choose the model by register, not by size. The best Russian model on encyclopedic prose (gsd, 94.4% UFeats) is the worst on web text (81.0%), and the best on web text (taiga) is tied for worst on encyclopedic prose (76.0%). Neither is the default that Stanza would have loaded.

What the vendors’ own tables cannot tell you#

For completeness, Stanza’s published per-treebank figures (from its performance page, UD 2.12 data table, read 2026-08-25), each model on its own test set:

Every Czech UFeats figure sits between 88 and 96, and every Russian one between 92 and 94. From this table, model choice looks like a two-to-eight-point decision. Cross-evaluated on fixed text, it is an eleven-to-nineteen-point decision. The difference is the whole reason the cross-evaluation was run: a diagonal table cannot show off-diagonal behavior, and users live off the diagonal.

Summary of measured findings#

1. The 1.140 pattern replicates. Model package inside one library moves UFeats by 11.4-19.2 points across four language/register settings, against 0.6-1.4 points for lemma.
2. The mechanism is different. Agreement on shared feature keys is 97.3-99.0% regardless of model match; the gap is which keys are emitted, not which values.
3. For Russian on GSD gold, the mismatched models are marginally more accurate on the features they emit (98.9% and 99.0% against the matched model’s 98.8%). The 18-point exact-match gap is entirely output shape.
4. The failure is silent. A .morph.get("Animacy") lookup returns empty with no error and no accuracy signal.
5. Register beats size. russian-gsd scores 94.4% UFeats on encyclopedic prose and 81.0% on web text; russian-taiga inverts it.
6. Stanza’s Russian and Czech defaults are syntagrus and pdt — the two treebanks in this survey that are also CC BY-NC-SA. The package string decides the license as well as the output shape, which lexicon-licensing.md takes up.

The Two Halves: Lexicon and Disambiguator#

Every tool in this category performs some part of a two-stage job.

Stage one — enumerate. Given a stretch of text, produce the set of analyses that are linguistically possible. For Czech ulici this is three readings of the lemma ulice (dative, accusative, locative singular). For Japanese 読んで it is a segmentation into 読ん + で together with the dictionary form 読む.

Stage two — choose. Pick one. Nothing about stage one can do this, because the information that decides it is outside the word.

The two stages fail differently, cost differently, and are licensed differently. A buyer who does not know which one they are buying will pick badly, and the rest of this survey is a demonstration of how badly.

Where each library sits#

“Implicitly” means the model was trained on annotated text and never exposes a candidate set. That is a design decision with consequences: you cannot ask a Stanza pipeline what else the word might have been, so you cannot build a human-in-the-loop correction flow on top of it without re-deriving the alternatives some other way.

Enumeration: three mechanisms#

Lattice plus path cost (the Japanese analyzers)#

MeCab, Sudachi, Janome, and Kuromoji all do the same thing. Overlay the input string with every dictionary entry that matches at every position; that gives a lattice of candidate segmentations. Each edge carries a cost from the dictionary, each adjacency carries a cost from a connection matrix, and the analysis is the minimum-cost path.

The consequence: segmentation and analysis are one operation, not two. There is no stage at which a Japanese analyzer has “the words” but not “the readings”. This is why every Japanese tool in this category ships its own dictionary and why none of the language-agnostic lemmatizers cover Japanese at all — see segmentation-first.md.

Finite-state or DAWG lookup (the Slavic analyzers)#

pymorphy3 stores 391,778 lexemes expanded into 5,140,211 word forms in a DAWG (directed acyclic word graph), with 3,456 inflection paradigms and 16,311 prediction suffixes for out-of-vocabulary guessing. Measured from the loaded dictionary’s own metadata, 2026-08-25; the whole payload is 16 MB on disk.

MorfFlex CZ, behind MorphoDiTa, is the same idea at Czech scale and is where UDPipe 2’s best Czech model gets its analyses from — the vendor’s own model page states that the PDT-C model “performs morphological analysis using the MorfFlex CZ 2.1 morphological dictionary via MorphoDiTa.”

The consequence: enumeration is nearly perfect and nearly free. Measured on UD_Czech-CAC, MorfFlex returns 4.37 analyses per token and the correct case is somewhere in that list 99.8% of the time. pymorphy3 on UD_Russian-Taiga returns a candidate list containing the correct case 99.7% of the time. Neither tool is short of answers.

Table lookup (simplemma, spaCy lookups)#

A dict from surface form to lemma, plus fallbacks. No paradigms, no features, no candidates. simplemma is 19 MB installed and covers 54 languages; spaCy’s spacy-lookups-data ships 512,867 Russian entries and 35,021 Czech ones.

This is the cheapest thing that works, and on lemmas it works better than its price suggests — 93-95% on Slavic gold, within a few points of tools a hundred times its size. What it cannot do is any part of stage two, because it never had a candidate set.

Choosing: the number that decides the category#

Enumeration is solved. Choosing is not, and the measurements say so loudly.

Czech, UD_Czech-CAC test, 250 sentences (3,520 tokens), gold tokenization, MorphoDiTa czech-morfflex2.1-pdtc2.0-250909, measured 2026-08-25:

Read that bottom row twice. The same dictionary, the same tool, the same text. Attaching a disambiguator moves lemma accuracy by six points and case accuracy by forty-five.

MorphoDiTa’s analyze output is in dictionary order rather than probability order, so 48.5% is a floor rather than a fair context-free baseline. pymorphy3 does rank its output, and gives the fair version of the same experiment:

Russian, gold tokenization, restricted to tokens where pymorphy3 offers at least one case, measured 2026-08-25:

pymorphy3’s ranking is better than dictionary order and still leaves twenty points on the table. The mechanism explains the number: the ranking prior P(t|w) in pymorphy3-dicts-ru covers 45,471 unique words — read from the dictionary’s own metadata — against a lexicon of 5.1 million forms. Outside those 45k words the ordering carries no information about which reading is likelier, and Russian text is mostly outside those 45k words.

The asymmetry that reorganizes the category#

Put the two experiments side by side and one shape appears in both languages, from two unrelated codebases:

The Russian lemma row is negative. On UD_Russian-GSD, measured on all 11,385 tokens with gold tokenization:

The context-free lexicon beats the context-aware pipeline on lemmas by two points, and loses to it on case by fifteen. Neither tool is better. They are answering different questions, and “which library is more accurate” has no answer until the question names an output column.

This is why the category cannot be filed by language. A Russian reader who needs lemmas for a search index and a Russian reader who needs case for a grammar drill should install different packages, and S1’s single per-language winner cannot express that.

Where “lemma” stops meaning the same thing#

The two-halves model also explains why the word lemma travels badly.

In Czech and Russian, a lemma is the citation form of an inflectional paradigm. městech → město. The paradigm exists in the lexicon, the mapping is deterministic once the reading is fixed, and the hard part was fixing the reading.

In Japanese, the lexicon hands back several distinct notions and the caller picks:

Measured with SudachiDict-core 20260723, split mode C, 2026-08-25.

dictionary_form undoes inflection. normalized_form additionally repairs a common misspelling, a variant okurigana, a script choice, and a colloquial conjugation — mapping SUMMER onto サマー and the ra-nuki たべれる onto the standard 食べる. Neither field is “the lemma”; they are two different products of the lexicon, and a downstream index built on the wrong one will either miss variants or over-merge them.

Nothing in the Slavic tools corresponds to normalized_form, and nothing in the Japanese tools corresponds to a case feature. Which brings us to the sharpest measurement in this pass.

The interchange format admits the languages are doing different work#

Counting distinct morphological feature keys in the FEATS column of each gold test split, 2026-08-25:

UD Japanese annotates a morphological feature on eight tokens in a thousand, and the only key it ever uses is Polarity, on 101 tokens in the whole test split. Czech uses twenty-eight keys on three tokens in four.

This is not a gap in the Japanese data. It is a correct description of Japanese: the language does not mark case, gender, or number on the noun, so the FEATS column has nothing to hold. What a Japanese analyzer produces instead — conjugation type 五段-マ行, conjugation form 連用形-撥音便, reading ヨン — has no UD feature equivalent and lives in XPOS and MISC.

The immediate consequence, reproduced on the tools themselves: spaCy’s Japanese pipeline publishes morph_acc: 0.0 in its model metadata, while its Russian pipeline publishes morph_acc: 0.972. Running both on a hand-checked sentence confirms the metadata is describing reality rather than a broken evaluation:

• ru_core_news_sm on городах returns Animacy=Inan|Case=Loc|Gender=Masc|Number=Plur.
• ja_core_news_sm on 読ん returns Inflection=五段-マ行;連用形-撥音便|Reading=ヨン — and no UD features at all.

The Japanese pipeline is not failing at morphology. There is no UD-shaped morphology in Japanese for it to do, and its morphologizer component has been repurposed to carry Sudachi’s inflection strings instead.

What this means for reading the rest of S2#

Three files follow the two halves into detail:

• segmentation-first.md — Japanese, where stage one is the whole problem and the lexicon edition changes the answer.
• disambiguation-first.md — Russian and Czech, where stage one is nearly free and stage two is where accuracy is bought.
• the-model-inside-the-library.md — what happens when stage two is a trained model and you load the wrong one.

Two more follow the halves into their non-technical consequences, which turn out to be the ones that bite in production:

• lexicon-licensing.md — the two halves are almost never under the same license, and the permissive one is almost always the code.
• footprint-and-throughput.md — the two halves have wildly different costs, and the expensive half is usually the data.

S3: Need-Driven Discovery — Approach#

Purpose: who has a morphological analysis problem, why, and what decides their answer before accuracy enters the room.

Why the personas are organized this way#

S2 established that this category is two products in one package: a lexicon that enumerates what a form could be, and a disambiguator that picks one. The accuracy tables split along that seam, and so do the licenses, the footprints, and the maintenance clocks.

Personas follow the same seam rather than following languages. A Japanese search engineer and a Czech search engineer have the same problem — they want a stable index key and do not care about case — while a Czech search engineer and a Czech grammar-app builder want opposite things from the same language.

Each persona file answers four questions:

1. Which half do they need? Lexicon, disambiguator, or both.
2. Which output column? Lemma, features, candidates, or boundaries.
3. What constrains them before accuracy? License, footprint, language count, offline operation, auditability.
4. What is the wrong answer, and what does it cost? Every persona gets a plausible mistake as well as a recommendation.

The personas#

Six personas, against the methodology’s four to five, because the license persona and the low-resource persona are the two that S1 had no way to represent and both change the answer completely.

What S3 does not do#

• No implementation guides. No installation commands, no pipelines, no code. Those belong to the reader and their agent.
• No new measurements. Every number quoted here was measured in S2 and is cited back to the file that produced it.
• No single winner per persona where the evidence supports two. Where a persona has a real fork — usually license against accuracy — both branches are stated with the condition that selects them.

A note on the triggering scenario#

Requests for this survey tend to arrive attached to a language-learning tool, which is why S1’s use_case block reads the way it does. That scenario appears here as one persona among six, judged on the category’s terms alongside the others. It does not drive the S1 or S2 verdicts, and it has no privileged status in the recommendation.

Persona: The Commercial Product Team#

Who#

A team shipping software that customers pay for, in an organization with a license policy. Someone — legal, procurement, or an engineer with an allowlist — has to approve every dependency, and “we’ll sort it out later” is not available because the analyzer’s data ships inside the artifact.

This persona exists as its own file because in this category the license decides the answer before accuracy is consulted, and that inverts the normal order of a software choice.

The problem#

The obvious workflow fails here. An engineer checks the PyPI license field, sees something permissive, adds the dependency, and moves on. From S2-comprehensive/lexicon-licensing.md, that workflow gets the wrong answer for most of this category:

The structural reason: pip install ufal.udpipe installs the MPL-2.0 code and no model. The restriction attaches later, at a manual download step that no requirements.txt records and no license scanner sees.

Which half they need#

Whichever half their product needs — but the license attaches to the data half, and the data half is what they will be shipping.

What decides their answer#

License, then everything else. Concretely:

1. Is any part of this non-commercial? If yes, it is out, regardless of accuracy.
2. Is any part share-alike? If yes, does our policy allow it for data that is redistributed but not linked?
3. Can a license scanner see it? A field that says MIT over a GPL-triple-licensed payload will pass an automated audit and fail a manual one.
4. Only then: is it accurate enough?

Recommendation#

The unencumbered set#

Everything here is permissive at both layers, verified from primary sources on 2026-08-25.

SudachiPy is the standout. Apache-2.0 at both layers, with a LEGAL file that enumerates the provenance of the incorporated UniDic and NEologd material rather than asserting a license and hoping. That is the cleanest large lexicon in this survey and the reason Japanese has an unencumbered answer while Czech does not.

The encumbered set#

The escape hatches#

Stanza can be made commercial-safe by pinning the package string. Its defaults trace to non-commercial data; CC BY-SA siblings sit one argument away — ru → gsd or taiga, cs → cac or cltt, ja → gsd. From S2-comprehensive/the-model-inside-the-library.md, that same string also decides the output shape and is worth 11-19 points of UFeats, so it wants choosing on merit anyway.

Czech has no unencumbered accuracy leader. spaCy has no Czech pipeline at all (S2-comprehensive/disambiguation-first.md), the UFAL tools are non-commercial, and what remains is Stanza on a CC BY-SA package or a lookup table. For lemmas, simplemma at 93.3-95.1% is MIT and closes most of the gap. For features, a Czech commercial product is choosing between Stanza-with-a-pinned-package and training its own model on CC BY-SA treebanks.

What they sacrifice#

• The top of the Czech accuracy table. UDPipe 2’s Czech models are the best measured in this survey and are unavailable.
• Japanese with spaCy’s convenience. spaCy’s Japanese pipelines are share-alike; a team whose policy forbids that uses SudachiPy directly, which is also faster.
• Some audit-tool coverage. pip-licenses reports “MIT” for unidic-lite, whose data payload is 99% of its bytes and is triple-licensed by a third party. Manual review is required for the data packages in this category, and an automated gate will not catch what it cannot see.

The wrong answer, and what it costs#

Trusting the PyPI license field. It describes the wrapper. Half the table above is a counterexample.

Prototyping on UDPipe. The likeliest path into trouble, because UDPipe is excellent, is what the previous survey recommended for Czech, and is what an academic colleague will suggest. The restriction is discovered at the productization review, after the pipeline is built and the accuracy expectations are set.

Recording the package version and not the model. A pip freeze shows stanza==1.14.0 and tells a future auditor nothing about syntagrus. The license lives on the model, and the model is not in the lockfile. Record model identities in the same place the versions live, and treat a model change as a license review.

Assuming a share-alike model is contagious, or assuming it is not. Whether a trained model is a derivative of its training corpus is unsettled, and this survey takes no position. What is verified is that spaCy has stamped "license": "CC BY-SA 4.0" into its Japanese model metadata, and a downstream consumer inherits that declaration whatever the underlying law says.

Decision criteria in one line#

Check the model’s license, not the package’s — and put the model identity in the lockfile, because the license lives there and pip freeze will not find it.

Persona: The Corpus Linguist#

Who#

A researcher annotating or querying a corpus — a linguistics department, a digital humanities project, a lexicographer, a computational sociolinguist. Their output is a claim about language that someone else will try to reproduce.

They are the only persona in this survey for whom the annotation scheme is part of the result, not an implementation detail.

The problem#

They want to ask questions like “how has the instrumental case been used in this genre over fifty years” or “which verbs take this complement”. Those questions are queries against a feature column, and the answer depends entirely on which convention the feature column follows.

S2-comprehensive/the-model-inside-the-library.md measured what that does. Running russian-syntagrus over UD_Russian-GSD text, against GSD gold:

Nine parts convention to one part error. A researcher who ran half a corpus through one model and half through another has produced a dataset in which a feature’s absence means “the model did not emit this key” in some documents and “this word does not have this feature” in others — and no accuracy metric will show it, because agreement on shared keys stayed at 98.9%.

The Czech version is subtler still: PDT-C annotates underspecified gender as a value set (Fem,Neut), FicTree commits to one value. Same key, same language, incompatible value vocabularies. A query for Gender=Fem returns different sets depending on which model produced the annotation.

Which half they need#

Both, plus the provenance of both.

The candidate list matters here for a reason none of the other personas share: it is the evidence. When a corpus linguist disagrees with an automatic annotation, the useful artifact is the set of readings the analyzer considered, not only the one it chose. Manual correction workflows are built on that set.

This rules out, or at least demotes, the pipelines that never expose alternatives. From S2-comprehensive/the-two-halves.md: Stanza, spaCy, and UDPipe are marked “implicitly” on enumeration — they were trained on annotated text and expose no candidate set, so you cannot ask them what else the word might have been.

What decides their answer#

Reproducibility, above accuracy. A slightly worse tagger whose exact version, model, and lexicon can be cited beats a better one that cannot. Tagset transparency, because the tagset is in the published result. Availability of the candidate set, for correction workflows. License, which for this persona is usually permissive enough: CC BY-NC-SA is a non-issue inside a university, which is why UFAL’s tools dominate this space.

Recommendation#

Czech — MorphoDiTa, with the model identity recorded. It ships MorfFlex CZ, a hand-maintained morphological dictionary with a documented positional tagset, and exposes both analyze (every reading) and tag (the choice) from the same model file. S2-comprehensive/disambiguation-first.md measured 4.37 analyses per token, a 99.8% case oracle, and 93.6% case after disambiguation on UD_Czech-CAC.

For parsing as well as tagging, UDPipe 2’s PDT-C model is the strongest measured Czech option, and the vendor’s own model page states that it “performs morphological analysis using the MorfFlex CZ 2.1 morphological dictionary via MorphoDiTa” — the neural model calls the hand-built lexicon. That lineage is the reason to prefer it for corpus work: the lexicon is citable in a way that a set of weights is not.

Russian — pymorphy3 for the candidate set, plus a neural tagger for the choice, with both recorded. pymorphy3’s dictionary metadata is itself citable and is unusually complete: source opencorpora.org, revision 417150, compiled_at 2022-01-08, 391,778 lexemes, 5,140,211 forms (S2-comprehensive/disambiguation-first.md). Few tools in this survey let you cite their lexicon that exactly.

Japanese — fugashi with the full unidic package rather than unidic-lite. UniDic 3.1.0 is the reference lexicon for Japanese corpus linguistics, its Short Unit Word convention is what the UD Japanese treebanks follow, and matching it is what produced fugashi’s 99.0% boundary F1 in S2-comprehensive/segmentation-first.md. The -lite package ships UniDic 2.1.2 from 2013 and is what fugashi.Tagger() loads by default — a silent thirteen-year version gap in a field where the lexicon is the citable object.

What they sacrifice#

• CC BY-NC-SA on the best Czech models, which is invisible inside a university and becomes a wall if the corpus is later licensed to a commercial partner. That transition is worth thinking about before the annotation runs, not after.
• Two tools for Russian, since no single Russian package offers a good candidate set and a good disambiguator together the way MorphoDiTa does for Czech.
• 770 MB for full UniDic, against 249 MB for the lite version.

The wrong answer, and what it costs#

Annotating a corpus with a default model. Stanza’s Czech default is pdt and its Russian default is syntagrus; neither was chosen, both change the feature key set, and both trace to CC BY-NC-SA data. The corpus inherits an annotation scheme and a license from a string nobody typed.

Mixing models across a corpus. The one failure mode with no accuracy signal whatsoever. Splitting a corpus across two models produces systematically different key sets in different documents, and a genre comparison run over that split will find a “difference” that is entirely the annotation pipeline.

Reporting accuracy from the vendor’s table. S2-comprehensive/the-model-inside-the-library.md shows why: every published per-treebank figure is a model on its own test set. Stanza’s Czech UFeats figures span 88 to 96 across four treebanks; cross-evaluated on fixed text, the same models span 78 to 97. The diagonal cannot describe off-diagonal behavior, and corpus work happens off the diagonal by definition.

Citing the package version and not the model. stanza==1.14.0 is not a reproducible annotation pipeline. stanza 1.14.0, package='cac' is. The same applies to ufal.udpipe==1.4.0.1 against czech-pdtc-ud-2.17-251125, and to fugashi==1.5.2 against unidic 1.1.0 (UniDic 3.1.0).

Decision criteria in one line#

Record the lexicon and the model, not the library — the version number in your lockfile is not the thing that produced your annotations.

Persona: The Language-Learning Tool Builder#

Who#

Someone building a reader, a drill, a graded-vocabulary tool, or a flashcard generator for a language that inflects. They take authentic text and want to show a learner what each word is doing: this is the locative, this verb is perfective, this noun is animate.

They may be a solo developer with a side project or a team inside an education company. Either way their output is shown to a person who is trying to learn the grammar, which changes what an error costs.

The problem#

For this persona the feature column is the product. A reader tool that tells a learner городах is a noun with lemma город has told them nothing they could not guess. Telling them it is locative plural is the entire value.

That puts them on the wrong side of every measurement in S2-comprehensive/disambiguation-first.md:

A learner tool built on a lexicon-only analyzer is showing wrong case labels to roughly one word in four in Russian, and one in two in Czech if it takes the first listed reading. Those are the words the learner is studying.

Which half they need#

Both, and they need to see the seam.

The disambiguator is required for the label. The candidate list is required for something the other personas do not need: knowing when to be quiet.

A grammar tool that says “locative plural” with 78% reliability teaches wrong grammar 22% of the time to someone with no way to check. A tool that says “this is either genitive singular or nominative plural — look at the verb” has taught something true, and has taught the learner the actual skill.

From S2-comprehensive/the-two-halves.md: pymorphy3 returns 3.72 analyses per token with 50.2% of tokens ambiguous, and MorfFlex returns 4.37. The ambiguity is present, measurable, and available for free from either tool. Half of Russian tokens have more than one reading, and a learner who does not know that has been given a false picture of the language.

What decides their answer#

Feature accuracy first, because it is the product. Candidate visibility second, because it converts an error into a teaching moment. License third, and this persona often ships commercially, which rules out the accuracy leaders — see persona-commercial-product. Offline operation, for a mobile or desktop reader.

Recommendation#

Russian, shipping commercially — pair the two halves explicitly.

• pymorphy3 (MIT, 16 MB) for the candidate set. It gives every reading with a score, and a UI can say “3 possible readings” instead of picking one.
• spaCy ru_core_news_sm (MIT, 14 MB) for the choice. 93.5% case, against pymorphy3’s 78.7%.

Running both costs 30 MB and answers the two questions separately, which is what this persona needs. Cross-checking them is a bonus that no single tool offers: where spaCy’s chosen case is absent from pymorphy3’s candidate list, something is wrong, and that is a flag the tool can act on rather than a silent error.

Czech, shipping commercially — the good options are closed. MorphoDiTa and UDPipe are the accuracy leaders and are CC BY-NC-SA (S2-comprehensive/lexicon-licensing.md). spaCy has no Czech pipeline at all. What remains is Stanza with an explicitly pinned CC BY-SA package — package='cac' or package='cltt' — accepting that Stanza’s own default (pdt) traces to non-commercial data, and accepting Stanza’s statement that model license information “is unclear”.

Czech or Russian, research or non-commercial — MorphoDiTa. It ships both halves in one model file: analyze for the candidate set, tag for the choice, from the same lexicon. That is exactly this persona’s shape, and no other tool in the survey offers it in one package.

Japanese — a different job, and easier. SudachiPy gives dictionary_form, reading_form (furigana, which a learner tool wants), normalized_form, and the conjugation type and form as strings such as 五段-マ行 and 連用形-撥音便. There is no case to disambiguate. From S2-comprehensive/the-two-halves.md, UD Japanese annotates a morphological feature on 0.8% of tokens using a single key — the feature column that is this persona’s whole product in Czech does not exist in Japanese, and the conjugation strings take its place.

What they sacrifice#

• The two-package Russian setup has two tokenizers. Aligning pymorphy3’s candidates to spaCy’s tokens requires feeding spaCy pre-split tokens, which is what S2 did throughout and which spaCy supports.
• Czech commercial builders give up the top of the accuracy table. UDPipe 2’s Czech models are the best measured here and are unavailable to them.
• Aspect is harder than case and less well served. Russian perfective/imperfective pairs are lexical, not inflectional — делать/сделать are different lemmas, and no analyzer in this survey links them into pairs. A learner tool that wants to teach aspect pairs needs a separate vocabulary resource. This was not measured and is flagged as a gap rather than solved.

The wrong answer, and what it costs#

Shipping pymorphy3’s top parse as the answer. This is the most likely mistake for this persona, because pymorphy3 is the obvious Russian package, its API returns a sorted list, and taking parse(word)[0] looks like the intended usage. It is 78.7% right on case. The mechanism, from S2-comprehensive/disambiguation-first.md: the ranking prior covers 45,471 words against a 5.1-million-form lexicon, so outside common vocabulary the ordering carries no information — and a learner tool works mostly on vocabulary the learner does not yet know.

Suppressing the ambiguity. Half of Russian tokens have multiple readings. A UI that hides that is teaching a false model of the language, and it forfeits the one thing a lexicon-based tool does better than a neural one.

Letting Stanza pick its own Czech package. The default is pdt. That decides both the output shape and the license, and neither decision was made by the builder.

Decision criteria in one line#

You need the disambiguator for the label and the lexicon for the caveat — buy both, and show the learner when the answer is uncertain.

Persona: The Low-Resource Language Worker#

Who#

Someone working on a language with no treebank, no PyPI package, and no Stack Overflow answers. A community linguist, a revitalization project, a Bible translation organization, a national language institute, a graduate student on a language with three hundred thousand speakers and no NLP.

They are in this survey because every recommendation in it assumes a resource they do not have, and because the answer for them reverses the survey’s own architecture.

The problem#

Every statistical option in this category needs annotated training data. Universal Dependencies, the source of that data for the neural tools, states its own scale on its home page (read 2026-08-25): “over 600 contributors producing over 200 treebanks in over 150 languages,” with release 2.18 dated 2026-05-15.

One hundred and fifty languages. The world has thousands. The tools that dominate this survey — Stanza with 95 default packages, UDPipe with 961 models, spaCy with 25 pipelines, simplemma with 54 languages — are all downstream of that same 150, and none of them can be pointed at a language outside it.

There is a second problem that bites hardest here. Many low-resource languages are morphologically rich — agglutinative, polysynthetic, with productive derivation. A single Inuktitut or Turkish or Sámi word can correspond to an English clause. The form space is unbounded, which means a table of forms is not a possible artifact and a statistical model would need vastly more data than a language with poor lexical resources can supply.

Which half they need#

The lexicon half, and they will be building it.

This inverts the survey’s usual advice. For Czech and Russian, S2 found that enumeration is solved and choosing is where the accuracy lives. For a language with no lexicon, enumeration is the entire project, and it takes years of linguist time rather than GPU time.

The compensating fact is that a finite-state morphology, once written, is complete in a way a trained model never is. It generates as well as analyzes, it covers forms no corpus contains, it can be inspected and corrected by a speaker who is not a programmer, and it does not degrade on out-of-domain text because it has no domain.

What decides their answer#

Whether a description of the morphology exists — a grammar, a dictionary, a speaker who can answer questions. Whether anyone has already started. The finite-state ecosystems are large and poorly advertised, and duplicating an existing transducer is the most common waste in this space. Sustainability. The output has to outlive the grant, the student, and the maintainer. License, because community language data carries obligations beyond the legal ones.

Recommendation#

First: check whether it already exists#

Two ecosystems hold most of the finite-state morphology for under-served languages, and neither shows up in a PyPI search. Counts read from the GitHub API on 2026-08-25:

• Apertium — 629 repositories in the apertium organization whose names begin apertium-, holding monolingual morphologies and bilingual pairs. lttoolbox, the engine, is GPL-2.0. Individual language packages are typically GPL-3.0 (apertium-kaz, Kazakh, is GPL-3.0).
• GiellaLT — 155 repositories in the giellalt organization named lang-*, concentrated on Sámi, Finno-Ugric, and circumpolar languages. giellalt/lang-sme (Northern Sámi) is GPL-3.0 and was last pushed 2026-08-25, the day this was checked. These are living projects, not archives.

An hour spent searching those two organizations is the highest-value hour in this persona’s project.

Second: the finite-state toolchain#

HFST — Helsinki Finite-State Technology, on PyPI as hfst (3.16.0.1, 2023-11-21), GPL-3.0. The repository hfst/hfst was last pushed 2026-08-25 with 141 open issues, which reads as an active project with a backlog rather than a stalled one. It is the engine underneath most GiellaLT work.

foma and Apertium’s lttoolbox are the other two engines in common use. All three consume broadly similar lexicon-plus-rules source formats, which means the descriptive work transfers between them better than the tooling suggests.

The GPL is close to universal here, and for this persona it is usually the right license rather than a constraint: community language resources benefit from a share-alike floor, and the copyleft is what keeps a national institute’s transducer from being enclosed.

Third: the hybrid path#

A finite-state analyzer produces the candidate set and no disambiguation, which puts this persona in exactly the position S2-comprehensive/disambiguation-first.md measured for pymorphy3: a complete lexicon with no way to choose. On Russian that cost 19.7 to 22.7 points of case; on Czech, dictionary order cost 45.1.

The available answer is the same one MorphoDiTa uses: annotate a small amount of text and train a disambiguator on top of the transducer. The transducer restricts the tagger’s job to choosing among a handful of readings the lexicon has already validated, which is a far smaller learning problem than tagging from scratch and needs correspondingly less data. UDPipe 2’s own Czech model page describes the same architecture from the other direction: a neural model that “performs morphological analysis using the MorfFlex CZ 2.1 morphological dictionary via MorphoDiTa.”

That is the shape to aim for, and it is reachable with far less annotated text than a from-scratch neural tagger.

Fourth: contribute upstream#

Contributing a treebank to UD moves the language into the 150 and makes every tool in this survey available to it at once — Stanza, UDPipe, and Trankit all train from UD data, and Trankit advertises 100 trainable languages against 56 pretrained ones. The multiplier on that work is larger than on any other action in this file.

What they sacrifice#

• Years, rather than an afternoon. A publishable finite-state morphology is linguist-months to linguist-years. There is no version of this that is a pip install.
• The disambiguation layer, until annotated text exists.
• Ecosystem convenience. No spaCy integration, no CoNLL-U out of the box, no pretrained embeddings. The output has to be adapted to whatever consumes it.

The wrong answer, and what it costs#

Fine-tuning a multilingual model and calling it done. It will produce fluent, plausible, unverifiable output for a language whose speakers cannot easily check it, and it will fail invisibly on exactly the productive morphology that makes the language interesting. For a revitalization or education context — where the output teaches people their own language — an analyzer that is right 80% of the time and cannot say which 80% is worse than no analyzer.

Building the transducer without a speaker in the loop. A finite-state morphology written from a grammar sketch encodes the sketch’s errors permanently and at scale.

Building it as a private artifact. The single most common failure in this space is a completed morphology that dies with its grant. GPL, a public repository, and a home in Apertium or GiellaLT are what convert linguist-years into a resource that outlives the project.

Not checking Apertium and GiellaLT first. 784 language repositories across two organizations, invisible to a PyPI search, several of them updated this week.

Decision criteria in one line#

For a language outside UD’s 150, the lexicon is the project and the finite-state ecosystems are where it already lives — search Apertium and GiellaLT before writing a line.

Persona: The Multilingual Platform Engineer#

Who#

Someone whose product handles many languages and will handle more. A content moderation platform, a translation memory, a customer-support router, a multilingual analytics tool. They are not choosing an analyzer for Czech; they are choosing a strategy that survives adding Czech, then Turkish, then Thai.

Their success metric is the marginal cost of language N+1.

The problem#

The per-language-specialist approach that S1 recommended — SudachiPy for Japanese, pymorphy3 for Russian, UDPipe for Czech — is correct for one language at a time and does not compose. Three languages means three APIs, three output schemas, three tagsets, three licenses, three maintenance clocks, and three sets of failure modes. Ten languages means ten of each.

Worse, the outputs do not line up. From S2-comprehensive/the-two-halves.md:

A platform that defines a “morphology” field in its schema and populates it from per-language specialists has defined a field that means twenty-eight things in Czech, nineteen in Russian, and nothing in Japanese. And from S2-comprehensive/the-model-inside-the-library.md, the key set changes within a language depending on which model was loaded.

Which half they need#

Whichever half is uniform. For most platforms that is the lexicon half producing a lemma, because lemma is the one output whose meaning survives the trip across language families — approximately. Even there, S2-comprehensive/the-two-halves.md shows Japanese offering dictionary_form and normalized_form as two different things and Slavic offering one.

What decides their answer#

Language coverage per dependency. Adding a language should be a config change, not an integration project. Schema uniformity. One output shape, or a documented per-language shape. Total footprint. N language packs is a deployment size problem before it is an accuracy problem. Ops surface. N maintenance clocks is N chances for a silent regression.

Accuracy matters less than for any other persona in this survey, because the platform’s downstream consumer is usually a matcher or a classifier that degrades gracefully rather than a human reading a grammar label.

Recommendation#

Tier 1: one dependency, most of Europe#

simplemma. MIT, 19 MB installed, 54 languages, no per-language download, no model, no service. Its published accuracy figures — verified independently for Czech in S2-comprehensive/approach.md, where its claimed 0.95 on UD CS-FicTree reproduced at 95.1% — sit between 0.91 and 0.97 for 34 of its languages.

For a platform whose morphology need is “give me a stable key per word”, this is close to a complete answer for the Indo-European and Uralic bulk of a language list, at a marginal cost per language of one BCP 47 tag.

Tier 2: the languages simplemma does not cover well#

Its own README marks low-coverage and experimental languages and names alternatives. Two categories need separate handling regardless:

• Unspaced languages — Japanese, Chinese, Thai. simplemma has no ja and structurally cannot: a form-keyed table needs a form (S2-comprehensive/segmentation-first.md). These get a real analyzer.
• Segmenting-with-clitics languages — Arabic, Hebrew. simplemma covers them and says what it costs: 0.91 for Arabic “on UD AR-PADT; real-world (unsegmented) input scores ≈0.85”, 0.93 for Hebrew dropping to ≈0.82. Eight to eleven points, paid for the segmentation the treebank did for it.

Tier 3: when features are required#

Stanza is the one library in this survey that covers this persona’s shape: 95 languages with default packages, one Python API, one output schema (UD CoNLL-U), one Apache-2.0 code license, and — from S2-comprehensive/disambiguation-first.md — the healthiest repository in the category, with 82 pull requests merged since 2026-02-01 against 1 open.

Three conditions attach:

1. Pin every package string. Defaults trace to CC BY-NC-SA data for Czech and Russian, and change the emitted feature key set by 11-19 points of UFeats.
2. Expect per-language key sets. UD normalizes the format, not the feature inventory. Treat “which keys does this language emit” as language configuration, not as a bug.
3. Model download and disk. Each language pack is a separate download at first use, which is an ops event in an immutable-container deployment.

UDPipe 2 offers 961 models through one REST interface and is the only option here with no local footprint at all — but every model set is CC BY-NC-SA (S2-comprehensive/lexicon-licensing.md), which rules it out for a commercial platform and makes it attractive for an internal or research one.

Trankit advertises exactly this persona — 56 pretrained languages, one API, and a README claim of outperforming Stanza. Its last release is 2024-10-13 and it draws 668 downloads a month, against Stanza’s 1,132,695. The stall test reads 4 open PRs against 0 merged since 2026-02-01. The idea is right; the project is not one to build a platform on today.

What they sacrifice#

• Peak accuracy per language. A specialist beats a generalist in its own language; that is the trade this persona is making on purpose.
• The candidate set. Stanza and UDPipe expose no alternatives (S2-comprehensive/the-two-halves.md), so a per-language correction workflow has to be built elsewhere.
• Japanese uniformity. Whatever the strategy, Japanese runs a different code path, and its output has no feature column to put in the schema.

The wrong answer, and what it costs#

Defining one morphology schema field and populating it from per-language specialists. The field will be a Czech feature bundle, an OpenCorpora tag string, and a Japanese conjugation type, in the same column. Every downstream consumer will special-case it, and the special cases will be written by different people at different times.

Assuming Universal Dependencies means universal features. UD standardizes the column format and the value vocabulary, not which keys a language uses. Twenty-eight keys in Czech, one in Japanese, and a key set inside Russian that changes with the model.

Adding languages one at a time without a coverage plan. Language eleven is where the per-language-specialist strategy becomes an integration backlog. Deciding the tier structure before language three is cheaper than discovering it at language eleven.

Betting on Trankit because the feature list matches. 668 downloads a month and nothing merged since February is a durability signal, not a quality verdict — the code is Apache-2.0 and forkable — but it is not a platform dependency.

Decision criteria in one line#

Buy language coverage per dependency, not accuracy per language — and treat the unspaced languages as a separate tier from the start.

Persona: The Search Indexer#

Who#

An engineer running full-text search over documents in an inflecting or unspaced language. Their users type книга and expect to find documents containing книгами. Their users type 図書館 and expect to find it whether the source wrote 図書館 or としょかん.

They are not a linguist. They are the person who gets a ticket saying “search is broken for Russian” and discovers the index is keyed on surface forms.

The problem#

Inflection multiplies the key space. A Czech noun has fourteen forms across seven cases and two numbers; a Russian verb has dozens. Indexing surface forms means a query matches one form out of many, and recall collapses in exactly the languages where users notice.

Stemming is the traditional answer and is poor in this family. Slavic stems are not prefixes of their forms — нести / несу / нёс share four characters of stem between them and diverge at the vowel. Japanese has no stems to cut, because it has no boundaries to cut at.

So the indexer needs a lemma: a stable key that every inflected form maps onto.

Which half they need#

The lexicon half, and almost none of the disambiguator half.

This is the persona S2’s central asymmetry serves best. From S2-comprehensive/disambiguation-first.md, on UD_Russian-GSD:

Disambiguation is worth fifteen points of case and minus two points of lemma. An indexer who buys a neural pipeline to fix their recall problem is paying for a column they will never read, and taking an 8× throughput penalty to do it (2,167 tokens/sec against pymorphy3’s 17,230).

The Czech version of the same table, from the same file: MorphoDiTa’s perceptron buys 45.1 points of case and 6.3 points of lemma. Six points is worth something for a large index, but it is a different order from forty-five.

What decides their answer#

Throughput, because indexing is a batch job over everything. Footprint, because the analyzer ships wherever the indexer runs, often in a sidecar. Determinism, because a lemma that changes between two runs silently splits an index.

Accuracy matters, but at the margin: the difference between 93.8% and 95.5% lemma accuracy is a recall difference measured in fractions of a percent of queries, and is dominated by tokenization and by the analyzer’s behavior on names and numbers.

Recommendation#

Slavic and most of Europe — simplemma. MIT, 19 MB for all 54 languages, no per-language download, no model, deterministic. 95.1% Czech and 93.8% Russian lemmas. Adding Polish, Ukrainian, Slovak, and Slovene costs nothing but a language tag.

Russian specifically, when the extra points matter — pymorphy3 with pymorphy3-dicts-ru. MIT, 16 MB, 95.5% lemma, 17,230 tokens/sec. Take the top parse and index its normal_form; ignore the rest of the candidate list.

Japanese — SudachiPy with SudachiDict-core, split mode A. Apache-2.0 throughout, 243,360 tokens/sec, and 208 MB of dictionary.

The Japanese recommendation comes with an extra: index normalized_form, not dictionary_form. From S2-comprehensive/the-two-halves.md, Sudachi maps シュミレーション → シミュレーション (a common misspelling), 打込む → 打ち込む (an okurigana variant), SUMMER → サマー (a script variant), and the colloquial たべれる → 食べる. Every one of those is a query a user will type and a document that will not match unless the index normalized it. Nothing in the Slavic tools corresponds to this field, and an indexer who ports a Slavic pipeline to Japanese and reaches for the lemma-shaped method gets the weaker of the two.

What they sacrifice#

• No case, gender, or number. Fine here; the index does not use them.
• No candidate set with simplemma. If a token is ambiguous between two lemmas, the table commits and the indexer never sees it. For search this is an acceptable loss — the alternative is indexing both keys, which trades precision for recall and is a product decision rather than a library one.
• simplemma has no Japanese, and cannot. See S2-comprehensive/segmentation-first.md: a form-keyed table needs a form, and Japanese does not supply one. A multilingual index that includes Japanese runs two code paths whatever it does.

The wrong answer, and what it costs#

Reaching for a full NLP pipeline. spaCy or Stanza will be the first search result, will be excellent software, and will cost this persona 8× throughput and two points of the one column they care about. On a 100-million-token corpus that is hours of batch time per re-index for negative accuracy.

Indexing dictionary_form in Japanese. Silently loses the misspelling and script-variant recall that was the reason to run an analyzer at all.

Letting the analyzer version float. A lemma is an index key. Upgrading SudachiDict from one quarterly release to the next can change segmentation for compounds that entered the dictionary — from S2-comprehensive/segmentation-first.md, スマートフォン is one token in core and two in small. Pin the dictionary version, and treat a dictionary upgrade as a re-index event rather than a patch release.

Decision criteria in one line#

If the only column you read is lemma, buy the lexicon and skip the disambiguator — it costs throughput and, in Russian, accuracy.

S3 Recommendation#

Six personas, and the answer changes for all six on the same two questions.

The two questions that decide everything#

1. Does the language need segmentation before analysis?

If yes (Japanese, Chinese, Thai; partially Arabic and Hebrew), the analyzer and the dictionary are one purchase and the dictionary edition is the decision. If no, go to question 2.

2. Do you need lemmas, or do you need features?

Lemmas are close to free. Features cost a disambiguator, and the gap is not small: from S2-comprehensive/, disambiguation is worth +6.3 points of Czech lemma and +45.1 points of Czech case, from the same lexicon.

That single asymmetry sorts the personas.

The persona table#

Three conclusions that hold across all six#

The library is not the decision#

For five of the six personas, the choice that moves their outcome most is not which library they install:

• The indexer’s outcome turns on which Japanese field they index — normalized_form or dictionary_form.
• The learning builder’s turns on whether they show the candidate list.
• The corpus linguist’s turns on which model package produced the annotation.
• The commercial team’s turns on which model they downloaded, which is not in their lockfile.
• The platform engineer’s turns on their tier structure, decided before language three.

Only the low-resource worker’s outcome turns on a tool, and that tool is a compiler rather than a library.

This is why the survey is organized by the two halves rather than by language. A per-language winner list answers a question none of these personas are asking.

Two personas are served by the cheapest tool in the survey#

simplemma is 19 MB, MIT, has no model and no download, and covers 54 languages at 93-95% lemma accuracy on the Slavic gold measured in S2 — ahead of spaCy’s Russian pipeline and within three points of MorphoDiTa’s Czech tagger.

Both the search indexer and the multilingual platform engineer should start there and add heavier machinery only where a measurement says they need it. Neither reads the feature column, and the feature column is the only thing the heavy tools do better.

The license persona exists because the license is not where you look for it#

persona-commercial-product is a persona rather than a footnote because the normal dependency-review workflow returns the wrong answer for most of this category. The PyPI field describes the wrapper. The models are separately downloaded, separately licensed, and invisible to pip freeze.

The three most-recommended Czech options in this field — UDPipe, MorphoDiTa, and Stanza’s default package — are all non-commercial or trace to non-commercial data, and none of them says so in the place an engineer looks.

What every persona should do regardless#

Pin the model package string explicitly, even when it matches the default. It decides the output shape (11-19 points of UFeats), the license (Stanza’s Czech and Russian defaults trace to CC BY-NC-SA data), and the reproducibility of everything downstream. Three words in a constructor.

Assert on the feature key set, not on accuracy. The characteristic failure in this category is a model that silently stops emitting a feature key while its accuracy on everything else stays flat. assert "Animacy" in doc[3].morph catches it; an accuracy threshold does not.

Treat a lexicon upgrade as a data migration. A dictionary release can change segmentation and therefore change every key downstream of it. SudachiDict ships quarterly. Pin it, and re-index when you move it.

What S3 could not settle#

• Aspect pairing in Russian. делать and сделать are separate lemmas in every analyzer measured, and no tool in this survey links imperfective to perfective. A learning tool that wants to teach aspect pairs needs a resource outside this category. Not measured; flagged.
• Arabic and Hebrew clitic segmentation. simplemma publishes the cost of unsegmented input for both (≈0.85 and ≈0.82 against 0.91 and 0.93), and this survey did not test the specialist tools for either language.
• Whether any of these personas’ accuracy numbers hold on their own text. Every figure here was measured on Universal Dependencies gold in a specific register, and S2-comprehensive/the-model-inside-the-library.md demonstrates that register moves the answer by more than the tool does. The right next step for any reader is to run the two candidate tools on a hundred sentences of their own data and read the disagreements.

S4: Strategic Selection — Approach#

Purpose: what happens to each of these choices over five to ten years, and which risks are structural rather than incidental.

Why this category’s durability question is unusual#

For most library categories, “will this survive?” is a question about a codebase: who maintains it, are patches landing, is the funding stable.

Here it is two questions, because S2 established that every option in this category is two products in one package. The code half and the data half have separate maintainers, separate release cadences, separate licenses, and separate failure modes — and it is routine for one to be healthy while the other is frozen.

The clearest instance, and the one S1 got wrong: pymorphy3 shipped eight releases between 2023 and 2025 and is by any code-side measure maintained. Its Russian dictionary, pymorphy3-dicts-ru, has one release in its entire history, dated 2022-01-08, confirmed both by the PyPI release index and by the compiled_at field in the dictionary the library loads at runtime. The library is maintained. The product is frozen.

So S4 asks the durability question twice for every option.

What is assessed#

The stall test#

Applied per ADDING-RESEARCH.md: open pull requests against pull requests merged since 2026-02-01, read from the GitHub API on 2026-08-25. A large open count with zero merges is the signal; a large open count with a large merge count is health.

The test needs one adjustment for this category. Several of these projects are academic tools with no PR culture at all — ufal/udpipe and ufal/morphodita both read 0 open and 0 merged, which under a naive reading looks like the same state as an abandoned project. It is not: development happens inside a university group and lands as releases, and both shipped to PyPI in late 2025. For those projects the release cadence and the institutional backing are the signals, and the PR ratio is noise.

The test also cannot fire on a project with an empty queue. mecab-python3 at 0 open and 0 merged is stable rather than stalled — nobody is waiting.

What S4 does not do#

• No ranking of the organizations behind these tools. Charles University, Works Applications, Stanford, and Explosion are compared on what their choices do to a downstream user, not on which is winning.
• No prediction that a specific project will die. Durability signals describe exposure, not fate. Every finding here is framed as “what happens to you if this changes.”
• No new measurements. S4 reasons over what S2 measured and over repository and registry facts read on 2026-08-25.

Output files#

Ecosystem Outlook#

Where this category is going, and what the arrival of large language models does to it.

Universal Dependencies is the center of gravity, and its limits are the category’s#

From UD’s own home page, read 2026-08-25: “over 600 contributors producing over 200 treebanks in over 150 languages,” most recent release 2.18 on 2026-05-15. The GitHub organization holds 481 repositories matching UD_.

Three of the five statistical tools in this survey — Stanza, UDPipe, Trankit — train directly from UD, and spaCy’s models cite UD treebanks as sources. Everything they can do is downstream of what UD has annotated.

This has produced two structural properties that will hold for the foreseeable future.

A shared interchange format, with real portability value. CoNLL-U and UD FEATS mean that spaCy, Stanza, and UDPipe are substitutable at the schema level, which is the single largest reduction in lock-in this category has ever had (lock-in-and-exit.md).

A hard ceiling at 150 languages. No amount of engineering moves a tool outside UD coverage, which is why the low-resource persona’s answer lives in a completely different ecosystem (Apertium, GiellaLT, HFST) that PyPI does not index.

The second property is also the category’s clearest growth path: contributing a treebank makes every UD-trained tool available to a language at once. The multiplier on treebank work is larger than on tool work.

One caution about UD’s own promise. UD standardizes the format and the value vocabulary, not the feature inventory. From S2-comprehensive/the-two-halves.md, UD Japanese uses one feature key and UD Czech uses 28; from the-model-inside-the-library.md, two Russian models in the same library emit different key sets on the same text, with the PDT-C convention writing underspecified gender as Fem,Neut where FicTree commits to one value. “Universal” describes the schema. It does not describe the data.

The lexicon is a durable asset and a fragile one#

The clearest long-run pattern in this survey is that hand-built lexicons outlast the software that reads them.

MorfFlex CZ is decades of Czech lexicographic work, and UDPipe 2 — a modern neural system that wins shared tasks — reaches for it rather than replacing it. The vendor’s own model page says the PDT-C model “performs morphological analysis using the MorfFlex CZ 2.1 morphological dictionary via MorphoDiTa.” The neural model does the choosing; the dictionary still does the enumerating, which is exactly the two-halves split this survey is organized around, asserted by the tool’s own authors.

UniDic has outlived MeCab’s development. IPAdic outlived the organization that built it. OpenCorpora’s data still powers 501,262 monthly pymorphy3 installs four years after the dictionary was last compiled.

The fragility is on the other side of the same fact: a lexicon is maintained by people, and people stop. pymorphy3-dicts-ru has one release in its history; unidic-lite ships a 2013 payload; IPAdic’s own packager wrote a section headed “You Shouldn’t Use This.”

The strategic reading: prefer lexicons with an institutional or commercial reason to be updated. SudachiDict is maintained because Works Applications uses it. MorfFlex is maintained because ÚFAL’s research depends on it. pymorphy3-dicts-ru is maintained by nobody’s need in particular, and it shows.

What large language models do to this category#

The tempting conclusion is that a general-purpose model makes morphological analyzers obsolete. The evidence in this survey argues against that for three specific reasons, and for one narrow case argues the opposite.

1. The measurements say the hard part is not the part an LLM is good at#

S2-comprehensive/ found that the lexicon half is close to solved — MorfFlex holds the correct Czech case for 99.8% of tokens; pymorphy3’s Russian oracle reaches 99.7% on web text — and that the disambiguator half is where accuracy is bought, worth +45.1 points of Czech case.

Disambiguation from sentence context is exactly what a language model does well, and it is the half where a language model could plausibly beat a perceptron. But it is also the half that already works: MorphoDiTa’s tagger reaches 93.6% on Czech case, UDPipe 2’s russian-gsd reached 97.1%, and each does it at thousands of tokens per second on a CPU with a deterministic, auditable output.

The half a language model would have to replace wholesale is the lexicon — and a lexicon is a completeness claim. “This form has exactly these four readings” is a statement a hand-built transducer can make and a probabilistic model cannot.

2. Determinism is a requirement in most of these use cases#

An index key that changes between runs splits an index. A grammar label that varies between two identical queries destroys a learner’s trust. A corpus annotation that cannot be reproduced is not a corpus annotation.

Every persona in S3 except the low-resource worker has a determinism requirement, and several have an offline requirement on top of it. A 16 MB DAWG that answers in microseconds and returns the same thing every time is not competing with an API call on quality alone.

3. Cost and footprint are not close#

From S2-comprehensive/footprint-and-throughput.md: pymorphy3 returns every reading for 17,230 tokens per second in a 16 MB package; SudachiPy segments and analyzes at 243,360 tokens per second. Corpus-scale morphology is a batch job over hundreds of millions of tokens. The economics do not favor a model call per token, and will not soon.

The narrow case where the opposite is true#

The low-resource persona. For a language outside UD’s 150 with no lexicon and no treebank, a multilingual model is the only thing that produces output at all, and “something at 70%” can beat “nothing” for triage, for bootstrapping annotation, or for prioritizing which forms a linguist should describe first.

persona-low-resource-language.md argues against shipping that output to speakers who cannot check it, and the argument holds. Using it as scaffolding for the human work is a different proposition and a reasonable one.

The likelier long-run shape, visible already in UDPipe 2’s architecture, is hybrid rather than replacement: a lexicon or transducer constrains the candidate set, and a learned model chooses within it. That is the design that already wins, and a language model is a better disambiguator than a perceptron in principle. Whether it is worth 10,000× the compute for the last two points is the question a reader should ask, and the answer today is usually no.

Five-year outlook by option#

What would change the picture#

• A permissively licensed Czech model trained only on CC BY-SA treebanks would close the largest gap this survey found. Nothing prevents it.
• A rebuild of pymorphy3-dicts-ru from current OpenCorpora data. The scripts are public and MIT; it is a weekend for someone with the corpus.
• A spaCy Czech pipeline, which would give the largest NLP ecosystem a Czech answer it has never had.
• UD relicensing PDT or SynTagRus, which is unlikely: the restrictions trace to source material whose rights sit with publishers, not with the tool maintainers.
• A hybrid tagger that uses a language model to disambiguate a transducer’s candidate set. The architecture UDPipe 2 already implements with a smaller model, and the place where a large model would add the most for the least.

Lexicon Durability: The Second Clock#

A morphological analyzer has two release cycles, and the one that matters is rarely the one on the badge.

The code half changes when the API changes, the Python version moves, or a bug is found — occasional, and visible in a lockfile. The data half changes when the language changes: new words, new proper nouns, new spellings, new borrowings. That clock never stops, and a lexicon that stops being rebuilt degrades continuously and silently against it.

The two clocks, measured#

Release dates read from the PyPI JSON API on 2026-08-25.

Three of these need naming individually.

SudachiDict: the counterexample that proves the point#

SudachiDict’s last eight releases: 2024-10-23, 2025-01-29, 2025-05-19, 2025-08-29, 2025-11-04, 2026-01-20, 2026-04-30, 2026-07-24. A quarterly cadence on a lexicon, sustained for years, from a company (Works Applications) that uses it in production.

That is what a maintained lexicon looks like, and it is the strongest durability signal in this entire survey. It is also the reason the Japanese column has an unencumbered, current, fast answer while the Czech column has to choose between non-commercial and second-best.

The corollary for a consumer: treat a SudachiDict upgrade as a data migration. From S2-comprehensive/segmentation-first.md, dictionary edition changes segmentation for compounds — スマートフォン is one token under core and two under small. A quarterly lexicon is a quarterly opportunity for a downstream index key to change. Pin it, and re-index when you move it.

pymorphy3-dicts-ru: maintained code, frozen product#

This is the finding S1 inverted. From the loaded dictionary’s own metadata (S2-comprehensive/disambiguation-first.md):

pymorphy3 the library is a fork created to keep pymorphy2 working on modern Python, and it has done that job — eight releases since 2023. But the fork’s dictionary packages were built once, in January 2022, and never rebuilt. The upstream build tooling exists and is public (no-plagiarism/pymorphy3-dicts, MIT); nobody has run it in three and a half years.

What that costs, and what it does not:

• It does not cost much on core vocabulary. Russian inflectional morphology is stable; город declined the same way in 2022 and does now. This is why pymorphy3 still measured 95.5% lemma accuracy in S2 — better than any other Russian tool tested.
• It costs on new words. Proper nouns, borrowings, product names, and neologisms from four years of Russian are absent, and fall through to the 16,311 prediction-suffix guesser.
• It costs most on the ranking. The P(t|w) prior over 45,471 words is a snapshot of a frequency distribution from a corpus older than the compile date, and frequency is the part of a language that moves fastest.

The exposure is a bus factor of approximately one on a package with 501,262 monthly downloads. The mitigation is real: MIT license, public build scripts, an upstream data source. Someone can rebuild it. Nobody has.

unidic-lite: a 2013 lexicon as the silent default#

The unidic-lite package README states the reason without evasion: “At the moment it uses Unidic 2.1.2, from 2013, which is the most recent release of UniDic that’s small enough to be distributed via PyPI.” The shipped ChangeLog dates version 2.1.2 to 2013-01-26.

The exposure is that this is what fugashi.Tagger() loads when nothing else is present — confirmed in S2 by reading Tagger.dictionary_info, which reported a sys.dic inside unidic_lite/dicdir with 756,264 entries. A developer following a 2026 tutorial gets a 2013 lexicon and no warning.

The escape is pip install unidic plus python -m unidic download, which brings UniDic 3.1.0 at roughly 770 MB. The barrier is the 770 MB and the post-install network step, which is why the default persists.

IPAdic is the same story one generation further back, and its own packager says so: the ipadic-py README’s section heading is “You Shouldn’t Use This”, and it notes the dictionary “hasn’t been updated since at least 2007”, that “the organization that created it no longer does this kind of work”, and that “the contact URLs listed in the source no longer resolve.” Janome ships it in the wheel.

Code-side durability: the stall test#

Open pull requests against pull requests merged since 2026-02-01, read from the GitHub API on 2026-08-25.

Three clear stalls, and each is instructive:

pymorphy2 is the stall that already resolved. Six patches waiting, nothing merged, last release 2020 — and a fork (pymorphy3) that took over the code half and left the data half where it was. This is what a successful fork looks like from the original’s side, and it is why the original still draws 211,607 downloads a month four years after its last release.

natasha is the stall to watch. 1,347 stars, seven patches waiting, last release 2023-07-24, and its sibling slovnet last released 2023-01-23. MIT and forkable, so the exposure is manageable, but a Russian-specific stack that has not shipped in three years is not something to start a project on.

Trankit is the stall with the largest gap between promise and reality. It advertises exactly what a multilingual platform wants — 56 pretrained languages, one API, a README claiming it outperforms Stanza — and draws 668 downloads a month against Stanza’s 1,132,695. Four patches waiting, none merged since February, last release 2024-10-13. Apache-2.0 and forkable; not a foundation.

Institutional backing and its shape#

The strongest and weakest durability arguments in this survey are both institutional.

Works Applications (Sudachi, SudachiDict) is a company that uses its own lexicon in production, which is the most durable arrangement available: the data gets updated because the company needs it updated, not because a grant renews.

Charles University / ÚFAL (UDPipe, MorphoDiTa, MorfFlex, the Prague Dependency Treebank) has maintained Czech language resources for decades and continues to ship — ufal.morphodita released 2025-12-02, ufal.udpipe 2025-11-20, UDPipe 2 models dated 2025-11-25. Institutional durability at its best.

That backing carries one durability caveat and one commercial one.

The commercial one is licensing-risk.md’s subject: CC BY-NC-SA on everything.

The durability one is a status flag. The ÚFAL project page for UDPipe 2 — the version behind the public REST service and behind every UD 2.17 model — records “Status: to be released”. UDPipe 2 has been serving models publicly, winning shared tasks, and being cited for years, and its own project page still describes it as a Python prototype pending release. That is not a prediction of failure; it is a signal that the maintained artifact and the released artifact are not the same thing, and a consumer depending on the REST endpoint should know that.

Explosion (spaCy) is venture-funded, which as a durability signal cuts both ways and is reported here as exposure rather than as a verdict: a company can change direction, and spaCy’s model licenses are already set per-model rather than by a company-wide policy. The library is MIT with 33,848 stars and a live contribution stream, so a fork is available if the direction changes.

Stanford / Stanza is the healthiest code-side project measured and the one that is most explicit that its model licensing is not settled — see licensing-risk.md.

Download trajectories#

Last-30-day figures read 2026-08-25, against the figures S1 recorded in November 2025.

Download counts are noisy — CI, mirrors, and dependency changes move them more than adoption does — and a single interval is weak evidence. The UDPipe halving is still the line to watch, because it coincides with the arrival of UDPipe 2 as a REST service, which would move usage off the PyPI package without any loss of users. Both readings are consistent with the data, and this survey does not have the evidence to choose between them.

Summary of durability findings#

1. The data clock is the one that matters, and it is not on the badge.
2. SudachiDict is the healthiest lexicon in the survey — quarterly releases, commercial user, Apache-2.0.
3. pymorphy3-dicts-ru was compiled once, on 2022-01-08, and never rebuilt — under a code package that ships regularly. S1 read the code clock and called the product actively maintained.
4. unidic-lite ships a 2013 lexicon and is the silent default for fugashi. Janome’s bundled IPAdic is worse and its own packager says so.
5. Three code-side stalls: pymorphy2 (already forked), natasha, Trankit.
6. Stanza and simplemma are the two healthiest projects by the stall test, and neither appeared in S1.
7. The academic tools need a different test. Zero-and-zero at ufal/udpipe is a PR-culture artifact, not a stall; their releases are the signal.

Licensing Risk#

S2-comprehensive/lexicon-licensing.md established what the terms are. This file asks the strategic questions: what can change, who can change it, and what a change would cost someone already depending on it.

The structural exposure#

The risk in this category is not that a license will be revoked. Creative Commons grants are irrevocable for material already received, and none of the projects here shows any sign of tightening.

The risk is that the terms were never where the engineer looked, so an organization can build for months on a restriction it does not know it has.

The mechanism is mechanical rather than anyone’s fault:

pip install ufal.udpipe        # installs MPL-2.0 code, no model
requirements.txt: ufal.udpipe==1.4.0.1
pip-licenses:      ufal.udpipe  MPL-2.0        ✓ passes
<later, at runtime or in a setup script>
  download a model             # CC BY-NC-SA, in nobody's lockfile

Every automated control in a normal dependency pipeline sees the first half and none of the second. The restriction attaches at a manual download step that no tool records.

The same shape recurs with different names:

Note how the errors run. In six of seven rows, the scanner sees something more permissive than what ships. A control that fails open on the majority of a category is not a control.

Risk 1: discovering the restriction after the pipeline is built#

Exposure: high, and concentrated on Czech.

The path is well-worn: an academic colleague or a previous survey recommends UDPipe or MorphoDiTa, which are the accuracy leaders and are excellent. Development proceeds for months. The restriction surfaces at a productization review, after the accuracy expectations have been set by a tool that cannot be shipped.

S1 walked straight into this. It recommended ufal-udpipe as the Czech answer on grounds of accuracy, academic backing, and Universal Dependencies conformance — every one of them defensible — and did not mention that every UDPipe model set is non-commercial. A team following S1 into a product would have found out late.

What it costs: for Czech, there is no drop-in permissive replacement at the same accuracy. spaCy has no Czech pipeline at all (S2-comprehensive/disambiguation-first.md). The remaining options are Stanza on a pinned CC BY-SA package, or training a model on CC BY-SA treebanks, or accepting simplemma’s 93-95% on lemmas with no features.

Mitigation: decide the license question before the accuracy question. It is the one constraint in this category that cannot be engineered around after the fact.

Risk 2: the model changes the license without a code change#

Exposure: moderate, and easy to miss because it looks like a tuning change.

From S2-comprehensive/the-model-inside-the-library.md, the package string inside Stanza decides three things at once: the emitted feature key set, the accuracy on the reader’s register, and — through the training treebank — the license.

stanza.Pipeline('cs')                  # -> pdt      -> UD_Czech-PDT, CC BY-NC-SA 4.0
stanza.Pipeline('cs', package='cac')   # -> cac      -> UD_Czech-CAC, CC BY-SA 4.0

An engineer tuning accuracy who switches Russian from gsd to syntagrus because the vendor’s table shows a higher UFeats figure has moved from CC BY-SA to CC BY-NC-SA in a one-word diff, in a file no license review will look at.

What it costs: a compliance finding on a change that reviewed as a performance tweak.

Mitigation: keep model identities in the same artifact as the version pins, and treat a model change as a license review. stanza==1.14.0 is not a record of anything; stanza==1.14.0, cs=cac, ru=gsd is.

Risk 3: the same library, different terms per language#

Exposure: moderate, and rises with every language added.

spaCy is MIT and its Russian and Ukrainian models are MIT (Nerus, Navec, Ukr-Synth). Its Japanese models are CC BY-SA 4.0 — CC BY-SA 3.0 for ja_core_news_trf, inherited from cl-tohoku/bert-base-japanese-char-v2. All from one vendor, in one release, under one MIT library.

An organization that approved “spaCy” as a dependency and later added Japanese has changed its obligations without changing its dependency list.

What it costs: less than the non-commercial cases — share-alike on a model is compatible with commercial distribution, subject to attribution and to whatever position one takes on whether a model is a derivative work. But it is a different obligation from the one that was approved.

Mitigation: approve models, not libraries. In a multilingual product the license review has to be per-language.

Risk 4: the unsettled question nobody has answered#

Exposure: unknowable, and this survey takes no position.

Whether a trained model is a derivative work of its training corpus has no settled answer in most jurisdictions. Stanza’s maintainers say so directly on their performance page:

License information for models built from the UD data is unclear, but users are encouraged to click on the git links below and check the license for the relevant data if they are unsure of the licenses for the data used to build these models. To the extent that The Trustees of Leland Stanford Junior University have ownership and rights over these language packs, all these Stanza language packs are made available under the Open Data Commons Attribution License v1.0.

That is a creditable disclosure and it puts the work on the user. spaCy has taken the opposite approach and answered the question for its own artifacts by stamping "license": "CC BY-SA 4.0" into the Japanese model metadata. UFAL has answered it a third way, by licensing the models directly rather than deriving their terms from the data.

Three vendors, three positions, one unsettled question. A downstream consumer inherits whichever position their vendor took.

Mitigation: prefer vendors who have declared. A declared CC BY-SA is a known obligation; an undeclared model trained on an unknown treebank is an open question carried indefinitely.

Risk 5: the audit gap on data payloads#

Exposure: low legal risk, high process risk.

unidic-lite reports “MIT License” to PyPI. Its data payload is 249 MB — the overwhelming majority of the package — and the COPYING file installed alongside it says the dictionary “is released under any of the GPL, the LGPL, or the BSD License.” A recipient may select BSD, so nothing is blocked. But pip-licenses will report MIT for a package whose substance is triple-licensed by a third party, and an audit that relies on it has not audited this package.

ipadic is the sharper version: no license field at all on PyPI, and a shipped COPYING that is a bespoke NAIST grant incorporating ICOT Free Software conditions. It is permissive in intent and is not an OSI-approved license, so any policy written as an allowlist rejects it — or, more likely, never sees it, because Janome bundles it inside the wheel rather than declaring it as a dependency.

Mitigation: manual review for every data package in this category, and a standing exception in the process that says automated license scanning does not cover model and dictionary payloads.

What could get better, and what could get worse#

Better: Universal Dependencies has been moving toward permissive licensing over time, and the CC BY-SA treebanks in this survey (Czech-CAC, Czech-CLTT, Russian-GSD, Russian-Taiga, Japanese-GSD, Japanese-GSDLUW) are already enough to train commercially usable models for all three languages. Someone releasing a permissively licensed Czech model trained only on CC BY-SA data would close the largest gap this survey found. Nothing prevents it; nobody has done it.

Worse: the non-commercial restrictions on PDT and SynTagRus are tied to corpus material whose rights sit with publishers and institutions rather than with the tool maintainers. UFAL cannot relicense PDT unilaterally. That is a structural reason not to expect the Czech situation to resolve on its own.

Unchanged: the audit gap. Nothing in the packaging ecosystem is moving toward recording model identity in a lockfile, so the gap between what a scanner sees and what ships stays open.

The single strategic instruction#

Record the model, not the package.

It is the mitigation for four of the five risks above. The license lives on the model, the accuracy lives on the model, the output shape lives on the model, and the model is the one thing a conventional dependency pipeline does not record.

Three words next to a version pin close most of this category’s compliance exposure.

Lock-In and Exit#

Swapping one morphological analyzer for another is a small code change and a large data change. This file separates the two, because teams consistently estimate the first and are surprised by the second.

The API is not the lock-in#

Every tool in this survey does approximately the same thing: text in, one record per token out, with a lemma, a part of speech, and some feature representation. The adapter between any two of them is an afternoon.

Two things make it look harder than it is, and both are surface:

• Tokenization differs, so a switch changes the record count. Handled by supplying gold or pre-computed tokens, which every tool measured in S2 accepts.
• The candidate-set tools return lists and the pipeline tools return one record. Handled by taking element zero, at the cost documented throughout S2.

Neither is a migration project.

The tagset is the lock-in#

What does not port is the vocabulary the features are written in, and everything downstream of an analyzer is written against that vocabulary.

Three incompatible schemes appear in this survey:

Converting between them is possible and lossy. S2 had to build two mappings to make any comparison at all — OpenCorpora case to UD case, and PDT tag position 5 to UD case — and both were written for one feature, deliberately narrow, because a full mapping is a research artifact rather than a utility function.

The scale of the loss: UD Czech uses 28 distinct feature keys, UD Russian 19-24, and UD Japanese one (S2-comprehensive/the-two-halves.md). PDT positional tags encode 15 positions including several with no UD equivalent. There is no lossless round trip.

So the migration unit is not the analyzer. It is every rule, query, template, index, and test that reads a feature.

Lock-in, ranked#

The middle row is the useful one and it is a point in favor of the UD-speaking tools that has nothing to do with accuracy: spaCy, Stanza, and UDPipe are mutually substitutable at the schema level. Moving between them changes the model and the numbers; it does not change the shape of the data or the code that reads it. That is a real strategic property, and it is the strongest argument for paying UD’s tax (a coarser feature inventory than PDT’s) up front.

The lock-in that is invisible: the key set#

S2-comprehensive/the-model-inside-the-library.md measured something that behaves like lock-in but is not usually recognized as such.

Two models in the same library, both speaking UD FEATS, emit different feature keys on the same text. russian-syntagrus on GSD text omits Case on 242 tokens, omits Animacy on 153, and adds NameType on 258 — while agreeing with gold on the keys it does emit at 98.9%.

So code written against one model’s output is coupled to that model, even though both models speak the same schema. Downstream logic of the form

if token.morph.get("Animacy") == ["Anim"]:

does not break on a model swap. It silently stops firing, with no exception and no change in any accuracy metric.

This is the most expensive form of lock-in in the category, because it is not visible in a type system, a schema, or a test that asserts on accuracy. The mitigation from S3 applies here too: assert on the key set, so a model swap that drops a feature fails a test instead of quietly changing behavior.

Exit cost by scenario#

Leaving pymorphy3 for a neural tagger (Russian)#

Reason: needing case accuracy above 78.7%. Cost: rewrite everything that reads OpenCorpora grammemes into UD FEATS. Accept losing the candidate list, which no UD pipeline provides. Accept losing two points of lemma accuracy. Verdict: moderate. Run both in parallel instead, as persona-language-learning-builder recommends — 30 MB and two answers is cheaper than a schema migration.

Leaving UDPipe or MorphoDiTa for license reasons (Czech)#

Reason: the product went commercial. Cost: high, and there is no equivalent destination. PDT positional tags do not map cleanly to UD, the permissive alternatives are less accurate, and spaCy has no Czech pipeline. Realistic exits: Stanza on a CC BY-SA package (a tagset migration and an accuracy drop), or simplemma for lemmas only (a capability drop). Verdict: the most expensive migration in this survey. Which is the argument for deciding the license question first.

Changing SudachiDict edition or version (Japanese)#

Reason: a quarterly release, or needing full for proper nouns. Cost: segmentation changes for compounds, so every derived key changes. Not a code change at all — a re-index. Verdict: low effort, high blast radius. Pin the dictionary version explicitly and schedule upgrades as data migrations.

Leaving simplemma#

Reason: needing features. Cost: near zero. It produces one string and nothing depends on its internals. Verdict: this is what makes it a good starting point. Starting with the cheapest tool that might work costs nothing if it turns out not to.

Leaving a finite-state transducer you wrote#

Reason: none, usually. Cost: there is nowhere to go, and no vendor to leave. The artifact is the asset. Verdict: maximum control, zero optionality. The correct trade for the low-resource persona and a poor one for anyone else.

Two properties that reduce exposure across the board#

Everything permissive here is forkable. SudachiPy, SudachiDict, pymorphy3, simplemma, spaCy, Stanza, and the pymorphy dictionary build scripts are all under licenses that permit a fork. pymorphy2 → pymorphy3 is a completed proof that this works in practice: the original stalled, someone forked the code half, and 501,262 downloads a month now flow through the fork.

Everything encumbered is still readable. CC BY-NC-SA restricts commercial use, not inspection. A team blocked from shipping a UFAL model can still use it to generate evaluation data, to check a permissive model’s output, and to decide whether the accuracy gap matters — which is exactly what S2 did.

The strategic reading#

Lemma-only consumers have almost no lock-in. Their output is a string and their tools are interchangeable. They should choose on footprint and throughput and revisit freely.

Feature consumers are locked in at the tagset, and should choose the tagset before the tool. UD is the portable choice and costs a coarser feature inventory; PDT and OpenCorpora are richer and strand you.

The Czech commercial case is the one to think about years ahead, because it is the only scenario in this survey with an expensive exit and no good destination.

And in every case, the key set is a contract. Write a test that asserts on it, because nothing else in the toolchain will notice when it changes.

Organizational Fit#

Which team can carry which option, and what each choice costs in people rather than in compute.

The hidden staffing question#

Every option in this category carries an obligation that does not appear in its documentation: someone has to know which model is loaded and why.

That obligation is small — a line in a runbook, a pinned string, a test — and it is the thing that is skipped when nobody owns it. The consequences from S2 and S4:

• The wrong model package costs 11-19 points of UFeats and changes the emitted feature key set (S2-comprehensive/the-model-inside-the-library.md).
• The wrong model package can change the license from CC BY-SA to CC BY-NC-SA in a one-word diff (licensing-risk.md).
• A lexicon upgrade changes segmentation and therefore every downstream key (lexicon-durability.md).

None of these produces an error. All of them produce a silent behavior change. A team with nobody assigned to the analyzer will absorb all three without noticing.

By team size#

Solo developer or side project#

Constraint: no time for a second dependency, no license review, no ops.

Fits: simplemma (one dependency, 54 languages, no model), pymorphy3 (one dependency, Russian, 16 MB), SudachiPy + SudachiDict (two dependencies, Apache-2.0, no download step).

Does not fit: anything with a manual model download, anything requiring a license judgment, anything with a per-language integration.

The one discipline to keep: pin the dictionary version. It is one line and it is the difference between a reproducible project and one that quietly changes behavior on the next pip install.

Small team, one or two languages#

Constraint: one person will own this and will also own four other things.

Fits: spaCy (largest ecosystem, most documentation, most likely that a teammate has used it) for Russian; SudachiPy for Japanese; the pymorphy3 + spaCy pairing from persona-language-learning-builder when both halves are needed.

Does not fit: MorphoDiTa or UDPipe unless someone is comfortable with a non-UD tagset and a manual model download; Stanza’s defaults without a decision about packages.

The one discipline to keep: write the package string explicitly, even when it matches the default. It documents a decision and it survives the owner leaving.

Larger team or platform#

Constraint: many languages, a license policy, an audit trail, and turnover.

Fits: Stanza as the uniform layer (95 languages, one API, one schema, Apache-2.0 code) with pinned CC BY-SA packages; simplemma as the cheap tier for lemma-only languages; specialists only where a measurement justifies them.

Requires: a per-language license record, a model-identity manifest alongside the lockfile, and a test that asserts on the feature key set. lock-in-and-exit.md argues that the key set is a contract; at this scale it has to be an enforced one.

Does not fit: a per-language-specialist strategy without a tier plan. It works through language three and becomes an integration backlog by language eleven.

University or research group#

Constraint: reproducibility above everything; commercial licensing is a non-issue.

Fits: the whole UFAL stack — MorphoDiTa, UDPipe 1 and 2, MorfFlex — which are the accuracy leaders for Czech and are CC BY-NC-SA, invisible in this context. Plus Stanza for breadth, and fugashi with full unidic for Japanese corpus work.

Requires: recording model identity in the methods section. stanza==1.14.0 is not a reproducible pipeline; stanza 1.14.0, package='cac' is (persona-corpus-linguist).

The risk to name early: a corpus annotated with CC BY-NC-SA-derived models cannot later be licensed to a commercial partner without rework. That transition is worth thinking about before the annotation runs, when it costs a package string, rather than after, when it costs a re-annotation.

Community language organization#

Constraint: the resource must outlive the grant, the student, and the maintainer.

Fits: the finite-state ecosystems — HFST, Apertium, GiellaLT — which are built for exactly this and are invisible to a PyPI search. GPL, community-maintained, 784 language repositories across two GitHub organizations (persona-low-resource-language).

Requires: linguist time rather than engineering time, and a speaker in the loop.

The discipline that matters most: publish it. The most common failure in this space is a completed morphology that dies with its funding.

By expertise available#

The third row is the one that catches teams. A license question that nobody is assigned to answer gets answered by whoever picked the dependency, which in this category is the wrong answer roughly half the time (S2-comprehensive/lexicon-licensing.md).

The three artifacts every team should have#

Small, and they cover most of what S2 and S4 found goes wrong.

1. A model manifest beside the lockfile. Not stanza==1.14.0, but:

stanza 1.14.0   ru=gsd (CC BY-SA 4.0)   cs=cac (CC BY-SA 4.0)
sudachidict-core 20260723 (Apache-2.0)

Version, model, license, in one place. This is the mitigation for four of the five risks in licensing-risk.md.

2. A key-set assertion in the test suite. One test per language that checks the feature keys the pipeline emits on a fixed sentence. It catches the silent model swap, which nothing else does.

3. A dictionary-upgrade runbook. A one-paragraph note saying that a lexicon version bump changes segmentation and requires a re-index. It is the difference between a scheduled migration and an incident.

What this costs in people#

• Solo or small team, permissive tools only: hours, once. Pin the versions, write one test, move on.
• Platform team, multiple languages: the manifest and the per-language license record are ongoing, at roughly an hour per language added.
• Research group: the model identity goes in the methods section, which is work the group already does.
• Community language project: linguist-months to linguist-years, and no software choice changes that.

The category’s total cost of ownership is dominated by knowing what is loaded. That is unusual for a library category, and it follows directly from the two-halves structure: the half that decides your accuracy, your license, and your output shape is data, and data does not appear in a dependency graph.

S4 Recommendation#

The strategic view of a category whose defining property is that the thing deciding your outcome is data, not code — and data does not appear in a dependency graph.

The three strategic paths#

Conservative — permissive, packaged, pinned#

For anyone shipping a product, and for anyone who cannot afford a surprise.

Everything here ships its data in the package or downloads it under permissive terms, nothing has a manual license judgment attached, and every option was verified from a primary source on 2026-08-25.

The known cost: Czech is second-best. The accuracy leaders for Czech are CC BY-NC-SA and there is no permissive equivalent.

Accuracy-first — take the best model, accept the terms#

For research, internal tools, and anything that will not be sold.

UDPipe 2’s russian-gsd reached 97.1% case and 94.4% UFeats on encyclopedic prose with gold tokenization; MorphoDiTa reached 93.6% Czech case and 96.3% lemma. These are the best numbers in this survey.

The known cost: the exit is the most expensive migration in the category (lock-in-and-exit.md). If there is any chance the work becomes a product, that decision is cheapest to make now.

Adaptive — start cheap, measure, escalate#

For anyone who does not yet know which output column they need.

1. Start with simplemma. 19 MB, MIT, 54 languages, 93-95% lemma accuracy on the Slavic gold measured in S2. Exit cost is near zero because its output is one string.
2. Measure on your own text. A hundred sentences and a reading of the disagreements answers more than any table in this survey, because register moves the answer by more than the tool does.
3. Escalate only where a measurement says to. If you need features, add a disambiguator — that is the 45-point step. If you need candidates, add a lexicon.
4. Pin whatever you land on, and record the model.

This is the path most readers should take, because most readers overestimate how much morphology they need and underestimate how much the model choice matters.

The five-year view#

The lexicons will outlive the libraries. MorfFlex predates UDPipe and is called by it. UniDic outlived MeCab’s development. IPAdic outlived the organization that built it. Choose the lexicon deliberately, because it is the durable asset — and prefer one with an institutional or commercial reason to be rebuilt.

Universal Dependencies will remain the center, and its 150-language ceiling will remain the category’s ceiling. UD FEATS portability between spaCy, Stanza, and UDPipe is the largest reduction in lock-in this category has, and is worth the coarser feature inventory. Outside UD, the answer is a different ecosystem entirely.

Large language models will not replace this category, and will improve half of it. The lexicon half is a completeness claim a probabilistic model cannot make; the disambiguator half is exactly what a language model is good at, and is also the half that already works at 93-97% on a CPU. The likely shape is what UDPipe 2 already does: a lexicon constrains the candidates, a learned model chooses among them.

The Czech commercial gap will not close on its own. The non-commercial restrictions trace to corpus material whose rights sit with publishers, not with the tool maintainers, so UFAL cannot relicense unilaterally. The gap closes when someone trains and releases a permissive Czech model on CC BY-SA treebanks. Nothing prevents it; nobody has done it.

Risk register#

The one-line version#

Pick the lexicon, not the library — then write its identity down.

Every strategic finding in S4 reduces to that. The lexicon decides your accuracy ceiling, your license, your output shape, and your maintenance exposure. The library is an adapter over it. And the lexicon is the one thing pip freeze will not tell you, which is why the highest-leverage artifact any team in this category can produce is a five-line manifest recording which model, at which version, under which terms.

What S4 could not settle#

• Whether ufal-udpipe’s 52% download decline is attrition or migration to the REST service. Both readings fit the data; one interval is weak evidence either way.
• Whether a trained model is a derivative work of its training corpus. Three vendors have taken three positions. This survey reports what each declares and offers no legal conclusion.
• OpenCorpora’s own dictionary terms. opencorpora.org returned HTTP 521 on 2026-08-25 and the upstream license could not be read from a primary source.
• Whether a hybrid LLM disambiguator over a transducer’s candidate set beats a perceptron by enough to matter. The architecture is obvious and the measurement does not exist yet. It is the most interesting open question in the category.