TMX File Format and Translation Memory: Domain Explainer#

Audience: Technical decision makers, product managers, architects without deep localization expertise

What This Solves#

The problem: Professional translators frequently encounter the same phrases, sentences, and terminology across different projects. Re-translating identical content wastes time, costs money, and introduces inconsistency.

Who encounters this:

• Freelance translators managing multiple clients
• Localization service providers (LSPs) coordinating teams
• Software companies releasing products in 50+ languages
• Content publishers with multilingual websites

Why it matters: Translation memory systems reduce translation costs by 30-70% by reusing previous translations. For a software company, this can mean the difference between $50K and $150K per release cycle. For freelance translators, it’s the difference between 500 words/hour and 1500 words/hour.

Accessible Analogies#

Translation Memory as a Bilingual Notebook#

Think of translation memory like a bilingual notebook where translators record every translation decision:

• Left page: Original text in source language
• Right page: Translation in target language
• Margin notes: Context, date, project name

When encountering new text, the translator flips through the notebook looking for similar entries. If “Click to continue” was translated as “Fare clic per continuare” in Italian last month, use the same translation today to maintain consistency.

Translation Memory eXchange (TMX) is the standardized format for this notebook, allowing different translators and tools to read each other’s notebooks.

TMX as Universal Currency for Translation#

Different translation tools (called CAT tools - Computer-Aided Translation) are like different banks:

• SDL Trados is like Bank of America
• memoQ is like Chase
• OmegaT is like a credit union

Each has its own internal format for storing translation records. TMX is like a universal currency that all banks accept, enabling translators to move their “translation wealth” between tools.

Why this matters: A translator switching from Trados to memoQ can bring 10 years of translation memory by exporting to TMX. Without TMX, they’d start from scratch.

TMX Level 1 vs Level 2: Plain Text vs Rich Text#

Level 1 TMX is like storing translations in a plain text file:

• “Click here to continue” → stored as “Click here to continue” (bold lost)
• Simpler, widely compatible, but loses formatting

Level 2 TMX is like storing translations in rich text:

• “Click here to continue” → preserves that “here” is bold
• More complex, but essential for software localization (buttons, menus have formatting)

Analogy: Level 1 is a handwritten recipe card. Level 2 is a recipe with highlighted ingredients, numbered steps, and timing notes.

When You Need This#

You NEED TMX libraries if:#

1. Building translation workflow automation

   • Example: Automatically extract translations from previous projects when starting new ones
   • Example: Pre-translate 70% of software release using last version’s translations

2. Integrating translation tools

   • Example: Your content management system needs to send content to translators and import results
   • Example: Your CI/CD pipeline auto-imports translations for deployment

3. Extracting translation data for machine learning

   • Example: Training a neural machine translation model requires millions of translated sentence pairs
   • Example: Bilingual glossary extraction from professional translation archives

4. Building custom CAT tools

   • Example: Developing industry-specific translation software (medical, legal, gaming)
   • Example: Creating internal translation platform for enterprise use

5. Converting between translation formats

   • Example: Freelancer receives TMX from client, needs to convert to gettext PO format for use in Django
   • Example: Agency needs to convert 50 projects from proprietary format to TMX for new toolchain

You DON’T need TMX libraries if:#

1. Using off-the-shelf CAT tools (Trados, memoQ, Smartcat)

   • These tools handle TMX internally - you just use the GUI

2. Basic internationalization (i18next, gettext, Rails i18n)

   • Framework-specific formats (JSON, PO) are simpler - convert to TMX only if needed

3. Machine translation APIs only (Google Translate, DeepL)

   • APIs handle everything - you send text, receive translation

4. One-time manual translation imports

   • Open TMX in Excel or text editor, copy/paste where needed

Trade-offs#

TMX-native vs Conversion-based Workflows#

TMX-native (using translate-toolkit or hypomnema directly):

• ✅ Pro: Direct manipulation, no format loss, full control
• ✅ Pro: Can handle Level 2 inline markup (bold, links, variables)
• ❌ Con: XML complexity, larger files, steeper learning curve

Conversion-based (using gettext PO, then converting to/from TMX):

• ✅ Pro: Simpler format, git-friendly (text not XML), developer-familiar
• ✅ Pro: Framework integration (Django, Flask, Rails all use PO)
• ❌ Con: Format conversion overhead, potential data loss, two-step workflow

Analogy: TMX-native is like working in Photoshop (powerful, complex). Conversion-based is like using Preview/Paint (simpler, occasional export to PSD when needed).

In-Memory vs Streaming Parsing#

In-memory (translate-toolkit, most libraries):

• ✅ Pro: Fast random access, easy to manipulate
• ✅ Pro: Simpler programming model
• ❌ Con: 10 MB TMX → 50 MB RAM (5x multiplier)
• ❌ Con: 500 MB TMX file crashes on laptop

Streaming (hypomnema only):

• ✅ Pro: Constant memory (50 MB regardless of file size)
• ✅ Pro: Can process multi-GB corpora on laptops
• ❌ Con: Sequential access only (can’t jump to middle)
• ❌ Con: More complex programming patterns

When streaming matters: Processing 1M+ translation units, batch data extraction, resource-constrained servers

Open Source (GPL) vs Permissive (MIT) Licensing#

GPL-2.0+ (translate-toolkit):

• ✅ Pro: Free, well-maintained, comprehensive features
• ❌ Con: If you import translate-toolkit in your code, your software must also be GPL
• ⚠️ Workaround: Use command-line tools (po2tmx, tmx2po) without GPL contamination

MIT (hypomnema, polib):

• ✅ Pro: Embed in commercial/proprietary software freely
• ✅ Pro: No viral licensing restrictions
• ❌ Con: Smaller community, potentially less stable

Analogy: GPL is like renting a powerful excavator - you can use it, but you can’t modify and resell it. MIT is like buying a shovel - do whatever you want with it.

Mature-Stable vs Modern-Evolving#

Mature libraries (translate-toolkit, polib):

• ✅ Pro: 10-15 years of battle-testing, known limitations
• ✅ Pro: Extensive documentation, large community
• ❌ Con: API conventions from Python 2 era (less Pythonic)
• ❌ Con: May lack modern features (type hints, async)

Modern libraries (hypomnema):

• ✅ Pro: Type safety, modern Python patterns (3.12+)
• ✅ Pro: Newer features (streaming, policy-driven parsing)
• ❌ Con: Pre-1.0 (API may change), smaller community
• ❌ Con: Less proven in production environments

Analogy: Mature libraries are like a Toyota Camry - reliable, boring, everyone knows how to fix it. Modern libraries are like a Tesla - innovative, exciting, repair shop network still growing.

Cost Considerations#

When Cost Matters#

Translation memory systems save money primarily through human translator efficiency, not library costs. The libraries themselves are free (open source), but there are indirect costs:

1. Development time: Building translation automation requires engineering effort

   • Simple PO↔TMX conversion: 1-2 days
   • Custom CAT tool integration: 2-4 weeks
   • Full translation memory platform: 3-6 months

2. Infrastructure: Large TMX processing requires server resources

   • Parsing 10 MB TMX: Negligible (runs on laptop)
   • Processing 1 GB corpus: Cloud instance ($50-200/month)
   • Real-time translation memory API: Load balancer, caching ($500-2000/month)

3. Licensing: Only matters if using GPL library in commercial product

   • Using translate-toolkit CLI tools: Free, no restrictions
   • Importing translate-toolkit in proprietary code: Must make code GPL or switch library
   • Using MIT libraries (hypomnema, polib): No restrictions

Build vs Buy#

Build with TMX libraries when:

• Workflow automation internal to company (not reselling)
• Need customization CAT tools don’t provide
• Processing sensitive data (can’t send to cloud services)
• High-volume processing (millions of translation units)

Buy CAT tool subscription when:

• Individual translators or small teams
• Standard translation workflows
• Don’t have engineering resources
• Need full-featured editor, QA tools, project management

Cost example:

• Building: $20K engineering (2 months) + $2K/year infrastructure = $22K year 1, $2K/year after
• Buying: SDL Trados Studio = $700/seat/year × 5 translators = $3.5K/year

Building makes sense at scale (10+ translators, high volume) or for specialized workflows.

Implementation Reality#

Realistic Timeline Expectations#

Simple TMX processing (reading, basic filtering):

• 1-3 days for proof of concept
• 1 week for production-ready script
• Suitable for: One-time migration, batch conversion

Translation memory integration (bidirectional sync with CAT tool):

• 2-4 weeks for MVP
• 2-3 months for production system
• Requires: TMX library + storage + API + error handling

Custom CAT tool (editor, fuzzy matching, terminology):

• 3-6 months for MVP
• 1-2 years for competitive product
• Requires: TMX + fuzzy matching algorithm + UI + project management

Team Skill Requirements#

Minimum viable skills:

• Python basics (loops, file I/O)
• XML awareness (structure, tags, attributes)
• Understanding of character encodings (UTF-8)

For production systems:

• Error handling (malformed XML, encoding issues)
• Memory management (large files)
• Testing (edge cases, round-trip integrity)

Advanced features:

• Fuzzy matching algorithms (Levenshtein distance)
• Translation memory segmentation (sentence splitting)
• CAT tool integration (format conversion, API design)

Common Pitfalls#

1. Assuming all TMX files are well-formed

   • Reality: Real-world TMX files have encoding errors, malformed XML, vendor-specific extensions
   • Solution: Use policy-driven parsing (hypomnema) or lenient parsing modes

2. Underestimating memory requirements

   • Reality: 10 MB TMX → 50 MB RAM (5x multiplier)
   • Solution: Use streaming API or process in batches

3. Ignoring TMX Level differences

   • Reality: Level 1 loses formatting (bold, links, variables in UI strings)
   • Solution: Use Level 2 library (hypomnema) if software localization

4. Not testing round-trip integrity

   • Reality: Read TMX → process → write TMX may lose data
   • Solution: Compare input/output XML, validate against TMX schema

5. Licensing confusion

   • Reality: Using translate-toolkit code in commercial product requires GPL compliance
   • Solution: Use CLI tools (no GPL contamination) or switch to MIT library (hypomnema, polib)

First 90 Days: What to Expect#

Weeks 1-2: Library selection and proof-of-concept

• Research libraries (S1 rapid discovery)
• Parse sample TMX files from actual workflow
• Verify Level 1/Level 2 requirements

Weeks 3-6: Core functionality implementation

• Read/write TMX with production data
• Handle edge cases (encoding, malformed XML)
• Unit tests for common operations

Weeks 7-12: Integration and production hardening

• Integrate with existing systems (CMS, CI/CD)
• Error monitoring and logging
• Performance optimization for large files
• Documentation for team

Common surprise: Real-world TMX files are messier than spec. Budget time for handling vendor-specific quirks, encoding issues, and malformed XML.

Summary#

TMX libraries enable translation memory automation - reusing previous translations to reduce costs and maintain consistency.

Choose based on your context:

• Production stability: translate-toolkit (GPL, comprehensive)
• Commercial products: hypomnema or polib (MIT license)
• PO-based workflows: polib + translate-toolkit conversion
• Large files: hypomnema (streaming API)

Realistic expectations:

• Simple automation: 1-2 weeks
• Translation memory integration: 2-3 months
• Custom CAT tools: 6-12 months

Key decision: TMX-native vs conversion-based workflows shape your architecture. Consider licensing, format requirements (Level 1 vs 2), and file sizes when selecting libraries.

S1 Rapid Discovery: TMX Libraries#

Critical Name Collision: TMX#

WARNING: “TMX” refers to TWO completely different file formats:

1. Translation Memory eXchange (TMX) - This research

   • XML-based standard for translation memory
   • Used in CAT (Computer-Aided Translation) tools
   • Contains source/target language pairs for translation reuse
   • Industry standard: GALA (Globalization and Localization Association)

2. Tiled Map XML (TMX) - NOT this research

   • Game development tile map format
   • Used by Tiled Map Editor
   • Contains sprite sheets, tile layers, collision data
   • Different XML schema entirely

Why This Matters#

The research brief mentioned “tmxlib” - this is a game tile map library, NOT a translation memory library. PyPI search for “tmx” returns both types, creating confusion.

Libraries excluded due to wrong TMX:

• tmxlib - Tiled map editor files (game development)
• pytmx - Pygame tile map loader (game development)
• python-tmx (DEPRECATED) - Was for translation, now superseded by hypomnema

Libraries Identified#

Native TMX Translation Memory Libraries:#

1. translate-toolkit (production-ready)

   • Part of comprehensive localization toolkit
   • TMX 1.4 Level 1 support
   • GPL-2.0+ license
   • 933 GitHub stars
   • Latest: v3.18.1 (Jan 2026)

2. hypomnema (modern, focused)

   • Full TMX 1.4b Level 2 support (nested inline markup)
   • Type-safe Python 3.12+
   • MIT license
   • 8 GitHub stars
   • Pre-1.0 active development
   • Note: Formerly called “python-tmx” (deprecated Dec 2025)

Indirect TMX Support:#

3. polib (gettext library with TMX conversion)
   • Mature PO/POT/MO file library
   • Does NOT natively parse TMX
   • Enables PO↔TMX via translate-toolkit conversion
   • MIT license
   • Latest: v1.2.0 (Feb 2023)

Initial Categorization#

By Format Support:#

• Native TMX: translate-toolkit, hypomnema
• TMX via conversion: polib (using translate-toolkit)

By Maturity:#

• Production-stable: translate-toolkit, polib
• Pre-1.0: hypomnema (active development, breaking changes expected)

By License:#

• GPL-2.0+: translate-toolkit (copyleft, requires derivative works to be GPL)
• MIT: hypomnema, polib (permissive, commercial-friendly)

Focus for S1#

Analyze all three libraries:

• translate-toolkit: The industry standard
• hypomnema: The modern alternative
• polib: The gettext ecosystem bridge

Document their TMX capabilities, use cases, and when to choose each.

hypomnema#

Overview#

Modern, type-safe Python library focused exclusively on TMX (Translation Memory eXchange) format. Built for Python 3.12+ with full type hints, streaming API, and TMX 1.4b Level 2 support (nested inline markup).

Package: hypomnema (PyPI, formerly python-tmx) License: MIT Status: Pre-1.0 (active development) Python: ≥3.12

Note: python-tmx package deprecated December 2025, superseded by hypomnema.

Maintenance Status#

Active Development (Pre-1.0)

• Current: Pre-release development phase
• History: Renamed from python-tmx (Dec 2025)
• GitHub: 8 stars (small community)
• Maintainer: Single-author project (Enzo Agosta)
• Breaking changes expected until 1.0 release

Key Capabilities#

TMX Conformance#

• Level: Full TMX 1.4b Level 2 support
• Parsing: Structured inline markup (<bpt>, <ept>, <it>, <ph>, <hi>)
• Writing: Roundtrip integrity with nested elements
• Validation: Policy-driven (strict/permissive/custom)

Architecture Highlights#

• Type-safe: Full Python 3.12+ type annotations
• Dataclass-based: Efficient memory representation (1.5-2 KB per unit)
• Backend-agnostic: Optional lxml (fast) or stdlib XML (portable)
• Policy-driven: Configure parsing strictness per use case

Streaming API#

Unique feature: Constant-memory streaming for large TMX files

• Parse multi-GB files without loading entire file into memory
• ~50 MB RAM regardless of file size
• Sequential access (suitable for ETL pipelines, corpus extraction)

Validation Policies#

• Strict: Reject malformed TMX (standards-compliant only)
• Permissive: Best-effort parsing (recover from real-world errors)
• Custom: Define error handling per field/element

Strengths#

TMX Level 2 support: ONLY Python library with full nested inline markup (essential for software localization with UI formatting)

Streaming capability: Process 1M+ translation units on laptops without memory exhaustion

Type safety: Full type hints enable IDE autocomplete, static analysis, early error detection

MIT license: Commercial-friendly, embed in proprietary software without copyleft restrictions

Modern Python: Built for Python 3.12+, idiomatic dataclass patterns, no legacy cruft

Backend flexibility: Choose lxml (2x faster) or stdlib XML (zero C dependencies)

Limitations#

Pre-1.0 status: API breaking changes expected, not recommended for risk-averse production

Small community: 8 GitHub stars, limited Stack Overflow coverage, single maintainer risk

Python 3.12+ only: Cannot use in codebases stuck on Python 3.11 or earlier

TMX-only: No multi-format support (no PO, XLIFF, TBX) - must use translate-toolkit for conversions

Documentation: README-based, no comprehensive tutorial or API reference site

No CLI tools: Python API only, requires scripting for automation (unlike translate-toolkit’s po2tmx)

Performance Characteristics#

• Parse speed (lxml backend): ~2 seconds per 100K units (fastest option)
• Parse speed (stdlib backend): ~20 seconds per 100K units (portable but slower)
• Memory overhead: 1.5-2 KB per unit (most efficient of three libraries)
• Streaming mode: Constant ~50 MB RAM for any file size

Best For#

NLP/ML pipelines: Extract parallel corpora from TMX for machine translation training (streaming API handles large datasets)

Software localization: UI strings with inline markup (<b>, <a>, variables) require Level 2 support

Commercial CAT tools: MIT license allows embedding in proprietary translation software

Modern Python projects: Type-safe codebases on Python 3.12+ benefit from full annotations

Custom validation: Policy-driven parsing handles vendor-specific TMX extensions gracefully

Not Ideal For#

Production-critical systems: Pre-1.0 API instability risks breaking changes during upgrades

Multi-format workflows: No native PO/XLIFF support (must integrate with translate-toolkit)

Legacy Python: Projects on Python 3.11 or earlier cannot use hypomnema

Risk-averse teams: Small community means limited support, single-maintainer dependency

Command-line automation: No CLI tools, requires Python scripting

Ecosystem Position#

Modern specialist: Hypomnema is the “precision tool” - focused on TMX Level 2 excellence, streaming performance, and type safety.

Strategic bet: Choosing hypomnema is betting on modern Python patterns over legacy stability. High reward (best-in-class TMX Level 2 + streaming) with risk (pre-1.0, small community).

Quick Stats#

Decision Factors#

Choose hypomnema if:

• TMX Level 2 required (nested inline markup)
• MIT license needed for commercial use
• Large files require streaming (>500 MB)
• Type safety priority (Python 3.12+)
• TMX-only focus acceptable

Choose alternative if:

• Production stability required (→ translate-toolkit)
• Pre-1.0 risk unacceptable (→ translate-toolkit)
• Multi-format needed (→ translate-toolkit)
• Python <3.12 required (→ translate-toolkit or polib)
• Large community desired (→ translate-toolkit)

Pre-1.0 Risk Mitigation#

If choosing hypomnema despite pre-1.0 status:

1. Pin exact version in requirements.txt
2. Monitor GitHub for breaking changes
3. Vendor the library if stability critical
4. Have migration path to translate-toolkit ready
5. Contribute to community (increase maintainer support)

polib#

Overview#

Mature Python library for manipulating gettext PO/POT/MO files. Does NOT natively parse TMX, but enables PO↔TMX workflows via translate-toolkit conversion. Widely used in Django, Flask, and other Python web frameworks.

Package: polib (PyPI) License: MIT Status: Stable/Maintenance mode Python: 2.7-3.11 (wide compatibility)

Maintenance Status#

Stable/Maintenance (Mature)

• Latest version: 1.2.0 (Feb 1, 2023)
• History: 10+ years of development
• GitHub: Established user base
• Release cadence: Infrequent (stable, feature-complete)
• Community: Used by Django, Flask localization ecosystems

Key Capabilities#

Native Format Support#

• PO files: Gettext Portable Object (human-readable translation format)
• POT files: Gettext template files
• MO files: Compiled binary translation files
• TMX: NOT natively supported (requires translate-toolkit for conversion)

Core Operations#

• Read/write/edit PO/POT/MO files
• Merge translations
• Detect encoding
• Handle plural forms
• Preserve comments and metadata

TMX Integration Pattern#

polib itself does NOT handle TMX. To work with TMX:

1. Use polib for PO file manipulation
2. Use translate-toolkit’s po2tmx / tmx2po for conversion
3. Result: PO-centric workflow with occasional TMX export/import

Strengths#

Zero dependencies: Pure Python with no external requirements (no lxml, no C extensions)

Wide Python support: Works on Python 2.7-3.11 (broadest compatibility of three libraries)

Framework integration: Native support in Django (makemessages, compilemessages) and Flask-Babel

Git-friendly format: PO files are line-based text (easy diffs, merge conflicts, code review)

Developer-centric: PO format designed for developers (comments, source code references, fuzzy flags)

Mature and stable: 10+ years in production, feature-complete, rare breaking changes

MIT license: Commercial-friendly, no copyleft restrictions

Limitations#

NO native TMX support: Cannot parse or write TMX directly (must use translate-toolkit as intermediary)

Format conversion overhead: PO→TMX→PO round-trip may lose PO-specific metadata (fuzzy flags, comments, locations)

Python 3.12+ unknown: No explicit Python 3.12+ testing/support claims

Minimal type hints: Limited type annotations compared to modern libraries

Not for TMX-only workflows: If project uses TMX as primary format, polib adds unnecessary complexity

Performance Characteristics#

• Parse speed: ~5 seconds per 100K entries (PO format, not TMX)
• Memory overhead: ~1 KB per entry (PO simpler than TMX XML)
• TMX conversion: Additional overhead via translate-toolkit (not measured)

Best For#

Django/Flask internationalization: Native framework integration, PO is standard format for these ecosystems

Developer-centric workflows: Developers prefer PO (text-based, git-friendly) over TMX (XML-based)

Version control: PO files merge better than TMX in git (line-based vs XML)

Python 2.7 compatibility: Only option if stuck on Python 2.7 (translate-toolkit and hypomnema require 3.11+/3.12+)

Zero dependency requirements: Environments where lxml or C extensions prohibited

Not Ideal For#

TMX-native workflows: If CAT tools and translators use TMX as primary format, polib adds conversion step

Direct TMX manipulation: No API for TMX parsing/writing (must shell out to translate-toolkit CLI)

Software localization: UI strings with inline markup need TMX Level 2 (polib + translate-toolkit only supports Level 1)

Large-scale translation memories: PO format not designed for million-unit translation memories (TMX is)

Ecosystem Position#

PO format specialist: polib is THE library for gettext PO files in Python. If your workflow is PO-centric, polib is mandatory. TMX support is a bonus via translate-toolkit integration.

Complementary tool: Use polib for PO manipulation + translate-toolkit for PO↔TMX conversion = best of both worlds (git-friendly PO for development, TMX for translator exchange)

Quick Stats#

Decision Factors#

Choose polib if:

• PO/POT/MO is primary format
• Django/Flask framework used
• Git-friendly format preferred
• Zero dependencies required
• Python 2.7-3.11 compatibility needed
• TMX is secondary (occasional export/import)

Choose alternative if:

• TMX is primary format (→ translate-toolkit or hypomnema)
• Need native TMX parsing (→ translate-toolkit or hypomnema)
• Python 3.12+ required (→ translate-toolkit or hypomnema)
• TMX Level 2 needed (→ hypomnema)

TMX Workflow Pattern#

Typical polib + TMX integration:

1. Development: Use polib to manage PO files (git-friendly, developer workflow)
2. Export for translators: Use translate-toolkit po2tmx to create TMX for CAT tools
3. Import from translators: Use translate-toolkit tmx2po to update PO files
4. Result: Developers work in PO, translators work in TMX, translate-toolkit bridges the gap

This pattern works when:

• Development team uses gettext (Django, Flask, GNU tools)
• Translation team uses CAT tools (Trados, memoQ, OmegaT)
• Both sides benefit from their preferred format

This pattern fails when:

• Frequent bidirectional sync required (conversion overhead adds latency)
• TMX Level 2 inline markup needed (conversion loses fidelity)
• Translation memory is primary asset (PO format not ideal for large TM databases)

S1 Rapid Discovery: Recommendation#

Quick Decision Guide#

For rapid selection of TMX translation memory libraries in Python:

Default Choice: translate-toolkit#

Use when: You need production-ready TMX support and don’t have specific constraints

Rationale:

• Battle-tested in enterprise localization workflows
• Active maintenance (v3.18.1, Jan 2026)
• Comprehensive format support (TMX + 20 other formats)
• Command-line tools for automation
• 933 GitHub stars, used by Weblate, Pootle

Trade-off: GPL-2.0+ license may restrict commercial use

Modern Alternative: hypomnema#

Use when: You need full TMX 1.4b Level 2 support or MIT licensing

Rationale:

• Full TMX Level 2 (nested inline markup)
• Type-safe Python 3.12+ with full type hints
• Streaming API for large files (constant memory)
• MIT license (commercial-friendly)
• Modern architecture, policy-driven validation

Trade-off: Pre-1.0 status means breaking API changes possible

PO Workflow Bridge: polib#

Use when: Your primary format is gettext PO, TMX is secondary

Rationale:

• Mature PO/POT/MO library (v1.2.0)
• MIT license
• Zero dependencies
• Convert PO↔TMX via translate-toolkit
• Wide Python version support (2.7-3.11)

Trade-off: Not native TMX support, requires translate-toolkit for conversion

Quick Comparison#

Decision Tree#

Need TMX support?
├─ Yes
│  ├─ Need Level 2 inline markup?
│  │  ├─ Yes → hypomnema (only option)
│  │  └─ No
│  │     ├─ GPL acceptable? → translate-toolkit (production-ready)
│  │     └─ Need MIT license? → hypomnema (modern) or polib+conversion (mature)
│  │
│  └─ Commercial product?
│     ├─ Yes → hypomnema or polib (MIT license)
│     └─ No → translate-toolkit (comprehensive)
│
└─ No, working with PO files?
   └─ Yes → polib (mature, purpose-built)

Use Case Recommendations#

Enterprise Localization Pipeline#

Choice: translate-toolkit

Reason: Production stability, multi-format support (TMX/PO/XLIFF), CLI tools for CI/CD, QA checks

NLP/Machine Translation Data Extraction#

Choice: hypomnema

Reason: Streaming API for large corpora, policy-driven parsing for messy real-world TMX, Level 2 support for inline markup

Django/Flask Internationalization#

Choice: polib (+ translate-toolkit for TMX)

Reason: Native framework integration, PO-friendly version control, convert to TMX when needed

Custom CAT Tool Development#

Choice: hypomnema

Reason: Full Level 2 support, type-safe API, MIT license for commercial distribution

Cross-Format Translation Management#

Choice: translate-toolkit

Reason: Handles 20+ formats, unified API, format conversion tools

Key Insights from S1#

1. Only 2-3 libraries exist for TMX translation memory in Python (small ecosystem)
2. Name collision: “tmxlib” is for game tile maps, NOT translation memory
3. Level 2 support rare: Only hypomnema supports full TMX 1.4b Level 2 (nested inline markup)
4. Licensing matters: GPL (translate-toolkit) vs MIT (hypomnema, polib) affects commercial use
5. PO ecosystem mature: polib + translate-toolkit enables PO↔TMX workflows

Next Steps#

After choosing a library:

1. Validate with your data: Test parsing/writing with actual TMX files
2. Check Level requirements: Verify if Level 1 or Level 2 features needed
3. License review: Ensure GPL or MIT aligns with project requirements
4. Integration test: Confirm compatibility with existing tools (CAT tools, CI/CD)
5. Performance baseline: Measure memory/speed with representative file sizes

When to Reconsider#

Upgrade from translate-toolkit to hypomnema if:

• TMX Level 2 features become required
• GPL licensing creates legal issues
• Large file processing needs streaming API

Downgrade from hypomnema to translate-toolkit if:

• Pre-1.0 API instability causes production issues
• Need multi-format support (PO, XLIFF, TBX)
• Require mature ecosystem and commercial support

Add polib to workflow if:

• Developers prefer PO format for version control
• Framework integration (Django/Flask) beneficial
• Git-friendly text format preferred over XML

Resources#

• translate-toolkit: https://docs.translatehouse.org/projects/translate-toolkit/
• hypomnema: https://github.com/EnzoAgosta/hypomnema
• polib: http://polib.readthedocs.org/
• TMX Spec: https://www.gala-global.org/tmx-14b

translate-toolkit#

Overview#

Production-ready localization toolkit with comprehensive TMX (Translation Memory eXchange) support. Part of Translate House’s ecosystem, widely used by major open-source localization platforms.

Package: translate-toolkit (PyPI) License: GPL-2.0-or-later Status: Production/Stable Python: ≥3.11

Maintenance Status#

Actively Maintained (January 2026)

• Latest version: 3.18.1 (Jan 14, 2026)
• Regular releases: Monthly cadence (3.17.0-3.18.1 in Nov 2025-Jan 2026)
• GitHub: 933 stars, 328 forks, 263 open issues
• Community: Active development, used by Weblate and Pootle
• History: 15+ years of continuous development

Key Capabilities#

TMX Conformance#

• Level: TMX 1.4 Level 1 conformance
• Parsing: Reads TMX translation memory files
• Writing: Generates TMX from translation data
• Markup: Preserves inline formatting (does NOT strip markup)

Format Ecosystem#

TMX is one of 20+ supported formats:

• Primary: PO, XLIFF, TMX, TBX
• Development: Properties, RC, RESX, DTD
• Office: OpenOffice.org GSI/SDF
• Other: Qt .ts, CSV, WordFast, subtitles

Command-Line Tools#

• po2tmx / tmx2po: Format conversion
• pofilter: 40+ translation quality checks
• pocount: Word count and analysis
• pretranslate: Apply translation memory
• poterminology: Extract terminology

Python API#

• Storage abstraction layer (unified API across formats)
• Programmatic TMX manipulation
• Batch processing capabilities
• Translation unit iteration and filtering

Strengths#

Comprehensive toolkit: Multi-format support means one tool for entire localization workflow (PO ↔ TMX ↔ XLIFF conversions)

Production-proven: Used by major platforms (Weblate, Pootle), 15+ years in production, extensive real-world validation

Command-line automation: Ready-to-use CLI tools for CI/CD pipelines, no Python knowledge required for basic operations

Quality assurance: Built-in 40+ translation checks (variables, punctuation, consistency) integrated with TMX workflows

Active maintenance: Monthly releases, responsive to issues, security-conscious (signed commits)

Limitations#

GPL licensing: Copyleft license requires derivative works to be GPL (may restrict commercial use in proprietary software)

TMX Level 1 only: Does NOT support TMX Level 2 nested inline markup (e.g., <bpt>, <ept> tags for complex formatting)

Memory usage: Loads entire TMX into memory (5x file size multiplier) - 10 MB file = 50 MB RAM

Python 3.11+ only: Dropped support for Python 3.10 and earlier (as of late 2025)

lxml dependency: Requires C-compiled lxml library (not pure Python)

Performance Characteristics#

• Parse speed: ~5 seconds per 100K translation units (adequate for most workflows)
• Memory overhead: 2-4 KB per translation unit (lxml DOM + wrapper objects)
• Scalability: Suitable for files up to 500 MB; larger files require splitting or batch processing

Best For#

Multi-format localization workflows: Teams working with PO, XLIFF, and TMX need unified conversion pipeline

Open-source projects: GPL license aligns with open-source ecosystems (Django, GNOME, KDE)

Enterprise localization platforms: Production stability and comprehensive tooling justify GPL restrictions

CLI automation: DevOps teams prefer command-line tools over Python API

Weblate/Pootle integration: Native support in these platforms

Not Ideal For#

Commercial proprietary software: GPL licensing complicates embedding in closed-source products (workaround: use CLI tools without importing library)

TMX Level 2 requirements: Software localization with nested inline markup needs hypomnema instead

Large file streaming: Files >500 MB need constant-memory streaming (hypomnema offers this)

Pure Python environments: lxml C dependency may complicate deployment in restricted environments

Modern type-safe codebases: Limited type hints compared to newer libraries

Ecosystem Position#

De facto standard for Python-based localization automation. Translate-toolkit is the “Swiss Army Knife” - if you need TMX + 20 other formats + QA tools + CLI automation, this is the proven choice.

Strategic dependency: Weblate (20K+ stars) and Pootle depend on translate-toolkit, ensuring long-term viability.

Quick Stats#

Decision Factors#

Choose translate-toolkit if:

• Production stability is priority
• Multi-format support needed (PO, XLIFF, TMX)
• Command-line automation required
• GPL licensing acceptable
• Weblate/Pootle integration planned

Choose alternative if:

• MIT license required (→ hypomnema, polib)
• TMX Level 2 needed (→ hypomnema)
• Streaming large files (→ hypomnema)
• Pure Python required (→ polib for PO, convert to TMX)

S2 Comprehensive Analysis: TMX Libraries#

Libraries Under Analysis#

1. translate-toolkit#

Focus areas:

• Storage abstraction architecture
• TMX parser implementation (lxml-based)
• Memory management strategies
• Format conversion pipeline
• Validation and conformance checking
• Command-line tooling architecture

2. hypomnema#

Focus areas:

• Policy-driven deserialization system
• Backend architecture (lxml vs stdlib)
• Streaming API implementation
• Type system and dataclass design
• Level 2 inline markup handling
• Error recovery mechanisms

3. polib#

Focus areas:

• PO format internal representation
• TMX conversion via translate-toolkit
• Pure Python implementation details
• Merging and deduplication algorithms
• Performance characteristics
• Gettext ecosystem integration

Analysis Framework#

Architecture Analysis#

For each library:

• Module structure: Package organization, dependency graph
• Core abstractions: Base classes, interfaces, protocols
• Data flow: From file → parsed object → serialized output
• Extension points: Subclassing, callbacks, plugins

Performance Analysis#

Metrics to examine:

• Memory usage: Overhead per translation unit, total file size impact
• Parsing speed: Time complexity, bottlenecks
• Concurrent access: Thread safety, multi-process considerations
• Scalability: Performance with 10K, 100K, 1M translation units

API Design Analysis#

Evaluation criteria:

• Type safety: Static typing, runtime validation
• Ergonomics: Pythonic patterns, ease of use
• Error handling: Exception hierarchy, recovery options
• Flexibility: Configuration options, customization points

TMX Conformance Analysis#

Standard compliance:

• TMX 1.4b spec: Which features supported
• Level 1 vs Level 2: Inline markup handling
• Validation strictness: Schema enforcement, error tolerance
• Edge cases: Namespace handling, encoding issues, malformed XML

TMX Technical Context#

TMX 1.4b Standard Overview#

Structural elements:

• <tmx>: Root, required attributes: version
• <header>: Metadata (creationtool, srclang, datatype, etc.)
• <body>: Container for translation units
• <tu>: Translation unit (one conceptual translation)
• <tuv>: Translation unit variant (language-specific segment)

Inline elements (Level 2):

• <bpt>: Begin paired tag (e.g., <b> open)
• <ept>: End paired tag (e.g., </b> close)
• <it>: Isolated tag (e.g., <br/>)
• <ph>: Placeholder (e.g., {0})
• <hi>: Highlight
• <sub>: Subflow (arbitrary nesting depth)

Level 1 vs Level 2:

• Level 1: Inline tags present but NOT nested deeply
• Level 2: Arbitrary nesting depth, complex inline structures

File Size Characteristics#

Real-world TMX files:

• Small: <1 MB, <10K translation units (project-specific)
• Medium: 1-100 MB, 10K-100K units (domain-specific TM)
• Large: 100 MB-1 GB, 100K-1M units (corporate memory)
• Very large: >1 GB, >1M units (industry-wide aggregations)

Memory implications:

• In-memory parsing: File size × 2-5x overhead (DOM structures)
• Streaming parsing: Constant memory (~10-50 MB regardless of file size)

Common TMX Challenges#

Encoding issues:

• UTF-8 vs UTF-16 (BOM handling)
• Legacy encodings (ISO-8859-1)
• Mixed encodings within file

XML edge cases:

• Namespace prefixes without declarations
• Malformed XML (unclosed tags, invalid nesting)
• CDATA sections
• Entity references

Performance bottlenecks:

• Large files (multi-GB)
• Deep inline nesting (Level 2)
• Many translation units with complex attributes
• Concurrent read/write access

Methodology for Technical Deep-Dive#

Source Code Examination#

Repository analysis:

1. Clone repositories
2. Analyze directory structure
3. Identify core modules
4. Trace execution paths (parse, serialize)
5. Document key algorithms

Example for translate-toolkit:

# translate/storage/tmx.py - main module
# translate/storage/base.py - storage abstraction
# translate/storage/pypo.py - PO integration

Performance Testing Framework#

Synthetic benchmark approach:

1. Generate TMX files of varying sizes (10K, 100K, 1M units)
2. Measure parse time, memory usage
3. Compare streaming vs in-memory
4. Test concurrent access patterns

Real-world file testing:

1. Obtain sample TMX from CAT tools (Trados, memoQ)
2. Test with malformed files (missing attributes, encoding issues)
3. Measure error recovery and validation behavior

API Surface Area Analysis#

Metrics:

• Public classes/functions count
• Configuration options
• Extension points (callbacks, subclassing)
• Type annotation coverage
• Documentation completeness

Example metrics to collect:

• translate-toolkit: Storage API classes, conversion functions
• hypomnema: Policy options, backend choices, dataclass fields
• polib: POEntry attributes, merge strategies

Deliverables#

File Outputs#

1. approach.md (this file): Methodology and framework
2. translate-toolkit.md: Deep technical analysis
3. hypomnema.md: Deep technical analysis
4. polib.md: Deep technical analysis (including TMX conversion mechanics)
5. feature-comparison.md: Technical feature matrix with measurements
6. recommendation.md: Technical selection criteria (HOW capabilities map to requirements)

Feature Comparison Matrix Structure#

Dimensions to compare:

• Parsing capabilities: Formats, validation levels, error recovery
• Writing options: Serialization formats, encoding control, pretty-printing
• Memory characteristics: Overhead, streaming support, scalability
• Performance metrics: Parse speed, write speed, benchmark results
• API surface: Classes, methods, configuration options
• Type safety: Annotation coverage, runtime checking
• Extension points: Subclassing, callbacks, plugins
• TMX conformance: Level 1/2, standard compliance, edge cases

Recommendation Document Structure#

Not “when to use” but “how to evaluate”:

• Technical requirement categories
• Measurable selection criteria
• Trade-off analysis frameworks
• Risk assessment for each choice

Key Questions to Answer#

Architecture#

• How do libraries represent TMX internally (data structures)?
• What XML parsing library do they use (lxml, stdlib, custom)?
• How is the storage abstraction designed?
• Where are the performance bottlenecks in parsing/serialization?

Implementation Details#

• How are inline elements (Level 2) handled?
• What validation strategies are used?
• How is error recovery implemented?
• What memory management strategies are employed?

API Design#

• How type-safe are the APIs?
• What configuration/customization is available?
• How extensible are the libraries?
• What error handling patterns are used?

Performance#

• What is the memory overhead per translation unit?
• How do libraries scale to large files (100K+ units)?
• Is streaming supported, and how is it implemented?
• What are the bottlenecks for read/write operations?

Edge Cases#

• How are malformed TMX files handled?
• What happens with encoding issues?
• How is concurrent access managed?
• What validation errors are caught vs ignored?

Success Criteria for S2#

S2 is successful if:

1. Technical depth achieved: Implementation details documented, not just API signatures
2. Performance characterized: Memory/speed measurements or estimates provided
3. Comparison is quantitative: Feature matrix with measurable criteria
4. Trade-offs explicit: Technical costs/benefits of each library clear
5. Foundation for S3: Enough technical detail to make informed selection decisions

S2 should enable a technical reader to:

• Understand HOW each library works internally
• Predict performance for their use case
• Evaluate technical risks and benefits
• Make informed architectural decisions

This analysis does NOT make the decision (that’s S3), but provides the technical foundation for decision-making.

Feature Comparison Matrix: TMX Libraries#

Overview#

This document provides a detailed, quantitative comparison of TMX translation memory libraries for Python. All measurements and characteristics are based on technical analysis of library internals, not use-case recommendations.

High-Level Comparison#

TMX Format Capabilities#

TMX Standard Conformance#

Supported TMX Elements#

Header Attributes Support#

Parsing Capabilities#

Input Formats#

Parsing Options#

Validation Features#

Writing Capabilities#

Serialization Options#

Format Conversion#

Performance Characteristics#

Memory Usage#

Parsing Speed (estimated, i7 CPU)#

Notes:

• translate-toolkit: lxml-based, storage abstraction overhead
• hypomnema (lxml): Fastest (optimized dataclasses)
• hypomnema (stdlib): Slowest (pure Python XML parser)
• polib: PO format faster to parse than TMX (simpler format)

Serialization Speed (estimated)#

Concurrent Access#

Scalability Limits (16 GB RAM)#

API Design#

Type Safety#

API Style#

Extensibility Points#

Configuration Options#

Data Model#

Internal Representation#

Memory Efficiency#

Error Handling#

Exception Types#

Error Recovery#

Dependencies and Portability#

Dependency Analysis#

For TMX conversion:

• polib + translate-toolkit: Requires lxml (C extension)

Installation Complexity#

Python Version Support#

Command-Line Tools#

CLI Tool Availability#

CLI Features#

Ecosystem Integration#

Framework Support#

CAT Tool Compatibility#

Documentation and Community#

Documentation Quality#

Community Support#

Licensing#

License Comparison#

License Implications#

Edge Case Handling#

Special Cases#

Summary Scoring Matrix#

Feature Coverage (0-5 scale)#

Use Case Fit (0-5 scale)#

Technical Decision Criteria#

Based on this feature comparison, here are measurable selection criteria:

Choose translate-toolkit if:#

• Multi-format support needed (PO, XLIFF, TMX, TBX, etc.)
• Command-line automation essential
• Production stability critical (mature, widely used)
• GPL licensing acceptable
• TMX Level 1 sufficient (inline markup preserved but unstructured)

Choose hypomnema if:#

• TMX Level 2 required (structured inline markup)
• Large files (>100 MB) via streaming API
• Type safety needed (Python 3.12+ type hints)
• MIT licensing required
• Policy-driven validation/error handling needed
• Pre-1.0 API instability acceptable

Choose polib if:#

• Primary format is PO (TMX secondary)
• Git-friendly text format preferred (PO diffs)
• Zero dependencies required
• Python 2.7 or PyPy support needed
• TMX fidelity not critical (lossy conversion acceptable)
• Framework integration (Django/Flask) important

This comparison provides the technical foundation for informed library selection based on measurable criteria and specific project requirements.

hypomnema: Deep Technical Analysis#

Architecture Overview#

Module Structure#

hypomnema/
├── __init__.py        # Public API exports
├── models.py          # Dataclass definitions (Tmx, Tu, Tuv, etc.)
├── parser.py          # Deserialization (XML → Python objects)
├── serializer.py      # Serialization (Python objects → XML)
├── backends/          # XML parsing backend abstraction
│   ├── base.py       # Backend interface
│   ├── lxml.py       # lxml implementation
│   └── stdlib.py     # xml.etree implementation
├── policies.py        # Validation and error handling policies
├── streaming.py       # Streaming API for large files
└── types.py           # Type definitions, enums

Design philosophy: Type-safe, policy-driven, backend-agnostic

Core Abstractions#

Dataclass-based models:

from dataclasses import dataclass
from typing import List, Optional

@dataclass
class Tmx:
    """Root TMX document"""
    header: Header
    body: Body
    version: str = "1.4b"

@dataclass
class Tu:
    """Translation unit"""
    tuvs: List[Tuv]
    tuid: Optional[str] = None
    datatype: Optional[str] = None
    props: List[Prop] = field(default_factory=list)
    notes: List[Note] = field(default_factory=list)

@dataclass
class Tuv:
    """Translation unit variant (language-specific)"""
    xml_lang: str
    segments: List[Segment]
    changedate: Optional[str] = None
    creationdate: Optional[str] = None

Key design decisions:

• Immutable by default (dataclasses frozen option available)
• Full type annotations (Python 3.12+ typing features)
• No inheritance hierarchy (flat dataclass structure)
• Composition over inheritance (segments as list, not tree)

Level 2 Inline Markup Representation#

Segment types:

from typing import Union

# Segment can be string or inline element
Segment = Union[
    str,                # Plain text
    Bpt,                # Begin paired tag
    Ept,                # End paired tag
    It,                 # Isolated tag
    Ph,                 # Placeholder
    Hi,                 # Highlight
    Sub,                # Subflow
]

@dataclass
class Bpt:
    """Begin paired tag"""
    i: str              # Tag identifier (for pairing with Ept)
    x: Optional[str]    # Cross-reference to external resource
    type: Optional[str] # Tag type hint
    content: str        # Original tag representation (e.g., "<b>")

@dataclass
class Sub:
    """Subflow (recursive nesting support)"""
    datatype: str
    segments: List[Segment]  # Can contain nested Bpt, Ept, Sub, etc.

Arbitrary nesting depth:

# Example: <seg> with deeply nested inline elements
Tuv(
    xml_lang="en",
    segments=[
        "Start ",
        Bpt(i="1", content="<p>"),
        "Paragraph with ",
        Bpt(i="2", content="<b>"),
        "bold and ",
        Bpt(i="3", content="<i>"),
        "italic",
        Ept(i="3", content="</i>"),
        Ept(i="2", content="</b>"),
        Ept(i="1", content="</p>"),
    ]
)

Validation of inline pairing:

• Policy-driven (can enable/disable)
• Checks i attribute matching between Bpt and Ept
• Detects unclosed tags, mismatched nesting
• Configurable strictness (error, warning, or ignore)

Backend Architecture#

Pluggable Backend System#

Interface:

from abc import ABC, abstractmethod

class Backend(ABC):
    @abstractmethod
    def parse_tmx(self, source: Union[str, bytes, Path]) -> Element:
        """Parse XML to element tree"""
        pass

    @abstractmethod
    def serialize_tmx(self, root: Element) -> bytes:
        """Serialize element tree to XML bytes"""
        pass

    @abstractmethod
    def create_element(self, tag: str, **attrs) -> Element:
        """Create XML element"""
        pass

lxml Backend#

Implementation (backends/lxml.py):

from lxml import etree

class LxmlBackend(Backend):
    def parse_tmx(self, source):
        parser = etree.XMLParser(
            remove_blank_text=False,  # Preserve whitespace
            resolve_entities=False,   # Security: no external entities
            no_network=True,          # Security: no network access
        )
        if isinstance(source, (str, Path)):
            return etree.parse(source, parser).getroot()
        else:
            return etree.fromstring(source, parser)

    def serialize_tmx(self, root):
        return etree.tostring(
            root,
            encoding='UTF-8',
            xml_declaration=True,
            pretty_print=True,
        )

Performance characteristics:

• Parse speed: Fast (C-compiled libxml2)
• Memory usage: Moderate (DOM tree + Python objects)
• Compatibility: Requires lxml package (C extension)

Trade-offs:

• Pro: 10-100x faster than stdlib
• Pro: Better error messages, advanced XML features
• Con: Platform-specific compilation (wheels usually available)

stdlib Backend#

Implementation (backends/stdlib.py):

import xml.etree.ElementTree as ET

class StandardBackend(Backend):
    def parse_tmx(self, source):
        if isinstance(source, (str, Path)):
            tree = ET.parse(source)
            return tree.getroot()
        else:
            return ET.fromstring(source)

    def serialize_tmx(self, root):
        # Manual XML declaration (ET doesn't add by default)
        xml_bytes = ET.tostring(root, encoding='UTF-8')
        return b'<?xml version="1.0" encoding="UTF-8"?>\n' + xml_bytes

Performance characteristics:

• Parse speed: Slower (pure Python)
• Memory usage: Lower (simpler tree structure)
• Compatibility: Zero dependencies (stdlib only)

Trade-offs:

• Pro: No external dependencies
• Pro: Works everywhere Python runs
• Con: 10-100x slower than lxml
• Con: Limited XPath, no schema validation

Backend Selection#

Default: lxml (if available), fallback to stdlib

from hypomnema import load

# Automatic backend selection
tmx = load("file.tmx")  # Uses lxml if installed, else stdlib

# Explicit backend choice
from hypomnema.backends import LxmlBackend, StandardBackend

tmx = load("file.tmx", backend=LxmlBackend())  # Force lxml
tmx = load("file.tmx", backend=StandardBackend())  # Force stdlib

Policy-Driven Validation#

Deserialization Policies#

Policy class:

@dataclass
class DeserializationPolicy:
    """Controls error handling during parsing"""
    strict: bool = True                    # Fail on any violation?
    skip_invalid_units: bool = False       # Skip bad <tu> elements?
    skip_invalid_tuvs: bool = False        # Skip bad <tuv> elements?
    validate_inline_pairing: bool = True   # Check <bpt>/<ept> matching?
    allow_missing_required_attrs: bool = False  # Allow missing mandatory attributes?
    log_warnings: bool = True              # Log issues to stderr?
    max_nesting_depth: Optional[int] = None     # Limit inline nesting (None = unlimited)

Usage:

from hypomnema import load, DeserializationPolicy

# Strict mode (default) - crash on any issue
tmx = load("file.tmx")  # Raises on malformed input

# Permissive mode - skip bad units, log warnings
policy = DeserializationPolicy(
    strict=False,
    skip_invalid_units=True,
    log_warnings=True,
)
tmx = load("messy_file.tmx", policy=policy)
# Returns partial result, logs errors

# Custom mode - specific validation only
policy = DeserializationPolicy(
    strict=True,
    validate_inline_pairing=False,  # Don't check <bpt>/<ept> matching
)
tmx = load("file.tmx", policy=policy)

Validation Levels#

What is validated (in strict mode):

1. XML well-formedness: Via backend parser
2. Required elements: <tmx>, <header>, <body>
3. Required attributes: version on <tmx>, srclang on <header>
4. Language codes: BCP 47 format validation (optional)
5. Inline tag pairing: <bpt i="1"> matched with <ept i="1">
6. Nesting depth: Maximum recursion limit (if configured)
7. Dataclass field types: Type checking during construction

Example validation errors:

# Missing required attribute
<tmx>  <!-- Missing version="1.4b" -->
# Raises: ValidationError("Missing required attribute: version")

# Mismatched inline tags
<seg><bpt i="1">...</bpt><ept i="2">...</ept></seg>
# Raises: ValidationError("Mismatched inline tag pairing: 1 vs 2")

# Invalid language code (if validation enabled)
<tuv xml:lang="invalid-lang-code">
# Raises: ValidationError("Invalid BCP 47 language code")

Error Recovery Mechanisms#

Skip invalid units:

policy = DeserializationPolicy(skip_invalid_units=True)
tmx = load("file.tmx", policy=policy)

# File contains:
# <tu>Valid unit</tu>
# <tu>Missing required child</tu>  <!-- Skipped -->
# <tu>Another valid unit</tu>

# Result: tmx.body.tu contains only 2 units (invalid one skipped)

Partial parsing:

# If <header> is malformed but <body> is valid
policy = DeserializationPolicy(
    strict=False,
    allow_missing_required_attrs=True,
)
tmx = load("file.tmx", policy=policy)
# Returns Tmx with default header values, valid body units

Logging:

import logging
logging.basicConfig(level=logging.WARNING)

policy = DeserializationPolicy(log_warnings=True)
tmx = load("file.tmx", policy=policy)
# Logs to stderr:
# WARNING: Skipped invalid <tu> at line 42: missing tuid attribute
# WARNING: Invalid language code 'en-us' (should be 'en-US')

Streaming API#

Motivation#

Problem: Large TMX files (GB-scale) don’t fit in memory

Solution: Event-driven parsing (SAX-style), process one <tu> at a time

Implementation#

Streaming function:

from hypomnema import stream_translation_units

def stream_translation_units(
    source: Union[str, bytes, Path],
    backend: Optional[Backend] = None,
    policy: Optional[DeserializationPolicy] = None,
) -> Iterator[Tu]:
    """
    Yield translation units one at a time.
    Memory usage: O(1) - constant, regardless of file size.
    """
    # Internal: Uses iterparse (lxml) or iterparse (stdlib)
    for event, element in iterparse(source):
        if element.tag == 'tu':
            tu = parse_tu_element(element, policy)
            yield tu
            element.clear()  # Free memory immediately

Internal mechanics:

# lxml backend (fast)
from lxml import etree

def _stream_lxml(source):
    context = etree.iterparse(source, events=('end',), tag='tu')
    for event, elem in context:
        yield elem
        elem.clear()  # Critical: free memory
        # Also clear parent references
        while elem.getprevious() is not None:
            del elem.getparent()[0]

# stdlib backend (portable)
import xml.etree.ElementTree as ET

def _stream_stdlib(source):
    for event, elem in ET.iterparse(source, events=('end',)):
        if elem.tag == 'tu':
            yield elem
            elem.clear()

Memory Characteristics#

Constant memory usage:

• Parsing overhead: ~10-50 MB (parser buffers)
• Per-unit processing: ~1-5 KB (single Tu object)
• Total: Independent of file size

Example: Processing 10 GB TMX file

• Traditional (DOM): ~40-50 GB RAM required
• Streaming: ~50 MB RAM required

Use Cases for Streaming#

1. Filtering large TMs:

from hypomnema import stream_translation_units, dump

# Extract subset without loading entire file
filtered_tus = []
for tu in stream_translation_units("huge.tmx"):
    if "medical" in tu.source_text.lower():
        filtered_tus.append(tu)

# Create new TMX from filtered units
tmx = Tmx(header=Header(...), body=Body(tu=filtered_tus))
dump(tmx, "medical_subset.tmx")

2. Parallel corpus extraction (NLP):

# Extract source/target pairs for MT training
with open("source.txt", "w") as src, open("target.txt", "w") as tgt:
    for tu in stream_translation_units("training_data.tmx"):
        if len(tu.tuvs) >= 2:
            src.write(tu.tuvs[0].text + "\n")
            tgt.write(tu.tuvs[1].text + "\n")
# Memory usage: Constant (only one TU in RAM at a time)

3. Statistics and analysis:

from collections import Counter

# Analyze language pair distribution
lang_pairs = Counter()
for tu in stream_translation_units("memory.tmx"):
    if len(tu.tuvs) >= 2:
        pair = (tu.tuvs[0].xml_lang, tu.tuvs[1].xml_lang)
        lang_pairs[pair] += 1

print(lang_pairs.most_common(10))
# No need to load entire file into memory

Streaming Limitations#

Cannot:

• Random access to units (sequential only)
• Modify file in-place (read-only)
• Build index of units (would require full pass)

Must:

• Process units in document order
• Either discard or accumulate in memory (if needed later)

Data Structures and Type System#

Dataclass Design#

Advantages:

from dataclasses import dataclass, field
from typing import List

@dataclass
class Tu:
    tuvs: List[Tuv] = field(default_factory=list)
    tuid: Optional[str] = None
    # ... other fields

# Automatic:
# - __init__()
# - __repr__()
# - __eq__()
# - __hash__() (if frozen=True)

Benefits:

• Boilerplate reduction (no manual __init__)
• Immutability option (frozen=True)
• Type checking via mypy/pyright
• IDE autocomplete for all fields

Type Annotations#

Complete type coverage:

from typing import List, Optional, Union

def load(
    source: Union[str, bytes, Path],
    backend: Optional[Backend] = None,
    policy: Optional[DeserializationPolicy] = None,
) -> Tmx:
    """Load TMX file (fully typed)"""
    ...

def dump(
    tmx: Tmx,
    target: Union[str, Path],
    backend: Optional[Backend] = None,
) -> None:
    """Save TMX file (fully typed)"""
    ...

Static type checking:

# mypy
mypy my_script.py
# Success: no issues found

# pyright
pyright my_script.py
# 0 errors, 0 warnings

Runtime type validation (optional):

from typeguard import typechecked

@typechecked
def process_tmx(tmx: Tmx) -> int:
    return len(tmx.body.tu)

# Raises TypeError if passed non-Tmx object

Memory Layout#

Per translation unit (estimated):

Tu(
    tuvs=[
        Tuv(xml_lang="en", segments=["Hello"]),
        Tuv(xml_lang="es", segments=["Hola"]),
    ],
    tuid="123",
    props=[],
    notes=[],
)

Memory breakdown:

• Tu object: ~500 bytes (dataclass overhead)
• Tuv objects: ~400 bytes × 2 = 800 bytes
• Strings: ~100 bytes (“Hello”, “Hola”, “en”, “es”, “123”)
• Lists: ~100 bytes (props, notes, segments, tuvs)
• Total: ~1.5 KB per unit (simple case)

With inline markup:

Tuv(
    xml_lang="en",
    segments=[
        "Text with ",
        Bpt(i="1", content="<b>"),
        "bold",
        Ept(i="1", content="</b>"),
    ]
)

Memory breakdown:

• Baseline: ~1.5 KB
• Inline objects: ~200 bytes × 2 (Bpt, Ept) = 400 bytes
• Total: ~1.9 KB per unit (with inline markup)

Comparison to translate-toolkit:

• hypomnema: ~1.5-2 KB per unit
• translate-toolkit: ~2-4 KB per unit (lxml overhead)
• Advantage: Dataclasses more memory-efficient than lxml tree

Enum Types#

TMX enumerations:

from enum import Enum

class Segtype(Enum):
    """Segmentation type"""
    BLOCK = "block"
    SENTENCE = "sentence"
    PHRASE = "phrase"
    PARAGRAPH = "paragraph"

class Datatype(Enum):
    """Content datatype"""
    PLAINTEXT = "PlainText"
    HTML = "HTML"
    XML = "XML"
    # ... other types

Type-safe usage:

header = Header(
    segtype=Segtype.SENTENCE,  # Type-checked
    datatype=Datatype.PLAINTEXT,
)

# Invalid: mypy error
header = Header(segtype="invalid")  # Type error: expected Segtype

Performance Characteristics#

Parsing Performance#

Benchmark setup (estimated, i7 CPU, 16 GB RAM):

• File: 100K translation units, 10 MB TMX
• Backend: lxml
• Policy: Default (strict validation)

Results:

• Parse time: ~2 seconds (lxml), ~20 seconds (stdlib)
• Memory usage: ~200 MB (lxml), ~150 MB (stdlib)
• Peak memory: ~300 MB (lxml), ~250 MB (stdlib)

Comparison to translate-toolkit:

• hypomnema (lxml): ~2 seconds, ~200 MB
• translate-toolkit: ~5 seconds, ~500 MB
• Advantage: Dataclasses more efficient than lxml-heavy storage abstraction

Streaming Performance#

Benchmark: 1M units, 100 MB TMX

• Streaming parse: ~20 seconds (lxml), ~200 seconds (stdlib)
• Memory usage: Constant ~50 MB (regardless of file size)
• Units processed per second: ~50K (lxml), ~5K (stdlib)

Comparison to in-memory:

• In-memory (1M units): ~2 GB RAM, ~20 seconds
• Streaming (1M units): ~50 MB RAM, ~20 seconds
• Trade-off: Similar parse time, 40x less memory

Serialization Performance#

Benchmark: 100K units → TMX file

• Serialize time: ~3 seconds (lxml), ~30 seconds (stdlib)
• Memory usage: ~300 MB (in-memory tree + output buffer)

Optimization: No incremental serialization

• Must construct full Tmx object before serialization
• No streaming write (future feature)

Concurrent Access#

Thread safety: NOT THREAD-SAFE (mutable dataclasses)

Safe patterns:

• Read-only: SAFE (if using frozen=True dataclasses)
• Concurrent parse (separate objects): SAFE
• Concurrent modify (shared object): UNSAFE (data races)

Workaround (immutable dataclasses):

from dataclasses import dataclass

@dataclass(frozen=True)
class ImmutableTu:
    # Fields cannot be modified after creation
    tuvs: tuple[Tuv, ...]  # Use tuple instead of list
    tuid: Optional[str]

# Thread-safe read access
# Modifications require creating new object (copy-on-write)

Scalability#

In-memory limits (16 GB RAM):

• Small: <100K units, <10 MB file
• Medium: 100K-1M units, 10-100 MB file
• Large: 1M-5M units, 100-500 MB file
• Very large: >5M units, >500 MB file (use streaming)

Streaming limits:

• No hard limit (constant memory)
• Practical: Disk I/O speed and parse time
• Very large files: 10+ GB feasible on modest hardware

API Patterns and Ergonomics#

Reading TMX#

In-memory (small files):

from hypomnema import load

tmx = load("file.tmx")  # Returns Tmx object
for tu in tmx.body.tu:
    print(f"{tu.tuvs[0].text} -> {tu.tuvs[1].text}")

Streaming (large files):

from hypomnema import stream_translation_units

for tu in stream_translation_units("large.tmx"):
    # Process one unit at a time
    if "keyword" in tu.source_text:
        print(tu)

With custom backend:

from hypomnema import load
from hypomnema.backends import StandardBackend

tmx = load("file.tmx", backend=StandardBackend())  # Force stdlib (no lxml)

Writing TMX#

Create from scratch:

from hypomnema import Tmx, Header, Body, Tu, Tuv, dump

tmx = Tmx(
    header=Header(
        creationtool="hypomnema",
        srclang="en",
        adminlang="en",
        datatype="PlainText",
    ),
    body=Body(tu=[]),
)

# Add translation units
tu = Tu(tuvs=[
    Tuv(xml_lang="en", segments=["Hello"]),
    Tuv(xml_lang="es", segments=["Hola"]),
])
tmx.body.tu.append(tu)

# Save to file
dump(tmx, "output.tmx")

Modify existing:

from hypomnema import load, dump

tmx = load("input.tmx")

# Add note to all units
for tu in tmx.body.tu:
    tu.notes.append(Note(text="Reviewed 2026-01-30"))

# Save changes
dump(tmx, "updated.tmx")

Level 2 Inline Markup#

Creating inline elements:

from hypomnema import Tuv, Bpt, Ept, Ph

tuv = Tuv(
    xml_lang="en",
    segments=[
        "Click ",
        Bpt(i="1", content="<b>"),
        "here",
        Ept(i="1", content="</b>"),
        " or press ",
        Ph(x="BUTTON_ID", content="{0}"),
    ]
)

Validating inline pairing:

from hypomnema import load, DeserializationPolicy

policy = DeserializationPolicy(validate_inline_pairing=True)
tmx = load("file.tmx", policy=policy)
# Raises if <bpt> and <ept> not properly paired

Extracting plain text (strip inline elements):

def plain_text(segments):
    return "".join(
        s if isinstance(s, str) else ""
        for s in segments
    )

text = plain_text(tuv.segments)  # "Click here or press "

Error Handling#

Strict mode (default):

from hypomnema import load
from hypomnema.exceptions import ValidationError

try:
    tmx = load("malformed.tmx")
except ValidationError as e:
    print(f"Validation failed: {e}")

Permissive mode:

from hypomnema import load, DeserializationPolicy

policy = DeserializationPolicy(strict=False, skip_invalid_units=True)
tmx = load("messy.tmx", policy=policy)
# Returns partial result (skips bad units)

Extension Points#

Custom Policies#

Subclass DeserializationPolicy:

from hypomnema.policies import DeserializationPolicy

class CustomPolicy(DeserializationPolicy):
    def on_invalid_unit(self, tu_element, error):
        # Custom error handling
        log_to_database(tu_element, error)
        if self.skip_invalid_units:
            return None  # Skip unit
        else:
            raise error  # Fail

policy = CustomPolicy(skip_invalid_units=True)
tmx = load("file.tmx", policy=policy)

Custom Backends#

Implement Backend interface:

from hypomnema.backends import Backend

class CustomBackend(Backend):
    def parse_tmx(self, source):
        # Custom XML parser (e.g., pugixml, rapidxml)
        ...

    def serialize_tmx(self, root):
        # Custom serialization
        ...

tmx = load("file.tmx", backend=CustomBackend())

Custom Validation#

Add validation to dataclasses:

from dataclasses import dataclass
from hypomnema import Tu

@dataclass
class ValidatedTu(Tu):
    def __post_init__(self):
        # Custom validation logic
        if not self.tuvs:
            raise ValueError("Tu must have at least one Tuv")
        if not self.tuid:
            raise ValueError("Tu must have tuid")
        # Inline tag pairing check
        self.validate_inline_pairing()

    def validate_inline_pairing(self):
        # Check <bpt> and <ept> matching
        ...

Dependencies#

Required#

• Python: ≥3.12 (for dataclasses, typing features)
• No mandatory dependencies (stdlib backend)

Optional#

• lxml: ≥4.6 (for LxmlBackend performance)
• typeguard: For runtime type checking
• mypy / pyright: Static type checking

Installation:

# Minimal (stdlib backend only)
pip install hypomnema

# With lxml (recommended)
pip install "hypomnema[lxml]"

# Development (type checking, testing)
pip install "hypomnema[dev]"

Edge Cases#

Namespace Handling#

Default namespace:

<tmx xmlns="http://www.lisa.org/tmx14" version="1.4b">

Handling: Transparent (stripped by backends)

Custom namespaces:

<tu xmlns:custom="http://example.com/custom">
  <custom:metadata>value</custom:metadata>
</tu>

Behavior:

• Preserved during parse (stored in element attributes)
• Serialized back to XML (roundtrip integrity)
• No typed access (must use XML API)

Encoding Edge Cases#

UTF-8 with BOM:

content = b'\xef\xbb\xbf<?xml version="1.0"?>...'
tmx = load(BytesIO(content))  # BOM automatically handled

UTF-16:

content = '<?xml version="1.0" encoding="UTF-16"?>...'.encode('utf-16')
tmx = load(BytesIO(content))  # Auto-detected from BOM

Invalid UTF-8 (strict mode):

content = b'<?xml version="1.0"?><tmx>\xff\xfe</tmx>'
tmx = load(BytesIO(content))  # Raises UnicodeDecodeError

Malformed XML Recovery#

Policy-driven recovery:

from hypomnema import load, DeserializationPolicy

# Skip malformed units, continue parsing
policy = DeserializationPolicy(skip_invalid_units=True)
tmx = load("broken.tmx", policy=policy)
# Returns partial result (valid units only)

Example: Unclosed tag

<tu>
  <tuv xml:lang="en"><seg>Text</seg></tuv>
  <!-- Missing </tu> -->
<tu>
  <tuv xml:lang="en"><seg>Valid</seg></tuv>
</tu>

Behavior:

• Strict mode: Raises XMLSyntaxError
• Permissive mode: Attempts to skip bad unit, continue

Duplicate Translation Units#

Behavior: Duplicates allowed (no deduplication)

tmx = Tmx(header=..., body=Body(tu=[
    Tu(tuvs=[...]),  # Unit 1
    Tu(tuvs=[...]),  # Unit 2 (duplicate source/target)
]))
dump(tmx, "output.tmx")  # Both units written

Manual deduplication:

def deduplicate_tmx(tmx):
    seen = set()
    unique_tus = []
    for tu in tmx.body.tu:
        key = (tu.source_text, tu.target_text)
        if key not in seen:
            unique_tus.append(tu)
            seen.add(key)
    tmx.body.tu = unique_tus
    return tmx

Deep Inline Nesting#

Arbitrary depth supported:

# Level 2: Deeply nested inline markup
segments = [
    Bpt(i="1", content="<p>"),
    Bpt(i="2", content="<b>"),
    Bpt(i="3", content="<i>"),
    Bpt(i="4", content="<u>"),
    "Text",
    Ept(i="4", content="</u>"),
    Ept(i="3", content="</i>"),
    Ept(i="2", content="</b>"),
    Ept(i="1", content="</p>"),
]

Nesting limit (configurable):

policy = DeserializationPolicy(max_nesting_depth=10)
tmx = load("file.tmx", policy=policy)
# Raises if inline nesting exceeds 10 levels

Technical Limitations#

Hard Constraints#

1. Python 3.12+: Cannot use on Python 3.11 or earlier
2. No incremental write: Must construct full Tmx before serialization
3. No in-place edit: Load → modify → save (no streaming write)
4. Pre-1.0 API: Breaking changes possible in future releases

Design Trade-offs#

Dataclass-based (pro/con):

• Pro: Type-safe, memory-efficient, Pythonic
• Con: Mutable by default (thread safety risk)

Policy-driven validation (pro/con):

• Pro: Flexible error handling, adapts to messy data
• Con: Complexity (many configuration options)

Backend abstraction (pro/con):

• Pro: Portable (stdlib) or fast (lxml)
• Con: Lowest common denominator (limited to ET features)

Comparison to translate-toolkit#

Technical Risk Assessment#

Stability Risks#

• Pre-1.0 status: HIGH (breaking API changes expected)
• Small community: MEDIUM (8 GitHub stars, limited support)
• Active development: LOW (regular commits, responsive maintainer)

Performance Risks#

• Large files (in-memory): MEDIUM (similar to translate-toolkit)
• Streaming: LOW (constant memory, good for GB-scale files)
• Concurrent access: MEDIUM (not thread-safe by default)

Compatibility Risks#

• Python 3.12+: HIGH (excludes older Python versions)
• Backend (lxml): LOW (fallback to stdlib available)
• Platform: LOW (pure Python + optional lxml)

Maintenance Risks#

• Author: MEDIUM (single maintainer, no organization)
• Commercial support: HIGH (no commercial backing)
• Dependency: LOW (lxml mature, stdlib always available)

Summary of Technical Characteristics#

This deep technical analysis provides the foundation for understanding hypomnema’s internal architecture, performance characteristics, and unique capabilities for TMX Level 2 processing.

polib: Deep Technical Analysis (TMX Conversion Context)#

Architecture Overview#

Module Structure#

polib/
└── polib.py          # Single-file module (~3000 lines)
    ├── POFile        # PO file representation
    ├── MOFile        # MO (compiled) file representation
    ├── POEntry       # Single translation entry
    ├── _POFileParser # PO parser
    └── _MOFileParser # MO parser

Design philosophy: Pure Python, single-file, zero dependencies

Core Abstractions#

Main classes:

class POFile:
    """Represents a PO (Portable Object) file"""
    def __init__(self, *args, **kwargs):
        self.metadata = {}           # Header metadata
        self.metadata_is_fuzzy = False
        self._encoding = 'utf-8'
        self.entries = []            # List of POEntry objects

class POEntry:
    """Represents a single translation entry"""
    def __init__(self, *args, **kwargs):
        self.msgid = ''              # Source text
        self.msgstr = ''             # Target text
        self.msgid_plural = ''       # Plural source (optional)
        self.msgstr_plural = {}      # Plural targets (optional)
        self.msgctxt = None          # Disambiguation context
        self.obsolete = False        # Commented-out entry
        self.encoding = 'utf-8'
        self.flags = []              # e.g., ['fuzzy', 'python-format']
        self.previous_msgid = None   # For fuzzy matching
        self.previous_msgid_plural = None
        self.previous_msgctxt = None
        self.comment = ''            # Translator comment
        self.tcomment = ''           # Automatic comment
        self.occurrences = []        # Source code locations

Key design decisions:

• List-based storage (not dict)
• Entries ordered as in file
• Metadata stored separately from entries
• Plural forms supported natively
• Context (msgctxt) for disambiguation

TMX Relationship#

Indirect TMX Support#

polib does NOT parse TMX directly. TMX support is via:

1. translate-toolkit conversion: External tool
2. PO as intermediate format: PO ↔ TMX bridge
3. Programmatic manipulation: polib for PO, then convert

Conversion Pipeline Architecture#

TMX → translate-toolkit (tmx2po) → PO → polib (manipulate) → PO → translate-toolkit (po2tmx) → TMX

Why this matters:

• TMX data model ≠ PO data model
• Conversion is lossy (some metadata lost)
• Round-trip not always perfect

PO Format Internals#

File Structure#

Example PO file:

# Translation file for MyApp
# Copyright (C) 2026
msgid ""
msgstr ""
"Project-Id-Version: MyApp 1.0\n"
"Language: es\n"
"Content-Type: text/plain; charset=UTF-8\n"

#: src/main.py:42
#, fuzzy, python-format
msgid "Hello, {name}!"
msgstr "¡Hola, {name}!"

#: src/main.py:56
msgctxt "greeting"
msgid "Hello"
msgstr "Hola"

#: src/main.py:60
msgid "One item"
msgid_plural "{count} items"
msgstr[0] "Un elemento"
msgstr[1] "{count} elementos"

Key components:

• Header entry: Empty msgid, metadata in msgstr
• Comments: # (translator), #. (extracted), #: (location), #, (flags)
• Context: msgctxt for disambiguation
• Plural forms: msgid_plural, msgstr[n]
• Flags: fuzzy, python-format, c-format, etc.

Internal Representation#

POEntry object for simple entry:

entry = POEntry(
    msgid='Hello',
    msgstr='Hola',
    occurrences=[('src/main.py', 42)],
    flags=['python-format'],
    comment='Greeting message',
)

POEntry for plural entry:

entry = POEntry(
    msgid='One item',
    msgid_plural='{count} items',
    msgstr_plural={
        0: 'Un elemento',
        1: '{count} elementos',
    },
)

Memory layout:

• POEntry object: ~500 bytes (Python object overhead)
• Strings (msgid, msgstr): string length × 2
• Lists (flags, occurrences): ~100 bytes + items
• Total: ~1 KB per entry (typical)

Metadata Handling#

Metadata storage (in header entry):

po = POFile()
po.metadata = {
    'Project-Id-Version': 'MyApp 1.0',
    'Language': 'es',
    'Content-Type': 'text/plain; charset=UTF-8',
    'Content-Transfer-Encoding': '8bit',
    'Plural-Forms': 'nplurals=2; plural=(n != 1);',
}

Serialized as:

msgid ""
msgstr ""
"Project-Id-Version: MyApp 1.0\n"
"Language: es\n"
"Content-Type: text/plain; charset=UTF-8\n"
...

TMX Conversion Mechanisms#

TMX → PO (via translate-toolkit)#

Conversion process:

tmx2po -i memory.tmx -o output.po

Internal steps (in translate-toolkit):

# Simplified from translate-toolkit tmx2po
from translate.storage import tmx, po

tmx_store = tmx.tmxfile()
tmx_store.parsefile('memory.tmx')

po_store = po.pofile()
po_store.encoding = 'UTF-8'

# Map TMX source/target to PO msgid/msgstr
for tmx_unit in tmx_store.units:
    po_unit = po_store.addsourceunit(tmx_unit.source)
    po_unit.target = tmx_unit.target
    # TMX notes → PO comments
    if tmx_unit.notes:
        po_unit.addnote(tmx_unit.notes)

po_store.save('output.po')

Data mapping:

TMX                     →  PO
<tu>                    →  POEntry
<tuv xml:lang="en">     →  msgid (source)
<tuv xml:lang="es">     →  msgstr (target)
<note>                  →  comment (translator comment)
<prop type="x-context"> →  msgctxt (if present)
tuid attribute          →  (lost - PO has no equivalent)

Lossy conversion:

• Lost from TMX: tuid, <prop> (most types), creation dates, inline markup structure
• Retained: Source/target text, notes (as comments), basic metadata
• Not preserved: Inline tags (<bpt>, <ept>) converted to plain text or escaped

PO → TMX (via translate-toolkit)#

Conversion process:

po2tmx -i translations.po -o memory.tmx

Internal steps (in translate-toolkit):

# Simplified from translate-toolkit po2tmx
from translate.storage import po, tmx

po_store = po.pofile.parsefile('translations.po')

tmx_store = tmx.tmxfile()
# Language detection from PO metadata
tmx_store.setsourcelanguage(po_store.sourceLanguage)
tmx_store.settargetlanguage(po_store.targetlanguage)

# Map PO msgid/msgstr to TMX source/target
for po_unit in po_store.units:
    if po_unit.istranslated() and not po_unit.isobsolete():
        tmx_unit = tmx_store.addsourceunit(po_unit.source)
        tmx_unit.target = po_unit.target
        # PO comments → TMX notes
        if po_unit.getnotes():
            tmx_unit.addnote(po_unit.getnotes())

tmx_store.save('memory.tmx')

Data mapping:

PO                      →  TMX
POEntry                 →  <tu>
msgid                   →  <tuv xml:lang="source">
msgstr                  →  <tuv xml:lang="target">
comment                 →  <note>
msgctxt                 →  <prop type="x-context">
flags                   →  (lost - no TMX equivalent)
occurrences             →  (lost - no TMX equivalent)

Lossy conversion:

• Lost from PO: flags, occurrences, previous_msgid, obsolete status
• Retained: Source/target text, comments (as notes), context (if used)
• Not supported: Plural forms (PO plurals don’t map cleanly to TMX)

Round-Trip Integrity#

PO → TMX → PO losses:

# Original PO
entry = POEntry(
    msgid='Hello',
    msgstr='Hola',
    flags=['fuzzy', 'python-format'],
    occurrences=[('src/main.py', 42)],
    comment='Greeting',
)

# After PO → TMX → PO
# Flags: LOST
# Occurrences: LOST
# Comment: RETAINED (as TMX <note>)

TMX → PO → TMX losses:

<!-- Original TMX -->
<tu tuid="123" datatype="plaintext">
  <prop type="x-domain">software</prop>
  <tuv xml:lang="en"><seg>Hello</seg></tuv>
  <tuv xml:lang="es"><seg>Hola</seg></tuv>
</tu>

<!-- After TMX → PO → TMX -->
<!-- tuid: LOST -->
<!-- <prop>: LOST (unless x-context) -->
<!-- Source/target: RETAINED -->

Recommendation: Don’t use PO as intermediate format if TMX fidelity is critical

polib API Patterns#

Reading PO Files#

import polib

# Load from file
po = polib.pofile('translations.po')

# Load from string
content = open('file.po').read()
po = polib.pofile(content)

# Access entries
for entry in po:
    print(f"{entry.msgid} -> {entry.msgstr}")

# Filter entries
translated = po.translated_entries()
fuzzy = po.fuzzy_entries()
untranslated = po.untranslated_entries()

Writing PO Files#

import polib

# Create new PO file
po = polib.POFile()
po.metadata = {
    'Project-Id-Version': '1.0',
    'Language': 'es',
    'Content-Type': 'text/plain; charset=UTF-8',
}

# Add entry
entry = polib.POEntry(
    msgid='Hello',
    msgstr='Hola',
    occurrences=[('main.py', 10)],
)
po.append(entry)

# Save to file
po.save('output.po')

# Save as MO (compiled)
po.save_as_mofile('output.mo')

Modifying PO Files#

import polib

po = polib.pofile('app.po')

# Update translations
for entry in po:
    if entry.fuzzy:
        # Review and unfuzzy
        entry.flags.remove('fuzzy')
        entry.msgstr = review_translation(entry.msgstr)

# Merge with another PO file
updates = polib.pofile('updates.po')
po.merge(updates)

# Save changes
po.save()

TMX Workflow with polib#

Complete workflow:

import polib
import subprocess

# Step 1: Convert TMX to PO (via translate-toolkit)
subprocess.run(['tmx2po', '-i', 'memory.tmx', '-o', 'temp.po'])

# Step 2: Load with polib
po = polib.pofile('temp.po')

# Step 3: Manipulate with polib
for entry in po:
    # Custom processing
    if needs_update(entry):
        entry.msgstr = update_translation(entry.msgstr)

# Step 4: Save modified PO
po.save('updated.po')

# Step 5: Convert back to TMX (via translate-toolkit)
subprocess.run(['po2tmx', '-i', 'updated.po', '-o', 'updated.tmx'])

PO Format Parsing Implementation#

Parser Architecture#

Internal parser (_POFileParser class):

class _POFileParser:
    def __init__(self, pofile, **kwargs):
        self.pofile = pofile
        self.transitions = {}  # State machine
        self.current_entry = None
        self.current_state = 'ST'  # Start state

    def parse(self, input):
        # Line-by-line state machine parser
        for linenum, line in enumerate(input.split('\n')):
            self.process_line(line, linenum)
        return self.pofile

State machine (simplified):

States:
- ST: Start (waiting for entry)
- TC: Translator comment (#)
- GC: Generated comment (#.)
- OC: Occurrence (#:)
- FL: Flags (#,)
- CT: Context (msgctxt)
- ID: Message ID (msgid)
- IP: ID Plural (msgid_plural)
- MP: Message Plural (msgstr[n])
- MS: Message String (msgstr)

Transitions:
- # → TC (translator comment)
- #. → GC (generated comment)
- #: → OC (occurrence)
- #, → FL (flags)
- msgctxt → CT
- msgid → ID
- msgstr → MS

Example parsing:

#: main.py:42
#, fuzzy
msgid "Hello"
msgstr "Hola"

Parser steps:

1. Line 1: #: main.py:42 → State OC, store occurrence
2. Line 2: #, fuzzy → State FL, add flag
3. Line 3: msgid "Hello" → State ID, store msgid
4. Line 4: msgstr "Hola" → State MS, store msgstr, finalize entry

Encoding Handling#

Auto-detection:

def detect_encoding(filename):
    # Check BOM
    with open(filename, 'rb') as f:
        bom = f.read(4)
        if bom[:3] == b'\xef\xbb\xbf':
            return 'utf-8-sig'
        if bom[:2] == b'\xff\xfe':
            return 'utf-16-le'
    # Parse Content-Type from metadata
    # Default: utf-8
    return 'utf-8'

Encoding issues:

• Invalid UTF-8: Falls back to Latin-1 (lenient)
• No strict mode (always attempts to parse)

Malformed PO Handling#

Error recovery:

• Syntax errors: Logged, entry skipped, parsing continues
• Missing msgstr: Creates entry with empty msgstr
• Unclosed strings: Best-effort recovery, may corrupt entry

Example:

msgid "Unclosed string
msgstr "Valid"

Behavior: Parser may corrupt next entry or crash

• No strict validation mode
• Assumes well-formed input

Performance Characteristics#

Parsing Performance#

Bottleneck: Line-by-line state machine (pure Python)

Estimated performance (i7 CPU, 16 GB RAM):

• 10K entries, 1 MB file: ~0.5 seconds
• 100K entries, 10 MB file: ~5 seconds
• 1M entries, 100 MB file: ~50 seconds

Comparison:

• polib: ~0.5 seconds (10K entries)
• translate-toolkit (lxml): ~0.3 seconds (10K entries)
• Trade-off: Slower but zero dependencies

Memory Usage#

Per entry overhead:

• POEntry object: ~500 bytes
• Strings (msgid, msgstr): text length × 2
• Lists (flags, occurrences): ~100 bytes
• Total: ~1 KB per entry

File vs memory:

• 10 MB PO file: ~20-30 MB RAM (2-3x overhead)
• Lower than translate-toolkit TMX (3-5x overhead)

Serialization Performance#

Bottleneck: String concatenation (pure Python)

Estimated:

• 10K entries: ~0.3 seconds
• 100K entries: ~3 seconds

Optimization: Uses str.join() internally (efficient)

Concurrent Access#

Thread safety: NOT THREAD-SAFE

Safe patterns:

• Read-only: SAFE (after parsing complete)
• Concurrent modifications: UNSAFE (shared mutable state)
• Multi-process: SAFE (separate POFile instances)

Scalability#

Practical limits (16 GB RAM):

• 1M+ entries feasible
• 100+ MB PO files supported
• No hard limit (list-based storage)

Comparison to TMX:

• PO files smaller than equivalent TMX (less XML overhead)
• Faster to parse (simpler format)

Extension Points#

Subclassing POFile#

import polib

class MyPOFile(polib.POFile):
    def save(self, fpath=None):
        # Custom pre-save validation
        self.validate()
        super().save(fpath)

    def validate(self):
        for entry in self:
            if not entry.msgstr and not entry.fuzzy:
                raise ValueError(f"Untranslated: {entry.msgid}")

Custom Entry Processing#

class ValidatedPOEntry(polib.POEntry):
    def __setattr__(self, name, value):
        # Custom validation on assignment
        if name == 'msgstr' and '{' in self.msgid and '{' not in value:
            raise ValueError("Placeholder missing in translation")
        super().__setattr__(name, value)

Custom Metadata Handling#

po = polib.POFile()

# Add custom metadata
po.metadata['X-Generator'] = 'MyTool 1.0'
po.metadata['X-Domain'] = 'software'

# Custom metadata preserved on save
po.save('output.po')

Dependencies#

Required#

• Python: 2.7, 3.6-3.11, PyPy
• No external dependencies: Pure Python stdlib only

Installation:

pip install polib
# No dependencies installed

For TMX Workflow#

• translate-toolkit: Required for PO ↔ TMX conversion
  • Brings lxml dependency
  • GPL-2.0+ license

Installation:

pip install polib translate-toolkit
# Now have: polib (MIT) + translate-toolkit (GPL) + lxml

Edge Cases#

Plural Forms Complexity#

PO plural handling:

msgid "One item"
msgid_plural "{count} items"
msgstr[0] "Un elemento"
msgstr[1] "{count} elementos"

No TMX equivalent:

• TMX has one source, one target per <tu>
• Plurals require multiple <tu> elements
• Conversion: Splits into separate entries or loses plural info

Context Disambiguation#

PO context:

msgctxt "button"
msgid "Save"
msgstr "Guardar"

msgctxt "menu"
msgid "Save"
msgstr "Guardar"

TMX conversion:

• Context stored in <prop type="x-context">
• Not all TMX tools recognize this property
• May lose disambiguation on import to CAT tools

Fuzzy Flag Handling#

PO fuzzy flag:

#, fuzzy
msgid "Hello"
msgstr "Hola"

TMX conversion:

• No TMX equivalent for fuzzy status
• Flag lost on PO → TMX conversion
• Translation treated as confirmed in TMX

Obsolete Entries#

PO obsolete entries (commented out):

#~ msgid "Old text"
#~ msgstr "Texto antiguo"

TMX conversion:

• Obsolete entries typically excluded from TMX export
• Manual inclusion requires custom conversion script

polib + translate-toolkit Integration Patterns#

Pattern 1: PO-Centric Workflow#

Use case: PO as source of truth, TMX for distribution

import polib
import subprocess

# Maintain translations in PO (version control friendly)
po = polib.pofile('app.po')

# Update translations
for entry in po:
    # Custom logic
    pass

po.save('app.po')

# Export to TMX for CAT tools
subprocess.run(['po2tmx', '-i', 'app.po', '-o', 'dist/memory.tmx'])

Pattern 2: TMX Import to PO Workflow#

Use case: Receive TMX from translator, merge into PO

import polib
import subprocess
import tempfile

# Convert TMX to temporary PO
with tempfile.NamedTemporaryFile(suffix='.po', delete=False) as tmp:
    subprocess.run(['tmx2po', '-i', 'from_translator.tmx', '-o', tmp.name])
    imported_po = polib.pofile(tmp.name)

# Merge into existing PO
existing_po = polib.pofile('app.po')
existing_po.merge(imported_po)
existing_po.save()

Pattern 3: Batch PO to TMX Conversion#

Use case: Combine multiple PO files into single TMX

import polib
import subprocess
import tempfile
import os

# Collect all PO files
po_files = ['locale/es.po', 'locale/fr.po', 'locale/de.po']

# Merge into single PO (temporary)
merged_po = polib.POFile()
for po_file in po_files:
    po = polib.pofile(po_file)
    merged_po.merge(po)

# Save merged PO
with tempfile.NamedTemporaryFile(suffix='.po', delete=False) as tmp:
    merged_po.save(tmp.name)
    # Convert to TMX
    subprocess.run(['po2tmx', '-i', tmp.name, '-o', 'combined.tmx'])
    os.unlink(tmp.name)

Technical Limitations#

Hard Constraints#

1. No native TMX support: Requires external tool (translate-toolkit)
2. No streaming API: Entire file loaded into memory
3. PO/POT/MO only: Cannot parse XLIFF, TBX, or other formats
4. No incremental save: Must write entire file on save

Design Trade-offs#

Pure Python (pro/con):

• Pro: Zero dependencies, works everywhere
• Pro: Easy to install, debug, understand
• Con: Slower than C-compiled parsers (lxml)

List-based storage (pro/con):

• Pro: Preserves entry order
• Pro: Simple, predictable
• Con: O(n) search, no indexing

Lenient parsing (pro/con):

• Pro: Accepts messy real-world files
• Con: May silently corrupt data on malformed input

Comparison to Direct TMX Libraries#

When polib is Relevant to TMX#

Scenarios:

1. PO-first workflow: PO files in version control, export to TMX
2. CAT tool integration: Import TMX, convert to PO for developers
3. Framework integration: Django/Flask i18n with TMX export
4. Git-friendly TM: PO diffs readable, TMX diffs not
5. Offline processing: No internet, no lxml compilation (pure Python)

Not relevant:

1. TMX-only workflow: No PO files involved
2. CAT tool to CAT tool: Direct TMX exchange
3. Level 2 inline markup: PO cannot represent structured inline tags

Technical Risk Assessment#

Stability Risks#

• Mature codebase: LOW (stable since 2010)
• Infrequent updates: LOW (feature-complete)
• Breaking changes: VERY LOW (stable API)

Performance Risks#

• Large files: LOW (efficient for PO format)
• Conversion overhead: MEDIUM (external tool dependency)
• Concurrent access: MEDIUM (not thread-safe)

Compatibility Risks#

• Python version: LOW (broad support 2.7-3.11)
• Platform: VERY LOW (pure Python, no compilation)
• TMX fidelity: HIGH (lossy conversion, metadata loss)

Maintenance Risks#

• Community: LOW (widely used, e.g., Weblate)
• Commercial support: MEDIUM (no official backing)
• Dependency: VERY LOW (zero dependencies)

Summary of Technical Characteristics#

This analysis provides the technical foundation for understanding how polib works internally, its performance characteristics, and how it fits into TMX workflows as an indirect (conversion-based) solution via the PO format bridge.

Technical Selection Criteria: TMX Libraries#

Purpose of This Document#

This document provides TECHNICAL EVALUATION CRITERIA for selecting TMX translation memory libraries, not prescriptive recommendations. It maps technical requirements to library capabilities, enabling informed decisions based on measurable characteristics.

Not included: Use case narratives (reserved for S3)

Included: Technical trade-off analysis, risk assessment, capability mapping

Technical Requirement Categories#

1. TMX Standard Compliance#

Requirement: Level of TMX 1.4b conformance needed

Evaluation criteria:

• Does application need to manipulate inline markup programmatically? → Level 2 required
• Is inline markup display-only (pass-through)? → Level 1 sufficient
• Is TMX export-only (from PO files)? → Conversion-based acceptable

Technical risk:

• Level 1 limitation: Cannot validate <bpt>/<ept> pairing, cannot reorder inline elements
• Level 2 requirement: Smaller library ecosystem, pre-1.0 stability risk
• Conversion loss: tuid, properties, flags lost in round-trip

2. File Size and Memory Constraints#

Requirement: Maximum file size and available RAM

Memory overhead calculations:

• translate-toolkit: File size × 3-5 (e.g., 100 MB → 300-500 MB RAM)
• hypomnema (in-memory): File size × 2-3 (e.g., 100 MB → 200-300 MB RAM)
• hypomnema (streaming): ~50 MB constant (file size independent)
• polib (PO): File size × 2-3 (PO files smaller than TMX)

Evaluation criteria:

• Available RAM < file size × 5? → Streaming required (hypomnema only)
• Batch processing multiple large files? → Streaming or multi-process (hypomnema)
• Embedded/resource-constrained? → polib (lowest overhead, pure Python)

Technical risk:

• In-memory parsing: Out-of-memory crash on files exceeding RAM budget
• Streaming limitation: Sequential access only, cannot modify in-place
• External chunking: Manual file splitting, coordination overhead

3. Performance Requirements#

Requirement: Parse/write speed and throughput

Evaluation criteria:

• Parse time critical (real-time)? → hypomnema (lxml backend)
• Memory critical? → polib (if PO format acceptable) or hypomnema (streaming)
• Concurrent writes needed? → Multi-process architecture (all libraries)
• Batch processing? → hypomnema (streaming) or parallel processes

Technical risk:

• lxml dependency: C-compiled, platform-specific builds (may fail on obscure platforms)
• stdlib backend: 10-100x slower, but zero dependencies
• Thread safety: No built-in locking, must use external synchronization

4. Validation and Error Handling#

Requirement: Strictness of validation, malformed file handling