Domain: Procurement & Contracts Analysis#

Status note, 2026-08-25. This explainer was written alongside the 2026-02 S1 pass, which concluded that no procurement analysis software exists. That conclusion was checked against primary sources and is wrong in three of its four parts — see 01-discovery/S2-comprehensive/. The domain description and the challenges below hold; the “why existing tools fall short” claims should be read against S2, which names the maintained tools that do the work.

What This Domain Is About#

Government procurement and contracts represent a massive, complex data ecosystem involving billions of dollars in public spending. This domain focuses on analyzing procurement data, contract relationships, and spending patterns to support transparency, oversight, and efficiency in government contracting.

Key Challenges#

1. Data Fragmentation#

• Federal spending data (USAspending.gov)
• State/local procurement systems (50+ different platforms)
• Contract documents (PDFs, databases, paper records)
• Vendor registrations (SAM.gov Unique Entity ID, legacy DUNS, state-level)
• No unified data model across jurisdictions

2. Entity Resolution Complexity#

• Vendors using multiple names, DBAs, subsidiaries
• Agencies with varying names across documents
• Subcontractor relationships often hidden
• Shell companies and related entities
• Mergers, acquisitions, name changes over time

3. Document Parsing Challenges#

• RFPs (Request for Proposals) in varied formats
• Contract awards with inconsistent structure
• Amendment documents referencing prior versions
• Multi-page tables spanning PDFs
• Unstructured text mixed with structured data

4. Network Analysis Needs#

• Prime contractor → subcontractor relationships
• Agency → vendor relationship patterns
• Grant flows: Federal → State → Local → Nonprofit
• Conflict of interest detection
• Monopolization and competition analysis

Why Existing Tools Fall Short#

General graph libraries (NetworkX, igraph) provide graph data structures but lack:

• Domain-specific entity types (Agency, Vendor, Contract)
• Procurement-specific metrics (bid concentration, award patterns)
• Built-in anomaly detection for procurement fraud signals
• Integration with government data sources

General NLP/parsing tools (pdfplumber, spaCy) handle text extraction but don’t:

• Understand procurement document structure (RFP sections, award criteria)
• Extract contract-specific entities (deliverables, milestones, pricing tables)
• Link documents across the procurement lifecycle (RFP → bid → award → contract → payment)
• Normalize across jurisdictions with different formats

General anomaly detection (scikit-learn, PyOD) provides algorithms but lacks:

• Domain knowledge of procurement red flags (bid rigging patterns, favoritism signals)
• Integration with procurement data schemas
• Interpretable results for non-technical oversight staff
• Temporal analysis of bidding patterns

Current State of Practice#

What Practitioners Do Today#

Investigative journalists:

• Manual spreadsheet analysis of spending data
• FOIA requests for contract documents
• Ad-hoc entity matching using fuzzy string matching
• Custom scripts for each investigation (not reusable)

Government auditors:

• Sampling-based contract review (can’t analyze everything)
• Manual document review for compliance
• Limited network analysis (basic contractor relationships)
• Siloed systems (federal vs state vs local)

Civic transparency organizations:

• Build custom scrapers for each jurisdiction
• Maintain vendor name dictionaries manually
• Create one-off visualizations per project
• Struggle with data updates and maintenance

Procurement officials:

• Manual competitive analysis of bids
• Spreadsheet-based vendor performance tracking
• Limited tools for detecting conflicts of interest
• No systematic pattern detection across contracts

The Gap This Domain Addresses#

There’s a missing layer of procurement-specific infrastructure between:

• Low-level tools (PDF parsers, graph libraries, ML frameworks)
• High-level applications (transparency portals, audit software)

This infrastructure should provide:

1. Standardized entity models for procurement domain

   • Contract, Vendor, Agency, Bid, Award, Payment
   • Relationships: subcontractor, prime_contractor, grantor, grantee
   • Attributes: Unique Entity ID (DUNS for historical records), NAICS codes, contract types, funding sources

2. Domain-aware parsing for procurement documents

   • RFP section identification (scope, evaluation criteria, terms)
   • Award document extraction (winner, amount, timeline)
   • Contract parsing (deliverables, milestones, pricing)
   • Amendment tracking (changes over contract lifecycle)

3. Procurement-specific analytics

   • Bid concentration metrics (HHI for vendor competition)
   • Award pattern analysis (favoritism signals)
   • Spending anomaly detection (outliers, unusual timing)
   • Network centrality (key vendors, agency dependencies)

4. Cross-jurisdiction data integration

   • Unified schemas for federal, state, local data
   • Entity resolution across data sources
   • Crosswalks between classification systems (NAICS, PSC, local codes)
   • Temporal tracking with boundary/jurisdiction changes

Example Use Cases#

Investigative Journalism#

Scenario: Reporter investigating whether a city favors certain contractors

Current approach:

# Manual spreadsheet work + custom scripts
import pandas as pd
contracts = pd.read_csv('city_contracts.csv')
# Hours of manual entity matching and analysis

With procurement infrastructure:

from procurement_analysis import ContractNetwork, AnomalyDetector

# Load and normalize data
network = ContractNetwork.from_sources([
    'city_contracts.csv',
    'sam_gov_data',
    'state_vendor_registry'
])

# Entity resolution built-in
network.resolve_entities(method='fuzzy_duns')

# Domain-specific analytics
detector = AnomalyDetector(network)
flags = detector.detect_favoritism(
    agency='city_public_works',
    timeframe='2020-2024',
    methods=['bid_concentration', 'award_timing', 'price_outliers']
)

# Interpretable results
for flag in flags.high_priority:
    print(f"{flag.vendor}: {flag.description} (confidence: {flag.score})")

Government Oversight#

Scenario: State auditor reviewing procurement compliance

Current approach:

• Sample 10% of contracts manually
• Request documents via email
• Review PDFs one by one
• Limited ability to spot systemic issues

With procurement infrastructure:

from procurement_analysis import ComplianceChecker, DocumentParser

# Parse all procurement documents
parser = DocumentParser()
contracts = parser.parse_directory('procurement_docs/',
                                   doc_types=['rfp', 'award', 'contract'])

# Automated compliance checking
checker = ComplianceChecker(rules='state_procurement_code')
violations = checker.audit(contracts,
                           checks=['competitive_bidding', 'conflict_of_interest',
                                   'minority_business_goals'])

# Generate audit report
report = checker.generate_report(
    format='pdf',
    include=['violations', 'patterns', 'recommendations']
)

Grant Management#

Scenario: Federal agency tracking grant flows to subrecipients

Current approach:

• Manual reporting from grantees
• Spreadsheet tracking of subawards
• Limited visibility into actual spending

With procurement infrastructure:

from procurement_analysis import GrantFlowMapper

# Map multi-level grant relationships
mapper = GrantFlowMapper()
mapper.load_federal_grants('HHS_grants.csv')
mapper.load_state_subawards('state_passthrough.csv')
mapper.load_local_contracts('county_spending.csv')

# Trace money flows
flow = mapper.trace_grant('HHS-12345-COVID-Relief')
print(f"Total disbursed: ${flow.total_amount}")
print(f"Levels: Federal → {len(flow.state_level)} states → "
      f"{len(flow.local_level)} counties → {len(flow.end_recipients)} vendors")

# Compliance checking
compliance = mapper.check_compliance(
    grant='HHS-12345-COVID-Relief',
    rules=['allowable_costs', 'reporting_requirements', 'conflict_of_interest']
)

Small Business Opportunities#

Scenario: Small business seeking government contract opportunities

Current approach:

• Check SAM.gov manually
• Miss opportunities in state/local systems
• Don’t know which agencies buy their services
• Hard to find teaming partners

With procurement infrastructure:

from procurement_analysis import OpportunityFinder

finder = OpportunityFinder()
finder.add_business_profile(
    naics=['541512', '541519'],  # Computer systems design
    capabilities=['cloud_migration', 'cybersecurity'],
    certifications=['8a', 'woman_owned']
)

# Find relevant opportunities across all jurisdictions
opps = finder.find_opportunities(
    sources=['federal', 'state', 'local'],
    filters={'contract_value': (50000, 500000), 'set_aside': True}
)

# Suggest teaming partners
partners = finder.suggest_partners(
    opportunity='RFP-2024-1234',
    criteria='complementary_capabilities'
)

Technical Challenges#

Entity Resolution#

• Challenge: Same vendor appears as “ABC Corp”, “ABC Corporation”, “ABC Co LLC”
• Complexity: Fuzzy matching at scale, handling subsidiaries, tracking changes over time
• Existing tools: Record linkage libraries exist but lack procurement context
• Gap: Need procurement-aware entity resolver spanning the retired DUNS and the current SAM.gov Unique Entity ID

Network Analysis at Scale#

• Challenge: Millions of contracts, vendors, agencies
• Complexity: Temporal networks (relationships change), multi-level (prime/sub), attributed (contract types)
• Existing tools: Graph databases handle scale, but lack procurement metrics
• Gap: Need efficient storage + domain-specific graph algorithms (bid concentration, relationship evolution)

Document Parsing Variability#

• Challenge: Every jurisdiction has different RFP templates
• Complexity: PDFs with varying structure, scanned documents, form fields
• Existing tools: General PDF parsers handle text extraction
• Gap: Need template matching + ML to identify sections across formats, extract structured contract data

Anomaly Detection Interpretability#

• Challenge: Statistical anomalies may have legitimate explanations
• Complexity: Domain expertise needed to distinguish true fraud signals from noise
• Existing tools: Generic anomaly detection flags outliers
• Gap: Need procurement-specific rules + explainable ML (why is this flagged?)

Related Research Domains#

From This Survey#

1.010-019: Graph & Network Analysis

• Foundation for contract network analysis
• Algorithms: centrality, community detection, path analysis
• Gaps: Procurement-specific network metrics not in general libraries

1.033: NLP Libraries

• Foundation for document parsing
• Entity extraction, text classification
• Gaps: Procurement document structure awareness

1.094: Constraint Solving

• Relevant for bid optimization, fair allocation
• Gaps: Procurement compliance constraints not in general solvers

1.101: PDF Processing

• Foundation for document parsing
• Table extraction, text extraction
• Gaps: Procurement document templates not recognized

1.310-319: Corporate Finance

• Shared infrastructure for financial analysis
• Different context: Public spending vs corporate finance
• Gaps: Fund accounting, multi-jurisdiction, transparency requirements

Beyond This Survey#

Academic Literature:

• Fraud detection in public procurement (many papers, few reusable tools)
• Network analysis of corruption (research code, not production libraries)
• NLP for contract analysis (proof-of-concepts, not maintained libraries)

Commercial Tools:

• GovWin IQ, Bloomberg Government (data platforms, not libraries)
• OpenGov Procurement (SaaS, not open source)
• Tyler Technologies (ERP systems, not analysis tools)

Government Initiatives:

• USAspending.gov (data portal, not analysis library)
• SAM.gov (vendor registry, not relationship analysis)
• DATA Act compliance (reporting format, not analysis tools)

Why This Matters#

Transparency & Accountability#

• Public has a right to understand how tax dollars are spent
• Journalists and advocates need tools to investigate
• Current barrier: Technical skills required too high

Fraud Prevention#

• Procurement fraud costs billions annually
• Manual auditing can only sample small percentage
• Systematic analysis could catch patterns early

Efficiency#

• Competitive procurement should reduce costs
• Identifying monopolization or favoritism improves outcomes
• Better tools help procurement officials make data-driven decisions

Small Business Access#

• Government contracting favors incumbents who know the system
• Small businesses struggle to find opportunities
• Better discovery tools level the playing field

Research & Policy#

• Evidence-based policy requires analyzing spending patterns
• Academic researchers reinvent infrastructure for each study
• Shared tools accelerate research and improve reproducibility

Success Criteria#

This domain succeeds if:

1. Investigative journalists can analyze procurement data without custom coding for each investigation
2. Government auditors can systematically review contracts instead of sampling
3. Civic tech builders can create transparency tools faster than starting from scratch
4. Researchers cite shared infrastructure instead of building one-off analysis pipelines
5. Small businesses can discover opportunities across all levels of government
6. Procurement officials have better tools to ensure competitive, fair contracting

Getting Started#

For Tool Builders#

If you’re building procurement analysis tools, this domain should help you:

1. Identify what already exists (don’t reinvent)
2. See what’s missing (where to contribute)
3. Understand the challenges (what makes this hard)
4. Learn from related domains (what infrastructure to reuse)

For Users#

If you analyze procurement data, this domain should help you:

1. Find existing tools that fit your use case
2. Understand limitations (why some tasks are hard)
3. Make better requests to tool builders (specific gaps to address)
4. Contribute domain knowledge (what practitioners need most)

For Researchers#

If you study procurement, this domain should help you:

1. Cite shared infrastructure (reproducibility)
2. Build on existing work (don’t start from scratch)
3. Publish reusable tools (contribute to the ecosystem)
4. Identify research gaps (where more work is needed)

Last Updated: 2026-02-05 Maintainer: research/crew/furiosa Related: docs/survey/1.300-309-structure.md

1.304 Procurement & Contracts - Discovery Synthesis#

Superseded in part, 2026-08-25. This pass concluded that no libraries exist for procurement analysis. Three of the four gaps below were checked against primary sources in S2 and are filled by named, maintained software: OCDS Cardinal implements eleven sourced corruption red flags, FollowTheMoney ships Contract and ContractAward as graph types, and the Open Contracting Data Standard is the cross-jurisdiction data model this pass reports as missing. The fourth gap, document parsing, holds. Read 01-discovery/S2-comprehensive/ for the current picture; the corrections are listed in 01-discovery/DISCOVERY_TOC.md.

Research Type: Gap Documentation#

This research piece documents identified gaps in the procurement and contracts analysis domain. Unlike library comparison research, this piece identifies missing infrastructure that should exist but does not.

Discovery Approach#

S1: Rapid Discovery - Confirmed no general-purpose libraries exist for:

1. Contract network analysis (entity relationships in procurement)
2. Bid pattern detection (anomaly detection for oversight)
3. Grant relationship mapping (multi-level money flows)
4. Procurement document parsing (RFPs, awards, contracts)

Existing tools are either:

• General-purpose (graph libraries, NLP parsers) - lack procurement domain awareness
• Commercial/proprietary (GovWin IQ, Bloomberg Government) - not open source libraries
• One-off scripts - not maintained or reusable

Key Findings#

1. Contract Network Analysis Gap#

What exists: General graph libraries (NetworkX, igraph) What’s missing: Procurement-specific entity types, metrics (bid concentration, award patterns), built-in anomaly detection for fraud signals

Why general tools fall short: Don’t understand procurement domain (vendors, agencies, contracts, subcontractors), lack integration with government data sources (USAspending.gov, SAM.gov)

Corrected 2026-08-25. The entity-types half is wrong. FollowTheMoney ships Contract, ContractAward and CallForTenders among 69 schemata, with the award relation declared as a directed edge from contract to supplier. The metrics half stands: no procurement metrics live in that model. See S2-comprehensive/followthemoney.md.

2. Bid Pattern Detection Gap#

What exists: General anomaly detection (scikit-learn, PyOD) What’s missing: Domain knowledge of procurement red flags (bid rigging patterns, favoritism signals), interpretable results for non-technical oversight staff

Current practice: Manual spreadsheet analysis, custom scripts per investigation

Corrected 2026-08-25. OCDS Cardinal (MIT, published by the Open Contracting Partnership) implements eleven named red flags, each cited to published corruption-risk literature and configurable per publisher. It requires OCDS input and, for most indicators, the bids extension. See S2-comprehensive/cardinal.md.

3. Grant Relationship Mapping Gap#

What exists: Database tools, data modeling libraries What’s missing: Multi-level tracking (Federal → State → Local → Nonprofit), compliance checking against grant requirements, money flow visualization

Pain point: Limited visibility into actual spending downstream from federal grants

4. Procurement Document Parsing Gap#

What exists: General PDF parsers (pdfplumber, Camelot, Tabula) What’s missing: Procurement document structure awareness (RFP sections, contract clauses), entity extraction for contract-specific entities, lifecycle linking (RFP → award → contract → payment)

Challenge: Every jurisdiction uses different formats, templates, and terminology

Confirmed 2026-08-25. This gap holds. OCP’s own attempt, kestrel, is archived as of 2026-07-25, and no maintained procurement-aware parser was found on PyPI or in the organizations searched. See S2-comprehensive/document-parsing.md.

Documentation Delivered#

Instead of library comparisons, this research provides:

1. Domain Explainer (DOMAIN_EXPLAINER.md)

   • Explains the procurement analysis domain
   • Why existing tools fall short
   • Current state of practice (journalists, auditors, civic tech)
   • Real-world use cases with code examples

2. Gap Specification (metadata.yaml)

   • Detailed description of each gap
   • Complexity estimates (moderate to complex)
   • Why existing tools are insufficient
   • Use cases and current pain points

3. Example Code (S3-example-code/)

   • Illustrative code showing what the API COULD look like
   • Demonstrates desired functionality
   • Shows integration patterns
   • Documents why these libraries don’t exist yet

Impact#

This research serves several purposes:

1. For tool builders: Identifies high-value gaps to fill
2. For users: Explains why their workflows are painful
3. For funders: Documents infrastructure gaps with societal impact
4. For researchers: Reference for civic tech infrastructure state

Related Research#

Superseding passes: 01-discovery/S2-comprehensive/ (twelve tools and data sources, verified against live endpoints and schema files) and 01-discovery/S3-need-driven/ (six personas).

Foundation (existing tools that would be building blocks):

• 1.010-019: Graph & Network Analysis
• 1.033: NLP Libraries
• 1.101: PDF Processing

Adjacent (related domains):

• 1.300: Public Finance Modeling
• 1.302: Budget Document Parsing

Recommendations#

For practitioners needing procurement analysis capabilities now:

1. Entity resolution: Start with RecordLinkage or dedupe libraries, add procurement context
2. Document parsing: Combine pdfplumber + spaCy, build templates for your jurisdiction
3. Network analysis: Use NetworkX, add procurement-specific metrics layer
4. Anomaly detection: Use scikit-learn outlier detection, add domain rules

Long-term: The gaps identified here represent significant library opportunities with real-world impact (fraud prevention, transparency, small business access).

Conclusion#

No single library provides end-to-end procurement analysis infrastructure. What exists instead is a toolchain organized around a shared data standard (see S2), which serves publishers of OCDS data and does not reach anyone else. For data outside that standard, current practice relies on:

• Manual analysis (slow, doesn’t scale)
• One-off scripts (not reusable, break when formats change)
• General-purpose tools + significant custom code

The gaps documented here represent missing middleware between low-level tools and high-level applications. Building these libraries would accelerate civic tech, investigative journalism, and government oversight.

S2: Comprehensive Analysis - Approach#

Research Date: 2026-08-25 (S1 originally 2026-02-05) Focus: Whether the data is in a shared schema, and what that decides

What This Pass Asks#

S1 asked “which libraries exist for procurement analysis” and answered “none.” That answer was reached by looking for a library shaped like the problem — a procurement_analysis package with a ContractNetwork class — and finding no such package on any registry. No such package exists. S1 was right about that and wrong about what follows from it.

S2 asks a different question, because this category does not sort into libraries. It sorts into a fault line that runs through the data:

Is your procurement data in a shared schema, or is it an agency dump?

On one side of that line sits a small, maintained toolchain built around the Open Contracting Data Standard: validate, merge, flatten, flag. Running eleven named corruption red flags over a national dataset is a command-line invocation, not a project. On the other side sits every dataset that was published in its own shape — including the largest one in this survey, US federal spending — and there the work is entity resolution and ETL with general-purpose tools, and none of the OCDS toolchain touches your files.

The line is not drawn by you. It is drawn by whoever published the data, before you open an editor. That is the organizing fact of this category, and it is why “what library should I use” has no answer until “who published this” is answered first.

CODE, API, or DATA#

This category mixes three kinds of thing that a feature matrix would flatten together. Every item below is labeled:

• CODE — something you install and import or invoke (ocdskit, splink)
• API — an endpoint you call, live (USASpending, TED, Find a Tender)
• DATA — a corpus you download in bulk, or a schema that governs one (the OCP Data Registry, the OCDS release schema)

A reader who confuses these will plan the wrong work. The OCDS toolchain is mostly CODE that is useless without DATA of the right shape. USASpending is an API with no CODE around it that anyone else maintains.

Method#

Primary sources, in this order:

1. The schemas themselves. The OCDS release schema was downloaded from standard.open-contracting.org and its properties counted in Python. The bids extension schema and FollowTheMoney’s Contract.yaml and ContractAward.yaml were read from their repositories as raw YAML/JSON.
2. The live endpoints. Every API named below was called. Response codes, record counts and field names quoted here came back from those calls on 2026-08-25; where an endpoint refused, the refusal is reported.
3. Registry and repository metadata — PyPI, crates.io and the GitHub API, queried directly rather than taken from S1 or from documentation.

Where something could not be verified it says so. opentender.eu sits behind a Cloudflare challenge that could not be cleared from the command line; its coverage is reported as the OCP Data Registry describes it, and labeled as such.

Structure of This Pass#

Scope Boundary#

This survey covers software and data sources for analyzing procurement. PDF table extraction is 1.101’s subject and graph algorithms are 1.010-019’s; both appear here only where the procurement context changes the answer. E-procurement systems that governments run to conduct purchasing are applications, not libraries, and are out of scope.

OCDS Cardinal (CODE)#

What it is: a command-line tool that reads compiled OCDS releases and outputs corruption red flags. Rust, MIT-licensed, published by the Open Contracting Partnership as open-contracting/cardinal-rs.

This is the tool S1 said does not exist.

The Eleven Indicators#

Version 0.0.8 ships these, each with an identifier that matches the published literature it came from:

Results are grouped under four keys — OCID, Buyer, ProcuringEntity, Tenderer — so a flag can attach to a single contracting process, or to a buyer that behaves this way across many of them. That distinction is the difference between “this tender looks odd” and “this office has a pattern.”

The Indicators Are Sourced, Not Invented#

Each indicator’s documentation cites where the rule comes from. R018 is “Based on ‘Single bidder’ in An Objective Corruption Risk Index Using Public Procurement Data.” R028 cites both Assessing the potential for detecting collusion in Swedish public procurement (Government Transparency Institute) and the European Commission’s Fraud in Public Procurement: A collection of Red Flags and Best Practices.

Each also states the mechanism it is looking for. R028’s, in the docs’ own words: “A corrupt buyer can award the pre-determined bidder by leaking competitors’ prices.” R018’s methodology is stated as a rule a reader can audit — flagged when the number of tenderers is 1 and /tender/procurementMethod is open or selective, with the competitive method list configurable in an INI settings file.

S1’s claim was that general anomaly detection “lacks domain knowledge of procurement red flags” and “interpretable results for non-technical oversight staff.” Cardinal is domain knowledge with citations, emitting named rules rather than outlier scores.

What It Needs From You#

Its documented workflow is four steps: collect OCDS data, prepare it (prepare identifies and corrects quality issues), explore it with a JSON processor to learn the publisher’s local vocabulary, then run indicators. Input is line-delimited JSON of compiled releases upgraded to 1.1.

Most of the eleven compare bids against each other, which means they need the bids extension’s bids.details[]. A publisher without it can be scored on very little. The extension boundary decides which of the eleven you get, and that is a fact about your publisher, not about the tool.

Maturity, Measured#

• Version 0.0.8, released 2025-11-22. Eight releases since 2023-02-13.
• 517 commits, 14 GitHub stars, MIT license, repository active (pushed 2026-08-13).
• Not on crates.io. The README carries a crates.io badge, but crates.io/api/v1/crates/ocdscardinal returns crate 'ocdscardinal' does not exist (checked 2026-08-25). Installation is a GitHub release binary or a build from source.
• Release-asset download counts for 0.0.8: linux-64 29, windows-64 9, linux-arm 7, macos-arm 5, macos-64 4, linux-32 2.

Twenty-nine downloads is the measure of how much of the oversight world is using this. The capability exists and is maintained by the organization that publishes the standard; the adoption does not yet.

One documentation drift worth knowing when reading its pages: Cardinal’s workflow page says the Data Registry offers “over 50 publishers” while the Registry’s own homepage says “over 100” and its search page lists 134 datasets.

Sources#

• Cardinal documentation, indicators — read 2026-08-25
• Cardinal docs source, docs/cli/indicators/R/*.md — read 2026-08-25
• GitHub API: repo, latest release and asset download counts — queried 2026-08-25
• crates.io API — queried 2026-08-25

Procurement Document Parsing (CODE — the gap that survives)#

Of S1’s four declared gaps, three are filled by named, maintained software. This one holds, and it holds for a reason the rest of the pass explains.

What Exists#

General PDF extraction, which is 1.101’s subject and is well served — pdfplumber 0.11.10 was released 2026-06-15 and is actively maintained. Table extraction, text extraction and layout analysis are solved problems that a procurement project inherits for free.

What is missing is the layer above: something that knows an RFP has a scope section, evaluation criteria and a submission deadline, and can find them across templates it has not seen.

The Closest Thing Stopped#

OCP’s kestrel — described as “Collect, classify and extract contracting data with AI/ML”, BSD-3-Clause — is archived, last pushed 2026-07-25. It is the nearest attempt by the organization with the most domain knowledge and the most sample data, and it is no longer running.

No maintained, procurement-aware document parser was found on PyPI or in the GitHub organizations checked. Reported as not found, not as absent: this pass searched open registries, and an in-house or commercial parser would not appear there.

Why the Gap Holds#

An RFP template belongs to a jurisdiction. There are more jurisdictions than there are maintainers willing to track their form changes, and a parser that covers one state’s format has done work that transfers to no other. Every incentive points at a one-off script, which is what S1 correctly observed practitioners writing.

The standard did something different with that same energy. Rather than parse the documents, OCDS moved the problem upstream: publishers emit structured data at the source, from the e-procurement system that produced the document in the first place. The 134 datasets in the Data Registry are 134 jurisdictions that no longer need parsing.

That reframes the remaining gap. Document parsing is the fallback for data that was never published structured — which, for a US state or municipal analyst, is most of it, and for a reader working with any of the 134 registry datasets, is none of it.

What a Project Here Looks Like#

Template matching for the handful of formats one jurisdiction actually uses, built on pdfplumber, maintained by whoever needs it, and rewritten when the form changes. That is the achievable scope. Treating it as a general library project is how the previous attempts ended.

The higher-leverage version of the same effort, for anyone with standing to do it, is helping the publisher emit OCDS — which moves one jurisdiction from the parsing side of the line to the toolchain side permanently.

Sources#

• GitHub API: open-contracting/kestrel archive status — queried 2026-08-25
• PyPI pdfplumber — queried 2026-08-25
• See 1.101 for the general PDF extraction comparison

Entity Resolution (CODE)#

The problem every reader hits, on both sides of the schema line: the same company appears under several names, and no shared key connects them.

Verified against PyPI 2026-08-25.

Ask the Identifier Question First#

Entity resolution is what you do when identifiers fail, and in procurement they sometimes do not. Two cases where the work collapses to a join:

• OCDS carries an Identifier definition — one of the 20 in the release schema — with scheme, id, legalName and uri. A publisher populating parties[].identifier with a national company-register scheme has handed you the key. Whether they populated it is a per-dataset question the Data Registry’s coverage notes and Pelican both answer.
• US federal records carry UEI and DUNS, both present in USASpending recipient records. Within one era the join is exact.

Reach for the libraries above when the answer is no: state and local systems with free-text vendor names, subcontractor tiers, cross-border matching, and the DUNS-to-UEI seam in longitudinal US work.

Choosing Among Them#

splink is the one to reach for at scale. It implements the Fellegi-Sunter probabilistic model against DuckDB or Spark, so linkage runs in the database rather than in memory, and it comes out of the UK Ministry of Justice’s analytical services — a public-sector team doing this on public-sector data. It is also the only one in the table with a 2026 release.

dedupe trades throughput for a lower start: it asks you to label example pairs and learns from them, which suits a journalist with one messy CSV and no training data. Last released 2024-08-15.

recordlinkage is the teaching-shaped toolkit — explicit blocking, comparison and classification stages you assemble yourself. Its last release was 2023-07-20, the oldest here.

cleanco is not a resolver. It normalizes “ABC Corporation”, “ABC Corp” and “ABC Co LLC” toward a common base before anything else runs. Run it first: it removes a large share of the near-duplicates for almost no effort.

nomenklatura only makes sense if your data is already in FollowTheMoney entities. Inside that model it is the resolution layer, and it inherits FtM’s Contract and ContractAward types.

The Part No Library Solves#

Subsidiaries, shell companies, mergers and name changes over time are ownership questions, not string-similarity questions. A resolver will not tell you that two differently-named bidders share a director. That needs beneficial-ownership data joined in from outside — a separate corpus, a separate acquisition problem, and out of scope for every package above.

Sources#

• PyPI JSON API for each package — queried 2026-08-25
• OCDS release schema Identifier definition — read 2026-08-25
• USASpending /api/v2/recipient/ response fields — observed 2026-08-25

FollowTheMoney (CODE — a graph model that already knows about contracts)#

What it is: an entity-graph data model and Python toolchain for anti-corruption data, maintained by OpenSanctions with an OCCRP-lineage copyright history (Copyright (c) 2017-2024 Journalism Development Network, Inc. / Copyright (c) 2025 OpenSanctions Datenbanken GmbH).

Contract and ContractAward Are In the Model#

The model ships 69 schema files, and three of them are this survey’s subject: Contract.yaml, ContractAward.yaml, CallForTenders. Read from the repository on 2026-08-25.

Contract — “A contract or contract lot issued by an authority” — extends Asset, with featured properties title, amount, authority, contractDate, a caption that falls back to procedureNumber, and a temporal extent anchored on contractDate. Its authority property is typed as an entity with range: LegalEntity and a reverse relation named contractAuthority (“Contracts issued”).

ContractAward — “A contract or contract lot as awarded to a supplier” — is declared as an edge:

edge:
  source: contract
  label: "awarded to"
  target: supplier
  directed: true

An award is a directed graph edge from contract to supplier, in the model, by declaration. Not something you assemble from a join.

This Refutes an S1 Claim Directly#

S1 and the DOMAIN_EXPLAINER both assert that general graph libraries “lack domain-specific entity types (Agency, Vendor, Contract)” and “procurement metrics.” The first half is wrong as stated. A maintained, MIT-licensed, actively released graph model ships Contract, ContractAward, CallForTenders, Payment, Company and Organization, with the award relation typed as an edge and the authority relation reversed for traversal.

The second half survives: FtM carries no bid-concentration or award-pattern metrics. Those live in Cardinal, on the OCDS side, and nothing bridges them.

What It Does Not Do#

FtM has no OCDS reader. No published OCDS-to-FtM mapping was found in the registries and repositories checked — reported as not found rather than absent, since a mapping could exist in an unindexed project. Writing one is a real piece of work with a real payoff: it is the join between the standard’s comparability and a graph model built for tracing ownership and control.

The surrounding ecosystem — Aleph and OpenAleph for document-plus-entity investigation, yente for matching as a service, documented graph exports for Neo4j-shaped work — is what investigative teams already run. That is an argument for meeting them in their model rather than inventing a new one.

Sources#

• Contract.yaml and ContractAward.yaml — read 2026-08-25
• GitHub API: opensanctions/followthemoney schema directory listing — queried 2026-08-25
• PyPI for followthemoney, nomenklatura, alephclient — queried 2026-08-25

Kingfisher and Pelican (CODE — infrastructure, not libraries)#

OCP’s own operational stack for collecting and assessing OCDS data at scale. All BSD-3-Clause, all in the open-contracting GitHub organization, all verified active on 2026-08-25.

What It Buys You#

Kingfisher is the answer to “the publisher’s API paginates, rate-limits, and changes shape, and I need all of it, repeatedly.” Collect handles the crawl per publisher; Process normalizes and stores; Summarize produces the SQL views that make a question like “how many awards had a single bidder, by buyer, by year” a query instead of a script.

Pelican sits alongside as the quality instrument — the thing that tells you a field is present in 4% of records before you build an analysis on it. For oversight work that distinction is the whole ball game: an indicator that returns nothing because nothing was flagged and an indicator that returns nothing because the field is empty look identical in the output.

The notebook layer is the visible surface. notebooks-ocds ships seven Colab templates, including a red flags checks template and usability checks templates that run against either a Kingfisher database, a field list, or the Data Registry directly — the last of which needs no Kingfisher deployment at all.

The Cost, Stated Plainly#

This is a database deployment, not a pip install. Process wants PostgreSQL and a message queue (yapw, OCP’s Pika wrapper, is a sibling repository). Collect is a Scrapy project you run and monitor. Star counts tell the story of the intended audience: 16, 4 and 3 respectively. This stack is built for the handful of teams that maintain national or multi-national OCDS mirrors, and it is public because OCP develops in the open, not because it is a product aimed at analysts.

For everyone else the Data Registry already ran Kingfisher for you and posted the output.

Sources#

• GitHub API repo metadata for open-contracting/kingfisher-*, pelican-*, notebooks-ocds, yapw — queried 2026-08-25
• notebooks-ocds README — read 2026-08-25
• PyPI ocdskingfishercolab — queried 2026-08-25

OCDS Kit and the Validators (CODE)#

The utility layer: get data into compiled-release shape, check it, get it into a spreadsheet. All four packages install from PyPI.

Verified against PyPI and the GitHub API on 2026-08-25.

What Each One Is For#

ocdskit is the command-line front door. Its documented command groups are Combine, Upgrade, Mapping Sheet, Packager, Schema, Normalize and Hierarchy. Two of those carry most of the traffic: compile, which turns releases into compiled releases, and upgrade, which lifts 1.0 data to 1.1. Cardinal’s workflow page names both as the prerequisite step for its own input.

ocdsmerge is the library ocdskit compile leans on, and is described by its own authors as “a library and reference implementation for merging OCDS releases.” A Rust port, ocds-merge-rs (MIT), was pushed as recently as 2026-08-21. Merging is versioned, rule-driven behavior, and the existence of a declared reference implementation is the reason two tools agree on what the latest value of a field is.

libcoveocds is the validator, and it is the license outlier of the set: AGPL-3.0-or-later where everything around it is BSD or MIT. It backs the hosted Data Review Tool at review.standard.open-contracting.org (HTTP 200, checked 2026-08-25). Its PyPI release is the oldest in the table, at 2024-10-19, while its repository was pushed 2026-07-30 — code moving without a release.

flattentool (110 GitHub stars, the most-starred package here) is what turns nested OCDS JSON into the tabular form that everyone outside the toolchain actually works in, and turns spreadsheets back into JSON. For a would-be publisher with data in Excel, this is the on-ramp.

A Correction for Anyone Reaching for Spoonbill#

spoonbill, OCP’s JSON-to-Excel converter, and spoonbill-web, its web frontend, are both archived — 2026-07-25 and 2026-07-24 respectively. So is sample-data (2026-07-27). Documentation and tutorials written before mid-2026 still point at Spoonbill. Use flattentool.

What This Layer Does Not Do#

Nothing here analyzes anything. It moves data between shapes and tells you whether the shape is legal. Every analytical claim in this survey comes from what is downstream of this layer, and every one of those tools assumes this layer has already run.

Sources#

• PyPI JSON API for each package — queried 2026-08-25
• OCDS Kit CLI docs — read 2026-08-25
• GitHub API repo metadata for open-contracting/* and OpenDataServices/flatten-tool — queried 2026-08-25

The Open Contracting Data Standard (DATA — a schema)#

What it is: a JSON schema, published by the Open Contracting Partnership, that defines a common shape for data about a public contracting process from planning through payment. It is not software. Nothing installs it. It is the thing that decides whether the software in the rest of this pass applies to your files.

Its stated purpose, from the standard’s own front page: “Governments around the world spend an estimated US$13 trillion through contracts every year. Yet, contracting information is often unavailable for public scrutiny.”

What Is In It#

The current version is 1.1.5. The release schema declares itself as https://standard.open-contracting.org/schema/1__1__5/release-schema.json against JSON Schema draft-04. Read and counted from that file on 2026-08-25:

• 13 top-level properties: ocid, id, date, tag, initiationType, parties, buyer, planning, tender, awards, contracts, relatedProcesses, language
• 20 definitions: Address, Amendment, Award, Budget, Classification, ContactPoint, Contract, Document, Identifier, Implementation, Item, Milestone, Organization, OrganizationReference, Period, Planning, RelatedProcess, Tender, Transaction, Value
• 169 declared properties across the root plus those 20 objects

ocid — the open contracting identifier — is the join key for the whole model. Every release about the same purchase carries the same ocid, which is what makes “trace this procurement from tender to final payment” a group-by rather than a research project.

Releases, Records, Compiled Releases#

The primer states the publishing model directly:

A release is JSON data that is published each time there is a change to a contracting process. […] Releases are immutable, which means they cannot be changed once published.

A record is JSON data that acts as an index of all releases for a single contracting process.

A record may carry a compiled release (“the latest value of each field”) and a versioned release (“a history of changes for each field”). Almost every analysis tool downstream expects compiled releases, one per line, in a line-delimited JSON file. Turning raw releases into that shape is a merge, and the merge has a reference implementation (ocdsmerge) because getting it right by hand is harder than it looks.

The change history is the part general spending databases usually lack. A tender whose deadline moved twice and whose value tripled leaves three immutable releases; a table of awards leaves one row.

The Second Boundary: Extensions#

The base schema is smaller than the problem, and the gap is filled by extensions. The registry at open-contracting/extension_registry lists 68 extensions, of which 6 are marked Core. Read from extensions.csv on 2026-08-25.

One of those six matters more than the rest here. Bid-level data is not in the base schema — individual submissions, their prices and their bidders live in the bids extension, which adds bids.details[] and bids.statistics[].

That single fact governs the whole oversight half of this category. A publisher can be fully OCDS-conformant, pass every validator, and still publish nothing about who else bid or what they offered. And every red flag that compares bids against each other needs exactly that. “Publishes OCDS” and “can be checked for bid rigging” are two different claims about a dataset.

Version Status#

The tag list on open-contracting/standard runs 1__0__0 through 1__1__5 and stops there. A 1.2 exists in the schemas as forward references — the bids extension marks Award.relatedBid as "deprecatedVersion": "1.2" in favor of relatedBids — but standard.open-contracting.org/1.2-dev/en/ returns 404 and no 1.2 documentation was published at the time of checking. Plan on 1.1.

What It Does Not Standardize#

• Local vocabularies. tender.procurementMethod is a codelist with a handful of values, but tender.procurementMethodDetails is free text and carries the publisher’s own procedure names. Cardinal’s own documentation demonstrates counting them with jaq as a first step, which tells you how much local knowledge survives the standardization.
• Item classification. Publishers use UNSPSC, CPV or a national system. Cross-country comparison of what was bought still needs a crosswalk.
• Coverage. Nothing in the schema requires a publisher to emit planning, contracts, or implementation data. Conformance is about shape, not completeness.

Sources#

• OCDS release schema 1.1.5 — downloaded and counted 2026-08-25
• Primer: How is OCDS data published? — read 2026-08-25
• bids extension release-schema.json — read 2026-08-25
• extension_registry extensions.csv — read 2026-08-25

The OCP Data Registry (DATA — bulk downloads)#

What it is: a catalog at data.open-contracting.org that has already crawled, merged and republished OCDS datasets from around the world, offering each as bulk JSON, Excel and CSV. In its own words, it provides data “as compiled releases with the latest value of each field” — the exact input format that Cardinal and the OCDS notebooks expect.

For most readers this replaces the entire Kingfisher stack.

What Is Actually In It#

The homepage says “over 100 publishers.” The search page enumerates 134 datasets (read 2026-08-25). The composition matters more than the headline:

By region: Europe 49, Latin America & Caribbean 43, Africa & Middle East 22, Asia 7, Eastern Europe & Central Asia 7, North America 4, Oceania 2.

By country, the top few: Mexico 18, Nigeria 13, United Kingdom 5, Honduras 4, Italy 4, Argentina 3, Canada 3.

Freshness, from the registry’s own date-range facet:

Update frequency: Unknown 52, Every 6 months 32, Daily 21, Monthly 9, Real time 7, Weekly 6, Annually 4, Every 3 months 2, Hourly 1.

Content coverage: Tender 131, Parties 130, Awards 127, Documents 107, Contracts 90, Planning 64, Amendments 28, Milestones 24.

Two readings follow. First, a majority of these datasets are historical rather than current — 17 of 134 carry data from the past month, and 52 have no known update frequency. Second, coverage thins as you move down the contracting lifecycle: nearly every dataset has tenders and awards, two-thirds have contracts, and under a fifth have milestones or amendments. Questions about what was tendered are well served. Questions about what was delivered are not.

The United States Entry#

The registry lists exactly one United States dataset: Procurement Services of the City of Portland, weekly, Apr 2015 – Mar 2026.

Not USASpending. Not FPDS. Not SAM.gov. Not any state. The largest procurement data publisher in the world does not appear in the standard’s registry, and a single US city does. Any reader whose subject is US federal contracting should take that as the survey’s central practical warning: the OCDS toolchain will not open your files.

Reading the Registry#

Registry entries carry OCP’s own assessment of source and coverage, including known quality issues, which makes them usable as a screening step before committing to a dataset. Dataset licensing is the publisher’s and varies; the registry site itself is CC BY 4.0. Per-dataset license terms were not verified in this pass and should be checked per dataset before republication.

Sources#

• OCP Data Registry home — read 2026-08-25
• OCP Data Registry search, all facets — read 2026-08-25

S2 Recommendation: The Schema Decides Before You Do#

S1 concluded that no libraries exist for procurement analysis and that the domain is a set of gaps waiting for someone to build middleware. Three of its four gaps are filled. What S2 replaces that conclusion with is a boundary.

The Category Is Two Territories, Not a Toolbox#

Almost every question in this survey resolves to “which column am I in,” and that is decided by the publisher. A journalist covering UK procurement and a journalist covering US federal procurement, asking the identical question, do different projects with non-overlapping tools.

The Four Findings That Should Change a Decision#

1. The red-flag tool exists, is MIT, and nobody uses it. OCDS Cardinal implements eleven indicators sourced from published corruption-risk literature, grouped by contracting process, buyer, procuring entity and tenderer. It is at version 0.0.8, is not on crates.io despite its own README badge, and its most-downloaded 0.0.8 binary has 29 downloads. The capability is real; the ecosystem around it is thin, and a team adopting it should expect to be an early user rather than a customer.

2. Conformance is not one bar — the bids extension is a second one. Bid-level data lives in a core extension, not the base schema. A publisher can pass every validator and still emit nothing about who else bid. Most of Cardinal’s eleven indicators need that extension. Check the extensions array in a release package before promising anyone a bid-rigging analysis.

3. Graph types for procurement already ship. FollowTheMoney declares Contract, ContractAward and CallForTenders among its 69 schemata, with the award relation typed as a directed edge from contract to supplier. S1’s “general graph libraries lack procurement entity types” is wrong as written. What is missing is the mapping between OCDS and that model, and no published one was found.

4. The largest dataset in the category is on the wrong side of the line. The OCP Data Registry’s only United States entry is the City of Portland. USASpending, FPDS and SAM.gov publish rich, open, unauthenticated data in a schema OCDS tooling cannot read — a 457-row element dictionary against OCDS’s 169 properties over 20 objects, describing different things in different shapes.

Choose By Question#

• “Is my data OCDS?” — check for a published release-package endpoint, or look the publisher up in the Data Registry. This is the first question, and answering it wrong wastes the whole project.
• “Score a national dataset for corruption risk.” → Registry download, ocdskit, Cardinal. Then check which of the eleven your publisher’s coverage actually supports.
• “US federal contracting.” → USASpending API and the FPDS ATOM feed directly, pandas, and splink for recipient resolution across the DUNS/UEI seam. None of the OCDS toolchain applies.
• “Trace ownership and control across a contract network.” → FollowTheMoney and its ecosystem, with the mapping from your source as your work.
• “Extract structure from RFP PDFs.” → pdfplumber and a template matcher you maintain. The general library does not exist and the one credible attempt is archived.

What S2 Did Not Settle#

Per-dataset license terms in the Data Registry, which vary by publisher and were not checked individually. OpenTender’s current coverage and download terms, which could not be read at source. Whether any OCDS-to-FollowTheMoney mapping exists outside the repositories searched.

TED, eForms and the European Feeds (API + DATA)#

Europe is the case where a bespoke publishing regime and OCDS coexist, with a published mapping between them. It is also where a reader can see what “publishes OCDS” looks like from the client side, in one HTTP request.

Tenders Electronic Daily#

TED is the EU’s notice publication system. Its v3 API at api.ted.europa.eu/v3/notices/search accepts a POST search with a query expression and a field list, unauthenticated — verified 2026-08-25, returning notice publication numbers with per-language XML and PDF links. A GET to the same path returns 405, so the method matters.

TED notices are structured by eForms, the EU’s notice format, documented at docs.ted.europa.eu (HTTP 200, checked 2026-08-25). eForms is a different standard from OCDS with a different origin: it is a regulatory publication format for notices, where OCDS is a data model for contracting processes.

The Bridge#

OCP maintains an OCDS profile for the EU at standard.open-contracting.org/profiles/eu/latest/en/ (HTTP 200), plus open-contracting/european-union-support (BSD-3-Clause, 10 stars, pushed 2026-08-12), described as “Support scripts for TED mapping.”

A profile is the mechanism the standard uses for exactly this: a named bundle of extensions and rules that says how a regional regime maps into the common model. It is what stops “EU procurement in OCDS” from being 27 independent translation projects.

UK Find a Tender: What a Live OCDS Feed Looks Like#

www.find-tender.service.gov.uk/api/1.0/ocdsReleasePackages returns OCDS release packages over plain HTTP with no key and no registration — verified 200 on 2026-08-25. The package declares "version": "1.1" and an extensions array that begins with the EU profile’s extension.json, followed by the amendment-rationale-classifications, budget-breakdown, contract-completion, documentation and pagination extensions.

That extensions array is the machine-readable answer to “what can I ask of this dataset.” A client reads it before touching a record and knows which fields to expect. Nothing on the bespoke side of the line offers an equivalent.

The registry lists five UK datasets — Find a Tender, Contracts Finder, Public Contracts Scotland, Sell2Wales, and OpenTender — three of them real-time.

OpenTender: The Comparability Layer#

The registry’s entry describes OpenTender, run by the Government Transparency Institute, as a platform that “collects, transforms and publishes contracting data from 35 jurisdictions: 27 European Union (EU) member states, EU Institutions, Georgia, Iceland, North Macedonia, Norway, Serbia, Switzerland, and the UK,” sourced from TED and from national portals. Registry-recorded data range Jan 2006 – Jan 2025, refreshed every 6 months.

This is the descendant of the DIGIWHIST research program, and it is the closest thing in this survey to a single cross-country procurement corpus with consistent semantics.

Not verified: opentender.eu itself sits behind a Cloudflare challenge that could not be cleared from the command line, so its download options, current coverage and license terms were not read at source. Everything above is as the OCP Data Registry describes it.

Sources#

• Live POST to api.ted.europa.eu/v3/notices/search — 2026-08-25
• Live GET to find-tender.service.gov.uk/api/1.0/ocdsReleasePackages — 2026-08-25
• OCDS profile for the EU — status checked 2026-08-25
• OCP Data Registry search — read 2026-08-25
• GitHub API: open-contracting/european-union-support — queried 2026-08-25

USASpending and FPDS (API + DATA — the other side of the line)#

The United States publishes more procurement data than anyone, through open APIs that need no key, in a schema of its own. Everything in this file was called live on 2026-08-25.

USASpending API#

api.usaspending.gov, version 2, unauthenticated. Observed:

The data dictionary endpoint returns 457 rows under four column sections: “Schema Data Label & Description”, “USA Spending Downloads”, “Database Download” and “Legacy USA Spending”. Set that against OCDS’s 169 declared properties across 20 objects and the difference in kind shows: USASpending documents a flat element dictionary mapping award fields to CSV columns and database tables, where OCDS describes a nested process model. Neither is a subset of the other, and translating between them is a mapping project.

The API’s implementation is open source. fedspendingtransparency/usaspending-api is CC0-1.0, 461 stars, pushed 2026-08-25. Public-domain source for a live government API is rare enough to plan around: the field semantics can be read out of the code rather than inferred.

The Identifier Correction#

Recipient records carry both duns and uei — one observed record shows "duns": "796528263", "uei": "JE73CDQUAPA7". The DUNS number is legacy; SAM.gov now assigns a Unique Entity ID as part of entity registration (“As part of registration, we will assign you a Unique Entity ID”, sam.gov, read 2026-08-25).

This survey’s metadata.yaml and DOMAIN_EXPLAINER both describe entity resolution in terms of “DUNS/EIN/SAM.gov integration.” Code written to that description will key on a retired identifier. Historical records carry DUNS, current registrations carry UEI, and any longitudinal vendor analysis has to carry both and know which era it is in. That is a real entity-resolution problem, and it is more specific than “vendors use multiple names.”

FPDS-NG#

The Federal Procurement Data System’s ATOM feed is public and needs no authentication. A query for LAST_MOD_DATE:[2026/08/01,2026/08/25] against fpds.gov/ezsearch/FEEDS/ATOM?FEEDNAME=PUBLIC returned HTTP 200 and a feed whose rel="last" link offsets to start=592800 — the scale of contract actions modified in a 25-day window, in a paginated XML feed.

FPDS is the transaction-level source USASpending aggregates from. Work that needs modification history rather than award summaries goes here.

SAM.gov Needs a Key, and Says So Badly#

Both api.sam.gov/entity-information/v3/entities and api.sam.gov/opportunities/v2/search returned HTTP 404 when called without an API key. Not 401, not 403 — 404. A client that treats 404 as “no such record” will silently report an empty result set rather than an auth failure. Budget for the key, and assert on the status code.

What This Means for Tooling#

None of the OCDS toolchain reads any of this. ocdskit, Cardinal, Pelican and the Kingfisher stack all take OCDS as input, and USASpending, FPDS and SAM.gov publish none. A US federal analyst either writes a mapping to OCDS — which is the work OCP’s Data Support team exists to help with, and which nobody has published for the federal government — or works in pandas against the native schema with general-purpose libraries.

Sources#

• Live calls to api.usaspending.gov v2 — 2026-08-25
• Live call to fpds.gov/ezsearch/FEEDS/ATOM — 2026-08-25
• Live calls to api.sam.gov v2/v3 without key — 2026-08-25
• sam.gov entity registration — read 2026-08-25
• GitHub API: fedspendingtransparency/usaspending-api — queried 2026-08-25

S3: Need-Driven Discovery - Approach#

Research Date: 2026-08-25 Focus: Who is on which side of the schema line, and what that costs them

What This Pass Asks#

S2 established that this category splits on one question — is your data in a shared schema or an agency dump — and that the answer is decided by the publisher rather than the analyst.

S3 asks what that does to real people. The pattern across the personas below: nobody chooses their side of the line, and the ones who suffer most are the ones who need to cross it. A researcher comparing five countries and a journalist tracing one vendor through US federal spending are doing the same intellectual work with no shared tooling, because one of them was handed OCDS and the other was not.

Two personas here are on the line rather than either side of it: the municipality deciding whether to publish in OCDS, and the vendor whose subject is opportunities that have not happened yet and therefore appear in neither column cleanly.

Method#

Each persona is a WHO with a WHY — a constraint that changes the answer, not a market segment. A persona earns its place only if flipping its constraint flips the recommendation. Every tool named was verified in S2 against a primary source; nothing new is asserted here.

Effort language is order-of-magnitude planning, derived from the integration surfaces measured in S2. It describes the shape of the work, not a quote.

Personas Covered#

1. The investigative journalist — one vendor, many agencies, no shared key
2. The oversight body — looking for bid-rigging patterns at scale
3. The vendor doing competitive analysis — the future, not the past
4. The publishing municipality — on the line, choosing which side to be on
5. The cross-country researcher — comparability is the whole requirement
6. The civic tech builder — a maintenance budget of approximately zero

Scope Boundary#

This is a category survey. No one of these readers is the reason it exists, and none of the recommendations below collapses into a single pick. A reader whose constraints span two personas will find the answers compose, because they are mostly different territories rather than competing tools.

Persona: The Civic Tech Builder#

Who: a volunteer brigade, small nonprofit or one developer building a public-facing transparency site for a city or a country.

Why this changes the answer: the maintenance budget is approximately zero, and the project outlives its author’s attention by years. Whatever breaks silently will stay broken.

The Constraint#

Every moving part is a future outage nobody is paid to fix. A scraper against a portal that changes its HTML is a liability; a nightly pipeline nobody watches is a page that goes stale without saying so.

The Answer#

Consume a published feed, do not build a collector. If the jurisdiction publishes OCDS, read the release packages — Find a Tender’s endpoint needs no key, no registration, and declares its own extensions in the payload, so a client can detect a change in what it is being given. If the jurisdiction is in the Data Registry, the bulk files are already merged and already refreshed.

Skip the Kingfisher stack. It is BSD-licensed and active, and it is a PostgreSQL deployment with a message queue built for teams that maintain national mirrors. Its star counts — 16, 4 and 3 — describe its intended population accurately. For one jurisdiction, the Registry already ran it.

Reach for flattentool rather than a schema of your own. Flattening to CSV gets OCDS into whatever the site is built on without inventing a second data model that will drift.

Effort: days for a site on a published feed. Months, recurring, for a site on a scraper.

The Trap#

Building the collector because the feed is imperfect. A feed that lags a week and keeps working outlasts a scraper that is current until the portal redesigns. The Registry’s 52 datasets with unknown update frequency are a warning about publishers, and a scraper inherits that risk plus its own.

What Would Flip It#

A jurisdiction that publishes nothing structured. Then this persona is doing document parsing with pdfplumber and a template matcher, which is the one kind of work in this survey with no library support and no upstream fix — and the point at which advocating for publication is a better use of the same hours.

Persona: The Cross-Country Researcher#

Who: an academic or policy analyst testing a hypothesis across many jurisdictions — does competition fall when deadlines shorten, does a reform change single-bidder rates.

Why this changes the answer: comparability is the requirement. A finding that holds in one country’s schema and cannot be reproduced in another’s is not a finding. This persona is the reason OCDS exists.

The Constraint#

Every hour spent normalizing is an hour of confound. And the standardization is partial: tender.procurementMethod is a codelist, but procurementMethodDetails is free text carrying each publisher’s own procedure names, and item classification splits across UNSPSC, CPV and national systems.

The Answer#

The OCP Data Registry is the corpus. 134 datasets, pre-merged into compiled releases, downloadable as JSON, Excel or CSV. Europe 49, Latin America and the Caribbean 43, Africa and the Middle East 22. No crawler to write, no per-portal ETL, one schema.

Screen on coverage before choosing the sample, not after. Tender data appears in 131 datasets, awards in 127, contracts in 90, milestones in 24. A question about delivery has a sample of at most 24 to work with, and a question about bids has a smaller one still.

Screen on freshness too. 40 datasets carry data from the past year; 124 from the past five. A panel design has to accommodate that most of this corpus is historical.

OpenTender is the alternative for Europe, described in the Registry as covering 35 jurisdictions from TED and national portals with consistent transformation — a single team’s semantics across the whole region, refreshed every six months, rather than 35 publishers’ own.

Effort: days to a working panel from the Registry. Weeks if the question needs anything the standard leaves local.

The Trap#

Treating conformance as comparability. Two datasets can both validate and mean different things by the same field, because coverage, local procedure vocabularies and classification systems all survive standardization. Cardinal’s own documentation demonstrates counting procurementMethodDetails values as a first step, which is the correct instinct: look at the vocabulary before trusting the field.

What Would Flip It#

Adding the United States to the sample. The Registry’s only US entry is the City of Portland. A comparison that needs US federal data leaves the standard behind and takes on a mapping project that nobody has published.

Persona: The Investigative Journalist#

Who: a reporter following one vendor across agencies, jurisdictions and years, on a story deadline.

Why this changes the answer: the subject is an entity, not a dataset, and entities do not respect the schema line. The vendor appears in a federal contract, a state award and a city purchase order, under three names, in three schemas.

The Constraint#

Deadline plus fragmentation. Every hour spent on ETL is an hour not spent on the story, and the vendor’s trail crosses exactly the boundary S2 says nothing crosses automatically.

The Answer#

Start from the identifier, not the tool. In US federal data the join is already free: USASpending recipient records carry both uei and duns, and the API needs no key. One /api/v2/recipient/ call and a few award lookups can establish the federal picture in an afternoon, against a corpus of over 18 million recipient records.

Then pay for the seam once. State and local data will not carry those keys. That is where cleanco plus splink earns its setup cost — normalize the legal suffixes first, then link probabilistically. dedupe is the alternative when there is one messy file and no training data.

Model the result in FollowTheMoney if the story is about relationships. Contract and ContractAward are already in the model, the award is already a directed edge, and the Aleph tooling around it is what investigative desks already run. Do not build a bespoke graph schema for a single story.

Effort: hours for the federal slice; days once state and local sources join.

The Trap#

Treating “no results” as a finding. SAM.gov’s API returns 404 rather than 401 when the key is missing. A script that reports “this vendor has no registrations” may be reporting that you forgot the key.

What Would Flip It#

If the vendor’s contracts are all in one OCDS-publishing jurisdiction, this persona becomes the cross-country researcher: download the compiled releases from the Data Registry, filter on parties[].identifier, and skip the resolution work entirely.

Persona: The Oversight Body#

Who: an audit office, inspector general or anti-corruption agency looking for bid-rigging and favoritism patterns across a whole procurement system rather than in a sampled handful of files.

Why this changes the answer: this persona needs bid-level data, which is the one thing the base OCDS schema does not carry. Their answer is decided by a single extension.

The Constraint#

Every pattern that distinguishes collusion from bad luck is a comparison between bids: identical prices, a lowest bid disqualified, a late bid winning, excessive unsuccessful bids. Award-only data cannot express any of them. It can express single-bidder rates and that is close to all.

The Answer#

If your data carries the bids extension: Cardinal, today. Eleven named indicators, each traceable to published corruption-risk literature, grouped by contracting process, buyer, procuring entity and tenderer. Results attach to repeat behavior by an office, which is what an audit plan can act on. MIT license, single binary, settings in an INI file.

Check coverage before promising anything. Run Pelican, or read the Data Registry’s coverage notes, and establish what share of records carry bids.details[]. An indicator returning zero because the field is empty looks identical in the output to one returning zero because nothing was flagged.

If your data does not carry it, the achievable scope is a much smaller analysis, plus a conversation with the publisher. Adding bid data to what a system already emits is a publishing change, and it is the highest-leverage thing an oversight body can ask for.

Effort: a day to first results on registry data. Weeks to tune away false positives — Cardinal’s own workflow documents that loop and expects it.

The Trap#

Reading indicator output as findings. Every one of the eleven is a red flag, not a conclusion, and the tool’s documentation says so by giving each one a mechanism rather than a verdict. An office that publishes flag counts as misconduct counts will burn its credibility on the first legitimate explanation.

What Would Flip It#

Non-OCDS data. A US state auditor working from a native ERP export gets nothing from Cardinal and has to implement the same eleven rules against their own schema — reading Cardinal’s per-indicator methodology as a specification rather than running the software.

Persona: The Publishing Municipality#

Who: a city, county or agency that has procurement data in its own systems and has decided to publish it — or is deciding whether to.

Why this changes the answer: this persona is not analyzing anything. Every other reader in this survey is downstream of a choice this one makes, and their tooling is validation and mapping rather than analysis.

The Constraint#

The data exists, in an e-procurement system or an ERP, in a shape nobody else uses. The work is a mapping, and the mapping has to survive staff turnover.

The Answer#

flattentool first, if the source is tabular. It converts between spreadsheets and structured JSON in both directions, MIT-licensed, and it is what turns an existing extract into a starting point rather than a rewrite. It replaced spoonbill, which OCP archived on 2026-07-25 — a correction that matters here, since older publishing tutorials still recommend it.

libcoveocds, or the hosted Data Review Tool, as the gate. Validation is the feedback loop that makes the mapping converge. The hosted version at review.standard.open-contracting.org needs no installation. The library is AGPL-3.0-or-later, which is the one license in the OCDS toolchain that a procurement office’s legal review should look at before it is embedded in anything.

Decide about bids early. The base schema carries no bid-level data. Publishing awards without submissions is conformant and is what most publishers do, and it is also the decision that determines whether anyone can ever run corruption indicators on your city’s data. Adding the extension later means revisiting the mapping.

Effort: the mapping is weeks. Keeping it running is the commitment that lasts, and the Registry’s 52 datasets with unknown update frequency are what happens when that commitment lapses.

The Trap#

Publishing once. A dataset that stops updating still appears in catalogs, still gets cited, and quietly becomes a historical archive that people read as current. Of 134 Registry datasets, 17 carry data from the past month.

What Would Flip It#

A national mandate. In the EU, notices go to TED under eForms regardless, and the OCDS question becomes whether to also publish through the EU profile rather than whether to publish at all.

Persona: The Vendor Doing Competitive Analysis#

Who: a company deciding which agencies to pursue, what to bid, and who they will be bidding against.

Why this changes the answer: this is the only persona whose primary subject is the future. Historical award data answers “who has been winning”; it does not list what is open now. The two live in different systems with different access rules.

The Constraint#

Forward-looking opportunity data is the part that is gated. Historical award data is open almost everywhere; SAM.gov’s opportunities API needs a key, and state and local opportunities are scattered across portals with no common feed.

The Answer#

Split the question in two and use different tools for each half.

Where has the money gone, and to whom — USASpending, unauthenticated. Award records carry piid, type, total_obligation, date_signed and base_and_all_options, and the agency list resolves 111 top-tier agencies. This half is a weekend of pandas.

What is open now — SAM.gov’s opportunities API, with a key. Register for one first; without it the endpoint returns 404 rather than an auth error, which reads as an empty result set.

In an OCDS jurisdiction, both halves come from one feed. UK Find a Tender serves OCDS release packages over plain HTTP with no key, covering tender through award in one schema with one ocid per process. A vendor operating there can watch the pipeline and the history with the same client.

Effort: days per jurisdiction, and it does not amortize across them.

The Trap#

Assuming award history predicts competition. Bid-level data — who else showed up and at what price — is the extension most publishers omit. In its absence, “three vendors have won everything here” and “three vendors are the only ones who ever bid” are indistinguishable, and they imply opposite strategies.

What Would Flip It#

A set-aside or certification requirement. Those attach to entity registration rather than to award records, which pushes this persona into SAM.gov’s entity API and its key, and away from spending analysis entirely.

S3 Recommendation: Find Your Side of the Line First#

Across six personas the tool choice is downstream of one fact none of them control: whether the data they need was published in a shared schema.

The Table#

The Two Rules That Cover Most Cases#

If the data is OCDS, use the toolchain and stop building. Compile with ocdskit, validate with libcoveocds, flatten with flattentool, flag with Cardinal. Four maintained tools, three of them permissively licensed, covering the path from raw releases to named corruption indicators. Anyone writing that pipeline from scratch is rebuilding work the standard’s own maintainers already did.

If it is not, the standard’s tooling gives you nothing, and the work is entity resolution. Not network analysis, not anomaly detection — those come later and are the easy part. The work is establishing that these two rows are the same company, and it is the hour count that decides whether the project finishes.

The Question to Ask First#

Who published this, and in what schema?

Look the publisher up in the OCP Data Registry, or fetch one record and look for an ocid. The answer determines the toolchain, the effort, and whether half this survey applies to you at all. Asking it in week one costs a browser tab. Discovering it in month two costs the pipeline.

The Asymmetry Worth Naming#

The three personas already inside the standard — the researcher, the oversight body on conformant data, the builder on a published feed — get a maintained, free, mostly permissive toolchain and finish in days. The three outside it pay in weeks and in bespoke code, and one of them is working with the largest and most open procurement dataset in the world.

Open data and standardized data are different goods. US federal procurement data is extraordinarily open — no key, public-domain API source, 18 million recipient records, a public ATOM feed of contract actions — and it is not comparable to anything. That gap is where most of the remaining effort in this category goes.

What This Pass Does Not Do#

It does not pick for any particular project. Each persona above is a constraint with an answer attached, and the answers compose rather than compete: a journalist working in an OCDS jurisdiction should read the researcher’s entry, and an oversight body outside one should read Cardinal’s indicator methodology as a specification rather than as software.

S4: Strategic Selection - Approach#

Research Date: 2026-08-25 Focus: What a choice in this category costs to reverse in three years

What This Pass Asks#

S2 mapped the tools and the data sources. S3 asked which constraint decides for whom. Both were about today. S4 asks the question that only shows up later:

If this turns out to be the wrong bet, what does it cost to get out?

That question has an unusual shape here, because the largest commitment in this category is not one you make. It is made by the organization that publishes your data, in the year before you arrive, and you inherit it. A survey that scored tools on their own switching costs would miss the thing that actually locks people in.

What S4 Scores#

Lock-in, by side of the schema line. A pipeline built on the Open Contracting Data Standard is portable across every publisher that emits it. A pipeline built on one agency’s export is portable across nothing. That asymmetry is the category’s central strategic fact and it is scored in lock-in.md.

Ecosystem durability. The OCDS toolchain is small, permissively licensed, and concentrated in one organization and, in several repositories, one person. Four repositories were archived in July 2026. ecosystem-durability.md measures that concentration from contributor data and from the maintainer’s own published finances, and reads it as a risk signal.

The United States as its own case. The OCP Data Registry’s only US entry is a single city. For anyone building US procurement analysis this is not a detail about coverage; it determines the whole shape of the project. the-us-problem.md treats it separately.

Exit. What leaving each position costs, and which insurance is cheap enough to buy on day one.

Method#

Contributor counts, archive dates and release histories came from the GitHub API on 2026-08-25. Funding figures came from the Open Contracting Partnership’s own published finances page, read the same day. Nothing here re-derives S2; the tool descriptions, versions and licenses live there and are cited rather than repeated.

Where a figure could not be collected, it says so. GitHub’s unauthenticated rate limit blocked several contributor queries on the first attempt; the ones reported below were retried until they returned, and any that did not are marked unverified.

How to Read the Funding Section#

Money appears in ecosystem-durability.md for one reason: to answer “will the maintenance survive?” It does not appear to rank anyone. A grant with an end date tells a reader what happens to them when it ends. That is a fact about the software’s future; whether the organization is admirable is not a question this survey asks or answers.

Ecosystem Durability: One Organization, and Often One Person#

The OCDS toolchain works, is permissively licensed, and is maintained. It is also thin in a specific, measurable way, and a team adopting it should know the shape of that thinness before it matters rather than after.

None of what follows is a reason to avoid these tools. Adoption in this survey has no better option and several worse ones. It is a reason to plan for the maintenance, not to assume it.

Maintainer Concentration, Measured#

Contributor counts from the GitHub API, 2026-08-25. Share is the top contributor’s commits as a fraction of all commits in the repository.

The same GitHub account, jpmckinney, is the top contributor to every open-contracting repository in that table. Cardinal’s own documentation names the person and the role: “Have questions, concerns, or feedback? Email James McKinney, OCP’s Head of Technology.”

Read that as a bus factor with a name attached. The standard itself is the healthiest line in the table — 35 contributors, top share 38% — which is the right way round: the specification is more distributed than the software that reads it. Cardinal is the least distributed, at 89% and two human contributors, and Cardinal is the tool doing the most interesting work in this category.

The one line not in the open-contracting organization, flatten-tool, is maintained by Open Data Services with a different top contributor. That is the only real diversification in the toolchain.

The July 2026 Archive Wave#

Four repositories were archived within four days:

Twenty-two of the seventy repositories listed in the organization are archived.

Two readings, and both are correct. The pruning is a sign of an organization managing its surface area rather than one collapsing — flattentool covers what spoonbill did, and consolidating onto it is a reasonable call. And it is also the mechanism by which a documented tool becomes a dead link: material written before mid-2026 still recommends spoonbill, and kestrel was the only credible attempt at the document-parsing gap S2 found still open.

Cardinal’s Install Path Is Broken, and That Is the Signal#

Cardinal’s README carries a crates.io badge. crates.io/api/v1/crates/ocdscardinal returns crate 'ocdscardinal' does not exist (checked 2026-08-25). Installation is a GitHub release binary or a build from source.

A broken badge is small. What it indicates is the absence of the feedback loop that would have caught it: enough users installing by the documented path that someone reports the failure. Cardinal’s most-downloaded 0.0.8 binary has 29 downloads. That number and that badge are the same fact seen twice.

Plan on being an early user: read the source when behavior surprises you, expect to open issues rather than find them already answered, and pin the binary you validated against.

Who Pays for the Maintenance#

The Open Contracting Partnership publishes its funding in full — funder, amount, duration, and whether the grant is restricted or unrestricted — on its own finances page, and describes itself in its site footer as “an independent non-profit public charity 501(c)(3).”

That disclosure is the useful part, because it makes the risk assessable rather than guessable. Read from that page on 2026-08-25, the largest listed grants and their end dates:

What this tells a reader planning three years out: the toolchain’s maintenance capacity is grant-funded, most of the grants are restricted to particular programs rather than to general engineering, and several of the largest carry end dates inside the planning horizon. Grants are renewed routinely and there is no prediction here that these will not be. The exposure is that maintenance attention in this ecosystem tracks program funding, and program funding has visible expiry dates — where a commercial vendor’s does not, and also cannot be read at all.

The licenses are what convert that exposure into something survivable. MIT and BSD across the toolchain mean a fork is available to anyone who needs one, and the specification — the part with 35 contributors — is the asset that would outlive any particular implementation of it.

Sources#

• GitHub API contributor lists, repository archive states — queried 2026-08-25
• Cardinal indicators documentation — read 2026-08-25
• OCP finances — read 2026-08-25
• crates.io API for ocdscardinal — queried 2026-08-25

Exit Strategy: What Leaving Each Position Costs#

Five positions appear across S2 and S3. Each has a different exit, and the differences are larger than the differences in getting in.

Leaving the OCDS Toolchain#

Cost: low. Days.

Every piece is small and single-purpose, and the data you hold is JSON that conforms to a published schema. Dropping ocdskit means writing the merge yourself against documented rules with a reference implementation to check against. Dropping flattentool means writing a flattener. Dropping libcoveocds means writing your own checks or using the hosted tool.

The reason the exit is cheap is that none of these tools own your data. They transform it and hand it back.

Leaving Cardinal#

Cost: low, and lower than it looks.

Cardinal’s real product is eleven documented rules with citations, and those are portable whether or not the binary is. If the project stalls, the methodology pages remain readable and implementable; if it does not stall, you have saved yourself implementing them.

Pin the binary version you validated against and record which indicators your publisher’s coverage actually supported. That record is the thing that makes a future reimplementation faithful rather than approximate.

Leaving a Bespoke ETL Pipeline#

Cost: high, and mostly invisible until you try.

The pipeline encodes knowledge that lives nowhere else: which columns mean what, which values are sentinels, which agency codes changed, and which records were excluded and why. None of that is in the output, and none of it is in the source.

The exit cost is dominated by the entity-resolution state described in lock-in.md. Persisting the mapping — source ids, decision, date, method — rather than the merged result converts this from a rebuild into a re-run.

Leaving a Publisher#

Cost: depends entirely on the column you are in.

Inside OCDS, changing publishers is changing a download URL. Outside it, changing publishers is a new project. The Registry lists 52 datasets with no known update frequency and 17 with data from the past month, so “the publisher stopped” is a routine event rather than a hypothetical, and it is the event that makes this dimension matter.

Leaving FollowTheMoney#

Cost: moderate.

FtM is a model rather than a store, and its entities serialize to documented JSON, so the data comes out. What does not come out cleanly is anything that depended on the model’s own semantics — the ContractAward edge, the reverse relations — which have to be re-expressed in whatever replaces it. That is schema work, not data recovery.

The Insurance Worth Buying On Day One#

Three things, all cheap while the project is small and expensive later:

1. Keep the raw source files. Everything downstream is reproducible from them and nothing is reproducible without them.
2. Persist entity decisions as a mapping table, not as merged records.
3. Write down which extensions and which fields your analysis depended on. In the OCDS case that is the extensions array from the release package, which the publisher hands you for free; in the bespoke case nobody hands it to you and nobody will remember.

Lock-In: You Inherit It, You Do Not Choose It#

The organizing statement from S2 — which side of the schema line are you on, and you do not choose — is already a lock-in statement. A publisher’s format decision, made before you arrive, determines your toolchain, your effort, and what your work is worth to anyone else. This page scores what that costs.

The Asymmetry, Stated As Portability#

Portability here means: how many other datasets does this pipeline run against, unchanged?

That is the whole finding. It is not a difference of degree. An analyst who writes a single-bidder-rate calculation against OCDS has written it once for every publisher in the standard; an analyst who writes the same calculation against a state ERP extract has written it once, for one state, until the ERP is upgraded.

Five Dimensions, Scored#

1. Data format — the dominant one, and the one you do not control. Moving a dataset from bespoke to OCDS is a publishing change inside someone else’s organization. No amount of engineering on your side changes which column you are in.

2. Tooling — low, on both sides. The OCDS tools are BSD or MIT, small, and each does one thing: ocdskit reshapes, flattentool flattens, Cardinal scores. Replacing any one of them is days, not months. The one license that constrains embedding is libcoveocds at AGPL-3.0-or-later, and the hosted Data Review Tool exists so most readers never need to embed it.

3. Analysis code — moderate, and asymmetric. Analysis written against OCDS field paths survives a change of publisher. Analysis written against transaction_fpds column names survives nothing.

4. Entity keys — high, and quiet. A pipeline that resolved entities by fuzzy name matching has an internal identifier space that nothing else shares. Rebuilding it against a different source means re-resolving from scratch, and the results will not be identical. This is the lock-in people discover late, because it does not look like lock-in until the second dataset arrives.

5. Extension dependence — narrow but sharp. An oversight pipeline built on bids.details[] runs against the subset of publishers who emit that extension. Moving to a publisher without it does not degrade the analysis; it removes it.

The Trap in Being On the Good Side#

An OCDS pipeline is portable across publishers and no further. The standard is the boundary of its portability in the other direction too: code written around ocid, releases and compiled releases does not transfer to USASpending, FPDS or a state ERP without the same rewrite everyone else faces. Portability across 134 datasets is a large number and a closed set.

The Cheap Insurance#

Keep the raw source and the derived model separate, and keep the raw. Both columns benefit, for the same reason: whatever you built on top can be rebuilt if the inputs are still on disk, and cannot be if they are not. Publishers withdraw datasets, portals redesign, and the Registry lists 52 datasets with no known update frequency.

For entity resolution specifically, persist the mapping, not the merged output. A table of “these source ids are the same company, decided on this date, by this method” survives a change of source, a change of resolver, and a disagreement about a match. A merged file does not.

S4 Recommendation: The Bet You Cannot Place#

Across three passes the same fact keeps returning in different clothes. S2 found it as a fault line, S3 as a constraint, and S4 as the thing that decides reversibility: the largest commitment in this category is made by your data’s publisher, and you inherit it.

What That Means for a Three-Year Plan#

Inside the standard, the strategic position is strong and the ecosystem is thin. A pipeline built on OCDS runs against 134 datasets unchanged, uses four small permissively-licensed tools that each cost days to replace, and inherits eleven documented corruption indicators. The exposure is not technical: it is that one organization maintains almost all of this, one account is the top contributor to every repository in it, and Cardinal — the most interesting piece — sits at 89% single-contributor share with a broken install badge and 29 downloads on its most popular binary. MIT and BSD licenses make that survivable. Plan on being an early user rather than a customer.

Outside the standard, the position is weak and no tool fixes it. The pipeline is portable across nothing, the domain knowledge has to be reimplemented rather than invoked, and the entity-resolution state accumulates quietly until it becomes the exit cost. Nothing in this category rescues that, because the thing that would rescue it is a decision inside somebody else’s organization.

The Three Findings That Should Change a Three-Year Decision#

1. Portability is binary here, not gradual. 134 datasets, or one. There is no middle position to engineer toward, and no amount of care on the analysis side moves you between them.

2. The standard is more durable than the software that reads it. The specification repository has 35 contributors and a 38% top share; the tools reading it run to 83-89%. If any part of this ecosystem outlives a funding change, it is the schema — which is also the part your analysis code should be written against.

3. Domain knowledge transfers even when software does not. Cardinal’s eleven indicator pages are auditable methodology with citations. A reader locked out of the toolchain by their publisher’s format can still take the rules. That is the single most portable asset in this survey and it costs nothing to copy.

Choose By Horizon#

• One jurisdiction, one question, this year. Use whatever is native. Do not build a mapping to a standard you will never compare against.
• Several jurisdictions, or a repeatable product. Get inside OCDS if your publishers are there, and treat the Data Registry as the corpus. If they are not, decide early whether the mapping is the project, because retrofitting it is worse than choosing it.
• US federal or US state work. Read the-us-problem.md first. The toolchain does not apply, ETL is the deliverable, and Cardinal’s methodology pages are the cheap way to import the domain knowledge without the schema.
• Anything where the answer must still be defensible in three years. Persist raw sources and entity-decision mappings from day one, and record which extensions your conclusions depended on.

What S4 Did Not Settle#

Whether OCP’s grant renewals will sustain current maintenance levels — the funding is disclosed with end dates, and the future is not a fact. Whether flatten-tool’s separate maintainership meaningfully diversifies the toolchain or is one repository’s accident. Whether any US federal OCDS mapping exists outside the repositories searched, which remains not found rather than demonstrated absent.

The United States Is Its Own Category#

The OCP Data Registry lists 134 datasets. Its only United States entry is Procurement Services of the City of Portland.

For a reader anywhere else in the world that is a coverage note. For a reader building US procurement analysis it decides the entire project, and it deserves to be said before any tool is named.

What Follows From It#

The OCDS toolchain does not apply. Not partially — at all. ocdskit compiles releases that do not exist. libcoveocds validates a schema nothing here emits. Cardinal reads compiled releases and there are none to read. Every strategic advantage catalogued in lock-in.md — portability across 134 datasets, published indicators, a shared field vocabulary — belongs to somebody else’s jurisdiction.

ETL is the project, not a phase of it. USASpending, FPDS-NG and SAM.gov are open, documented and unauthenticated in the places that matter, and each has its own shape. The work of getting them into one queryable model, and keeping them there as they change, is the deliverable. Analysis is what happens afterward, on a smaller budget than you planned.

Entity resolution has a dated seam in it. Federal records carry both a legacy DUNS and the current SAM.gov Unique Entity ID. Any analysis spanning the transition holds two identifier spaces and has to know which era each record belongs to. This is the specific version of “vendors have many names” that US work actually hits, and it is more tractable than the general problem and easier to get quietly wrong.

State and local is a different problem again. Fifty-plus procurement systems, no shared schema, and no registry that has already crawled them. This is where the document-parsing gap S2 left open does its damage, because it is where data was never published structured in the first place.

Two Strategic Responses, and What Each Costs#

Build the mapping. Write US federal spending data into OCDS and the whole toolchain switches on: validation, the indicator suite, comparability with 134 other datasets. Nobody has published such a mapping — searched and not found, which is not proof it does not exist somewhere unindexed. The cost is a serious schema-mapping project against a 457-row element dictionary, and the ongoing cost is tracking two moving targets rather than one.

Do not build the mapping. Work natively, in pandas, against the richest and most open procurement data in the world, and accept that the result compares to nothing and that every analytical rule has to be implemented rather than invoked. This is what almost everyone does, and it is a defensible choice, not a failure.

The second is right for a single-jurisdiction question. The first is the only route to a US answer that sits alongside anyone else’s, and it is the unbuilt thing in this survey with the most leverage behind it.

The Cheap Middle Path#

Cardinal’s per-indicator documentation states each rule as an auditable methodology — R018 flags a process where the number of tenderers is 1 and /tender/procurementMethod is open or selective — with a citation to the literature it came from. Read those eleven pages as a specification and implement the rules against native US fields. That transfers the domain knowledge, which is the scarce part, without the schema mapping, which is the expensive part.

It gets you the indicators and none of the portability. For a US oversight office with one dataset and no plans to compare internationally, that trade is usually the right one.

The Fact Behind the Fact#

US federal procurement data is extraordinarily open: no API key, a public-domain implementation, over 18.2 million recipient records, a public feed of contract actions. It is also comparable to nothing. Open and standardized are different goods, and this is the clearest case of the two coming apart anywhere in the survey. A jurisdiction can lead the world on one and be absent from the other, and readers who treat “open data” as a single achievement will plan the wrong project.