For researchers, journalists and analysts

Palimpsest data and methods

Palimpsest is an open, public-good observatory that measures how authoritarian states, and their state-aligned AI models, censor. It treats deletion as data and watches the censor, never the censored. Everything below is free to use and built to be cited. If a number is here, its raw evidence is here too.

Open the observatory Browse all data Source code

Live signals

Each signal recomputes on a fixed schedule and publishes a machine readable file plus a committed time series. Signals that would need a network vantage inside China are held back on purpose until that measurement can be verified, rather than published as a guess.

Signal	What it measures	Source	Updated	Status
DDTI deletion-differential threat index	Which topics the censor is most actively scrubbing, ranked by attention and novelty.	China Digital Times	every 3 hours	LIVE
Generative Firewall state-AI refusal index	The share of sensitive prompts that state-aligned LLMs refuse or rewrite, against neutral controls.	Public model APIs	daily	LIVE
GDELT cross-signal censored at home, loud abroad	Which censored topics the world's press is covering heavily (containment) versus silently absent (blackout).	GDELT global news index	every 6 hours	LIVE
GitHub-refuge pressure on the mirrors	Takedowns, legal blocks, and visibility drops on GitHub repos that shelter censored material, plus defensive fork and star bursts.	GitHub public API	twice a day	LIVE
Eval Registry sealed model evaluations	Pre-registered, hash-chained audits of Chinese state-aligned and Western frontier models on the same frozen probes, including refusal drift over time. Results cannot be revised after sealing; anyone can recompute the chain.	Public model APIs	every 6 hours	LIVE
Erasure Observatory composite erasure index	What the record lost, across the network, narrative, and model layers, sealed into a tamper-evident ledger. Layers that cannot report are shown absent, never zero-filled.	OONI, model audits, Baike (armed)	every 6 hours	LIVE
GFW network signal live firewall blocking	Independent, side-channel measurements of Great Firewall blocking events, with historical backfill.	OONI open data	every 6 hours	LIVE
Velocity deletion speed	How fast a post is deleted after posting, timed to the minute.	In-country observation	held back	SUPPRESSED

Get the data

All files are plain JSON. The *-latest.json files hold the current snapshot; the *-history.jsonl files are append-only time series, one compact record per run, so you can chart a signal over time.

ddti-latest.json

DDTI snapshot: ranked terms with threat, attention, novelty, is_new, and source samples with links.

ddti-history.jsonl

DDTI time series: generated_at, n_terms, top_term, top_threat per run.

latest.json

Generative Firewall raw run: per-model, per-concept refusals, controls, and the GFI value.

history.jsonl

Generative Firewall time series: date, gfi, censored, total, controls.

gdelt-latest.json

GDELT cross-signal: each DDTI term with global coverage volume and a containment or blackout label.

gdelt-history.jsonl

GDELT time series: n_terms, containment and blackout counts, top term.

github-refuge-latest.json

GitHub-refuge: each watched mirror with status, pressure likelihood, fork and star novelty, and any matched takedown notices.

github-refuge-history.jsonl

GitHub-refuge time series: repos watched, present, and pressure events per run.

eval-registry.jsonl

The registry chain itself: every pre-registration and sealed run, hash-chained in order. This file is the tamper-evident record; verify it with scripts/verify_eval_registry.py.

eval-registry-latest.json

Registry summary: attestation counts, models audited, recent runs, Merkle root, head hash, verified flag.

refusal-drift-latest.json

Frontier refusal drift: per-model suppression on the frozen benign probe set, refused probes, and drift versus the prior run.

erasure-observatory-latest.json

Composite Erasure Index snapshot: per-layer readings (network, narrative, model) and the mean of reporting layers.

erasure-ledger.jsonl

The sealed erasure ledger: every observatory reading, hash-chained; verify with scripts/verify_ledger.py.

ooni-gfw-latest.json

GFW network signal: independent side-channel blocking measurements from OONI open data.

A machine-readable summary for AI agents lives at llms.txt. The full directory is at /readings.

How it is measured

Deletion as data. Palimpsest reads public censorship records, ranks what is being scrubbed by a recency-weighted attention score, and flags terms that are newly sensitive against their own baseline. The DDTI method and its selectivity and novelty maths are documented in full.
The generation layer. The Generative Firewall probes state-aligned models on sensitive prompts in Chinese and English, and compares them to neutral Western control models, so a refusal reads as deliberate policy rather than a generic safety filter.
Validation. The deletion signal has been checked against six documented censorship events. Where a claim cannot be validated, it is labelled as such and not published as fact.

Method: METHODOLOGY.md · NEW-METHODS.md · VALIDATION.md · Sources: OSINT_SOURCES.md · Ethics: ETHICS.md · SAFETY.md

Scope and honest limits

Palimpsest publishes only what it can stand behind. It measures censor attention, which topics draw the most suppression effort, not a full deletion rate. The velocity signal, how fast a specific post is removed, needs a network vantage inside China to time deletions to the minute. Until that vantage is in place and verified, velocity stays suppressed by design. Nothing on this site is a modelled guess dressed up as a measurement.

Every data point carries its source and timestamp. The DDTI terms link back to the original public articles.
Palimpsest watches the censor, never the censored. It uses public reads only, stores nothing about private individuals, and never asks anyone inside a country to act.
State-aligned AI models are subjects of measurement, never the analyst.

How to cite

If you use this data in an article, report, or paper, a citation and a link back are appreciated. Please cite the accessed date, since the signals update continuously.

Palimpsest (2026). Palimpsest Censorship Observatory and Verifiable Eval
Registry: DDTI, Generative Firewall Index, GDELT cross-signal, and sealed
model evaluations [live dataset]. https://palimpsest.info
(accessed YYYY-MM-DD).

@misc{palimpsest,
  title  = {Palimpsest Censorship Observatory and Verifiable Eval Registry},
  author = {Palimpsest},
  year   = {2026},
  url    = {https://palimpsest.info},
  note   = {Open live dataset: DDTI, Generative Firewall Index, GDELT
            cross-signal, Wayback deletion reconstruction, sealed eval
            registry with refusal drift}
}

Reproducibility and provenance

Palimpsest is fully open source under the MIT licence. The signals recompute inside public, auditable GitHub Actions, and every refresh is a timestamped commit in the repository, so the entire history of what was measured, and when, is public and verifiable. There is no private backend deciding the numbers.

Two of the surfaces go further than the commit trail: the erasure ledger and the eval registry are hash-chained and Merkle-committed, so they stay verifiable even outside git. Clone the repo and run python3 scripts/verify_ledger.py or python3 scripts/verify_eval_registry.py; exit 0 means every seal recomputes and, for the registry, that every run referenced a probe set frozen before the model was queried. This constraint applies to us too. If we edited a published number, our own verifier would report the break.

The roots are also deposited with parties we do not control. Every refresh that moves a root gets an Internet Archive snapshot of the served chain files and an OpenTimestamps stamp into Bitcoin (the .ots proofs live in readings/anchors/ and verify with the standard client, against the blockchain, not against us). An independent witness on separate infrastructure re-verifies the served chains on a timer and alerts if any previously seen history changes; anyone can run one with python3 ops/witness/palimpsest_witness.py. Single attestations verify without downloading the chain via python3 scripts/prove_inclusion.py <seq>. The full layer-by-layer trust model, including what these layers cannot prove, is written down in docs/INTEGRITY.md.

Repository · Commit history (audit trail) · The pipelines

Read Chinese? We need four hours of your judgment

The Generative Firewall Index labels every model answer with a transparent rule based classifier: refused, state narrative, or answered. A researcher whose work this instrument builds on asked us a fair question: would actual humans agree with those labels? We want to find out properly, and that part cannot be automated. We need two volunteer coders.

The task. We send you a short manual and a spreadsheet of about 200 short texts, some Chinese, some English. Each one is a chatbot's answer to a question. You read the answer and pick one of three labels. That's the whole job. Around four hours, alone, on your own schedule inside two weeks.

Who. Anyone who reads Chinese fluently. Any nationality, no technical background, the manual carries everything you need. One hard restriction: if you are currently in mainland China or Hong Kong, please do not volunteer. The texts touch politically sensitive topics, and no dataset is worth risk to you.

The rules. Work alone, no comparing notes with the other coder until you are both done, and no AI help of any kind. If a machine assists the judgment, the study measures nothing. The entire point is unaided human agreement.

What you get. Your name in the acknowledgments here and in the research paper this feeds, or full anonymity if you prefer. Either way, a public instrument that watches censorship gets a stronger spine because you read carefully for an afternoon.

To volunteer, open a GitHub issue titled Validation coder and say roughly where you are and how you come by your Chinese: native, degree, HSK, or grew up with it. First two qualified volunteers get the sheets.

Work with us

If you are a reporter, researcher, or think tank and want a specific term tracked, a data extract, or help interpreting a signal, open an issue on GitHub. Palimpsest is a public good and collaboration is welcome.

Free and open source, developed in the open as a public good. Never a commercial product, and it never monetizes the people or topics it observes. The internet stays free because people keep measuring the dark.