How member papers are tagged

The expert panel

The codebook is too large for one reading. It is divided among several readers, each given one group of related dimensions and a description of the kind of specialist who would judge them: a research-methods specialist for study design, a research-data specialist for what kind of knowledge and data a paper contributes, an Earth-system specialist for the physical setting, and so on. Each reader sees the whole paper but assigns values only on its own dimensions.

Each reader is shown, for every value it may assign, the value’s definition, what qualifies for it, and what must not be tagged with it. That full wording is what the atlases use. It is not a list of bare labels.

A paper whose subject matches one of the published atlases is also read by that atlas’s own specialists. A paper is matched to an atlas only when its DOI resolves to a species or subject already in that atlas. Where it matches nothing, or matches two, only the shared panel reads it. The subject is never inferred from the model’s own description of the paper.

The three conditions a tag must meet

A proposed tag is stored only if it passes all three.

The judge sometimes declines to answer on papers about disease organisms and biological control, which a safety classifier misreads. A declined answer is recorded as the judge being unavailable, not as a rejection, and such tags are marked and are excluded from the accuracy measurement below rather than counted as correct.

Papers outside the codebook’s subject area

Researchers publish in every field. The codebook covers the environment, biodiversity, ecology, conservation, Earth systems, natural products and environmental health. Before the panel reads a paper, a short first pass names the paper’s field and decides whether the codebook covers it.

A paper outside that area is offered only the dimensions that describe any study at all, such as study type, methods, and what kind of knowledge it contributes. The ecological dimensions are not offered to it. A paper in condensed-matter physics is described as a laboratory study in the physical sciences and is not given a habitat or a threat.

What the researcher sets themselves

A researcher can set any field on their own paper. A field they have set is theirs: the tagging never overwrites it, and it is shown as the author’s own statement rather than as a produced tag. The tagging fills empty fields only, unless the author asks for the paper to be tagged again.

How the accuracy is measured

Accuracy is not assumed from the method. It is measured by taking a sample of stored tags and showing each one to a third model, different from both the reader and the judge, with the paper, the dimension, every allowed value for that dimension, the asserted value with its definition and rules, and the quoted evidence. The model is asked to confirm or reject. It is not told that the tag was already accepted.

The sample is drawn in two halves. One half is drawn at random from all stored tags, which is what a reader browsing tags actually meets. The other half takes one tag per dimension, which gives the rarely used dimensions enough weight to be seen. The result is reported as the share confirmed, with a 95 percent interval, for the whole sample and for each half.

This is the same procedure, the same wording and the same sampling design used for the atlas tags, so the two numbers are comparable rather than two different kinds of measurement. Papers a member has marked private are never included in the sample.

Corrections by the author

When a researcher changes a tag on their own paper, that change is kept, with the previous value and the new one. An author correcting the description of their own work is the most direct evidence available that a tag was wrong.

It is reported next to the accuracy measurement and never added into it. They answer different questions: one is an independent model’s verdict on a drawn sample, the other is an author’s judgement on one paper they wrote. A field the researcher is the first to fill is not a correction and is not counted as one.

Current state of the measurement

The method described above is implemented and the measurement runs on demand. No accuracy figure for member papers is published yet, because the measurement has not been run on a body of real member papers. When it has been, the share confirmed, the interval, the sample size, the date and the sampled tags will be published here, together with the correction rate.

The same procedure is run against the published atlas corpus, and that result is reported with the atlas it describes. It is a separate measurement of a separate body of papers and is not a figure for member papers.

What the procedure produces is one independent model’s second opinion on a sample, not ground truth. A reviewer applying a strict reading will reject some tags another reviewer would accept, so the share confirmed is better read as the rate at which an independent reviewer agrees than as a count of correct tags. The two halves of the sample are reported separately for the same reason: the randomly drawn half is the closer estimate of what a reader browsing tags actually meets.

Papers tagged before this method was introduced keep the tags they have. Those tags were produced by a single reading with no evidence requirement and no judge. They are not known to be wrong, they are unmeasured, and they are marked as such rather than removed.

Provenance

Every stored tag records the quoted evidence, how much of that quote occurs in the paper’s own text, which model proposed it, which model confirmed it, and a prompt identifier.

The prompt identifier is derived from the exact wording given to every reader and judge together with the full codebook, so two tags carrying the same identifier were produced under identical instructions and an identical codebook, and any change to either produces a different identifier. The wording it refers to is stored, so a tag can always be traced back to the instructions that produced it.