WG21 AUDIT
WG21.org Paper Review Process
Each paper submitted to the WG21.org mailing list is processed through a structured review system using AI assistance to provide an overview of the paper through automated analysis. Information is returned as an Audit for each paper in the enhanced mailings.
CORE OBJECTIVE
The purpose of the AI review is to determine whether the claims inside a paper are supported by the evidence provided, using a consistent, structured, and verifiable procedure — while also checking alignment with SD-4 as a required baseline. It’s intended to be used as an additional tool for readers, not as a standalone or definitive reflection of the paper’s contents.
IMPORTANT NOTE
The Audit is not intended to be read as a recommendation of acceptance or rejection of any paper. The goal is to provide an automated review of the paper that is as unbiased as possible.
THE ROLE OF AI IN THIS PROCESS
AI assistance is used to:
- Identify missing information (in evidence, rationale, or benchmarks)
- Flag missing information across sections
- Generate initial audit notices for human review
- Apply standard test patterns consistently
- Highlight compound dynamics that may not be obvious on a single read
The review is made public so readers may examine the findings and verify them against the original paper.
REVIEW PROCESS – STEP BY STEP
STEP 1: INTAKE
The paper is cataloged, citations are verified, and the document is divided into sections for analysis.
Extracts claims, evidence, concessions, questions, dependencies, scope statements, and explicit requests. Record exact text and line numbers.
Identifies missing information (claims without evidence, sections without rationale, performance assertions without benchmarks). The AI tags the data with Categories of Inspection and assigns severity levels.
After both reads, a one-sentence summary is written identifying load-bearing claims.
STEP 2: RESEARCH REVIEW
For each Category of Inspection, up to three research searches are performed. Only findings connected to the paper’s thesis are retained.
- Citations compared against extraction annotations
- Sections checked against standard test patterns
- Compliance checklist based on SD-4 (WG21 Practices and Procedures) applied
Categories of Inspection
Category
Description
Performance
Claims about runtime, memory, compile time, or throughput
Design
Architectural choices, interfaces, and behavioral models
Specification
Wording, consistency with the standard, and normative references
Usability
Impact on end-users, learnability, and error handling
Ecosystem
Interaction with existing libraries, tools, and platforms
Rationale
Justifications for design decisions and problem statements
| Category | Description |
|---|---|
| Performance | Claims about runtime, memory, compile time, or throughput |
| Design | Architectural choices, interfaces, and behavioral models |
| Specification | Wording, consistency with the standard, and normative references |
| Usability | Impact on end-users, learnability, and error handling |
| Ecosystem | Interaction with existing libraries, tools, and platforms |
| Rationale | Justifications for design decisions and problem statements |
STEP 3: VERIFICATION
Each finding within a paper is re-examined against five verification criteria below. Findings that are already addressed, self-resolved, or not relevant are removed.
The report records how many audit notices were generated, survived, and removed. Surviving findings are reviewed for compound dynamics.
Verification Criteria
Applied during Step 3 to determine which findings remain in
the final report:
-
Already acknowledged
The paper names this limitation, trade-off, or counterargument elsewhere
-
Directly claimed
The finding is based on an actual statement in the paper, not an inference
-
Self-resolved
The paper itself provides the answer elsewhere (even implicitly)
-
Practically relevant
A committee member would reasonably raise this in discussion
-
Structural
The issue relates to logic, evidence, or claims (not formatting or style)
Items that are editorial, formatting, or stylistic are not recorded
as structural findings.
Applied during Step 3 to determine which findings remain in the final report:
-
Already acknowledged
The paper names this limitation, trade-off, or counterargument elsewhere
-
Directly claimed
The finding is based on an actual statement in the paper, not an inference
-
Self-resolved
The paper itself provides the answer elsewhere (even implicitly)
-
Practically relevant
A committee member would reasonably raise this in discussion
-
Structural
The issue relates to logic, evidence, or claims (not formatting or style)
Items that are editorial, formatting, or stylistic are not recorded as structural findings.
FINAL REPORT CONTENTS
After the three-step review process, an overall assessment is applied. Assessment outcomes indicate information status using the meanings below:
Three Possible Outcomes
The submission contains the necessary information. Any notes are minor or informative.
The submission has useful content but would benefit from additional information in specific areas.
The submission is missing substantial information needed for proper evaluation. Revision recommended.
The final report will also summarize and return the following notices from the audit:
- Requests the paper makes to the committee
- Structural assessment
- Any compound dynamics
- Major audit notices (in order of severity)
- Regular audit notices
- Strengths
- Rationale checklist
- Reference table
- Complete inventory of what was extracted, filed, verified, and survived
- Ratio of audit notices generated to those surviving verification
When the audit notices section is empty, the paper contained the expected information across all checked categories.
SUMMARY
The Audit provides an automated review of each paper. It identifies where information may be missing and where a paper might benefit from additional detail. Readers may use the findings as part of their own review of the paper. The Audit is presented to highlight observations; it is not intended to be read as a recommendation of acceptance or rejection of any paper.