When several language models give the same answer, does that make it more trustworthy? Comparing responses to synthetic Korean-medicine classification cases led me to an earlier question: what counts as the same answer? Exact wording, broader categories and a set of classification axes do not preserve the same information.
This was an exploratory reanalysis of an archived response record without an expert reference. It was not a study of patients receiving care, and agreement between models could not establish diagnostic accuracy. I revised the analysis and submission packages, then put reworking and resubmission on hold on 21 July 2026 after five rejected submissions.
The comparison begins before the number
Exact text matching separates responses that differ in wording or format, even when their meaning is close. A broader category can bring more responses together while discarding distinctions in the original text. Before favoring one measurement, I needed to explain what each representation treated as equivalent.
I distinguished surface wording, categories produced by an archived collapsing rule, and a representation split into classification axes. These were different ways to encode the same response record, not interchangeable measurements. Their definitions belonged alongside any interpretation of agreement.
Missing information introduced another choice. Two responses with the same absent field could be counted as matching, treated as nonmatches, or omitted in favor of complete comparisons. Each approach changes what is being compared. Reporting only an agreement value would conceal the contribution of those handling rules.
Choosing the models also shapes the comparison
A collection of models from several vendors is still a selected comparison group. Different models or access routes from one vendor can produce different response patterns. I examined this selection of requested endpoints—the panel—as another condition on the interpretation.
The archived record also had limits: verbatim completions and detailed version and request provenance were not sufficiently retained. I could not describe the work as fresh data collection under fully recoverable conditions, or turn it into a claim that one vendor or model was intrinsically superior. Its scope was the sensitivity of agreement measurement to representation and panel choices within a fixed archive.
Revising the manuscript, then pausing
I revised successive submission packages to clarify the scope and contribution, but the lineage accumulated five rejections. At least one was an editorial-screening decision without external peer review. Those decisions neither validated nor disproved the findings; I chose not to continue submitting the same framing.
The decision recorded on 21 July was to pause before redesign. Preparing another manuscript would first require a clearer response-recording plan and a reference against which to define the claims. The unresolved question is not merely whether several answers converge, but how the process of encoding and comparing them makes that convergence measurable.