What makes an advisory board publishable

Most conversations I have about consensus studies start the same way.

A team has run an advisory board. The discussion was genuinely good — experienced clinicians, real disagreement, a few insights nobody had written down before. Somebody in the room says, “this should be published.” And then the question arrives: can we actually publish this, or will a journal dismiss it as expert opinion?

It is a fair question, and the honest answer has two halves.

The uncomfortable half

A write-up of an unstructured discussion is expert opinion. Reviewers will treat it that way, and they are not wrong to. If five people talked for three hours and one of them was more senior, more confident or simply louder than the others, there is no way for a reader to know whose view the conclusions represent. There is no denominator. There is no way to tell a statement that everyone endorsed from a statement nobody objected to out of politeness.

That is not a failure of the people in the room. It is a failure of design.

The encouraging half

The same group of experts, convened differently, can produce something that stands up to peer review — because formal consensus methods are a recognised way of generating evidence, not a workaround for lacking it.

Jones and Hunter set this out clearly three decades ago: consensus methods such as the Delphi technique and the nominal group technique (the expert panel) are a means of synthesising information, and where published information is inadequate or non-existent, they let you harness the insight of appropriate experts so that decisions can be made.¹

That last condition matters. Consensus is not a substitute for a trial that could have been run. It is the appropriate method when the evidence is limited, inconsistent, emerging or absent — and when a decision has to be made anyway.

And this is not a fringe approach. Several of the reporting standards that journals and health technology assessment (HTA) bodies now rely on were themselves built by consensus: the CARE guideline for case reports,² the CERT template for exercise interventions,³ and the COSMIN standards for patient-reported outcome measures⁴ were all developed through formal Delphi processes. The machinery that defines good reporting was, in several cases, produced by the very method that people worry is not rigorous enough.

What actually separates rigorous from weak

Here is the part that should interest anyone commissioning this work.

Black et al. reviewed the evidence on consensus development methods and found that the output is shaped by a specific set of design choices: how the task is set, who is selected to participate, how scientific information is presented to them, how the interaction is structured — the number of rating rounds, whether participation is equitable — and how individual judgements are synthesised, including how agreement is defined and how outliers are handled. Their conclusion is the sentence worth remembering: adherence to best practice enhances the validity, reliability and impact of what is produced.⁵

Credibility is not a property of the method. It is a property of the execution.

And execution is, empirically, where most studies fall down. Diamond et al. examined a random sample of 100 published Delphi studies. Of those, 98 claimed to assess consensus, but only 72 defined what consensus meant — and only 42 defined it before starting. A total of 70 stopped after a fixed number of rounds rather than when agreement was actually reached.⁶

Read that again, because it is the whole argument: in most published consensus studies, the authors decided what counted as agreement after seeing how people voted, or never decided at all.

Journals now have a standard to hold you to

The reporting side has caught up. In 2024, ACCORD — the ACcurate COnsensus Reporting Document — was published in PLoS Medicine as the first reporting guideline applicable to all consensus methods in biomedical research. It was developed through the EQUATOR Network’s methodology, with a 72-member Delphi panel across six continents, and produced a 35-item checklist covering how a consensus study should be described.⁷

Its stated purpose is instructive: to make the rigour of the consensus methods clear to readers, and so to enhance trust in the recommendations consensus panels make. A reporting guideline is, in practice, a scoring rubric. It tells you what a reviewer will look for — which means you can design to it from the start.

What this looks like in practice

I co-authored a Thai expert consensus on bone health optimisation for instrumented spine surgery.⁸ The design was deliberately conventional: a targeted literature review to draft the statements, review and validation with subject matter experts, an agreement threshold agreed before any rating took place, two rounds of independent rating by the panel, and then a face-to-face meeting to work through the statements that had not reached the threshold — discuss, reword, re-rate, finalise.

The output was 25 statements and 45 sub-statements covering screening, assessment and treatment by risk category, in a context where local guidance had been limited. Clinicians can use it. It is citable. And every methodological choice is visible to a reader.

That matters beyond the citation. What the panel produced is practical treatment guidance for bone health optimisation — a local reference a surgeon can use to decide who to screen, what to assess and how to treat before instrumented spine surgery. Where guidance had been thin, there is now a stated standard to work to, which is how a consensus study earns its place: not as a publication, but as something that helps the right patients get the right treatment at the right time.

None of it was exotic. What it required was deciding, before the meeting, what kind of output we were building — and planning for the fact that a rating panel is a separate group from the people who draft and validate the statements, with its own recruitment and its own budget line. That separation is the part most often underestimated, and it is the part that cannot be improvised once the discussion is over.

The one thing you cannot retrofit

You cannot add rigour to a discussion that has already happened. You cannot decide the agreement threshold after seeing the votes, or reconstruct an anonymous rating round from a transcript, or claim independence for a panel that was assembled to be agreeable.

So if you are planning an advisory board and there is any chance you will want a publication out of it, the question is worth asking at the design stage, not afterwards: should this be a consensus study?

Often the honest answer is no — some advisory boards exist to inform internal strategy, and that is a perfectly good reason to hold one. But when the answer is yes, the difference in cost is modest and the difference in what you end up owning is substantial.

← Back to Insights