To quantify the clinical consistency of expert panelists' ratings of appropriateness of pre-operative and post-operative chemotherapy plus radiation for rectal cancer.
A panel of nine physicians (two surgeons, four medical oncologists, three radiation oncologists) rated the appropriateness of providing pre-operative and post-operative treatments for rectal cancer, utilizing a modified-Delphi (RAND/UCLA) approach. Clinical scenarios were paired so that each component of a pair differed by only one clinical feature (e.g. tumor stage). A pair of appropriateness ratings was defined as inconsistent when the clinical scenario that should have had the higher (or at least equal) appropriateness rating was given a lower rating. The rate of inconsistency was analyzed for panelists' ratings of pre- and post-operative chemotherapy plus radiation.
The final panel rating was inconsistent for 1.19% of pre-operative scenario pairs, and 0.77% of post-operative scenario pairs. Using the conventional RAND/UCLA definition of appropriateness, the magnitude of the inconsistency would produce inconsistent appropriateness ratings in 0.43% of pre-operative and 0.11% of post-operative scenario pairs. There was significant variation in the rate of inconsistency among individual panelists' final ratings of both pre-operative (range: 0.43-5.17%, P