Ultrasound G.E.L. – Emergency Physician Accuracy in Identifying and Grading Hydronephrosis

Originally published on Ultrasound G.E.L. on 5/21/18 – Visit HERE to listen to accompanying PODCAST! Reposted with permission.

Follow Dr. Michael Prats, MD (@PratsEM), Dr. Creagh Bougler, MD (@CreaghB), and Dr. Jacob Avila, MD (@UltrasoundMD) from Ultrasound G.E.L. team!

Emergency physician interpretation of point-of-care ultrasound for identifying and grading of hydronephrosis in renal colic compared with consensus interpretation by emergency radiologists

Acad Emerg Med April 2018 – Pubmed Link

Take Home Points

1. There was disagreement in diagnosing hydronephrosis on POCUS, especially mild hydronephrosis. Emergency physicians interpretation of POCUS was 85.7% sensitive for detecting hydronephrosis compared to radiology interpretation. Both had similar sensitivity compared to CT and overall poor specificity.

2. Specificity by emergency physicians improved when diagnosing ≥ moderate hydronephrosis (94.6%).

3. There was significant difference in interpretation based on level of training.

Background

Pain from kidney stones is a common reason to come to the emergency department. Ultrasound has been shown to be a good first step in assessment – to determine the presence and degree of hydronephrosis. Although computed tomography is still standard for finding the size and location of the stone, studies have shown that an ultrasound-first approach can reduce the need for CT without increasing risk to the patient. As many are switching to this approach and since management might change based on the degree of hydronephrosis found, it is important to establish that point of care ultrasound is accurate in this determination. This study wanted to see how emergency physicians (EP) interpretation of POCUS compared to radiologists interpretation of the same images.

Questions

How accurate is emergency physician interpreted POCUS for hydronephrosis compared to emergency radiologists?

How accurate are these interpretations compared to CT?

Population

Urban academic tertiary care center in Doha, Qatar

Inclusion:

• Enrolled in previous study
• Moderate to severe renal colic
• POCUS performed
• CT performed within 24 hours of POCUS

Exclusion:

• Incomplete POCUS
• No CT scan

Design

Study using previously collected renal US data

Original trial enrolled consecutive patients with moderate to severe renal colic, had POCUS performed.

The present study used the patients who had both POCUS and a CT within 24 hours (66.3% of original population).

So they took patients who had both POCUS and CT. Six different EPs evaluated each clip. Asked to evaluate presence, grade, and (if bilateral) worse side of hydronephrosis. Their interpretation of the degree of hydronephrosis was compared to a consensus interpretation from emergency radiologists. Two radiologists evaluated and if disagreement, settled by a third. Both parties blinded to clinical data.

Then, these interpretations were compared to CT scan results (the interpreter of CT was blinding to POCUS).

Interrater agreement also assessed.

Who did the ultrasounds?

Emergency physicians or research assistants performed original ultrasounds.

Their training was 30-minute didactic teaching session + performance of 35 renal US under supervision

Unfortunately, these images were not interpreted by the people who acquired the images.

Interpreted by:

• Two attending EPs
• Two “advanced EM” fellows
• Two residents

Received 45 minute training session including 15 minute didactic + 25-30 practice scan interpretations with feedback

The Scan

Curvilinear transducer, abdominal settings

Their protocol:

• Longitudinal and transverse views of kidneys bilaterally, all 6 second clips

Their hydronephrosis definitions:

• Mild: separation of the renal sinus and enlargement of calicies, preservation of renal papillae
• Moderate: blunting or rounding of the calices or obliteration of renal papillae without affecting cortical thickness
• Severe: caliceal ballooning, cortical thinning

*Clinical pearl: if you’re not sure if there is hydro or not → probably doesn’t matter (because mild hydro is not very specific for requiring an intervention or for the diagnosis of kidney stone). Note: this assumes you have some experience under your belt and can accurately exclude more significant hydro.

Learn POCUS for hydronephrosis at 5 Minute Sono

Results

The Patients

N = 651 patients

• Median age 34
• 83.7% male
• Median BMI 26.6

69.6% had hydronephrosis on ultrasound as interpreted by radiologist

• None 30.4%
• Mild 59.3%
• Moderate 9.5%
• Severe 0.8%

72.7% had hydronephrosis by CT

• Median stone size 4 mm (IQR 3-7)

Primary Outcomes

Accuracy of EP identifying presence of hydronephrosis compared to radiologist interpretation

Sensitivity 85.7% (84.3-87)

Specificity 65.9% (63.1-68.7)

+LR 2.5 (2.3-2.7)

-LR 0.22 (0.19-0.24)

Attending EPs vs Training EPs

Attending EPs Sensitivity 96.9% (95.6-97.9)

Training EPs Sensitivity 80.1 (78.2-81.9)

Attending EPs Specificity 57.2% (57.1-62.2)

Training EPs Specificity 70.3% (66.9-73.5)

Secondary Outcomes

Compared to CT diagnosis of hydronephrosis

EP sensitivity 81.1% (79.6-82.5)

Radiology sensitivity 85% (82.5-87.2)

EP Specificity 59.4(56.4- 62.5)

Radiology specificity 79.7% (75.1- 83.7)

Post-HOC analysis of moderate and severe hydronephrosis grouped together

EP specificity 94.6% (93.7-95.4)

EP sensitivity 34.2% (31.2-37.3)

Attending EPs specificity 98.4% (97.4-99.1)

Training Eps specificity 92.7% (91.4-93.8)

Radiology specificity 97.3%

Radiology sensitivity – not reported

Other Findings

Median time between POCUS and CT was 104 minutes after (but ranged from many hours before to many hours after). 21 CTs had “signs of recently passed stone”.

Radiologists disagreed on presence of hydro in 10.1% cases (all mild hydro).

Radiologist interobserver agreement K = 0.77 (presence or absence) and 0.82 (grading)

Radiologist vs CT K = 0.47 (presence or absence) and 0.64 (grading)

False negatives – 11 had moderate and 365 had mild compared to radiology

Compared to CT:

• 14/148 moderate hydro called normal by radiology, 86/506 moderate called normal by EP
• 2 cases where EP called no hydro but severe hydro on CT

Limitations

Single center, not in US

Did not use the actual interpretation by the treating physician. Used a second interpretation by the study investigators. They stated this is because the original interpretations were incompletely recorded.

Time between CT and POCUS could lead to changes in the presence and grade of hydronephrosis between studies.

No color doppler used to confirm that hydronephrosis was not actually renal vessels.

Relatively few moderate and severe hydro exams to draw conclusions from.

Relatively small stones may be less likely to cause hydronephrosis.

This study only assessed for the presence of hydronephrosis, not the diagnosis of nephrolithiasis. This is distanced from an actual patient centered outcome.

Discussion

If radiologist did not correlate that well to CT, should that be the comparison for EPs? Interesting that the sensitivities with comparison to the standard of CT were fairly similar. If that is the case, could have just as easily compared EP to radiologist, and made the conclusion that the radiologists have decreased accuracy for interpreting POCUS images. It is true that the radiologist had great specificity than EPs compared to CT. Most likely, they are more accurate, based on this comparison, but that assumption was embedded in the study to start.

You have to ask yourself – what am I going to do with this information? Trying to rule in nephrolithiasis? – might want to use only findings of moderate or severe hydro. Trying to rule out nephrolithiasis? – well since there is significant disagreement with mild hydro, POCUS may not be the best study.

It is unfortunate that this study took out the acquisition component of POCUS and compared only the interpretations. Is the difference between POCUS and comprehensive radiologist performed study based on the acquisition (equipment + operator skill) or interpretation (EP vs radiologist)? Unable to determine that from these results.

Take Home Points

1. There was disagreement in diagnosing hydronephrosis on POCUS, especially mild hydronephrosis. Emergency physicians interpretation of POCUS was 85.7% sensitive for detecting hydronephrosis compared to radiology interpretation. Both had similar sensitivity compared to CT and overall poor specificity.

2. Specificity by emergency physicians improved when diagnosing ≥ moderate hydronephrosis (94.6%).

3. There was significant difference in interpretation based on level of training.

Our score