Assess-AI FAQs

What is Assess-AI?

Assess-AI is a quality assurance/quality improvement registry under the umbrella of ACR’s National Radiology Data Registry (NRDR) that provides real-world monitoring of AI algorithms deployed in a clinical workflow.

What is the purpose of Assess-AI?

Assess-AI compares the performance metrics of a site’s locally deployed AI algorithms to national benchmarks and facilities with similar characteristics (i.e., region, facility type, trauma level, urban/rural).

What services does Assess-AI offer?

Enrollment in Assess-AI provides sites with access to the following services:

• Monitoring of algorithm inputs including imaging equipment, protocols, and software version
• Continuous monitoring and concordance with radiology reports
• Reports and dashboards to share with your AI governance team

Where can sites access performance reports, and what types of reports are available?

Participating sites will be able to access performance reports via ACR Dart. Aggregated data will be analyzed to monitor the performance of AI products deployed at a site. The information provided includes:

• Concordance and discordance rates between AI results and the radiology report
• Comparisons to national benchmarks
• Comparisons to similar facilities

For more information about Assess-AI reports, click here.

What data is needed in order to participate?

Data submitted by participating sites can be classified into four categories:

• Image study metadata
• AI Results
• Patient demographics
• Radiology report text

This data allows the Assess-AI registry to produce rich, high-fidelity analysis and information to guide your organization on the performance of your AI solutions.

For more information about Assess-AI data submission, click here. You may also view the Assess-AI Data Dictionary and Assess-AI Data Ingest Specification.

Does Assess-AI require the submission of Protected Health Information (PHI)? 

The only PHI that is required is the imaging study date. No other PHI is needed to be submitted to the registry. Assess-AI provides tools to assist with the de-identification process and follows the DICOM standard; however, it is the responsibility of each site to ensure that data is properly de-identified before submission. To aid in this process, sites have the ability to preview the data before it is sent to the registry.

How is data submitted to the registry?

Data is submitted to the registry via ACR Connect. Assess-AI registry data ingestion, local processing, and submissions are handled on the Assess-AI app running on ACR Connect at each facility. Facilities with ACR Connect installed for other NRDR registries’ data submissions can utilize the same platform for the Assess-AI registry.

What is ACR Connect?

ACR Connect is a software only system that is installed locally at a facility. It expands the local processing capabilities and minimizes the need to move data offsite. It is a cloud native solution and is an API based, standards friendly platform that promotes integration, collaboration, and customization. It supports multiple initiatives on the same ACR Connect platform using an extensible “app model”.

What are the technical requirements for installing ACR Connect?

ACR Connect system requirements and installation details can be found on the ACR Connect knowledgebase.  

Is there a cost to install ACR Connect?

There is no cost for installing the ACR Connect software; however, the ACR does charge for some of the services provided through ACR Connect.

What steps are involved in the enrollment process for Assess-AI?

The enrollment process for Assess-AI depends on whether an organization is already enrolled in NRDR.

If an organization has not yet enrolled in NRDR, they will need to initiate the NRDR application process. This involves submitting a Participation Agreement (which can be signed online) and BAA. A one-time NRDR registration fee of $500 will be incurred during this process.

If an organization is already enrolled in NRDR, the corporate administrator for the organization simply needs to log in to the NRDR portal and check the box next to Assess-AI to add their facilities. More information for adding a new registry can be found here. There is no additional paperwork required.

Is Assess-AI subject to IRB approval?

Assess-AI operates as part of NRDR, which is IRB exempt. To view the exemption, click here.

What is the fee schedule for participating in Assess-AI?

My organization has multiple facilities. Which facilities should I enroll in Assess-AI?

We generally recommend enrolling in the Assess-AI registry for all facilities for which you have implemented AI solutions. If you have not implemented AI solutions for all of your facilities yet, you can enroll them at a later time as you expand the implementation.

What is the next step after enrolling in Assess-AI?

After enrolling in Assess-AI, please verify that ACR Connect is installed at your facility. If not (or if you are unsure), you can contact ACR Connect support team by either submitting a ticket here or by e-mailing acrconnect@acr.org.

Once ACR Connect is installed, an overview session with your team will be scheduled to help your organization gain a better understanding of the registry’s features.

Finally, a training session will be arranged for your team in order to activate and configure the registry.

Use of Large Language Models (LLMs) in Assess-AI  

How are the Radiology Results processed?

Radiology results are processed using commercial large language models (LLMs) to extract findings from the reports. However, only the ACR will have access to the data from these reports. LLM developers will not have access to the data, which ensures that it is not used for training or any other purposes beyond the registry’s scope.

How are LLMs used in Assess-AI?

For Assess-AI, participating facilities will be submitting AI results and de-identified radiology reports. To make a concordance calculation between the two, the surrogate ground truth must be inferred from the radiology report. Radiology reports are highly unstructured and differ to large degrees between institutions and radiologists. LLMs will be used to extract structured findings from unstructured text in an automated way, thus enabling surrogate ground truth extraction at scale.

What is surrogate ground truth?

ACR terms the LLM extracted radiological findings as “surrogate ground truth” instead of “ground truth” to reflect the fact that the findings will be inferred with LLMs and not annotated by a radiologist. ACR acknowledges that LLMs have limitations and can be prone to issues such as hallucinations and overconfidence. However, ACR takes precautions to ensure the highest performance possible such as rigorous prompt development, testing, and model selection as well as ongoing performance monitoring.  

Which LLMs are being used for Assess AI?

Assess AI is utilizing a HIPAA compliant AWS managed service called Amazon Bedrock. Amazon Bedrock provides access to ~20 foundation models (FMs), or off-the-self LLMs, developed by Amazon and third-party companies such as AI21 Labs, Anthropic, Mistral, and more.

What does the Assess-AI registry do with my de-identified report text?

ACR will append your de-identified reports to a use case specific LLM prompt and send it to a secure LLM API to extract surrogate ground truth. For example, appending a report to the following prompt

“In the report above, does the radiologist report intracranial hemorrhage? Respond with positive or negative.”

This may yield the following result:

“Positive”

The result will be captured in a database and used to calculate concordance with an imaging AI result.

Will my de-identified report text be used by Amazon or Bedrock third party model providers?

No, your de-identified report will not be used by Amazon or other Bedrock third party model providers.