Diagnostic performance of rapid antigen tests (RAT) for COVID-19 and factors associated with RAT-negative results among RT-PCR-positive individuals during Omicron BA.2, BA.5 and XBB.1 predominance

Abstract

Background

While numerous studies have evaluated the real-world performance of rapid antigen tests (RATs), data on the effect of Omicron sublineages such as XBB and reinfections on RAT performance is limited. We assessed the performance of RATs and factors associated with RAT-negative results among individuals who tested SARS-CoV-2-positive by reverse transcription-polymerase chain reaction (RT-PCR).

Methods

We conducted a retrospective study among Singapore residents who underwent testing for SARS-CoV-2 with RAT (Acon Flowflex or SD Biosensor) and RT-PCR in the same clinical encounter between 9 May 2022 and 21 November 2022. RT-PCR served as a reference standard for RAT performance. Logistic regression was used to estimate the odds ratios (OR) of factors associated with negative RAT results among RT-PCR-positive cases.

Results

Of 8,620 clinical encounters analysed, 3,519 (40.8%) were SARS-CoV-2-positive on RT-PCR. Overall sensitivity and specificity of RAT was 84.6% (95% CI 83.3–85.7%) and 99.4% (95% CI 99.1–99.6%) respectively. Acon Flowflex consistently achieved higher sensitivity and specificity than SD Biosensor test kit. Among RT-PCR-positive cases, individuals who had a previous documented SARS-CoV-2 infection, coinfection with another respiratory pathogen or tested ≥ 6 days from symptom onset had higher odds of testing RAT-negative, but the associations were attenuated after adjustment for cycle threshold values (proxy for viral load). There was no significant difference in RAT performance between Omicron sublineages BA.2, BA.5 and XBB.1.

Conclusion

Diagnostic performance of RAT was not affected by changes in predominant circulating Omicron sublineages. However, reinfection cases may be under ascertained by RAT. In individuals with a previous SARS-CoV-2 infection episode or symptom onset ≥ 6 days prior to testing, a confirmatory RT-PCR may be considered if there is high clinical suspicion.

Introduction

While the reverse transcription-polymerase chain reaction (RT-PCR) test remains the gold standard to detect SARS-CoV-2, it is time-consuming and costly, requiring trained laboratory personnel and dedicated facilities. Since the development of rapid antigen tests (RATs) for SARS-CoV-2 detection, they have served as an important diagnostic tool for COVID-19, being less costly and resource-intensive while allowing prompt detection to inform clinical management and public health decisions. In settings with ongoing community transmission, RATs may be used for diagnosis of COVID-19 among individuals who are symptomatic or had recent exposure to SARS-CoV-2. Self-testing with RATs can also be used for screening purposes among individuals without symptoms or known exposure to SARS-CoV-2, irrespective of the level of community transmission [1]. With RATs becoming ubiquitous, it is important that there are clear instructions and training of healthcare personnel in using RATs for valid sample collection and reliable results.

In Singapore, RATs have been the main testing modality for SARS-CoV-2 since early 2022 [2], and diagnosis of COVID-19 based on RAT is used to inform clinical management and other policy decisions such as easing of safe distancing measures and recommended timings of COVID-19 vaccinations [3, 4]. Furthermore, there has been increasing use of self-testing with RATs, with frequent distribution exercises of RAT kits by the government to each household [5]. It is thus important to monitor and evaluate the performance of RATs.

Numerous studies have evaluated the real-world performance of RATs in various settings, with one 2022 meta-analysis which included over 150 studies reporting the sensitivity and specificity among symptomatic patients to be 73.0% and 99.1% respectively [6]. Viral load has also been found to be the most important factor in determining RAT sensitivity, based on a systematic review which included 83 studies [7]. With the genesis of new SARS-CoV-2 variants with enhanced fitness driven by mutations in the spike protein responsible for target recognition and cellular entry [8], studies have also assessed RAT performance over time. A United Kingdom study which analysed data collected between November 2020 and March 2022 found that RATs were able to detect most SARS-CoV-2 infections throughout vaccine roll-out and across the pre-alpha or Alpha, Delta, and Omicron BA.1/BA.2 variants [9].

However, the landscape of COVID-19 has evolved rapidly, with the emergence of numerous Omicron subvariants including XBB and its descendent lineages, and a high rate of reinfections [10, 11]. A Japanese study which evaluated seven kinds of RATs found that all showed similar sensitivity to the Omicron subvariants BA.5, BA.2.75, BF.7, XBB.1, and BQ.1.1 [12]. Data on the effect of reinfection on the performance of RATs remain limited. A Czechia study reported that RAT sensitivity for Omicron cases with a previous infection was 79.2% (95% confidence interval [CI] 77.8–80.5%) compared with 81.9% (95% CI 81.3–82.5%) for those without a confirmed previous infection, but the study was conducted during the early Omicron period with data up to February 2022 [13].

In this study, we aimed to assess the performance of RATs compared with RT-PCR as a gold standard, and identify factors associated with negative RAT results among cases who tested SARS-CoV-2 positive by RT-PCR during the period of Omicron BA.2, BA.5 and XBB.1 predominance in Singapore.

Methods

Study design, setting, and participants

We conducted a retrospective study among individuals who were tested for SARS-CoV-2 using RAT and RT-PCR during the same clinical encounter between 9 May 2022 and 21 November 2022. Under the Acute Respiratory Infection (ARI) surveillance programme at 11 polyclinics in Singapore which served as sentinel outpatient primary care sites, individuals presenting with acute respiratory symptoms of cough, runny nose, sore throat and/or fever were randomly selected and offered to join the surveillance programme by their clinician. Individuals who were willing to participate then underwent testing with a healthcare-administered RAT and RT-PCR. We evaluated the two RAT kits predominantly used in the sentinel clinics, Flowflex SARS-CoV-2 Antigen Rapid Test (Acon Laboratories) and STANDARD Q COVID-19 Ag Home Test (SD Biosensor).

Specimens for RT-PCR were tested using the BioFire® Respiratory 2.1 (RP2.1) Panel (bioMérieux, France), a multiplex PCR which allows the simultaneous detection of multiple viral and bacterial respiratory organisms, including SARS-CoV-2 [14]. RT-PCR-positive cases were those which were SARS-CoV-2-positive on RP2.1 panel. The SARS-CoV-2-positive specimens were then tested using TaqPath™ COVID-19 Combo Kit (ThermoFisher, USA) and selected for whole genome sequencing (WGS) based on the cycle threshold (Ct) and S-gene target failure (SGTF) status. 40 PCR cycles were run as recommended by the manufacturer, and specimens with Ct value < 30 could be selected for WGS.

WGS was conducted on specimens using the ARTIC nCoV-2019 amplicon panel (Integrated DNA Technologies) and the Nextera XT DNA Library Preparation Kit (Illumina) on the Illumina MiSeq platform in accordance with the manufacturers’ instructions. The viral genome sequences were assembled from raw data by in-house pipelines and the viral lineages were determined by Pangolin.

Data sources and variables of interest

Data were collected from databases maintained by the National Public Health Laboratory and Ministry of Health, Singapore. Paired RAT and RT-PCR results for SARS-CoV-2, as well as age, sex, ethnicity, brand of RAT kit, presence of co-infection with other viral respiratory pathogen(s), clinic attended, vaccination status at time of infection, previous infection status, days between symptom onset and sample collection and SARS-CoV-2 lineage (if sequenced) were collected. Ct value was used as a proxy indicator of viral load, based on the lowest Ct value of the three targets in the TaqPath™ COVID-19 Combo Kit assay. Swab collection site for RT-PCR (nasopharyngeal, nasal, oropharyngeal and mid-turbinate, or throat) was recorded by healthcare staff and collected, but anatomical collection site of RAT was not known. Completion of the primary vaccination series was defined as two doses of Pfizer-BioNTech/Comirnaty or Moderna-Spikevax, three doses of Sinovac-CoronaVac or Sinopharm BBIBP-CorV, or two doses of non-mRNA vaccines approved under the World Health Organization (WHO) Emergency Use Listing besides Sinovac-CoronaVac and Sinopharm BBIBP-CorV [15]. Individuals who received additional vaccine doses after the primary vaccination series were considered boosted.

Those who had a SARS-CoV-2 infection notified to the Ministry of Health at least 90 days before the date of study inclusion were considered to have a previous documented infection [16]. In January 2022, Omicron overtook Delta as the predominant strain in Singapore and comprised over 91% of local cases which were sequenced [17]. Hence, individuals with a documented infection episode before 1 January 2022 were considered to have a previous pre-Omicron infection, while those with a previous documented infection from 1 January 2022 were assumed to have an Omicron infection. Individuals who tested RT-PCR-positive during the study period and did not have a previous documented infection were considered first infections.

Statistical analyses

RAT sensitivity, specificity, negative predictive value, and positive predictive value were calculated using BioFire RP2.1 Panel RT-PCR as the reference standard, with 95% CI calculated. Logistic regression models were used to estimate odds ratios (OR) of factors with negative RAT results among those who were SARS-CoV-2-positive by RT-PCR. Two multivariable logistic regressions were constructed. Model 1 adjusted for demographics, COVID-19 vaccination status, RAT brand, presence of co-infection with other respiratory pathogens, previous known SARS-CoV-2 infection, days between symptom onset and sample collection, SARS-CoV-2 lineage, PCR sample type and clinic visited, while Model 2 adjusted for factors in Model 1 and Ct value, to elucidate the effect of Ct values on the associations. Adjusted ORs of Models 1 and 2 were labelled aOR1 and aOR2 respectively. All data analysis was performed using Stata version 15.0 (StataCorp, College Station, TX, USA). A p-value of <0.05 was considered statistically significant.

Ethics approval

The study received ethics approval from the NHG Domain Specific Review Board (DSRB Ref: 2023/00131) with waiver of informed consent.

Results

Diagnostic performance of RAT for COVID-19

A total of 8,620 clinical encounters were analysed, comprising 3,519 (40.8%) SARS-CoV-2-positive and 5,101 (59.2%) SARS-CoV-2-negative samples by RT-PCR based on the BioFire RP2.1 Panel (Table 1). 2,976 of RT-PCR-positive cases tested positive on RAT, achieving an overall sensitivity of 84.6% (95% CI 83.3–85.7%), while 5,068 of RT-PCR-negative cases tested negative on RAT, giving a specificity of 99.4% (95% CI 99.1–99.6%). Positive and negative predictive values were 98.9% (95% CI 98.5–99.2%) and 90.3% (95% CI 89.5–91.1%) respectively.

Table 1 Diagnostic performance of rapid antigen tests

By brand of RAT kits, Acon Flowflex achieved a sensitivity of 85.9% (95% CI 84.4–87.2%) and specificity of 99.4% (95% CI 99.0–99.6%), while sensitivity and specificity of SD Biosensor were 81.6% (95% CI 79.1–83.9%) and 99.4% (95% CI 98.8–99.7%) respectively (Fig. 1). At Ct values ≤ 25, sensitivity was 92.7% (95% CI 91.7–93.6%) overall, 93.6% (95% CI 92.5–94.6%) for Acon Flowflex and 90.5% (95% CI 88.4–92.3%) for SD Biosensor. At Ct values > 25 to ≤ 30, sensitivity decreased to 53.1% (95% CI 47.0–59.1%) overall, 56.4% (95% CI 48.6–63.9%) for Acon Flowflex and 47.6% (95% CI 37.8–57.6%) for SD Biosensor. RAT sensitivity further decreased to 14.3% (95% CI 8.9–21.2%) overall at Ct values > 30.

Fig. 1

RAT sensitivity by cycle threshold (Ct) value & brand of RAT kit

Stratification by number of days between symptom onset and test collection showed that RAT sensitivity peaked at 88.4% (95% CI 86.2–90.4%) 2–3 days after symptom onset and declined to 58.2% (95% CI 45.5–70.2%) when the test was administered ≥ 6 days from symptom onset (Fig. 2).

Fig. 2

RAT sensitivity by days between symptom onset and test

Among those who were RT-PCR-positive, 320 (9.1%) had a previous pre-Omicron or Omicron infection. RAT sensitivity was 75.6% (95% CI 70.5–80.2%) for reinfection cases, compared with 85.5% (95% CI 84.2–86.7%) for first infections.

Factors associated with RAT-negative results among RT-PCR-positive cases

The odds of a negative RAT result increased substantially with increasing Ct values. Compared with Ct < 20, Ct > 20 and ≤ 25 had 3.26 times (95% CI 2.38–4.46) higher odds of producing a negative RAT result, and Ct > 25 to ≤ 30 had an adjusted OR of 16.11 (95% CI 11.15–23.28) (Table 2).

Table 2 Association of factors with negative antigen rapid tests among RT-PCR-positive cases

The predominant circulating SARS-CoV-2 variants during the study period were Omicron BA.2, BA.5 and XBB.1. On logistic regression with Omicron BA.2 as the reference group, there was no significant association between the various Omicron sublineages and discordant RAT results among RT-PCR-positive cases (Table 2).

Compared with Acon Flowflex test kit, SD Biosensor test kit was associated with higher odds of a negative RAT result, even after adjustment for all covariates (aOR2 1.58 [95% CI 1.15–2.15]). Individuals who received booster doses had higher odds of testing RAT-negative (aOR2 3.85 [95% CI 1.74–8.52]) than those who only completed the primary vaccination series.

Having a previous documented SARS-CoV-2 infection with pre-Omicron variant (aOR1 3.46 [95% CI 2.14–5.59]) or Omicron variant (aOR1 2.14 [95% CI 1.37–3.32]), or a coinfection with another respiratory pathogen (aOR1 2.03 [95% CI 1.34–3.07] were significantly associated with a negative RAT result on adjustment for covariates except Ct value (model 1). However, these associations were substantially attenuated and no longer statistically significant after further adjustment for Ct value (model 2).

Compared with testing within 1 day of symptom onset, individuals who were tested 6 days or more from symptom onset had higher odds of testing RAT-negative (aOR1 4.02 [95% CI 2.29–7.08], while those tested 2–3 days after symptom onset had lower odds of testing RAT-negative (aOR1 0.67 [95% CI 0.52–0.86]. These associations were also attenuated after adjusting for Ct value, with only ≥ 6 days between symptom onset and test administration remaining significant (aOR2 2.25 [95% CI 1.09–4.63]).

Discussion

While both Acon Flowflex and SD Biosensor achieved the minimum criteria set by the WHO of ≥ 80% sensitivity and ≥ 97% specificity, we found that Acon Flowflex consistently performed better than SD Biosensor, consistent with similar previous studies [18, 19]. Our sensitivity estimates of 93.6% for Acon Flowflex and 90.5% for SD Biosensor at Ct values ≤ 25 were comparable with a German study evaluating diagnostic tests using a common panel of SARS-CoV-2 specimens, which reported sensitivities of 94.1% and 88.9% for Acon Flowflex and SD Biosensor respectively [18]. With increasing Ct values, there was a substantial decrease in RAT sensitivity and increase in odds of testing RAT-negative among RT-PCR-positive cases, consistent with systematic reviews that viral load is the most important factor influencing sensitivity [7, 20].

RAT performance did not significantly change between Omicron sublineages BA.2, BA.5 and XBB.1. This is aligned with current knowledge given that RATs target the nucleocapsid protein [21], while majority of mutations in SARS-CoV-2 variants of concern (VOC) occur in the spike protein [22]. These findings reassure us that the performance of RAT is not affected by the emergence of new variants and sublineages.

We found a marked decline in RAT sensitivity when the test was administered ≥ 6 days from symptom onset, and this association was mainly mediated by higher Ct values suggestive of lower viral load. This was congruent with a Brazilian study which found that greatest sensitivity of the antigen test was observed when the test was performed within 5 days of symptom onset [23], suggestive that viral load markedly decreases 5 days after symptom onset.

We also observed that individuals who were tested 2–3 days after symptom onset had lower odds of having a false negative RAT result compared with those tested within 1 day of symptom onset. A study conducted in Switzerland similarly found that sensitivity of the rapid test peaked at 2 days post onset of symptoms [24]. We further showed that the association was mediated by viral loads (using Ct value as a proxy marker), aligned with previous studies which reported peak viral loads in upper respiratory specimens around 3 days from symptom onset [25, 26]. Individuals with an initial negative RAT result may consider repeating the test 1–2 days after symptom onset or obtaining a RT-PCR test.

Of note, RAT sensitivity was 75.6% for known reinfection cases, around 10% less compared to first infections and below the WHO minimum criteria. Our results also showed that the association between having a previous pre-Omicron or Omicron infection and a negative RAT result was similarly mediated by viral loads. This is consistent with previous studies which found that infection-induced immunity was associated with shorter duration of viral shedding and lower viral loads [27]. In settings where RAT is the main testing modality, there may be under-detection of reinfection cases. This is relevant if clinical decisions such as commencement of oral antiviral treatment are made based on RAT results, and a confirmatory RT-PCR may be considered if there is high clinical suspicion. Under-ascertainment of reinfection cases from reduced RAT sensitivity should also be considered in the formulation of public health policies and COVID-19 control measures. Furthermore, RAT sensitivity is likely to decline as the seroprevalence among the population and proportion of reinfection cases continue to increase, thus the performance of RATs should be continuously evaluated.

Individuals who received booster doses were found to have higher odds of testing RAT-negative than those who only completed the primary vaccine series, even after adjustment for viral loads. A study conducted in Germany also observed that the antigen test was potentially less sensitive for samples with medium (20–30) Ct values in vaccinated than unvaccinated cases [28]. This may suggest differences in RAT performance by vaccination status, and further studies will be required to evaluate this association.

The strengths of this study include the use of comprehensive data collected under a surveillance programme, and paired sample collections in the same clinical encounter. However, there are also several limitations. First, we did not have data on the anatomical collection site for RAT specimen, which has been found to be a factor affecting RAT sensitivity [7]. RAT performance could also be affected by variable swab techniques between different healthcare workers, although we attempted to mitigate this by adjusting for the clinic which the patient had presented to in the logistic regression models. Data on the comorbidities of participants were also not available, hence we were unable to assess the effect of factors such as immunosuppression which may affect RAT performance. Second, the classification of previous infection status may not be precise due to under-ascertainment of mild or asymptomatic cases who did not present to healthcare facilities. Third, Ct value was used as a proxy indicator for viral load but is not a direct marker as it is affected by factors related to sample collection and processing and performance of the RT-PCR assay [29]. To mitigate this, all RT-PCR tests in this study were conducted in a single laboratory with the same assay, and specimen type was included as a covariate in the multivariable logistic regression model. Fourth, as the study only included symptomatic individuals who were selected for ARI surveillance, the results may not be generalisable to asymptomatic cases.

The results of our study can help to inform testing strategies to increase RAT sensitivity among symptomatic patients, such as the choice of test kits and testing within 5 days of symptom onset. While RAT performance did not significantly change between Omicron sublineages BA.2, BA.5 and XBB.1, we highlight that reinfections are less likely to be picked up on RAT, which can contribute to under-ascertainment of reinfection cases. The diagnostic performance of RATs may thus be periodically evaluated as the proportion of reinfection cases increases and the SARS-CoV-2 virus continues to mutate over time.