Advertisement

Saliva as a diagnostic specimen for testing respiratory virus by a point-of-care molecular assay: a diagnostic validity study

  • Author Footnotes
    † K.K.W. To and C.C.Y. Yip contributed equally to this study.
    K.K.W. To
    Footnotes
    † K.K.W. To and C.C.Y. Yip contributed equally to this study.
    Affiliations
    Department of Microbiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pokfulam, Hong Kong, China

    Department of Microbiology, Queen Mary Hospital, Pokfulam, Hong Kong, China

    State Key Laboratory for Emerging Infectious Diseases, The University of Hong Kong, Pokfulam, Hong Kong, China

    Carol Yu Centre for Infection, The University of Hong Kong, Hong Kong, China

    Department of Clinical Microbiology and Infection Control, The University of Hong Kong-Shenzhen Hospital, Shenzhen, China
    Search for articles by this author
  • Author Footnotes
    † K.K.W. To and C.C.Y. Yip contributed equally to this study.
    C.C.Y. Yip
    Footnotes
    † K.K.W. To and C.C.Y. Yip contributed equally to this study.
    Affiliations
    Department of Microbiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pokfulam, Hong Kong, China

    Department of Microbiology, Queen Mary Hospital, Pokfulam, Hong Kong, China
    Search for articles by this author
  • C.Y.W. Lai
    Affiliations
    Department of Microbiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pokfulam, Hong Kong, China

    Department of Microbiology, Queen Mary Hospital, Pokfulam, Hong Kong, China
    Search for articles by this author
  • C.K.H. Wong
    Affiliations
    Department of Family Medicine and Primary Care, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong, China
    Search for articles by this author
  • D.T.Y. Ho
    Affiliations
    Department of Microbiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pokfulam, Hong Kong, China
    Search for articles by this author
  • P.K.P. Pang
    Affiliations
    Department of Microbiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pokfulam, Hong Kong, China
    Search for articles by this author
  • A.C.K. Ng
    Affiliations
    Department of Microbiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pokfulam, Hong Kong, China
    Search for articles by this author
  • K.-H. Leung
    Affiliations
    Department of Microbiology, Queen Mary Hospital, Pokfulam, Hong Kong, China
    Search for articles by this author
  • R.W.S. Poon
    Affiliations
    Department of Microbiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pokfulam, Hong Kong, China

    Department of Microbiology, Queen Mary Hospital, Pokfulam, Hong Kong, China
    Search for articles by this author
  • K.-H. Chan
    Affiliations
    Department of Microbiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pokfulam, Hong Kong, China
    Search for articles by this author
  • V.C.C. Cheng
    Affiliations
    Department of Microbiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pokfulam, Hong Kong, China

    Department of Microbiology, Queen Mary Hospital, Pokfulam, Hong Kong, China
    Search for articles by this author
  • I.F.N. Hung
    Affiliations
    Carol Yu Centre for Infection, The University of Hong Kong, Hong Kong, China

    Department of Clinical Microbiology and Infection Control, The University of Hong Kong-Shenzhen Hospital, Shenzhen, China

    Department of Medicine, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong, China
    Search for articles by this author
  • K.-Y. Yuen
    Correspondence
    Corresponding author. K-Y. Yuen, Carol Yu Centre for Infection, Department of Microbiology, The University of Hong Kong, 19th Floor, Block T, Queen Mary Hospital, Pokfulam Road, Hong Kong, China.
    Affiliations
    Department of Microbiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pokfulam, Hong Kong, China

    Department of Microbiology, Queen Mary Hospital, Pokfulam, Hong Kong, China

    State Key Laboratory for Emerging Infectious Diseases, The University of Hong Kong, Pokfulam, Hong Kong, China

    Carol Yu Centre for Infection, The University of Hong Kong, Hong Kong, China

    Department of Clinical Microbiology and Infection Control, The University of Hong Kong-Shenzhen Hospital, Shenzhen, China
    Search for articles by this author
  • Author Footnotes
    † K.K.W. To and C.C.Y. Yip contributed equally to this study.
Open ArchivePublished:June 12, 2018DOI:https://doi.org/10.1016/j.cmi.2018.06.009

      Abstract

      Objectives

      Automated point-of-care molecular assays have greatly shortened the turnaround time of respiratory virus testing. One of the major bottlenecks now lies at the specimen collection step, especially in a busy clinical setting. Saliva is a convenient specimen type that can be provided easily by adult patients. This study assessed the diagnostic validity, specimen collection time and cost associated with the use of saliva.

      Methods

      This was a prospective diagnostic validity study comparing the detection rate of respiratory viruses between saliva and nasopharyngeal aspirate (NPA) among adult hospitalized patients using Xpert® Xpress Flu/RSV. The cost and time associated with the collection of saliva and nasopharyngeal specimens were also estimated.

      Results

      Between July and October 2017, 214 patients were recruited. The overall agreement between saliva and NPA was 93.3% (196/210, κ 0.851, 95% CI 0.776–0.926). There was no significant difference in the detection rate of respiratory viruses between saliva and NPA (32.9% (69/210) versus 35.7% (75/210); p 0.146). The overall sensitivity and specificity were 90.8% (81.9%–96.2%) and 100% (97.3%–100%), respectively, for saliva, and were 96.1% (88.9%–99.2%) and 98.5% (94.7%–99.8%), respectively, for NPA. The time and cost associated with the collection of saliva were 2.26-fold and 2.59-fold lower, respectively, than those of NPA.

      Conclusions

      Saliva specimens have high sensitivity and specificity in the detection of respiratory viruses by an automated multiplex Clinical Laboratory Improvement Amendments-waived point-of-care molecular assay when compared with those of NPA. The use of saliva also reduces the time and cost associated with specimen collection.

      Keywords

      Introduction

      Respiratory tract infection is frequently caused by respiratory viruses [
      • Jain S.
      • Self W.H.
      • Wunderink R.G.
      • Fakhran S.
      • Balk R.
      • Bramley A.M.
      • et al.
      Community-acquired pneumonia requiring hospitalization among U.S. adults.
      ,
      • To K.K.
      • Lau S.K.
      • Chan K.H.
      • Mok K.Y.
      • Luk H.K.
      • Yip C.C.
      • et al.
      Pulmonary and extrapulmonary complications of human rhinovirus infection in critically ill patients.
      ]. Rapid and accurate detection of respiratory viruses is important in guiding antimicrobial treatment and infection control precautions to improve patient outcome and to prevent nosocomial transmission, respectively [
      • Semret M.
      • Schiller I.
      • Jardin B.A.
      • Frenette C.
      • Loo V.G.
      • Papenburg J.
      • et al.
      Multiplex respiratory virus testing for antimicrobial stewardship: a prospective assessment of antimicrobial use and clinical outcomes among hospitalized adults.
      ,
      • Muthuri S.G.
      • Venkatesan S.
      • Myles P.R.
      • Leonardi-Bee J.
      • Al Khuwaitir T.S.
      • Al Mamun A.
      • et al.
      Effectiveness of neuraminidase inhibitors in reducing mortality in patients admitted to hospital with influenza A H1N1pdm09 virus infection: a meta-analysis of individual participant data.
      ]. The World Health Organization has included the diagnosis of and diagnostic tests for severe acute respiratory infections as a key research agenda [
      • World Health Organization
      Research needs for the battle against respiratory viruses (BRaVe).
      ].
      Automated multiplex molecular assay has significantly improved respiratory virus testing through its simplicity, short turnaround time and high accuracy [
      • Merckx J.
      • Wali R.
      • Schiller I.
      • Caya C.
      • Gore G.C.
      • Chartrand C.
      • et al.
      Diagnostic accuracy of novel and traditional rapid tests for influenza infection compared with reverse transcriptase polymerase chain reaction: a systematic review and meta-analysis.
      ,
      • Huang H.S.
      • Tsai C.L.
      • Chang J.
      • Hsu T.C.
      • Lin S.
      • Lee C.C.
      Multiplex PCR system for the rapid diagnosis of respiratory virus infection: systematic review and meta-analysis.
      ,
      • Brendish N.J.
      • Malachira A.K.
      • Armstrong L.
      • Houghton R.
      • Aitken S.
      • Nyimbili E.
      • et al.
      Routine molecular point-of-care testing for respiratory viruses in adults presenting to hospital with acute respiratory illness (ResPOC): a pragmatic, open-label, randomised controlled trial.
      ,
      • Chan K.H.
      • To K.K.W.
      • Li P.T.W.
      • Wong T.L.
      • Zhang R.
      • Chik K.K.H.
      • et al.
      Evaluation of NxTAG respiratory pathogen panel and comparison with xTAG respiratory viral panel fast v2 and film array respiratory panel for detecting respiratory pathogens in nasopharyngeal aspirates and swine/avian-origin influenza a subtypes in culture isolates.
      ], and has been shown to lower the risk of hospital admission, reduce length of hospital stay, optimize the use of antivirals, shorten duration of antimicrobial treatment and reduce radiological investigations [
      • Rappo U.
      • Schuetz A.N.
      • Jenkins S.G.
      • Calfee D.P.
      • Walsh T.J.
      • Wells M.T.
      • et al.
      Impact of early detection of respiratory viruses by multiplex PCR assay on clinical outcomes in adult patients.
      ,
      • Busson L.
      • Mahadeb B.
      • De Foor M.
      • Vandenberg O.
      • Hallin M.
      Contribution of a rapid influenza diagnostic test to manage hospitalized patients with suspected influenza.
      ,
      • Mitchell S.L.
      • Chang Y.C.
      • Feemster K.
      • Cardenas A.M.
      Implementation of a rapid influenza A/B and RSV direct molecular assay improves emergency department oseltamivir use in paediatric patients.
      ]. Despite the favourable turn-around-time and accuracy of these automated assays, significant delay occurs in real-life clinical practice due to the delay in specimen collection, transportation and testing.
      The delay in specimen collection is often due to the waiting time for healthcare workers to collect nasopharyngeal aspirate (NPA) or nasopharyngeal swab (NPS), which are currently the recommended specimen types [

      U.S. Food and Drug Administration. List of microbial tests. Available at: https://www.fda.gov/MedicalDevices/ProductsandMedicalProcedures/InVitroDiagnostics/ucm330711.htm#microbial. [Accessed 2 March 2018].

      ]. The collection of nasopharyngeal specimens also causes significant discomfort to patients and is associated with infection control risk to healthcare workers [
      • Frazee B.W.
      • Rodriguez-Hoces de la Guardia A.
      • Alter H.
      • Chen C.G.
      • Fuentes E.L.
      • Holzer A.K.
      • et al.
      Accuracy and discomfort of different types of intranasal specimen collection methods for molecular influenza testing in emergency department patients.
      ,

      Centers for Disease Control and Prevention. Influenza specimen collection desk reference guide. Available at: https://www.cdc.gov/flu/pdf/freeresources/healthcare/flu-specimen-collection-guide.pdf. [Accessed 22 February 2018].

      ]. Despite a lower sensitivity than nasopharyngeal specimens, nasal and throat swabs are also specimen types recommended by clinical guidelines [
      • Baron E.J.
      • Miller J.M.
      • Weinstein M.P.
      • Richter S.S.
      • Gilligan P.H.
      • Thomson Jr., R.B.
      • et al.
      A guide to utilization of the microbiology laboratory for diagnosis of infectious diseases: 2013 recommendations by the Infectious Diseases Society of America (IDSA) and the American Society for Microbiology (ASM)(a).
      ,
      • Lambert S.B.
      • Whiley D.M.
      • O'Neill N.T.
      • Andrews E.C.
      • Canavan F.M.
      • Bletchly C.
      • et al.
      Comparing nose-throat swabs and nasopharyngeal aspirates collected from children with symptoms for respiratory virus identification using real-time polymerase chain reaction.
      ,
      • Meerhoff T.J.
      • Houben M.L.
      • Coenjaerts F.E.
      • Kimpen J.L.
      • Hofland R.W.
      • Schellevis F.
      • et al.
      Detection of multiple respiratory pathogens during primary respiratory infection: nasal swab versus nasopharyngeal aspirate using real-time polymerase chain reaction.
      ]. The collection of nasal or throat specimens, though less invasive than nasopharyngeal specimens, is still uncomfortable and healthcare workers are required for their collection [
      • Frazee B.W.
      • Rodriguez-Hoces de la Guardia A.
      • Alter H.
      • Chen C.G.
      • Fuentes E.L.
      • Holzer A.K.
      • et al.
      Accuracy and discomfort of different types of intranasal specimen collection methods for molecular influenza testing in emergency department patients.
      ].
      Saliva has been proposed as an alternative specimen type in the diagnosis of respiratory virus infection [
      • To K.K.
      • Lu L.
      • Yip C.C.
      • Poon R.W.
      • Fung A.M.
      • Cheng A.
      • et al.
      Additional molecular testing of saliva specimens improves the detection of respiratory viruses.
      ,
      • Kim Y.G.
      • Yun S.G.
      • Kim M.Y.
      • Park K.
      • Cho C.H.
      • Yoon S.Y.
      • et al.
      Comparison between saliva and nasopharyngeal swab specimens for detection of respiratory viruses by multiplex reverse transcription-PCR.
      ,
      • Robinson J.L.
      • Lee B.E.
      • Kothapalli S.
      • Craig W.R.
      • Fox J.D.
      Use of throat swab or saliva specimens for detection of respiratory viruses in children.
      ,
      • Yoon J.
      • Yun S.G.
      • Nam J.
      • Choi S.H.
      • Lim C.S.
      The use of saliva specimens for detection of influenza A and B viruses by rapid influenza diagnostic tests.
      ]. Saliva can be provided easily by the patient. The use of saliva instead of nasopharyngeal specimens would avoid patient discomfort, reduce the risk of nosocomial transmission to healthcare workers and other patients, and shorten the time to diagnosis. Kim et al. showed that the detection rate was comparable between NPS and saliva (77.5% versus 76.3%) among military recruits [
      • Kim Y.G.
      • Yun S.G.
      • Kim M.Y.
      • Park K.
      • Cho C.H.
      • Yoon S.Y.
      • et al.
      Comparison between saliva and nasopharyngeal swab specimens for detection of respiratory viruses by multiplex reverse transcription-PCR.
      ]. In our previous study, 92% of patients with respiratory virus detected in their NPA also had the same virus detected in their saliva [
      • To K.K.
      • Lu L.
      • Yip C.C.
      • Poon R.W.
      • Fung A.M.
      • Cheng A.
      • et al.
      Additional molecular testing of saliva specimens improves the detection of respiratory viruses.
      ].
      Despite these advantages, saliva is not used widely in clinical practice. The major hurdle is that saliva is not a recommended specimen type for respiratory virus testing in approved rapid molecular assay platforms [
      • Merckx J.
      • Wali R.
      • Schiller I.
      • Caya C.
      • Gore G.C.
      • Chartrand C.
      • et al.
      Diagnostic accuracy of novel and traditional rapid tests for influenza infection compared with reverse transcriptase polymerase chain reaction: a systematic review and meta-analysis.
      ,
      • Huang H.S.
      • Tsai C.L.
      • Chang J.
      • Hsu T.C.
      • Lin S.
      • Lee C.C.
      Multiplex PCR system for the rapid diagnosis of respiratory virus infection: systematic review and meta-analysis.
      ]. Furthermore, the diagnostic validity of saliva testing by automated molecular assays has not been reported. In this study, we compared the diagnostic validity of saliva and NPA in the detection of respiratory viruses using an automated multiplex molecular assay, which has recently received the Clinical Laboratory Improvement Amendments waiver. We also estimated the reduction in cost and time associated with the collection of saliva when compared with those of nasopharyngeal specimens.

      Materials and methods

       Study design, participants and setting

      This was a prospective diagnostic validity study conducted in Queen Mary Hospital, a teaching hospital in Hong Kong with 1600 beds. Written informed consent was obtained from study participants. Research nurses screened patients for eligibility from Monday to Friday except for holidays during the study period. Saliva and NPA specimens from adult hospitalized patients with respiratory tract infection were tested for influenza A virus, influenza B virus and respiratory syncytial virus (RSV) using Xpert® Xpress Flu/RSV assay (GeneXpert System, Cepheid, Sunnyvale, CA, USA). Monoplex RT-PCR was performed when there were discrepant results between saliva and NPA in the Xpress Flu/RSV assay [
      • To K.K.
      • Lu L.
      • Yip C.C.
      • Poon R.W.
      • Fung A.M.
      • Cheng A.
      • et al.
      Additional molecular testing of saliva specimens improves the detection of respiratory viruses.
      ]. All amplification curves were inspected manually. Cycle threshold (Ct) values from the Xpress Flu/RSV assay were used as a surrogate for viral load.
      Data collection was planned before patients were recruited. The results of the molecular diagnostic assays were not available to the attending physicians. Clinical information was not available to the performers of the laboratory tests. Patients were considered to have severe disease if they required oxygen supplementation, were admitted into the intensive care unit or coronary care unit, or died, as we described previously [
      • To K.K.
      • Hung I.F.
      • Li I.W.
      • Lee K.L.
      • Koo C.K.
      • Yan W.W.
      • et al.
      Delayed clearance of viral load and marked cytokine activation in severe cases of pandemic H1N1 2009 influenza virus infection.
      ,
      • To K.K.W.
      • Lu L.
      • Fong C.H.Y.
      • Wu A.K.L.
      • Mok K.Y.
      • Yip C.C.Y.
      • et al.
      Rhinovirus respiratory tract infection in hospitalized adult patients is associated with TH2 response irrespective of asthma.
      ,
      • To K.K.
      • Zhou J.
      • Song Y.Q.
      • Hung I.F.
      • Ip W.C.
      • Cheng Z.S.
      • et al.
      Surfactant protein B gene polymorphism is associated with severe influenza.
      ]. This study has been approved by the Institutional Review Board of the University of Hong Kong/Hospital Authority Hong Kong West Cluster (UW 17-266). Please refer to supplementary methods (Appendix S1) for details on the inclusion and exclusion criteria, Xpress Flu/RSV assay, monoplex RT-PCR, and specimen and data collection.

       Specimen collection time and cost

      To estimate the time required for the collection of saliva, NPS and NPA, we have recorded the specimen collection time used by three different healthcare workers in collecting each of the specimen types. For cost analysis, resource use and associated costs, including staff costs, accrued to the hospital were taken into consideration, adopting a hospital setting perspective (see Supplementary material, Appendix S1).

       Statistical analysis

      The detection rate of paired saliva and NPA specimens was compared using McNemar's test. Agreement between saliva and NPA was performed using κ statistics [
      • Landis J.R.
      • Koch G.G.
      The measurement of observer agreement for categorical data.
      ]. For sensitivity and specificity, the reference standard for a positive result was either concordant result between saliva and NPA, or discordant result, but either saliva and NPA tested positive for the same respiratory virus by monoplex RT-PCR. Ct values were compared using Wilcoxon matched-paired signed rank test. The correlation of Ct values between saliva and NPA was assessed using Spearman correlation coefficient. A p value of <0.05 was considered statistically significant. All statistical analysis was performed using GraphPad PRISM® version 6.07 or SPSS 23.0.

      Results

       Patients' characteristics

      Between 10 July and 6 October 2017, a total of 732 patients were screened, and 214 patients were eligible for the study (Fig. 1, and Supplementary material, Table S1). The median age was 71 years, and 55.6% (119/214) were women. The most common symptoms were cough and shortness of breath (79%; 169/214). Hypertension was the most common underlying disease (54.2%; 116/214), followed by chronic heart disease (33.2%; 71/214) and diabetes mellitus (33.2%; 71/214). Fifty-nine patients (27.6%) had severe disease and required oxygen supplementation.

       Comparison of respiratory virus detection between saliva and NPA

      Among the 214 eligible patients, 210 had valid results in their NPA and saliva specimens by Xpress Flu/RSV assay (Fig. 2). There was overall high agreement between saliva and NPA (93.3%; 196/210, κ 0.851, 95% CI 0.776–0.926). Among patients with concordant results, 32.7% (64/196) had the same respiratory virus detected in both saliva and NPA, including 21.9% (43/196) with influenza A virus, 1.5% (3/196) with influenza B virus, and 9.2% (18/196) with RSV. The detection rate of respiratory viruses in saliva was lower than that of NPA, but not reaching statistical significance (32.9%; 69/210 versus 35.7%; 75/210; p 0.146).
      Fig. 2
      Fig. 2Summary of saliva and nasopharyngeal aspirate results of Xpress® Flu/RSV in 214 patients.
      Fourteen patients had discordant results between saliva and NPA by Xpress Flu/RSV assay (Fig. 2 and Table 1), including three patients with virus detected in saliva but not in NPA (Patients 1–3 in Table 1), nine patients with virus detected in NPA but not in saliva (Patients 4–12 in Table 1), and two patients with multiple viruses in their NPA but only one virus detected in their saliva (Patients 13 and 14 in Table 1). For Patients 1–3, the viruses detected by Xpress Flu/RSV assay in their saliva were also detected by monoplex RT-qPCR for all three patients. The saliva testing results could have changed patient's antiviral treatment or infection control precaution measures. For Patients 4–12, the viruses detected in their NPA were also detected by monoplex RT-qPCR in seven patients. For Patient 12, for whom Xpress Flu/RSV assay detected RSV in NPA but not in saliva, monoplex RT-qPCR detected RSV in both NPA and saliva. For Patients 13 and 14, only the virus with the lowest Ct value (highest viral load) in the NPA specimen was detected in their saliva specimen by Xpress Flu/RSV assay.
      Table 1Patients with discordant results between saliva and nasopharyngeal aspirate by Xpress Flu/RSV assay
      Patient no.Gender/AgeXpress Flu/RSV assayMonoplex RT-PCRUnderlying diseaseDiagnosisPotential impact of testing result on patient treatment and infection control precautions
      NPASalivaNPASaliva
      Only saliva tested positive by Xpress Flu/RSV
       1M/61NDInfluenza A virusND
      Monoplex RT-PCR for influenza A virus was performed.
      Influenza A virusHypopituitarism, gallstonesStreptococcus pneumoniae pneumonia and meningitisTreatment with neuraminidase inhibitor
       2F/61NDRSVND
      Monoplex RT-PCR for RSV was performed.
      RSVIschaemic heart disease, diabetes mellitus, hypertension, hyperlipidaemiaAcute coronary syndromeNil
       3F/68NDRSVND
      Monoplex RT-PCR for RSV was performed.
      RSVAcute myeloid leukaemiaNeutropenic feverContact precaution; treatment with ribavirin
      Only NPA tested positive by Xpress Flu/RSV
       4F/91Influenza A virusNDInfluenza A virusND
      Monoplex RT-PCR for influenza A virus was performed.
      Ischaemic heart disease, peripheral vascular disease, renal stonesPneumoniaTreatment with neuraminidase inhibitor
       5M/82Influenza A virusNDInfluenza A virusND
      Monoplex RT-PCR for influenza A virus was performed.
      Renal stonesPneumoniaTreatment with neuraminidase inhibitor
       6M/78Influenza A virusNDND
      Monoplex RT-PCR for influenza A virus was performed.
      ND
      Monoplex RT-PCR for influenza A virus was performed.
      COPDCOPD exacerbationTreatment with neuraminidase inhibitor
       7F/81Influenza A virusNDInfluenza A virusND
      Monoplex RT-PCR for influenza A virus was performed.
      Hypertension, diabetes mellitus, hyperlipidaemia, strokeUpper respiratory tract infectionTreatment with neuraminidase inhibitor
       8M/53Influenza A virusNDInfluenza A virusND
      Monoplex RT-PCR for influenza A virus was performed.
      Good past healthNonarteritic anterior ischaemic optic neuropathyTreatment with neuraminidase inhibitor
       9F/71Influenza B virusNDInfluenza B virusND
      Monoplex RT-PCR for influenza B virus was performed.
      Hypertension, hyperlipidaemiaUpper respiratory tract infectionTreatment with neuraminidase inhibitor
       10F/60RSVNDND
      Monoplex RT-PCR for RSV was performed.
      ND
      Monoplex RT-PCR for RSV was performed.
      Colonic polypUpper respiratory tract infectionNil
       11M/66RSVNDRSVND
      Monoplex RT-PCR for RSV was performed.
      Diabetes mellitus, SchizophreniaUpper respiratory tract infectionNil
       12F/91RSVNDRSV
      The viral load in the NPA and saliva using monoplex qRT-PCR was 9.97 × 105 copies/reaction and 26 copies/reaction, respectively.
      RSV
      The viral load in the NPA and saliva using monoplex qRT-PCR was 9.97 × 105 copies/reaction and 26 copies/reaction, respectively.
      Hypertension, atrial flutterPneumoniaNil
      Co-infection detected by Xpress Flu/RSV
       13F/70Influenza A virus
      The Ct values are: influenza A virus A 1: 27.4; influenza A virus A 2: 28.6; influenza B virus: 33.6; RSV: 35.3.


      Influenza B virus
      The Ct values are: influenza A virus A 1: 27.4; influenza A virus A 2: 28.6; influenza B virus: 33.6; RSV: 35.3.


      RSV
      The Ct values are: influenza A virus A 1: 27.4; influenza A virus A 2: 28.6; influenza B virus: 33.6; RSV: 35.3.
      Influenza A virusInfluenza A virus
      Influenza B virus and RSV were not detected by monoplex RT-PCR.
      Influenza A virus
      Influenza B virus and RSV were not detected by monoplex RT-PCR.
      HypertensionUpper respiratory tract infection with vasovagal attackTreatment with neuraminidase inhibitor
       14M/81Influenza A virus
      The Ct values are: influenza A virus A1: 35.6; influenza A virus A 2: 39; RSV: 15.6.


      RSV
      The Ct values are: influenza A virus A1: 35.6; influenza A virus A 2: 39; RSV: 15.6.
      RSVRSV
      Influenza A virus and influenza B virus were not detected by monoplex RT-PCR.
      RSV
      Influenza A virus and influenza B virus were not detected by monoplex RT-PCR.
      Chronic rheumatic heart disease, atrial fibrillationAcute bronchitisNil
      Abbreviations: COPD, chronic obstructive pulmonary disease; F, female; M, male; ND, not detected; NPA, nasopharyngeal aspirate; RSV, respiratory syncytial virus.
      a Monoplex RT-PCR for influenza A virus was performed.
      b Monoplex RT-PCR for influenza B virus was performed.
      c Monoplex RT-PCR for RSV was performed.
      d The viral load in the NPA and saliva using monoplex qRT-PCR was 9.97 × 105 copies/reaction and 26 copies/reaction, respectively.
      e The Ct values are: influenza A virus A 1: 27.4; influenza A virus A 2: 28.6; influenza B virus: 33.6; RSV: 35.3.
      f The Ct values are: influenza A virus A1: 35.6; influenza A virus A 2: 39; RSV: 15.6.
      g Influenza B virus and RSV were not detected by monoplex RT-PCR.
      h Influenza A virus and influenza B virus were not detected by monoplex RT-PCR.
      We have further analysed the four patients with invalid/error results (see Supplementary material, Table S2). For the two patients with error results in saliva specimens (Patients 1 and 2 in Table S2), RSV was detected in the NPA specimens of both patients by Xpress Flu/RSV assay, and RSV was also detected in the saliva by monoplex RT-qPCR. For patient 1 in Table S2, manual inspection of the amplification curve of Xpress Flu/RSV assay showed that RSV RNA was amplified in saliva, indicating successful detection of RSV despite the machine indicating an error result. For the two patients with invalid results in their NPA specimens (Patients 3 and 4 in Table S2), respiratory virus was not detected by Xpress Flu/RSV assay in their saliva specimens; and monoplex RT-PCR for influenza A virus, influenza B virus and RSV were all negative in both of their NPA and saliva specimens.

       Analysis of sensitivity and specificity

      Excluding the four patients with either invalid results or more than one virus detected in their NPA, the overall sensitivity and specificity for NPA was 96.1% (95% CI 88.9%–99.2%) and 98.5% (95% CI 94.7%–99.8%), respectively (Table 2). Excluding the two patients with error results in saliva, the overall sensitivity and specificity for saliva was 90.8% (95% CI 81.9%–96.2%) and 100% (95% CI 97.3%–100%), respectively.
      Table 2Results of Xpress Flu/RSV assay for saliva and nasopharyngeal aspirate when compared with patients' infection status
      Xpress Flu/RSV resultPatient with infection by A, B or RSVPatient without infection by A, B or RSVTotalSensitivity (95% CI) (%)Specificity (95% CI) (%)
      NPAInfluenza A virusPositive4825098.0 (89.1–99.9)98.8 (95.6–99.9)
      Negative1161162
      Total49163212
      Excluded two patients with invalid results from Xpert Xpress Flu/RSV assay in their NPA.
      Influenza B virusPositive415100 (39.8–100)99.5 (97.4–100)
      Negative0207207
      Total4208212
      Excluded two patients with invalid results from Xpert Xpress Flu/RSV assay in their NPA.
      RSVPositive2322592.0 (74.0–99.0)98.9 (96.2–99.9)
      Negative2185187
      Total25187212
      Excluded two patients with invalid results from Xpert Xpress Flu/RSV assay in their NPA.
      TotalPositive7327596.1 (88.9–99.2)98.5 (94.7–99.8)
      Negative3132135
      Total76134210
      Excluded two patients with invalid results and two patients with more than one virus detected by Xpert Xpress Flu/RSV assay in their NPA.
      SalivaInfluenza A virusPositive4504591.8 (80.4–97.7)100 (97.8–100)
      Negative4163167
      Total49163212
      Excluded two patients with error results from Xpert Xpress Flu/RSV assay in their saliva.
      Influenza B virusPositive30375.0 (19.4–99.4)100 (98.2–100)
      Negative1208209
      Total4208212
      Excluded two patients with error results from Xpert Xpress Flu/RSV assay in their saliva.
      RSVPositive2102191.3 (72.0–98.9)100 (98.1–100)
      Negative2189191
      Total23189212
      Excluded two patients with error results from Xpert Xpress Flu/RSV assay in their saliva.
      TotalPositive6906990.8 (81.9–96.2)100 (97.3–100)
      Negative7136143
      Total76136212
      Excluded two patients with error results from Xpert Xpress Flu/RSV assay in their saliva.
      Abbreviations: A, influenza A virus; B, influenza B virus; NPA, nasopharyngeal aspirate; RSV, respiratory syncytial virus.
      a Excluded two patients with invalid results from Xpert Xpress Flu/RSV assay in their NPA.
      b Excluded two patients with invalid results and two patients with more than one virus detected by Xpert Xpress Flu/RSV assay in their NPA.
      c Excluded two patients with error results from Xpert Xpress Flu/RSV assay in their saliva.

       Viral load analysis

      Among the 64 patients with the same respiratory virus detected in both NPA and saliva, the Ct values were significantly lower in NPA than those in saliva for FluA1 (p < 0.0001), FluA2 (p < 0.0001) and RSV (p < 0.004) channel (see Supplementary material, Fig. S1a). Notably, 14.0% (6/43), 18.6% (8/43), 33.3% (1/3) and 22.2% (4/18) of patients had lower Ct values in their saliva than NPA for FluA1, FluA2, influenza B virus and RSV, respectively. There was no significant correlation between saliva and NPA for the Ct values for influenza A (FluA1, r2 0.004; and FluA2, r2 0.008) and RSV (r2 0.039) (p > 0.05) (see Supplementary material, Figs S1b and S2).

       Comparison of the time and cost associated with collection of saliva, NPS and NPA

      The mean specimen collection time required for the collection of saliva (114 seconds) was 2.26-fold and 1.38-fold shorter than that required for the collection of NPA (259 seconds) and NPS (157 seconds), respectively (Fig. 3a). The mean cost of saliva collection ($1.16) was 2.59-fold and 2.09-fold lower than the costs of NPA ($2.77) and NPS ($2.03), respectively (Fig. 3b).
      Fig. 3
      Fig. 3Time (a) and cost (b) required for specimen collection per patient. The specimen collection time includes the time associated with the specimen collection, informing and instructing the patient about the procedure, and the procedure of putting on and removing masks and gloves. Data on the specimen collection time represents the mean ± standard deviation of the length of time measured for three healthcare workers. NPA, nasopharyngeal aspirate; NPS, nasopharyngeal swab.

      Discussion

       Principal findings

      In this study, we showed a high overall agreement (93.3%) between saliva and NPA specimens when tested by an automated multiplex molecular assay approved for point-of-care testing. The sensitivity of saliva was lower than that of NPA (90.8% versus 96.1%). However, the specimen collection time and the cost associated with the use of saliva were much lower than those of using nasopharyngeal specimens. Hence, saliva is a feasible specimen type for respiratory virus testing when used with automated molecular assays.

       Comparison with other studies

      Previous studies tested saliva for respiratory viruses using molecular assays that require a separate nucleic acid extraction step [
      • To K.K.
      • Lu L.
      • Yip C.C.
      • Poon R.W.
      • Fung A.M.
      • Cheng A.
      • et al.
      Additional molecular testing of saliva specimens improves the detection of respiratory viruses.
      ,
      • Kim Y.G.
      • Yun S.G.
      • Kim M.Y.
      • Park K.
      • Cho C.H.
      • Yoon S.Y.
      • et al.
      Comparison between saliva and nasopharyngeal swab specimens for detection of respiratory viruses by multiplex reverse transcription-PCR.
      ,
      • Robinson J.L.
      • Lee B.E.
      • Kothapalli S.
      • Craig W.R.
      • Fox J.D.
      Use of throat swab or saliva specimens for detection of respiratory viruses in children.
      ,
      • Yoon J.
      • Yun S.G.
      • Nam J.
      • Choi S.H.
      • Lim C.S.
      The use of saliva specimens for detection of influenza A and B viruses by rapid influenza diagnostic tests.
      ,
      • Sueki A.
      • Matsuda K.
      • Yamaguchi A.
      • Uehara M.
      • Sugano M.
      • Uehara T.
      • et al.
      Evaluation of saliva as diagnostic materials for influenza virus infection by PCR-based assays.
      ,
      • Bilder L.
      • Machtei E.E.
      • Shenhar Y.
      • Kra-Oz Z.
      • Basis F.
      Salivary detection of H1N1 virus: a clinical feasibility investigation.
      ]. The current study tested saliva in a US Food and Drug Administration-approved automated multiplex molecular assay. The use of saliva in automated molecular assays can reduce the real-world turnaround time. One of the concerns of using saliva in automated molecular assays is that the viscous saliva may block the fluid channels within the system. However, in this study, the number of invalid/error results was the same for saliva and NPA. Therefore, saliva is a suitable specimen type for this kind of automated system.
      The current study is also unique in that we estimated and analysed the time and costs associated with specimen collection. The shorter specimen collection time is important for busy clinical settings and the reduction in cost is important in resource-limited settings. This analysis has strengthened our recommendation to implement the use of saliva for respiratory virus testing in the clinical setting.
      Most of the previous studies on saliva testing were conducted in children or young adults, or at outpatient clinics or at emergency departments [
      • Kim Y.G.
      • Yun S.G.
      • Kim M.Y.
      • Park K.
      • Cho C.H.
      • Yoon S.Y.
      • et al.
      Comparison between saliva and nasopharyngeal swab specimens for detection of respiratory viruses by multiplex reverse transcription-PCR.
      ,
      • Robinson J.L.
      • Lee B.E.
      • Kothapalli S.
      • Craig W.R.
      • Fox J.D.
      Use of throat swab or saliva specimens for detection of respiratory viruses in children.
      ]. The current study enrolled hospitalized adult patients, including patients who developed severe disease during the hospitalization. This is important because patients with severe disease are likely to benefit most from laboratory confirmation.
      The lower sensitivity of saliva, when compared with NPA, is probably due to the lower viral load in the saliva for most patients. In the current study, seven patients (9.2%) had respiratory viruses detected in their NPA but not in their saliva by both Xpress Flu/RSV assay and our in-house monoplex RT-PCR assay. Therefore, for patients with high clinical suspicion of respiratory virus infection but negative saliva result, NPA should be tested. However, there was a poor correlation in the viral load between paired saliva and NPA, and some patients had higher viral load in the saliva than in the NPA. Notably, respiratory virus could be detected in the saliva for three patients with negative NPA (3.8%). The saliva test results for these three patients have potential impact on the decisions on antiviral treatment and infection control precautions.
      The sensitivity of saliva in the current study is much higher than that found in a previous paediatric study, which showed a much lower sensitivity (74%) for saliva specimens [
      • Robinson J.L.
      • Lee B.E.
      • Kothapalli S.
      • Craig W.R.
      • Fox J.D.
      Use of throat swab or saliva specimens for detection of respiratory viruses in children.
      ]. The higher sensitivity of saliva in the current study may be related to the assay used. In Xpress Flu/RSV, 300 μL of specimen in viral transport medium is used for nucleic acid extraction and all extracted nucleic acid is used for the PCR. However, in systems where nucleic acid extraction is performed as a separate step, the amount of nucleic acid used for the PCR is usually much less. Hence the high sensitivity of this automated multiplex assay has circumvented the problem of low viral load for some patients.

       Limitations of this study

      First, we only recruited adult patients. Further evaluation should be conducted in the paediatric population. Second, this study only included hospitalized patients with more severe disease. As viral load may be lower in patients with milder symptoms, further studies should be conducted in the outpatient setting. Third, Xpress Flu/RSV only detects influenza A virus, influenza B virus and RSV. Further studies should evaluate the accuracy for other respiratory viruses. Fourth, during the study period, there were very few patients with influenza B virus infection and no patients infected with avian influenza viruses. As the tissue tropism of avian influenza viruses can be different from that of seasonal influenza viruses [
      • To K.K.
      • Chan J.F.
      • Chen H.
      • Li L.
      • Yuen K.Y.
      The emergence of influenza A H7N9 in human beings 16 years after influenza A H5N1: a tale of two cities.
      ], the use of saliva for these patients should be confirmed in future studies. Finally, among 321 patients fulfilling the inclusion criteria, 107 were excluded because these patients could not spit out a sufficient volume of saliva. Therefore, some patients would still require nasopharyngeal aspirate or swab for respiratory virus testing.

      Conclusions and implications on clinical practice and research studies

      The higher sensitivity of automated molecular assays has circumvented the problem of lower viral load in saliva specimens. As saliva can be collected easily with minimal equipment and manpower, saliva is a viable option for the detection of respiratory viruses. The use of saliva instead of nasopharyngeal specimens may also enhance recruitment of subjects in clinical studies. In community surveillance studies, the use of saliva allows self-collection. Saliva should be added to the list of recommended diagnostic specimen types for traditional or automated respiratory virus molecular assays in clinical or research settings.

      Transparency declaration

      All authors declare no conflict of interest.

      Funding

      This work was partly supported by donations from the Shaw Foundation Hong Kong, Richard Yu and Carol Yu, Michael Seak-Kan Tong, Respiratory Viral Research Foundation Limited, Hui Ming, Hui Hoy and Chow Sin Lan Charity Fund Limited, and Chan Yin Chuen Memorial Charitable Foundation; and the Collaborative Innovation Centre for Diagnosis and Treatment of Infectious Diseases of the Ministry of Education of China. The funding sources had no role in the design and conduct of the study, in the collection, analysis and interpretation of data, or in the preparation, review or approval of the manuscript.

      Acknowledgements

      We acknowledge the kind generosity of Cepheid for providing the Xpert Xpress Flu/RSV kits used in this study. Cepheid had no role in the design and conduct of the study, in the collection, analysis and interpretation of data, or in the preparation, review or approval of the manuscript. The data on agreement, sensitivity and specificity were presented at the 28th ECCMID in Spain on 22 April 2018.

      Appendix A. Supplementary data

      The following are the supplementary data related to this article:
      Fig. S1. Cycle threshold (Ct) values of Xpress® Flu/RSV assay on saliva and nasopharyngeal aspirate specimens.
      Fig. S2. Bland–Altman plot showing the difference in the Ct values between nasopharyngeal aspirate and saliva of each patient with influenza A virus or respiratory syncytial virus detected by Xpress Flu/RSV assay against the mean Ct values.
      Table S1. Demographics, clinical characteristics and outcome of patients recruited in this study.
      Table S2. Patients with Xpress Flu/RSV assay results reported as invalid or error.
      Appendix S1. Supplementary methods.

      Contributions to authorship

      KKWT, CCYY, CYWL, RWSP, IFNH and KYY designed the study. KKWT, CCYY, CYWL, DTYH, PKPP, ACKN and KHL acquired the data. KKWT and CKHW carried out the statistical analysis. All authors interpreted the data, revised the manuscript critically for important intellectual content and approved the final report.

      Data sharing

      The investigators will share data used in developing the results presented in this manuscript on request to the corresponding author. Anonymized record level data will be made available on proposal for analysis by those who have received ethical clearance from their host institution.

      References

        • Jain S.
        • Self W.H.
        • Wunderink R.G.
        • Fakhran S.
        • Balk R.
        • Bramley A.M.
        • et al.
        Community-acquired pneumonia requiring hospitalization among U.S. adults.
        N Engl J Med. 2015; 373: 415-427
        • To K.K.
        • Lau S.K.
        • Chan K.H.
        • Mok K.Y.
        • Luk H.K.
        • Yip C.C.
        • et al.
        Pulmonary and extrapulmonary complications of human rhinovirus infection in critically ill patients.
        J Clin Virol. 2016; 77: 85-91
        • Semret M.
        • Schiller I.
        • Jardin B.A.
        • Frenette C.
        • Loo V.G.
        • Papenburg J.
        • et al.
        Multiplex respiratory virus testing for antimicrobial stewardship: a prospective assessment of antimicrobial use and clinical outcomes among hospitalized adults.
        J Infect Dis. 2017; 216: 936-944
        • Muthuri S.G.
        • Venkatesan S.
        • Myles P.R.
        • Leonardi-Bee J.
        • Al Khuwaitir T.S.
        • Al Mamun A.
        • et al.
        Effectiveness of neuraminidase inhibitors in reducing mortality in patients admitted to hospital with influenza A H1N1pdm09 virus infection: a meta-analysis of individual participant data.
        Lancet Respir Med. 2014; 2: 395-404
        • World Health Organization
        Research needs for the battle against respiratory viruses (BRaVe).
        2013 (Available at:)
        • Merckx J.
        • Wali R.
        • Schiller I.
        • Caya C.
        • Gore G.C.
        • Chartrand C.
        • et al.
        Diagnostic accuracy of novel and traditional rapid tests for influenza infection compared with reverse transcriptase polymerase chain reaction: a systematic review and meta-analysis.
        Ann Intern Med. 2017; 167: 394-409
        • Huang H.S.
        • Tsai C.L.
        • Chang J.
        • Hsu T.C.
        • Lin S.
        • Lee C.C.
        Multiplex PCR system for the rapid diagnosis of respiratory virus infection: systematic review and meta-analysis.
        Clin Microbiol Infect. 2018; 24: 1055-1063
        • Brendish N.J.
        • Malachira A.K.
        • Armstrong L.
        • Houghton R.
        • Aitken S.
        • Nyimbili E.
        • et al.
        Routine molecular point-of-care testing for respiratory viruses in adults presenting to hospital with acute respiratory illness (ResPOC): a pragmatic, open-label, randomised controlled trial.
        Lancet Respir Med. 2017; 5: 401-411
        • Chan K.H.
        • To K.K.W.
        • Li P.T.W.
        • Wong T.L.
        • Zhang R.
        • Chik K.K.H.
        • et al.
        Evaluation of NxTAG respiratory pathogen panel and comparison with xTAG respiratory viral panel fast v2 and film array respiratory panel for detecting respiratory pathogens in nasopharyngeal aspirates and swine/avian-origin influenza a subtypes in culture isolates.
        Adv Virol. 2017; 2017: 1324276
        • Rappo U.
        • Schuetz A.N.
        • Jenkins S.G.
        • Calfee D.P.
        • Walsh T.J.
        • Wells M.T.
        • et al.
        Impact of early detection of respiratory viruses by multiplex PCR assay on clinical outcomes in adult patients.
        J Clin Microbiol. 2016; 54: 2096-2103
        • Busson L.
        • Mahadeb B.
        • De Foor M.
        • Vandenberg O.
        • Hallin M.
        Contribution of a rapid influenza diagnostic test to manage hospitalized patients with suspected influenza.
        Diagn Microbiol Infect Dis. 2017; 87: 238-242
        • Mitchell S.L.
        • Chang Y.C.
        • Feemster K.
        • Cardenas A.M.
        Implementation of a rapid influenza A/B and RSV direct molecular assay improves emergency department oseltamivir use in paediatric patients.
        J Med Microbiol. 2018; 67: 358-363https://doi.org/10.1099/jmm.0.000676
      1. U.S. Food and Drug Administration. List of microbial tests. Available at: https://www.fda.gov/MedicalDevices/ProductsandMedicalProcedures/InVitroDiagnostics/ucm330711.htm#microbial. [Accessed 2 March 2018].

        • Frazee B.W.
        • Rodriguez-Hoces de la Guardia A.
        • Alter H.
        • Chen C.G.
        • Fuentes E.L.
        • Holzer A.K.
        • et al.
        Accuracy and discomfort of different types of intranasal specimen collection methods for molecular influenza testing in emergency department patients.
        Ann Emerg Med. 2017; 71: 509-517https://doi.org/10.1016/j.annemergmed.2017.09.010
      2. Centers for Disease Control and Prevention. Influenza specimen collection desk reference guide. Available at: https://www.cdc.gov/flu/pdf/freeresources/healthcare/flu-specimen-collection-guide.pdf. [Accessed 22 February 2018].

        • Baron E.J.
        • Miller J.M.
        • Weinstein M.P.
        • Richter S.S.
        • Gilligan P.H.
        • Thomson Jr., R.B.
        • et al.
        A guide to utilization of the microbiology laboratory for diagnosis of infectious diseases: 2013 recommendations by the Infectious Diseases Society of America (IDSA) and the American Society for Microbiology (ASM)(a).
        Clin Infect Dis. 2013; 57: e22-e121
        • Lambert S.B.
        • Whiley D.M.
        • O'Neill N.T.
        • Andrews E.C.
        • Canavan F.M.
        • Bletchly C.
        • et al.
        Comparing nose-throat swabs and nasopharyngeal aspirates collected from children with symptoms for respiratory virus identification using real-time polymerase chain reaction.
        Pediatrics. 2008; 122: e615-e620
        • Meerhoff T.J.
        • Houben M.L.
        • Coenjaerts F.E.
        • Kimpen J.L.
        • Hofland R.W.
        • Schellevis F.
        • et al.
        Detection of multiple respiratory pathogens during primary respiratory infection: nasal swab versus nasopharyngeal aspirate using real-time polymerase chain reaction.
        Eur J Clin Microbiol Infect Dis. 2010; 29: 365-371
        • To K.K.
        • Lu L.
        • Yip C.C.
        • Poon R.W.
        • Fung A.M.
        • Cheng A.
        • et al.
        Additional molecular testing of saliva specimens improves the detection of respiratory viruses.
        Emerg Microbe. Infect. 2017; 6: e49
        • Kim Y.G.
        • Yun S.G.
        • Kim M.Y.
        • Park K.
        • Cho C.H.
        • Yoon S.Y.
        • et al.
        Comparison between saliva and nasopharyngeal swab specimens for detection of respiratory viruses by multiplex reverse transcription-PCR.
        J Clin Microbiol. 2017; 55: 226-233
        • Robinson J.L.
        • Lee B.E.
        • Kothapalli S.
        • Craig W.R.
        • Fox J.D.
        Use of throat swab or saliva specimens for detection of respiratory viruses in children.
        Clin Infect Dis. 2008; 46: e61-e64
        • Yoon J.
        • Yun S.G.
        • Nam J.
        • Choi S.H.
        • Lim C.S.
        The use of saliva specimens for detection of influenza A and B viruses by rapid influenza diagnostic tests.
        J Virol Methods. 2017; 243: 15-19
        • To K.K.
        • Hung I.F.
        • Li I.W.
        • Lee K.L.
        • Koo C.K.
        • Yan W.W.
        • et al.
        Delayed clearance of viral load and marked cytokine activation in severe cases of pandemic H1N1 2009 influenza virus infection.
        Clin Infect Dis. 2010; 50: 850-859
        • To K.K.W.
        • Lu L.
        • Fong C.H.Y.
        • Wu A.K.L.
        • Mok K.Y.
        • Yip C.C.Y.
        • et al.
        Rhinovirus respiratory tract infection in hospitalized adult patients is associated with TH2 response irrespective of asthma.
        J Infect. 2018; 76: 465-474
        • To K.K.
        • Zhou J.
        • Song Y.Q.
        • Hung I.F.
        • Ip W.C.
        • Cheng Z.S.
        • et al.
        Surfactant protein B gene polymorphism is associated with severe influenza.
        Chest. 2014; 145: 1237-1243
        • Landis J.R.
        • Koch G.G.
        The measurement of observer agreement for categorical data.
        Biometrics. 1977; 33: 159-174
        • Sueki A.
        • Matsuda K.
        • Yamaguchi A.
        • Uehara M.
        • Sugano M.
        • Uehara T.
        • et al.
        Evaluation of saliva as diagnostic materials for influenza virus infection by PCR-based assays.
        Clin Chim Acta. 2016; 453: 71-74
        • Bilder L.
        • Machtei E.E.
        • Shenhar Y.
        • Kra-Oz Z.
        • Basis F.
        Salivary detection of H1N1 virus: a clinical feasibility investigation.
        J Dent Res. 2011; 90: 1136-1139
        • To K.K.
        • Chan J.F.
        • Chen H.
        • Li L.
        • Yuen K.Y.
        The emergence of influenza A H7N9 in human beings 16 years after influenza A H5N1: a tale of two cities.
        Lancet Infect Dis. 2013; 13: 809-821