Outlier detection for questionnaire data in biobanks

Rieko Sakurai, Masao Ueki, Satoshi Makino, Atsushi Hozawa, Shinichi Kuriyama, Takako Takai-Igarashi, Kengo Kinoshita, Masayuki Yamamoto, Gen Tamiya

Research output: Contribution to journalArticlepeer-review

4 Citations (Scopus)

Abstract

Background: Biobanks increasingly collect, process and store omics with more conventional epidemiologic information necessitating considerable effort in data cleaning. An efficient outlier detection method that reduces manual labour is highly desirable. Method: We develop an unsupervised machine-learning method for outlier detection, namely kurPCA, that uses principal component analysis combined with kurtosis to ascertain the existence of outliers. In addition, we propose a novel regression adjustment approach to improve detection, namely the regression adjustment for data by systematic missing patterns (RAMP). Result: Application to epidemiological record data in a large-scale biobank (Tohoku Medical Megabank Organization, Japan) shows that a combination of kurPCA and RAMP effectively detects known errors or inconsistent patterns. Conclusions: We confirm through the results of the simulation and the application that our methods showed good performance. The proposed methods are useful for many practical analysis scenarios.

Original languageEnglish
Article numberdyz012
Pages (from-to)1305-1315
Number of pages11
JournalInternational Journal of Epidemiology
Volume48
Issue number4
DOIs
Publication statusPublished - 2019 Aug 1

Keywords

  • Outlier detection
  • anomaly detection
  • kurtosis
  • principal component analysis
  • regression adjustment

ASJC Scopus subject areas

  • Epidemiology

Fingerprint Dive into the research topics of 'Outlier detection for questionnaire data in biobanks'. Together they form a unique fingerprint.

Cite this