Drug classification with a spectral barcode obtained with a smartphone Raman spectrometer

Measuring, recording and analyzing spectral information of materials as its unique finger print using a ubiquitous smartphone has been desired by scientists and consumers. We demonstrated it as drug classification by chemical components with smartphone Raman spectrometer. The Raman spectrometer is based on the CMOS image sensor of the smartphone with a periodic array of band pass filters, capturing 2D Raman spectral intensity map, newly defined as spectral barcode in this work. Here we show 11 major components of drugs are classified with high accuracy, 99.0%, with the aid of convolutional neural network (CNN). The beneficial of spectral barcodes is that even brand name of drug is distinguishable and major component of unknown drugs can be identified. Combining spectral barcode with information obtained by red, green and blue (RGB) imaging system or applying image recognition techniques, this inherent property based labeling system will facilitate fundamental research and business opportunities.

Miniaturization of optical spectrometers has been an active area of research because the demand for portable scientific and industrial characterization tools remains high1,2,3,4,5. Furthermore, smartphones are ubiquitous devices that provide numerous applications and services. Recently, many efforts have focused on converting smartphone cameras into optical spectrometers for mobile food inspection6,7 beauty care8, health care9, and other applications10,11,12,13,14. In these cases, the image sensor of the smartphone detects optical signals from the object of interest—such as reflectance, fluorescence, and Raman emissions. Then, the smartphone’s application processor (AP) and communication chip can together perform on-device or cloud-linked analysis12, providing identification of specimens or evaluation of physical or chemical conditions.

Most research on smartphone-based spectrometers uses gratings as a dispersion component, assembled in an external optics module6,7,8,9,10,11,12,13. Gratings is an excellent optical component in spectrometer to disperse optical signals with high spectral resolution, but is not easy to minimize its form factor to fit into smartphone. To overcome this issue, mini spectrometers by replacing conventional grating with such as photonic crystals14,15, metasurfaces16,17,18, quantum dots19 and silicone nanowires20 integrated on charge coupled detector (CCD) or CMOS image sensors have been investigated. To calculate the input spectrum, s(λ)
out of measured intensity, I(x)
at the detector, numerical analysis needs to be done as expressed by the equation below due to its low Q-factor or complicated form of response function, r(λ,x)
at each pixel where x
is the position of each pixel at the detector.

Thus, experimental results in the literature14,15,16,17,18,19,20 have substantial limitations—especially in terms of capturing weak and high spectral resolution required for Raman signatures.

Due to the increasing online pharmacies and supply chain, counterfeit drugs have become threatening even to public health safety. This issue becomes more critical since increasing the online pharmacies and supply chain can provide blind spots for counterfeit or substandard drugs to be distributed into the public health market21. Current smartphone applications (such as DrugID, ID My Pill, Pill Identifier, Pill Finder, and Drug Info) can distinguish drug types and models either by entering the name, shape, color, and/or etched marks of the drugs; or by comparing the drug pill’s RGB images (acquired with the camera) with the U.S. Food and Drug Administration database. The identification accuracy is insufficient due to similar appearance, absence in the database, or other technical issues. In this sense, Raman spectrum can provide valuable information on drugs, and there have been some researches in the literature on classifying drugs by Raman spectroscopy with the aid of machine learning22,23,24,25,26. Classifying pharmaceutical ingredients, and detection of newly emerging psychoactive substance and illicit drugs were demonstrated by partial least squares-discriminate analysis (PLS-DA)22, principal component analysis (PCA)23 and CNN24, respectively. Detection of illicit drugs25 or psychoactive drugs26 were demonstrated even in human urine and finger marks to prevent patients from overdose or misuse of it by support vector machines (SVM) and PLS-DA, respectively.

We demonstrated smartphone based Raman spectrometer which are enough for drug classification. The Raman spectrometer is composed of 2D periodic array of band pass filters on the image sensor of a Samsung Galaxy Note 9, with a compact external Raman module. Raman intensity map captured by the image sensor is defined as Raman spectral barcode by the analogy of conventional barcodes, machine-readable optical labels that enable location, identification, and/or tracking. As a demonstration, we experimentally investigated 54 commonly used drugs for diabetes, hyperlipidemia, hypertension, painkillers, and nutritional supplements; which frequently come in almost identical shapes, sizes, and colors. Since each spectral barcode of drug contains unique Raman signatures of the material, we conducted the identification of spectral barcodes of drugs with a convolutional neural network (CNN) embedded in the smartphone. In addition, identification accuracy can be further enhanced by information fusion with spectral barcode and conventional RGB images taken by the smartphone camera. Another advantage of spectral barcode-based classification is that we can identify chemical component of unknown drugs once other drugs with the same chemical component are in the database.

Integrating with AI capability in the smartphone spectrometer allows users to analyze the spectrum at various places and situations. This will enhance its portability and usability of smartphone spectrometer in numerous disciplines including drug classification. Our proposed concept of a CNN powered spectral barcode will facilitate many research and business opportunities for smartphone spectrometers.

Smartphone Raman spectrometer and spectral barcode
Figure 1 shows schematics of the smartphone Raman spectrometer and spectral barcode; which is the 2D Raman intensity map acquired with the smartphone Raman spectrometer, and an artificial intelligence algorithm embedded in the smart phone for classification. Raman signals are generated and collected by a compact external module integrated with a 785 nm laser diode. The miniaturized external Raman module is attached to the rear-wide camera of the Samsung Galaxy Note 9, and its detailed optical components and configurations is shown in Supplementary Fig. 1 with a photograph. The Raman emission, which is excited by positioning the specimen at the focal point, i.e. contacting at the objective lens, simultaneously illuminates several sets of 128 channels (CHs) located near the center of the image sensor. For 120 CHs out of 128 CHs, its band pass filters transmit 120 distinct wavelengths in the range of 830–910 nm. The rest CHs are blocked by metal as position indicators exhibited as black squares in Fig. 1. The spectral width and transmission rate of the band pass filters range from 1–1.2 nm and 0.45–0.6, respectively (Supplementary Fig. 2). Each band pass filter consisted of a pair of Si/SiO2 distributed Bragg reflectors (DBRs), its resonant wavelength is adjusted by the thickness of the Si cavity layer in the center27,28. The details of the filter structure and fabrication can be found in Methods. In Supplementary Table 1, the smartphone Raman spectrometer of this work is compared with miniaturized spectrometers which are controllable by android smartphones, or embedded in the smartphone12,29,30. The compared details are shown in the caption of Supplementary Table 1. As the role of the external module in this work is just to excite and collect Raman signals from the specimen without additional connecting electronic board to the smartphone, the smartphone Raman spectrometer becomes more compact and versatile with minimized external module.

This entry was posted in Uncategorized. Bookmark the permalink.