(1039-B) ot2eye: Detection of labware conditions to monitor and extend affordable liquid-handling robots

Abstract: In the automation of life science experiments, the types, positions, states, and wears of labwares change according to modifications in experimental protocols and the progress of experimental operations. Accordingly, verifying and recording that the experimental robot is operating in the intended environment is essential for quality assurance and traceability of actions and results. For such purpose, general webcams are used to record fixed-point images inside automated liquid-handling robots, primarily for investigating what happened when abnormalities occur. However, a large number of images and long-duration recordings can make the verification process cumbersome, hindering traceability. To overcome this difficulty, we hypothesized that automatic detection of labwares from images and videos would make it easier to acquire labware states and search logs. Here, to demonstrate the effectiveness of this idea, we developed ot2eye, a labware detection system for Opentrons OT-2. ot2eye can detect labwares in images taken inside the OT-2 using the YOLO object detection algorithm. We built a unique dataset by annotating images of the OT-2 with labware in place and fine-tuned the YOLO object detection algorithm using this dataset. As a result of validation, ot2eye demonstrated high-precision detection of various labware types, such as micro-well plates and tip racks. Furthermore, it was able to accurately detect not only labwares but also the presence or absence of tips in tip racks from a single image. Our proposed method is expected to provide vision to affordable automated pipetting robots and promote AI-Robot-driven life sciences by making the states of labwares machine-readable.