Machine vision-based driving and feedback scheme for digital microfluidics system

2.1 EWOD device structure and materials

The EWOD device comprises an electrode array of photolithography patterned metal (indium tin oxide) on a glass substrate with a ground plane (indium tin oxide on glass) connected parallel to it. The electrode array and the ground plane are separated by a gasket of known thickness (H). Some individual droplets are contained between the electrode array and the ground plane. To reduce friction, silicone oil (mass fraction: 10%) was used as the lubricant surrounding the droplets in previous studies.

A hydrophobic insulating layer is spin coated to insulate the droplets from the electrode array. In our study, Teflon AF 1600 (800 nm) is used as a dielectric as well as a hydrophobic layer in the EWOD device. Note that the Teflon AF 1600 is a biocompatible polymer with an average static contact angle of 116°. Its dielectric constant is approximately 2.4. This characteristic is conducive to various droplet operations [25]. The top view of a closed EWOD chip is shown in Figure 2(a). It consists of 10 (3 mm × 3 mm) square electrodes with a 50 μm separation between adjacent electrodes. The height between the upper plate and the lower plate is approximately 1.5 mm. To ensure the stability of the EWOD chip and the reliability of the circuit connection, we designed a printed circuit board (PCB) foil as the interface between the EWOD chip and the external driver circuit. Each electrode is connected to an external driving chip interface by a separate electrical wire. The spacing between the adjacent electrical wires is 1 mm. Detailed parameters and dimensions of the PCB foil are shown in Figure 2(b).

Figure 2

(a) Top view of the closed EWOD chip. (b) Detailed parameters and dimensions of the PCB foil.

2.2 Machine vision-based driving and feedback scheme

Figure 3 shows a schematic illustration of the machine vision-based DMF driving and feedback scheme in a double-plate EWOD chip configuration. The four major components include the double-plate EWOD chip, a high-resolution camera, a computer with a graphical image analysis system, and the driving control system.

Figure 3

A schematic illustration of the EWOD chip machine vision-based driving and feedback scheme.

There are four key modules in the design of the EWOD chip machine vision-based driving and feedback scheme: image acquisition, image processing and recognition, chip status analysis, and feedback and driving. The original EWOD chip image, including droplets, captured by the camera must be processed through an image algorithm. The complete image processing and the analysis solution are shown in Figure 4. The background of the EWOD chip and the droplet shapes have obvious differences. Background subtraction and droplet extraction are the two main algorithms used in our study to capture the real-time position of droplets on the EWOD chip.

Figure 4

The complete image processing and analysis solution.

First, the K-nearest neighbor (KNN) background model of the original image is built by the image analysis system. Then, we use morphology to analyze the processed images and set appropriate “expansion” parameters. We also develop a median-based Gaussian weighted filter (MGWF) that is effective for image edge filtering. To eliminate salt-and-pepper noise, an improved adaptive filtering (IAMF) algorithm is used in this study. Finally, the positions of droplets are recognized by a circular edge detection algorithm.

The proposed driving control system is programmed to control a series of droplet actuations and acquire image information from the image analysis system to manage the control logic for the sequential operations. Different droplet operations (e.g., move, dispense, merge) can be automatically controlled based on the recognition and feedback.

2.3 Driving control system

The EWOD device driving control system architecture diagram is shown in Figure 5. We use an embedded microcontroller (STM32F7) as the control core of the driving control system. Its frequency can reach 180 MHz, and it has abundant communication interfaces (i.e., SPI, I²C, and a serial port). To actuate the droplet from one electrode to another electrode, a high voltage signal is necessary. Most previous studies used a “boost converter + relay” as the voltage output module. However, the problem with this design is that the voltage rise time is high (e.g., 200–500 ms). To avoid this problem, an SSD1627 chip is used as a voltage output module in the proposed driving control system. It has 98 I/O ports, each of which can work together or independently and output 18–40 V of driving voltage [26]. These characteristics can meet the driving requirements of EWOD devices very well. All the real-time data are stored in an external electrically erasable programmable read-only memory (EEPROM; 32 kB, M95320-R, STMicroelectronics). The serial port is used as a data exchange interface between the image analysis system and the driving control system. The proposed driving control system has the advantages of easy fabrication, portability, and a high level of integration.

Figure 5

Driving control system architecture diagram.

2.4 Feedback model for droplet actuation

As described earlier, the real-time positions of droplets on the EWOD chip can be obtained by the proposed location system. Furthermore, the droplet position data are provided to the driving control system as feedback.

In this study, we present a priority adjustment strategy for driving parameters based on feedback (Table 1). The criterion of the proposed feedback model is to improve DMF application efficiency without affecting the EWOD chip stability as much as possible.

For the proposed system, users must enter values for parameters required to actuate the droplets on the device before the system starts. These parameters mainly include electrode radius, droplet radius (i.e., one and a half of the electrode size), initial single driving time (i.e., time duration for one pulse), incremental time (i.e., time duration for one adjustment), initial driving voltage (i.e., initial voltage applied to the electrode), incremental voltage, initial electrode, and destination electrode.

To ensure the reliability and safety of the EWOD chip, the highest driving parameter adjustment priority is given to the single driving time. Generally, for most DMF applications, the single drive time is less than 2,000 ms. A change in the driving time will not affect the EWOD chip reliability.

Droplet operations (e.g., moving or splitting) are highly dependent on driving voltage. Due to various defects on the electrode surface, a larger driving voltage is beneficial to overcome the resistance to droplet movement. However, excessive driving voltage may cause degradation of the dielectric layer, reducing the lifetime of EWOD device. Consequently, it takes second priority. It should be noted that the incremental voltage should not be set too large in most applications. In our study, it was set to 2 V.

In this study, we first proposed the droplet motion path as a modifiable parameter in the feedback model. The droplet motion path has the lowest priority. Generally, the default droplet motion path is the optimal path. Some experiments revealed that there can be droplet motion failure even if the driving voltage and time are sufficiently large. In this case, the driving control system must adjust the droplet motion path. For soluble droplets, the motion path recalculation for a particular droplet must consider the motion paths of all droplets (i.e., at least two electrodes should separate the droplets). This requires that the control system not only have accurate droplet positioning capability but also have powerful calculation and analysis ability. In our software, the shortest path method is used to recalculate the new droplet motion path.

The feedback control pseudo code is presented in Table 2. The software used in this article is written in the standard C/C++ language. By using the modular design, the efficiency, expansibility, and maintainability of the software are improved. The driving voltage and time are accumulated based on cyclic iteration in the proposed system. Because the voltage rise time of the SSD1627 chip is approximately 10 ms, the modification of driving parameters for droplet control can occur quickly.

1. Ethical approval: The conducted research is not related to either human or animal use.

3.1 Droplet location experiment

The image analysis system is programmed based on the OpenCV library, which is used to develop and verify the proposed detection and recognition scheme in the Visual Studio (VS) 2015 environment.

First, we set up a droplet location experiment to demonstrate the proposed system’s algorithm flow. To detect the droplet position, four operations were executed every 300 ms to judge whether the droplet had dispensed from the reservoir electrode or transferred successfully over the activated electrode. The algorithm flow of the machine vision-based driving and the feedback system is shown in Figure 6.

Figure 6

Algorithm flow of the machine vision-based driving and feedback system. (a) Original image of EWOD device. (b) Image graying and background extraction. (c) Image binaryzation. (d) Droplet location.

Step 1: The image analysis system acquires an original frame captured by the high-resolution camera. Step 2: The image analysis system calculates a difference image for the droplet by subtracting a reference image. Step 3: The image analysis system binarizes the difference image using a series of image algorithms. Step 4: The image system uses a Hough transform function to detect droplets on electrodes. Step 5: The image analysis system returns a successful or unsuccessful result to the driving control system. A failed droplet motion will trigger driving parameter adjustment (as presented in Table 1, there are three priorities) until the motion is successful. If the droplet successfully moved to the target electrode, it will be driven to the next electrode based on the default droplet path. These procedures are designed to set up a closed loop, so that droplet detection and driving are executed continuously and simultaneously.

In this instance, a droplet was stationary at the x electrode. The driving control system activated the y electrode. An original frame was captured by the high-resolution camera, as shown in Figure 6(a). A grayscale frame was obtained from the original frame (Figure 6(b)). In our study, a reference image of the EWOD chip was acquired without visible droplets on any of the electrodes. This reference image is acquired for droplet edge detection and droplet position subtraction techniques. Furthermore, a binary frame is created using image-processing algorithms. From this frame, a Hough transform function can be used to detect circles (droplets), as shown in Figure 6(d).

After sensing the droplet’s position, the feedback model is applied to ensure that droplets can reach the target electrode smoothly. As described earlier, the system will first adjust the single driving time if the droplet motion has failed.

3.2 Droplet motion experiment: shuttling

Surface defects on a hydrophobic layer may cause unsteady motion. If droplet motion is unexpectedly slowed, sequential electrode activation without feedback will result in an out-of-control droplet. In this section, droplet shuttling was selected as a verification experiment for demonstrating the machine vision-based driving and feedback scheme. The purpose of this experiment is to demonstrate the effect of the proposed scheme on the success rate of continuous droplet motion. Shuttling means that the droplet moves back and forth frequently in the EWOD chip. In this experiment, a droplet shuttled in a linear EWOD chip consisting of eight electrodes. The electrode diameter is 5 mm, and their spacing is 20 µm. The number of activated electrodes ranges from 1 to 16. Each case was repeated 10 times.

From the experimental result shown in Figure 7(a), we can see that the success rate of droplet shuttling decreases with the increasing number of activated electrodes without a feedback scheme. When the number of activated electrodes was 16, the success rate was only 40%. The reason for the failure is that the droplet was blocked on electrode No. 6. A larger driving time and voltage can be set to overcome this problem. However, such a configuration will not only affect the real-time performance of the DMF system but also reduce the stability of the EWOD device (the larger voltage will reduce the life of the hydrophobic layer). To solve this issue, the proposed driving and feedback scheme is integrated with the DMF system. The experimental result shows that the proposed scheme can effectively improve the success rate of droplet shuttling (i.e., the success rate can reach at least 80% with feedback). Furthermore, the proposed scheme can be applied to several DMF system droplet operations, including splitting, dispensing, and merging.

Figure 7

The comparison results of the two droplet motion methods. (a) Success rate of droplets shuttled under different driving schemes. (b) The time of droplet movements under different control schemes.

In addition, we conducted a comparison between the capacitance-based position method and the proposed scheme. Figure 7(b) shows the comparison results of the droplet motion under different control schemes. As mentioned earlier, compared with conventional capacitance-based sensing methods, the advantage of the machine vision-based position method is high efficiency. With an increased number of droplet movements, the detection time required by capacitance-based sensing methods is much longer than that for the machine vision-based position method. This is because the capacitance-based position method must collect the capacitance values of all electrodes every time. For EWOD chips with many electrodes, the locating time will far exceed the droplet movement time.

3.3 Droplet motion experiment: merging

Next, we used the proposed scheme to track droplets merging on the EWOD chip. Droplet merging is a commonly performed operation on EWOD chips. Merging is defined as an operation in which multiple droplets are driven to an electrode for fusion, and the merged large droplet can move successfully. This operation is widely used in chemical and biological fields. Some previous studies detected droplet merging by capacitive sensing and feedback [27,28]. However, this method is time consuming because it needs to scan every electrode. Moreover, this method cannot identify the size of the merged droplet.

In this section, a universal chemical reaction (the droplet on the left is d-glucose, the one on the right is H₂O) is controlled and detected by the proposed scheme on the EWOD chip. The aim of this study is to observe the proposed system’s performance as the droplet volume changes. As shown in Figure 8(a), two droplets of equal volume (3 μL) are captured by the proposed system. Two droplets are driven to move toward the middle region simultaneously (as shown in Figure 8(b)). The mole values were converted to the number of molecules in 6 μL, which is the volume of the merged droplet. After the two droplets containing chemicals were merged (as shown in Figure 8(c)), the system deduced that the merged droplet volume is approximately 6.3 μL by analysis and calculation. The experimental results show that the measured droplet volume agrees with the theoretical droplet volume.

Figure 8

A chemical reaction controlled by the proposed system. (a) Two droplets of equal volume (d-glucose and H₂O) are observed on the EWOD chip. (b) The two droplets are driven toward the middle electrode. (c) The two droplets merge on the EWOD chip. (d) The merged large droplet is captured by the proposed system and moved to the left electrode.

After this experiment, we found that the driving control system applied 30 V to drive the merged droplet at the beginning, but failed. According to the feedback model, the driving control system adjusted the driving parameters many times (i.e., the driving voltage reached 40 V) until the merged droplet moved successfully (Figure 8(d)).

Of note, droplet motion can accelerate the rate of a chemical reaction, similar to shaking a test tube to increase the chemical reaction rate. This is an advantage of the DMF system in chemical and biological fields.

3.4 Droplet status detection

In addition to droplet shuttling and merging, we also validated the proposed scheme by evaluating droplet movement effects for different liquids. All of them (i.e., DI water, PBS, and HBSS) are commonly used in chemical and biological experiments. In our experiment, different liquids were driven across a linear EWOD chip consisting of six electrodes; this was repeated 10 times for a total of 60 motions.

In general, the droplet velocity was measured for each movement. It is the ratio between the electrode length (L; i.e., L = 5 mm) and the single driving time (T _d; i.e., V = L/T _d) without a feedback system. However, for the proposed scheme, the image analysis system needs some time to locate the droplets. This time (T _p) is approximately 200 ms (±50). Moreover, the driving chip (T _v) output time is approximately 20 ms. Hence, the droplet velocity was defined as V = L × N K d × ( T d + T p + T v ) , where K _d is the number of electrode actuations and N is the number of electrodes that the droplet actually crosses in the experiment.

The droplet movement performance for different liquids on the EWOD chip without feedback is shown in Figure 9. We know that T _d not only affects the droplet movement success rate but also determines droplet velocity. In this experiment, different liquids, DI water, PBS, and HBSS were driven across six electrodes at different velocities (i.e., different single driving times: 300, 500, 800, 1,200, 1,500, and 1,800 ms) with 10 repetitions for a total of 60 actuations. Figure 9 shows that DI water maintains a higher motion performance at high velocities (short single driving time). However, higher velocities generally result in poor droplet movement for liquids containing NaCl (PBS and HBSS). Although a longer single driving time can improve the droplet movement success rate, it may aggravate surface fouling of the hydrophobic layer [29] and reduce the real time of the system. This experiment shows that it is not appropriate to use a fixed single driving time to drive droplets. Therefore, the feedback model has good application significance for a DMF system.

Figure 9

The droplet movement performance for different liquids on the EWOD chip without feedback.

The droplet movement performances for different liquids on the EWOD chip with feedback are shown in Table 3. Obviously, improvements are observed in this experiment. The number of successful movements (out of 60) increased significantly with the proposed scheme for the same droplet velocity. Particularly in complex DMF systems (i.e., multiple droplets moving simultaneously on the EWOD chip), droplet velocity should be taken into account. In addition, in this experiment, we found that a short driving time (300 ms) is favorable for liquids containing no proteins (PBS and DI water), while a long driving time (1,200 ms) is favorable for protein-rich liquids (HBSS). This observation is similar to that of the previous study [30], where a short single driving time was not enough to account for the liquid viscosity.

If the droplet is conductive and the driving voltage is DC, the actuation force in the absence of a top dielectric layer depends on the equivalent capacitance of the bottom dielectric layer [31]. On the same EWOD device, the droplet motion performance will depend on the friction force. Also, the higher friction forces is associated with high-viscosity liquids [32]. The pH of the droplet has no direct effect on the droplet movement performance.

Clearly, the experimental data generated by the proposed feedback model much better estimated the actual kinetics than those generated without feedback control. Hence, this experiment proved that the proposed driving and feedback scheme helps improve the motion performance of liquids containing NaCl by automatically optimizing the driving parameters.

3.5 Adaptive experiment

There are inevitably various failures in practical DMF applications. To demonstrate the reliability of the proposed scheme in complex DMF applications, we set up an adaptive experiment. The specific experimental parameters are presented in Table 4. The initial driving voltage and single driving time are 0. In this experiment, we artificially burned the hydrophobic layer of electrode No. 6 using high voltage (100 V). The whole experiment is automatically controlled by the proposed system. A series of experimental images is shown in Figure 10.

Figure 10

A series of real-time EWOD chip images. All driving parameters are set and adjusted adaptively by the proposed system. (a) Droplet stays at No. 5 electrode. (b–e) Droplet changes motion path. (f) Droplet reaches the destination electrode.

After this experiment, we printed out and reviewed all the system records. First, the system calculated the droplet motion path (i.e., the shortest path method: No. 5 → 6 → 7 → 8). Next, the system activated the No. 6 electrode according to the droplet motion path. However, the droplet could not pass over the No. 6 electrode because it had been damaged (this process lasted approximately 9 s). During this process, the driving voltage and single driving time were adjusted many times (i.e., the voltage reached 40 V and the time reached 1,800 ms). In this condition, the system recalculated the droplet motion path (i.e., new path: No. 5 → 1 → 2 → 3 → 7 → 8). Finally, the droplet successfully moved to the destination electrode (No. 8). Table 5 presents the number of driving parameter modifications in this experiment. The driving time was modified 18 times. The results are consistent with our feedback model presented in Table 1. Notably, the increment of driving voltage (3 V) and time (300 ms) were set relatively low in this experiment. Such a setup is beneficial to protect the EWOD chip. However, it will affect the DMF system efficiency. Specific feedback model rules can be fine-tuned for different applications (e.g., increment or driving voltage range).

Electrode damage is a kind of device defect that cannot be repaired. In this experiment, the proposed system successfully transported the droplet to the target electrode by modifying the droplet’s motion path. Furthermore, the system will record the electrode number and the newest droplet motion path. The droplet will not become stuck on the damaged electrode again the next time. The experimental results revealed that the proposed mechanism has good reliability and ability to avoid interference (i.e., a damaged electrode).

The properties comparison of the proposed scheme with some previously reported control system is summarized in Table 6. The position sensing of the proposed scheme is dynamic, indicating detection, feedback, and actuation can be simultaneously realized. The driving control system has high portability and integration. Moreover, the preparation of the proposed system before detection is simple. The properties of self-adaptive of actuation parameters and recalculation of droplet motion path are also very useful for practical applications of the DMF system.