Deep industrial transfer learning at runtime for image recognition

2.1 Deep industrial transfer learning

In humans and animals, ‘natural’ intelligence allows the transfer of knowledge from known problems and their solution towards unknown ones or to adapt previously learned lessons based upon new information – both while retaining older information or skills. This is commonly called ‘continual learning’ [13], [21].

In ‘artificial’ deep learning, such transfer or adaption is not easily achieved, as new information simply tends to overwrite old information without gaining a considerable advantage compared to learning from scratch using only new information. This displacement process is called ‘catastrophic forgetting’ [3].

The term ‘continual learning’ describes approaches to overcome the problem of catastrophic forgetting in machine learning. More specifically, it describes a field focused on solving multi-task problems in a common feature space X1=⋯=Xt with differing distributions of input data PX1≠…≠P(Xt) and known source and target labels. The goal is to learn those tasks sequentially without forgetting previously learned tasks, so that eventually all tasks can be solved by a single deep learning algorithm. Building upon [6], [20], three different problem cases can be differentiated:

1. Incremental Task Learning (ITL) describes problems, in which a deep learning algorithm is sequentially trained on multiple observations Dt={(x,y)∣xi∈X;yi∈Yt;i=1,…,nt} corresponding to t∈N+ domains and tasks in order to be able to infer any yj∈Yt given (xj∈X,t). This is commonly achieved using a multi-headed output layer of which only the head corresponding to task Tt is active.

2. Incremental Domain Learning (IDL) describes problems, in which a deep learning algorithm is sequentially trained on multiple observations Dt=(x,y)∣xi∈X;yi∈Yt;i=1,…,nt corresponding to t∈N+ domains and tasks in order to be able to infer any yj∈Y given xj∈X.

3. Incremental Class Learning (ICL) describes problems, in which a deep learning algorithm is sequentially trained on multiple observations Dt=(x,y)∣xi∈X;yi∈Yt;i=1,…,nt corresponding to t∈N+ domains and tasks in order to be able to infer any yj∈Yt given xj∈X.

1. ITL describes this algorithm’s ability to correctly label new images when given information as to which label space Yt shall be used – i. e., returning e. g., one of the two possible answers ‘1’ and ‘2’ for an input image associated with task T1.

2. IDL, in contrast, describes the algorithm’s ability to correctly label new images without specifying the label space Yt used at all – i. e., returning one of the merely two possible answers ‘1 or 3’ and ‘2 or 4’.

3. ICL, finally, describes the algorithm’s ability to correctly label new images using all labels from all label spaces Yt – i. e., returning one of the four possible answers ‘1’, ‘2’, ‘3’ and ‘4’.

Figure 1

Continual learning problem classes using the example of hand-written digit recognition.

However, these continual learning problem classes only address a sub-space of the much greater problem space regarding the transfer of knowledge. More precisely, continual learning focusses on retaining old skills and thereby leads to an algorithm’s generalization, focusing on similarities between different (sub-)problems, e. g., recognizing characteristics of the same problem described in different datasets. Contrastingly, ‘transfer learning’ focusses just on the new skills [22] and thereby leads to an algorithm’s differentiation, focusing on differences between different (sub-)problems, e. g., forming clusters of very similar problems described in different datasets. As useful such a distinction might be in classifying algorithms, it loses its usefulness when applied to practical problems: Industrial automation use cases as described in Section 1 usually require the generalization of information contained across potentially dispersed sub-datasets, e. g., at different locations, and across small changes over time as well as the differentiation between different variants of a problem. This double requirement leads the authors of this article to define the term ‘industrial transfer learning’ as encompassing both, continual and transfer learning. More specifically, deep industrial transfer learning shall be examined as a method to mitigate the problems of conventional deep learning as described in Section 1 [17].

Luckily, both, continual learning and transfer learning, need to overcome the so-called ‘stability-plasticity dilemma’, which refers to the opposing aims of having an algorithm stable enough to keep connections once learned and flexible enough to learn new connections once encountered. This justifies the hypothesis, that at least some continual learning approaches should deliver good results when used for industrial transfer learning.

2.2 Dual memory method for image recognition

One such promising continual learning approach suitable for deep industrial transfer learning as described above is the dual-memory method [21], [9], [15]: A slow-learning module (inspired by a mammalian brain’s neocortex section) and a fast-learning module (inspired by a mammalian brain’s hippocampus section) combine their strengths to mitigate the stability-plasticity dilemma. The slow-learning module, hereon called module A, is used to extract general information from the input data and generalize it. In contrast, the fast-learning module, hereon called module B, is used to extract specific information of the input data and to save it as new memory.

This approach is not to be confused with the state-of-the-art process of using pre-trained convolutional neural networks to solve image recognition tasks. Instead of merely pre-training on a similar dataset, which would indeed solve the problem of training data availability, the proposed approach continuously and efficiently re-trains on every new instance provided and therefore solves the problem of continuously changing dynamic processes as well.

Today, feature extraction, i. e., the extraction of general information from input data, is usually carried out by deep neural networks (DNN). Different DNNs can be used for this purpose, but for image recognition tasks as examined here, convolutional neural networks (CNN) have proven most capable. Tab. 2 lists some of the most accurate CNNs, whose performance shall be compared in Section 2. Although the algorithms listed are full classification algorithms, only their feature extraction components will be used. Therefore, they are referred to as feature extraction algorithms.

While many CNN-based feature extractors can be used as module A, the requirements for module B are more demanding. In order to facilitate transfer learning for the use case described in Section 1, an appropriate algorithm must

1. be trainable on a data stream, where samples of the different classes appear randomly in time and order,

2. be able to classify already seen and trained classes at all times,

3. show a limited growth of computational complexity and memory consumption as the number of known classes grows and

4. not need to store any raw data for training.

4.1 Incremental learning: comparison of different feature extractors on ImageNet-10

The following experiments compare the different feature extraction algorithms from Tab. 2 regarding their performance when used in module A. The experiments were executed on the ImageNet-10-dataset, a subset of the larger ImageNet-dataset. It can be characterized by a small number of different classes (10) and a large number of highly different natural images per class (>1250), challenging any image recognition algorithm to distinguish small as well as large differences.

The training of DICLA was executed with one epoch per incremental step, so that every training image is seen just once. As described in Section 3, DICLA contains different hyperparameters, which were optimized based on a grid search. Because the feature extractor, module A, was fixed for these experiments (αA=0), solely hyperparameters of module B, threshold value ρ and the learning rate αB, were optimized. Both parameters can range from 0 to 1 and were investigated in this range with a step width of 0.1, resulting in 121 combinations for the grid search. To ensure meaningful results, we performed cross validation. We split the training data (<1250 images) into a training set of 20 and a validation set of 100 randomly drawn images per execution. This was repeated five times per possible combination, resulting in a total of 606 tests. Based on this hyperparameter optimization, the following parameter were chosen for conducting the final results:

1. Training images per class: 10 to 1250 – randomly drawn

2. Test images per class: 50 (all available images)

3. Threshold value ρ=0.5 (fix threshold)

4. Learning rate of module B αB=0.2

Tab. 3 lists the average training time, the average final accuracy and the average final accuracy’s double standard deviation for every feature extraction algorithm from Tab. 2 on different numbers of randomly drawn training images ranging from 1 to 1250. Averages and deviations are calculated based upon five runs. It can be seen that with an increase in the number of training images per class, the final recognition accuracy rises at the cost of a higher training time. However, while the calculative load increase for higher numbers of training images per class rises sharply, the accuracy gain decreases rapidly. While this is true for every feature extraction algorithm, the extend differs, e. g., MobileNet-V2 already comes close to its peak accuracy using only 50 training images and taking 23.4 s whereas Inception-V3 needs at least 500 images and 493.4 s to do the same. Furthermore, the algorithms greatly vary in their peak accuracy and the training time needed, ranging from 58 % (Inception-V3) to 76.3 % (MobileNet-V2) and from 439.7 s (MobileNet-V2) to 2888.5 s (ResNet152V2).

To further compare the different algorithms’ accuracies against each other, Fig. 3 shows their performance on 10, 100 and 1000 training images per class and 1, 2, 5 and 10 incremental learning steps, each averaged over five runs for each of the six feature extraction algorithms listed in Tab. 2 respectively used for module A.

The diagrams on the left side depict the accuracy after each group’s training in a scenario where the ten classes are split into ten groups of one class each and learned sequentially using 10 (top), 100 (middle) or 1000 (bottom) training images per class. While the small number of training samples in the top diagram leads to several algorithms not starting out with 100 % accuracy after just one trained class, this gets better with an increase in the number of training images. Inception-V3 consistently performs much weaker than the other feature extraction algorithms, which perform similarly well except for MobileNet-V2 and to a lesser extend DenseNet121 reaching better accuracies.

The right diagrams depict the final accuracy after the complete, sequential training of a scenario where the ten classes are split into 1, 2, 5 or 10 training groups using 10 (top), 100 (middle) or 1000 (bottom) training images per class. Altogether, the number of training groups does not appear to have a large influence on the algorithms’ performance. This supports the claim that DICLA is capable of performant incremental learning, thereby mitigating the plasticity-stability-dilemma so that old skills are retained while new ones are learned.

Figure 3

Results for DICLA using different feature extraction algorithms and different parameter sets on the ImageNet-10 dataset.

Concludingly, the experiments on ImageNet-10 clearly point towards the use of MobileNet-V2 in module A, combining the highest final accuracies with the lowest training time needed among all feature extractors compared. Furthermore, MobileNet-V2 is the feature extractor with the smallest memory footprint (see Tab. 2).

4.2 Incremental learning: evaluation of DICLA using MobileNet-V2 on ImageNet

Based upon the findings in the previous section, a version of DICLA using MobileNet-V2 in module A is further evaluated. To this extend, experiments are executed on the full ImageNet-dataset, which can be characterized by a large number of different classes (1000) and a large number of highly different natural images per class (>1250), challenging any image recognition algorithm to distinguish small as well as large differences. The well-published algorithms iCaRL and Learning without Forgetting (LwF) [11] are used for comparison (accuracies obtained from [28]).

Training of DICLA is executed using the same parameters as in the previous section and 10 and 100 training images per class.

Figure 4

Results of DICLA using 10 training images per class, DICLA using 100 training images per class, iCaRL and LwF on the full ImageNet dataset.

Fig. 4 and Tab. 4 show the resulting accuracies: iCaRL and LwF start out at 90 % accuracy, DICLA using 100 training images per class (DICLA100) at 75.1 % and using 10 training images per class (DICLA10) at 67.9 %. However, over the course of the incremental learning process, this gap between DICLA and the other algorithms narrows considerably. In the end, DICLA10 is even slightly more accurate than LwF (39.3 % to 39 %) and DICLA100 considerably more accurate than iCaRL (49.6 % to 44 %). This is despite the fact, that iCaRL as well as LwF utilize a ResNet-Architecture for feature extraction, which generally achieves a better classification accuracy than MobileNet-V2 (see Tab. 2) and both perform 100 epochs per incremental step and use all about 1,300 training images per class of which iCaRL saves about 20,000.

4.3 Distributed learning: evaluation of DICLA using MobileNet-V2 on ImageNet-10

Finally, DICLA using MobileNet-V2 in module A is experimentally evaluated regarding its distributed learning capabilities. To this end, the ten classes of ImageNet-10 are evenly distributed to 1, 2, 5 and 10 edge devices conducting the training, so that every edge device has the same number of classes to learn. Training is executed using the same parameters as in the previous sections and 100 training images per class. After the training, the knowledge acquired is then shared between the edge devices and the algorithm’s performance is tested on the evaluation dataset consisting of the test images.

Tab. 5 shows the results of this experiment: It can be seen that the average final accuracy is almost fully independent from the number of edge devices. Because of the design of DICLA, this could be expected, because representations and class assignments can easily and losslessly be exchanged between different edge devices. Small differences could be explained by the still rather large spread of the results and a slight advantage for more centralized learning, i. e., on a smaller number of edge devices, due to cross-effects.