Estimating adsorption isotherm parameters in chromatography via a virtual injection promoting double feed-forward neural network

A Structure of the feed-forward neural network

This appendix introduces what feed-forward neural network (FNN) is used for, its model structure, and how the model is trained to be valid.

An FNN contains an input layer that takes in the independent variables (also called features) values, an output layer that produces values for model predictions, and the hidden layers (if any) whose structure is controlled by hyper-parameters set by users. Figure 7 is an example of FNN for a regression problem with two independent variables { X 0 , X 1 } and two dependent variables { Y 0 , Y 1 } , where the following holds.

• The value pair { x 0 , x 1 } is a realization of { X 0 , X 1 } and also the input to the model.

• The 𝑏 terms (called bias terms) and the weights { w i ⁢ j } are parameters. The node values (except the bias nodes and the ones in the input layer) are transformations of the weighted sums of the outputs from the previous layer, i.e.

  a i ⁢ j := σ ⁢ ( w 0 ⁢ i ⁢ b 0 ( j - 1 ) + ∑ k = 1 n j - 1 w k ⁢ i ( k ) ⁢ a k , j - 1 ) , j = 1 , 2 , 3 ,

  where 𝜎 is called the activation function which is pre-selected, n j - 1 denotes the number of nodes in the ( j - 1 ) th layer. Further, a i ⁢ j = x i if j = 0 , and a i ⁢ j = o i if j = 3 . Normally, the activation is the same for every node except the ones in the output layer. For classification problems, the activation function for the output layer nodes may be the softmax function, which converts real numbers to probabilities of classes, while for regression problems the activation function for the output layer may just be an identity function.

• The 𝑜 terms are outputs from the neural net. They are supposed to match the data for the dependent variables.

• Note that this example has only one bias term in each layer (except the input layer). Other neural networks may have one bias associated with each node.

The role of the activation function is to control the amount of contribution made by the corresponding node to the model’s output, and there are different choices for this function. The widely used ones are sigmoid, ReLU, and tanh, each having its own advantages and disadvantages. For example, ReLU is computationally efficient to use and can avoid vanishing gradient, but it may lead to the problem of “dead neuron”, i.e. during the training, when the ReLU activation function is used, if a neuron is not activated in some step, it will never be activated in all the following steps even if it should be activated in the true model.

To find values of the parameters (the weights and bias terms) so that the model’s outputs are close to data of the dependent variables, a process called model training is performed in the following procedures.

1. Pre-select hyper-parameters such as the loss function and the hidden layer structure.

2. Assign initial values to the parameters (weights and bias terms).

3. Compute the partial derivative of the loss with respect to each weight and bias term through the back-propagation algorithm.

4. Based on the partial derivatives, update the parameter values to decrease the loss value.

5. Repeat steps (3)–(4) until the loss converges (meaning it does not decrease any more) or reach some threshold.

The number of hidden layers and nodes are treated as hyper-parameters which can be adjusted according to the model’s performance on the validation data. Refer to [19, Chapter 11] for more details on neural network.