Fuzzy neural network with backpropagation for fuzzy quadratic programming problems and portfolio optimization problems

1 Introduction

Neural network was introduced by Rosenblatt (1958) in his seminal paper. The technique was highly appraised by scientists and emerged as an important mechanism for solving the real world’s complex problems [1]. In the field of artificial intelligence and machine learning the neural network plays a significant role. The initial application of the neural network was reported by LeCun et al. [2], where it was adopted for recognizing the handwritten Zip codes. The technique was applied to pattern recognition problem in Bishop [3]. These series of successful implementations motivated researchers to explore the technique keenly and apply it to various real-world scenarios. Research studies have prioritized the steepest descent method of Jacobs over the momentum method of Polyak. The idea of Nesterov accelerated gradient approach evolved as the batch gradient descent and stochastic gradient descent. Moreover, the evolution of support vector machines (SVM) initiated a new era in the domain of machine learning and can be perceived as an extension of perceptron. The technique of SVM has been successfully implemented both for linear and nonlinear classification problems, whereby the problem is categorically divided into separating hyperplanes. Moreover, the SVM classification problems constructed quadratic program problems. Both the primal problem and the dual primal problem can be formulated as quadratic programming problems. The primal-dual method became insufficient as our observations went high, so, for the large-scale optimization LASSO regression method was developed. The importance of quadratic programming increased due to the fact that the LASSO solution developed the problem as an augmented quadratic programming problem.

Initially, very few optimization scientists had expertise with quadratic optimization [4]; like Mann, Markowitz can be justly recognized as the “father of quadratic programming.” In 1956, Markowitz put forward a technique to solve quadratic programming problems [5]. In studies [6,7,8], different approaches were presented to quadratic programming problems.

Quadratic programming problems gained prominence in many areas and played a pivotal role in many scientific and engineering problems. Quadratic programming is a noteworthy approach in which the objective function is quadratic subject to linear constraint. Moré and Toraldo [9], solved quadratic programming problems with bounded constraints. In [10], a complete bibliography of the development of quadratic programming techniques was presented. In the early 90’s, the researchers adopted traditional optimization techniques using artificial neural networks. Linear and quadratic programming problems were solved with the help of neural networks by Maa and Shanblalt and Xia [11,12]. The authors studied bounded quadratic programming problems using the neural network approach [13]. Zak et al. [14] proposed a neural network approach for solving linear programming problems. High-performance neural network was used by Wu et al. [15] for solving linear and quadratic problems. Xia and Wang [16] tried primal neural networks for solving convex quadratic programming problems.

The technique of portfolio optimization theory was first introduced in the seminal paper of Markowitz [17], and after that became a noteworthy topic for emerging researchers. Morkowitz explained the anticipated return of the portfolio as a desiring phenomenon and the fluctuations of the returns undesirable. The mean-variance model of the Markowitz efficiently computed the optimized portfolio [18]. In the study by Richard and Roncalli [19], risk-constrained portfolio optimization problems were addressed. Diversification in investment strategies using maximum entropy approach was addressed by Bera and Park [20]. Brodie et al. [21] addressed the concept of stable investment strategies. The successful and incredible implementations of artificial intelligence and machine learning approaches have attracted researchers to reconsider conventional portfolio optimization. Markowitz proposed a quadratic programming model for solving portfolio optimization problems [22]. Financial specialists proposed models to minimize the risk of returns and maximize their benefits in portfolio optimization problems [23,24,25]. Financial specialists broadly utilized the mean-variance optimization model. Roa’s quadratic entropy [26] maximized portfolio returns by ordering and diversification. Bayesian optimization approach was adopted by Gonzalvez et al. [27]. Lezmi et al. [28] studied portfolio allocation problems with skewed risk. Perrin and Roncalli [29] used the CCD algorithm for solving portfolio optimization problems.

The idea of fuzzy sets proposed by Zadeh [30] has significant applications in the fields of science, technology, engineering, and management. Fuzzy modeling has been applied to optimization problems [31,32,33,34,35].

The idea of fuzzy modeling has been extensively used in the decision-making process. Ala et al. [36] used Multi-Objective Grey Wolf Optimization for the security and efficiency of green energy systems. Riaz et al. [37] studied the efficiency of the supply chain system in a city using a fuzzy decision-making process. Jana et al. [38] evaluated the performance of green sustainable approaches for parcel delivery using fuzzy integral techniques. Alsattar et al. [39] utilized three-stage decision-making process with Pythagorean fuzzy sets for the development of a green sustainable lifestyle. Khan et al. [40] proposed a simplified technique to solve fuzzy linear programming problems. Khan and Khan [41] solved intuitionistic fuzzy linear programming problem using duality properties. In the study by Khan and Karam [42], Data Envelopment Analysis DEA techniques were adopted to measure the efficiency of American Airlines. The study by Faiza and Khalil [43] adopted machine learning techniques to predict flight delay problems. Machine learning techniques were adopted in Zhang et al., [44] for various real world problems. Khan and Aftab [45] utilized dynamic programming problems to address fuzzy linear programming problems. In the study by Silva et al. [46], two-phase approach was used to solve fuzzy quadratic programming problems. Liu [47,48] used fuzzy parameters in solving quadratic programming problems. Quadratic programming problems were addressed by Kheirfam [49] and sensitivity analysis by Kheifam et al. [50]. Molai [51] used fuzzy relational techniques for quadratic programming problems. Probability approaches were adopted by Barik and Biswal [52]. In the study by Gani and Kumar [53], pivoting techniques were used to solve fuzzy quadratic programming problems. The study by Zhou [54] proved the necessary optimality conditions for fuzzy quadratic programming problems. Bai and Bao [55] solved quadratic programming problems in a fuzzy uncertain environment. In the study by Ghanbari and Moghadam [56], an ABS algorithm was adopted to solve the fuzzy quadratic programming problem. Umamaheswari and Ganesan [57] proposed a new method for fuzzy quadratic programming problems. Dasril et al. [58] presented an enhanced constraint method to solve fuzzy quadratic programming problems. Elshafei [59] addressed a fuzzy quadratic programming problem in which the variables were unrestricted. In the studies by Rout et al. and Biswal and De [60,61], multi-objective quadratic programming problems were addressed. In the study by Fathy [62], the multi-objective problem with integer variables was addressed. The study by Khalifa [63] presented an interactive method for multi-objective quadratic programming. In the study by Taghi-Nezhad and Babakordi [64], the problems were decomposed into sub-problems for solution. Fuzzy quadratic programming was successfully applied in irrigation systems [65] and irrigation planning and portfolio optimization [66]. Neural networks were used in the study by Malek and Oskoei [67] to solve constrained quadratic problems. Fuzzy nonlinear programming problems were solved with neural network techniques in [68]. Mansoori et al. [69] utilized neural networks for solving fuzzy quadratic programming problems. In the study by Coelho [70], a dual method is used to solve fuzzy quadratic programming problems.

In this study, a backpropagation fuzzy neural network is used to solve fuzzy quadratic programming problems. The method is applied to the portfolio optimization problem in the Pakistan Stock Exchange. Table 1 presents a summary of the techniques used in the literature and the gap of the proposed back propagation neural network over the existing techniques. It depicts that different methods are used to solve fuzzy quadratic programming problems, and the proposed backpropagation neural network techniques have not been applied so far for the same. Thus, the proposed method has a methodological and population gap with the existing techniques.

The statement of the proposed research problem is “How back propagation fuzzy neural network can be utilized to address the quadratic programming problems in fuzzy uncertain environment?”

The objectives of the research are as follows:

1. To study back propagation fuzzy neural network to solve quadratic programming problems in a fuzzy uncertain environment.

2. To implement the proposed methodology to the mean-variance Markowitz model in capital market structure.

3. To solve the model on Python machine learning software using the PyTorch module.

4. To implement the proposed approach on the datasets of six leading companies in the Pakistan Stock Exchange.

5. To analyze the results and propose suggestions for optimal portfolio management.

The main contributions of the proposed study are listed as follows:

1. Incorporation of backpropagation techniques to solve fuzzy quadratic programming problems with the techniques of fuzzy neural network.

2. Construction of Lagrange’s function in matrix form and chain rule for backpropagation in a fuzzy quadratic programming problem.

3. Python machine learning software is used to solve the proposed model. The module used is PyTorch.

4. Implementing the proposed technique for portfolio optimization in the Pakistan Stock Exchange.

5. Analysis of results and insight into perspective investors in Pakistan Stock Exchange.

The foremost advantages of the proposed technique are listed as follows:

1. The main advantage of the proposed back propagation neural network is that it can easily adjust and fine-tune the weights of the network from the error rate obtained at the previous layer.

2. The chain and power rules of the derivative allow backpropagation and successively update the weights of the network to perform efficiently.

3. Thus, the gradient of the loss function is calculated by iterating backward layer by layer but one at a time to reduce the difference between the desired and the predicted outputs.

4. The prime utility of the proposed back propagation model is its successful implementation in the Pakistan Stock Exchange. In all three cases of the implementation, the convergence is obtained at 475 iterations. Thus, it is faster than the previous studies.

5. The proposed technique brings an improvement of 28.77% in the objective function of the mean variance optimization MVO which is greater than reported in the literature.

An epitomic road map of the proposed study is presented below and shown in Figure 1.

1. Proposition of the fuzzy quadratic programming problem with back propagation neural network.

2. Formulate the lower model of the proposed fuzzy quadratic programming problem with a back propagation neural network.

3. Formulate the central model of the proposed fuzzy quadratic programming problem with a back propagation neural network.

4. Formulate the upper model of the proposed fuzzy quadratic programming problem with a backpropagation neural network.

5. Implement the proposed fuzzy QP model to mean-variance optimization MVO in the Pakistan Stock Exchange.

6. Formulate the lower fuzzy QP model for the mean-variance optimization MVO in the Pakistan Stock Exchange.

7. Formulate the central fuzzy QP model for the mean-variance optimization MVO in Pakistan Stock Exchange.

8. Formulate the upper fuzzy QP model for the mean-variance optimization MVO in Pakistan Stock Exchange.

9. Sole all three models (lower, central, and upper) using Python machine learning software and obtain results using the PyTorch module.

10. Combine the Results of all three models and present a single fuzzy solution.

11. Compare and contrast the results with existing studies in the literature.

12. Conclusions and summary of the research.

Figure 1

Road map of the proposed study.

This study is organized as follows. After the introduction, the backpropagation method for solving quadratic programming problems with neural networks is given in Section 2. This is followed by Section 3 in which a backpropagation method for solving fuzzy quadratic programming problems with neural networks is proposed. In Section 4, the proposed approach adopted in three is converted into lower, central, and upper models. In Section 5, the proposed methodology is adopted for portfolio optimization problems in Pakistan Stock Exchange. This section has two subsections, 5.1 and 5.2, for the lower and upper models, respectively. Results and discussions are given in Section 6, and the study is concluded in Section 7.

2 Back propagation artificial neural network for quadratic programming problems

A quadratic layer program using a neural network can be defined as (1).

s.t.

where the current layer is z i + 1 ∈ R n , the previous layer is z i ∈ R n , optimization variable is z ∈ R n , and parameters of the optimization problem are Q ( z i ) ∈ R p , p ( z i ) ∈ R n , A ( z i ) ∈ R m ⁎ n , b ( z i ) ∈ R m , G ( z i ) ∈ R p ⁎ n , and h ( z i ) ∈ R p . These parameters depend in a differentiable manner on the past layer z i , and can be optimized by adjusting its weights in a neural network. The backpropagation model consists of a reverse pass as well as a forward pass. It is required to determine the gradient of loss function in optimizing the QP problem with respect to the variable parameters. To get these derivatives, we separate the Karus Kuhn Tucker (KKT) conditions (sufficient and necessary conditions for optimality) of the QP problem and arrange it utilizing techniques from matrix differential calculus. The Lagrangian of the QP problem is given (2).

where λ ≥ 0 are the dual multipliers on the inequality constraints and ν ≥ 0 are the dual multipliers of the equality constraints. For stationary, primal feasibility, and complementary slackness, the KKT conditions are (3).

Here, D(·) is used for creating the diagonal matrix from the vector. Introducing slacks for inequality constraints in (1) and iteratively minimizing the residuals from KKT conditions over primal and dual problems. The affine scale directions are calculated as (4).

The matrix K in (4) is (5).

The center and correct direction is given by (6).

Here, µ is the duality gap and ρ > 0 . Moreover, “v” is updated as Δ v = Δ v aff + Δ v cc . In the forward pass, we introduce (7) to control the forward pass.

In K sym , only D λ s changes between successive iterations. Thus, in the forward pass, the system is decomposed into smaller symmetric systems like (7). For backward pass, the KKT of (2) are given (8).

The model (9) is obtained by taking the differential of the conditions (8).

Model (9) in matrix form is (10).

Utilizing these equations, we find the Jacobin of z* (or λ* and v*) with regard to any parameters. For example, in the event to compute the Jacobian ∂ z ∂ b ∈ R m ⁎ n , substitute db = I (and set all other differential terms within the right-hand side to zero), solve the equation, and the resulting value of d z would be the specified Jacobin. In the backpropagation calculation, we need to multiply the left matrix-vector product with the past reverse pass vector ∂ l ∂ z ⁎ ∈ R n , for example ∂ l ∂ b = ∂ l ∂ z ⁎ . ∂ z ⁎ ∂ b . Thus, we calculate (dz, dλ, dν) by multiplying the inverse of the left-hand-side matrix with the right-hand sides. This can be done by multiplying the reverse pass vector by the transpose of the differential matrix as shown in (11).

Then, with respect to all QP parameters, the relevant gradients can be given by (12).

Whereas all these terms are at most the size of the parameter matrices in standard backpropagation. Finally, system (13) presents the symmetric matrix computing the backpropagated gradient.

where d ∼ λ = D ( λ ⁎ ) d λ . Thus, all the reverse pass gradients can be computed using factorized KKT matrix.

3 Proposed back propagation fuzzy neural network for fuzzy quadratic programming problems

Fuzzy quadratic layer in a neural network using Klir and Yuan’s study [71] can be presented as (14).

s.t.

where the current layer is ( z i + 1 ) ( s , l , r ) ∈ R n , the previous layer is ( z i ) ( s , l , r ) ∈ R n , the optimization variable is z ( s , l , r ) ∈ R n , and the parameters of the proposed optimization problem are ( Q ( z i ) ) ( s , l , r ) ∈ R p , ( p ( z i ) ) ( s , l , r ) ∈ R n , ( A ( z i ) ) ( s , l , r ) ∈ R m ⁎ n , ( b ( z i ) ) ( s , l , r ) ∈ R m , ( G ( z i ) ) ( s , l , r ) ∈ R p ⁎ n , and ( h ( z i ) ) ( s , l , r ) ∈ R p . These parameters can depend in differentiable ways on the past layer ( z i ) ( s , l , r ) and can be optimized similarly to other weights in a neural network. An efficient model requires having a reverse pass as well as a forward pass. It is required to determine the derivative of the solution to the QP with respect to the variable parameters. The Lagrangian of problem is given by (15).

where λ ≥ 0 are the dual multipliers of the inequality constraints and ν ≥ 0 for equality constraints. For KKT conditions are (16).

Here, D(·) creates the diagonal matrix of the vector, introducing slacks for inequality constraints in (14), and minimizing the residuals from KKT conditions over primal and dual problems. The affine scale directions are calculated as (17).

The matrix K ^{(s, l, r)} in (17) is (18).

The center and correct direction is presented (19).

Here, µ is the duality gap and ρ > 0 . Moreover, “v” is updated as Δ v ( s , l , r ) = Δ v aff ( s , l , r ) + Δ v cc ( s , l , r ) . In the forward pass, we introduce (20) to control the forward pass.

In K sym ( s , l , r ) only D λ s changes between successive iterations. Thus, in forward pass, the system is decomposed into smaller symmetric systems (20).

For backpropagation, the KKT of (15) are given (21).

Taking the differentials of (21), gives (22).

Model (22) in matrix form gives (23).

These equations help find the Jacobin of z ⁎ ^{(s, l, r)} (or λ ⁎ ^{(s, l, r)} and v ⁎ ^{(s, l, r)}) with regard to any parameters. For example, Jacobin ∂ z ∂ b ( s , l , r ) ∈ R m ⁎ n , put the value of db ^(s,l,r) = I ^(s,l,r) and set all other differential terms within the right-hand side to zero. Then, solve the equation and the resulting value of dz ^(s,l,r) gives the specified Jacobin. In the backpropagation calculation, multiply the left matrix-vector product with some past reverse pass vector ∂ l ∂ z ⁎ ( s , l , r ) ∈ R n , for example ∂ l ∂ b ( s , l , r ) = ∂ l ∂ z ⁎ ⋅ ∂ z ⁎ ∂ b ( s , l , r ) , thus calculating (dz, dλ, dν)^(s,l,r) by multiplying the inverse of the left-hand-side matrix with the right-hand sides. Thus, the backward pass vector is multiplied by the transpose of the differential matrix as (24).

The gradients are presented in (25).

Finally, (26) presents the symmetric matrix computing the backpropagated gradients.

where d ∼ λ ( s , l , r ) = ( D ( λ ⁎ ) d λ ) ( s , l , r ) . Thus, factorized KKT matrix gives all the gradients in the backpropagation path.

4 The proposed backpropagation fuzzy artificial neural network for lower, central, and upper bounds for quadratic programming problem model

Fuzzy quadratic layer program in triangular fuzzy numbers can be given as follows (27):

s.t.

The current layer is ( z i + 1 ) ( s − l , s , s + r ) ∈ R n , previous layer is ( z i ) ( s − l , s , s + r ) ∈ R n , optimization variable is z ( s − l , s , s + r ) ∈ R n , and parameters of the optimization problem are ( Q ( z i ) ) ( s − l , s , s + r ) ∈ R p , ( p ( z i ) ) ( s − l , s , s + r ) ∈ R n , ( A ( z i ) ) ( s − l , s , s + r ) ∈ R m ⁎ n , ( b ( z i ) ) ( s − l , s , s + r ) ∈ R m , ( G ( z i ) ) ( s − l , s , s + r ) ∈ R p ⁎ n , and ( h ( z i ) ) ( s − l , s , s + r ) ∈ R p . The parameters depend on the past layer ( z i ) ( s − l , s , s + r ) , which can be optimized by updating the weights of the neural network.

For backpropagation, it is required to determine the derivative of the solution to the QP problem with respect to the variable parameters. The Lagrangian of the problem is (28).

where λ ≥ 0 are the dual multipliers of the inequality constraints and ν ≥ 0 are the dual multipliers of the equality constraints. The KKT conditions are (29).

Here, D ( . ) creates the diagonal matrix from the vector. We introduce slacks for inequality constraints in (27) and minimize the residuals from KKT conditions. The affine scale directions are calculated (30).

The value of K ^{(s−l,s,s+r)} in (30) is (31).

The center and correct direction is presented (32).

Here, μ is the duality gap and ρ > 0 . Moreover, “v” is updated as Δ v ( s − l , s , s + r ) = Δ v aff ( s − l , s , s + r ) + Δ v cc ( s − l , s , s + r ) . The system (33) controls the forward pass.

In the matrix, K sym ( s − l , s , s + r ) only D λ s changes between successive iterations. Thus, in forward pass, the system is decomposed into smaller symmetric systems as (33). For reverse pass, the KKT of (28) are (34).

The model (35) is obtained by taking the differentials of (34).

Model (35) in matrix form is (36).

These equations help find the Jacobin of z ⁎ ^{(s–l,s,s+r)} (or λ ⁎ ^{(s–l,s,s+r)} and v ⁎ ^{(s–l,s,s+r)}) with regard to any parameters. To compute the Jacobin ∂ z ∂ b ( s − l , s , s + r ) ∈ R m ⁎ n , substitute db ^{(s−l, s, s+r)} = I ^{(s−l, s, s+r)} and set all other differential terms within the right-hand side to zero. Solving for dz ^{(s−l, s, s+r)} would be the specified Jacobin. In the backpropagation calculation, the left matrix-vector is multiplied with past reverse pass vector ∂ l ∂ z ⁎ ( s − l , s , s + r ) ∈ R n , for example, ∂ l ∂ b ( s − l , s , s + r ) = ∂ l ∂ z ⁎ ⋅ ∂ z ⁎ ∂ b ( s − l , s , s + r ) . Thus, we are able to calculate (dz,dλ,dν)^{(s−l, s, s+r)} by multiplying the inverse of the left-hand-side matrix with the right-hand sides. Thus, the reverse pass vector is multiplied by the transpose of the differential matrix as (37).

The gradients with respect to QP parameters are presented in (38).

Finally, the system (39) presents the symmetric matrix computing the backpropagated gradients.

Here, d ∼ λ ( s − l , s , s + r ) = ( D ( λ ⁎ ) d λ ) ( s − l , s , s + r ) . Thus, gradients in the backpropagation path are calculated.

5 Proposed QP model formulation for portfolio optimization in Pakistan stock exchange

In this section, the proposed fuzzy backpropagation neural network is applied to the fuzzy quadratic portfolio optimization problem in the Pakistan Stock Exchange. Markowitz’s MVO model is utilized to formulate the problem. The solutions identify profitable portfolios for potential investors in the Pakistan Stock Exchange based on historical records of the assets. Returns data of the six leading stocks traded in the Pakistan stock exchange are taken into consideration. The companies include Archroma Pakistan Limited (ARPL), IGI Holdings Limited. (IGIHL). International Industries Limited (INIL), Atlas Honda Limited (ATLH), MCB Bank Limited (MCB), and Pakistan Oilfields Limited (POL). For each asset, monthly data between January 2016 and October 2020 of the returns (close price) is obtained from Data Portal, Pakistan Stock Exchange [72]. Table 2 represents the closing prices for the selected stocks from January 2016 to October 2020 [72].

Let’s k i l stands for above “Total returns” for assets i = 1 , 2 , 3 , . . . , 6 , and l = 1 , 2 , 3 , … , L where l = 0 corresponds to January 2016 and l = L to October 2020. Utilizing (40), the source data k i l , l = 0 , 1 , 2 , … , L , be transformed into rates of return r i l for assets i = 1 , 2 , 3 , . . . , 6 as given in Table 3.

From (40), the arithmetic mean and geometric mean for each asset can be calculated (41) and (42) and shown in Table 4.

Covariance and volatility are calculated in (43) and (44) and are shown in Tables 4 and 5, respectively.

The geometric mean is utilized rather than the arithmetic mean because the return rate is multiplicative over period.

Finally the correlation matrix is Table 6.

Ultimately, the central quadratic programming problem for portfolio optimization is constructed.

The proposed central quadratic programming mean-variance model for the portfolio optimization is (45) from Tables 4 and 5.

s.t