Extract the information from big data with randomly distributed noise
-
Jin Cheng
Abstract
In this manuscript, a purely data-driven statistical regularization method is proposed for extracting the information from big data with randomly distributed noise.
Since the variance of the noise may be large, the method can be regarded as a general data preprocessing method in ill-posed problems, which is able to overcome the difficulty that the traditional regularization method is unable to solve, and has superior advantage in computing efficiency.
The unique solvability of the method is proved, and a number of conditions are given to characterize the solution.
The regularization parameter strategy is discussed, and the rigorous upper bound estimation of the confidence interval of the error in the
Funding source: National Natural Science Foundation of China
Award Identifier / Grant number: 11971121
Award Identifier / Grant number: 11871149
Funding statement: J. Cheng is supported by the NSFC (Grant No. 11971121). M. Zhong is supported by the NSFC (Grant No. 11871149) and supported by Zhishan Youth Scholar Program of SEU.
References
[1] R. A. Adams, Sobolev Spaces, Pure Appl. Math. 65, Academic Press, New York, 1975. Search in Google Scholar
[2] R. S. Anderssen and M. Hegland, For numerical differentiation, dimensionality can be a blessing!, Math. Comp. 68 (1999), no. 227, 1121–1141. 10.1090/S0025-5718-99-01033-9Search in Google Scholar
[3] J. Cheng, X. Z. Jia and Y. B. Wang, Numerical differentiation and its applications, Inverse Probl. Sci. Eng. 15 (2007), no. 4, 339–357. 10.1080/17415970600839093Search in Google Scholar
[4] P. Ciarlet, The Finite Element Method for Elliptic Problems, North-Holland, Amsterdam, 1978. 10.1115/1.3424474Search in Google Scholar
[5] P. Craven and G. Wahba, Smoothing noisy data with spline functions, Numer. Math. 31 (1978), 377–403. 10.1007/BF01404567Search in Google Scholar
[6] S. R. Deans, The Radon Transform and some of its Applications, John Wiley & Sons, New York, 1983. Search in Google Scholar
[7] E. J. M. Delhez, A spline interpolation technique that preserves mass budgets, Appl. Math. Lett. 16 (2003), no. 1, 17–26. 10.1016/S0893-9659(02)00139-8Search in Google Scholar
[8] R. Gorenflo and S. Vessella, Abel Integral Equations, Analysis and Applications, Lecture Notes in Math. 1461, Springer, Berlin, 1991. 10.1007/BFb0084665Search in Google Scholar
[9] C. W. Groetsch, Differentiation of approximately specified functions, Amer. Math. Monthly 98 (1991), no. 9, 847–850. 10.1080/00029890.1991.12000802Search in Google Scholar
[10] C. W. Groetsch, Optimal order of accuracy in Vasin’s method for differentiation of noisy functions, J. Optim. Theory Appl. 74 (1992), no. 2, 373–378. 10.1007/BF00940901Search in Google Scholar
[11] C. W. Groetsch, Lanczos’ generalized derivative, Amer. Math. Monthly 105 (1998), no. 4, 320–326. 10.1080/00029890.1998.12004888Search in Google Scholar
[12] C. W. Groetsch and O. Scherzer, Optimal order of convergence for stable evaluation of differential operators, Electron. J. Differential Equations 1993 (1993), Paper No. 04. Search in Google Scholar
[13] M. Hanke and O. Scherzer, Inverse problems light: Numerical differentiation, Amer. Math. Monthly 108 (2001), no. 6, 512–521. 10.1080/00029890.2001.11919778Search in Google Scholar
[14] D. N. Hào, L. H. Chuong and D. Lesnic, Heuristic regularization methods for numerical differentiation, Comput. Math. Appl. 63 (2012), no. 4, 816–826. 10.1016/j.camwa.2011.11.047Search in Google Scholar
[15] J. Huang and Y. Chen, A regularization method for the function reconstruction from approximate average fluxes, Inverse Problems 21 (2005), no. 5, 1667–1684. 10.1088/0266-5611/21/5/010Search in Google Scholar
[16] J. G. Huang and S. T. Xi, On the finite volume element method for general self-adjoint elliptic problems, SIAM J. Numer. Anal. 35 (1998), 1762–1774. 10.1137/S0036142994264699Search in Google Scholar
[17] X. Z. Jia, Y. B. Wang and J. Cheng, The numerical differentiation of scattered data and its error estimate, Numer. Math. J. Chinese Univ. 25 (2003), no. 1, 81–90. Search in Google Scholar
[18] S. A. Lu and Y. B. Wang, The numerical differentiation of first and second order with Tikhonov regularization, Numer. Math. J. Chinese Univ. 26 (2004), no. 1, 62–74. 10.1007/s11464-006-0014-xSearch in Google Scholar
[19] D. A. Murio, Automatic numerical differentiation by discrete mollification, Comput. Math. Appl. 13 (1987), no. 4, 381–386. 10.1016/0898-1221(87)90006-XSearch in Google Scholar
[20] A. G. Ramm and A. B. Smirnova, On stable numerical differentiation, Math. Comp. 70 (2001), no. 235, 1131–1153. 10.1090/S0025-5718-01-01307-2Search in Google Scholar
[21] L. B. Scott and L. R. Scott, Efficient methods for data smoothing, SIAM J. Numer. Anal. 26 (1989), no. 3, 681–692. 10.1137/0726040Search in Google Scholar
[22] G. Wahba, Smoothing noisy data with spline functions, Numer. Math. 24 (1975), no. 5, 383–393. 10.1007/BF01437407Search in Google Scholar
[23] Y. B. Wang, X. Z. Jia and J. Cheng, A numerical differentiation method and its application to reconstruction of discontinuity, Inverse Problems 18 (2002), no. 6, 1461–1476. 10.1088/0266-5611/18/6/301Search in Google Scholar
[24] Y. B. Wang and T. Wei, Numerical differentiation for two-dimensional scattered data, J. Math. Anal. Appl. 312 (2005), no. 1, 121–137. 10.1016/j.jmaa.2005.03.025Search in Google Scholar
[25] T. Wei, Y. C. Hon and Y. B. Wang, Reconstruction of numerical derivatives from scattered noisy data, Inverse Problems 21 (2005), no. 2, 657–672. 10.1088/0266-5611/21/2/013Search in Google Scholar
© 2021 Walter de Gruyter GmbH, Berlin/Boston
Articles in the same Issue
- Frontmatter
- The 𝑟-𝑑 class predictions in linear mixed models
- Stability results for weak solutions to backward one-dimensional semi-linear parabolic equations with locally Lipschitz source
- Parameter estimation for a chaotic dynamical system with partial observations
- Extract the information from big data with randomly distributed noise
- A data and knowledge driven approach for SPECT using convolutional neural networks and iterative algorithms
- Nonlinear dynamics of pulsating detonation wave with two-stage chemical kinetics in the shock-attached frame
- A partial inverse problem for quantum graphs with a loop
- Guided image filtering based ℓ0 gradient minimization for limited-angle CT image reconstruction
- Recovery of the time-dependent implied volatility of time fractional Black–Scholes equation using linearization technique
- Inverse problems for finite Jacobi matrices and Krein–Stieltjes strings
- Soliton orthogonal frequency division multiplexing with phase–frequency coding on the base of inverse scattering transform
Articles in the same Issue
- Frontmatter
- The 𝑟-𝑑 class predictions in linear mixed models
- Stability results for weak solutions to backward one-dimensional semi-linear parabolic equations with locally Lipschitz source
- Parameter estimation for a chaotic dynamical system with partial observations
- Extract the information from big data with randomly distributed noise
- A data and knowledge driven approach for SPECT using convolutional neural networks and iterative algorithms
- Nonlinear dynamics of pulsating detonation wave with two-stage chemical kinetics in the shock-attached frame
- A partial inverse problem for quantum graphs with a loop
- Guided image filtering based ℓ0 gradient minimization for limited-angle CT image reconstruction
- Recovery of the time-dependent implied volatility of time fractional Black–Scholes equation using linearization technique
- Inverse problems for finite Jacobi matrices and Krein–Stieltjes strings
- Soliton orthogonal frequency division multiplexing with phase–frequency coding on the base of inverse scattering transform
