Startseite Kalman filter with impulse noised outliers: a robust sequential algorithm to filter data with a large number of outliers
Artikel
Lizenziert
Nicht lizenziert Erfordert eine Authentifizierung

Kalman filter with impulse noised outliers: a robust sequential algorithm to filter data with a large number of outliers

  • Bertrand Cloez EMAIL logo , Bénédicte Fontez , Eliel González-García und Isabelle Sanchez
Veröffentlicht/Copyright: 17. April 2024
Veröffentlichen auch Sie bei De Gruyter Brill
Manuskript einreichen Informationen für Autor*innen
The International Journal of Biostatistics
Aus der Zeitschrift The International Journal of Biostatistics Band 20 Heft 2

Abstract

Impulse noised outliers are data points that differ significantly from other observations. They are generally removed from the data set through local regression or the Kalman filter algorithm. However, these methods, or their generalizations, are not well suited when the number of outliers is of the same order as the number of low-noise data (often called nominal measurement). In this article, we propose a new model for impulsed noise outliers. It is based on a hierarchical model and a simple linear Gaussian process as with the Kalman Filter. We present a fast forward-backward algorithm to filter and smooth sequential data and which also detects these outliers. We compare the robustness and efficiency of this algorithm with classical methods. Finally, we apply this method on a real data set from a Walk Over Weighing system admitting around 60 % of outliers. For this application, we further develop an (explicit) EM algorithm to calibrate some algorithm parameters.

Keywords: outlier detection; filter algorithm; Gaussian model; EM algorithm; walk over weighing
Corresponding author: Bertrand Cloez, MISTEA, Univ Montpellier, INRAE, Institut Agro, Montpellier, France, E-mail:

Funding source: TechCare

Award Identifier / Grant number: 862050

  1. Research ethics: Not applicable.

  2. Author contributions: The authors have accepted responsibility for the entire content of this manuscript and approved its submission.

  3. Competing interests: The authors state no conflict of interest.

  4. Research funding: This work was financially supported by the European Union's Horizon 2020 Innovation Action program for funding the project Integrating innovative TECHnologies along the value Chain to improve small ruminant welfARE management (TechCare; grant agreement 862050).

  5. Data availability: The raw data can be obtained on request from the corresponding author.

References

1. Fisch, ATM, Eckley, IA, Fearnhead, P. Innovative and additive outlier robust Kalman filtering with a robust particle filter. IEEE Trans Signal Process 2022;70:47–56. https://doi.org/10.1109/tsp.2021.3125136.Suche in Google Scholar

2. Pimentel, MAF, Clifton, DA, Clifton, L, Tarassenko, L. A review of novelty detection. Signal Process 2014;99:215–49. https://doi.org/10.1016/j.sigpro.2013.12.026.Suche in Google Scholar

3. Chandola, V, Banerjee, A, Kumar, V. Anomaly detection: a survey. ACM Comput Surv 2009;41:1–58. https://doi.org/10.1145/1541880.1541882.Suche in Google Scholar

4. Maity, A, Chatterjee, R. Impulsive noise in images: a brief review. Comput Vis Graph Image Process 2018;4:6–15. https://doi.org/10.19101/tipcv.2017.39025.Suche in Google Scholar

5. Pardeshi, V, Patil, R. Suppression of random valued impulse noise in image processing: a review. Int J Adv Res Electrical Electron Instrum Eng 2015;4. https://doi.org/10.15662/ijareeie.2015.0401017.Suche in Google Scholar

6. Cappé, O, Moulines, E, Ryden, T. Inference in hidden Markov models. New York, NY: Springer; 2007.Suche in Google Scholar

7. Del Moral, P. Mean field simulation for Monte-Carlo integration. Monogr Stat Appl Probab 2013;126:26.10.1201/b14924Suche in Google Scholar

8. Yao, H, Zhu, F. Using PSO particle filter to recovery image. In: 2011 3rd international workshop on intelligent systems and applications; 2011:1–4 pp.10.1109/ISA.2011.5873251Suche in Google Scholar

9. Pai, P, Cong, C, Yongsheng, Y. Particle swarm optimization based on hybrid Kalman filter and particle filter. J Shanghai Jiaot Univ 2020;25:681–8. https://doi.org/10.1007/s12204-020-2229-5.Suche in Google Scholar

10. Anderson, BDO, Moore, JB. Optimal filtering. Mineola, New York: Dover Publications; 2012.Suche in Google Scholar

11. Gelb, A, Kasper, JFJr., Nash, RAJr., Prices, CFJr., Sutherland, AAJr. Applied optimal estimation. Massachusetts: MIT Press; 1974.Suche in Google Scholar

12. Kalman, RE, Bucy, RS. New results in linear filtering and prediction theory. ASME. J Basic Eng 1961;83:95–108. https://doi.org/10.1115/1.3658902.Suche in Google Scholar

13. Kalman, RE. A new approach to linear filtering and prediction problems. J Basic Eng 1960;82:35–45. https://doi.org/10.1115/1.3662552.Suche in Google Scholar

14. Maybeck, PS. Stochastic models, estimation, and control. New York, San Francisco, London: Academic Press; 1982.Suche in Google Scholar

15. Ting, J-A, Theodorou, E, Schaal, S. Machine learning: ECML 2007. In: 18th European conference on machine learning. Warsaw, Poland: Springer; 2007:748–56 pp.Suche in Google Scholar

16. Morris, J. The Kalman filter: a robust estimator for some classes of linear quadratic problems. IEEE Trans Inf Theor 1976;22:526–34. https://doi.org/10.1109/tit.1976.1055611.Suche in Google Scholar

17. Agamennoni, G, Nieto, JI, Nebot, EM. An outlier-robust Kalman filter. In: 2011 IEEE international conference on robotics and automation. IEEE; 2011:1551–8 pp.10.1109/ICRA.2011.5979605Suche in Google Scholar

18. Masreliez, C, Martin, R. Robust Bayesian estimation for the linear model and robustifying the Kalman filter. IEEE Trans Automat Control 1977;22:361–71. https://doi.org/10.1109/tac.1977.1101538.Suche in Google Scholar

19. Meinhold, RJ, Singpurwalla, ND. Robustification of Kalman filter models. J Am Stat Assoc 1989;84:479–86. https://doi.org/10.2307/2289933.Suche in Google Scholar

20. Schick, IC, Mitter, SK. Robust recursive estimation in the presence of heavy-tailed observation noise. Ann Stat 1994;22:1045–80. https://doi.org/10.1214/aos/1176325511.Suche in Google Scholar

21. Sorenson, HW, Alspach, DL. Recursive Bayesian estimation using Gaussian sums. Automatica 1971;7:465–79. https://doi.org/10.1016/0005-1098(71)90097-5.Suche in Google Scholar

22. West, M. Robust sequential approximate Bayesian estimation. J Roy Stat Soc B 1981;43:157–66. https://doi.org/10.1111/j.2517-6161.1981.tb01165.x.Suche in Google Scholar

23. Wu, WR, Kundu, A. Recursive filtering with non-Gaussian noises. IEEE Trans Signal Process 1996;44:1454–68. https://doi.org/10.1109/78.506611.Suche in Google Scholar

24. Huber, PJ. Robust estimation of a location parameter. Ann Math Stat 1964;35:73–101. https://doi.org/10.1214/aoms/1177703732.Suche in Google Scholar

25. Cipra, T, Romera, R. Kalman filter with outliers and missing observations. Test 1997;6:379–95. https://doi.org/10.1007/bf02564705.Suche in Google Scholar

26. Zoubir, AM, Koivunen, V, Ollila, E, Muma, M. Robust statistics for signal processing. Cambridge: Cambridge University Press; 2018.10.1017/9781139084291Suche in Google Scholar

27. Chang, L, Li, K. Unified form for the robust Gaussian information filtering based on M-estimate. IEEE Signal Process Lett 2017;24:412–16. https://doi.org/10.1109/lsp.2017.2669238.Suche in Google Scholar

28. Gandhi, MA, Mili, L. Robust Kalman filter based on a generalized maximum-likelihood-type estimator. IEEE Trans Signal Process 2009;58:2509–20. https://doi.org/10.1109/tsp.2009.2039731.Suche in Google Scholar

29. Huang, Y, Zhang, Y, Zhao, Y, Chambers, JA. A novel robust Gaussian–student’s t mixture distribution based Kalman filter. IEEE Trans Signal Process 2019;67:3606–20. https://doi.org/10.1109/tsp.2019.2916755.Suche in Google Scholar

30. Durovic, ZM, Kovacevic, BD. Robust estimation with unknown noise statistics. IEEE Trans Automat Control 1999;44:1292–6. https://doi.org/10.1109/9.769393.Suche in Google Scholar

31. González-García, E, Alhamada, M, Nascimento, H, Portes, D, Bonnafe, G, Allain, C, et al.. Measuring liveweight changes in lactating dairy ewes with an automated walk-over-weighing system. J Dairy Sci 2021;104:5675–88. https://doi.org/10.3168/jds.2020-19075.Suche in Google Scholar PubMed

32. González-García, E, de Oliveira Golini, P, Hassoun, P, Bocquier, F, Hazard, D, González, LA, et al.. An assessment of walk-over-weighing to estimate short-term individual forage intake in sheep. Animal 2018;12:1174–81. https://doi.org/10.1017/s1751731117002609.Suche in Google Scholar

33. González-García, E, Alhamada, M, Pradel, J, Douls, S, Parisot, S, Bocquier, F, et al.. A mobile and automated walk-over-weighing system for a close and remote monitoring of liveweight in sheep. Comput Electron Agric 2018;153:226–38. https://doi.org/10.1016/j.compag.2018.08.022.Suche in Google Scholar

34. Leroux, E, Llach, I, Besche, G, Guyonneau, JD, Montier, D, Bouquet, PM, et al.. Evaluating a walk-over-weighing system for the automatic monitoring of growth in postweaned Mérinos d'Arles ewe lambs under Mediterranean grazing conditions. Animal – Open Space 2023;2:100032.10.1016/j.anopes.2022.100032Suche in Google Scholar

35. Rauch, HE, Tung, F, Striebel, CT. Maximum likelihood estimates of linear dynamic systems. AIAA J 1965;3:1445–50. https://doi.org/10.2514/3.3166.Suche in Google Scholar

36. Murphy, KP. Switching Kalman filters, technical report. Berkley: University of California; 1998.Suche in Google Scholar

37. Murphy, KP. Dynamic Bayesian networks: representation, inference and learning. Berkeley: University of California; 2002.Suche in Google Scholar

38. Nguyen, LH, Goulet, J-A. Anomaly detection with the switching Kalman filter for structural health monitoring. Struct Control Health Monit 2018;25:e2136. https://doi.org/10.1002/stc.2136.Suche in Google Scholar

39. Royer, F, Lutcavage, M. Filtering and interpreting location errors in satellite telemetry of marine animals. J Exp Mar Biol Ecol 2008;359:1–10. https://doi.org/10.1016/j.jembe.2008.01.026.Suche in Google Scholar

40. Speekenbrink, M, Shanks, DR. Learning in a changing environment. J Exp Psychol Gen 2010;139:266. https://doi.org/10.1037/a0018620.Suche in Google Scholar PubMed

41. Wu, W, Black, MJ, Mumford, D, Gao, Y, Bienenstock, E, Donoghue, JP. Modeling and decoding motor cortical activity using a switching Kalman filter. IEEE Trans Biomed Eng 2004;51:933–42. https://doi.org/10.1109/tbme.2004.826666.Suche in Google Scholar PubMed

42. Baum, LE, Petrie, T. Statistical inference for probabilistic functions of finite state Markov chains. Ann Math Stat 1966;37:1554–63. https://doi.org/10.1214/aoms/1177699147.Suche in Google Scholar

43. MacDonald, IL, Zucchini, W Hidden Markov and other models for discrete-valued time series. CRC Press; 1997, vol 110.Suche in Google Scholar

44. Bar-Shalom, Y, Fortmann, TE, Cable, PG. Tracking and data association. J Acoust Soc Am 1990;87:918–9.10.1121/1.398863Suche in Google Scholar

45. Kim, C, Li, F, Ciptadi, A, Rehg, JM. Multiple hypothesis tracking revisited. In: Proceedings of the IEEE international conference on computer vision; 2015:4696–704 pp.10.1109/ICCV.2015.533Suche in Google Scholar

46. Boyen, X, Koller, D. Tractable inference for complex stochastic processes. arXiv preprint arXiv:1301.7362; 2013.Suche in Google Scholar

47. Gorynin, I, Gangloff, H, Monfrini, E, Pieczynski, W. Assessing the segmentation performance of pairwise and triplet Markov models. Signal Process 2018;145:183–92. https://doi.org/10.1016/j.sigpro.2017.12.006.Suche in Google Scholar

48. Lehmann, F, Pieczynski, W. Suboptimal Kalman filtering in triplet Markov models using model order reduction. IEEE Signal Process Lett 2020;27:1100–4. https://doi.org/10.1109/lsp.2020.3002420.Suche in Google Scholar

49. Chen, Y, Li, W, Wang, Y. Robust Gaussian approximate fixed-interval smoother with outlier detection. IEEE Signal Process Lett 2020;27:1505–9. https://doi.org/10.1109/lsp.2020.3015422.Suche in Google Scholar

50. Li, H, Medina, D, Vilà-Valls, J, Closas, P. Robust variational-based Kalman filter for outlier rejection with correlated measurements. IEEE Trans Signal Process 2020;69:357–69. https://doi.org/10.1109/tsp.2020.3042944.Suche in Google Scholar

51. Wang, H, Li, H, Fang, J, Wang, H. Robust Gaussian Kalman filter with outlier detection. IEEE Signal Process Lett 2018;25:1236–40. https://doi.org/10.1109/lsp.2018.2851156.Suche in Google Scholar

52. González, LA, Bishop-Hurley, G, Henry, D, Charmley, E. Wireless sensor networks to study, monitor and manage cattle in grazing systems. Anim Prod Sci 2014;54:1687–93. https://doi.org/10.1071/an14368.Suche in Google Scholar

53. Le Gall, JF. Brownian motion, martingales, and stochastic calculus. Switzerland: Springer International Publishing; 2016, 274.10.1007/978-3-319-31089-3Suche in Google Scholar

54. Supplementary material available on line. Available from: https://forgemia.inra.fr/isabelle.sanchez/kfino_outputs.Suche in Google Scholar

55. Cloez, B, Sanchez, I. kfino: Kalman filter for impulse noised outliers. R package version 1.0.0; 2022.10.32614/CRAN.package.kfinoSuche in Google Scholar

56. Fitzgerald, DW, Murphy, FE, Wright, WMD, Whelan, PM, Popovici, EM. Design and development of a smart weighing scale for beehive monitoring. In: 2015 26th Irish Signals and Systems Conference (ISSC); 2015:1–6 pp.10.1109/ISSC.2015.7163763Suche in Google Scholar

57. Nyalala, I, Okinda, C, Kunjie, C, Korohou, T, Nyalala, L, Chao, Q. Weight and volume estimation of poultry and products based on computer vision systems: a review. Poultry Sci 2021;100:101072. https://doi.org/10.1016/j.psj.2021.101072.Suche in Google Scholar PubMed PubMed Central

58. Brunet, H, Creach, P, Concordet, D. Optimal estimation of broiler movement for commercial tracking. Smart Agric Technol 2023;3:100113. https://doi.org/10.1016/j.atech.2022.100113.Suche in Google Scholar

Supplementary Material

This article contains supplementary material (https://doi.org/10.1515/ijb-2023-0065).

Received: 2023-06-28
Accepted: 2024-03-05
Published Online: 2024-04-17

© 2024 Walter de Gruyter GmbH, Berlin/Boston

Artikel in diesem Heft

  1. Frontmatter
  2. Research Articles
  3. Random forests for survival data: which methods work best and under what conditions?
  4. Flexible variable selection in the presence of missing data
  5. An interpretable cluster-based logistic regression model, with application to the characterization of response to therapy in severe eosinophilic asthma
  6. MBPCA-OS: an exploratory multiblock method for variables of different measurement levels. Application to study the immune response to SARS-CoV-2 infection and vaccination
  7. Detecting differentially expressed genes from RNA-seq data using fuzzy clustering
  8. Hypothesis testing for detecting outlier evaluators
  9. Response to comments on ‘sensitivity of estimands in clinical trials with imperfect compliance’
  10. Commentary
  11. Comments on “sensitivity of estimands in clinical trials with imperfect compliance” by Chen and Heitjan
  12. Research Articles
  13. Optimizing personalized treatments for targeted patient populations across multiple domains
  14. Statistical models for assessing agreement for quantitative data with heterogeneous random raters and replicate measurements
  15. History-restricted marginal structural model and latent class growth analysis of treatment trajectories for a time-dependent outcome
  16. Revisiting incidence rates comparison under right censorship
  17. Ensemble learning methods of inference for spatially stratified infectious disease systems
  18. The survival function NPMLE for combined right-censored and length-biased right-censored failure time data: properties and applications
  19. Hybrid classical-Bayesian approach to sample size determination for two-arm superiority clinical trials
  20. Estimation of a decreasing mean residual life based on ranked set sampling with an application to survival analysis
  21. Improving the mixed model for repeated measures to robustly increase precision in randomized trials
  22. Bayesian second-order sensitivity of longitudinal inferences to non-ignorability: an application to antidepressant clinical trial data
  23. A modified rule of three for the one-sided binomial confidence interval
  24. Kalman filter with impulse noised outliers: a robust sequential algorithm to filter data with a large number of outliers
  25. Bayesian estimation and prediction for network meta-analysis with contrast-based approach
  26. Testing for association between ordinal traits and genetic variants in pedigree-structured samples by collapsing and kernel methods
https://doi.org/10.1515/ijb-2023-0065
Schlagwörter für diesen Artikel
outlier detection; filter algorithm; Gaussian model; EM algorithm; walk over weighing

Artikel in diesem Heft

  1. Frontmatter
  2. Research Articles
  3. Random forests for survival data: which methods work best and under what conditions?
  4. Flexible variable selection in the presence of missing data
  5. An interpretable cluster-based logistic regression model, with application to the characterization of response to therapy in severe eosinophilic asthma
  6. MBPCA-OS: an exploratory multiblock method for variables of different measurement levels. Application to study the immune response to SARS-CoV-2 infection and vaccination
  7. Detecting differentially expressed genes from RNA-seq data using fuzzy clustering
  8. Hypothesis testing for detecting outlier evaluators
  9. Response to comments on ‘sensitivity of estimands in clinical trials with imperfect compliance’
  10. Commentary
  11. Comments on “sensitivity of estimands in clinical trials with imperfect compliance” by Chen and Heitjan
  12. Research Articles
  13. Optimizing personalized treatments for targeted patient populations across multiple domains
  14. Statistical models for assessing agreement for quantitative data with heterogeneous random raters and replicate measurements
  15. History-restricted marginal structural model and latent class growth analysis of treatment trajectories for a time-dependent outcome
  16. Revisiting incidence rates comparison under right censorship
  17. Ensemble learning methods of inference for spatially stratified infectious disease systems
  18. The survival function NPMLE for combined right-censored and length-biased right-censored failure time data: properties and applications
  19. Hybrid classical-Bayesian approach to sample size determination for two-arm superiority clinical trials
  20. Estimation of a decreasing mean residual life based on ranked set sampling with an application to survival analysis
  21. Improving the mixed model for repeated measures to robustly increase precision in randomized trials
  22. Bayesian second-order sensitivity of longitudinal inferences to non-ignorability: an application to antidepressant clinical trial data
  23. A modified rule of three for the one-sided binomial confidence interval
  24. Kalman filter with impulse noised outliers: a robust sequential algorithm to filter data with a large number of outliers
  25. Bayesian estimation and prediction for network meta-analysis with contrast-based approach
  26. Testing for association between ordinal traits and genetic variants in pedigree-structured samples by collapsing and kernel methods
Jetzt anmelden und 20% sparen
Institutioneller Zugang
Wie funktioniert Authentifizierung?
Haben Sie eine Idee, wie wir unsere Website verbessern können?
Bitte schreiben Sie uns.
© 2025 De Gruyter Brill
Heruntergeladen am 26.10.2025 von https://www.degruyterbrill.com/document/doi/10.1515/ijb-2023-0065/html?recommended=sidebar
Button zum nach oben scrollen