Home Kalman filter with impulse noised outliers: a robust sequential algorithm to filter data with a large number of outliers
Article
Licensed
Unlicensed Requires Authentication

Kalman filter with impulse noised outliers: a robust sequential algorithm to filter data with a large number of outliers

  • Bertrand Cloez EMAIL logo , Bénédicte Fontez , Eliel González-García and Isabelle Sanchez
Published/Copyright: April 17, 2024
Become an author with De Gruyter Brill
Submit Manuscript Author Information
The International Journal of Biostatistics
From the journal The International Journal of Biostatistics Volume 20 Issue 2

Abstract

Impulse noised outliers are data points that differ significantly from other observations. They are generally removed from the data set through local regression or the Kalman filter algorithm. However, these methods, or their generalizations, are not well suited when the number of outliers is of the same order as the number of low-noise data (often called nominal measurement). In this article, we propose a new model for impulsed noise outliers. It is based on a hierarchical model and a simple linear Gaussian process as with the Kalman Filter. We present a fast forward-backward algorithm to filter and smooth sequential data and which also detects these outliers. We compare the robustness and efficiency of this algorithm with classical methods. Finally, we apply this method on a real data set from a Walk Over Weighing system admitting around 60 % of outliers. For this application, we further develop an (explicit) EM algorithm to calibrate some algorithm parameters.

Keywords: outlier detection; filter algorithm; Gaussian model; EM algorithm; walk over weighing
Corresponding author: Bertrand Cloez, MISTEA, Univ Montpellier, INRAE, Institut Agro, Montpellier, France, E-mail:

Funding source: TechCare

Award Identifier / Grant number: 862050

  1. Research ethics: Not applicable.

  2. Author contributions: The authors have accepted responsibility for the entire content of this manuscript and approved its submission.

  3. Competing interests: The authors state no conflict of interest.

  4. Research funding: This work was financially supported by the European Union's Horizon 2020 Innovation Action program for funding the project Integrating innovative TECHnologies along the value Chain to improve small ruminant welfARE management (TechCare; grant agreement 862050).

  5. Data availability: The raw data can be obtained on request from the corresponding author.

References

1. Fisch, ATM, Eckley, IA, Fearnhead, P. Innovative and additive outlier robust Kalman filtering with a robust particle filter. IEEE Trans Signal Process 2022;70:47–56. https://doi.org/10.1109/tsp.2021.3125136.Search in Google Scholar

2. Pimentel, MAF, Clifton, DA, Clifton, L, Tarassenko, L. A review of novelty detection. Signal Process 2014;99:215–49. https://doi.org/10.1016/j.sigpro.2013.12.026.Search in Google Scholar

3. Chandola, V, Banerjee, A, Kumar, V. Anomaly detection: a survey. ACM Comput Surv 2009;41:1–58. https://doi.org/10.1145/1541880.1541882.Search in Google Scholar

4. Maity, A, Chatterjee, R. Impulsive noise in images: a brief review. Comput Vis Graph Image Process 2018;4:6–15. https://doi.org/10.19101/tipcv.2017.39025.Search in Google Scholar

5. Pardeshi, V, Patil, R. Suppression of random valued impulse noise in image processing: a review. Int J Adv Res Electrical Electron Instrum Eng 2015;4. https://doi.org/10.15662/ijareeie.2015.0401017.Search in Google Scholar

6. Cappé, O, Moulines, E, Ryden, T. Inference in hidden Markov models. New York, NY: Springer; 2007.Search in Google Scholar

7. Del Moral, P. Mean field simulation for Monte-Carlo integration. Monogr Stat Appl Probab 2013;126:26.10.1201/b14924Search in Google Scholar

8. Yao, H, Zhu, F. Using PSO particle filter to recovery image. In: 2011 3rd international workshop on intelligent systems and applications; 2011:1–4 pp.10.1109/ISA.2011.5873251Search in Google Scholar

9. Pai, P, Cong, C, Yongsheng, Y. Particle swarm optimization based on hybrid Kalman filter and particle filter. J Shanghai Jiaot Univ 2020;25:681–8. https://doi.org/10.1007/s12204-020-2229-5.Search in Google Scholar

10. Anderson, BDO, Moore, JB. Optimal filtering. Mineola, New York: Dover Publications; 2012.Search in Google Scholar

11. Gelb, A, Kasper, JFJr., Nash, RAJr., Prices, CFJr., Sutherland, AAJr. Applied optimal estimation. Massachusetts: MIT Press; 1974.Search in Google Scholar

12. Kalman, RE, Bucy, RS. New results in linear filtering and prediction theory. ASME. J Basic Eng 1961;83:95–108. https://doi.org/10.1115/1.3658902.Search in Google Scholar

13. Kalman, RE. A new approach to linear filtering and prediction problems. J Basic Eng 1960;82:35–45. https://doi.org/10.1115/1.3662552.Search in Google Scholar

14. Maybeck, PS. Stochastic models, estimation, and control. New York, San Francisco, London: Academic Press; 1982.Search in Google Scholar

15. Ting, J-A, Theodorou, E, Schaal, S. Machine learning: ECML 2007. In: 18th European conference on machine learning. Warsaw, Poland: Springer; 2007:748–56 pp.Search in Google Scholar

16. Morris, J. The Kalman filter: a robust estimator for some classes of linear quadratic problems. IEEE Trans Inf Theor 1976;22:526–34. https://doi.org/10.1109/tit.1976.1055611.Search in Google Scholar

17. Agamennoni, G, Nieto, JI, Nebot, EM. An outlier-robust Kalman filter. In: 2011 IEEE international conference on robotics and automation. IEEE; 2011:1551–8 pp.10.1109/ICRA.2011.5979605Search in Google Scholar

18. Masreliez, C, Martin, R. Robust Bayesian estimation for the linear model and robustifying the Kalman filter. IEEE Trans Automat Control 1977;22:361–71. https://doi.org/10.1109/tac.1977.1101538.Search in Google Scholar

19. Meinhold, RJ, Singpurwalla, ND. Robustification of Kalman filter models. J Am Stat Assoc 1989;84:479–86. https://doi.org/10.2307/2289933.Search in Google Scholar

20. Schick, IC, Mitter, SK. Robust recursive estimation in the presence of heavy-tailed observation noise. Ann Stat 1994;22:1045–80. https://doi.org/10.1214/aos/1176325511.Search in Google Scholar

21. Sorenson, HW, Alspach, DL. Recursive Bayesian estimation using Gaussian sums. Automatica 1971;7:465–79. https://doi.org/10.1016/0005-1098(71)90097-5.Search in Google Scholar

22. West, M. Robust sequential approximate Bayesian estimation. J Roy Stat Soc B 1981;43:157–66. https://doi.org/10.1111/j.2517-6161.1981.tb01165.x.Search in Google Scholar

23. Wu, WR, Kundu, A. Recursive filtering with non-Gaussian noises. IEEE Trans Signal Process 1996;44:1454–68. https://doi.org/10.1109/78.506611.Search in Google Scholar

24. Huber, PJ. Robust estimation of a location parameter. Ann Math Stat 1964;35:73–101. https://doi.org/10.1214/aoms/1177703732.Search in Google Scholar

25. Cipra, T, Romera, R. Kalman filter with outliers and missing observations. Test 1997;6:379–95. https://doi.org/10.1007/bf02564705.Search in Google Scholar

26. Zoubir, AM, Koivunen, V, Ollila, E, Muma, M. Robust statistics for signal processing. Cambridge: Cambridge University Press; 2018.10.1017/9781139084291Search in Google Scholar

27. Chang, L, Li, K. Unified form for the robust Gaussian information filtering based on M-estimate. IEEE Signal Process Lett 2017;24:412–16. https://doi.org/10.1109/lsp.2017.2669238.Search in Google Scholar

28. Gandhi, MA, Mili, L. Robust Kalman filter based on a generalized maximum-likelihood-type estimator. IEEE Trans Signal Process 2009;58:2509–20. https://doi.org/10.1109/tsp.2009.2039731.Search in Google Scholar

29. Huang, Y, Zhang, Y, Zhao, Y, Chambers, JA. A novel robust Gaussian–student’s t mixture distribution based Kalman filter. IEEE Trans Signal Process 2019;67:3606–20. https://doi.org/10.1109/tsp.2019.2916755.Search in Google Scholar

30. Durovic, ZM, Kovacevic, BD. Robust estimation with unknown noise statistics. IEEE Trans Automat Control 1999;44:1292–6. https://doi.org/10.1109/9.769393.Search in Google Scholar

31. González-García, E, Alhamada, M, Nascimento, H, Portes, D, Bonnafe, G, Allain, C, et al.. Measuring liveweight changes in lactating dairy ewes with an automated walk-over-weighing system. J Dairy Sci 2021;104:5675–88. https://doi.org/10.3168/jds.2020-19075.Search in Google Scholar PubMed

32. González-García, E, de Oliveira Golini, P, Hassoun, P, Bocquier, F, Hazard, D, González, LA, et al.. An assessment of walk-over-weighing to estimate short-term individual forage intake in sheep. Animal 2018;12:1174–81. https://doi.org/10.1017/s1751731117002609.Search in Google Scholar

33. González-García, E, Alhamada, M, Pradel, J, Douls, S, Parisot, S, Bocquier, F, et al.. A mobile and automated walk-over-weighing system for a close and remote monitoring of liveweight in sheep. Comput Electron Agric 2018;153:226–38. https://doi.org/10.1016/j.compag.2018.08.022.Search in Google Scholar

34. Leroux, E, Llach, I, Besche, G, Guyonneau, JD, Montier, D, Bouquet, PM, et al.. Evaluating a walk-over-weighing system for the automatic monitoring of growth in postweaned Mérinos d'Arles ewe lambs under Mediterranean grazing conditions. Animal – Open Space 2023;2:100032.10.1016/j.anopes.2022.100032Search in Google Scholar

35. Rauch, HE, Tung, F, Striebel, CT. Maximum likelihood estimates of linear dynamic systems. AIAA J 1965;3:1445–50. https://doi.org/10.2514/3.3166.Search in Google Scholar

36. Murphy, KP. Switching Kalman filters, technical report. Berkley: University of California; 1998.Search in Google Scholar

37. Murphy, KP. Dynamic Bayesian networks: representation, inference and learning. Berkeley: University of California; 2002.Search in Google Scholar

38. Nguyen, LH, Goulet, J-A. Anomaly detection with the switching Kalman filter for structural health monitoring. Struct Control Health Monit 2018;25:e2136. https://doi.org/10.1002/stc.2136.Search in Google Scholar

39. Royer, F, Lutcavage, M. Filtering and interpreting location errors in satellite telemetry of marine animals. J Exp Mar Biol Ecol 2008;359:1–10. https://doi.org/10.1016/j.jembe.2008.01.026.Search in Google Scholar

40. Speekenbrink, M, Shanks, DR. Learning in a changing environment. J Exp Psychol Gen 2010;139:266. https://doi.org/10.1037/a0018620.Search in Google Scholar PubMed

41. Wu, W, Black, MJ, Mumford, D, Gao, Y, Bienenstock, E, Donoghue, JP. Modeling and decoding motor cortical activity using a switching Kalman filter. IEEE Trans Biomed Eng 2004;51:933–42. https://doi.org/10.1109/tbme.2004.826666.Search in Google Scholar PubMed

42. Baum, LE, Petrie, T. Statistical inference for probabilistic functions of finite state Markov chains. Ann Math Stat 1966;37:1554–63. https://doi.org/10.1214/aoms/1177699147.Search in Google Scholar

43. MacDonald, IL, Zucchini, W Hidden Markov and other models for discrete-valued time series. CRC Press; 1997, vol 110.Search in Google Scholar

44. Bar-Shalom, Y, Fortmann, TE, Cable, PG. Tracking and data association. J Acoust Soc Am 1990;87:918–9.10.1121/1.398863Search in Google Scholar

45. Kim, C, Li, F, Ciptadi, A, Rehg, JM. Multiple hypothesis tracking revisited. In: Proceedings of the IEEE international conference on computer vision; 2015:4696–704 pp.10.1109/ICCV.2015.533Search in Google Scholar

46. Boyen, X, Koller, D. Tractable inference for complex stochastic processes. arXiv preprint arXiv:1301.7362; 2013.Search in Google Scholar

47. Gorynin, I, Gangloff, H, Monfrini, E, Pieczynski, W. Assessing the segmentation performance of pairwise and triplet Markov models. Signal Process 2018;145:183–92. https://doi.org/10.1016/j.sigpro.2017.12.006.Search in Google Scholar

48. Lehmann, F, Pieczynski, W. Suboptimal Kalman filtering in triplet Markov models using model order reduction. IEEE Signal Process Lett 2020;27:1100–4. https://doi.org/10.1109/lsp.2020.3002420.Search in Google Scholar

49. Chen, Y, Li, W, Wang, Y. Robust Gaussian approximate fixed-interval smoother with outlier detection. IEEE Signal Process Lett 2020;27:1505–9. https://doi.org/10.1109/lsp.2020.3015422.Search in Google Scholar

50. Li, H, Medina, D, Vilà-Valls, J, Closas, P. Robust variational-based Kalman filter for outlier rejection with correlated measurements. IEEE Trans Signal Process 2020;69:357–69. https://doi.org/10.1109/tsp.2020.3042944.Search in Google Scholar

51. Wang, H, Li, H, Fang, J, Wang, H. Robust Gaussian Kalman filter with outlier detection. IEEE Signal Process Lett 2018;25:1236–40. https://doi.org/10.1109/lsp.2018.2851156.Search in Google Scholar

52. González, LA, Bishop-Hurley, G, Henry, D, Charmley, E. Wireless sensor networks to study, monitor and manage cattle in grazing systems. Anim Prod Sci 2014;54:1687–93. https://doi.org/10.1071/an14368.Search in Google Scholar

53. Le Gall, JF. Brownian motion, martingales, and stochastic calculus. Switzerland: Springer International Publishing; 2016, 274.10.1007/978-3-319-31089-3Search in Google Scholar

54. Supplementary material available on line. Available from: https://forgemia.inra.fr/isabelle.sanchez/kfino_outputs.Search in Google Scholar

55. Cloez, B, Sanchez, I. kfino: Kalman filter for impulse noised outliers. R package version 1.0.0; 2022.10.32614/CRAN.package.kfinoSearch in Google Scholar

56. Fitzgerald, DW, Murphy, FE, Wright, WMD, Whelan, PM, Popovici, EM. Design and development of a smart weighing scale for beehive monitoring. In: 2015 26th Irish Signals and Systems Conference (ISSC); 2015:1–6 pp.10.1109/ISSC.2015.7163763Search in Google Scholar

57. Nyalala, I, Okinda, C, Kunjie, C, Korohou, T, Nyalala, L, Chao, Q. Weight and volume estimation of poultry and products based on computer vision systems: a review. Poultry Sci 2021;100:101072. https://doi.org/10.1016/j.psj.2021.101072.Search in Google Scholar PubMed PubMed Central

58. Brunet, H, Creach, P, Concordet, D. Optimal estimation of broiler movement for commercial tracking. Smart Agric Technol 2023;3:100113. https://doi.org/10.1016/j.atech.2022.100113.Search in Google Scholar

Supplementary Material

This article contains supplementary material (https://doi.org/10.1515/ijb-2023-0065).

Received: 2023-06-28
Accepted: 2024-03-05
Published Online: 2024-04-17

© 2024 Walter de Gruyter GmbH, Berlin/Boston

Articles in the same Issue

  1. Frontmatter
  2. Research Articles
  3. Random forests for survival data: which methods work best and under what conditions?
  4. Flexible variable selection in the presence of missing data
  5. An interpretable cluster-based logistic regression model, with application to the characterization of response to therapy in severe eosinophilic asthma
  6. MBPCA-OS: an exploratory multiblock method for variables of different measurement levels. Application to study the immune response to SARS-CoV-2 infection and vaccination
  7. Detecting differentially expressed genes from RNA-seq data using fuzzy clustering
  8. Hypothesis testing for detecting outlier evaluators
  9. Response to comments on ‘sensitivity of estimands in clinical trials with imperfect compliance’
  10. Commentary
  11. Comments on “sensitivity of estimands in clinical trials with imperfect compliance” by Chen and Heitjan
  12. Research Articles
  13. Optimizing personalized treatments for targeted patient populations across multiple domains
  14. Statistical models for assessing agreement for quantitative data with heterogeneous random raters and replicate measurements
  15. History-restricted marginal structural model and latent class growth analysis of treatment trajectories for a time-dependent outcome
  16. Revisiting incidence rates comparison under right censorship
  17. Ensemble learning methods of inference for spatially stratified infectious disease systems
  18. The survival function NPMLE for combined right-censored and length-biased right-censored failure time data: properties and applications
  19. Hybrid classical-Bayesian approach to sample size determination for two-arm superiority clinical trials
  20. Estimation of a decreasing mean residual life based on ranked set sampling with an application to survival analysis
  21. Improving the mixed model for repeated measures to robustly increase precision in randomized trials
  22. Bayesian second-order sensitivity of longitudinal inferences to non-ignorability: an application to antidepressant clinical trial data
  23. A modified rule of three for the one-sided binomial confidence interval
  24. Kalman filter with impulse noised outliers: a robust sequential algorithm to filter data with a large number of outliers
  25. Bayesian estimation and prediction for network meta-analysis with contrast-based approach
  26. Testing for association between ordinal traits and genetic variants in pedigree-structured samples by collapsing and kernel methods
https://doi.org/10.1515/ijb-2023-0065
Keywords for this article
outlier detection; filter algorithm; Gaussian model; EM algorithm; walk over weighing

Articles in the same Issue

  1. Frontmatter
  2. Research Articles
  3. Random forests for survival data: which methods work best and under what conditions?
  4. Flexible variable selection in the presence of missing data
  5. An interpretable cluster-based logistic regression model, with application to the characterization of response to therapy in severe eosinophilic asthma
  6. MBPCA-OS: an exploratory multiblock method for variables of different measurement levels. Application to study the immune response to SARS-CoV-2 infection and vaccination
  7. Detecting differentially expressed genes from RNA-seq data using fuzzy clustering
  8. Hypothesis testing for detecting outlier evaluators
  9. Response to comments on ‘sensitivity of estimands in clinical trials with imperfect compliance’
  10. Commentary
  11. Comments on “sensitivity of estimands in clinical trials with imperfect compliance” by Chen and Heitjan
  12. Research Articles
  13. Optimizing personalized treatments for targeted patient populations across multiple domains
  14. Statistical models for assessing agreement for quantitative data with heterogeneous random raters and replicate measurements
  15. History-restricted marginal structural model and latent class growth analysis of treatment trajectories for a time-dependent outcome
  16. Revisiting incidence rates comparison under right censorship
  17. Ensemble learning methods of inference for spatially stratified infectious disease systems
  18. The survival function NPMLE for combined right-censored and length-biased right-censored failure time data: properties and applications
  19. Hybrid classical-Bayesian approach to sample size determination for two-arm superiority clinical trials
  20. Estimation of a decreasing mean residual life based on ranked set sampling with an application to survival analysis
  21. Improving the mixed model for repeated measures to robustly increase precision in randomized trials
  22. Bayesian second-order sensitivity of longitudinal inferences to non-ignorability: an application to antidepressant clinical trial data
  23. A modified rule of three for the one-sided binomial confidence interval
  24. Kalman filter with impulse noised outliers: a robust sequential algorithm to filter data with a large number of outliers
  25. Bayesian estimation and prediction for network meta-analysis with contrast-based approach
  26. Testing for association between ordinal traits and genetic variants in pedigree-structured samples by collapsing and kernel methods
Sign up now to receive a 20% welcome discount
Institutional Access
How does access work?
Have an idea on how to improve our website?
Please write us.
© 2025 De Gruyter Brill
Downloaded on 26.10.2025 from https://www.degruyterbrill.com/document/doi/10.1515/ijb-2023-0065/html?lang=en
Scroll to top button