Evaluation of Basic Trainings for Rescue Forces

1.1 Firefighter Basic Trainings

Internationally, the structure of fire brigades is quite diverse, and different nations have different distributions of volunteer and full-time firefighters (Brushlinsky et al. 2019). Nonetheless, Bukowski and Tanaka (1991) proposed four central points as international performance code for firefighters: “Prevent the fire or retard its growth and spread, protect building occupants from the fire effects, minimize the impact of fire [and] support fire-service operations” (175–176). Thus, firefighters’ goals are to save lives, property, and the environment. For these purposes every firefighter must receive appropriate training.

In Germany, fire protection is provided by professional fire brigades as well as volunteer fire brigades (cf. Brushlinsky et al. 2019). In both cases, future firefighters must first complete appropriate basic training (cf. FwDV 2 2012), which is provided by municipalities, districts and cities. At the voluntary level, basic training includes troop training and technical training. At the professional level, basic training involves several years of full-time training. Troop training is dedicated to the smallest tactical fire brigade unit, the troop. This usually consists of two firemen: A troop leader and a troop man. The troop leader bears the responsibility for his troop man and usually takes orders from a group leader, who, in turn, is subordinate to a platoon leader. Accordingly, the troop training can be subdivided into troop man training and troop leader training. The curriculum for troop training at the voluntary fire brigade level consists of various modules on topics such as legal bases, vehicle knowledge, equipment knowledge and life-saving emergency measures. It comprises a total of 185 h. Complementing and building on this, the second part of basic training at the voluntary fire brigade level is technical training (FwDV 2 2012). According to FwDV 2 (2012), the technical training includes, for example, additional training courses for radio operators, respirator wearers and machine operators as well as special training courses on how to behave when working with hazardous substances. The combination of troop and technical training at volunteer fire brigades is roughly comparable with the training required to become a professional firefighter at the professional fire brigade level. A central difference is that the basic training at the professional fire brigade level additionally includes training as a paramedic as well as internships at fire stations.

Regulations (FwDV 2 2012; VAP1.2-Feu 2015) specify the design of basic trainings, including components such as adequate content, learning objectives, time targets and methods. However, these training descriptions only represent minimum requirements. They are regarded as recommendations that can be supplemented. Accordingly, slight differences exist in the implementation of basic trainings depending on the respective federal state, district, municipality and trainer (Buchenau 2020). For example, even though the specified learning objectives must be achieved in the specified time frame (FwDV 2 2012), trainers can use methods and didactic approaches that are not described in the regulations. Furthermore, selection of trainers for the professional fire brigade focuses more on the trainers’ technical rather than pedagogical aptitude (VAP1.2-Feu 2015; Meyer and Stiegel 2012).

This training situation underlines the need for an evaluation to assess the current quality of basic training and to identify potential areas for further improvement. The desired result, namely high-quality firefighter basic trainings, will not only support good practices in emergency situations but will also support the physical and psychological health of firefighters. Specific evaluations can be helpful for recording and improving the quality of firefighters’ initial trainings in order to reduce the risks associated with the profession. For this reason, the concept of program evaluations will be explained in more detail in the following section.

1.2 Program Evaluation

Widely used in the context of evaluating training programs is the four-level model by Kirkpatrick (1979) (Blau et al. 2012). According to the model, the evaluation of trainings should take place at the four consecutive levels of reaction, learning, behavior and results (see Figure 1). According to Kirkpatrick (1979), the first level reaction covers the subjective, emotional evaluation of a training. This can be operationalized with questions about training relevance, materials and exercises, and reactions to the trainer or premises (Blanchard and Thacker 2010; Kirkpatrick 2007). Positive reactions not only indicate that participants are highly motivated but also that they are paying close attention – processes that are presupposed for participants’ learning success (Blanchard and Thacker 2010). Secondly, according to Kirkpatrick (1979), evaluations should investigate the level of learning. This refers to the extent to which participants expand their knowledge, develop their skills or change attitudes through a training. The trainees’ learning can be assessed with items such as “After the training, I know substantially more about the training contents than before” (Grohmann and Kauffeld 2013, 142). This evaluation criterion is of central importance to determine how well trainers promote the learning of participants and to uncover potential improvements (Kirkpatrick 2007). Thirdly, at the level behavior, an evaluation should examine whether a change in behavior, i.e., a transfer of learned training content to everyday situations, has taken place as a result of a training (Kirkpatrick 1979). Finally, at the level of results, an evaluation should record the broad organizational effects of a training. For example, a training might lead to reduced costs or improved service quality (Kirkpatrick 1979). Kirkpatrick and Kirkpatrick (2006) stress that the four levels of the model are to be understood as successive stages. Each individual level has an influence on the next level. For this reason, Kirkpatrick (1979) suggests evaluating a program on higher levels only after it has been shown to be successful on lower levels. However, implementing subsequent evaluation levels steadily becomes more difficult and resource intensive (Kennedy et al. 2014).

Figure 1:

Kirkpatrick’s four levels of training evaluation.

Therefore, we aim to develop an evaluation questionnaire that covers the levels of reaction and learning. At the reaction level, we will record firefighters’ affective reactions to the trainings and their perceived usefulness (cf. Blanchard and Thacker 2010). At the learning level, self-assessments of whether the firefighters acquired specific competences should provide information on subjective learning success (cf. Kirkpatrick and Kirkpatrick 2006). The present work aims at covering both of these levels in a standardized questionnaire (cf. Blanchard and Thacker 2010; Kirkpatrick and Kirkpatrick 2006). Before describing the questionnaire, the next section examines the extent to which evaluations are common in firefighter basic trainings.

1.3 Evaluation in the Context of Firefighter Basic Trainings

Regarding firefighter trainings, evaluations are critically important to achieve optimum training outcomes. In basic training courses, it is already common to evaluate participants’ performances after practical exercises. Such evaluations often take place in the form of “lessons learned” (Berlin and Carlström 2014, 199), conversations that happen after the training exercises with the aim of discussing in the training group what worked and what could use improvement. Also, Childs (2005) states the importance of such critical reflection in firefighter education programs. Similarly, Sommer and Njå (2011) propose that sharing experiences is a good learning method in firefighter basic trainings.

Less common is the evaluation of firefighter basic trainings themselves. However, such an evaluation is equally important, as various training conditions can impede learning. For instance, learning effects are absent or very low if a training is badly structured, if exercises are unrealistic, if a training mainly focuses on existing knowledge or if trainers impart new knowledge using bad didactics (Berlin and Carlström 2014). To the best of our knowledge, the Feedback Instrument for Rescue forces Education (FIRE, Schulte and Thielsch 2019) questionnaire is currently the only systematically developed and validated published general evaluation tool in the context of firefighter education programs. The instrument is directed at future group and platoon leaders who can use it to rate the quality of firefighter leadership trainings in which they participate. The FIRE questionnaire consists of the six scales trainers’ behavior, structure, overextension, group, competence and transfer.

To better understand what constitutes a validated questionnaire, the concept of validity and the aim of the present research is explained in more detail in the following section.

1.4 Validation of Questionnaires and Aim of the Present Study

In general, validity is understood as the measure of accuracy with which a questionnaire measures the concept that it intends to measure (Goldstein and Simpson 2002). Goldstein and Simpson (2002) propose to assess validity by examining three different facets of validity: Content validity, construct validity and criterion validity. First, content validity examines the extent to which a questionnaire represents the characteristic to be measured (Haynes, Richard, and Kubany 1995). It can be achieved by involving experts from the respective field in the development of the construct’s definition and items for the questionnaire (American Educational Research Association et al. 2014). Next, construct validity can be confirmed if relationships between the behavior in the test situation and underlying psychological characteristics can be demonstrated (Goldstein and Simpson 2002). As such, construct validity can include the investigation of a questionnaire’s factorial structure: Factorial validity exists if there is a good fit between the theoretical model on which a questionnaire is based on and the empirical data obtained with it (Guilford 1946; Thompson and Daniel 1996). Yet, as described by Campbell and Fiske (1959), construct validity can be divided into two subtypes: convergent validity and divergent validity. A questionnaire is regarded as being convergently valid if high correlations with construct-related questionnaires can be proven. The idea is to test whether constructs that are expected to be related are, in fact, related. In turn, a questionnaire can be described as being divergently valid if only low correlations with other independent constructs are found. Thus, it is the idea to check whether constructs measured with other questionnaires that are not expected to be related do not, in fact, have any relationship. The third aspect of validity, criterion validity, aims at demonstrating that questionnaire scores are related to or, more precisely, predict concrete real-life outcomes. Again, two subtypes can be specified: A questionnaire is said to have concurrent validity if it can predict criteria measured at the same time (e.g., expert or global ratings), whereas it has predictive validity if it forecasts criteria measured some point after the questionnaire scores were obtained (e.g., learning or performance outcomes) (Cronbach and Meehl 1955).

The present study aims to respond to the need for a valid tool to evaluate firefighter basic trainings^[1]. Therefore, our first goal is to systematically develop a profound theoretical basis for appropriate items (Study 1). Further, we aim at a piloting these items as well as comprehensively validating the developed questionnaire using different samples. To examine the questionnaire’s factorial validity, we conduct exploratory and confirmatory factor analyses. Beyond that, to investigate convergent and divergent construct validity as well as concurrent criterion validity, we make use of correlative validation methods (Study 2 & 3).

2.1 Method

In Study 1, in a qualitative research approach we asked N = 21 experts (n = 13 trainees, n = 4 trainers, n = 4 persons mainly having a managing function in a fire brigade school) what they personally consider to be important aspects of good training at the municipal and district level. All participants were German and male. Ages ranged from 18 to 31 years (M = 22.46; SD = 3.60) for the trainees, from 25 to 49 years (M = 34.00; SD = 10.52) for the trainers, and from 33 to 46 years (M = 37.50; SD = 5.80) for the persons with a managing function. The survey was answered with regard to trainings for professional fire brigades by 48% of the participants, whereas 52% answered it with regard to trainings for voluntary fire brigades. Factors related to the success and quality of basic trainings were recorded in an online survey by means of the Critical Incident Technique (CIT, Flanagan 1954). The CIT is a qualitative research method based on expert surveys. It is frequently used as an effective exploratory tool to better understand specific human activities or to get an information base for further research (Butterfield et al. 2005). The idea is to let experts clearly and comprehensibly describe critical situations (i.e., situations including particularly effective or ineffective behaviors) from which critical categories or items can be derived. Additionally, participants were able to directly name important aspects of a good training in four open questions: The first concerned aspects of good training in the fire brigade. The second asked for characteristics of a good trainer and his or her teaching style. The third asked about relevant framework conditions, and the fourth asked for ways in which trainees themselves can contribute to good training. The study was available online from the end of August to October 2017. Participation was voluntary and anonymous. As compensation, the respondents received a result report after Study 1 was completed.

2.2 Results and Discussion

Based on a qualitative content analysis (Mayring 2015), the experts’ statements were clustered into categories of good firefighter basic trainings by two independent evaluators. The analyses led to eight categories of a good basic training at a fire brigade: Didactics, motivation & engagement, personality, content & methods, structure & organization, materials & facilities, group and achievement of learning objectives (see Table 1). A standard procedure in questionnaire construction is the creation of a large item pool for a first draft of a questionnaire (e.g., Kline 2000; Nunnally 1975; Rossi, Wright, and Anderson 2013). This makes it possible to remove unsuitable items after subsequent item analyses while still retaining a sufficiently large number of items. Thus, in accordance with the identified categories, a pool of 51 items, which comprehensively and fully depicted the mentioned success-critical aspects, was created for a preliminary version of the questionnaire (see Table A1 in the online supplement at https://doi.org/10.5281/zenodo.3948173).

The results of Study 1 indicate that the categories experts described as being important for good firefighter basic trainings are similar to the success factors for firefighter leadership trainings (FIRE questionnaire). However, the results also revealed aspects that are not covered by the existing FIRE questionnaire (Schulte and Thielsch 2019), such as specific teaching methods and outcomes, motivational aspects, personality aspects of the trainers and required materials and facilities. Beyond that, the results of Study 1 revealed parallels to the characteristics of good teaching at universities, as many items were similar to items that have been described in established evaluation instruments used at universities (TRIL, Gläßer et al. 2002; MFE-ZGr, Grötemeier and Thielsch 2010a; MFE-ZHa, Grötemeier and Thielsch 2010b; MFE-V, Hirschfeld and Thielsch 2009; HILVE, Rindermann and Amelang 1994; FEVOR, FEPRA, Staufenbiel 2000; MFE-Sr, Thielsch and Hirschfeld 2012).

Based on these findings, we developed an adapted questionnaire for the evaluation of firefighter basic trainings. Thus, the initial item pool consisted of 51 questions newly created based on interview results of Study 1 as well as items originating from existing instruments that were adapted to the technical context of the fire brigade as well as to the training context at municipal and district level (see Table A1 in the online supplement at https://doi.org/10.5281/zenodo.3948173). The resulting questionnaire was named Feedback Instrument for Rescue forces Education – Basic education (FIRE-B). The aims of the following studies were thus to shorten this draft version by removing items proven to be unsuitable in item analyses, to facilitate practical application and to check the psychometric quality of the resulting final instrument.

3.1 Method

3.2 Results and Discussion

4.1 Method

4.2 Results and Discussion

4.2.1 Confirmatory Factor Analysis

A confirmatory factor analysis (CFA) was conducted to review the factor structure proposed in Study 2. Modification indices indicated a high correlation between item 1a (“I think the training was clearly structured”) and item 1b (“I could always follow the training process during the training”) of the scale structure & didactics. After considerations of content, both items were judged to be redundant. Since item 1a seemed more global and understandable, item 1b was removed from the model. According to Schermelleh-Engel, Moosbrugger, and Müller (2003), the model fit for the final FIRE-B with 29 items on six scales can be classified as good (RMSEA = 0.05; SRMR = 0.04) to acceptable (CFI = 0.95; TLI = 0.95). The χ²-test was significant ( χ² (362) = 585.12, p < 0.001), which is common for large samples (Tanguma 2001). However, related to the degrees of freedom, the χ²-value is good (χ²/df = 1.62). Results thus provide support for a six-factorial structure. Figure 2 illustrates the specified model including all path coefficients. The intercorrelations of the scales are medium to high and can be found in Table A1 in the appendix.

Figure 2:

Results of confirmatory factor analysis. Standardized coefficients are reported.

In sum, the results of the CFA are in line with the findings of the EFA in Study 2, confirming that the items of the FIRE-B load on six distinct factors (structure & didactics,support& encouragement, group, practice, materials & facilities, competence). Overall, the six-dimensional questionnaire structure demonstrates that various quality factors contribute to good firefighter basic trainings and that firefighters should have a wide range of skills for successful action (cf. Kleinmann et al. 2010).

5.1 Practical Application

On a practical level, the FIRE-B for the first time offers the possibility to assess the quality of firefighter basic trainings at municipal and district level from the trainee’s point of view. The differentiation of process and outcome scales according to Kirkpatrick (1979) helps to distinguish information on learning outcomes from information on possible ways to adapt the training process. In this way, the process-related items of the FIRE-B capture judgments about the trainer, the organization of the training, the group as well as about exercises, materials and facilities. The result-related items assess the extent to which the training has contributed to the acquisition of knowledge and skills. Thus, the FIRE-B provides information on the current quality of basic firefighting trainings and helps to identify possible areas for improvement within the implementation of the training and the achievement of the learning objectives. If, for example, trainees indicate not having achieved certain learning objectives, the trainer can check whether and at what point in the process there was a problem. To facilitate the questionnaire’s practical application, we provide additional information in the online supplement at https://doi.org/10.5281/zenodo.3948173, including questionnaire templates (in English and German) and scoring instructions.

Generally, an evaluation with the FIRE-B should take place directly after a firefighter training course. The 29 items can be answered quickly (in our experience in about 4–5 min), and the six different scales provide a comprehensive picture of the training quality. The items are consistently to be rated on a seven-point response scale with a denial option, allowing a simple data analysis and interpretation. Depending on the evaluation context, we recommend the additional use of four optional items (see Table A5 in the online supplement at https://doi.org/10.5281/zenodo.3948173). Also, as it is important to ensure anonymous evaluation, one should not collect variables (e.g., demographic) that allow conclusions about individual persons. If scales are not applicable or if the use of all scales is considered too time consuming, single FIRE-B scales can be omitted, as they have been validated separately. However, one should not remove individual items. As the scales are already very brief, this may impair psychometric quality. Similarly, one should not change the wording of the individual questions. Exceptions are minor adjustments to ensure the questionnaire’s comprehensibility and its optimal adaptation to the evaluation context.

In the analysis, mean values are calculated for the individual scales across participants and courses (see scoring instructions in the online supplement at https://doi.org/10.5281/zenodo.3948173). A high number of missing values (see scoring instructions in the online supplement at https://doi.org/10.5281/zenodo.3948173) from many participants may indicate a lack of fit of the questionnaire in the respective evaluation context. Additionally, an evaluation analysis should only take place if a minimum number of completed evaluation questionnaires are available (see scoring instructions in the online supplement at https://doi.org/10.5281/zenodo.3948173). Further, to keep effort and calculation errors to a minimum, we recommend that evaluations are analyzed using simple data evaluation programs.

Beyond that, organizers and trainers should consider the subjective nature of this type of evaluation: feedback gathered with the FIRE-B questionnaire is an opportunity for organizers and trainers to obtain important information about their own teaching activities from their trainees’ points of view. Thus, organizers and trainers should meet with the trainees to discuss the results of the evaluation. In addition, the responsible organization should support the evaluation both technically and in terms of its content. Particularly, organizers should offer trainers help if evaluations repeatedly reveal areas for improvement or offer them praise for good teaching quality.

Finally, while the FIRE-B was developed specifically for the context of fire brigades, in this context it pursues a rather global approach, meaning that the questionnaire can be used to evaluate a wide range of firefighter basic trainings. If, beyond that, people wish to investigate more specific aspects as part of an evaluation, we recommend using more specific evaluation scales, such as scales for the evaluation of firefighter examinations, mission exercises or command unit trainings (Röseler et al. 2020; Schulte and Thielsch 2019; Thielsch, Busjan, and Frerichs 2018).

5.2 Limitations and Future Research

This study has some limitations but also some possibilities for future research. Both will be discussed below.

Regarding the application of the FIRE-B, one must consider that German firefighters served as the basis for constructing the questionnaire, such that the proportion of voluntary versus professional firefighters as well as the gender distributions were representative of German fire brigades (cf. Deutscher Feuerwehrverband (DFV) 2015). Fire brigades in other countries may have different distributions or different work and trainings structures. Thus, organizers in different countries should first check whether the FIRE-B validly covers the relevant areas of the respective training courses or whether it needs to be adapted. Similarly, the use of the FIRE-B questionnaire in other training contexts is conceivable, as its items do not contain a fire brigade-specific vocabulary. The content should be generally relevant within rescue training courses, such as paramedic trainings. Before applying a translation or adapted version, the validity should first be checked. Therefore, we recommend as a minimum requirement performing an expert assessment of the content validity for the intended context, and we highly welcome specific validation studies.

Another aspect to consider when using the FIRE-B is its subjectivity: The FIRE-B asks for judgments from the trainees’ points of view. As indicated above, such information is primarily helpful for trainers to reflect on their own teaching activities and, if necessary, to think about possibilities for improvement. At the same time, the evaluation’s subjectivity increases the risk of misunderstandings. We, therefore, recommend that organizers meet with the trainers and trainees after the evaluation to discuss the results as a group, to collect ideas for improvement, and to uncover and clarify possible misunderstandings. In this regard, an evaluation from the trainers’ viewpoints might also be of interest, especially as Schulte and Thielsch (2019) showed that the judgments of trainees are, to some extent, different from those of the trainers. For example, trainers rated trainees’ overextension to be higher than the trainees actually experienced. As such, considering the views of both parties could contribute to a more global assessment of the quality of firefighter basic trainings and can also benefit trainees’ learning (Berlin and Carlström 2014; Childs 2005; Sommer and Njå 2011).

Additionally, the results of our study also carry some limitations. First, there was no proof of a relationship between the FIRE-B scales and passing the final exam. In view of the marginal failure rate, the question of passing the final examination for basic trainings seems to be an inadequate validation criterion. In the case of a uniform grading system, the grade achieved in a final examination might be a more suitable criterion. Furthermore, an objective proof of examination (e.g., training diplomas) could prevent possible effects of social desirability. Second, to avoid high exclusion rates, the date of the evaluated training was used as a filter criterion only in Study 2 but not in Study 3 (see Figures A1/A3 in the appendix). In Study 3, there were few significant correlations between the number of years since the start of training and FIRE-B scales. However, this finding does not affect the validity of this work, since controlling partial correlations were used for statistical data analysis in Study 3.

Regarding further research, the four-level model according to Kirkpatrick (1979) calls for methodically expanded follow-up studies investigating how to evaluate firefighter basic trainings at the levels of behavior and results. For the assessment of transfer effects at the behavioral level, future studies should collect data during subsequent work as a firefighter. Conceivable sources are again subjective self-judgments (e.g., “I successfully manage to apply the training contents in my everyday work” (Grohmann and Kauffeld 2013, 142), but also judgments from the perspectives of trainers, colleagues or superiors. In addition, objective observations of behavior based on standardized evaluation criteria can be a useful supplement (Blanchard and Thacker 2010). For the evaluation at the level of results, follow-up studies could examine whether a high quality of training leads to better organizational outcomes and whether, for example, fire brigade operations can be carried out more quickly or more successfully. Equally, future studies could check whether high-quality trainings imply a lower accident rate among firefighters themselves. A particularly challenging aspect of such an evaluation at the result level is the difficulty in attributing positive effects only to the object of evaluation and not to other unrecorded influences (Kennedy et al. 2014; Kirkpatrick 1979; Praslova 2010).

5.3 Conclusion

The present paper provides, for the first time, a systematically developed and validated evaluation questionnaire for firefighter basic trainings. The FIRE-B is a useful, efficient, reliable and valid feedback instrument. Thus, it can and should be used in rescue service education. The regular and long-term use of the evaluation questionnaire may not only contribute to the recording of current quality standards but may also reveal areas for improvement or possible changes in the basic training of firefighters. By providing a tool that may help improve the quality of firefighter basic trainings, we hope to ultimately contribute to society as a whole, as an optimized education hopefully leads to the most competent firefighters capable of responding optimally to various forms of emergencies.