This document represents the iTRACK Deliverable 6.12 iTRACK final evaluation tool demonstration, under the iTRACK Task 6.5 Real-Time Evaluation and Reporting. This deliverable is one of the three deliverables of the iTRACK that are associated with iTRACK evaluation and measurements or the iTRACK evaluation framework, namely D6.10 Evaluation method for process and policies, D6.11 Evaluation methods for integrated system, and D6.12 Final evaluation tool demonstration. The iTRACK Deliverable 6.12 is a computer system used for reporting and evaluating the quality and usefulness of iTRACK technology. Hereafter this system will be referred to as the iTRACK reporting and evaluation system.
The integrated system evaluation framework introduced in the iTRACK deliverable D6.11 assists in measuring the quality and usefulness of the iTRACK integrated system. The iTRACK deliverable D6.11 is the basis for this deliverable D6.12. The metrics and methods developed and introduced in D6.11 support the design of automated and digital reporting feedback tools, which provides real-time evaluation of individuals and teams during training sessions and missions. This deliverable D6.12 is the implementation of these metrics and methods in the form of a computer system.
As proposed in D6.11, the D6.12 presents the iTRACK system quality in terms of the results achieved in WP4 and WP5, while implementing the usefulness in terms of administering questionnaires to the iTRACK users. The D6.12 presents the results of these surveys as well. Lastly, D6.12 presents the iTRACK system usefulness in terms showing the performance indicators logged by the iTRACK components, as well as staff compliance with SOPs/policies suggested in the iTRACK deliverable D2.6.
The iTRACK reporting and evaluation system is developed with extensibility in mind. Extensibility in terms of the system's capability of allowing administrators to add new development indicators, performance indicators, surveys, and SOPs.
The iTRACK reporting and evaluation system is capable of showing all the development indicators suggested in the iTRACK deliverable D6.11, however, only the results of the unit tests are currently available in the iTRACK work-package WP5. Some of these results, like code coverage from the iTRACK deliverable D5.2, were included in the current implementation of the iTRACK reporting and evaluation system as an example. Furthermore, results of the perceived usefulness and ease of use questionnaire[i] and the User experience Questionnaire (UEQ)[ii] that were filled by the iTRACK system users during the iTRACK’s “April 2018 Exercise”—see the iTRACK deliverable D7.3 for more information– were included in the iTRACK reporting and evaluation system as an example of users surveys as well.
Finally, D6.11 proposed a set of performance indicators. These proposed performance indicators were accepted by the iTRACK technical partners independently from the D6.11 itself. This step was taken to assure the possibility of calculating these performance indicators and implementing their needed code inside the different iTRACK system’s components. Nonetheless, in the current iTRACK reporting and evaluation system, these performance indicators were randomly generated for presentation purposes until the final development of the iTRACK system, except for the components developed by UiA which are already generating the performance indicators as proposed. In addition, as currently there are no real mission’s SOPs/policies filled surveys, such results were randomly generated and included for presentation purposes as well.
[i] Fred D. Davis, “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology,” MIS Quarterly 13, no. 3 (September 1989): 319–40, https://doi.org/10.2307/249008.
[ii] Bettina Laugwitz, Theo Held, and Martin Schrepp, “Construction and Evaluation of a User Experience Questionnaire,” in HCI and Usability for Education and Work, ed. Andreas Holzinger, vol. 5298 (Berlin, Heidelberg: Springer Berlin Heidelberg, 2008), 63–76, https://doi.org/10.1007/978-3-540-89350-9_6.
