D6.11 - Evaluation methods for integrated system

This document represents iTRACK Deliverable 6.11 iTRACK Evaluation Methods for Integrated System, which is a framework to measure quality and usefulness of iTRACK technology, under the iTRACK Task 6.5 Real-Time Evaluation and Reporting. This deliverable is one of the three deliverables of iTRACK that are associated with iTRACK evaluation and measurements or the iTRACK evaluation framework, namely D6.10 Evaluation method for process and policies, D6.11 Evaluation methods for integrated system, and D6.12 Final evaluation tool demonstration. This deliverable D6.11 in addition to 6.10 is the basis for the subsequent deliverable D6.12. The metrics developed in D6.10 support the design of automated and digital reporting feedback tools, which will provide a real-time evaluation of individuals and teams during training sessions. D6.11 embraces the iTRACK integrated system evaluation. Finally, in D6.12 the final evaluation tool/demonstrator will be presented.
The integrated system evaluation framework will assist in measuring the quality and usefulness of the iTRACK integrated system, by proposing methods and metrics to evaluate the ITRACK system quality in terms of meeting the specified requirements as well as user satisfaction. The former is achieved via testing the software system against the requirements in terms of specifications, use cases, design documents, etc. which is conducted in WP4 and WP5, while the latter is achieved via testing the iTRACK system components usability directly with users both in office and field settings, and subjectively via questionnaires to administered to these users. Lastly, the framework will guide the measurement of the iTRACK system usefulness in terms of how the system is helping users in accomplishing their job tasks quicker, improving their job performance, productivity, effectiveness, and in general making their job easier. In other words, the framework will measure the enhancement in performance indicators as a result of using the iTRACK system.
The iTRACK April 2018 exercise, deliverable D6.3, is an example of applying the iTRACK integrated system evaluation framework, as it was the first iTRACK technology testing with users. During this exercise a set of participants tested the ready iTRACK technology components. The participants were asked to complete certain tasks using the iTRACK technology. The suitable usability and usefulness metrics and questionnaires studied in the D6.11 were used during this exercise.