In this deliverable D7.3 – Result of Integrated System Evaluation, we present the results of the full-week simulation exercises to test the usability of iTRACK technology as well as the integration of this technology into the humanitarian logistics, information management and security risk management (SRM) processes and policies. The test was prepared in deliverable D6.3, which outlined the general ideas for carrying out the tests on the premises of TUD.
Two types of tests were carried out: usability tests for different parts of the web-based software and the developed apps on the mobile phone, and tests of the entire software suite in simulated field scenarios. After each usability test, participants filled out a questionnaire to test the quality of the software to determine possible improvements in the next versions. For the field tests, a small convoy of one truck and two cars was used, where the truck carried threat detection hardware. A small simulated warehouse with goods was set-up, as well as a mission control room for the mission leader, Information Management Officer (IMO) and security officer(s). To simulate threats, a drone was deployed, and a checkpoint was built on campus using sandbags. In addition to the communication using the developed mobile phone apps, radios were available as a back-up and for fast communication, e.g., in case of a threat. In total 5 two-hour usability tests and 6 half-day missions were carried out during the test week.
As the usability tests for the software preceded the field tests, the role players were able to use the software, and knew how to use it. For every field test, the role players were extensively briefed. Video, audio, and screen capture hard- and software was used to record the field experiments for post exercise analysis which is reported in this deliverable. Log files of the three deployed test servers were kept for analysis for future development.
Before each field experiment, consent forms were signed in line with the iTRACK ethics handbook (D1.1) and the T3.3 ethical considerations for the experiments. Each field experiment used a number of injects (planned and pre-defined events, such as a disruptions, within the scenario), selected from the total set of injects in D6.3. Thereby, the field experiments were able to test different aspects of the developed hardware and software, and enabled us to gather different experiences from the role players. Some of the role players were students (16), others were humanitarian aid workers (4) and some roles were played by employees from the project partners (11). The humanitarian aid workers were able to provide a good reference to the relation between the developed components and the possible deployment of these components in reality. By also using the project partners as players, immediate feedback of the components' use in an actual scenario was provided, which helped a lot in the understanding of the partners how their components can serve as a part of an overall usage in the field. After each field test, a debriefing session with all participants was conducted to immediately gather the most important comments for the used hardware and software. All technical and other partners were present at the debrief, so feedback could immediately be gathered and potentially processed by the partner that implemented the component mentioned in the debrief.
As the developers of the components were also present during the test week, several new component versions were made available during the week for analysis in a later field experiment. Many of the suggested changes will, however, be processed for the next (and final) version and tested in the final comprehensive demonstration which will be prepared in T6.4 and carried out in T7.4. Especially the usability tests yielded many detailed comments where and how to improve the mobile apps and the backend software, which will be addressed in more detail in WP4 and WP5 (D5.3). The field tests gave additional comments about the integration of the components in realistic scenarios, which will be addressed in WP4, and more specifically in D4.2.
Although not all components fully functioned as intended during the test week, and many suggestions for improvement were given, the overall comments of the users were positive. Especially the experts from the WFP and from IMMAP indicated that the components could be a huge improvement for the efficiency, effectiveness and security of humanitarian missions.
