Industry 4.0
Vol. 8 (2023), Issue 3, pg(s) 99-104

SOCIETY & ”INDUSTRY 4.0”

Evaluation of the learning success in the implementation phase of Extended Reality

  • 1 FH JOANNEUM, Kapfenberg, Austria

Abstract

Extended Reality (XR) stands as a hypernym for Augmented Reality (AR), Mixed Reality (MR) and Virtual Reality (VR) technologies and represents an increasingly important Industry 4.0 technology rank. XR technologies are enriching or replacing the environment with digital information. This introduces new possibilities for learning, collaboration, product presentation or other fields of application in companies. Various meta- and individual studies prove a positive influence of XR elements on learning success. However, the appraisal of results must be made with caution since no standard is currently available to measure learning success. This paper contributes to this situation by providing a framework to perform a generic measurement of learning success when using XR. The developed framework follows the 4-level evaluation model according to Kirckpatrick focussing on response, learn, apply, and benefit to the organization and is further developed meeting the specific requirements of XR training. The developed framework adresses two dimensions: the evaluation of the short term as well as the long-term training success. Furthermore, the paper gives insights in the practical validation of the framework done by an industrial company using a VR Training within the onboarding process. The framework is intended to help organizations to systematically assess the learning success of XR-supported training compared to traditional paper-based training.

Keywords

References

  1. PWC (2014), Anticipating problems, finding solutions Global Annual Review 2014, (https://www.pwc.com/gx/en/global-annual-review/assets/pwc-global-annual-review-2014.pdf)
  2. B. C. Group (2023) Industry 4.0, (https://www.bcg.com/capabilities/manufacturing/industry- 4.0)
  3. S. U. Rehman and S. Ejaz (2020), AN IMPLEMENTATION OF 9 PILLARS OF INDUSTRY 4.0 IN CONVENTIONAL FOOTWEAR INDUSTRY MODEL, Int. J. Eng. Appl. Sci. Technol., vol. 04, no. 12, pp. 283–286, (doi: 10.33564/IJEAST.2020.v04i12.047)
  4. G. Erboz (2017), How To Define Industry 4.0: Main Pillars Of Industry 4.0, (https://www.researchgate.net/publication/326557388_Ho w_To_Define_Industry_40_Main_Pillars_Of_Industry_40)
  5. B. Marr (2022), The top 10 tech trends in 2022 everyone must be ready for now, Forbes, (https://www.forbes.com/sites/bernardmarr/2022/02/21/the -top-10-tech-trends-in-2022-everyone-must-be-ready-for-now/?sh=30cd8098827d)
  6. B. Göbl (2022), Deloitte Tech Trends 2023, (https://www2.deloitte.com/at/de/seiten/enterprise-performance/articles/tech-trends.html)
  7. B. Rohleder (2021), Industrie 4.0 – so digital sind Deutschlands Fabriken (https://www.bitkom.org/sites/default/files/2021- 04/bitkom-charts-industrie-4.0-07-04-2021_final.pdf)
  8. K. Alena and K. Sobania (2022), Zeit für den digitalen Aufbruch, (https://www.dihk.de/resource/blob/65850/53d8cb00755f2 a2ce14532eb3fc9d45e/digitalisierungsumfrage-2022- data.pdf)
  9. C. Zerres, K. Israel, and O. Ernst (2021), EXTENDED REALITY CHANCEN, HERAUSFORDERUNGEN UND ANWENDUNGSBEISPIELE FÜR KLEINE UND MITTELSTÄNDISCHE UNTERNEHMEN.
  10. J. Peddie (2023), Augmented Reality, 2. Cham: Springer International Publishing.
  11. S. Romina Sorko and J. Komar (2020), Qualitative Acceptance Model of Augmented Reality from the Perspective of Personalists, Teh. Glas., vol. 14, no. 3, pp. 352–359, (doi: 10.31803/tg-20200719183209)
  12. J. Chen, J. Dai, K. Zhu, and L. Xu (2022), Effects of extended reality on language learning: A meta-analysis, Front. Psychol., vol. 13, (doi: 10.3389/fpsyg.2022.1016519)
  13. X. Xie (2022), Research on Immersion Teaching Method Based on 5G +XR Technology and Reinforcement Learning Model, Adv. Multimed., vol. 2022, pp. 1–12, (doi: 10.1155/2022/7092100)
  14. J. Radianti, T. A. Majchrzak, J. Fromm, and I. Wohlgenannt (2020), A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda, Comput. Educ., vol. 147, (doi: 10.1016/j.compedu.2019.103778)
  15. M. Hernandez-de-Menendez, C. Escobar Díaz, and R. Morales-Menendez (2020), Technologies for the future of learning: state of the art, Int. J. Interact. Des. Manuf., vol. 14, no. 2, pp. 683–695, (doi: 10.1007/s12008-019-00640- 0)
  16. N. H. Al-Kumaim, A. K. Alhazmi, F. Mohammed, N. A. Gazem, M. S. Shabbir, and Y. Fazea (2021), Exploring the Impact of the COVID-19 Pandemic on University Students’ Learning Life: An Integrated Conceptual Motivational Model for Sustainable and Healthy Online Learning, Sustainability, vol. 13, no. 5, p. 2546, (doi: 10.3390/su13052546)
  17. M. Treve (2021), What COVID-19 has introduced into education: challenges Facing Higher Education Institutions (HEIs), High. Educ. Pedagog., vol. 6, no. 1, pp. 212–227, (doi: 10.1080/23752696.2021.1951616)
  18. S. R. Sorko and M. Brunnhofer (2019), Potentials of Augmented Reality in Training, Procedia Manuf., vol. 31, pp. 85–90, (doi: 10.1016/j.promfg.2019.03.014)
  19. G. Rangel-de Lázaro and J. M. Duart (2023), You Can Handle, You Can Teach It: Systematic Review on the Use of Extended Reality and Artificial Intelligence Technologies for Online Higher Education, Sustainability, vol. 15, no. 4, p. 3507, (doi: 10.3390/su15043507)
  20. J. Komar and S. R. Sorko (2021), Mixed Reality Application Development for Practitioners. A ‘Do it Yourself (DIY)’ Use Case within a Learning Factory, SSRN Electron. J., (doi: 10.2139/ssrn.3862073)
  21. E. S. I. Yaqoot, W. S. Wan Mohd Noor, and M. F. Mohd Isa (2021), The Predicted Trainer and Training Environment Influence toward Vocational Training Effectiveness in Bahrain, J. Tech. Educ. Train., vol. 13, no. 1, (doi: 10.30880/jtet.2021.13.01.001)
  22. D. L. Kirkpatrick (1959), Techniques for Evaluation Training Programs, J. Am. Soc. Train. Dir., no. 13, pp. 21–26.
  23. S. Mohammed, A. Norsiah, and M. Norsiah (2013), Evaluation of effectiveness of training and development: The Kirkpatrick model, Asian J. Bus. Manag. Sci., vol. 2, no. 11, pp. 14–24.
  24. R. Bates (2004), A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence, Eval. Program Plann., vol. 27, no. 3, pp. 341–347, (doi: 10.1016/j.evalprogplan.2004.04.011)
  25. P. Tamkin, J. Yarnall, and M. Kerrin, Tamkin, P.; Yarnall, J.; Kerrin (2002), M. Kirkpatrick and Beyond: A Review of Training Evaluation. Brighton: The Institute for Employment Studies.
  26. J. Kirkpatrick (2021), An Introduction to The New World Kirkpatrick Model, Kirkpatrick Partners, (https://www.kirkpatrickpartners.com/wp-content/uploads/2021/11/Introduction-to-the-Kirkpatrick- New-World-Model.pdf)
  27. D. L. Kirkpatrick and J. D. Kirkpatrick (2007), Implementing the Four Levels: A Practical Guide for Effective Evaluation of Training Programs.
  28. L. Hamminger (2020), Lernerfolg aus Sicht berufstätiger Studierender. Erste Schritte auf dem Weg zu einer Messmethode, Mag. erwachsenenbildung.at, no. 4, pp. 1- 12.,(https://www.pedocs.de/volltexte/2020/20692/pdf/Erwa chsenenbildung_40_2020_Hamminger_Lernerfolg_aus_Si cht.pdf)
  29. N. A. A. S. Administration (2020), NASA TLX Task Load Index, (https://humansystems.arc.nasa.gov/groups/tlx/)
  30. V. Venkatesh and F. D. Davis (2000), A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies, Manage. Sci., vol. 46, no. 2, pp. 186–204, (doi: 10.1287/mnsc.46.2.186.11926)
  31. J. Brooke (1996), SUS: A ‘Quick and Dirty’ Usability Scale, P. W. Jordan, B. Thomas, I. L. McClelland, and B. Weerdmeester, Eds. CRC Press, (doi: 10.1201/9781498710411)
  32. K. Holzkamp (1995), Lernen. Subjektwissenschaftliche Grundlegung, Informationen Deutsch als Fremdspr., vol. 22, no. 2–3, pp. 242–248, (https://publishup.uni-potsdam.de/opus4- ubp/frontdoor/deliver/index/docId/393/file/HOLZLERN.p df)
  33. REFA (missing year), lndustrial Engineering Standardmethoden zur Produktivitätssteigerung und Prozessoptimierung.
  34. P. Häußler (2010), Physikdidaktik, E. Kircher, R. Girwidz, and P. Häußler, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, p. 266, (doi: 10.1007/978-3-642-01602-8)
  35. H. Clasen (2010), Die Messung von Lernerfolg: Eine grundsätzliche Aufgabe der Evaluation von Lehr- bzw. Trainingsinterventionen, Technischen Universität Dresden, (https://d-nb.info/1008623563/34)
  36. A. Frick-Salzmann (2017), Gedächtnis: Erinnern und Vergessen. Wiesbaden: Springer Fachmedien Wiesbaden, (doi: 10.1007/978-3-658-16720-2)
  37. H. Ebbinghaus (2013), Memory: A Contribution to Experimental Psychology, Ann. Neurosci., vol. 20, no. 4, (doi: 10.5214/ans.0972.7531.200408)
  38. D. L. Kirkpatrick and J. D. Kirkpatrick (2005), Transferring Learning to Behavior: Using the Four Levels to Improve Performance. Berrett-Koehler Publisher.
  39. Preparedness and Emergency Response Learning Center (missing year), Kirkpatrick Level 3 (Behavior) Evaluation Strategies, (http://www.phf.org/programs/preparednessresponse/evalu ationrepository/Documents/Kirkpatrick_Level_3_Behavior_Evaluation_Strategies.pdf)
  40. R. S. Kaplan and D. P. Norton (2007), Balanced Scorecard, in Das Summa Summarum des Management, Wiesbaden: Gabler, pp. 137–148. (doi: 10.1007/978-3-8349-9320- 5_12)
  41. B. Ahrendt, U. Heuke, W. Neumann, and F. Tubbesing (2021), Erfolgsfaktor Sozialkompetenz: Mitarbeiterpotenziale systematisch identifizieren und entwickeln, 1. Haufe Lexware GmbH.

Article full text

Download PDF