Implementing predictive analysis using self-learning digital twins and image analysis with GPT-4 turbo with vision for inspection and repair of construction

  • 1 Department of Computer-Aided Engineering – UACEG – Sofia, Bulgaria


Nowadays, many structures should be inspected, analyzed, and repaired. This is a complex and expensive process that also includes predictive analytics to prevent possible construction failures.
One of the most used predictive analytics applications involves extracting necessary metadata from images and videos to evaluate the condition of real-world systems and recommend measures to sustain these systems. Image analysis is not a new concept – many solutions have been used for several decades.
The current paper mainly focuses on OpenAI-based capabilities to implement Image Analysis and Cognitive Digital Twins and proposes faster, cheaper implementation and more adaptive approaches to offering predictive analysis for constructions.
ChatGPT (Chat Generative Pre-Trained Transformer) is one of the trending technologies in modern Artificial Intelligence (AI), and experts in this area expect to have a very high impact on the industry shortly.
One of the latest versions – GPT-4 Turbo with Vision, developed by OpenAI, is a significant multimodal model (LMM) capable of interpreting images and providing text-based answers to queries regarding those images. It combines capabilities in natural language processing and visual comprehension.
The proposed approach considers using OpenAI LLM and Digital Twins for three different aspects of predictive analysis for Construction: image analysis, case decomposition, and creation of self-adaptive models to find possible trends to compromise structures and offer preventive actions. This research compares traditional methods for inspection and repair of Construction, including the time required for predictive analysis, the correctness of the proposed actions, and the cost of the methodology.



  1. T. Fernandez, "How to Choose the Best OpenAI Model for Your AI Application," Semaphore CI, Aug. 10, 2023. [Online]. Available:
  2. Lu, Yuqian, et al. "Digital Twin-driven Smart Manufacturing: Connotation, Reference Model, Applications and Research Issues." 2020,
  3. "Generative Pre-Trained Transformer." Wikipedia, 26 July 2023,
  4. "GPT-4 Technical Report". Arxiv.Org, 2023, Accessed 8 Aug 2023.
  5. M., Mateev. "INDUSTRY 4.0 AND THE DIGITAL TWIN FOR BUILDING INDUSTRY". Industry 4.0, vol 5, no. 1, 2020, pp. 29-32., Accessed 9 Aug 2023.
  6. Mateev M. Predictive Analytics Based on Digital Twins, Generative AI, and ChatGPT, Proceedings of the 27th World Multi- Conference on Systemics, Cybernetics and Informatics: WMSCI 2023, pp. 168-174 (2023);
  7. HeidiSteen. 2024. "RAG and Generative AI - Azure AI Search." RAG and Generative AI - Azure AI Search | Microsoft Learn. Accessed June 12.
  8. Mateev, Mihail. "Using Azure AI Vision and Open AI to analyze images for predictive analysis," BSC Technology AI Bootcamp. [Online]. Available:

Article full text

Download PDF