Pál Vásárhelyi Doctoral School of Civil Engineering and Earth Sciences 

Faculty of Civil Engineering, Department of Structural Engineering

Supervisor: Dr. KŐRIS Kálmán/Dr. Marek SALAMAK

BIM-based smart bridge health monitoring using the mixed reality application developed in UNITY 3D

Introducing the research area

The use of digital technologies in the construction industry is becoming a powerful tool for the evaluation of serviceability, monitoring of structural health, and visualization of Structural Health Monitoring (SHM) data. This concern has been focused on in my research where the Finite Element Analysis (FEA) method is used to analyze the serviceability of bridges and helped propose the SHM system of the bridges. Thus, low-cost Internet of Things (IoT)-based wireless sensors are developed in this study to establish a linkage between Building Information Modelling (BIM) technology and the SHM system. For this purpose, the BIM model of the bridge is used in the UNITY game engine to develop a smart Augmented Reality (AR)/Virtual Reality (VR) app that can directly be used in Cyber-Physical devices like Oculus, HoloLens (HL), etc. for the visualization, monitoring, and management of SHM data. Further, the measured data will be trained using Machine Learning (ML) algorithms to automatically detect bridge damage. In this way, smart bridge health monitoring can be performed onsite or remotely in an AR/VR environment.

Brief introduction of the research place

In the recent technological boom, the bridge industry is trying to catch up with modern tools to bring cutting-edge technology to bridge monitoring systems [1]. This research tried to apply this concept by bringing different emerging fields like Building Information Modelling (BIM), Structural Health Monitoring (SHM), Artificial Intelligence (AI), Extended Reality (AR/MR), and UNITY game engines on one platform to establish a smart bridge health monitoring system.

History and Context of the Research

SHM of bridges has attracted the interest of many bridge engineers, and state-of-the-art research is available on this topic [2], [3]. To automate this process, many researchers choose to integrate SHM systems with the applications of BIM technology to develop the BIM-based SHM system using the Internet of Things (IoT) technology [4], [5]. A similar trend has been followed in my research but not merely to integrate the SHM system as done by Boddupalli et. al., [6], but also to link it with AR/VR. The basic purpose of this additional integration is to utilize available technologies for advanced data management, rapid visualization of bridge damages, and remote, and robust SHM. As the use of this technology is still part of the research work and practical implementations of such devices in the field are still questionable for SHM purposes, Rios et., al., presented a detailed review of the main shortcomings identified in the implementation of AR technology in bridge monitoring and highlighted future trends that could be addressed by the Digital Twins (DT) for the bridges [7]. I focused on one of these shortcomings and my research findings proposed a solution that overcomes the limitation of the HoloLens tool and links the BIM-based SHM system with the AR technology.

The research goals, open questions

My research aims to integrate Structural Health Monitoring (SHM) system with AR devices on the industrial scale. To ensure this integration, the following goals were set to achieve:

                i.          Develop an automatic FE model using the Visual Programming Language (VPL) script generated by BIM technology.

              ii.          Automate of SHM system using low-cost wireless sensors and IoT-based web platforms.

             iii.          Develop an MR application using the UNITY game engine and deploy it on AR/VR devices for visualization and management of SHM data.

             iv.          Training a dataset of monitoring results using the ML algorithms that can detect bridge damages intelligently well in advance to develop an intelligent SHM system for a bridge.

These objectives are primarily set to meet the requirements of the bridge industry, where traditional bridge inspection methods still lack onsite data visualization and damage identification, leading to erroneous decisions and increased monitoring costs. Thus, achieving the above objectives could introduce an intelligent bridge inspection method where much of the work is performed in an augmented environment.

Methods

This multidisciplinary research uses a variety of software and analytical tools to achieve the research goals as shown in Figure 1. It starts with the FEA of the bridges using linear and nonlinear analysis methods. These analyses are performed using AXISVM, and Autodesk Robot software. Inn this way, a simulation of the existing bridge condition is carried out to identify the preliminary evaluation of the bridge. This method helps identify the need for the installation of SHM devices and location points. According to the proposed SHM deployment plan, low-cost wireless sensors are developed using the locally available low-cost sensors and Arduino microcontrollers and installed on real bridges. The IoT-based web platform for these wireless sensors (a free version of IoT) will be developed using Arduino coding, which will not only help in controlling these devices but also in data management and storage. Then, BIM technology will be leveraged by developing the BIM model of bridges using Autodesk Revit. This 3D model is used as a base platform for the automation of the SHM system using the BIMification approach. This approach will provide the direct control of IoT-based SHM system through the BIM model, thus fulfilling the automation objectives. Furthermore, the BIM model of the bridge is also used to automate the development of the FE model, where it directly develops the FE model geometry and attempts to control the complete FEA of bridges.

The BIM model will also be used to develop the MR application to integrate the SHM system with AR technology. This application will be developed using the UNITY 3D platform and deployed directly on AR/VR devices to help visualize SHM data in AR/VR. This tool will also help detect bridge damage on the spot and take necessary measures immediately. For long-term SHM, SHM data is collected and analyzed over a period of time. The measured response of bridge health is compared with the simulated response (done in FE analysis) to verify the results and improve the outcomes. Furthermore, these data are trained using ML algorithms to automatically detect/predict damages and generate warning alarms. In this way, an intelligent SHM system capable of automatic damage detection and visualization of structural health data in AR/VR has been developed.

 

Figure 1. The layout of the research work.

Results

According to the suggested methodology, the FEA of a reinforced concrete box girder bridge in Hungary is carried out to analyze the existing condition and damage identification. Based on the FEA, the SHM system of the bridge is also proposed to perform the long-term bridge health monitoring of the bridge. The proposed SHM system of the bridge is shown in Figure. 2.

Figure 2. SHM proposed for a bridge in Hungary.

 

Similarly, the research considers a complete case study of a real bridge in Poland to develop a prototype of an intelligent SHM system. It includes FEA, static and dynamic load testing, BIM modelling, analysis of existing SHM system, automation of the SHM system of the bridge, development of VPL script for automated FE model development, and development of AR application for the visualization of SHM data. The outcomes of this research have successfully integrated the SHM system and BIM technology using the BIMification approach as shown in Figure 3.

Figure 3. BIM-based SHM system integrated with the IoT-based wireless sensors.

 

In order to test the prototype developed, the research will consider a Panewnicka bridge in Poland. This testing will include the development of the SHM system, the installation of wireless sensors according to the developed SHM plan, data collection for a period of three weeks (limited time period because of certain limitations and the risk of sensor theft), and the comparison of simulated and measured data to detect associated bridge damage. Furthermore, an MR (Mixed reality) application is developed for the SHM system of the bridge, deployed to the HoloLens device, and successfully tested in the MR environment. In this way, AR is successfully implemented on a real bridge and the novelty of this research is provided by the visualization of SHM data in the MR environment. This has led to the basis of the intelligent bridge health monitoring. The experimental results are shown in Figure 4.

Figure 4. Visualization of SHM results in MR, using the HoloLens device.

Currently, the measured data is already in the training phase, where genetic algorithms will be tested on the measured data and the appropriate algorithm will be applied to develop an automatic damage detection and alerting system of the developed SHM system.

Expected impact and further research

This research primarily focuses on field implementation; therefore, the developed application has been specifically tested on a real bridge SHM system. The main advantage of this research is for bridge inspectors, as the current practice uses the traditional way of data collection, where bridge inspectors have to collect the data using external drives and can visualize results in a separate step, after data processing. Thus, inspectors do not have the possibility of onsite damage detection, and the work is cumbersome and time-consuming. Recent developments have attempted to resolve this issue by using a web platform which can store data directly on a web server from where the data can be accessed directly without having to go to the site, but it still lacks data visualization and on-site damage identification. In this way, some immediate decisions are neglected which costs way more than the monitoring cost. Thus, the findings of my research proposed a solution to this problem. The future focus is to link BIMified SHM system with the applications of Augmented Reality (AR) and Mixed Reality (MR).

Publications

1.      M. Fawad, M. Salamak, K. Koris, “BIMification of bridge SHM system and BIM based FE model development,” Nature Scientific Reports 2023, (Paper Under Review)

2.      Hijazeen A., M. Fawad, M. SALAMAK, K. Koris, “Implementation of Digital Twin and Support Vector Machine in Structural Health Monitoring of Bridges”, Achieves of Civil Engineering, 2023. (Paper Accepted)

3.      M. Fawad, M. Salamak, Hijazeen A. K. Koris, “Case study of using AR/MR technology for the assessment of the bridge concepts-Part-2”, e-BrIM Volume 1, 2023, Pg. 57–66, https://e-brim.com/bim-for-bridges-february-2023

4.      M. Fawad, K. Koris., M. Salamak, M. Gerges, G. Bednarski, R. Sienko “Non-linear modelling of a bridge with damage evaluation and proposal of SHM system”, Achieves of Civil Engineering, 2022, DOI: 10.24425/ace.2022.141903.

5.      Fawad M., Salamak M., K. Koris., Lazinski P., Poprawa G., “Load testing of Kurow extradosed bridge with FE modelling of its concrete structure”, 13th Central European Congress on Concrete Engineering, Poland, September 2022. (Special Mention award for poster presentation)

6.      M. Fawad, M. Salamak “Case study of using VR/MR technology for the assessment of the bridge concepts-Part-1”, e-BrIM Volume 3, 2022, Pg. 31–37.

Links

https://docs.revit.connect.trimble.com/

https://learn.microsoft.com/en-us/windows/mixed-reality/

https:/mixed-reality development using unity

https://blynk.cloud/dashboard

References

[1]  O. S. Sonbul and M. Rashid, “Algorithms and Techniques for the Structural Health Monitoring of Bridges: Systematic Literature Review,” Sensors, vol. 23, no. 9, pp. 1–29, 2023, doi: 10.3390/s23094230.

[2]  H. Li, S. Li, J. Ou, and H. Li, “Reliability assessment of cable-stayed bridges based on structural health monitoring techniques,” Struct. Infrastruct. Eng., vol. 8, no. 9, pp. 829–845, 2012, doi: 10.1080/15732479.2010.496856.

[3]  H. Li and J. Ou, “The state of the art in structural health monitoring of cable-stayed bridges,” J. Civ. Struct. Heal. Monit., vol. 6, no. 1, pp. 43–67, 2016, doi: 10.1007/s13349-015-0115-x.

[4]  M. Theiler, K. Dragos, and K. Smarsly, “BIM-based design of structural health monitoring systems,” Struct. Heal. Monit. 2017 Real-Time Mater. State Aware. Data-Driven Saf. Assur. - Proc. 11th Int. Work. Struct. Heal. Monit. IWSHM 2017, vol. 1, pp. 829–836, 2017, doi: 10.12783/shm2017/13941.

[5]  A. Scianna, G. F. Gaglio, and M. La Guardia, “Structure Monitoring with BIM and IoT: The Case Study of a Bridge Beam Model,” ISPRS Int. J. Geo-Information, vol. 11, no. 3, 2022, doi: 10.3390/ijgi11030173.

[6]  C. Boddupalli, A. Sadhu, and E. Rezazadeh Azar, “An integrated structural health monitoring tool using building information modeling,” 6th Int. Struct. Spec. Conf. 2018, Held as Part Can. Soc. Civ. Eng. Annu. Conf. 2018, pp. 87–95, 2018.

[7]  A. Jiménez Rios, V. Plevris, and M. Nogal, “Bridge management through digital twin-based anomaly detection systems: A systematic review,” Front. Built Environ., vol. 9, no. April, pp. 1–18, 2023, doi: 10.3389/fbuil.2023.1176621.