Examples of Knowledge Graphs: Fully Automatic Detection and Reasoning System
Ensuring construction site safety remains paramount within the industry, prompting extensive research efforts. As one of examples of knowledge graphs in construction project, this study aims to create a fully automated detection and inference system tailored for safety. Central to this endeavor is the integration of a knowledge graph, which not only standardizes industry specifications (written in natural language) and on-site images (collected by sensors) into a computationally tractable format, but also facilitates subsequent decision-making processes and knowledge-based question answering. Unlike existing research approaches, this system endeavors to merge front-end on-site detection with back-end intelligent reasoning, culminating in a knowledge-driven end-to-end solution.
System Framework
Category List
- Artificial Intelligence
- Asset Management
- Digital Twin
- Featured
- Hardware
- Knowledge Graph
- LiDAR
- News
- Site Monitoring
- Vegetation Monitoring
- Virtual Reality
Recent Post
As shown in Fig.1, we divide the entire system into two phases: knowledge graph construction and application. The knowledge graph construction phase includes three stages: data collection, data processing and graph generation, which converts industry specifications and on-site data into the knowledge graph. The knowledge graph application phase includes two stages: graph inference and downstream application. Intelligent reasoning is performed based on the previously obtained knowledge graph, and finally fed back to the construction site, forming a fully automatic closed-loop process.
Technical Details
1. Regulation Knowledge Graph Construction
After obtaining construction safety regulations in Stage 1 (this study takes regulations Construction Industry: OSHA Safety and Health Standards (29 CFR 1926/1910) [1] as an example), the system builds a regulatory knowledge graph as shown in Stage 3 based on natural language processing technology in Stage 2.
Specifically, this study first selected the corpus in the standard and manually annotated it, then used a pre-trained model to encode it, and used a machine learning algorithm to learn its features (this study uses the GPLinker [2] method as an example, as shown in Fig. 2). This enables feature extraction of named entities and relationships. After obtaining the triples, we can construct a regulatory knowledge graph.
2. On-Site Query Graph Construction
After collecting image data of the construction site in the first stage, this study utilizes image segmentation technology to identify objects such as people, equipment, and environment on site. Then the spatial relationship recognition algorithm [3] is used to estimate the location relationship on site. The identified objects serve as nodes of the query graph, and the spatial relationships serve as edges of the query graph. The scene graph generation process is shown in Fig. 3.
3. Reasoning Over Knowledge Graph
Knowledge graph reasoning refers to discovering the relationships hidden between data through logical reasoning or statistical algorithms based on the facts and relationships in the knowledge graph, filling gaps in the knowledge graph, or verifying whether existing facts are true. The current reasoning process only includes on-site compliance inspections and determines whether safety regulations are met by querying and comparing the actual on-site conditions with regulatory standards. Features currently under development include:
1) Knowledge completion: Infer unknown information or hidden patterns through existing facts and relationships to fill gaps in the knowledge graph. This stage relies on graph embedding technology with representation capabilities and graph neural networks that enable link prediction capabilities.
2) Complex logical inference: Based on existing knowledge and rules, perform complex logical inferences, and obtain prediction results. This stage relies on logic-based reasoning engines (such as the logic programming language Prolog).
4. Downstream applications
The reason why the knowledge graph runs through this system is not only its advantage of being easy to modify and optimize (compared to large models, etc.), but also its rich downstream applications, including but not limited to the following points:
1) Visualization
The nodes and edges of the knowledge graph can carry rich semantic information, such as entity types, attribute values, relationship types, etc., so this information can be displayed during visualization, allowing users to intuitively understand the meaning and structure of the graph. Knowledge graphs built through graph databases such as Neo4J (shown as Fig. 4) can be easily stored and queried, and represented intuitively.
2) Operation Recommendation
This system is not only dedicated to detecting whether construction safety complies with specifications, but also strives to provide guidance on how to improve it. This advantage relies on the recommendation ability of the knowledge graph. Recommendation systems based on knowledge graphs can alleviate the cold start problem of traditional recommendation systems, that is, they can still provide effective recommendations in new scenarios. In addition, the recommendation system of the knowledge graph has high interpretability and can clearly explain the reasons and basis for the recommendation results, helping users understand the rationality and applicability of the recommendations.
3) Question & Answer (Q&A) System
To further improve the ease of use, another development application of this research is the Q&A system. Complete knowledge inquiry and feedback by answering natural language questions raised by users. It can help users quickly obtain the information they need and improve information retrieval efficiency and user experience. Knowledge graphs provide structured knowledge representation that can support more precise and deeper question answering. This research is dedicated to developing a Q&A system based on prompt learning, using prompt learning to perform chain reasoning from the knowledge graph [4] (as shown in Fig. 5).
Currently, this system has initially completed the research and development of the knowledge graph construction phase, and is committed to developing and improving the knowledge graph application phase.
We can use this system not only for for construction site compliance testing, but also for infrastructure testing, urban planning recommendations and many other fields in the future. We look forward to cooperating and communicating with colleagues in various industries. For more information or specific inquiries, please feel free to contact us.
Reference
[1] OSHA, U, Construction Industry: OSHA Safety and Health Standards (29 CFR 1926/1910), US Department of Labor, Occupational Safety and Health Administration, Washington, DC, 2020. https://www.osha.gov/laws-regs/regulations/standardnumber/1926
[2] Chen, Jiyu, et al. “GPBP: Pipeline Extraction of Entities and Relations for Construction of Urban Utility Tunnel Knowledge Graph.” 2023 CAA Symposium on Fault Detection, Supervision and Safety for Technical Processes (SAFEPROCESS). IEEE, 2023. doi: 10.1109/SAFEPROCESS58597.2023.10295733.
[3] Fang, Weili, et al. “Knowledge graph for identifying hazards on construction sites: Integrating computer vision with ontology.” Automation in Construction 119 (2020): 103310. https://doi.org/10.1016/j.autcon.2020.103310
[4] Choudhary, Nurendra, and Chandan K. Reddy. “Complex Logical Reasoning over Knowledge Graphs using Large Language Models.” arXiv preprint arXiv:2305.01157 (2023). https://doi.org/10.48550/arXiv.2305.01157
[5] Ma, Z.; Zhang, S.; Jia, H.; Liu, K.; Xie, X.; Qu, Y. A Knowledge Graph-Based Approach to Recommending Low-Carbon Construction Schemes of Bridges. Buildings 2023, 13, 1352. https://doi.org/10.3390/buildings13051352