Towards a self-cognitive complex product design system: A fine-grained multi-modal feature recognition and semantic understanding approach using large language models in mechanical engineering

Facing the promising tendency of human-artificial intelligence (AI) collaborative product design, fine-grained and multi-modal mechanical part recognition and semantic understanding have become a basic task for achieving a self-cognitive product design system. However, traditional semantic understanding approaches for mechanical parts can only handle single-modal data, which is either textual or image data, resulting in the following limitations 1) insufficient mining on fine-grained part’s functional/behavioral/structural information, and 2) ineffectiveness on multi-modal part information alignment, therefore restricting the intelligence level of the previous product design assistants. To mitigate these challenges, this paper proposes a fine-grained multimodal reasoning approach for mechanical part semantic understanding. The proposed approach utilizes a pre-trained Convolutional Neural Network (CNN) for visual feature extraction, a large language model (LLM) called LLaMA3 for advanced textual analysis, and a Unified Feature Fusion Module (UFFM) to facilitate robust cross-modal interactions. A positive and negative sample generation mechanism is implemented to refine the model’s ability to discern subtle variations in complex components. Experimental evaluations on the Industrial Part Multimodal Dataset (IPMD) demonstrate a significant improvement in classification accuracy, providing a more precise and intelligent solution for the semantic understanding in complex product design systems.