Scholarlink

This paper presents a novel application of multimodal large language models (LLMs) to enhance the learning and application of building energy modeling. The study leverages Retrieval-Augmented Generation (RAG) models integrated with a dataset of 59 publicly available YouTube video tutorials focused on EnergyPlus and OpenStudio. Unlike traditional LLM methods, this approach is unique in its utilization of three LLMs to process and integrate different types of data: text, screenshots, and video references. The preprocessing phase utilizes Google’s transcription service to convert video content into text, which is then summarized using the T5 model and embedded with the Instructor Embedder model. Meta’s Llama v2 7b model handles user queries, extracting relevant information and providing detailed responses enriched with visual references, including exact video minutes and screenshots. Unlike traditional LLM models, the proposed model delivers comprehensive responses that go beyond text, providing users with reference videos with exact timestamps and screenshots. Furthermore, the proposed web interface presents these enriched responses in one page, significantly reducing the time needed for users to understand and apply complex energy modeling concepts. This framework demonstrates the potential of multimodal LLMs in creating powerful educational tools for architects, engineers, and students. The entire workflow was completed on a laptop with a single 4GB GPU, demonstrating the feasibility of implementing such a system on relatively modest hardware.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Leveraging Multimodal Large Language Models for Enhanced Learning and Application in Building Energy Modeling

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Rania Labib

摘要

This paper presents a novel application of multimodal large language models (LLMs) to enhance the learning and application of building energy modeling. The study leverages Retrieval-Augmented Generation (RAG) models integrated with a dataset of 59 publicly available YouTube video tutorials focused on EnergyPlus and OpenStudio. Unlike traditional LLM methods, this approach is unique in its utilization of three LLMs to process and integrate different types of data: text, screenshots, and video references. The preprocessing phase utilizes Google’s transcription service to convert video content into text, which is then summarized using the T5 model and embedded with the Instructor Embedder model. Meta’s Llama v2 7b model handles user queries, extracting relevant information and providing detailed responses enriched with visual references, including exact video minutes and screenshots. Unlike traditional LLM models, the proposed model delivers comprehensive responses that go beyond text, providing users with reference videos with exact timestamps and screenshots. Furthermore, the proposed web interface presents these enriched responses in one page, significantly reducing the time needed for users to understand and apply complex energy modeling concepts. This framework demonstrates the potential of multimodal LLMs in creating powerful educational tools for architects, engineers, and students. The entire workflow was completed on a laptop with a single 4GB GPU, demonstrating the feasibility of implementing such a system on relatively modest hardware.