Scholarlink

Generating large datasets for training natural language processing (NLP) algorithms typically involves considerable effort in curation and crowdsourcing, requiring humans with domain-specific expertise and time. Our study analyzes an approach utilizing Large Language Models (LLMs) to autonomously produce book summaries and question–answer pairs from textual data of book PDFs. Leveraging two openly accessible LLMs—“DeciLM-7B-instruct-GGUF” and “gemma-2b-it-GGUF”, supported by GPU offloading for enhanced processing efficiency, our methodology aims to streamline data generation for NLP applications. The language model chains are zero-shot prompt-engineered to generically perform the NLP tasks. The entire source text is split into chunks on which individual tasks were carried out. The results of our study are stored in CSV files, showcasing the potential of LLMs in automating the creation of NLP training datasets. For access to our codebase, please visit our repository at https://github.com/lomash-relia/QAPipleline-using-LLMs

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Transforming Text into Knowledge: Leveraging LLMs, Zero-Shot Chains, and LangChain for Automated Generation of Book Summaries and Q&A Datasets

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Lomash Relia,
  • Jaimin Patel,
  • Nisha Panchal,
  • Dweepna Garg

摘要

Generating large datasets for training natural language processing (NLP) algorithms typically involves considerable effort in curation and crowdsourcing, requiring humans with domain-specific expertise and time. Our study analyzes an approach utilizing Large Language Models (LLMs) to autonomously produce book summaries and question–answer pairs from textual data of book PDFs. Leveraging two openly accessible LLMs—“DeciLM-7B-instruct-GGUF” and “gemma-2b-it-GGUF”, supported by GPU offloading for enhanced processing efficiency, our methodology aims to streamline data generation for NLP applications. The language model chains are zero-shot prompt-engineered to generically perform the NLP tasks. The entire source text is split into chunks on which individual tasks were carried out. The results of our study are stored in CSV files, showcasing the potential of LLMs in automating the creation of NLP training datasets. For access to our codebase, please visit our repository at https://github.com/lomash-relia/QAPipleline-using-LLMs