Scholarlink

Transformer-based models (e.g. BERT, RoBERTa) have revolutionized modern natural language understanding. However, their large parameter counts and intensive computation makes deployment on resource-constrained devices unfeasible (e.g. smartphones, IoT boards, edge servers). This analysis reviews techniques to optimize BERT-style architectures to run efficiently under strict memory, computation power, and energy budgets. We compare and contrast widely used compression techniques: pruning, quantization, knowledge distillation, and architecture redesign. We evaluated these models on three standard NLP tasks: Sentiment Analysis (SST-2 Dataset), Paraphrase Detection (Microsoft Research Paraphrase Corpus - MRPC), and Topic Classification (AG News Dataset). We report the models’ performance on General Language Understanding Estimation (GLUE) benchmarks, and compare hardware-level metrics under edge device constraints, a simulated Raspberry Pi environment was created using QEMU on an Ubuntu host, replicating a Raspberry Pi 3 Model B+ (1 GB RAM, quad-core ARM processor). Our analysis aims to guide practitioners and researchers in selecting and designing BERT-derived models for efficient on-device NLP. The experiment shows the following analysis, BERT-base sets a high accuracy (89.88%), but has prohibitive resource demands (latency: 1504 ms, memory: 397 MB). MobileBERT offers a balanced solution (90.28%), achieving similar accuracy with better efficiency. TinyBERT maximizes efficiency (latency: 41 ms, memory: 14 MB) but sacrifices performance (80.19%), while DistilBERT strikes a moderate trade-off, suitable for various on-device NLP applications.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Lightweight Transformer Models for Resource–Constrained Environments: A Comparative Analysis

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Pranav Kini,
  • Omkar Sista,
  • Rishikesh Prabhu,
  • Shubha Puthran

摘要

Transformer-based models (e.g. BERT, RoBERTa) have revolutionized modern natural language understanding. However, their large parameter counts and intensive computation makes deployment on resource-constrained devices unfeasible (e.g. smartphones, IoT boards, edge servers). This analysis reviews techniques to optimize BERT-style architectures to run efficiently under strict memory, computation power, and energy budgets. We compare and contrast widely used compression techniques: pruning, quantization, knowledge distillation, and architecture redesign. We evaluated these models on three standard NLP tasks: Sentiment Analysis (SST-2 Dataset), Paraphrase Detection (Microsoft Research Paraphrase Corpus - MRPC), and Topic Classification (AG News Dataset). We report the models’ performance on General Language Understanding Estimation (GLUE) benchmarks, and compare hardware-level metrics under edge device constraints, a simulated Raspberry Pi environment was created using QEMU on an Ubuntu host, replicating a Raspberry Pi 3 Model B+ (1 GB RAM, quad-core ARM processor). Our analysis aims to guide practitioners and researchers in selecting and designing BERT-derived models for efficient on-device NLP. The experiment shows the following analysis, BERT-base sets a high accuracy (89.88%), but has prohibitive resource demands (latency: 1504 ms, memory: 397 MB). MobileBERT offers a balanced solution (90.28%), achieving similar accuracy with better efficiency. TinyBERT maximizes efficiency (latency: 41 ms, memory: 14 MB) but sacrifices performance (80.19%), while DistilBERT strikes a moderate trade-off, suitable for various on-device NLP applications.