Rapid progress in machine learning has led to the development of Large Language Models (LLMs), which have extraordinary capabilities, such as creative writing and report summarization. However, applications powered by these models require significant amounts of compute resources and usually run on expensive and power-hungry accelerators, such as Graphics Processing Units (GPUs). To meet the demands of these applications and lower their operating costs, new custom LLM inference serving systems have been developed. These systems exploit the unique characteristics of LLMs to serve inference requests in a more performant and resource-efficient manner, primarily by leveraging scheduling and memory management techniques. Our early work highlights the difference in performance of these systems across use cases with different characteristics, such as question-answering and text summarization. Furthermore, we identify significant room for improvement in the performance of specific use cases via improved memory management. Overall, this line of work plans to build novel LLM inference serving systems considering different objectives, such as application performance, model accuracy, and energy efficiency, while taking into account the unique characteristics of different use cases and the available hardware.