SoC design has shifted towards heterogeneous architectures with CPUs and domain-specific accelerators to keep up with the growing computational demands of current-day applications. These accelerators demand high on-chip and off-chip memory bandwidth and require efficient management of shared system resources. We characterize Internet Protocol Security (IPsec), a high-throughput application, by collecting the memory traces of this application running on the accelerators and CPU cores of NXP LX2160A SoC. We use this characterization to design a simulation infrastructure for simulating IPsec on possible domain-specific architectural extensions and perform a design-space exploration across various general-purpose memory management policies. We propose APPAMM, an application-specific predictive packet-aware memory management policy using the knowledge of IPsec to improve performance for next-generation SoCs. Using our approach to manage memory for different input packet streams, we report improvements of up to 22x in the packet drop rate and peak throughput.