Large Language Model (LLM) inference heavily relies on cloud systems, resulting in inherent limitations such as prolonged delays, escalated bandwidth expenditure, and potential data security risks. Edge computing emerges as a solution by enabling LLM execution on edge devices, closer to data sources. However, challenges such as device heterogeneity and limited memory capacity pose significant barriers to conventional tensor parallelism of LLM. Therefore, we propose EdgeInfer-TP, a resource-aware collaborative tensor parallelism inference system tailored for heterogeneous edge devices. It integrates MILP-based model allocation optimization and multi-level dynamic offloading of KV cache to enable efficient and stable collaborative inference on resource-constrained devices. The experimental results across multiple real-world heterogeneous devices and diverse network environments demonstrate that our approach achieves up to 22.23% latency reduction and 27.87% throughput improvement over baseline approaches.