This paper presents a reinforcement learning framework for optimizing last-mile delivery in an intermodal logistics system, using a configurable theoretical urban environment. Unlike prior case-specific studies, this work introduces a generalizable simulation setup with Euclidean distance routing, synthetic node placement, and time-dependent congestion modeled using a synthetic Travel Time Index (TTI). Public transport is integrated as a transshipment option within a pickup and delivery structure and can also transport goods between stations of the same network, enabling inter-station freight transfers. The custom environment supports operational-level decisions including vehicle routing, dynamic request assignment, and simplified tramway-based transshipment. A Proximal Policy Optimization (PPO) agent is trained and evaluated on policy stability, convergence, and reward efficiency. Over the training period, the agent demonstrated consistent improvement in mean reward, despite volatility, and a gradual reduction in entropy loss. This work contributes a scalable, traffic-aware simulation framework and emphasizes the importance of refined exploration strategies for agent generalization.