Effective investment techniques rely on portfolio optimization and financial forecasting, which need models that integrate historical market data and emerging patterns. However, conventional forecasting approaches frequently have trouble adapting to market changes, so more sophisticated techniques are required. This paper presents BL-TGAN (Black-Litterman Transformer-GAN), incorporating the refining methods of Generative Adversarial Networks (GANs) and the prediction powers of Transformer Networks into the Black-Litterman model. The transformer component captures temporal relationships and detects market trends, while GAN adds adaptability by adjusting predictions to explain financial conditions. The proposed model is evaluated using a daily stock dataset comprising seven organizations listed on the Australian Securities Exchange (ASX). Its performance is benchmarked against standalone Transformer and GAN models utilizing Mean Absolute Error (MAE), Mean Squared Error (MSE), and Normalized Mean Squared Error (NMSE). The results exhibit that BL-TGAN performs the highest accuracy across all metrics, with an MAE of 0.0380, MSE of 0.2102, and NMSE of 0.0027. BL-TGAN offers a dependable and adaptable method for making investment decisions by combining time-series forecasting with market dynamics modeling.