This chapter explores the challenges and solutions for processing multimodal information, specifically in the context of multimodal sentiment analysis and hate speech detection. The increasing amount of multimodal data, such as text, images and videos, presents unique challenges for machine learning algorithms. These challenges include the integration and fusion of information from multiple modalities to acquire the overall context. In this chapter, first, we present an overview of recent developments on multimodal learning techniques in the context of sentiment and hate speech detection; second, we present a multimodal model that combines different visual aspects and features for multimodal sentiment detection; and third, we present a multi-task multimodal model for misogyny detection in multimodal memes.