In this contribution, we explore the effects of algorithmic transparency and algorithmic accountability on organizations. Algorithmic transparency refers to the idea that algorithms should be transparent to the people who use, regulate, and are affected by those algorithms. This means that the input, processes, and their use should be known, even though the algorithms do not necessarily need to be fair. Algorithmic accountability refers to the idea that organizations should be held responsible for decisions made by the algorithms they use, even if they are unable to explain how the algorithms produce their results. Considering that algorithms are increasingly substituting many human activities—making decisions in place of humans—organizational actors may not always agree with algorithmic decisions, yet they remain legally accountable for their choices. We posit that organizational actors may face an internal legitimacy challenge regarding rationalization and consensus when their preferences do not align with the opaque decisions made by the algorithm. We discuss the role played by organizational actors, as Centaurs or Cyborgs, depending on whether they totally delegate specific tasks to AI or deeply interact with AI to enhance their own decisions. We maintain that the contemporary emphasis on AI does not include an accurate and comprehensive reflection on the long-term effects on organizational systems.