Although accountability is often seen as the best way to address the dangers of algorithmic decision making, too little attention has been given to understanding accountability and how it is constituted. Seeking to fill that gap, a general account of accountability is provided here and then extended to algorithmic decision making. Drawing on the work of Mark Bovens, accountability is presented as a social practice consisting of actors and forums who share a set of beliefs central to which is the belief that the actor has an obligation to explain certain kinds of behavior to a forum with the possibility that the forum may render sanctions. Importantly, the shared beliefs that constitute accountability are embedded in social norms. Extending this account to algorithmic decision making reveals a good deal of uncertainty about the actors, forums, and norms that currently constitute accountability for algorithmic decision making. Finally, when it comes to the role of algorithms in social practices of accountability, although algorithms are efficacious in algorithmic decision making, only human actors can be the actors called to account.