Critical Code Sudies with AI: conversing with LLMs about code
摘要
In this paper, we explore some theoretical approaches and practical uses of AI to assist with the work of Critical Code Studies to explicate codes as cultural objects. Although programmers, from novices to seasoned professionals, use LLMs to write, debug, and even summarize or explain code, LLMs are not yet used pervasively for critical readings of code. We offer a complement to the code generation paradigm with code interpretation paradigm, using the LLM as an interpretive “conversational partner.” Misinterpretation and specifically AI hallucination must be negotiated differently: there is no run-button or test suite to check that an interpretation will not compile. Many researchers and scholars initially struggle with the daunting undertaking of Critical Code Studies due in part to a widespread cultural bias that code is a functional system consisting only of denotations of instructions—and thus presumptively an inappropriate object for critical interpretive methods designed for non-technical systems, such as literature and the arts. Yet, LLMs freely produce commentary on code using semiotic hermeneutics, easily modeling an “agnostic” attitude toward the relationship between any Python program and psychoanalysis. Furthermore, LLMs are code language polyglots, making a wide variety of historical and code objects potentially more accessible to cultural critics who lack specific fluencies. LLMs also possess a useful aptitude for disrespecting disciplinary boundaries between technical and cultural objects when prompted. This has a particular utility to CCS methods and the field. We offer a case study using LLMs in CCS research on an important long-form code object, the source code from Apollo 11. This case study offers initial findings and some suggestions for best practices. Our initial explorations point to the potential of the tools to help with interpretation. However, our experiments reveal one very urgent caution: because of the way LLMs generate output, they can be overly influenced by natural language elements in the code. Moreover, code readers unversed in the languages the LLM is analyzing will have no ability to recognize when a misrepresentation occurs.