We explore in-context learning (ICL), a popular paradigm for inference with Large Language Models (LLMs), in a controlled experimental setup using synthetic training data. Using a range of small transformer models trained from scratch, we focus on a mathematical task with simple yet precise prompts: learning a linear function f from a sequence of inputs \(x_i\) and their corresponding function values \(f(x_i)\) . Our findings challenge the prevailing narrative that transformers adopt algorithmic approaches like linear regression to in-context learn (ICL) a linear function. We observe that all models have “boundary values” that limit generalizability. While we can extend boundary values with training distributions over a wider range, we lose the precision of models trained on distributions with more restricted ranges. Thus, we see a dilemma for ICL at least in some tasks: either models will lack generalizability or precision.