<p>We study fundamental block-structured integer programs called tree-fold and multi-stage IPs. Tree-fold IPs have a constraint matrix with independent blocks linked together by few constraints in a recursive pattern. Transposing this constraint matrix yields the constraint matrix of multi-stage IPs. The state-of-the-art algorithms to solve these IPs have an exponential gap in their running times, making it natural to ask whether this gap is inherent. We answer this question in the affirmative. Assuming the Exponential Time Hypothesis, we prove lower bounds showing that the exponential difference is necessary. This also proves that the known algorithms are essentially optimal. Moreover, we prove unconditional lower bounds on the size of the Graver basis elements, a fundamental building block of all known algorithms to solve these IPs. This shows that none of the current approaches can be improved beyond this bound unconditionally.</p>