For each positive integer N, define \( S'_N \ =\ \{1 < d < \sqrt{N}: d|N\} \text{ and } L'_N \ =\ \{\sqrt{N} < d < N : d|N\}. \) Recently, Chentouf characterized all positive integers N such that the set of small divisors \(\{d\le \sqrt{N}: d|N\}\) satisfies a linear recurrence of order at most two. We nontrivially extend the result by excluding the trivial divisor 1 from consideration, which dramatically increases the analysis complexity. Our first result characterizes all positive integers N such that \(S'_N\) satisfies a linear recurrence of order at most two. Moreover, our second result characterizes all positive N such that \(L'_N\) satisfies a linear recurrence of order at most two, thus extending considerably a recent result that characterizes N with \(L'_N\) being in an arithmetic progression.