This paper focuses on the problem of developing automated methods for writer identification from handwriting. It introduces convolutional transformer encoder (CTE), a novel architecture designed for determining authorship from handwritten text, be it modern or historical handwriting. CTE is the first of its kind designed to operate on historical handwritten fragments, setting it apart from existing methods that rely on entire document pages. The implementation of CTE across multiple handwriting datasets is shown to enjoy superior performance over state-of-the-art lexical-segmentation methods for modern handwriting datasets and competitive performance with folio-directed methods on historic handwriting datasets. CTE tracks subtle features and cues from handwriting strokes to distinguish authorship.