Examining the colon wall in transabdominal ultrasound images emerges as a promising, non-invasive approach for diagnosing and managing ulcerating colitis, a widespread inflammatory bowel disease affecting millions of people worldwide. However, due to its intricacies, this examination has thus far been confined to experts with specialized training. To the best of our knowledge, we are the first to evaluate automated colon wall segmentation using several advanced deep learning segmentation architectures in combination with established and specialized loss functions. To this end, we publish a new open-source dataset, named C-TRUS, including expert annotations for 827 transabdominal ultrasound images as well as image quality categorizations. Furthermore, we establish inter-observer variability, and find that colon wall segmentation is challenging even for medical experts, reaching a moderate average consensus Dice score of 0.6134. The best performing model is the Mask R-CNN architecture, achieving an average Dice score of 0.7249 across all image quality categories and a Dice score of 0.8218 on high quality images. We provide the C-TRUS dataset at https://github.com/wwu-mmll/c-trus .