RC-MDNet: Risk-Controlled Malicious Domain Detection for Open-World Network Security

Open-world Internet traffic routinely violates the i.i.d. assumption, causing malicious domain detectors to become overconfident on out-of-distribution (OOD) inputs and to incur costly false alarms or misses. We introduce RC-MDNet, a risk-controlled framework that couples (i) post-hoc probability calibration via temperature scaling, (ii) a dual-threshold uncertainty-aware selective predictor that trades coverage for reliability, and (iii) conformal prediction (overall and class-conditional) to furnish distribution-free risk guarantees. Using a lightweight character-level BiLSTM backbone, we evaluate across internal splits (Random, Apex, Cold-start, TLD-holdout) and external zero-shot corpora. RC-MDNet maintains high utility while reducing effective error through abstention, achieving near-target automatic accuracy (Auto-F1 \(\ge 0.995\) ) with tunable coverage and certified risk. Robustness studies under character-level perturbations (InsDel, CharRep, Homoglyph) show stable performance, and ablations confirm the importance of calibration and the double-threshold design, while class-conditional conformal control supports asymmetric cost preferences relevant to security operations. Overall, RC-MDNet delivers graceful degradation and interpretable, deployment-grade reliability under shift, offering a practical path to risk-aware DNS threat detection.