Robust Hilbert space fragmentation in group-valued loop models