Towards absolutely stable ergodicity breaking in two and three dimensions