The authors prove that the kernel of the action of the modular group on the centre of a semisimple factorisable Hopf algebra is a congruence subgroup whenever this action is linear. If the action is only projective, they show that the projective kernel is a congruence subgroup. To do this, they introduce a class of generalised Frobenius-Schur indicators and endow it with an action of the modular group that is compatible with the original one.