Two closely related topics, higher order Bohr sets and higher order almost automorphy, are investigated in this paper. Both of them are related to nilsystems. In the first part, the problem which can be viewed as the higher order version of an old question concerning Bohr sets is studied: for any $din mathbb{N}$ does the collection of ${nin mathbb{Z}: Scap (S-n)capldotscap (S-dn)neq emptyset}$ with $S$ syndetic coincide with that of Nil$_d$ Bohr$_0$-sets? In the second part, the notion of $d$-step almost automorphic systems with $dinmathbb{N}cup{infty}$ is introduced and investigated, which is the generalization of the classical almost automorphic ones.