The same shifting occurs also with the QAAC encoder. See :

 

 

When the HE-AAC mode is selected, a negative audio delay has to be specified in order to compensate the shifting.

It must exactly -5184 samples OR -125 miliseconds.

 

Example with QAAC (I don't know FDKAAC commands) :

• qaac.exe --cbr 64 --he --adts --delay -5184s --quality 2 --rate keep - -o "output"
• qaac.exe --cbr 64 --he --adts --delay -0:0:0.125 --quality 2 --rate keep - -o "output"