The same shifting occurs also with the QAAC encoder. See :
When the HE-AAC mode is selected, a negative audio delay has to be specified in order to compensate the shifting.
It must exactly -5184 samples OR -125 miliseconds.
Example with QAAC (I don't know FDKAAC commands) :
- qaac.exe --cbr 64 --he --adts --delay -5184s --quality 2 --rate keep - -o "output"
- qaac.exe --cbr 64 --he --adts --delay -0:0:0.125 --quality 2 --rate keep - -o "output"