Phone Inadvertently Upgraded to 4.6.4. Is downgrade to 4.6.1 via ADB Safe?

I just received two phones (from the excellent eBay deal that I'm sure will drive more popularity of this forum), and stupidly, while working on the first one (getting CM11 on which went seamlessly), I forgot to disable updates on the 2nd. When I turned to the 2nd phone, my heart sank as I saw that it was now on 4.6.4 (464000220). My immediate thought was to boot into recovery and sideload 4.6.1 (820) but I remember reading that there is danger in downgrading this way?

Can anyone shed some light if downgrading this was is safe, or if I am no royally screwed? While Kingroot has no problem rooting 4.6.4, I'm trying to get to CM11 so I need Safestrap which apparently require 4.6.1 (820).

Any advice would be much appreciated!

Thanks a bunch!