I don’t think I ever could be.
God is generally held to be omnipotent (all-powerful), omniscient (all-knowing) and omnibenevolent (all-good). So perhaps Frank was referring to this conception of God when in the previous lines he spoke of never being alone. If you believe in a Deity figure like the one described above, well how could such a figure ever NOT have your back!
To help improve the meaning of these lyrics, visit "Frank Ocean's Open Letter on Tumblr" by Frank Ocean and leave a comment on the lyrics box