For some strange reason, the SSH keys aren't working on CentOS. I cannot make any sense out of it, since everything is more or less identical (the only difference is that the comment in my SSH key was omitted and replaced by the local path).
However, after i've copied in the key with ssh-copy-id I can then replace it with the old one, and it starts to work.
Needless to say, it's a very odd issue.