I should really remember this as I have to use it often. Posting it here in the hope that it will stick eventually. I guess using it all the while instead of ssh-copy-id would do the trick.
$ cat ~/.ssh/id_ed25519.pub | ssh vincent@vlara.co.uk "mkdir -p ~/.ssh && \
cat >> ~/.ssh/authorized_keys"