We do it sometimes. Haven't in quite a while, but it wasn't weird for us at all.
Just like some people use "baby" as a term or endearment... and it doesn't mean you think you're with a baby, nor does it make you think of your own baby (if you have one). "Daddy" is the same way. It clearly doesn't refer to one's own father, nothing of the sort.
There is a previous discussion on this topic around here somewhere, I'll see if I can drag it up for you. Might be helpful, or you at least might find it interesting!
Edit: Well, here are the results
from my forum search. The thread I remembered is on there, but so are a bunch more! So instead of linking to each of them individually, I thought I'd just share this link so you can see all of them. Thought you might enjoy checking them out!