Your baby is just getting used to recognizing faces, when you introduce the Hats Off game! Your baby won’t be fooled for long, but he’ll enjoy the fun of taking off the hat and putting it back on again.
Sometimes babies get scared when people’s appearances change. If your baby starts to get upset, put the hat on only briefly, then remove it and show him you’re still Mom/Dad. If he continues to be upset, play the game at a later date.