Go to the Trophy leaderboard “FRIENDS” tab. Then tap your friend's name to send them an invite to play!

I did that and it said we needed the same server

Right. I do that, with the same result.

The game connects you to the closest server to limit lag. We are still working on a way to instead connect to the server that your friend is on. I am sorry for the trouble!