How does focus priority work when playing a Rengar, Trophy Hunter

It depends on whose focus window you're in when he hits the table — Rengar is a [Reaction] play (his Ambush lets him go to a battlefield with enemy units even if you have none there), so you can only play him when you hold focus in a Showdown Open State or priority on an existing chain (Rule 347.1). Once his chain item closes and the turn returns to Open, focus moves to the next player in turn order (Rule 346), so you can't play him and then immediately start another chain — your opponent gets first crack.

Two things people usually get wrong here. First, nobody can respond to Rengar being played: units finalize and enter play immediately during the Finalize step (Rule 337.2), so there's no reaction window to the play itself — only triggered abilities (a "when you play me," "when a unit enters here," etc.) go on the chain and can be responded to.

Second, playing him doesn't earn you focus or make you the attacker. The player who applied Contested keeps the Attacker designation and the opening focus (Rule 345); Rengar entering a battlefield you don't control makes you the Defender (FAQ #10056), and any Shield/Assault only matter once combat actually begins. If combat has already started, the combat chain opened from triggered abilities, so focus does not pass when that chain empties (Rule 346.1).