If my opponent moves Vex Cheerless to an empty battlefield and I answer with Rengar Trophy Hunter, what units move back to base and does anyone stay at the battlefield?

To answer your question regarding the showdown between Vex, Cheerless and Rengar, Trophy Hunter:

Sequence of Events

1. Opponent's Move: Your opponent moves Vex, Cheerless to an open battlefield. This initiates a non-combat showdown, and your opponent is the Attacker (Rule 442.1.a.1).
2. Your Response: You play Rengar, Trophy Hunter to that battlefield as a [Reaction]. Because there is an enemy unit (Vex) present, you satisfy the condition for Rengar's ability. By doing so, you become the Defender (Rule 442.1.a.2).
3. Showdown Conclusion: The chain empties, both players pass priority, and the non-combat showdown transitions into a combat showdown.

Combat Resolution

• Combat Damage Step: Both Vex, Cheerless and Rengar, Trophy Hunter deal damage to each other based on their Might.
• Combat Cleanup:
  1. Heal: All units heal (Rule 461.1.a.1).
  2. Recall: According to Rule 461.1.a.2, "Recall Attackers present at the Battlefield if Defenders are still present."

Result

• Who moves back to base: The Attacker (Vex, Cheerless) is recalled to the opponent's base because the Defender (Rengar, Trophy Hunter) remains at the battlefield.
• Who stays: If Rengar, Trophy Hunter survives the combat damage, he stays at the battlefield.
• Conquering: If Rengar remains on the field as the only unit after the Vex is recalled, you will establish control of the battlefield and potentially "Conquer" it (resulting in a point), provided you have not already scored that battlefield this turn.

This interpretation is based on the rules for Attacker/Defender designations (Rule 442.1.a), the specific combat cleanup steps (Rule 461.1), and the ability of Rengar, Trophy Hunter as retrieved.

Note: I could not fully verify this ruling against the retrieved rules, specifically regarding the Might values used.