what rule says rengar is defender when he ambushes into a battlefield the enemy moved into which was open before. with no friendly unit

According to the game's rules for Attacker/Defender designations, your status is determined by who initiated the contest of the battlefield and when you joined it.

The Relevant Rules

• Rule 442.1.a.1: The player who first moves a unit to an unoccupied (open) battlefield causes it to become Contested and is designated as the Attacker.
• Rule 442.1.a.2: A player who subsequently plays a unit to a battlefield that is already Contested by an opponent is designated as the Defender.

Why this applies to Rengar, Trophy Hunter

When your opponent moves a unit to an open battlefield, they fulfill the condition of Rule 442.1.a.1, becoming the Attacker and making the battlefield Contested.

Because Rengar, Trophy Hunter has a unique ability that allows him to be played as a [Reaction] to any battlefield with enemy units, you can play him there even if you have no other presence on the board. Once he enters that Contested battlefield, you fulfill the requirement of Rule 442.1.a.2, which results in you being designated as the Defender.

This interpretation is based on Rule 442.1.a of the Riftbound core rules and is consistent with multiple RiftJudge FAQs (e.g., #9992, #9820, and #10056) regarding this specific card interaction.