What would happen if I activated the effect of Baited Hook on a unit that was under the effect of Tactical Retreat?

If you activate Baited Hook targeting a friendly unit that is under the effect of Tactical Retreat, the unit will be saved, and you will not be able to play a unit from your deck.

According to Rule 369.1 and FAQ #4178, FAQ #8203, and FAQ #3536, Tactical Retreat creates a replacement effect that intercedes during the resolution of the "kill" instruction. Because the replacement effect changes the "kill" action to a "recall" action, no unit is actually killed.

Sequence of Events:

1. Activation: You activate Baited Hook by paying its costs and declaring a target friendly unit.
2. Resolution: Baited Hook begins to resolve and attempts to kill the targeted unit.
3. Replacement: Tactical Retreat's replacement effect intercedes, replacing the "kill" with a "recall." The unit is sent to your base instead of the trash.
4. Check: Baited Hook looks for the Might value of the "killed unit" to determine what you can play. Because no unit was actually killed (the kill action was replaced), there is no Might value to reference (it returns "null").
5. Finalization: You look at the top 5 cards of your Main Deck, but because there is no valid Might value to compare against, you cannot play any unit. You then recycle all 5 cards.

Summary: You will look at the top 5 cards of your deck, but you will be unable to play any of them, and they will all be recycled. This is because Baited Hook specifically requires a "killed unit" to reference for its Might value, and the save effect from Tactical Retreat prevents that game action from occurring.