If I use Thrill of the Hunt on a unit and banish it, does it still keep the minus might from spells?

No, the unit will not retain the minus Might (or any other modifiers) from previous spells.

Reasoning

When your unit is banished by Thrill of the Hunt, it leaves the battlefield and enters the Banishment zone. As soon as the unit leaves the board, it ceases to be the same game object it was previously.

According to Rule 705, when a unit leaves play, all buffs/modifiers are removed from it. Furthermore, when Thrill of the Hunt instructs you to play that unit to a battlefield, it enters as a "new object." Because it is a new object, any previous effects or spells that were targeting or modifying the original unit no longer apply to it.

This is consistent with the logic established in RiftJudge FAQ #9687, which clarifies that a unit moved to Banishment and then re-played enters as a new object, losing any previously attached equipment or persistent effects.