If I retreat my only unit and my opponent has attacker, do they lose the attacker buff?

Ruling: No, they do not lose the attacker designation. Once a combat has started and a unit has gained attacker designation, they retain that designation until the end of the combat.

Nuances:

• Retreating or flashing a unit away from the battlefield doesn't stop it from being a combat