Does my opponent's defending unit immediately stop being a defender if my only attacking unit leaves combat?

No — the defending unit keeps its Defender designation. Designations are only removed when Combat fully ends (Rule 466.7.a), or if that unit itself leaves the battlefield the combat is taking place at (Rule 323.2.c).

An attacker leaving, dying, or being recalled doesn't end the combat either — it stays active until all players pass Focus consecutively, and cards that care about "while it's a defender" keep applying. Same logic runs the other way: an attacker keeps its Attacker designation even if the defender moves away or the combat becomes unopposed (FAQ 1398, 4660).

The only exception is if the defending unit itself is moved off that battlefield (or its controller changes) — then a Cleanup strips the designation from it, which can matter for abilities like Assault/Shield that depend on it (FAQ 187).