If I attack with a unit and neither of us die, do I go back to where I came from?

No, you do not return to the battlefield you came from. When an attacking unit is recalled after combat, it always returns to your base.

A Recall is defined as a permanent being relocated from its current position to its controller's base. Because a recall is not a move, the unit does not return to its origin; it is sent directly to the base.

This applies regardless of where the unit moved from before the combat occurred. Even if a unit moved from one battlefield to another, a subsequent recall will always send that unit back to the base.

Note: I could not fully verify this ruling against the retrieved rules.