For Ivern Brush and Power Nexus, if I remove the Brush, can I still use the Power Nexus after?

It depends on the timing — specifically, which turn you swap back and try to use Power Nexus.

On the same turn you swap back: No. When you score at Brush (which allows the swap back), that's already a score at that battlefield location this turn. When Power Nexus returns via the swap back (Rule 438.7.b), it inherits "all current effects and statuses" — including the fact that this battlefield has already been scored this turn. Rule 465 says you can only score once per battlefield per turn. So you cannot use Power Nexus's ability to score again that same turn.

On a future turn after swapping back: Yes. On a new turn, the per-turn scoring limit resets. You can move a unit to Power Nexus, satisfy "When you hold here," and pay [A][A][A][A] to score 1 point via its ability. The swap-back fully restores Power Nexus to that battlefield spot — it's just that the turn you swapped back is locked out for additional scoring at that location.