I have a 6 might unit and opponent plays a spell that says give a unit -2 might. Does this mean that my unit only has an attack value of 4?

That is correct. In Riftbound, a unit's Might value determines the amount of combat damage it contributes during the Combat Damage Step. When a spell grants your 6-Might unit -2 Might, its current Might becomes 4.

According to Rule 472.3, arithmetic effects like Might increases or decreases are applied in the Arithmetic layer. Because your unit's current Might is now 4, it will contribute 4 combat damage during the Combat Damage Step.

It is important to remember that in Riftbound, damage is not "subtracted" from a unit's Might stat; rather, Might is a value that is modulated by spells and effects. Your unit still has its full printed Might potential, but its current Might is 4 for the duration of that effect.