Opponent played Charm on my unit and wants to move it, but I play Not So Fast. They defied, I play Not So Fast again, can I do that or no?

Yes, you can. Each Not So Fast is its own Chain item, and your second copy resolves before the Defy does.

The Chain stacks like this: Charm (bottom) → your Not So Fast #1 → their Defy → your Not So Fast #2 (top). Newest item resolves first (Rule 340.1), so your second Not So Fast counters the Charm before the Defy even matters. Then Defy counters your first Not So Fast, which is now irrelevant — Charm is already countered and the move never happens.

The only way around it is if they Defy your second Not So Fast too, so keep priority flow in mind when you play it.