In a past life, you had complete control and power over this person, and shadows of that old struggle are felt in this lifetime. You could feel threads of that past-life dynamic, too, in a heavy sense of familiarity that settles in early on. Were you lovers in a past life?
Perhaps, but most likely your relationship was much more like that of jailer and prisoner. You kept your partner down, using emotional or even physical force to control their every movement. There's no reason why it should be nearly so intense in this lifetime, but you feel the power of the dynamic, at least subtly. When your lover gets frustrated with you, they could feel scared to voice their irritation -- and that could be an old, subconscious memory of your former life together and your old, destructive relationship. Or your sex life could take on a dynamic of domination, which isn't necessarily fun for either of you; that, too, would be a holdover from that past life. If you want things to work out smoothly in this one, you're going to have to try in a conscious way to encourage your lover to be who they are, fully and completely, despite any feelings of fear or restriction they might feel; and to allow them to do, say and be whoever and whatever they want. It's the only way you can break this old cycle of control and domination, short of refusing to be involved again.