The bond between you is difficult to maintain because this aspect lends your relationship a fractious, irritating energy that brings both of you down. The sexual attraction was probably really strong from the start, which was more likely the factor that convinced you that pursuing a long-term relationship would be worthwhile, when really, you might have been better off apart. Sex and sensuality can be strong motivating factors; when it's good, it's amazing, and you don't want to give it up!
But it's not enough to base an entire relationship on. In fact, it might actually feed your sexual connection with other sources of friction in your relationship. Conflict is second nature between you; your lover won't watch their mouth or their behavior when you're together, and they end up saying things that they'd never say to someone else -- hurtful or tactless things that end up getting under your skin in a big way. Irritation grows, resentments store up, as do arguments...but there is that strong attraction that you rely on as one way that you both know you can relate to each other in a positive manner. Instead of perpetuating this pattern, why not try to change it? If you do have a long term connection, you can work together -- with conscious effort, of course -- to find the source of your irritating influence on each other, and find ways to relate more gently and kindly.