The attraction between you could be very one-way, which is difficult for both of you. You are probably less invested in the relationship than your lover is, and that could hurt for them at times. You could start off strong, on equal footing, but soon things start to get skewed between you.
They're much more into the fantasy of a real, long-term relationship, complete with love and attraction, affection and intimacy, trust and communication. They really, really want to create this with you, but you just don't cooperate! After a while, they may start to feel as if you're much more self-centered than they realized, more interested in yourself than you are in them or in the relationship. If they're normally a bit insecure in a love relationship, needing lots of reassurance and affection to feel secure and comfortable, this relationship could push them even further into pessimism and insecurity. They could become jealous and possessive of your time, not caring that whatever you felt for them in the beginning of the relationship faded fast. Their dependency and insecurity turns you off, and that attraction that was strong in the beginning withers. It doesn't have to stay this way between you, though. If the relationship ends up really meaning something to both of you, it's salvageable through careful, conscious communication. Make sure to reassure your lover of your feelings and your intentions.