I'm a holy knight but every girl I like ends up joining this other guy’s harem. Someone please tell me what I’m doing wrong?!

"Yamadai Ichiro and Oji Ryuguin both get summoned to the Kingdom of Arcana, a magical realm where they have to face 12 dungeons in order to save the world! A classic otherworld adventure! Ryuguin is revealed to be a hero class, but spends his days slacking off. Ichiro is dissapointed when he is revealed to be a holy knight but he vows to do his best and find the item that will allow him to get back to his own world as soon as possible! The road ahead is long and full of dangers but luckily for Ichiro there are plenty of lovely heroines to support him. Except that... they all keep falling for the slacker Ryuguin? Why does this keep happening?"