The biggest reason why Zhai Yanjun and Xiang Wai didn't work out was the fact that at the end of the day, they both didn't know how to love. Xiang Wai craved attention and love since her childhood. Even if she didn't remember her early childhood, her brain subconsciously looked for the love and attention that she received living in the Xiang family. And when she couldn't find it, it actually became her obsession.
As for Zhai Yanjun, he had long forgotten what love even felt like. His traumatic childhood made him withdrawn and quiet. And no desire was actually left inside of him. That was why he slowly even drifted away from Shui Xian's life. Zhai Yanjun was someone who felt like his existence didn't really matter at all.