I might have chosen active skills before since it was the most priority to get before the passive due to the way the Versatile earns points for skills. Just by earning them with a normal attack with the corresponding weapon of the class, it seems I can earn 1 sp randomly per kill. I usually kill enemies with skills so I didn't notice it but seeing the low count of the other class sp even though I used some of them to kill enemies, it seems like they only get a lot of SP by using a skill normally.
Now that I have plenty of skills to use and plenty of SP to spend, that is not a problem anymore. Problem is, Passives are expensive and there are lots of passive every class. I know some of them but the others are passives I am not so familiar with unless I try to look at the description.