Did he pull you aside in the corridor before talking to the Matchmaker? If he did, I think that unlocks his path. Stuff that helps includes inviting him to your first event (he always says no, but it doesn't use up a slot and probably gets you some romance with him), visiting your room every week (I usually do it 2/3 times and still get his confession), and being nice to your servants. You might also have to do well in the Week 4 trial and figure out the Historians' secret chain to do his romance, but I haven't tried it yet, so I'm not sure.
Viewing post in Game Discussion (Full Spoilers)
I beelined for Jasper first as soon as the demo came out and also aimed for him first as soon as the game got EA, and I can confirm that you don't need to finish the historian chain to successfully romance Jasper, but the ending is definitely a lot happier if you do (yup, I totally went back for him multiple times). To unlock his epilogue though, you need to finish the historian chain AND complete some legendary requirements, which I have yet to do. Like the previous post commented, visit your room every week, be nice to servants (stand up for Sarya, offer them your breakfast, etc). For Imogen's trial make sure you ask him why the head butler won't let you question Imogen, BUT make sure you don't push him for more info a little bit afterwards when you notice that he is tired and invite him to sit down and talk with you.