The trick is to keep the discussion from drifting into whether or not you and Kayla are dating. "Are you setting us up on a date?" you ask.
"Aren't you already dating?" Janice asks.
You need to lean into Janice asking inappropriate questions to people she oversees. "I trust you're not speculating about the personal lives of two people you're in charge of."
"Your department chair is directly responsible—"
"But you oversee all of the academic faculty, right?" With Janice on the back foot, you're not going to stop pressing your advantage. "I appreciate that you're thinking about our long-term careers and what effect dating can have on them. Trust me when I say that Kayla and I are focused on succeeding at Winfield Phillips."
Bill beams. "That's the kind of attitude I enjoy seeing in our faculty. That can-do spirit has built the college into what it is today."
Today it's a lesser-known New England liberal arts school. But that wouldn't be tactful to say.
"Yes, of course," Janice says. "I meant nothing inappropriate. We look forward to both of you having a long career here."
"Thank you." You get the sense that Kayla is thanking both Janice and you.
"Well done," Alex says. "Though you perhaps missed an opportunity to make Janice think more highly of you."
Gee, Thanks