So this stems from two times when Jim talks about this.
In “The Secret” - Season 2:
“Hey oh, listen, um, I told Michael on the booze cruise. It’s so stupid. Um, I told Michael that I had had a crush on you when you first started here.”
In “Launch Party” - Season 4:
Jim: Do you remember what you said to me on my first day of work, just before you walked me over to my desk? Pam: Yeah, enjoy this moment, because you’re never going to go back to this time before you met your desk mate Dwight.
I think it’s generally accepted that both Jim and Pam started working at around the same time, but Pam started before Jim. So hence Pam would still be fairly new there when Jim joined but Pam would still be there long enough to know things like how weird Dwight was.
“We were up at the front, and we were talking about what’s really important, and Pam, I think enough is enough. I think we should set a date for our wedding. How about June 10th? Come on, let’s do it! Come on, Pam!”
“Look, about you and Jim-”
“Oh, no, that’s, you don’t have to.”
“No, I feel it’s my responsibility as your boss-slash-friend.”
“No, really, it’s okay. I know that Jim had, like, a crush on me when he first started. But that was a long time ago, so.”
“It wasn’t that long ago. It was on the booze cruise.”
“Jim had a crush on me on the booze cruise or he told you about it on the booze cruise?”
“Yeah, okay, shut it Michael. I’m done. That’s it. I’m out.” - ep 213