There was a time where I had no idea who Daisy Calloway was. But I know her now. She’s my exuberant little sister, and i’ve grown to love her more than blood. I love her as a friend, and I selfishly don’t want to let her go.
These two lines are so important; despite the wording-making it sound as if Fitz is angry at Jemma for not returning his feelings-it makes it clear that Fitz is angry that Jemma left-because of him. He’s angry at himself for ever telling her how he felt; in his mind, it’s why she left. At the start, he wanted to tell her. Of course he did. But he couldn’t, not with their friendship at stake. He nearly begged her to stop demanding to know why he was sacrificing himself for her in the Pod; he didn’t want to tell her anymore. He was, quite frankly, willing to die without her knowing.
And now, Jemma’s gone. Left and didn’t come back-Fitz can’t even contact her. He was lied to, not only by Jemma but by both Coulson and May, and now he knows that she isn’t with her parents, safe and sound. She’s far from safe, she’s undercover at Hydra. She could die at any minute. And he believes it’s his fault. Why else would she lie to him? So, he automatically goes to the only explanation he can think of: because he told her how he felt. Yes, he’s upset that Jemma left and didn’t tell him why, but he’s more upset with himself for ruining their friendship by telling her in the first place. Hence, his quote in the second gif-which was also his way to try and fit in with the group.
He’s bitter at first by this outcome, then tries to accept his line of reasoning, but even as 2x04 goes off, you can tell there’s still something nagging at him, more doubts swirling around his head.