In my mind, Jasper eventually gets fed up with the Cullen's treatment of him and leaves, and the occasional visit because Alice. I think once Jasper finally does this, Rosalie decides she's done being the scapegoat and leaves, too. I can see Rosalie and even Alice following the "diet" still, but Jasper probably doesn't angst as much when he slips and Emmett is just happy to be out of Calculus in my head. What do you think?
This is undoubtedly possible.
The most crucial thing when writing found families, I think, is giving all the characters a clear reason for loving each other and wanting to be together. In more dramatic works, it can be something as basic as “we want to save the world” or “everyone else is dead”. Because Twilight has lower stakes, all the Cullens need to have distinct, convincing motivations. Jasper, Alice, Rosalie and even Emmett, as you point out, don’t necessarily have those.
To begin with Alice, it seems like she had a series of visions and decided that she was destined to join the Cullens. Still, she acknowledges that her gift is fickle. Her visions certainly didn’t imply an eternity with the family, and she may very well decide to leave, taking Jasper with her. And, speaking of Jasper, it’s hard to pinpoint anything in the text which suggests that he’s happier with the Cullens than he would be with just Alice. Either of them could easily grow impatient/bored/stifled and leave. (From a purely pragmatic standpoint, it seems like Alice and Jasper bring a lot of benefits with them. She’s responsible for the Cullens’ wealth, whereas he keeps the peace and is a competent fighter. In other words, there’s nothing keeping them in the family.)
Although Rosalie is tremendously loyal to the Cullens, I’m not really sure why. Is it just the concept of family? Does she like them? Do they provide her with emotional support/intimacy/connection? None of that makes it onto the page, and the amount of sheer malice they ascribe to her is astounding. Anyway, I could see her getting tired of that– as you say– and leaving, Emmett going with her. He already appreciates “breaks” from the family; I can’t imagine him minding seeing them periodically instead of all the time.