It would take a lot for Luffy to let you go especially if he can feel in your heart that you’d rather do anything but that. Luffy understands his nakama better than they think.
Sanji will take on everything by himself. Sanji never wants to be a burden. Sanji forced himself to fight Luffy and kick him hard enough to knock a tooth out that hasn’t come back yet. It tore him apart to have to do that. Sanji deserves all of our love even if he doesn’t think he deserves it.