I have an idea about why the attackers retreated.
Lord Polaris (the one whom I think is the knife dude at the townhouse, since he’s a crazy knife dude at Sphere Music Hall) might not have been supposed to attack at all. He’s a butler and only supposed to follow orders. However, he’s hung up on his “pride as a butler” and the idea of not dying “without action” during this downfall of the “Blue Star”. We have to now assume that the Blue Star is our earl, Blavat’s targeted prize. This all seems to be for the purpose of destroying the earl’s support and trapping him/reeling him in.
If this planned attack at the townhouse had been perfect, only Lord Sirius (the earl’s twin) would have been seen there, Soma would be dead, and Agni would be alive to tell everyone that the earl committed the murder.
However, Lord Polaris’ pride and desire for action prompted him to rush in and attack, too. Yes, Agni was trying to fight the gunman, but only two shots were fired, and that gun holds more than that (at least five. How many bullets does that Bulldog revolver carry? 5? 6?). Anyway, Lord Polaris was probably not needed, but he attacked Agni anyway. Now Soma is alive instead. And he’s seen not ONLY the earl look-alike but also this cloaked dude who is nothing like Sebastian (except, perhaps, for inhuman speed and agility) going wild with serrated knives. Instead of killing Soma, too, they fled the scene because someone needs to be alive to accuse the earl of the attack. Now they are cutting their losses and regrouping to plan their next move, now that things have changed.
The attackers don’t know about Agni seeing old family photos. They probably don’t have a clue that Agni MIGHT have shoved photographic evidence into Soma’s hand, either…. If he did, then the plan is double-botched, since Soma will see the photo and put all the evidence together to realize the earl isn’t the gunman….