I thought they should have taken him out after the HBP or even the batter before that. Like you mentioned, the HR changed the dynamic of that inning.
It seems like a reoccurring thing with him - pretty damn solid for 3 or 4 innings, falls apart in the 5th or 6th inning. Which is understandable because he's a swing starter like you mentioned. But CC needs to have a better plan, IMO