The decision to leave is definitely now, and Rory does take the chance to slip away. He'll come back later after their battle's been worn out. Plus if he didn't he would most likely not hear the end of it from Clint.