It is tough to walk away from something you both want and can afford, but in this case I think you made the right call.