In case it helps anyone else, I found the easiest way to do this is to turn the diffuser sideways, ie. perpendicular to its usual orientation, and it'll just slide right out. In doing this, I also realized the two center ones weren't broken like I thought, they were just turned all the way...