Using the 4ohm speakers might require you to turn the volume knob a little further to the right, but other than that it won't hurt.
Generally if an amp is rated to produce a certain amount of power at 2ohms the power would be halved at 4ohms and halved against at 8ohms. Just as it would double at 1ohm and then double again at 1/2ohm if the amp could handle it before either burning up or going into a protection mode.
So if an amp is rated at 100 watts at 8ohms it would be 200@4, 400@2, 800@1, 1600@1/2, and 3200@1/4. I use to run autotek MM3000s in competition back in the day and they were rated 3000@2ohms and we would run them down to 1/2ohm to try and get 12000 watts for just a few seconds. At one point we were using one RE MT 18 with dual 1ohm voice coils paralleled to 1/2ohm in a huge vented box tuned to 40HZ. A 3 second burp inside the vehicle would make your skin crawl and your hair stand up. It was tough to keep 12+ Volts going to the amps though.