Also for someone that lives on the east coast at sea level, I was thinking about coming a bit early to start to acclimate to the higher altitudes. I've personally never had an issue with it but not sure about the others that will be coming with. How long should I plan to come before hand to try and acclimate to the altitude?
Silverton sits at 9,000 feet ASL. That's a high elevation for anyone living below 3,000 feet. The bonus is that we'll all be driving to that area and your body will be acclimating along the way. It's much more pronounced for folks flying to Colorado and heading straight up to the mountains. You'll have the benefit of slowly climbing in elevation enroute to Silverton. For those traveling through Kansas on the way here, you'll gain 4,000 feet just crossing that state.
Altitude Sickness is a real thing and isn't really something that can be predicted. Regardless of what kind of physical shape you're in, you'll either do well at altitude, or you wont. Altitude Sickness has nothing to do with physical condition and everything to do with your body's ability to acclimate.
Being out of breath after walking across the campground on day one is normal. You may feel a little nauseous and even a little dizzy. You may end up with a pretty sizable headache and may not sleep well your first night here. The important thing to monitor is whether you're getting better or worse.
Let your body do what it needs on day one. Relax, don't do a bunch of hiking or running around, avoid lots of alcohol, drink lots of water, eat a high carb meal, take a nap, etc. Act like you're on vacation. This gives your body the time it needs to adjust. It's difficult to keep kids from overdoing it on day one. They're psyched to be out of the vehicle, there are tons of other kids running around, and kids generally love just being kids. They're just as susceptible to Altitude Sickness as adults, but they may not recognize the symptoms.
With all of that being said, this is Colorado. The ski/snowboard capitol of the United States. Think about how many people fly from sea level to Denver airport, then drive straight to the mountains and ski all weekend. (not a good idea, but something that's done fairly often)
@aegustafson is a paramedic in Aspen (and part of the Hooligans Crew) and can really shed some light on what he sees throughout the year from folks flying in from sea level.
Unless you or your family members are getting markedly worse by the hour, it's not something that should ruin your trip. We will travel over 13,000 feet almost every day. Even I feel lightheaded above 13,000 feet.
Here are some elevations around the area if symptoms seem to be worsening or not improving at all. Usually, a few hours at a lower altitude will fix everything. You don't need to drive all the way home to feel better. A few thousand feet can make a huge difference.
Silverton - 9,305 feet
Ouray - 7,800
Ridgway - 6,900
Durango - 6,500
Montrose - 5,800