Oh, you were running a single battery - that's probably the single biggest problem right there. You simply don't have enough capacity to adequately run the winch on one battery, especially when the winch is relatively slow and making long pulls. If the winch is basically decoration, and you might do a 20' pull once in a while, then a single battery will get you by. If you're really going to use the winch, then you need a strong, well-maintained charging system.
A note on batteries: I make no secret about it that I am not a fan of Optima batteries. In my experience, the Optima battery that gave them a reputation for quality and power is a thing of the past. I do not have all that much respect for those batteries anymore as I see a large number of premature failures, and they absolutely do not survive (for even weeks in some cases) in 24V electrical systems which we work on a lot.
Up here, north of the 49th, in Canuckistan, we actually see more 24V systems and dual battery 12V systems due to the large number of diesels we work on than just about anything else. If there's one thing that doesn't work well in pretty much any dual battery system, it's running 2 batteries that are not of identical construction, age, type, size and manufacture. Yes, you can do it and get away with it for a while, but in the long term you're not going to be getting the best performance out of your system if it's mix and match. If you don't want to believe me, then just do a brief search in the Diesel section here on 'Mud to see how often battery related questions crop up.
If you're going to be running dual batteries, then, I would suggest that you make them the same ones - blue, green, yellow, red, black, or whatever. You don't really need to have a rainbow of colours under hood... we're not after the decorative value of the battery case, we're looking for solid performance.
The other thing about batteries and battery maintenance is that virtually all automotive charging systems do not properly charge or maintain AGM batteries. AGMs will perform far better if they are recharged and maintained with a relatively high-tech (usually external) voltage regulator/charge monitor that feeds them the correct voltage and current during the recharge cycle. I can't think of too many people, other than boaters and overlanders, who will spend that kind of money on a proper charging system.
So, what would I suggest? Get two identical batteries of the largest group size that you can reasonably put under your hood. Try a Group 27 or maybe even a Group 31. I personally run (Made in the USA) East Penn / Deka Group 27s, but there are others out there that are very good as well. A really good group 27 may not list the highest overall CCAs, but really high CCAs usually means thin plates (read: not very robust) that will break down more quickly over time. Deep Cycle batteries usually have thicker plates and hefty plate construction for durability and, as a result, have a lower CCA value. In the middle, somewhere, is a happy medium.
When shopping for a battery look for features like forged battery posts, good grid construction of both grid plates (and not just cheap expanded mesh), a large case (since the amount of electrolyte present does play a role), good weight, and anchored plate envelopes (plates are set into resin to resist vibration).
You do not need to isolate the batteries in your system, but if you feel you really need to, then using an isolator like the Redarc Smart Start is a really good way to go. Diode based isolators, by nature of their design, introduce voltage drop across the system, and voltage drop is evil.
As for the winch motor, the 6HP unit is going to be a good upgrade. It might be a good time to replace the solenoids with something a little more reliable than the Warn offerings if you're going to be using the winch more than just a little (Albright solenoid).
And, yes, there are people who go out looking to winch then they go out fourwheeling
~John
