I think it's pretty fair to think that not only is using a drill press with an xy vise as a mill not going to go too well, but using an x-y vise just to locate a hole is also questionable IMO. (I have one and never used it with my drill press, in fact.) Sure, you can put the part just right under the bit, perhaps by using some sort of fine pointer -which means extra work- or by having very good eyes, but you can usually do just as well and as fast with a good centerpunch divot and letting the part locate itself with the drill bit, I think. Of course, then you have to be careful to hold it securely (I learned my lesson the skin-torn hard way) but that's not so difficult. The x-y vise also takes a bunch of vertical space and adds a lot of weight on the table.
I would look carefully at how the dovetails are worn if I were to buy a used mill. If it's all at the usual center positions, it's hard to work around with the gibs. And also look at the screws and nuts, you can get backlash and locating issues if worn unevenly. And, if you can't resist buying one -who could?- on a budget keep in mind that you will also need to spend a ton of money on tooling to get much done. That's easy to overlook in the initial excitement.
Yes, some of the earlier HF machines made in Taiwan were OK and much better than the following early chinese ones. The more recent generic chinese machines may have gotten a bit better as the QC improves, though, although still a long way from serious US brands, I think.
Have fun!