removing the power steering will not cause that
first thing i would suggest is to have your alignment checked
it is most likely out of alignment and you probably have some bad parts in there somewhere (bushings, ball joints, tie rods, etc) causing the looseness in the front end
even if your car seems to track rather straight the toe can still be off causing it to get "darty" when turning especially at speed
or, though much less common, it is possible that your issue is in the steering rack or column
either way an alignment shop should be able to let you know whats going on there
or you could get under there yourself and take a look at everything and even align the car yourself using the string box method (it is intimidating but not overly difficult if you take your time and make sure you are precise with your measurements)