> I am a Physics (specifically Engineering Physics) major who works with programming, and yes, you will definitely be able to compete.
Hell yes. It will mean that clueless recruitment agencies won't even bother contacting you or doing anything for you, because they don't see you as suitable for anything related to IT. Whether that's a good or bad thing is questionable. It does however mean that getting the foot in the door, i.e. getting into a position that requires a Bachelor in IT is a lot more difficult, in them not wanting to take any risks. The only thing you need to do is prove to them that there is no risk.
Between casual IT work (Sendmail, Exchange, Adobe, Linux/Mac OS, Windows Server and end-user support skills) next to my degree and my Physics degree, I was able to pick up enough skills to get a job. It helped that my Physics degree forced me to learn all of Assembly, C, LabVIEW, LaTeX, Mathematica, MATLAB, Python (Project Euler, actually, is the thing that started forcing me to really know Python) and to a lesser extent Bash and Scheme.
Being able to put ten languages on top of Perl, PHP, JavaScript and others I knew already or know outside of my degree, probably really helped in getting a position programming in a niche language. Given the ten programming languages you're likely to learn in a Physics degree (with a bit of interest) and considering that's likely to be more than you'll get from a Bachelor in IT or maybe even computer science, you're in luck :)
So, in a nutshell, and riffing of another comment here, learn version control and unit testing and know at least one programming language like Python or JavaScript inside out and you're pretty good. Brush up on your design skills and you're done - for now :)