According to Scientific American, Inc., this is a myth. Birds do not abandon their young as a consequence of smelling a human scent. But they will abandon their offspring and nest in response to disturbance. So, you can probably safely pick-up a baby bird that has fallen from it’s nest, but you probably shouldn’t be messing around with the nest itself.
(Source: Scientific American, Inc.)