Ever wondered why Google returned search results that lead to sites that require a registration? How did Google index the site without a registration? Many sites want their site indexed in Google to receive more hits, so they allow Googlebots in. Because of this reason we can take advantage of this. All we have to do is disguise ourself as a Googlebot and many sites will let us in without registration.
To do this we have to change the User Agent. The user agent defines the browser and what version you are using. For example if you go to this site it will tell you what browser you are using, and it finds this info out by the user agent.
(Source: PinGUY’s Website)