Crawlers and bots are not regular users so they shouldn’t be treated normally. After several users feedback and SEO experts consultation we decided to do the following with bots:
- Bots are excluded from redirections: While a while ago Google promised that bots were going to have worldwide IP most of the time they are US based. That also happens with the vast majority of crawlers around. We found that redirecting bots like they were regular users has a bad impact on SEO and indexed pages so they are exlcluded from redirections. Also if you create custom redirections you should exclude them. Check this gist as reference: https://gist.github.com/timersys/31309f9ff41d64ea412e
- New settings to choose country of bots: We added a new setting to set country for bots. If no country is specified in the settings bot will be treated as a regular user and it’s real IP will be used instead.
As always if you have any question please open a support ticket.