Skip to content

Many bots not being detected as bots #134

@fluffy-critter

Description

@fluffy-critter

The user-agent string Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) fits all standards for bot declaration, but is not being detected as a bot:

>>> ua=user_agents.parse('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)')
>>> ua.is_bot
False

Is there some reason why this library isn't using the presence of r'\+https?://' as a bot/spider indicator?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions