Security: Add default robots.txt #680
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
anonymoususer72041
force-pushed
the
security/add-default-robots-txt
branch
from
fc4ef71 to
ae4e93c
Compare
anonymoususer72041
added a commit
to anonymoususer72041/OpenCATS
that referenced
this pull request
Dec 11, 2025
commit ae4e93c Author: anonymoususer72041 <247563575+anonymoususer72041@users.noreply.github.com> Date: Sun Dec 7 15:36:04 2025 +0100 Add default robots.txt
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This change adds a default robots.txt that blocks crawling of the entire OpenCATS installation by default while explicitly allowing the public careers portal and the static assets it needs.
The file whitelists the /careers/ path as the preferred public entry point for job listings and job detail/apply pages, and allows common asset paths such as /js/, /images/, /main.css, /careersPage.css, /ie.css and /not-ie.css so that search engines can properly render the public careers pages.
For installations that expose public careers pages via query-based URLs, the robots.txt includes commented Allow rules for /index.php?m=careers variants together with an explanatory comment.
Motivation
OpenCATS is primarily an internal ATS, so most of the application (including candidate data, client data, attachments, and admin interfaces) should not be discoverable or indexed by search engines. At the same time, many deployments rely on the public careers portal to advertise open positions and want those job pages to remain indexable.
Adding a conservative robots.txt with a "block everything by default, explicitly allow the public careers portal and its assets" approach helps prevent accidental indexing of internal areas while still supporting SEO for job postings.