Add sitemap and robots.txt for SEO by ooloth · Pull Request #16 · ooloth/michaeluloth.com

ooloth · 2025-12-19T03:57:41Z

✅ What

Adds /sitemap.xml with all static pages and dynamic blog posts from Notion
Adds /robots.txt that allows all crawlers and references the sitemap
Adds sitemap link to HTML <head> for search engine discovery
Uses trailing slashes in URLs to match existing RSS feed format
All files statically generated at build time

🤔 Why

Search engines can now efficiently discover and index all site content
Better SEO means more organic traffic to blog posts
Sitemap auto-updates when new blog posts are published in Notion

Copilot

Pull request overview

This PR implements SEO improvements by adding XML sitemap generation and a robots.txt file. The sitemap includes all static pages (home, blog index, likes) and dynamically fetches blog posts from Notion, while robots.txt allows all crawlers and references the sitemap location.

Key changes:

Added sitemap generation with static pages and dynamic blog posts from Notion
Added robots.txt configuration allowing all crawlers with sitemap reference
Added sitemap link to HTML head via layout metadata

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
app/sitemap.ts	Generates sitemap with static pages and blog posts fetched from Notion
app/sitemap.test.ts	Comprehensive test coverage for sitemap generation logic
app/robots.ts	Configures robots.txt to allow all crawlers and reference sitemap
app/robots.test.ts	Test coverage for robots.txt configuration
app/layout.tsx	Adds sitemap link to HTML head metadata
app/layout.test.tsx	Test coverage for layout metadata including sitemap link

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-19T03:58:16Z

app/sitemap.ts

+
+  const blogPosts: MetadataRoute.Sitemap = posts.map((post) => ({
+    url: `${SITE_URL}${post.slug}/`,
+    lastModified: new Date(post.firstPublished),


The field name 'firstPublished' suggests the initial publication date, but 'lastModified' typically represents the most recent update. Consider using a 'lastUpdated' or 'lastModified' field from the post data if available, or rename this to 'published' to better reflect that it represents the publication date rather than the last modification date.

Copilot · 2025-12-19T03:58:16Z

app/sitemap.ts

+
+  const staticPages: MetadataRoute.Sitemap = [
+    { url: SITE_URL, priority: 1 },
+    { url: `${SITE_URL}blog`, changeFrequency: 'weekly', priority: 0.8 },


The 'blog' URL is missing a trailing slash while 'likes' and blog post URLs include them. This creates inconsistency in URL formatting. For consistency with the trailing slash pattern used elsewhere (line 18), add a trailing slash to the blog URL.

Suggested change

{ url: `${SITE_URL}blog`, changeFrequency: 'weekly', priority: 0.8 },

{ url: `${SITE_URL}blog/`, changeFrequency: 'weekly', priority: 0.8 },

claude · 2025-12-19T03:59:52Z

Pull Request Review: Add sitemap and robots.txt for SEO

Thank you for this well-structured PR! The implementation is clean and follows good practices. Here's my detailed review:

✅ Strengths

Excellent Test Coverage: All new files have comprehensive tests with good edge case coverage
- sitemap.test.ts covers static pages, dynamic posts, empty states, and integration scenarios
- robots.test.ts validates configuration completely
- layout.test.tsx ensures metadata updates are verified
Consistent Patterns: Implementation follows existing codebase conventions
- Uses force-static export (matches RSS feed pattern in app/rss.xml/route.ts:8)
- Leverages existing getPosts function with proper error handling
- Follows Result pattern with .unwrap() (consistent with codebase error handling)
SEO Best Practices: Proper implementation of standard SEO features
- Sitemap includes priority and changeFrequency attributes
- robots.txt references sitemap location
- Sitemap link added to HTML <head> for search engine discovery
URL Consistency: Trailing slashes match existing RSS feed format (app/rss.xml/route.ts:40)

⚠️ Issues & Recommendations

1. Error Handling Missing (High Priority)

Issue: app/sitemap.ts:9 calls .unwrap() which will throw if getPosts fails, causing build failures.

Location: app/sitemap.ts:9

const posts = (await getPosts({ sortDirection: 'descending' })).unwrap()

Problem: Unlike RSS feed which has error handling in a loop (app/rss.xml/route.ts:30-33), sitemap has no fallback for API failures.

Recommendation: Add error handling to gracefully degrade:

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const postsResult = await getPosts({ sortDirection: 'descending' })
  
  const staticPages: MetadataRoute.Sitemap = [
    { url: SITE_URL, priority: 1 },
    { url: `${SITE_URL}blog`, changeFrequency: 'weekly', priority: 0.8 },
    { url: `${SITE_URL}likes`, priority: 0.5 },
  ]

  // Return static pages if posts fetch fails
  if (!postsResult.ok) {
    console.error('[Sitemap] Failed to fetch posts:', postsResult.error)
    return staticPages
  }

  const blogPosts: MetadataRoute.Sitemap = postsResult.value.map((post) => ({
    url: `${SITE_URL}${post.slug}/`,
    lastModified: new Date(post.firstPublished),
    priority: 0.7,
  }))

  return [...staticPages, ...blogPosts]
}

Why: This ensures the sitemap still generates with static pages even if Notion API fails, preventing complete build failures.

2. Test Gap: Error Handling Not Tested (Medium Priority)

Issue: Tests don't cover the getPosts failure scenario, which is critical for build reliability.

Location: app/sitemap.test.ts

Recommendation: Add test case:

import { Err } from '@/utils/errors/result'

it('returns static pages when getPosts fails', async () => {
  vi.mocked(getPosts).mockResolvedValue(Err(new Error('API failure')))

  const urls = await sitemap()

  // Should still return the 3 static pages
  expect(urls).toHaveLength(3)
  expect(urls.every((u) => u.url.startsWith('https://michaeluloth.com/'))).toBe(true)
})

3. Sitemap Link in `<head>` - Non-Standard (Low Priority)

Issue: app/layout.tsx:18-26 adds sitemap link via metadata.icons.other, which is semantically incorrect.

Context: While this works, sitemaps aren't icons. The <link rel="sitemap"> tag is informational only (search engines discover sitemaps via robots.txt or manual submission).

Options:

Keep as-is: Works fine, no functional impact
Remove: Sitemap in robots.txt is sufficient (app/robots.ts:13)

Use Next.js metadata.other (if you want to keep it):

export const metadata: Metadata = {
  // ... other fields
  other: {
    'sitemap': '/sitemap.xml',
  },
}

Recommendation: Consider removing since it's redundant with robots.txt, but no harm in keeping it.

4. Minor: SITE_URL Duplication (Low Priority)

Observation: SITE_URL constant duplicated across files:

app/sitemap.ts:3
app/robots.ts:3
app/rss.xml/route.ts:6

Recommendation: Consider extracting to shared constant (e.g., lib/constants.ts) to ensure consistency and easier updates. Not critical for this PR.

🔒 Security

✅ No security concerns. All data from getPosts is already validated via Zod schemas at the I/O boundary (io/notion/getPosts.ts:24-60).

⚡ Performance

✅ Excellent:

Static generation at build time (force-static)
No runtime overhead
Leverages existing cache system via getPosts

📝 Code Quality

✅ Generally excellent:

Clean, readable code
Follows TypeScript best practices
Proper use of Next.js 16 metadata routes
Consistent with existing patterns

🎯 Summary

Must Fix Before Merge:

Add error handling to sitemap.ts to prevent build failures

Nice to Have:
2. Add test for getPosts failure scenario
3. Consider removing or relocating sitemap <head> link

Overall Assessment: Strong implementation with one critical fix needed. The test coverage and code quality are exemplary. Once error handling is added, this will be production-ready.

Great work! 🚀

Ensures all page URLs have consistent trailing slash format: - https://michaeluloth.com/ - https://michaeluloth.com/blog/ - https://michaeluloth.com/likes/ - https://michaeluloth.com/slug/

Previously, if fetching blocks failed for any post, the RSS feed would silently skip that post and continue building. This meant we could deploy an incomplete RSS feed without knowing. Now using .unwrap() to crash the build if any post's blocks can't be fetched, ensuring we never deploy a broken RSS feed. Updated test to verify build crashes on block fetch failures.

The <link rel="sitemap"> tag in HTML is: - Not part of the HTML spec - Not used by major search engines - Redundant with robots.txt (which is the standard way to reference sitemaps) Search engines discover sitemaps via robots.txt, manual submission to search consoles, or by crawling /sitemap.xml directly.

Centralizes SITE_URL in utils/constants.ts to ensure consistent trailing slash usage across sitemap, robots.txt, and RSS feed. The trailing slash is critical for URL consistency - all page URLs use trailing slashes (/, /blog/, /likes/, /slug/). Having a single source of truth prevents accidental inconsistencies.

ooloth added 5 commits December 18, 2025 22:38

add sitemap

36f16b5

sitemap: add tests

57f6ce3

sitemap: add trailing slashes

9c32835

robots: add robots.txt

ceb8c02

layout: add link to sitemap to head

d344f20

Copilot AI review requested due to automatic review settings December 19, 2025 03:57

Copilot AI reviewed Dec 19, 2025

View reviewed changes

ooloth added 5 commits December 18, 2025 23:01

sitemap: add trailing slashes to blog and likes URLs

6e33f04

Ensures all page URLs have consistent trailing slash format: - https://michaeluloth.com/ - https://michaeluloth.com/blog/ - https://michaeluloth.com/likes/ - https://michaeluloth.com/slug/

fix: formatting

0e3c34c

ooloth merged commit 2e55f2b into main Dec 19, 2025
4 checks passed

ooloth deleted the add-sitemap branch December 19, 2025 04:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add sitemap and robots.txt for SEO#16

Add sitemap and robots.txt for SEO#16
ooloth merged 10 commits intomainfrom
add-sitemap

ooloth commented Dec 19, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Dec 19, 2025

Uh oh!

Copilot AI Dec 19, 2025

Uh oh!

claude bot commented Dec 19, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	{ url: `${SITE_URL}blog`, changeFrequency: 'weekly', priority: 0.8 },
	{ url: `${SITE_URL}blog/`, changeFrequency: 'weekly', priority: 0.8 },

Conversation

ooloth commented Dec 19, 2025

✅ What

🤔 Why

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

claude bot commented Dec 19, 2025

Pull Request Review: Add sitemap and robots.txt for SEO

✅ Strengths

⚠️ Issues & Recommendations

1. Error Handling Missing (High Priority)

2. Test Gap: Error Handling Not Tested (Medium Priority)

3. Sitemap Link in <head> - Non-Standard (Low Priority)

4. Minor: SITE_URL Duplication (Low Priority)

🔒 Security

⚡ Performance

📝 Code Quality

🎯 Summary

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

3. Sitemap Link in `<head>` - Non-Standard (Low Priority)