I recommend you use the next-sitemap generator by "iamvishnusankar".
GitHub: "https://github.com/iamvishnusankar/next-sitemap"
Install the package "npm i next-sitemap" or "yarn add next-sitemap".
In your ENV file create a SITE_URL property and set the value as your public domain. Create this property in your ".env.local" file if you're testing local. Also add these values to you Environment Variables in you deployment settings.
SITE_URL=https://my-url.com/
Then create a "next-sitemap.config.js" file in your root folder and paste this code in it. If you have a large website set the "sitemapSize: 7000", if not you can leave it out. Also set the change frequency to 'daily'.
/** @type {import('next-sitemap').IConfig} */
module.exports = {
siteUrl: process.env.SITE_URL || 'https://my-url.com/',
generateRobotsTxt: true,
changefreq: 'daily',
priority: 0.7,
sitemapSize: 7000,
transform: async (config, path) => {
return {
loc: path,
changefreq: config.changefreq,
priority: config.priority,
lastmod: config.autoLastmod ? new Date().toISOString() : undefined,
alternateRefs: config.alternateRefs ?? [],
}
},
additionalPaths: async (config) => [
await config.transform(config, '/additional-page'),
],
robotsTxtOptions: {
policies: [
{
userAgent: '*',
allow: '/',
},
{
userAgent: 'test-bot',
allow: ['/path', '/path-2'],
},
{
userAgent: 'black-listed-bot',
disallow: ['/sub-path-1', '/path-2'],
},
],
additionalSitemaps: [
'https://doc.my-url.com/', //Add other releases or sub domains here
],
},
}
Then create a "robot.txt' file in your public folder and set the values as you see fit. Here is an example.
# *
User-agent: *
Allow: /
# test-bot
User-agent: test-bot
Allow: /path
Allow: /path-2
# black-listed-bot
User-agent: black-listed-bot
Disallow: /sub-path-1
Disallow: /path-2
# Host
Host: https://my-url.com
# Sitemaps
Sitemap: https://my-url.com/sitemap.xml
Sitemap: https://docs.my-url.com/sitemap.xml
Finally, run "npm run build' or "yarn run build". This will generate and sitemap-0.xml and/or a sitemal.xml in your public folder.