User-agent: Specifies which web crawlers the rules apply to (* means all)

Disallow: Tells crawlers not to access specific paths or files

Allow: Explicitly allows access to specific paths (overrides Disallow)

Crawl-delay: Minimum delay between requests (in seconds)

Sitemap: Location of your XML sitemap for search engines

Upload: Place the generated robots.txt file in your website's root directory