robots.txt controls crawling, not indexing
Two plain files sit at the root of most sites, both read by crawlers, and they are confused with each other constantly. A robots.txt is a set of instructions about fetching. A sitemap.xml is a list of URLs you would like considered. One restricts, one suggests, and neither does what most people assume.
What each file actually is
robots.txt is fetched before anything else and read as groups. A group opens with a User-agent line naming which crawler it applies to, then lists Disallow and Allow path prefixes:
User-agent: *Allow: /Disallow: /drafts
It is a request, not enforcement. A well-behaved crawler reads it and complies. Nothing in it compels a badly-behaved one, and the file is public at a predictable URL, so every path you name in it is published to anyone who looks.
sitemap.xml is a list of URLs with optional metadata, pointing crawlers at pages they might not otherwise find — something buried deep, or not linked from anywhere prominent. It helps discovery. It does not oblige a search engine to index anything on it, and a page can be indexed perfectly well without ever appearing in one.
The contradiction that catches people
Here is the part that trips up people who have run sites for years. robots.txt controls crawling, not indexing. Disallow means "do not fetch this". It does not mean "do not list this".
If another site links to a disallowed URL, a search engine can learn that the URL exists from the link alone. It never fetched the page, so it has no title and no description to show — but it can still put the bare URL in results, sometimes with a note that it could not be crawled. The page you were trying to hide is now in the index, listed less usefully than if you had left it alone.
To keep a page out of results you need a noindex signal, which lives in the response itself: a <meta name="robots" content="noindex"> tag in the page's HTML, or an X-Robots-Tag: noindex response header, which is the option for files that are not HTML.
And now the trap closes. A crawler has to fetch the page to see either of those. If the same URL is disallowed in robots.txt, the crawler never fetches it, never sees the noindex, and the page can sit in results indefinitely while you keep tightening the file that is causing the problem. The two instructions cancel each other out.
So the order matters. If a page is currently in results and should not be, allow it to be crawled and serve noindex on it. Wait until it has dropped out. Only then, if you want to save crawl effort on a large section, is adding the Disallow back worth considering — and for a handful of pages there is rarely any need.
And neither file is a privacy control
Nothing in this pair keeps anybody out of anything. A crawl directive is advisory, a noindex tag only shapes what search engines display, and both leave the page fully reachable to anyone with the URL. Content that genuinely must not be seen needs authentication, or needs not to be published. That is a different mechanism entirely, and no root file substitutes for it.
Getting the sitemap right
The Sitemap line in robots.txt is independent of the user-agent groups — it applies to all of them wherever you put it, and it takes an absolute URL rather than a path.
Beyond that, the useful discipline is to list only canonical URLs you actually want considered. A sitemap padded with URLs that redirect, that return 404, or that are duplicate versions of a page you would rather have indexed under a different address, is telling a search engine that you consider those canonical. That is a worse signal than a short, accurate list, and a sitemap that quietly rots after a redesign is one of the more common ways a site ends up arguing with itself about which URLs matter.
The tool
The robots.txt + Sitemap generator takes your domain and a comma-separated list of paths to disallow, and writes a robots.txt with a single User-agent: * group, Allow: /, one Disallow line per path you gave it — adding the leading slash if you left it off — and a Sitemap line pointing at sitemap.xml on your domain. Underneath it prints a one-page sitemap.xml starter you extend by copying the url element per page. Both files belong at your site root.
Use the disallow field for what it is: a way to keep crawlers away from sections that are not worth fetching. If your goal is to keep a page out of search results, leave it crawlable and serve noindex instead.