How adding a Robots.txt file will help indexing website

[NextGenMarketing]

Tweet

Add any pages that you do not want search engines indexing after the “disallow:” command in the form of http://www.yoursite.com. Save this file into your website files and upload it to your website. Search engines look for this file. Even if you don’t have to use it at this point, you should put the file in and you can add to it later.

robots.txt file. This file has to be added in the root file of your website.

User-agent: *
Disallow:

[Mogga]

You can also put in the location of yor sitemap

sitemap: http://www.example.tld/sitemap.xml