This file is added
to the root
directory of your website. It can be created in 'Notepad'. The file
name is robots.txt.
Here is an example of a robots.txt file:
User-agent:
*
Disallow: /landingpages/
Disallow: /campaigns/ppc.html
|
The 'User-agent' is the search engine. To specify
every search engine, * is used.
'Disallow' tells the search robots to ignore certain folders (in this
example: 'landingpages') and files (ppc.html in the 'campaigns'
folder). You can have as many 'Disallow' lines as you require.
Note that a sensible approach is to always put all the folders and
files that you do not wish to be indexed by search engines into one
folder. That way you do not need to constantly update the robots.txt
file.
The robots text file is therefore useful to 'hide' pages on your
website that you would not wish to feature on search engine results.
Examples of files in this category include confidential reports or
pages involved in payments processing. Particularly sensitive
information can be presented in a pdf document and password protected.
Robots text files are a better option to robots commands within web
pages as some search engines will still list pages with 'noindex'
commands.
To find out more about the use of robotos.txt files and all the
necessary website design features in
the context of affiliate marketing,
you can consult our ebook, Successful Affiliates Guide, the
complete guide to affiliate marketing.
|