| |
|
|
| |
RoboGen Editor Description |
|
| |
RoboGen is a visual editor for Robot Exclusion Files. It allows the user to quickly and easily create the ROBOTS.TXT files required to instruct web search engines which parts of a web site are not to be indexed and made searchable by the general web public. RoboGen does this by providing the user a way to log onto his FTP server and then select the documents and directories which are not to be made searchable. RoboGen maintains a database of over 180 search engine user-agents, which are selectable from a drop down menu. |
|
| |
|
|
|
|