Languages
Description
The robots.txt file is the first thing search engine "spiders" look for when indexing a website. The robots.txt file tells search engine spiders (robots) which files and/or directories they are NOT allowed to index. This helps to prevent incomplete site indexing as well as prevent exposing the files and directories that you don't want "all over the world wide web". It's fast, easy and FREE.
Related:
User Reviews:
Related:
0ds Statusbarbounce ?0ds Statusbarbounce is multi-platform compatible. This script displays a bounce-text in the status barBy: dseffects.com |
Freeware |
0ds Statusbarclock ?0ds Statusbarclock is multi-platform compatible. This script displays the clock time in the status bar.By: dseffects.com |
Freeware |
0ds Statusbarshoot ?0ds Statusbarshoot is multi-platform compatible. This script displays a shoot-text in the status bar.By: dseffects.com |
Freeware |
99 Website Designs With Javascript MenusAdvanced yet both Easy and Quick to implement. With these pre-built and highly configurable web sites you will save a great deal of time and money...By: Chris O'Brien |
Shareware |
A Zen Advice Script ?A Zen Advice Script is both a Unix and Linux compatible script that uses Math.random() and arrays to give the user random Zen-like advice....By: Zen Advice |
Freeware |
XLSReadWriteII.NET 2.00XLSReadWriteII.NET for Delphi 2005/2006 and others, Win Forms Use this version for creating Win Forms applications with Delphi, Visual Studio and...By: Axolot Data |
Shareware |
TGPS 1.11Interface with your GPS receiver.By: Axolot Data |
Shareware |
pgISQL (Interactive SQL for PostgreSQL) ver. 2.2.0 2.2.0Interactive SQL for PostgreSQL is a powerfull database utility. It allows: Build SQL queries using SQL Editor with SQL syntax highlighting and SQL...By: Sujit |
Shareware |
Syndicate Pro 1.0Syndicate Pro multi-platform compatible. Syndicate Pro is server software for managing and distributing news headlines and other content over the ...By: Sujit |
Shareware |
ESBPCS for VCL 4.2.2ESB Professional Computation Suite (ESBPCS for VCL) provides a huge collection of Borland Delphi Routines and Components to make Data Entry and...By: Glenn Crouch |
Shareware |
Related
- Robots.txt Generator
Information on the use of robots.txt as well as a robots.txt generator and a list of search engine robots. - Webmasterworld: Robots.txt
Brett Tabke experiments with writing a weblog in a text file usually read only by robots. Commentary on the world of... - Robots.txt Validation
Creator and validator of robots.txt files. - Robots.txt Analysis
Tool that fetches robots.txt from a given URL and parses it against the URL. Includes different user-agents. - Robots.txt Checker
Analyzes a robots.txt file searching for syntax and "logical" errors, and shows a summary of what effect it will have. - Robots.txt Syntax Checker
Online tool for syntax verification to robots.txt files, provided by Simon Wilkinson. - Global Robots Ltd
Buys, sells and installs new and used industrial robots. Specialising in used 6-axis models. Includes stock list. Based in... - Active Robots, Ltd.
Sells robot kits, controllers, and accessories suitable for educational resource requirements and amateur robotics... - Mutant Robots
Video, pictures, news and detailed history on Tazbot, Diesector, Root Canal, Easty Beast and their creator. - Snake Robots
Features several robotic snakes. Includes photos and video clips.