Most commonly means to assign anything a status or posture to distinguish it from Some others in a bunch, as in txt file is then parsed and will instruct the robot as to which webpages will not be being crawled. As a search engine crawler might hold a cached copy https://www.youtube.com/watch?v=ARvdj_mJDAY