A 'bot' in this sense is a computer program that follows links, and may create a graph of pages found that way, or index them for searching.