5 Sitemap: http://www.chiplist.com/sitemap.txt
11 Disallow: /ChipList2/scripts/
13 Disallow: /ChipList2/styles/
16 Disallow: /ChipList2/ads/
17 Disallow: /advertisements/
18 Disallow: /ChipList2/advertisements/
21 Disallow: /ChipList2/graphics/
23 #Disallow: /ChipList1/
26 # robots.txt for http://www.wikipedia.org/ and friends
28 # Please note: There are a lot of pages on this site, and there are
29 # some misbehaved spiders out there that go _way_ too fast. If you're
30 # irresponsible, your access to the site may be blocked.
32 # Inktomi's "Slurp" can read a minimum delay between hits; if your
33 # bot supports such a thing using the 'Crawl-delay' or another
34 # instruction, please let us know.
36 # *at least* 1 second please. preferably more :D
42 # Crawlers that are kind enough to obey, but which we'd rather not have
43 # unless they're feeding search engines.
44 User-agent: UbiCrawler
53 # Some bots are known to be trouble, particularly those designed to copy
54 # entire sites. Please obey robots.txt.
55 User-agent: sitecheck.internetseer.com
61 User-agent: MSIECrawler
64 User-agent: SiteSnagger
67 User-agent: WebStripper
76 User-agent: Offline Explorer
82 User-agent: TeleportPro
94 User-agent: Microsoft.URL.Control
109 User-agent: Download Ninja
113 # Sorry, wget in its recursive mode is a frequent problem.
114 # Please read the man page and use it properly; there is a
115 # --wait option you can use to set the delay between hits,
122 # The 'grub' distributed client has been *very* poorly behaved.
124 User-agent: grub-client
128 # Doesn't follow robots.txt anyway, but...
134 # Hits many times per second, not acceptable
135 # http://www.nameprotect.com/botinfo.html
139 # A capture bot, downloads gazillions of pages with no public benefit
140 # http://www.webreaper.net/
141 User-agent: WebReaper
145 # Provided courtesy of http://browsers.garykeith.com.
146 # Created on February 13, 2008 at 7:39:00 PM GMT.
148 # Place this file in the root public folder of your website.
149 # It will stop the following bots from indexing your website.
152 User-agent: ALeadSoftbot
153 User-agent: BeijingCrawler
157 User-agent: BOTW Spider
158 User-agent: bumblebee
159 User-agent: Bumblebee
160 User-agent: BuzzRankingBot
161 User-agent: Charlotte
164 User-agent: CydralSpider
165 User-agent: DataFountains
166 User-agent: DiamondBot
167 User-agent: Dulance bot
169 User-agent: EARTHCOM.info
173 User-agent: Exabot-Images
174 User-agent: Exabot-Test
175 User-agent: exactseek-pagereaper
176 User-agent: Exalead NG
177 User-agent: FANGCrawl
178 User-agent: Feed::Find
179 User-agent: flatlandbot
181 User-agent: GigabotSiteSearch
182 User-agent: GurujiBot
183 User-agent: Hatena Antenna
184 User-agent: Hatena Bookmark
185 User-agent: Hatena RSS
186 User-agent: HatenaScreenshot
188 User-agent: HiddenMarket
189 User-agent: HyperEstraier
190 User-agent: iaskspider
192 User-agent: InfociousBot
194 User-agent: iVia Page Fetcher
196 User-agent: Kolinka Forum Search
197 User-agent: KRetrieve
198 User-agent: LetsCrawl.com
199 User-agent: Lincoln State Web Browser
200 User-agent: Links4US-Crawler
202 User-agent: Lsearch/sondeur
203 User-agent: MapoftheInternet.com
204 User-agent: NationalDirectory
205 User-agent: NetCarta_WebMapper
206 User-agent: NewsGator
207 User-agent: NextGenSearchBot
212 User-agent: Nudelsalat
214 User-agent: OmniExplorer_Bot
215 User-agent: OpenIntelligenceData
216 User-agent: Oracle Enterprise Search
218 User-agent: panscient.com
219 User-agent: PeerFactor 404 crawler
220 User-agent: PeerFactor Crawler
221 User-agent: PlantyNet
222 User-agent: PlantyNet_WebRobot
226 User-agent: QuickFinder Crawler
227 User-agent: Radiation Retriever
229 User-agent: RedCarpet
230 User-agent: ScorpionBot
233 User-agent: searchbot
234 User-agent: Seeker.lookseek.com
235 User-agent: SeznamBot
238 User-agent: snap.com beta crawler
240 User-agent: SnapPreviewBot
243 User-agent: Speedy Spider
244 User-agent: Speedy_Spider
245 User-agent: SpeedySpider
247 User-agent: SquigglebotBot
248 User-agent: SurveyBot
249 User-agent: SynapticSearch
250 User-agent: T-H-U-N-D-E-R-S-T-O-N-E
251 User-agent: Talkro Web-Shot
252 User-agent: Tarantula
253 User-agent: TerrawizBot
254 User-agent: TheInformant
255 User-agent: TMCrawler
256 User-agent: TridentSpider
257 User-agent: Tutorial Crawler
259 User-agent: unwrapbot
260 User-agent: URI::Fetch
262 User-agent: Vonna.com b o t
264 User-agent: Votay bot
265 User-agent: WebAlta Crawler
267 User-agent: Webclipping.com
269 User-agent: Webinator
272 User-agent: Xerka WebBot