rennea
Gold Member
- 3,662
Follow along with the video below to see how to install our site as a web app on your home screen.
Note: This feature may not be available in some browsers.
The thread revolves around participants' confusion and curiosity regarding the Yahoo Slurp Spider, a web-indexing robot used by Yahoo. Various participants express their thoughts and experiences related to the topic, with some seeking clarification on its function.
Views differ significantly, with some participants expressing confusion and others providing technical explanations. No clear consensus emerges regarding the understanding of the Yahoo Slurp Spider.
The discussion reflects a mix of technical curiosity and personal anecdotes, highlighting varying levels of familiarity with web indexing and related technologies among participants.
Participants interested in web technologies or those seeking to understand the implications of web crawlers may find this discussion relevant.
janetupnorth said:Yahoo! Slurp is Yahoo!'s web-indexing robot. The Yahoo! Slurp crawler collects documents from the Web to build a searchable index for search services using the Yahoo! search engine. These documents are discovered and crawled because other web pages contain links directing to these documents.
As part of the crawling effort, the Yahoo! Slurp crawler will take robots.txt standards into account to ensure we do not crawl and index content from those pages whose content you do not want included in Yahoo! Search Technology. If a page is disallowed to be crawled by robots.txt standards, Yahoo! will not read or use the contents of that page. The URL of a protected page may be included in Yahoo! Search Technology as a "thin" document with no text content. Links and reference text from other public web pages provide identifiable information about a URL and may be indexed as part of web search coverage.
(From Yahoo)
The_Kitchen_Guy said:I think I liked my answer better.![]()
One more piece of software snooping around your stuff.dannyzmom said:Ok - and can you translate that to English???:
![]()
The Yahoo Slurp Spider is a web crawler used by Yahoo to index web pages for its search engine. It helps gather data from websites to improve search results and provide users with relevant information.
The Yahoo Slurp Spider works by following links on web pages to discover new content. It reads the HTML of each page, collects information, and stores it in Yahoo's database for indexing purposes.
The Yahoo Slurp Spider is important for website owners because it helps ensure their content is indexed and appears in Yahoo search results. Proper indexing can lead to increased visibility and traffic to their websites.
To optimize your website for the Yahoo Slurp Spider, ensure that your site is easily navigable, use relevant keywords in your content, create a sitemap, and avoid using excessive Flash or JavaScript that may hinder crawling.
If you notice the Yahoo Slurp Spider crawling your site, it's generally a good sign that your website is being indexed. However, if you have concerns about the frequency of crawling, you can adjust your robots.txt file to manage its access.