Most search engines are a hybrid of at least three expert systems: a gatherer, an
indexer and an extractor.(1)
Electronic Gatherers or Crawlers
The gatherer visits websites, scans for content, links, meta-tags and images and
electronically fingerprints information objects excluding duplicates and objects
from ineligible sites (by policy). With millions of sites to visit, gatherers work
in parallel to speed-up delivery. They are also called spiders, crawlers or knowledge
robots. It takes several days for a search engine to identify new links and web pages.
Electronic Indexers
The indexer takes over to classify content and map it with context to create a
database called a catalog or a knowledge repository. It cross-references content
and other objects by keyword, titles, format, source, dates, and other attributes
specific to the search engine. Some connect related objects using background
knowledge like synonyms, clusters like SIC classification or drill-down hierarchical
lexicons. In addition to these cross-references, smart engines also map content to
subject matter, company, geographic location, people or other contexts and fuzzy sets.
This clustering can create rich content/context relationships. Indexers organize and
store the product in the search-engine databases. They also work in parallel and use
the frequency of past searches collected by extractors to create new pointers, and
build redundancy in high-density search areas to alleviate search-engine traffic jams.
Note that creating an index is a complex task even for humans whose experience and
educational background influence the keywords, phrases and topics to index. The
electronic indexers play it safe on keywords, but have a long way to go on subject
matter cataloguing.
Extractors and Knowledge Brokers
Sometimes called knowledge brokers, extractors interact with users, validate
requests to signal potential errors (typos, dates, Boolean operators), suggest
correct wording, search and match the query with the right indexed content.
They rank the results and report a summary of the findings along with URLs and
links for source tracing and detailed browsing. Smart extractors keep track of
various search patterns, compositions, frequencies, hit rates and other statistics
to optimize the total performance of the search-engine.
