# Search Engine Indexing ## How Search Engines Work - Search engines operate in three distinct stages: - **Crawling** - think of this as a librarian collecting every book available in the world - **Indexing** - imagine the librarian reading each book and creating a detailed catalogue system - **Ranking** - like a librarian recommending the best books based on your specific question ## Crawling - Web pages are discovered through **software programs called crawlers** (also known as spiders, bots, or robots) - Crawlers navigate the web by **following links from one webpage to another**, systematically visiting pages - They begin from a **set of seed URLs** and explore other pages linked from those starting points - Website crawlers **follow rules and guidelines established by website owners**, using mechanisms like the `robots.txt` file - These guidelines **direct crawlers on which areas to explore or avoid**, respecting website preferences and ensuring privacy - Once a crawler reaches a webpage, it **fetches the HTML content** of that page - The crawler **examines the HTML structure** and retrieves information including: - Text content - Headings - Links - Metadata - To understand the webpage structure, **the HTML is broken down into individual components** - This process involves **identifying elements, tags, and attributes** that hold valuable information like titles and headings

Crawlers don't read webpages like humans do - they parse the HTML structure to extract structured data for indexing

## Indexing - The data extracted from webpages is **indexed by storing the collected information in a structured manner** within a search engine's database - **Each word in the document is included in the page's index as an entry**, along with the word's position on the page - The index enables **quick retrieval and ranking of relevant web pages** in response to user queries

Indexing is the process of organizing and storing webpage data in a searchable database structure

## Ranking - When a user enters a query, the search engine **searches the index for matching pages** - The search engine **returns results they believe are the highest quality and most relevant** to the user's query ## Benefits of Search Engine Crawling & Indexing - The process of search engine indexing is **essential for search engines to collect, examine and arrange online content** - It involves **collecting and storing information from web pages in a searchable index** - There are several key benefits: - Improved search results - Efficient retrieval - Ranking and relevance - Freshness and updates ### Improved Search Results - Indexing webpages means search engines can: - **Provide users with relevant and up-to-date search results** - **Match user searches with content** which increases the chances of accurate and valuable results - Users are **more likely to find what they're looking for quickly**, ideally on the first page of search results - This eliminates the need to navigate to additional pages

Well-indexed content improves the user experience by reducing the time needed to find relevant information

### Efficient Retrieval - Indexing enables **efficient retrieval of information** - Search engines **don't need to scan the entire web for every search query** - They can **search their indexed data to produce search results quickly** | Without Indexing | With Indexing | |-----------------|---------------| | Must scan billions of pages for each query | Searches pre-organized index | | Extremely slow response times | Near-instant results | | High computational cost per search | Efficient resource usage | ### Ranking & Relevance - Indexing enables search engines to **assess the relevance and quality of web pages** - **Search result rankings are determined by various ranking algorithms** that analyse indexed data - These algorithms consider factors such as: - **Keyword relevance** - how well the page matches search terms - **Backlinks** - links from other websites pointing to the page - **User engagement** - how users interact with the page

Ranking algorithms use indexed data to determine which pages best answer a user's query

### Freshness & Updates - Search engine crawlers **periodically revisit indexed web pages to detect updates and changes** - This process ensures that **search results display the latest content** currently accessible on the Internet - If a webpage has been updated and not re-crawled, **the page may no longer be relevant for the user's search** - Think of it like a news website: if the index isn't updated regularly, users might see yesterday's headlines when searching for today's news

Outdated indexes can lead to poor user experiences by showing irrelevant or obsolete information

## Summary Table: Search Engine Indexing Process | Stage | Purpose | Key Activities | |-------|---------|----------------| | **Crawling** | Discover web pages | Follow links, fetch HTML, respect robots.txt | | **Indexing** | Organize content | Parse HTML, extract data, store in database | | **Ranking** | Serve relevant results | Apply algorithms, assess quality, order results |