基于Lucene與Heritrix的搜索引擎構(gòu)建

摘    要
在互聯(lián)網(wǎng)蓬勃發(fā)展的今天，互聯(lián)網(wǎng)上的信息更是浩如煙海。人們?cè)谙硎芑ヂ?lián)網(wǎng)帶來的便利的同時(shí)，卻面臨著一個(gè)如何在如此海量的內(nèi)容中準(zhǔn)確、快捷地找到自己所需要的信息的問題，由此互聯(lián)網(wǎng)搜索引擎應(yīng)運(yùn)而生。
本文在對(duì)搜索引擎的原理、組成、數(shù)據(jù)結(jié)構(gòu)和工作流程等方面深入研究的基礎(chǔ)上，對(duì)搜索引擎的三個(gè)核心部分即網(wǎng)絡(luò)蜘蛛、網(wǎng)頁(yè)索引和搜索的分析及實(shí)現(xiàn)過程進(jìn)行闡述。網(wǎng)絡(luò)蜘蛛部分采用了基于遞歸和歸檔機(jī)制的Heritrix網(wǎng)絡(luò)爬蟲；網(wǎng)頁(yè)索引部分利用開源的Lucene引擎架構(gòu)設(shè)計(jì)并實(shí)現(xiàn)了一個(gè)可復(fù)用的、可擴(kuò)展的索引建立與管理子系統(tǒng)；搜索部分在Ajax技術(shù)支持上，設(shè)計(jì)并實(shí)現(xiàn)了一個(gè)靈活、簡(jiǎn)潔的用戶接口。本系統(tǒng)具有抓取網(wǎng)頁(yè)、建立和管理索引、建立日志以及搜索信息等功能，具備一定的應(yīng)用前景。

關(guān)鍵詞：搜索引擎；中文分詞；索引

 
The Construction of Search Engine Based on Lucene and Heritrix

Abstract
The contents on the Web are increasing exponentially as the rapid development of the Internet. A problem how to obtain the useful information from vast contents quickly and accurately is facing us while people are enjoying the convenience of the Internet. The solver of this problem is Web Search Engine.
The analysis and implementation process of three basic components of search engine(Crawler, Indexer and Searcher) is described in this paper on the basis of further study on the principles, composition, data structure and work flow of search engine. The crawler component is implemented with Heritrix crawler based on the mechanism of recursion and archiving; A reusable, extensible index establishment and management subsystem are designed and implemented by open-source package named “Lucene” in the indexer component; The Searcher component based on the Ajax technology is designed and realized as a flexible, concise user interface. The system has some functions, such as crawling web page, establishment and management index, establishment log and search information, it has a certain application prospect.

Key Words：Search Engine；Chinese Word Segmentation；Index

 
目    錄
摘    要 I
Abstract II
1 緒論 1
1.1 項(xiàng)目背景 1
1.2 國(guó)內(nèi)外發(fā)展現(xiàn)狀 1
2 系統(tǒng)的開發(fā)平臺(tái)及相關(guān)技術(shù) 3
2.1 系統(tǒng)開發(fā)平臺(tái) 3
2.2 系統(tǒng)開發(fā)技術(shù) 3
2.2.1 Heritrix網(wǎng)絡(luò)爬蟲簡(jiǎn)介 3
2.2.2 Lucene技術(shù)簡(jiǎn)介 4
2.2.3 Ajax技術(shù)簡(jiǎn)介 4
3 系統(tǒng)分析與設(shè)計(jì) 6
3.1 系統(tǒng)需求分析 6
3.1.1 系統(tǒng)架構(gòu)分析 6
3.1.2 系統(tǒng)用例模型 6
3.1.3 系統(tǒng)領(lǐng)域模型 10
3.2 系統(tǒng)概要設(shè)計(jì) 11
3.3 系統(tǒng)詳細(xì)設(shè)計(jì) 12
3.3.1 索引建立子系統(tǒng) 13
3.3.2 用戶接口子系統(tǒng) 17
4 系統(tǒng)的實(shí)現(xiàn) 18
4.1 系統(tǒng)包框架的構(gòu)建 18
4.1.1 索引建立子系統(tǒng) 18
4.1.2 用戶接口子系統(tǒng) 19
4.2 系統(tǒng)主要功能實(shí)現(xiàn) 19
4.2.1 索引建立子系統(tǒng) 19
4.2.2 用戶接口子系統(tǒng) 22
結(jié)    論 24
參 考 文 獻(xiàn) 25
致    謝 26