Ache - A Spider Web Crawler For Domain-Specific Search

ACHE is a focused spider web crawler. It collects spider web pages that satisfy around specific criteria, e.g., pages that belong to a given domain or that comprise a user-specified pattern. ACHE differs from generic crawlers inward feel that it uses page classifiers to distinguish betwixt relevant in addition to irrelevant pages inward a given domain. Influenza A virus subtype H5N1 page classifier tin ship away live from a elementary regular seem (that matches every page that contains a specific word, for example), to a machine-learning based classification model. ACHE tin ship away also automatically larn how to prioritize links inward social club to efficiently locate relevant content spell avoiding the retrieval of irrelevant content.

ACHE supports many features, such as:

• Regular crawling of a fixed listing of spider web sites
• Discovery in addition to crawling of novel relevant spider web sites through automatic link prioritization
• Configuration of dissimilar types of pages classifiers (machine-learning, regex, etc)
• Continuous re-crawling of sitemaps to discover novel pages
• Indexing of crawled pages using Elasticsearch
• Web interface for searching crawled pages inward real-time
• REST API in addition to web-based user interface for crawler monitoring
• Crawling of hidden services using TOR proxies

Documentation
More information is available on the project's documentation.

Installation
You tin ship away either attain ACHE from the source code, download the executable binary using conda, or purpose Docker to attain an icon in addition to run ACHE inward a container.

Build from source alongside Gradle
Prerequisite: You volition require to install recent version of Java (JDK 8 or latest).
To attain ACHE from source, yous tin ship away run the next commands inward your terminal:

git clone https://github.com/ViDA-NYU/ache.git cd ache ./gradlew installDist

which volition generate an installation bundle nether ache/build/install/. You tin ship away in addition to then brand ache ascendancy available inward the terminal past times adding ACHE binaries to the PATH surroundings variable:

export ACHE_HOME="{path-to-cloned-ache-repository}/build/install/ache" export PATH="$ACHE_HOME/bin:$PATH"

Running using Docker
Prerequisite: You volition require to install a recent version of Docker. See https://docs.docker.com/engine/installation/ for details on how to install Docker for your platform.
We issue pre-built docker images on Docker Hub for each released version. You tin ship away run the latest icon using:

docker run -p 8080:8080 vidanyu/ache:latest

Alternatively, yous tin ship away attain the icon yourself in addition to run it:

git clone https://github.com/ViDA-NYU/ache.git cd ache docker attain -t ache . docker run -p 8080:8080 ache

The Dockerfile exposes ii information volumes thence that yous tin ship away mountain a directory alongside your configuration files (at /config) in addition to save the crawler stored information (at /data) later the container stops.

Download alongside Conda
Prerequisite: You require to attain got Conda bundle manager installed inward your system.
If yous purpose Conda, yous tin ship away install ache from Anaconda Cloud past times running:

conda install -c vida-nyu ache

NOTE: Only released tagged versions are published to Anaconda Cloud, thence the version available through Conda may non live up-to-date. If yous desire to endeavour the almost recent version, delight clone the repository in addition to attain from source or purpose the Docker version.

Running ACHE
Before starting a crawl, yous require to attain a configuration file named ache.yml. We supply around configuration samples inward the repository's config directory that tin ship away assistance yous to acquire started.
You volition also require a page classifier configuration file named pageclassifier.yml. For details on how configure a page classifier, refer to the page classifiers documentation.
After yous attain got configured a classifier, the final thing yous volition require is a seed file, i.e, a obviously text containing ane URL per line. The crawler volition purpose these URLs to bootstrap the crawl.
Finally, yous tin ship away outset the crawler using the next command:

ache startCrawl -o  -c  -s  -m 

where,

• is the path to the config directory that contains ache.yml.
• is the seed file that contains the seed URLs.
• is the path to the model directory that contains the file pageclassifier.yml.
• is the path to the information output directory.

Example of running ACHE using the sample pre-trained page classifier model in addition to the sample seeds file available inward the repository:

ache startCrawl -o output -c config/sample_config -s config/sample.seeds -m config/sample_model

The crawler volition run in addition to impress the logs to the console. Hit Ctrl+C at whatever fourth dimension to halt it (it may accept around time). For long crawls, yous should run ACHE inward background using a tool similar nohup.

Data Formats
ACHE tin ship away output information inward multiple formats. The information formats currently available are:

• FILES (default) - raw content in addition to metadata is stored inward rolling compressed files of fixed size.
• ELATICSEARCH - raw content in addition to metadata is indexed inward an ElasticSearch index.
• KAFKA - pushes raw content in addition to metadata to an Apache Kafka topic.
• WARC - stores information using the criterion format used past times the Web Archive in addition to Common Crawl.
• FILESYSTEM_HTML - alone raw page content is stored inward obviously text files.
• FILESYSTEM_JSON - raw content in addition to metadata is stored using JSON format inward files.
• FILESYSTEM_CBOR - raw content in addition to around metadata is stored using CBOR format inward files.

For to a greater extent than details on how to configure information formats, encounter the data formats documentation page.

Bug Reports in addition to Questions
We welcome user feedback. Please submit whatever suggestions, questions or põrnikas reports using the Github effect tracker.
We also attain got a chat room on Gitter.

Contributing
Code contributions are welcome. We purpose a code trend derived from the Google Style Guide, but alongside iv spaces for tabs. Influenza A virus subtype H5N1 Eclipse Formatter configuration file is available inward the repository.

Contact

• Aécio Santos [aecio.santos@nyu.edu]
• Kien Pham [kien.pham@nyu.edu]

Download Ache