Welcome

• Internet World Wide Web Search Engines
• Employ Amazon Elastic Compute Cloud (Amazon EC2) and Simple Storage Service (Amazon S3)
• Use open source software
• Rank pages using proprietary link analysis algorithms

We use the open source spider heritrix to crawl the web and hadoop distributed file system HDFS to store data. Our java based crawler is programmed to politely crawl and honor robots.txt, please send us an email if our spider is overloading your server.