About Site Map Submit Contact Us Log in | Create an account
Create an account Log In
Average Rating
User Rating:
Visitors Rating:
My rating:

Write review
  • License: Freeware
  • Last update: 5 years ago
  • Total downloads: 150
  • Price: Free |
  • Operating system: Linux
  • Publisher: Emil Romanus
See full specifications

linux default iconMethabot Publisher's description

File system crawler

The Methabot software is a speed-optimized, scriptable and highly configurable web, ftp and local file system crawler. It supports scripted filetype parsing, a wide variety of customization options and is easily configured to fit anyones particular needs.

With the use of the module system and scripting language, users are able to take full or partial control of the crawling process and decide however Methabot should store web data, statistics and much more.

Just by running Methabot from command line you are able configure custom filetypes, filtering expressions, behaviour, and much more, so you don't have to be a scripter!

Here are some key features of "Methabot":

В· It's fast, designed from the ground and up with speed-optimization in mind.
В· Scriptable through javascript with E4X
В· User-defined filetype filtering (according to MIME type, file extension or UMEX expression)
В· Multi-threaded
В· Highly configurable from command line
В· Extensible module system, supporting custom data parsers and filters.
В· Simple yet powerful filtering of URLs through UMEX.
В· Automated downloading
В· Support for automatic cookie handling when running over HTTP
В· Reliable, fault-tolerant networking
В· Portable, tested with success on 32-bit/64-bit Linux 2.6, 32-bit/64-bit FreeBSD 6.x/7.0, Windows XP and Mac OS X. Should work on almost any Unix-like OS.

What's New in This Release:

В· Bugfix, when external-peek was used the depth limit was messed up.
В· Memory usage cleanup fixes
В· dynamic-url option is no longer set to lookup by default, since it slows down the crawling significantly
В· Build system now creates and installs some header files that modules can use when linking
В· metha-config tool added
В· lmm_mysql moved outside of this package

System Requirements:

В· SpiderMonkey headers
В· curl and libcurl
Program Release Status: Minor Update
Program Install Support: Install and Uninstall

Is Methabot your software?

Manage your software

Most Popular

linux default icon Psiphon 2.5
At Psiphon we are committed to an open and accessible Internet.
linux default icon iPad forums 20101002
Only forums, only iPad
linux default icon Webalizer Xtended 2.23-05 RB29
Webalizer Xtended is a fork of Webalizer External link (a server log file analysis program)......
linux default icon JAWStats 0.7 Beta
A Free Web Site Statistics tool
linux default icon Phzilla (formerly PhProxy - InBasic) 3.5.1A
Allows you to open block pages automatically using web Proxy