Selected Tags

Click on a tag to remove it

More Tags

Click on a tag to add it and filter down

Web Crawling gems

Showing projects tagged as Web Content Scrapers and Web Crawling

  • anemone

    7.5 0.0 L5 Ruby
    Anemone web-spider framework

  • Spidr

    5.8 4.1 Ruby
    A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

  • LinkThumbnailer

    5.0 5.5 L5 Ruby
    Ruby gem that fetches images and metadata from a given URL. Much like popular social website with link preview.

  • 🕷 Tanakai

    3.4 5.0 Ruby
    Tanakai is a modern web scraping framework written in Ruby. A fork of Kimurai.

  • html2rss

    2.4 8.6 Ruby
    📰 Build RSS 2.0 feeds from websites (and JSON APIs) automatically or with a few CSS selectors.

  • The Hawker Ruby gem

    2.0 0.0 Ruby
    The Hawker gem is a web scraper which allows you to pull the basic information for given social media profile URL

  • Kimurai

    0.6 2.1 Ruby
    DISCONTINUED. Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with Javascript rendered websites