Selected Tags

Click on a tag to remove it

More Tags

Click on a tag to add it and filter down

Web Crawling gems

Showing projects tagged as Web Crawling

  • Mechanize

    9.0 7.1 L4 Ruby
    Mechanize is a ruby library that makes automated web interaction easy.
  • anemone

    7.6 0.0 L5 Ruby
    Anemone web-spider framework
  • Upton

    7.1 0.0 L5 HTML
    A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)
  • FastImage

    6.6 6.7 L5 Ruby
    FastImage finds the size or type of an image given its uri by fetching as little as needed
  • Wombat

    6.6 2.2 L5 Ruby
    Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
  • MetaInspector

    6.3 6.4 L5 Ruby
    Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
  • Spidr

    5.7 6.4 Ruby
    A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
  • pismo

    5.5 0.0 L5 Ruby
    Extracts machine-readable metadata and content from Web pages
  • LinkThumbnailer

    4.9 0.0 L5 Ruby
    Ruby gem that fetches images and metadata from a given URL. Much like popular social website with link preview.
  • Vessel

    4.6 5.6 Ruby
    Fast high-level web crawling Ruby framework
  • Screencap

    3.3 0.0 JavaScript
    A gem to screencap webpages in ruby. Uses Phantom.js under the hood.
  • 🕷 Tanakai

    3.1 6.1 Ruby
    Tanakai is a modern web scraping framework written in Ruby. A fork of Kimurai.
  • instabot.rb

    2.8 0.0 Ruby
    An instagram bot works without instagram api, only needs your username and password. written in ruby
  • html2rss

    2.2 6.4 Ruby
    📰 Build RSS 2.0 feeds from websites (and JSON APIs) with a few CSS selectors.
  • The Hawker Ruby gem

    1.9 0.0 Ruby
    The Hawker gem is a web scraper which allows you to pull the basic information for given social media profile URL
  • Google Search Results in Ruby

    1.7 5.1 Ruby
    Google Search Results via SERP API Ruby Gem
  • Supplejack API

    1.3 7.4 Ruby
    Supplejack API Mountable Engine
  • Kimurai

    0.6 2.1 Ruby
    DISCONTINUED. Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with Javascript rendered websites