Ruby Web Crawling

17 Web Crawling gems and projects

• Supplejack API

  1.3 7.6 Ruby

  Supplejack API Mountable Engine
  

• MetaInspector

  6.3 6.4 L5 Ruby

  Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
  

WorkOS - The modern identity platform for B2B SaaS

The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

Promo workos.com



• FastImage

  6.5 6.4 L5 Ruby

  FastImage finds the size or type of an image given its uri by fetching as little as needed

• LinkThumbnailer

  4.9 0.0 L5 Ruby

  Ruby gem that fetches images and metadata from a given URL. Much like popular social website with link preview.

• Wombat

  6.6 2.2 L5 Ruby

  Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

• Kimurai

  0.6 2.1 Ruby

  Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with Javascript rendered websites

• The Hawker Ruby gem

  1.9 0.0 Ruby

  The Hawker gem is a web scraper which allows you to pull the basic information for given social media profile URL

• anemone

  7.6 0.0 L5 Ruby

  Anemone web-spider framework

• Mechanize

  9.0 7.2 L4 Ruby

  Mechanize is a ruby library that makes automated web interaction easy.
  

• Upton

  7.1 0.0 L5 HTML

  A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)
  

• Screencap

  3.3 0.0 JavaScript

  A gem to screencap webpages in ruby. Uses Phantom.js under the hood.

• pismo

  5.5 0.0 L5 Ruby

  Extracts machine-readable metadata and content from Web pages

• instabot.rb

  2.8 0.0 Ruby

  An instagram bot works without instagram api, only needs your username and password. written in ruby

• Vessel

  4.6 5.6 Ruby

  Fast high-level web crawling Ruby framework
  

• Google Search Results in Ruby

  1.7 5.1 Ruby

  Google Search Results via SERP API Ruby Gem
  

• Spidr

  5.7 6.4 Ruby

  A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

• 🕷 Tanakai

  3.0 6.1 Ruby

  Tanakai is a modern web scraping framework written in Ruby. A fork of Kimurai.

* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.

InfluxDB - Power Real-Time Data Analytics at Scale sponsored www.influxdata.com

Popular Comparisons

• Wombat
  vs
  Mechanize
• Mechanize
  vs
  anemone
• Mechanize
  vs
  Kimurai
• The Hawker Ruby gem
  vs
  Kimurai
• MetaInspector
  vs
  Mechanize

SaaSHub - Software Alternatives and Reviews sponsored www.saashub.com