Ruby Web Crawling

15 Web Crawling gems and projects

• Mechanize

  8.9 6.8 L4 Ruby

  Mechanize is a ruby library that makes automated web interaction easy.
  

• anemone

  7.6 0.0 L5 Ruby

  Anemone web-spider framework

OPS - Build and Run Open Source Unikernels

Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.

Promo github.com/nanovms



• Upton

  7.2 0.0 L5 HTML

  A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)
  

• Wombat

  6.5 0.2 L5 Ruby

  Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

• FastImage

  6.4 4.3 L5 Ruby

  FastImage finds the size or type of an image given its uri by fetching as little as needed

• MetaInspector

  6.1 1.9 L5 Ruby

  Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
  

• pismo

  5.6 0.0 L5 Ruby

  Extracts machine-readable metadata and content from Web pages

• LinkThumbnailer

  4.8 1.3 L5 Ruby

  Ruby gem that fetches images and metadata from a given URL. Much like popular social website with link preview.

• Vessel

  3.5 5.8 Ruby

  Fast high-level web crawling Ruby framework
  

• Screencap

  3.2 0.0 JavaScript

  A gem to screencap webpages in ruby. Uses Phantom.js under the hood.

• instabot.rb

  2.7 0.0 Ruby

  An instagram bot works without instagram api, only needs your username and password. written in ruby

• The Hawker Ruby gem

  1.7 0.0 Ruby

  The Hawker gem is a web scraper which allows you to pull the basic information for given social media profile URL

• Supplejack API

  1.3 9.4 Ruby

  Supplejack API Mountable Engine
  

• Google Search Results in Ruby

  1.1 5.2 Ruby

  Google Search Results via SERP API Ruby Gem
  

• Kimurai

  0.6 2.1 Ruby

  Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with Javascript rendered websites

* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.

Less time debugging, more time building sponsored scoutapm.com

Popular Comparisons

• Mechanize
  vs
  Wombat
• anemone
  vs
  Mechanize
• Kimurai
  vs
  Mechanize
• Mechanize
  vs
  MetaInspector
• Upton
  vs
  Mechanize

Deliver Cleaner and Safer Code - Right in Your IDE of Choice! sponsored www.sonarlint.org