Ruby Web Crawling

16 Web Crawling gems and projects

• Mechanize

  9.0 0.0 L4 Ruby

  Mechanize is a ruby library that makes automated web interaction easy.
  

• anemone

  7.6 0.0 L5 Ruby

  Anemone web-spider framework

Collect and Analyze Billions of Data Points in Real Time

Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.

Promo www.influxdata.com



• Upton

  7.1 0.0 L5 HTML

  A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)
  

• Wombat

  6.6 0.0 L5 Ruby

  Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

• FastImage

  6.5 0.0 L5 Ruby

  FastImage finds the size or type of an image given its uri by fetching as little as needed

• MetaInspector

  6.2 0.0 L5 Ruby

  Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
  

• pismo

  5.6 0.0 L5 Ruby

  Extracts machine-readable metadata and content from Web pages

• Spidr

  5.6 10.0 Ruby

  A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

• LinkThumbnailer

  4.9 0.0 L5 Ruby

  Ruby gem that fetches images and metadata from a given URL. Much like popular social website with link preview.

• Vessel

  4.4 0.0 Ruby

  Fast high-level web crawling Ruby framework
  

• Screencap

  3.2 0.0 JavaScript

  A gem to screencap webpages in ruby. Uses Phantom.js under the hood.

• instabot.rb

  2.8 0.0 Ruby

  An instagram bot works without instagram api, only needs your username and password. written in ruby

• The Hawker Ruby gem

  1.8 0.0 Ruby

  The Hawker gem is a web scraper which allows you to pull the basic information for given social media profile URL

• Google Search Results in Ruby

  1.5 0.0 Ruby

  Google Search Results via SERP API Ruby Gem
  

• Supplejack API

  1.3 0.0 Ruby

  Supplejack API Mountable Engine
  

• Kimurai

  0.6 2.1 Ruby

  Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with Javascript rendered websites

* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.

Learn any GitHub repo in 59 seconds sponsored getonboard.dev

Popular Comparisons

• Wombat
  vs
  Mechanize
• anemone
  vs
  Mechanize
• Mechanize
  vs
  Kimurai
• Mechanize
  vs
  MetaInspector
• Upton
  vs
  Mechanize

Clean code begins in your IDE with SonarLint sponsored www.sonarlint.org