16 Web Crawling gems and projects
-
Upton
7.1 0.0 L5 HTMLA batteries-included framework for easy web-scraping. Just add CSS! (Or do more.) -
Wombat
6.6 0.0 L5 RubyLightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages. -
FastImage
6.5 0.0 L5 RubyFastImage finds the size or type of an image given its uri by fetching as little as needed -
MetaInspector
6.2 0.0 L5 RubyRuby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images... -
Spidr
5.6 10.0 RubyA versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use. -
LinkThumbnailer
4.9 0.0 L5 RubyRuby gem that fetches images and metadata from a given URL. Much like popular social website with link preview. -
instabot.rb
2.8 0.0 RubyAn instagram bot works without instagram api, only needs your username and password. written in ruby -
The Hawker Ruby gem
1.8 0.0 RubyThe Hawker gem is a web scraper which allows you to pull the basic information for given social media profile URL -
Kimurai
0.6 2.1 RubyKimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with Javascript rendered websites
Collect and Analyze Billions of Data Points in Real Time
Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.
Promo
www.influxdata.com
* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.