Description
This Ruby Gem is meant to scrape and parse results from Google, Bing, Baidu, Yandex, Yahoo, Ebay, Youtube, and more using SerpApi.
Google Search Results in Ruby alternatives and similar gems
Based on the "Web Crawling" category.
Alternatively, view google-search-results-ruby alternatives based on common mentions on social networks and blogs.
-
FastImage
FastImage finds the size or type of an image given its uri by fetching as little as needed -
Wombat
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages. -
MetaInspector
Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images... -
Spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use. -
LinkThumbnailer
Ruby gem that fetches images and metadata from a given URL. Much like popular social website with link preview. -
instabot.rb
An instagram bot works without instagram api, only needs your username and password. written in ruby -
The Hawker Ruby gem
The Hawker gem is a web scraper which allows you to pull the basic information for given social media profile URL -
Kimurai
DISCONTINUED. Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with Javascript rendered websites
Scout Monitoring - Performance metrics and, now, Logs Management Monitoring with Scout Monitoring
* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest.
Do you think we are missing an alternative of Google Search Results in Ruby or a related project?
README
Google Search Results in Ruby
This Ruby Gem is meant to scrape and parse results from Google, Bing, Baidu, Yandex, Yahoo, Ebay and more using SerpApi.
The following services are provided:
SerpApi.com provides a script builder to get you started quickly.
Installation
Modern Ruby must be already installed:
$ gem install google_search_results
Tested Ruby versions:
- 2.2
- 2.5.2
- 3.0.0
see: Git hub actions.
Quick start
require 'google_search_results'
search = GoogleSearch.new(q: "coffee", serp_api_key: "secret_api_key")
hash_results = search.get_hash
This example runs a search about "coffee" using your secret api key.
The SerpApi.com service (backend)
- searches on Google using the search: q = "coffee"
- parses the messy HTML responses
- return a standardizes JSON response The class GoogleSearch
- Format the request to SerpApi.com server
- Execute GET http request
- Parse JSON into Ruby Hash using JSON standard library provided by Ruby Et voila..
Alternatively, you can search:
- Bing using BingSearch class
- Baidu using BaiduSearch class
- Yahoo using YahooSearch class
- Yandex using YandexSearch class
- Ebay using EbaySearch class
- Home depot using HomeDepotSearch class
- Youtube using YoutubeSearch class
See the playground to generate your code.
Summary
- Google Search Results in Ruby
- Summary
- Guide
- How to set the private API key
- Search API capability for Google
- Example by specification
- Location API
- Search Archive API
- Account API
- Search Google Images
- Search Google News
- Search Google Shopping
- Google Search By Location
- Batch Asynchronous search
- Supported search engine
- Google search API
- Bing search API
- Baidu search API
- Yahoo search API
- Yandex search API
- Ebay search API
- Generic SerpApi search
- Error management
- Change log
- Roadmap
- Conclusion
- Contributing
Guide
How to set the private API key
The api_key can be set globally using a singleton pattern.
GoogleSearch.api_key = "secret_api_key"
search = GoogleSearch.new(q: "coffee")
or api_key can be provided for each search.
search = GoogleSearch.new(q: "coffee", api_key: "secret_api_key")
To get the key simply copy/paste from serpapi.com/dashboard.
Search API capability for Google
search_params = {
q: "search",
google_domain: "Google Domain",
location: "Location Requested",
device: "desktop|mobile|tablet",
hl: "Google UI Language",
gl: "Google Country",
safe: "Safe Search Flag",
num: "Number of Results",
start: "Pagination Offset",
api_key: "private key", # copy paste from https://serpapi.com/dashboard
tbm: "nws|isch|shop",
tbs: "custom to be search criteria"
async: true|false # allow async
}
# define the search search
search = GoogleSearch.new(search_params)
# override an existing parameter
search.params[:location] = "Portland,Oregon,United States"
# search format return as raw html
html_results = search.get_html
# search format returns a Hash
hash_results = search.get_hash
# search as raw JSON format
json_results = search.get_json
(the full documentation)[https://serpapi.com/search-api].
More search API are documented on SerpApi.com.
You will find more hands on examples below.
Example by specification
We love true open source, continuous integration and Test Drive Development (TDD). We are using RSpec to test our infrastructure around the clock to achieve the best QoS (Quality Of Service).
The directory test/ includes specification/examples.
Set your api key.
export API_KEY="your secret key"
Install RSpec
gem install rspec
To run the test:
rspec test
or if you prefers Rake
rake test
Location API
location_list = GoogleSearch.new(q: "Austin", limit: 3).get_location
pp location_list
it prints the first 3 location matching Austin (Texas, Texas, Rochester)
[{:id=>"585069bdee19ad271e9bc072",
:google_id=>200635,
:google_parent_id=>21176,
:name=>"Austin, TX",
:canonical_name=>"Austin,TX,Texas,United States",
:country_code=>"US",
:target_type=>"DMA Region",
:reach=>5560000,
:gps=>[-97.7430608, 30.267153],
:keys=>["austin", "tx", "texas", "united", "states"]},
...]
Search Archive API
This API allows to retrieve previous search. To do so run a search to save a search_id.
search = GoogleSearch.new(q: "Coffee", location: "Portland")
original_search = search.get_hash
search_id = original_search[:search_metadata][:id]
Now let retrieve the previous search from the archive.
search = GoogleSearch.new
archive_search = search.get_search_archive(search_id)
pp archive_search
it prints the search from the archive.
Account API
search = GoogleSearch.new
pp search.get_account
it prints your account information.
Search Google Images
search = GoogleSearch.new(q: 'cofffe', tbm: "isch")
image_results_list = search.get_hash[:images_results]
image_results_list.each do |image_result|
puts image_result[:original]
end
To download the image: wget #{image_result[:original]}
this code prints all the images links, and download image if you un-comment the line with wget (linux/osx tool to download image).
Search Google News
search = GoogleSearch.new({
q: 'cofffe', # search search
tbm: "nws", # news
tbs: "qdr:d", # last 24h
num: 10
})
3.times do |offset|
search.params[:start] = offset * 10
news_results_list = search.get_hash[:news_results]
news_results_list.each do |news_result|
puts "#{news_result[:position] + offset * 10} - #{news_result[:title]}"
end
end
this script prints the first 3 pages of the news title for the last 24h.
Search Google Shopping
search = GoogleSearch.new({
q: 'cofffe', # search search
tbm: "shop", # shopping
tbs: "tbs=p_ord:rv" # by best review
})
shopping_results_list = search.get_hash[:shopping_results]
shopping_results_list.each do |shopping_result|
puts "#{shopping_result[:position]} - #{shopping_result[:title]}"
end
this script prints all the shopping results order by review order with position.
Google Search By Location
With SerpApi.com, we can build Google search from anywhere in the world. This code is looking for the best coffee shop per city.
["new york", "paris", "berlin"].each do |city|
# get location from the city name
location = GoogleSearch.new({q: city, limit: 1}).get_location.first[:canonical_name]
# get top result
search = GoogleSearch.new({
q: 'best coffee shop',
location: location,
num: 1, # number of result
start: 0 # offset
})
top_result = search.get_hash[:organic_results].first
puts "top coffee result for #{location} is: #{top_result[:title]}"
end
Batch Asynchronous search
We do offer two ways to boost your searches thanks to async
parameter.
- Non-blocking - async=true (recommended)
- Blocking - async=false - it's more compute intensive because the search would need to hold many connections.
company_list = %w(microsoft apple nvidia)
puts "submit batch of asynchronous searches"
search = GoogleSearch.new({async: true})
search_queue = Queue.new
company_list.each do |company|
# set search
search.params[:q] = company
# store request into a search_queue - no-blocker
result = search.get_hash()
if result[:search_metadata][:status] =~ /Cached|Success/
puts "#{company}: search done"
next
end
# add result to the search queue
search_queue.push(result)
end
puts "wait until all searches are cached or success"
search = GoogleSearch.new
while !search_queue.empty?
result = search_queue.pop
# extract search id
search_id = result[:search_metadata][:id]
# retrieve search from the archive - blocker
search_archived = search.get_search_archive(search_id)
if search_archived[:search_metadata][:status] =~ /Cached|Success/
puts "#{search_archived[:search_parameters][:q]}: search done"
next
end
# add result to the search queue
search_queue.push(result)
end
search_queue.close
puts 'all searches completed'
This code shows a simple implementation to run a batch of asynchronously searches.
Supported search engine
Google search API
GoogleSearch.api_key = ""
search = GoogleSearch.new(q: "Coffee", location: "Portland")
pp search.get_hash
https://serpapi.com/search-api
Bing search API
BingSearch.api_key = ""
search = BingSearch.new(q: "Coffee", location: "Portland")
pp search.get_hash
https://serpapi.com/bing-search-api
Baidu search API
BaiduSearch.api_key = ""
search = BaiduSearch.new(q: "Coffee")
pp search.get_hash
https://serpapi.com/baidu-search-api
Yahoo search API
YahooSearch.api_key = ""
search = YahooSearch.new(p: "Coffee")
pp search.get_hash
https://serpapi.com/yahoo-search-api
Yandex search API
YandexSearch.api_key = ""
search = YandexSearch.new(text: "Coffee")
pp search.get_hash
https://serpapi.com/yandex-search-api
Ebay search API
EbaySearch.api_key = ""
search = EbaySearch.new(_nkw: "Coffee")
pp search.get_hash
https://serpapi.com/ebay-search-api
Youtube search API
YoutubeySearch.api_key = ""
search = YoutubeSearch.new(search_query: "Coffee")
pp search.get_hash
https://serpapi.com/youtube-search-api
Homedepot search API
HomedepotSearch.api_key = ""
search = HomedepotSearch.new(q: "Coffee")
pp search.get_hash
https://serpapi.com/homedepot-search-api
Walmart search API
WalmartSearch.api_key = ""
search = WalmartSearch.new(query: "Coffee")
pp search.get_hash
https://serpapi.com/walmart-search-api
Duckduckgo search API
DuckduckgoSearch.api_key = ""
search = DuckduckgoSearch.new(query: "Coffee")
pp search.get_hash
https://serpapi.com/duckduckgo-search-api
Generic SerpApi search
SerpApiSearch.api_key = ENV['API_KEY']
query = {
p: "Coffee",
engine: "youtube"
}
search = SerpApiSearch.new(query)
hash = search.get_hash
pp hash[:organic_results]
see: google-search-results-ruby/test/search_api_spec.rb
Error management
This library follows the regular raise an exception when something goes wrong provided by Ruby. Any networking related exception will be returned as is. Anything related to the client layer will be returned as a SerpApiException. A SerpApiException might be caused by a bug in the library. A networking problem will be caused by either SerpApi.com or your internet.
Change log
- 2.1 - Add more search engine: Youtube, Duckduckgo, Homedepot, Walmart
- improve error management and documentation.
- 2.0 - API simplified( GoogleSearchResults -> GoogleSearch), fix gem issue with 2.6+ Ruby, Out Of Box step to verify the package before delivery.
- 1.3.2 - rename variable client to search for naming consistency
- 1.3 - support for all major search engine
- 1.2 - stable versino to support goole and few more search engine
- 1.1 - client connection improvement to allow multi threading and fiber support
- 1.0 - first stable version with Google engine search with Google image
Roadmap
- 2.1 Improve exception / HTTP status handling
Conclusion
SerpApi supports all the major search engines. Google has the more advance support with all the major services available: Images, News, Shopping and more.. To enable a type of search, the field tbm (to be matched) must be set to:
- isch: Google Images API.
- nws: Google News API.
- shop: Google Shopping API.
- any other Google service should work out of the box.
- (no tbm parameter): regular Google search.
The field tbs
allows to customize the search even more.
The full documentation is available here.
Contributing
Contributions are welcome, feel to submit a pull request!
To run the tests:
export API_KEY="your api key"
rake test