Ruby gem to inspect web pages. It scrapes a given URL and returns its title, description, meta tags, links, images, and more.
Add this line to your application's Gemfile:
gem 'webinspector'And then execute:
$ bundle
Or install it yourself as:
$ gem install webinspector
page = WebInspector.new('http://example.com')page = WebInspector.new('http://example.com', {
timeout: 30, # Request timeout in seconds (default: 30)
retries: 3, # Number of retries (default: 3)
headers: {'User-Agent': 'Custom UA'} # Custom HTTP headers
})page.response.status # 200
page.response.headers # { "server"=>"apache", "content-type"=>"text/html; charset=utf-8", ... }
page.status_code # 200
page.success? # true if the page was loaded successfully
page.error_message # returns the error message if anypage.url # URL of the page
page.scheme # Scheme of the page (http, https)
page.host # Hostname of the page (like, example.com, without the scheme)
page.port # Port of the page
page.title # title of the page from the head section
page.description # description of the page
page.links # array of all links found on the page (absolute URLs)
page.images # array of all images found on the page (absolute URLs)
page.meta # meta tags of the page
page.favicon # favicon URL if availablepage.meta # all meta tags
page.meta['description'] # meta description
page.meta['keywords'] # meta keywords
page.meta['og:title'] # OpenGraph titlepage.domain_links('example.com') # returns only links pointing to example.com
page.domain_images('example.com') # returns only images hosted on example.compage.find(["ruby", "rails"]) # returns [{"ruby"=>3}, {"rails"=>1}]page.javascripts # array of all JavaScript files (absolute URLs)
page.stylesheets # array of all CSS stylesheets (absolute URLs)page.language # detected language code (e.g., "en", "es", "fr")page.structured_data # array of JSON-LD structured data objects
page.microdata # array of microdata items
page.json_ld # alias for structured_datapage.security_info # hash with security details: { secure: true, hsts: true, ... }page.load_time # page load time in seconds
page.size # page size in bytespage.content_type # content type header (e.g., "text/html; charset=utf-8")page.technologies # hash of detected technologies: { jquery: true, react: true, ... }page.tag_count # hash with counts of each HTML tag: { "div" => 45, "p" => 12, ... }page.feeds # array of RSS/Atom feed URLs found on the pagepage.social_links # hash of social media profiles: { facebook: "url", twitter: "url", ... }page.robots_txt_url # URL to robots.txt
page.sitemap_url # array of sitemap URLspage.cms_info # hash with CMS details: { name: "WordPress", version: "6.0", themes: [...], plugins: [...] }page.accessibility_score # hash with score (0-100) and details: { score: 85, details: [...] }page.mobile_friendly? # true if the page has viewport meta tag and responsive CSSpage.to_hash # returns a hash with all page dataNew Features:
- RSS/Atom feed detection with
feedsmethod - Social media profile extraction with
social_linksmethod - CMS detection and information with
cms_infomethod (WordPress, Drupal, Joomla, Shopify, Wix, Squarespace) - Accessibility scoring with
accessibility_scoremethod - Mobile-friendly detection with
mobile_friendly?method - Robots.txt and sitemap URL detection with
robots_txt_urlandsitemap_urlmethods
Improvements:
- Enhanced
Requestmodule withvalid?andssl?methods for better URL validation - Improved
Metamodule with author and publisher extraction - Better error handling across all modules
- Performance improvements with internal caching
- Steven Shelby (@stevenshelby)
- Sam Nissen (@samnissen)
The WebInspector gem is released under the MIT License.
- Fork it ( https://github.com/davidesantangelo/webinspector/fork )
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create a new Pull Request