Class CraigScrape::Scraper
In: lib/scraper.rb
Parent: Object

Scraper is a general-pupose base class for all libcraigscrape Objects. Scraper facilitates all http-related functionality, and adds some useful helpers for dealing with eager-loading of http-objects and general html methods. It also contains the http-related cattr_accessors:

logger - a Logger object to debug http notices too. Defaults to nil

retries_on_fetch_fail - The number of times to retry a failed uri download. Defaults to 8

sleep_between_fetch_retries - The amount of seconds to sleep, between successive attempts in the case of a failed download. Defaults to 30.

retries_on_404_fail - The number of times to retry a Resource Not Found error (http Response code 404). Defaults to 3.

sleep_between_404_retries - The amount of seconds to sleep, between successive attempts in the case of a Resource Not Found error. Defaults to 3.

Methods

downloaded?   new   uri  

Constants

URL_PARTS = /^(?:([^\:]+)\:\/\/([^\/]*))?(.*)$/
HTML_TAG = /<\/?[^>]*>/

Attributes

url  [R]  Returns the full url that corresponds to this resource

Public Class methods

new(init_via = nil)

Scraper Objects can be created from either a full URL (string), or a Hash. Currently, this initializer isn‘t intended to be called from libcraigslist API users, though if you know what you‘re doing - feel free to try this out.

A (string) url can be passed in a ‘http://’ scheme or a ‘file://’ scheme.

When constructing from a hash, the keys in the hash will be used to set the object‘s corresponding values. This is useful to create an object without actually making an html request, this is used to set-up an object before it eager-loads any values not already passed in by the constructor hash. Though optional, if you‘re going to be setting this object up for eager-loadnig, be sure to pass in a :url key in your hash, Otherwise this will fail to eager load.

[Source]

# File lib/scraper.rb, line 78
  def initialize(init_via = nil)
    if init_via.nil?
      # Do nothing - possibly not a great idea, but we'll allow it
    elsif init_via.kind_of? String
      @url = init_via
    elsif init_via.kind_of? Hash
      init_via.each_pair{|k,v| instance_variable_set "@#{k}", v}
    else
      raise BadConstructionError, ("Unrecognized parameter passed to %s.new %s}" % [self.class.to_s, init_via.class.inspect])
    end
  end

Public Instance methods

downloaded?()

Indicates whether the resource has yet been retrieved from its associated url. This is useful to distinguish whether the instance was instantiated for the purpose of an eager-load, but hasn‘t yet been fetched.

[Source]

# File lib/scraper.rb, line 93
  def downloaded?; !@html_source.nil?; end
uri()

A URI object corresponding to this Scraped URL

[Source]

# File lib/scraper.rb, line 96
  def uri
    @uri ||= URI.parse @url if @url
    @uri
  end

[Validate]