Day 4: Model layer

The database problem is solved. My limited understanding pins the failure on my pre-set “id” attribute. Currently I’m setting :id manually after calling the superclass (ActiveRecord::Base) constructor and before saving the object, which appears to work.

A second problem occurred when I saved objects more than once, specifically when the ID is the same but non-key fields changed. I got around this by doing an update if the ID exists in the database and a save if it does not. This results in an extra query with every save, but caching is done so rarely that this should not make a difference

app/models/person.rb constructor:

  def initialize( person_info )
    if person_info.instance_of? Basewrapper::Recordwrapper
      hash = person_info.get_hash
      id = hash["id"]
      super hash
      write_attribute :id, id
      if Person.exists? id
        update
      else
        save
      end
    end
  end

Timeentry, Project and Company look very similar to this. To create basic associations, I declare that Project and Person “has_many :timeentries” and Timeentry “belongs_to” Project and Person.

I added the custom attribute “api_user” to the basecamp objects. This will allow me to clear the cache for each user at the end of the session.

I also added the static function “update_cache(url, login, pass, use_ssh)” to every model object. The cache will be initialized when the user authenticates, and this will be our only basecamp interaction. All subsequent data is read from my local database.

Where this is going:
Once the model layer is complete, I will write the controller layer to generate graphs of hours per person, hours per project and (if I have time) a pivot view of hours by person by project.

Day 4: Basecamp Caching Considerations

Caching is taking far more resources than I expected. Overall I’m pulling ~ 3 megs of data for all people, companies, projects, and 30 days of time entries of a mid-size web development firm.

Since I’m taking advantage of ActiveRecord for relational object access, dynamic (on request) caching is going to be tricky. I’d have to hook into ActiveRecord’s primary key lookup mechanism and request the data from basecamp IFF we fail to find it in the DB.

I don’t think I will accomplish dynamic caching within this 7-day learning experience, but suggestions are welcome as always! =)

Day 3: Caching

Kept working on the basecamp wrapper and flattened the basecamp response to a legally formatted, one-dimensional mapping that I can use to initialize ActiveRecord objects.

At about 4 AM last night I started having doubts about my approach. There is so much re-mapping, and 37 Signals uses Ruby on Rails as well — they are either making this difficult intentionally or I’m missing a significant tool.

I’m trudging through this but I’m almost there — still, it bothers me that I haven’t found a simpler process to turn the 37 Signals Record into an ActiveRecord::Base child.

Update: 6:18 PM
I’ve created the database definitions (example below) and generated the model for Project, Company, Person, and Timeentry. For some reason the hash isn’t written to the test DB though. The inserted rows are NULL for every value, although the hash has proper values.

/db/migrate/002_create_projects.rb:

class CreateProjects  false
      t.column :status, :string, :null => false
      t.column :created_on, :date
      t.column :last_changed_on, :datetime
      t.column :company_name, :string
      t.column :company_id, :string, :null => false
      t.column :announcement, :string
      t.column :start_page, :string
      t.column :show_writeboards, :boolean
      t.column :show_announcement, :boolean
    end
  end

  def self.down
    drop_table :projects
  end
end

model/project.rb:

class Project < ActiveRecord::Base

  def initialize( project_recwrap )
    super project_recwrap.get_hash
  end

end

I’m fried, and there’s an art gallery opening and some house parties in my neighborhood. I’ll either take a look at this again tonight or tomorrow. By the end of tomorrow I need to have ActiveRecord-enabled basecamp objects for my data in order to stay on track, but I need a break.

Solution to Snag 2.B.1

require 'basecamp'

# Wrapper for 37's Basecamp class. This provides all
# basecamp data with ActiveRecord friendly names and
# data.
class Basewrapper < Basecamp

  class Recordwrapper < Record

    # Mapping of illegal (hyphenated) to legal object types.
    # This mapping is used to rename these object types'
    # names, as well as the attributes that correspond to
    # them.
    @@name_translation = {
      "attachment-category" => "attachmentcategory",
      "post-category" => "postcategory",
      "time-entry" => "timeentry",
      "todo-item" => "todoitem",
      "todo-list" => "todolist"
    }

    # Translates all names to the mysql-safe format
    # by removing and replacing underscores.
    def initialize( type, hash )

      # Translate title if neccessary
      if @@name_translation.include? type
          type = @@name_translation[type]
      end

      # Hyphens will cause things to break eventually, so they stop here
      if type.to_s.include? "-"
        raise "unexpected hyphen in basecamp record response type"
      end

      # Change the hash attribute names to correspond
      # to our translated object title
      @@name_translation.each do |orig, trans|
        # The hash is multi-dimensional, so we
        # do this recursively.
        translate_hash_keys(hash) do |key|
          if key.include? orig
            key = key.sub orig, trans
          end
          key
        end
      end # @@name_translation.each...

      # replace the hyphens with underscores
      translate_hash_keys hash do |key|
        key = dedashify key, "_"
      end

      # now, initialize the 37 signals record
      super type, hash
    end

    # This function replaces all instances of the string
    # specified by orig in hash key names with the string
    # specified by trans.
    # This supports multi-dimensional hashes
    def translate_hash_keys( hash, &action )

      hash.each do |key, value|
        orig_key = key
        key = action.call key
        unless key == orig_key
          hash[key] = value
          hash.delete orig_key
        end
        # recurse if neccessary
        if value.instance_of? Hash
          translate_hash_keys( value, &action )
        end
      end
    end

    def to_s
      "#<Recordwrapper(#{@type}) #{@hash.inspect[1..-2]}>"
    end

    private

    # OVERRIDES Record#dashify
    # I broke dashify to allow def [](name) to parse "_"
    def dashify( name )
      name
    end

    # Removes all hyphens "-" from name by default. If
    # replacement is specified, "-" is replaced.
    def dedashify( name, replacement="" )
      name.to_s.tr "-", replacement
    end

  end # class Recordwrapper

  # OVERRIDES Basecamp#record
  # This wraps the single result in a Recordwrapper object.
  def record( path, parameters={} )
    result = request path, parameters
    (result && !result.empty?) ? Recordwrap.new(result.keys.first, result.values.first) : nil
  end

  # OVERRIDES Basecamp#records
  # This wraps an array of results in an array of Recordwrapper objects.
  def records( node, path, parameters={} )
    result = request(path, parameters).values.first or return []
    result = result[node] or return []
    result = [result] unless Array === result
    result.map { |row| Recordwrapper.new(node, row) }
  end

end

Day 2: Visualization + Caching

At the end of day 1, we are using the 37 signals Basecamp wrapper library in our ROR project. At this point, we can do basic basecamp queries (get all projects/people/etc) and toy with generic “Record” objects for these instances.

Right now I will set up my graphing mechanism, later tonight I will enhance the basecamp wrapper to allow handling of timeentries AND cache basecamp objects to a local database. Let’s get to it!

Part A: Visualization

I’ve chosen Scruffy as my graphing library, because it’s (a) known to work and widely used and (b) has an open architecture while (c) allowing for rapid prototyping. Scruffy takes care of the graph generation, leveraging RMagick (and in turn, ImageMagick) to generate the images.

require 'rubygems'
require 'scruffy'

graph = Scruffy::Graph.new
graph.title = "Phil's affection for Ruby"
graph.renderer = Scruffy::Renderers::Split.new(:split_label => 'Internal (Top) / External (Bottom)')
graph.add :area, 'Dreamed', [0,20,60,80,110,100], :category => :top
graph.add :area, 'Heartfelt', [0, 0, 40, 60, 100, 120], :category => :top
graph.add :bar, 'Vocalized', [0, 0, 20, 50, 90, 106], :category => :bottom
graph.add :bar,  'Blogged', [0, 0, 5, 3, 5, 20], :category => :bottom
graph.point_markers = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun'
graph.render(:width => 640, :as => 'JPG', :to => 'graphtest.jpg', :min_value => 0 )

Ok, so now graphing works — time to combine this with some real data!

 

 

 

 

 

Part B: Caching

The basecamp API includes a provision that bans “abuse”. We’ll be doing some statistical analysis of basecamp data (requiring frequent access to the same data sets) so some caching would be polite.

As it stands, the 37 signals basecamp wrapper returns a self-defined “Record” object with the following signature (Example is a “time-entry”):

#<Record(time-entries) "time-entry"=>{
  "person-id"=>314351, "date"=>#<Date: 4907793/2,0,2299161>,
  "todo-item-id"=>nil, "project-id"=>486815, "id"=>646586,
  "description"=>"description goes here", "hours"=>"8.0"
}>

My goal is to store this in a local database and access the data via a “Time-Entry” Active Record object.

Snag 1:

The variables couldn’t be named worse. From my limited ROR knowledge, the ActiveRecord class for “time-entry” should be named “Timeentry” and the table “timeentries.” Hyphens are illegal in mysql and will probably be (mis)interpreted as minus signs in other contexts as well.

The only easy way around this that I see is pattern-matching the offensive object and attribute names to strip hyphens from object names, and convert all other hyphens to underscores. For example, the variable “time-entry” becomes “timeentry” and “time-entry-id” becomes “timeentry_id”.

This requires hacking the basecamp wrapper and keeping a static list of object names, but I can’t think of a graceful quicker solution.

Snag 2:

The basecamp objects have globally unique pre-established “id” attributes, but the Rails Migration mechanism assumes an auto_increment “id” attribute. I’m considering two approaches:

1. drop the rails auto_increment “id” column with remove_column and create a non-auto_increment column using add_column

2. globally replace all “*_id” or “id” strings with “*_bid” or “bid”

Here I’ll try #1 first, since it seems a little more graceful.

Day 1: The workspace

My work setup

On day 0, I installed ROR on my development system (my trusty Sony VAIO PCG-V505EX), running Ubuntu Feisty Fawn with Apache on port 80 and Mongrel on port 3000. For my IDE, I’m trying out Aptana +RDT and good old emacs with a ruby mode.

I started reading “Why’s Poignant guide to Ruby” last night but found myself skipping over too much content — it’s cute and I’ll keep scanning it, but the Ruby API is better suited for my schedule (Rails API too). I’ll undoubtedly reference it enough in the future to start getting used to it.

Also started browsing the Basecamp API forums and downloaded the 37 signals Ruby Wrapper. Looking for a guide for my initial challenges, I found Beyond the Type’s BC / ROR tutorial.

First snag

Attempting to launch script/console gave me the following error:

/usr/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/commands/console.rb:25:in `exec':
No such file or directory - irb  -r irb/completion -r script/../config/../config/environment -r
console_app -r console_with_helpers --simple-prompt (Errno::ENOENT)

Gross! It turns out that irb was not installed on my system (“No such file or directory – irb”) and a quick apt-get install irb fixed this.

Second snag:

After installing irb and including the basecamp library, I get

/usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require':
no such file to load -- net/https (MissingSourceFile)

This is a little bit disconcerting — HTTPS is something I expect to be included alongside HTTP, but maybe ROR or Ubuntu is weird like that. A stab in the dark “sudo gem install net-https” didn’t work (no such gem), so I googled until I found a Spanish-speaking gentleman who encountered and solved this problem. Lo and Behold, the same fix worked for me.

Third Snag:

Script/console now starts, but I get WARNINGs about redefining a bunch of constants.

/usr/lib/ruby/gems/1.8/gems/xml-simple-1.0.11/lib/xmlsimple.rb:275:
warning: already initialized constant KNOWN_OPTIONS

/usr/lib/ruby/gems/1.8/gems/xml-simple-1.0.11/lib/xmlsimple.rb:280:
warning: already initialized constant DEF_KEY_ATTRIBUTES

I’ll take this warning to heart, since we’ll be transforming XML data into objects all day long and something is definitely amiss here. Again, this turns out to be a solved problem. Thanks, community!

“Are you using a gem such as ‘flickr’? Well, the reason you’re getting these warnings is that Ruby on Rails includes its own copy of the XmlSimple library. But XmlSimple is also available as a Ruby Gem, and the flickr gem depends on it. Ruby tries to avoid loading the same library twice, but it does this based on the file name. Problem is, the gem’s file is “xmlsimple.rb” but the Ruby on Rails’ file is “xml_simple.rb”. Bogus.”

Sure enough, at the top of basecamp.rb I find:

 begin
  require 'xmlsimple'
rescue LoadError
  begin
    require 'rubygems'
    require_gem 'xml-simple'
  rescue LoadError
    abort <<-ERROR
The 'xml-simple' library could not be loaded. If you have RubyGems installed
you can install xml-simple by doing "gem install xml-simple".
ERROR
  end
end

Changing this to the following took care of the double-loading:

begin
    require 'rubygems'
    gem 'xml-simple'
  rescue LoadError
    abort <<-ERROR
The 'xml-simple' library could not be loaded. If you have RubyGems installed
you can install xml-simple by doing "gem install xml-simple".
ERROR
end

Voila! Now I have a ROR environment, and a ruby interface to basecamp, all in time for lunch.