When it comes to crossdomain quirks with frames and Ajax, there’s not usually a “good” solution — there’s just one that works. Something I was working on the other week had a “well it works” moment, although the solution was far from ideal.

The Problem

The page in question is the Sponsored These Tweeters page at SponsoredTweets. It’s a pretty basic Wordpress page with an iFrame that contains the list of Tweeters. Real easy way of adding a dynamic touch to a Wordpress site. The issue is that the iframe contains a dynamic amount of content, and could potentially grow or shrink based on the length of amount of tweeters being shown. The kicker? The page with the list of tweeters is at a subdomain, and under https.

Monitoring with Jquery

My first idea with anything like this is “Can I just listen for a jquery event on the element?”. Well, sure, but the height of the iframe won’t change. You’ll only be able to get the height/width of the iframe itself, not the content within it from the page itself, so watching for a resize or load event on the iframe won’t work. What you’re really wanting to listen for is a change in the height of the content

Works? No.

Reaching into the frame with contentWindow

If your frame and the page containing it both exist on the same domain, you can reach into DOM of the frame and get the height that way. At anytime you can do something like this to get the height of the page:

document.getElementById("iframeid")
         .contentWindow.document.body.scrollHeight

This bit of code will return an integer — the height of the content of the frame in pixels. Even if the iframe has a height of (for example) 300px, the content itself could be smaller or larger. Unfortunately you can’t set a jQuery event watcher on document.getElementById("iframeid").contentWindow.document.body either (at least from what I’ve seen), so I can’t see a way to watch for a change in height.

The “not-so-nice” fix for this is to make a really fast function that repeatedly checks a change in the height, and resizes the iframe accordingly. Here’s some sample code for this using jQuery.

var $iframe = $("#iframeid");
function resize_iframe() {
  var current_height = $iframe.css("height");
  if(current_height != $iframe[0].contentWindow.document.body.scrollHeight) {
    $iframe.css("height", $iframe[0].contentWindow.document.body.scrollHeight);
  }
}
setInterval("resize_iframe();", 100);

Everytime I use setTimeout() or setInterval() in Javascript, it’s immediately a red flag. These (along with the evil-eval) can be an obvious code smell, but also a trigger that the code itself isn’t designed in the best way.

The problem with this though is that it won’t quite be realtime. Also the contentWindow.document object is not available cross-domain. Even doing things like contentWindow.document.location.href to get the current location of the frame doesn’t work if the calling page and the frame are on different domains/subdomains. This might work for some cases, but not if your iframe is on a different domain.

Works? Yes.
Works on the same domain? Yes.
Works on different subdomains on the same domain? No.

The document.domain hack

If both your frame and your calling page exist on the same parent domain, there’s a little hack you can do to get this to work. Each page has a document.domain variable that contains the domain from which ajax calls can be made to. If you rails application is up at https://app.sponsoredtweets.com, then your document domain will be set to app.sponsoredtweets.com. Of course your main website sitting at http://sponsoredtweets.com will have a document.domain ot sponsoredtweets.com. Because of this they won’t be able to talk to each other. What you can do is manually change the document.domain value.

Surprisingly enough, you can tweak this value as long as you only get broader in your domain. For instance, if document.domain is app.sponsoredtweets.com, then you’ll be able to manually set this using something like this: document.domain = "sponsoredtweets.com" From that point on, that page will be treated as those requests were coming/going to that domain. Using this, you could set your iframe and the page containing the iframe to be on the same root domain, then reach into it’s contentHeight as described above. After you’ve set this to “sponsoredtweets.com” though, you won’t be able to change it again, as that’s the most general a document.domain can get.

Works on the same domain? Yep.
Works on different subdomains on the same domain? Yes.
Works crossdomain? No.
Works across https? No.

Frame within a Frame within a Frame

Unfortunately, none of this works across domain, of from http connections to https. Here’s how it works for SponsoredTweets:

The main page is http://sponsoredtweets.com/tweeters/sponsor-these-tweeters/, aka, the parent page.
Which has a frame to https://app.sponsoredtweets.com/tweeters, aka, the content page.
Which has a frame to http://sponsoredtweets.com/iframe.html, aka, the placeholder page.

The middle frame is the one that will change in height of course. For the same reasons as above, the middle frame cannot reach into it’s parent frame and call methods there either. It can, however, control it’s own iframe in a very limited sense. For instance, take this line from the

$("#parent_domain")[0].contentWindow.location = 'http://sponsoredtweets.com/iframe.html#' + height;

Whenever the height of the content page changes, it updates the location of it’s frame to include a hash tag followed by the current height of the content. By itself this won’t do anything, but if that iframe.html page is watching for changes to it’s location, we can start from there.

The placeholder page is what makes this all possible. All it has to do is monitor for changes to it’s hash, and when it sees them, send them up to the parent page (that is, the top level page). Now, although the content page can’t talk to it’s parent page, the placeholder page can talk to it since they are both on the same domain. Is it a hack? Oh yeah, and of the worst kind.

<html>
<script type="text/javascript">
<!--
  function sendHash() {
    var location = window.location.href.split("#")
    if(location.length != 2) { return; }
    parent.parent.receiveHash(location[1]);
  }
  setInterval("sendHash();", 100);
-->
</script>
<body></body>
</html>

parent.parent…? Yeah, thanks HTML. All the receiveHash() function on the parent page does is take a number and set it as the height of it’s iframe. Real simple implementation, it’s just the craziness that is cross-domain security that makes it difficult to grasp.

Works across https? Yes!

Got a Better Solution?

I can’t believe this would be the ideal way to do something as simple as resize an iframe. Do you know of a better way of doing this given that the pages exist on separate domains? I’d love to hear about it.

Long loading pages make people leave your site. Amazon and Google know this, and have optimized their sites accordingly. For Google, a half second in page load time decreased traffic and ad revenues by 20%! For Amazon every 100 ms resulted in decreased sales by 1%. For these companies, a half a second could mean a billion dollars in potential earnings. The book Website Optimization goes into this in more detail on the psychology of web performance, which poses a number of advantages of performance. If you’re taking speed seriously, you’re going to want to do all you can make a better user experience — both on the front end loading speed and the backend.

So what’s this delayed_job thing?

Optimizing some of the worst offenders is where delayed_job comes in. It shifts long running tasks within a page request to a later time, where it won’t impact the user. Delayed_job installs as a Ruby on Rails plugin from it’s github location. It acts like a queuing service for your application, with your database storing the queue itself. This has a number of advantages over another messaging queue like MQ, XMPP or Starling and Workling. With your database storing your queue, all you need is a delayed_job process running that will repeatedly query your database for new items. Removing a job from the queue is as easy as removing a row from a table.

What should be processed in the background?

So we know page load time is important, but what can we speed up? One of the easiest things is long running processes that don’t need immediate feedback to the user. Hitting a credit card gateway wouldn’t be a good example — in that case you’d want to give feedback to the user immediately. But for processes where the user might not even know something is happening, or if when you can show a placeholder while their request is processing, delayed_job makes a lot of sense. Here’s a few examples:

• Sending emails (surprisingly slow!)
• Resizing images
• Processing uploads (like importing a big csv)
• Hitting external services (like updating twitter stats for instance)
• more listed on the delayed_job github page

Getting it Setup

Just install the plugin as you would any other.

$ ruby script/plugin install git://github.com/tobi/delayed_job.git

Create a migration and fill it in with the database setup in delayed_job/spec/database.rb. At this time here’s what’s included.

$ ruby script/generate migration create_table_for_delayed_job

class CreateTableForDelayedJob < ActiveRecord::Migration
  def self.up
    create_table :delayed_jobs, :force => true do |t|
      t.integer  :priority, :default => 0
      t.integer  :attempts, :default => 0
      t.text     :handler
      t.string   :last_error
      t.datetime :run_at
      t.datetime :locked_at
      t.string   :locked_by
      t.datetime :failed_at
      t.timestamps
    end
  end

  def self.down
    drop_table :delayed_jobs
  end
end

Just save it and run your migrations.

$ rake db:migrate

Column Explanation

Most of the columns in the table are self explanatory. Priority can be anything, positive or negative, and it will be processed from largest to smallest. You can change priority of items in the queue at anytime, and they’ll be picked up that order. Attempts is, of course, how many times the job has been attempted. The number of attempts determines how long to wait before the next attempt. Originally the job is picked up in priority order, and will not be run again for 5+N^4 seconds, where N is the number of attemtps. By default, delayed_job items will be attempted 25 times, with just over 4 days between try 24 and 25. Handler is the YAML representation of the handler objects and it’s input (discussed more later). last_error is the message of the last error, while locked_by contains the server name, and process ID of delayed_job process that’s currently processing that item. This allows for multiple servers to process the queue, and for an easy overview of what each server is doing.

Creating Delayed Jobs Handlers

Creating a delayed job handler is probably the easiest part. There’s a great article on RailsTips with a few more examples, including using delayed_job to send out emails. I actually prefer using deliver_later for this, so that we can archive emails, but clear out processed jobs.

As for the handler, just create an object that receives whatever input you need. This can be full out ActiveRecord objects if you want, but those will be serialized and deserialized at runtime, so it makes more sense to just store references and load the objects at runtime. For instance, lets say you want to grab all of a users friends from twitter after they logged in, you might do something like this.

class LoadTwitterFriendsJob < Struct.new(:user_id, :twitter_screen_name)
  def perform
    user = User.find(user_id)
    user.update_friends!
  end
end

The twitter_screen_name parameter here isn’t used at all, but helps when debugging items in the queue without having to query multiple tables. If this throws any error it’ll be reprocessed later. In the case of connecting to Twitter this comes in very handy. Watch out though — if there’s any situation where you’d want this job to be marked as completed on error make sure you handle errors accordingly. For instance, if you’re hitting Twitter and a user has blocked access to your application, it won’t matter how many times you try — it’s going to fail.

There look to be a bit more elegant ways of creating the job handler, but I know this simple method has worked for me, and is processing requests on SponsoredTweets just fine.

Adding Delayed_jobs

Let’s say you want to update friends when a user logs in. In your sessions_controller, or maybe in an after_create in your Session model, you could add that user onto the queue.

Delayed::Job.enqueue(LoadTwitterFriendsJob.new(user.id, user.twitter_screen_name), 0, 5.minutes.from_now)

In this case, the item will be added with a priority of 0 (lowest), and set to run no sooner than 5 minutes. The LoadTwitterFriendsJob object will be saved as a YAML object with the payload passed in.

Processing the Queue

Delayed_job comes with a helpful task to get you off the ground fast – rake jobs:work, which you can keep open and monitor the progress of the queue. You can also pass in a rails environment here as well, such as rake jobs:work RAILS_ENV=qa. The github page has directions on how to daemonize the process using the daemon ruby gem.

# start the worker daemon
$ ruby script/delayed_job start -- production

# stop it
$ ruby script/delayed_job stop -- production

This will save a log file and pid file for the delayed_job process for you. You can setup something like Monit to monitor this process to make sure it’s running and not acting out at all times.

Update your Deploy Scripts

The last bit can be pulled directly from the MagnionLabs article — updating your dpeloy script. Delayed_job needs to be restarted everytime you deploy, otherwise it’ll be using an outdated version of your application to process the queue. This might not even be noticeable at first, but eventually this might end up in catastrophe.

# add this to config/deploy.rb
namespace :delayed_job do
  desc "Start delayed_job process"
  task :start, :roles => :app do
    run "cd #{current_path}; script/delayed_job start -- #{stage}"
  end

  desc "Stop delayed_job process"
  task :stop, :roles => :app do
    run "cd #{current_path}; script/delayed_job stop -- #{stage}"
  end

  desc "Restart delayed_job process"
  task :restart, :roles => :app do
    run "cd #{current_path}; script/delayed_job restart -- #{stage}"
  end
end
after "deploy:start", "delayed_job:start"
after "deploy:stop", "delayed_job:stop"
after "deploy:restart", "delayed_job:restart"

If you’re using multistage Capistrano deploys, it’ll use the stage you’re currently deploying to. Otherwise, you’ll probably need to replace that with RAILS_ENV. Also, the restart task depends on the delayed_job process already running. If you’re using monit and can mostly guanantee this you can leave it as it is, but otherwise you might want to change the restart role to include stopping and starting. Also you might have to chown the resulting log file generated.

Finishing Up

By default jobs will be deleted from the delayed_jobs table after being run. You can customize this yourself, as with the other defaults for max # of tries and the max runtime of any single process. Read more about that in the readme that comes with the plugin. If ever you want to clear out the queue altogether you can always run rake jobs:clear and start over fresh!

As some others have mentioned, it’s that time of year where most people with blogs make their public announcements on what languages they’ll be concentrating on for the upcoming year. Last year I think my top focus was Ruby on Rails and Javascript; so safe to say I’ve gotten a little experience in those. Things have taken a bit of a turn though, and I’m going to switch things up for 2009 and concentrate a little on my weaknesses and a little on new growth.

Weaknesses? Yeah, everyone has them. Those areas where when you have to implement something you bite your lip and find a way to get through it — usually with a 10ft poll. If they’re something that ties into your daily routine in some way, this is kind of task that can be holding you back from potentially getting more work done, or higher quality work.

One of my weaknesses has always been testing. Maybe I’ve been surrounded by too many testers, but its definitely something I could improve on. I wouldn’t consider myself fully proficient in a language to the point where I’d feel confident teaching someone else until I was proficient testing in it. Some languages don’t make this easy of course. Testing Javascript isn’t the most common thing, but it’s important, especially with such a Javascript heavy application as CloudShout. There are some great new tools like FireUnit for testing in Javascript too. Testing in Python, especially within Django has to have great support — as long as you actually do it. With Rails there’s a million different ways to test; but I’d like to concentrate mostly on mochs and stubs for a swiftly running test suite. No point in having a test suite that takes so long to run that no one has the time to run it after all.

There are a few other topics of interest for the new year. Things I’d love to look into and play around with at least a least a little. XMPP and especially bosh look extreme interesting to me, and I look forward to cramming together the little I’ve learned so far into an Adogo presentation in a little over a week.

Collective intelligence looks like another interesting topic that I’ve always wanted to dig a little deeper into. The Netflix prize looks an great way to experiment a little with it in some different language — Python, C and Erlang seem to be the most commonly used for it. There’s at least one active Rails plugin for collective intelligence, acts_as_recommendable, that uses a C implementation for the heavy lifting and might be worth checking out as well.

Last year git took over in a lot of programming world thanks to an amazing killer app, Github. Getting familiar with git isn’t a big thing, but it’s worth it to have a little more shared language with your peers. Moving from CVS to Subversion wasn’t something that most people did overnight, but after the switch no one ever went back. Git is shaping up to be same, but has a long road of public support and client side tools ahead of it before it makes it that far. People that like git, really really like git. I’m ashamed to say I’ve made improvements to a few projects I’ve cloned from Github, but never got around to branching and committing my changes (or writing tests for my changes of course). This is something I’d love to get better at — more for fun and familiarity with git than anything else. That’s the goal for this year — work on projects that are fun!