Going Dark. Again

May 6th, 2008

UPDATE: I reckon this article has nailed it

This weekend was the Anzac weekend, a time for reflection and remembering the things our Grand Fathers and Great Grand Fathers (and possibly Mothers) did for us and our way of life. I miss you Pa.

I had the opportunity on Friday night to sit down with some code, uninterrupted. No internet, therefore no Email, IM, IRC, Twitter, RSS. It was grand.

It started me thinking, how the hell can us “knowledge” (I hate that term) workers balance being as productive as you know you can with keeping in touch? The vast majority of my friends are online most of the time, at work, at home and even in between sometimes (Hi Lachlan!). I like being in the loop, I like knowing what’s going on. Twitter has made that a process alot easier than it was 12 months ago.

So I have decided that my work day is now going to consist mostly of “off-line” time. 9 – Lunch will be offline, that is, working without distractions. No Email, IRC, IM, Twitter. At Lunch time i’ll surface again for a little while, probably an hour tops. After lunch i’ll hook back in. After the kids are in bed I may come back on, maybe not. Dunno yet.

I tend to embrace things half-heartedly, this is one thing i’m going to try and stick to. If you need me urgently, SMS or Phone is where it’s at.

The other side effect I hope this will address is my new level of bored-ness with the interwebs, seriously, I can’t find anything interesting out there atm.

Apologies for the above brain dump, upon re-reading it wasn’t very well thought out. Oh Well.

Matta

4 comments »

Fluid.app

December 18th, 2007

Have been using Fluid for the past week.

It’s an OS X (Leopard only) app that lets you create standalone cocoa based web apps, basically a wrapper for web kit.

I’ve got one for gmail and unfuddle setup. It’s nice to have them survive browser crashes normally due to unruley javascript.

UPDATE: Here’s a sexy big icon the UF guys sent me

Highly resposive Unfuddlers

December 6th, 2007

To follow on from my last post, I have spent the day to-ing and fro-ing with the Unfuddle.com guys and I have come up with an SVN hook for non-UF-hosted svn repos.

Here’s for code for anyone interested

Move over lighthouse, Unfuddle it is

November 30th, 2007

So, I blogged earlier about Lighthouse and how much I was in love it with. It has all the features I like but unfortunately for me, it missed a few critical ones my Boss wanted.

Enter Unfuddle

We’ve been using this for a few months now and it works a charm. It’s a rails app and is run buy some very passionate dudes in Hawaii.

The feature that really stands out for us is the work flow for tickets. See image below.

Boss opens it, I accept it, I resolve it, boss closes it.

Boss can reopen it if need be. So easy for me to see what tickets are left to work on in this release, also what tickets need to be verified by him before I can push the release live.

Anyone who is dealing with a large ongoing site needs a decent bug tracker, this one is up there with the best i’ve used, both OSS and commercial.

Print this out, give it to your boss

November 27th, 2007

I don’t actually have to, he sent me the link.

This made me a very happy programmer.

Wide vs. Deep

Going Dark

October 17th, 2007

Not sure if it’s just me, but lately the constant barrage of IM/IRC/Twitter/Email has been making me feel less productive than I know I can be so for the majority of the day I have been Going Dark. That is, turn comms off and the music up.

I don’t go completely off net as I need my other brain to remind me of things I constantly forget.

I miss my IRC buddies from #roro on freenode.net heaps, they are a constant source of knowledge and entertainment but they are pretty good at posting to the roro tumblr, so that keeps me in the loop mostly.

I actually jump on IM and IRC once or twice during the day, mostly at lunch, or in a lull between pushing out code.

I’d like to hear how everyone else deals with the “always on” side of things?

When Testing Caching ...

October 15th, 2007

Don’t forget to set:

config.action_controller.perform_caching = true

in test.rb in environments. You’ll wonder why nothing is working until you do.

-33.702635,151.099434 : Hornsby

September 17th, 2007

So what does a north western Sydney suburb have to do with rails? Not very much, but it has lent it’s name to a very very cool app written by Lachie Cox.

The main aim of Hornsby is to get rid of the dreaded rails fixtures. As a newcomer to Rails about a year ago, testing was a new and exciting thing for me. It took a bit of discipline but I got into it. Then I got out of it. Fixtures where p*ssing me off, badly. They were a nightmare to maintain and actually made me write worse code than normal due to the fact I didn’t want to rewrite all my fixtures.

I then started using Rspec, I love writing specs but fixtures were still making me swear at my MBP.

Hornsby makes me an extremely happy Rails programmer. Here is Lachie’s description.

Here’s my code. It’s an example of setting up a “scenario” that I used for testing my Equipment

1
2
3
4
5
6

scenario :equipment do
  @manufacturer = EquipmentManufacturer.create!(:name => "Ben Hogan")
  @category = EquipmentCategory.create!(:title =>"Fairway Woods")
  @equipment = EquipmentProduct.new(:name => "Edge CFT Ti  Hybrid",:equipment_manufacturer_id => @manufacturer.id,:equipment_category_id => @category.id)
  @equipment.save(false)
end

Here is the bit from Rspec test that loads it up.

1
2
3
4
5
6
7

describe EquipmentController do

  before(:each) do
    hornsby_scenario :equipment
  end
   ...
end

You’ll notice I hand in “false” to the save method, this allows us to bypass all the validations and setup just the bare minimum to pass tests. This is helpful sometimes but be aware it can lead to other weird stuff than you might need to be aware of, assuming your validations are setup for a reason. Another nice side effect of Hornsby scenarios is the fact that if your DB tables are not the rails default ones you don’t have to jump though hoops to get your Db populated.

This software has made my testing life a joy yet again, knowing the exact state of your DB when testing is precisely what was missing from fixtures.

Here’s those links again:

Lachie’s write up

Source Code

Out of Bandwidth Cache Expiry

August 29th, 2007

So, you’ve been merrily caching away and using the cache plugin I wrote and you’re still getting some heat about your site sometimes “hanging”. I reckon the problem might be that the person complaining is getting stuck behind a slow running process that is re-caching a page on your site. So what to do?

What I’ve done is started an extra mongrel in my pack but not included it in the standard mongrel pack, my (cut down) mongrel config looks like this:

1
2
3
4
5
6

--- 
port: "2222"
environment: production
address: 127.0.0.1
pid_file: log/mongrel.pid
servers: 7

As you can see there is 7 mongrels there. Next, setup a second cluster that contains only your 7th mongrel

1
2
3

<Proxy balancer://expirecluster>
  BalancerMember http://127.0.0.1:2228
</Proxy>

Next, tell apache to redirect any of your magic expiration URLs to that cluster

1
2

  RewriteCond  %{QUERY_STRING} magic_url_expire_param=1$
  RewriteRule ^/(.*)$ balancer://expirecluster%{REQUEST_URI} [P,QSA,L]

Restart apache and you’re done. Any cron triggered expire requests will now go through the out of bandwith mongrel and not slow down visitors to you site.

This method can actually be used to segregate any URLs off to different clusters, I have a project coming up that will have a lot of slow processes that rely on an external SOAP service running. They’ll all be farmed off to a cluster running on a totally different machine with trimmed down, DB free mongrels.

Seesaw, restarting your mongrels with zero downtime

August 21st, 2007

Last weekend was the first Sydney Rails Group Hax Day, held by our very own Lachie. What a day! about 10 rails geeks showed up including a few guys we had either never met or only at the previous meet-up.

Max and I started the day talking about what we’d like to see in a web-based app that managed other rails apps. In discussing this I mentioned that over that iseekgolf.com we’re running getting a serious amount of traffic and a lot of that is e-commerce related, i.e stuff that would really screw me up if the mongrel is was being executed on was killed half way though a process. This led to me suggesting that it’d be great if you could restart mongrels bit by bit.

After about 30 mins of talking it though we decided that the best way was to split the mongrel pack down the middle and bounce each half separately while the other half handled the load. This would potentially slow the site down however it shouldn’t shut the site down entirely.

What we came up with was Seesaw. This does precisely what we needed including configuring the web server on the fly.

This morning I installed it live at iseekgolf.com and after a few config tweaks it was working a treat. It brought a huge smile to my face as the logs kept rolling buy as the mongrels were restarting.

One gotcha is that by default Seesaw names your mongrel cluster “mongrel_pack”, be sure to replace all instances of your mongrel cluster name in your apache config file to this new name, especially the SSL one ;)

Hopefully some people may find a use for this plugin.

We’ll be doing some minor changes to it over the coming days, including some definable pauses in between the cluster restarts and apache redirecting traffic to them to give them time to warm up.

Max has written up an awesome blog entry, with pretty pictures and all!

Lighthouse App, Svn and you

August 13th, 2007

Last week we decided that our ad-hoc issue tracking system wasn’t good enough for the serious development we are doing now. We tried Trac, Basecamp and iGTD with email submission. None of them really impressed us for various reasons. Trac was the best of the bunch for bug tracking but never really let us spread our wings. Basecamp is a great app for what it is meant to do but not the best for bug tracking. I setup iGTD to accept tasks from the boss via email, the problem then was closing the loop, when I was done it was up to me to let the boss know.

Enter Lighthouse

What a brilliant piece of software, it’s written by the Active Reload guys who also wrote Mephisto the blogging software this blog is written in. The brilliance of this software comes of it’s simplicity. It does the job I want it to do almost perfectly.

It doesn’t make any assumptions about things like Bug Priority and such, however it does have a standard tagging system and you can save searches into “bins” that are available on the bug searching screen, we have settled on, low,normal,high,highest and URGENT. See below.

I also implemented the SVN integration, the FAQ can be found here Now all I do is put a proper comment in the SVN commit that explains the changes and has the lighthouse syntax in it, an example might be “Changes to log form [#15 state:resolved]”. A commit with that comment in it closes the ticket as well, brilliant.

So the aim of this post was really just to send props to Rick and the team at Active Reload, it’s great to see an app by the geeks, for the geeks.

The Caching Gap(tm) ... mixin' it with Cron

August 10th, 2007

So the plugin i mentioned in my last post is alive and kicking, it has totally solved the problem we were having.

A new problem arose today, now that the pages are begin re-cached as part of the sweeping process, it was taking ages. Ages being up to 60 seconds to re-cache them all, also ages with the boss sitting there waiting for thing to happen. Not so good from a Usability perspective.

Again, I was throwing the idea of using DRB around on IRC and Lachie Cox suggested good old cron. We are already using cron to expire the front page on the hour to keep the info fresh so I thought i’d expand it a little.

I created a new model called CacheQueue that had two columns, cache_type and cache_data. The two types of cache I hose now are the regular esxpressions that get handed into an expire_fragment method, or the urls that get the special param handed in and get re-cached up.

The sweeper now looks like this

1
2
3
4
5
6
7
8
9
10

## The  Sweeper
class TournamentSweeper < ActionController::Caching::Sweeper
  observe ...models...

  def after_save(record)
      ...  
      CacheQueue.create({:cache_type => "url", :cache_data => '/?special_param_to_hose_cache=1'})
      CacheQueue.create({:cache_type => "regex", :cache_data => "tournaments/#{tournament.id}-*"})
      ...
  end

So we’re creating lots of entires in the cache_queues table, all having an expired = 0 column by default.

The next step was to write a method that could be called from a rake task that expires the caches, here it is

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

## The  CacheQueue Class
class CacheQueue < ActiveRecord::Base
  def self.expire
    begin
      require 'action_controller/integration'
      # wite a disc lock
      lockfile = RAILS_ROOT + "/log/cache.expire"
      if File.exists? lockfile
        $stderr.puts "Locked since " + File.open(lockfile,"r").atime.to_s
        return
      end
      FileUtils.touch lockfile
      caches_to_be_hosed = CacheQueue.find(:all,:conditions => {:expired => false}, :order => "cache_type DESC, id ASC")
      hosed_url_caches = []
      hosed_regex_caches = []
      sess = ActionController::Integration::Session.new
      sess.host! LIVE_HOST
      c = ActionController::Base.new 
      skipped = 0
      caches_to_be_hosed.each do |cache|
        # update the db
        cache.expired = true
        cache.save!
        case cache.cache_type
          when "url"
            if hosed_url_caches.include? cache.cache_data
              skipped += 1
              next 
            end
            sess.get cache.cache_data
            hosed_url_caches << cache.cache_data
          when "regex"
            if hosed_regex_caches.include? cache.cache_data
              skipped += 1
              next
            end
            c.expire_fragment(Regexp.new(cache.cache_data))
            hosed_regex_caches << cache.cache_data
        end
      end
      FileUtils.rm lockfile
    rescue
      FileUtils.rm lockfile
    end
    $stderr.puts ((hosed_regex_caches.size + hosed_url_caches.size).to_s + " caches removed") if (hosed_regex_caches.size + hosed_url_caches.size > 0)
    $stderr.puts skipped.to_s + " caches skipped" if skipped > 0
  end
end
 
1
2
3
4
5
6

## The  Rake task
namespace :cache do
  task :expire =>  :environment do
    CacheQueue.expire
  end
end

This write out to $stderr, when it actually does something it send me an email. It also implements locking so it doesn’t run over itself.

One of the cool parts of this is that it’ll actually skip over a cache if it already hosed it, so if in the space of the 4 minutes when it was run last the boss had edited the same article twice, which would schedule the re-caching of the front page and other busy pages, this method would only do each one once, your machine will thank you.

The Caching Gap(tm)

August 8th, 2007

So my talk at the July RoRo meetup was mostly about caching rails apps, and how we go about it. The main part of our site is the tournament sweeper that monitors everything that can belong to a tournament, articles, scores, courses etc. It also removes some of the busiest pages on the site, namely the front page and the News and Tournaments page.

This has worked all well and good, that was until we got huge amounts of traffic due to our coverage of the US PGA Bridgestone Invitational, the boss and the photographer where over there taking photos and video, most of the articles on that tourney have 15+ images on them.

What happened next was pretty inevitable, it was always going to happen just didn’t quite know when. I call it:

The Caching Gap

It’s the time between when you expire a cache and the next time its written out, The Caching Gap is usually closed up by the next poor bastard to come along and request the page, he’ll be sitting there twiddling his thumbs while the DB gets spanked and the cache rendered. So, what happens if that page takes, say 2-4 seconds to load and that page is getting >200 requests a second? I’ll tell you what happens, all your mongrels go “ok, no cache file present, better spank the DB and write one out”. Once this happens as many times as mongrels you have, things get messy. Enter my newest, and first, plugin, Cache Fixer.

The plugin itself only really does one minor thing, it allows you to hand in a force attribute to the cache method. At present, the cache method looks for the existence of the fragment you’re trying to cache and if it exists, just uses it, if not it’ll render and cache a new one. The changes are backwards compatible as the new force argument is false by default. Here’s an example:

1
2
3
4
5
6
7
8
9
10

## The Controller

class FrontpageController < ApplicationController
   def index
       @expire = params[:special_param_to_hose_cache] == "1" ? true : false
       if !read_fragment("frontpage") || @expire
          ... heavy lifiting goes here
       end
   end
end

 

1
2
3
4
5

## The View

<% cache("frontpage" , @expire) do %>
   ... render page here ...
<% end %>

The @expire variable get set in the controller and used in the view, thus allowing a single call to the method to re-render the cache. This solved half the problem. The other half is sweeping. I first tackled this in a very punk, quick-and-dirty way.

1
2
3
4
5
6
7

## The Quick and Dirty Sweeper
class TournamentSweeper < ActionController::Caching::Sweeper
  observe ...models...

  def after_save(record)
    system("wget #{LIVE_URL}?special_param_to_hose_cache=1 -O /dev/null &")
  end

This worked quite well, LIVE_URL is a constant that is set in each rails environment and this was even quite fast due the the & sending the process into the background however not very railsy, not very railsy at all.

After a bit of head banging, the good folk in #ror_au gave me a few heads up, specifically toolmantim and freelancing_god. This is what I came up with.

1
2
3
4
5
6
7
8
9
10
11

## The more Railsy Sweeper
class TournamentSweeper < ActionController::Caching::Sweeper
  observe ...models...

  def after_save(record)
        require 'action_controller'
        require 'action_controller/integration'
        sess = ActionController::Integration::Session.new
        sess.host! LIVE_HOST
        sess.get '/?special_param_to_hose_cache=1
  end

This code was yoinked from the console app on rails, works a treat. Obviously LIVE_HOST is a constant like LIVE_URL is in my app.

So that is how I solved my caching problem, I’m throwing it out there to anyone who can think of a better or faster way to do this. What I may do next is flick the re-caching to a DRB server.

The plugin is available at http://rails-oceania.googlecode.com/svn/mattallen/plugins/cache_fix/ it’s not tested beyond me playing around. The plugin includes this patch too. Solves a nasty cache_sweeper bug when called with the :only param

Triple J Twitter

August 7th, 2007

To scratch an itch of mine earlier this week I setup a new twitter account that broadcasts what program is currently on on Triple J

The account is at twitter.com/whatsonjjj

The code looks like this, it’s obviously specific to JJJ but could be easily modified to any other static HTML source. It gets run from cron on my server at one and 31 minutes past the hour.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

require 'rubygems'
require 'hpricot'
require 'open-uri'

now = Time.now
hour = now.hour.to_s.rjust(2,"0")
minute = now.min < 30 ? "00" : "30"
day = now.strftime("%A")
search =  hour + ":" + minute
url = "http://www.abc.net.au/triplej/guide/#{day}.htm"
doc = Hpricot(open(url))
doc.search("strong").each do |ele|
  re = Regexp.new("^" + search+"-")
  if ele.inner_text.match(re)
    time = ele.inner_text
    program = ele.parent.parent.parent.search("a").first.inner_text.gsub("'","&apos;")
    puts [day,time,program].join(" ")
    system("curl --basic --user 'whatsonjjj:xxxx' \
                --data-ascii \"status=#{[day,time,program].join(" ")}\" \
               'http://twitter.com/statuses/update.json'")
  end
end

Simple walk-though:

Grab the time now, figure out if we’re on the hour or the half-hour. Grab today’s guide. Parse it, look for a string tag that starts with HH:[00|30], grab the name of the program, send that over to twitter via their API

There is probably a more elegant way to get the name of the program than that, but that works.

I plan to grabbing out some of the other data too, host name and the like.

Here we go, here we go, here we go

August 7th, 2007

So today marks a new dawn of blogging for me. I have tried before and have failed miserably, I just didn’t think i had anything interesting to say.

As it turns out, some people think I do have interesting things to say. Last month at our Rails Meeting I gave a talk about how we do things over at iseekgolf.com. It was the first time i’d spoken publicly for, maybe, 10+ years? I think i was a little rusty to start with but soon found my groove. I have been known to be able to talk for way to long on things I know about. The video was featured video of the day on the 6th of August over at viddler

Anyway, the guys on IRC prodded me into starting to write some of my ramblings down, so here we go. Watch this space for a plugin i wrote today to handle a specific caching problem I was having.

Here’s the video.