Create Your Own Rails 3 Engine

Engine is an interesting feature of Rails. Engines are miniature applications that live inside your application and they have structure that you would normally find in a typical Rails application. If you have used the Devise gem, which itself is an engine, you know the benefits of being able to add functionality to your application with just a few lines of code. Another great benefit of engines is when you or your team are maintaining a number of applications the common functionalities can be extracted into engines.

Engines are already available prior to Rails 3 but it is not a core feature of the framework. As such, engine developers resorted to monkey-patching which, oftentimes, lead to engines breaking when Rails gets updated. In Rails 3.1, engines are now supported by the framework and there is now a clealy defined place where to hook your engines into Rails.

Now, let us go through the steps of building a simple engine. We will be working on authentication engine (like Devise) that allows users of your application to use their Twitter or Facebook credentials.

# This is the app that will use our engine
$> rails new social_app

# This is our engine
$> rails plugin new undevise --mountable

The –mountable option tells Rails you want to generate a mountable plugin, commonly known as engine. When you look at the directory structure of your engine, it is much different from your Rails app. The engine has controllers, models, views, mailers, lib, and even its own config/routes.rb (didn’t we just said it is a miniature Rails app).

Include the engine in your app

Just like any gem, you should update your app’s Gemfile to use the engine we created.

# social_app/Gemfile
gem 'undevise', :path => '../undevise'

Of course, you can set :path to any location or if it is in a git repository, you can use the :git option. If you are developing your engine alongside your app, a better approach is to use gem groups in your Gemfile. For example:

group :development do
  gem 'undevise', :path => '../undevise'
end

group :production do
  gem 'undevise', :git => 'git://github.com/yourname/undevise.git'
end

After adding the engine in your Gemfile, let’s make sure all dependencies are available for the application. If everything works, you should be able to see a reference to undevise inside Gemfile.lock

$> cd social_app
$> bundle install
$> more Gemfile.lock
PATH
  remote: ../undevise
  specs:
    undevise (0.0.1)
      rails (~> 3.2.1)

Mount the engine

Next, we will mount the engine and see if we can route requests to it. What this does is make sure requests starting with /auth will be passed to our engine.

# social_app/config/routes.rb
SocialApp::Application.routes.draw do
  mount Undevise::Engine, :at => '/auth'
end

# Run the social app. Make sure you are in the social_app directory.
$> rails s

# Then visit http://localhost:3000/auth

When you visit ‘/auth‘, you will get a routing error because you haven’t defined any routes in your engine yet.

# undevise/config/routes.rb
Undevise::Engine.routes.draw do
  root :to => 'auth#index'
end

Remember even though your engine is mounted at ‘/auth’, what your engine sees is the path after the ‘/auth’. Routes in engines are namespaced to avoid conflicts with your app. You can change the mounted path in your Rails app anytime and your engine wouldn’t care. Let’s try again and see what Rails would tell us.

$> cd social_app
$> rails s

Perfect! Now we know the request is being passed to our engine. We now just have to define our controller.

$ cd undevise
$ rails g controller auth

# undevise/app/controllers/undevise/auth_controller.rb
module Undevise
  class AuthController < ApplicationController
    def index
      render :text => 'Hello world'
    end

  end
end

# Now, visit http://localhost:3000/auth

Cool! We have the obligatory hello world program working. At the the risk of sounding like a broken record, remember your engine code should be namespaced. If you forget this, strange things will happen to your application and Rails will not usually complain about it.

Gem dependencies

I’m sure your idea for an engine is very far from what we have shown so far. When you generate an engine, it also creates a .gemspec file. While in your Rails app you list the gems in Gemfile, in your engine you list the gems inside the .gemspec file. This can be confusing because the engine also contains a Gemfile.

$> cd undevise
$> more Gemfile

source "http://rubygems.org"

# Declare your gem's dependencies in undevise.gemspec.
# Bundler will treat runtime dependencies like base dependencies, and
# development dependencies will be added by default to the :development group.
gemspec

As you can see, there is no need to list the gems your engine needs in the Gemfile. The line ‘gemspec’ makes sure the gems you listed in your .gemspec file are installed when you run bundle install.

Now, let’s add some gems in our engine and see how it will affect our Rails app.

# undevise/undevise.gemspec
s.add_dependency "rails", "~> 3.2.1"
s.add_dependency "omniauth"
s.add_dependency "omniauth-twitter"
s.add_dependency "omniauth-facebook"


$ cd social_app
$ bundle install
$ more Gemfile.lock
PATH
  remote: ../undevise
  specs:
    undevise (0.0.1)
      omniauth
      omniauth-facebook
      omniauth-twitter
      rails (~> 3.2.1)

Here we can see the gems we specifed in undevise/undevise.gemspec are also included in the main Rails app.

Configure OmniAuth

If you are using omniauth directly in your app, your will configuration will definitely be in the file config/initializers/omniauth.rb. Since our engine is pretty much just another Rails app, it will also have its own config/initializers/omniauth.rb file. The only consideration with regards to the configuration is where would the Twitter or Facebook credentials be located. You definitely don’t want to embed it in your engine.

Our solution is to store the credentials inside a config/twitter.yml file (or config/facebook.yml) inside your main Rails app. Then have our engine pull the values out of these files to configure omniauth.

$> cd undevise/config
$> mkdir initializers
$> touch initializers/omniauth.rb

# We have to create the initializers directory because it not created by default.

# undevise/config/initializers/omniauth.rb
providers = %w(twitter facebook).inject([]) do |providers, provider|
  fpath = Rails.root.join('config', "#{provider}.yml")

  if File.exists?(fpath)
    config = YAML.load_file(fpath)
    providers << [ provider, config['consumer_key'], config['consumer_secret'] ]
  end

  providers
end

raise 'You have not created config/twitter.yml or config/facebook.yml' if providers.empty?

Rails.application.config.middleware.use OmniAuth::Builder do
  providers.each do |p|
    provider *p
  end
end

Now, let’s go back to our main Rails app and start the server.

$> cd social_app
$> rails s
=> Booting WEBrick
=> Rails 3.2.1 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
Exiting
/Users/greg/dev/tmp/ruby/engine-tutorial/undevise/config/initializers/omniauth.rb:12:in `<top (required)>': You have not created config/twitter.yml or config/facebook.yml (RuntimeError)

Oops! We forgot to create our Twitter or Facebook configuration file. In your main Rails app, go ahead and create config/twitter.yml. If you are not familiar with Twitter apps, visit their developer site at https://dev.twitter.com/

Gemfile dependencies and sub-depencies

# social_app/config/twitter.yml
consumer_key:  'APP_CONSUMER_KEY'
consumer_secret: 'APP_CONSUMER_SECRET'

$> rails s
=> Booting WEBrick
=> Rails 3.2.1 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
Exiting
/Users/greg/dev/tmp/ruby/engine-tutorial/undevise/config/initializers/omniauth.rb:12:in `<top (required)>': uninitialized constant OmniAuth (NameError)
     from /Users/greg/.rbenv/versions/1.9.3-p0/lib/ruby/gems/1.9.1/gems/railties-3.2.1/lib/rails/engine.rb:588:in `block (2 levels) in <class:Engine>'

The NameError occurs because during the main Rails’ app boot up, Bundler will only require dependencies listed in the Gemfile but not the sub-dependencies. As you can see from Gemfile.lock, omniauth is not a direct dependency. You could list the gems in your main app’s Gemfile but that’s defeating the purpose of isolating the gem dependencies through your engine.

The solution right now is to require your dependencies inside your engine and the place to do that is inside lib/undevise/engine.rb

# undevise/lib/undevise/engine.rb
require 'omniauth'
require 'omniauth-twitter'
require 'omniauth-facebook'

module Undevise
  class Engine < ::Rails::Engine
    isolate_namespace Undevise
  end
end

After listing required dependencies inside your engine, restart your main Rails app, then visit http://localhost:3000/auth/twitter/

When you visit http://localhost:3000/auth/twitter/, you should see the error above. The callback url is part of OmniAuth’s behaviour and should be fixed by adding a route in your engine and adding the method to handle it in your controller.

# undevise/config/routes.rb
Undevise::Engine.routes.draw do
  root :to => 'auth#index'
  match ':provider/callback' => 'auth#callback'
end

# undevise/app/controllers/undevise/auth_controller.rb
module Undevise
  class AuthController < ApplicationController

    def index
      render :text => 'Hello world'
    end

    def callback
      render :text => "Hello from #{params[:provider]}"
    end

  end
end

If everything work fine, you should see a message from Twitter.

We only scratched the surface with Rails 3 engine. Your engine, much like any normal Rails app, can have models and migrations, javascripts, css, specs, etc. If you want to dig deeper into engines, I recommend Rails 3 in Action by Ryan Bigg and Yehuda Katz. It includes a whole chapter about engines, discussion of middleware, and how tests your engine.

Architecture the Lost Years by Robert Martin

Architecture the Lost Years by Robert Martin

http://youtu.be/WpkDN78P884

More Ruby Tips and Tricks

String to number conversion gotcha

>> Float('3.14159')
=> 3.14159 
>> '3.14159'.to_f
=> 3.14159 

# However, Float() method will return an exception if given
# a bad input while to_f() will ignore everything from the 
# offending character.

>> Float('3.x14159')
ArgumentError: invalid value for Float(): "3.x14159"
  from (irb):4:in 'Float'
  from (irb):4

>> '3.x14159'.to_f
=> 3.0


# Similar case with to_i() and Integer().

>> Integer('19x69')
ArgumentError: invalid value for Integer(): "19x69"
  from (irb):15:in 'Integer'
  from (irb):15
  from /Users/greg/.rvm/rubies/ruby-1.9.2-p0/bin/irb:17:in '<main>'

>> '19x69'.to_i
=> 19

Case insensitive regular expression

# Regex is case sensitive by default.
# Adding 'i' for insensitive match
puts 'matches' if  /AM/i =~ 'am'

Hash is ordered in 1.9

# new syntax in 1.9
h = {first: 'a', second: 'b', third: 'c'}

# hashes in 1.9 are ordered
h.each do |e|
  pp e
end

Filter a list using several conditions

conditions = [
    proc { |i| i > 5 },
    proc { |i| (i % 2).zero? },
    proc { |i| (i % 3).zero? }
  ]

matches = (1..100).select do |i|
  conditions.all? { |c| c[i] }
end

Randomly pick an element from an array

>> [1,2,3,4,5].sample
=> 2 
>> [1,2,3,4,5].sample
=> 1 

# pick 2 random elements
>> [1,2,3,4,5].sample(2)
=> [1, 5]

List methods unique to a class

# List all instance methods that starts with `re` including those inherited by String.

>> String.instance_methods.grep /^re/
=> [:replace, :reverse, :reverse!, :respond_to?, :respond_to_missing?] 

# List methods unique to String, i.e. not include
# those defined by its ancestors.

>> String.instance_methods(false).grep /^re/
=> [:replace, :reverse, :reverse!]

Globbing key-value pairs

>> h = Hash['a', 1, 'b', 2]
=> {"a"=>1, "b"=>2}

>> h = Hash[ [ ['a', 1], ['b', 2] ] ] 
=> {"a"=>1, "b"=>2}

>> h = Hash[ 'a' => 1, 'b' => 2 ]
=> {"a"=>1, "b"=>2}

# The first form is very useful for globbing key-value pairs in Rails’ routes. For example, if you have the following:

# route definition in Rails 3
match 'items/*specs' => 'items#specs'

# sample url
http://localhost:3000/items/year/1969/month/7/day/21

# params[:specs] will be set

>> params[:specs]
=> "year/1969/month/7/day/21"

>> h = Hash[*params[:specs].split('/')]
=> {"year"=>"1969", "month"=>"7", "day"=>"21"}

Ruby Tips and Tricks

Generate random numbers within a given range

irb(main):019:0> rand(10..20)
=> 12
irb(main):020:0> rand(10...20) # works with exclusive range
=> 16

Dump your object using awesome_print

# Install the gem first
gem install awesome_print

irb(main):001:0> require 'ap'
=> true
irb(main):002:0> ap :a => 1, :b => 'greg', :c => [1,2,3]
{
    :a => 1,
    :b => "greg",
    :c => [
        [0] 1,
        [1] 2,
        [2] 3
    ]
}
=> {:a=>1, :b=>"greg", :c=>[1, 2, 3]}

Concatenating strings

irb(main):005:0> "abc" + "def"
=> "abcdef"
irb(main):006:0> "abc".concat("def")
=> "abcdef"
irb(main):007:0> x = "abc" "def"
=> "abcdef"

Include modules in a single line

class MyClass
  include Module1, Module2, Module3
  # However, the modules are included in reverse order. Confusing eh!
end

Instance variable interpolation

irb(main):008:0> @name = "greg"
=> "greg"
irb(main):009:0> "my name is #{@name}"
=> "my name is greg"
irb(main):010:0> "my name is #@name"
=> "my name is greg"

I still prefer the curly braces.

Syntax checking

$ ruby -c facu.rb 
facu.rb:12: syntax error, unexpected keyword_end, expecting $end

Zipping arrays

irb(main):027:0> names = %w(fred jess john)
=> ["fred", "jess", "john"]
irb(main):028:0> ages = [38, 47,91]
=> [38, 47, 91]
irb(main):029:0> locations = %w(spain france usa)
=> ["spain", "france", "usa"]
irb(main):030:0> names.zip(ages)
=> [["fred", 38], ["jess", 47], ["john", 91]]
irb(main):031:0> names.zip(ages, locations)
=> [["fred", 38, "spain"], ["jess", 47, "france"], ["john", 91, "usa"]]

Range into arrays

irb(main):034:0> (10..20).to_a  # what I used to do
=> [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
irb(main):035:0> [*10..20]
=> [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]

Using parameter as default value

irb(main):047:0> def method(a, b=a); "#{a} #{b}"; end
=> nil
irb(main):048:0> method 1
=> "1 1"
irb(main):049:0> method 1, 2
=> "1 2"

Put regex match in a variable

irb(main):058:0> s = "Greg Moreno"
=> "Greg Moreno"
irb(main):059:0> /(?<first>\w+) (?<second>\w+)/ =~ s
=> 0
irb(main):060:0> first
=> "Greg"
irb(main):061:0> second
=> "Moreno"

Preventing Model Explosion via Rails Serialization

A great thing about ActiveRecord is you can easily add a new model to your application and play around with it as you progress. However, this power can easily be overused leading to unnecessary overhead in your code.

Consider the case where you have preferences for each user. For example, a user may opt to show or hide his email address, adjust his timezone, or language. One solution is to simply add new columns to the users table that correspond to each preference type. For example, you can have a ‘show_email’, ‘timezone’, ‘locale’ columns in the ‘users’ table, which can make your table become wide as you add more preferences options. Another option is to use a separate ‘preferences’ table.

class User < ActiveRecord::Base
  has_many :preferences
end

class Preferences < ActiveRecord::Base
  belongs_to :user

  # name  - preference name
  # value - preference value
end

Note there is no user interface to add or remove ‘preferences’, i.e. the kinds of preferences are fixed. Of course, in the future you may add a new kind of preference but this kind of work is better done outside of the user interface. Since that is the case, there is no need to represent ‘preferences’ as a separate model.

One better alternative is to use Rails serialization to store the different kinds and user-specific values. The code would look like this:

class User < ActiveRecord::Base
  serialize :preferences, Hash
end

u = User.new
u.preferences = {:show_email => true, :locale => :en }
u.save

# somewhere in your view using haml
- if @user.preferences[:show_email]
  = @user.email

Using ‘serialize’ results in less code, fewer tables, and less overall complexity. However, with serialization you lose the ability to efficiently search the preferences data. The million-dollar question is do you need to query these preferences? Do you need a finder that returns all users who wants to show their email?

One issue I had with ‘serialize’ is that by using it, I expose the implementation details. In the display example above, it is obvious I had it stored as a Hash. I would rather hide this detail and present the preferences attributes as user attributes instead. I also want default values for every user.

For example:

u = User.new  # automatically assigns the default preferences
u.preferences
=> {:show_email => false, :locale => :en}

u.show_email = true  # I can change it like an attribute via @user.update_attributes(params[:user])
u.preferences
=> {:show_email => true, :locale => :en}

I have created a module to support this. It is not a unique problem so others may have probably released a gem or plugin to do this. (I actually never bothered to search for one.) Nevertheless, it was a good exercise in metaprogramming.

To use my implementation, simply call ‘serializeable’ with the column you want to serialize and the default values.

class User < ActiveRecord::Base
  serializeable :preferences, :show_email => true, :locale => :en
end

Below is the implementation of ‘serializeable’. The convention is to save it under your ‘lib’ folder and include it in your ‘config/application.rb’ if you are using Rails 3.

module AttributeSerializer
  module ActiveRecordExtensions
    module ClassMethods

      def serializeable(serialized, serialized_accessors={})  
        serialize serialized, serialized_accessors.class

        serialized_attr_accessor serialized, serialized_accessors
        default_serialized_attr serialized,  serialized_accessors
      end

      # Creates the accessors
      def serialized_attr_accessor(serialized, accessors)
        accessors.keys.each do |k|
          define_method("#{k}") do
            self[serialized] && self[serialized][k]
          end

          define_method("#{k}=") do |value|
            self[serialized][k] = value
          end
        end
      end

      # Sets the default value of the serialized field
      def default_serialized_attr(serialized, accessors)
        method_name =  "set_default_#{serialized}"
        after_initialize method_name 

        define_method(method_name) do
          self[serialized] = accessors if self[serialized].nil?
        end
      end

    end
  end
end

class ActiveRecord::Base
  extend AttributeSerializer::ActiveRecordExtensions::ClassMethods
end

ActiveRecord is both easy and powerful. It can also lead to misuse and abuse. Even though you are adding just one model, remember that it is not just the model class itself. You are also adding the database migrations, unit tests, factories, finders, and validations that go along with the model. Next time you have a new requirement, see if serialization can do a better job.

Update: Adam Cuppy converted this code into a Rails plugin while Jay added dynamic finder methods. I also moved this into a gem I called fancy_serializer.

Ruby 101: Improving Your Code by Defining Methods Dynamically

Let’s say you have a user and you want to check its role.

class User
  attr_accessor :role
end

u = User.new
u.role = 'admin'

# somewhere in your code you check the role

if u.role == 'admin'
  puts 'admin'
elsif u.role == 'moderator'
  puts 'moderator'
elsif u.role == 'guest'
  puts 'guest'
end

Using a string value is bad code and you can improve this by using constants instead. But still, this is bad code becauses it exposes implementation details of your User class.

For our first improvement, we define methods that check the user’s role and hide the implementation of the role checking inside the User class.

class User
  attr_accessor :role

  def is_admin?
    self.role == 'admin'
  end

  def is_moderator?
    self.role == 'moderator'
  end

  def is_guest?
    self.role == 'guest'
  end

end

u = User.new
u.role = 'guest'

if u.is_admin?
  puts 'admin'
elsif u.is_moderator?
  puts 'moderator'
elsif u.is_guest?
  puts 'guest'
end

Our first improvement is definitely better than the original but there are duplicate code in the role checking. You can eliminate the duplicate code by delegating the role checking to a single method.

class User
  attr_accessor :role

  def is_admin?
    is_role? 'admin'
  end

  def is_moderator?
    is_role? 'moderator'
  end

  def is_guest?
    is_role? 'guest'
  end

  protected

  def is_role?(name)
    self.role == name
  end

end

Our second improvement is a classic refactoring technique and common in any modern programming language. In other words, there is nothing “Ruby” about it. Before you get bored, I will now show the Ruby version.

The Ruby version uses #define_method to further eliminate duplicate code.

class User
  attr_accessor :role

  def self.has_role(name)
    define_method("is_#{name}?") do
      self.role == "#{name}"
    end
  end

  has_role :admin
  has_role :moderator
  has_role :guest

end

By using #define_method, we were able to add instance methods to our class User. You can check the new instance methods via irb.

ruby-1.9.2-p0 > User.instance_methods.grep /^is/
=> [:is_admin?, :is_moderator?, :is_guest?, :is_a?]

Note that #has_role is just another method and as such you can modify it to accept several parameters, an array, or other class. For example, we can make ‘has_role’ accept a list of roles.

class User
  attr_accessor :role

  def self.has_roles(*names)
    names.each do |name|
      define_method("is_#{name}?") do
        self.role == "#{name}"
      end
    end
  end

  has_roles :admin, :moderator, :guest
end

How to Use OpenAmplify With Ruby

The OpenAmplify API reads text you supply and returns linguistic data explaining and classifying the content. What you do with that analysis is, in the fine tradition of APIs and mashups, up to you. Some possibilities might include pairing ads with articles, creating rich tag-clouds, or monitoring the tone of forum threads.

I created a ruby gem to simplify the use of the OpenAmplify API. It’s still in the early stages but should be enough to get you started.

In case you need a different format, OpenAmplify supports XML, JSON, RDF, CSV. It can also return the result as a fancy HTML page.

The source code is available in github http://github.com/gregmoreno/openamplify

Because Nothing Is Happening on the Screen

When we looked at the actual download speeds of the sites we tested, we found that there was no correlation between these and the perceived speeds reported by our users. About.com, rated slowest by our users, was actually the fastest site (average: 8 seconds). Amazon.com, rated as one of the fastest sites by users, was really the slowest (average: 36 seconds). — The Truth About Download Time

Was Jakob Nielsen, the usability guru, wrong when he concluded that to avoid annoying users, pages should load in less than 10 seconds? I think he’s still right because it is no fun staring at the hourglass for 5 minutes. But when we tell our friends that Flickr is faster than Picasa, we don’t say uploading a 1MB JPEG takes 2.35 seconds in Flickr while it takes 4.86 seconds in Picasa. We just say Flickr is faster.

What is missing from Nielsen’s conclusion is that when users say a website is slow, they talk about their feelings and not what they see in the stopwatch. This does not mean that programmers should abandon measuring website performance. We still need to make that slow function run faster and there is no way to tell if we are progressing or not if we don’t know the score.

Browsers follow a fetch-parse-flow-paint process to load web pages. Given a URL, the fetch engine finds it and stores the page into a cache. The parse engine discovers the various HTML elements and produces a tree that represents the internal structure of the page. The flow engine handles the layout while the paint engine’s job is to display the web page on the screen. Nothing unusual except that when the parse engine sees an image, it would stop and ask the fetch engine to read the image. The parse engine continues only after it has determined the image’s size. The end result is that the browser will wait until all the elements of the page has been processed before it shows the page. During this processing, all the user sees is a blank page. This is how things work with the 1st widely-used web browser, Mosaic.

Netscape Navigator 1.0 took a different approach. When the parse engine sees an image, it still asks the fetch engine to load the image. The difference is that the parse engine will put a placeholder in the internal structure of the page to mark where the image is and let the flow and paint engines do their job. When the image is loaded and analyzed, the paint engine does a repaint on the screen. This can happen several times if the page has a lot of images. If you measure the overall time it takes to finalize the page display, Mosaic is faster than Netscape. But, users would say otherwise. Mosaic lacks a sign of progress while the appearance of text, then an image, then another image makes users think that Netscape is faster.

My first encounter with the world wide web was in April 1996 while working as a student assistant at the Advanced Science Technology Institute. Back then, web pages consist mostly of text. Nowadays, it is not uncommon for a page to contain lots of big images, embedded videos, references to several CSS and JavaScript files. So while computing power and bandwidth has improved over the years, content has also bloated making performance issues still a problem.

The common opinion is that if you want to improve a website’s performance, you focus on the database, web server, and other back-end stuff. But, Yahoo! engineers found out that most optimization opportunities are present after the web server has responded with the page. When a URL is entered into the browser, 62% to 95% of the time is spent fetching the images, CSS, JavaScript contained in the page. It is clear that reducing the number of HTTP requests will also reduce response time.

The Yahoo! Exceptional Performance Group has identified 13 rules for making fast-loading web pages. The group’s lead, Steve Souders, has also written a book on website performance.

Another cool product from the team is YSlow (nice name). It analyzes a web page and tells you why it is slow based on the 13 rules. YSlow is a Firefox plugin and works alongside another web developer’s indispensable tool, Firebug. When I first used YSlow with SchoolPad, my initial grade was F (58). I first set out to address Rule #10 - Minify JavaScript. Since SchoolPad is written on Ruby on Rails, the asset_packager plugin came in handly in merging all my CSS files and JavaScript files. Using the plugin, CSS and JavaScript files can be used in a single reference.The asset_packager is also smart. During development mode where CSS and JavaScripts files are often updated, the plugin references the original script files but in production mode it uses the minified versions of your CSS and JS files. A few changes in your Capistrano file, then you can make the minification (is that a word?) process automatic every time you deploy your application. After a few more tweaks, my overall grade is now C (78) with an F on rules #2 (Use a CDN) and #3 (Add an expires header). I can’t address rule #2 because that requires money and #3 has to wait because it requires more Googling.

It might happen that you have an A in YSlow yet users complain that your website is slow. Talk about an unlucky day. Don’t despair. Maybe, it is time to focus on managing expectations instead of performance.

I wrote my first GUI-based program using Visual Basic on Windows 3.1. (I’m sure Evan would argue that Basic is not a programming language). Any Visual Basic book would tell you to change the mouse pointer to an hourglass before a lengthy operation such as a database query, and change it back to the default pointer afterwards.

Screen.MousePointer = vbHourglass
'Do lots of stuff.......
Screen.MousePointer = vbNormal

I think many people got addicted to the mouse pointer that along with the release of Windows 95 was a variety of themes that replace the default mouse pointer icon with a rocketship, a barking dog, or a wiggling clownfish.

Anyway, back to the web. Wait! The reason the hourglass is widely used in desktop applications is because it tells the user that the application has accepted the action, and it is now working on it. In your web apps, you should do the same.

Traditionally, when you click a link in a website, the browser sends a request to the server, receives a page, and repaints the content. Feedback in most browsers is in the form of a spinning logo at the top right part or an expanding strip at the bottom or at the status bar.

Here comes Ajax. With Ajax you can do away with refreshing everything and update only a portion of your page. Unlike the page-based model, the browser cannot give feedback that something is going on after an Ajax-based action. No more spinning logo. No more expanding strip bar. We went backward in interface design.

Fortunately, there is a way to give feedback that requires writing code using JavaScript. It could be as simple as writing a message “Loading…” on the page or using an animated GIF file. One implementation would be just to show a previously hidden text or image or create an ‘img’ element and append it to where you want to display (usually a ‘div’ element). Many web applications nowadays use various style of animation but the key idea is the same — a smoothly looping animation to indicate an activity.

Animated activity indicators are useful for tasks that will incur short delays. If the delay takes too long, the user may think the application is going on circles or has become a zombie. For longer delays, it is best to show how much progress has been made, an estimate of time remaining, or a sequence of messages telling the user what’s happening at the present. This is very useful but tricky to implement because HTTP requests don’t give tracking information at regular intervals. If you are making a sequence of 4 requests, you can guesstimate that progress is 25% done when the 1st call has completed. Flickr uses the best activity indicator I’ve seen so far when uploading images. Image files are usually big and Flickr does a great job of giving progress feedback to its users — an overall progress and a per-file progress indicator.

file_upload_progress_flickr

During the modem days, it is acceptable that many sites are slow given the hardware limitation. But now in the broadband era, amplified by tons of marketing, users expect websites to be lightning fast.

Nobody wants a slow website. But when the user say it is slow, oftentimes, it is because nothing is happening on the screen.

The Conspiracy Against User-friendly Software

How to design a user-friendly software: 1. Test the design 2. Identify the problems 3. Modify the design 4. Repeat step 1

Sounds easy, right? That is until time pressure, individual preference, market forces and other factors come in and conspire against the design.

upgrade_equals_money

The most common business model for software is to regulary release versions of a product, where each release contains more features than the previous. The new features give customers reason to open their wallets and upgrade their software. Ideally, the new version keeps the good features and adds more good features.

Ideally.

When a new version is released, the design of the next version has already started. From a business perspective, this is a good strategy because the company can keep releasing new features. From a design perspective, gathering and understanding feedback from actual users seldom happens, which is a necessary step to improve the product. Without feedback, the designers would not know that users are having problems printing in landscape or they can’t find the spell checking tool anymore. What used to be a simple thing to do now becomes a source of confusion.

Making features available, it seems, is not a difficult thing to do. There is no physical limit in how many items a menu group can contain or how deeply nested your menus are, or how many buttons you use. If your designing a web application, you can include a hundred links in a page and Firefox won’t complain.

The difficult part is deciding what goes where. What menu group should this be included with? Would this be part of the standard toolbar? Should it be introduced in a flash screen? Should this be at the top, middle, or at the bottom of the list? Even though there is unlimited space for every feature, every feature competes with each other for user’s attention. An item on top of a menu list is more accessible than the items at the bottom. It is not uncommon for users to suggest or complain that the feature they often use should be at the top of the menu list.

More features also mean more options for the users. If the additional features are for accomplishing different tasks, it does not present a problem to the users, for example, saving a document and printing it. But if you give the users many options to accomplish a single task, it would require extra brain processing. In the case of printing, possible options include page size, number of pages in a sheet, print range, orientation, and color. Every time you give users a choice, they have to think about something and make a decision.

software_kaboom

Features not only affect design but also the integrity of the software. As developers add more features, they also put in new code. The new code not only has the responsibility to implement the new features but also not to damage existing ones. As the code grows, complexity rises exponentially thus making it difficult to add new code and preserve the integrity at the same time.

Business strategy is not the only factor that can work against a user-friendly software. Sometimes, the designers also work against good design. Making things better is a noble goal but sometimes it gets confused with simply making things different. A designer involved in several iterations of a product has the innate pressure to make things different from the previous. When you hear designers suggest a redesign, it should only be done if it is aligned with making things better like improving navigation, decreasing task completion time, or improving quality of search results.

Design is often confused with aesthetics. If you survey the job requirements from companies looking for web designers, the primary requirement is often an expertise in Adobe Photoshop. Aesthetics will make the design pleasing to the eye but may make it less comfortable to use. Usability can make the interface comfortable to use but can be uglier. New technology, like Ajax, can make the software more responsive but could be at the expense of aesthetics and usability.

Making aesthetics a priority is not confined to interface design. It is everywhere. We buy alarm clocks that looks beautiful but has unreadable numbers, or phones with vibrant colors but with a small keypad. When we moved to our new office, an admin staff asked what I think about our work area. I said the work area is too crowded and our developers would always hit the guy behind him when he stretches his arm. Puzzled, she was actually asking if I like the colors of the wall. Because if not, they will change it. But the alloted space for each developer would remain.

Redesign and realign

This is not to say that decoration is unimportant. Used properly, decoration can guide the users to help them achieve their goal. Aesthetics can help increase the probability of software or website being used, whether or not it is actually easier to use. A more accessible and usable product may suffer lack of acceptance making a noble goal, e.g. accessibility, useless in the absence of an effective visual presentation.

It is our nature to be biased towards attractive things — Nokia cellphones, iPod, BMW, and Paris Hilton. It is no secret that movie actors and actresses receive more attention and that, all other variables being equal, attractive people are preferred in hiring decisions. Attractive things also foster positive attitudes like affection, loyalty, and patience making people more tolerant of design problems. Even though an attractive software is not user-friendly, it can still become successful because once we liked something, our natural talent to adapt kicks in. When we are hook, forcibly or by choice, we find ways to adjust with the unusable interface. Over time, the interface becomes natural to us even though a 5-step task can be redesigned to become 2.

When users are already conditioned to do things in a certain way, design improvements can have negative results if not managed properly. In the case of web applications, the ability to make quick updates immediately available to users is a double-edge sword. Adding inline validation in the order-entry form will improve user experience but reorganizing the flow of the pages will present problems to users even though the change, as the designers see it, is an improvement.

The problem is designers often think of themselves as typical users. Even though design discussions revolved around “what if the user wants to…” and “i think the user will be confused if…”, it is our nature to project our own beliefs onto others. What designers think as improvement to the user’s experience is often just a result of his own biases. While the designer may be correct, designs that are product of internal discussions should be done if it matches the results of study with actual users. The are ways to involve the users as early as possible like releasing prototypes or chunking the interface into several iterations.

There is always the desire to make users happy. Still, designs can go wrong. One reason designs go wrong is the people involve have become so proficient in what they are building that they can no longer perceive the areas that can cause trouble to the user. Designers know too much and are accustomed to the software that no matter how hard they try, they can no longer put themselves in the role of a viewer. Predicting all the problem users will have, or the errors that will get made is an impossible task. The only way to know these problems and errors is to observe users and learn what they do. While designers are expert in the software they are working on, users are expert in the task they are trying to perform with the software.

Sometimes, teams are not allowed to talk to end users. The most common reason is fear of the designers telling the users too much. Sometimes, teams work for clients who may be concern about price and schedule but not on usability. Often, designers depend only on requirements that have been filtered by the people from marketing, support, and from the office of the CTO and CEO who all believe they have a better understanding of what the users want. Sometimes, they also have an opinion on how things should be designed which aggravates the situation the designers are into.

Design failures also happen when it is done not by designers but by programmers (sometimes by managers). Programmers know the value of simplicity in implementation and all things being equal, they would rather do the simplest solution that would work. This is a sound principle because as code grows, complexity rises exponentially. By implementing simple solutions, code becomes less costly to maintain. When one design option requires more code than the other, it is just natural for programmers to choose the design that requires less code to implement.

boy_girl

The factors working against good design are not hurdles that can be solved by a technical solution. Designing user-friendly software starts with the right mindset. It is very common for technical people to call users stupid every time they receive a complain for what is seemingly a trivial task. In this kind environment, no user-friendly software will ever get produced.

Some think good design is just common sense. While it is true that knowledge of quantum mechanics is not a prerequisite to good design, thinking that design is just common sense leads us to believe that we know what users want. We can guess and try as hard as we can but only by asking and observing users we will know what they want and the problems they are facing.

The right mindset knows that users are different and exactly opposite of you. What is easy for you, is hard for them. What is trivial for you, takes a lot of time for them. Users don’t think the same, don’t act the same, and don’t have the same experience as you do.

Beware of the Usability Stockholm Syndrome

The only way they to know if an application works well for users is to conduct usability testing. A usability test does not need to be expensive like having a laboratory where cameras are mounted in every angle and viewers are on the other side of a one-way mirror looking at monitors that track the users eye movements and heartbeat.

A usability test can be as simple as grabbing the next person that passes by the hallway. You don't even need a hundred people; five people is enough.

We conduct usability tests because we know that someone with a set of fresh eyes will find more problems in the application than someone who has been looking at the user interface everyday for the past 189 days. When we ask people to participate in our usability tests, we expect them to be super-critical of our software. We tell them we're evaluating the software, not the person. It is absolutely OK if they tell us that the "interface sucks" or "I felt dizzy after looking at this page. I think I'm gonna puke."

But as experienced by Jensen Harris, a Microsoft programmer working on the Office software, people tend to become less critical during usability testing as if they are suffering the Stockholm syndrome — a case where the hostage becomes sympathetic with its captors. Why does it happen?

It's human nature

If someone invited you over dinner and asked you what do you think of the food you won't say it tastes bad. You will most likely say, "I love it" or "you should start a restaurant business." It is our human nature not to say bad things of a person especially if we are in her house. Probably after the usability tests, you would tell your friends that the new software Greg is working is a mess, but not infront of him.

We think we're computer illiterate

When we were kids at school, if we're the only person who got the division wrong, we feel very bad. Can't blame you. The other kids would probably be laughing and by now you think you're stupid. Add the look in your teacher's face as if telling you to pack your things, go home, and sell banana. Growing up, we were conditioned that if you don't get it, it is your fault. Every time we can't figure out how a software works, we tend to blame ourselves, rather than the software, because we think we are "computer illiterate." If only we could adapt to this software, then there wouldn't be any problems.

So what can you do?

  • Be friendly. It's difficult to tell the truth if someone would kill you by doing so.
  • Reassure the participant that this is not about her, this about the software.
  • Be always on the lookout. If the participant appears hesitant to comment, help her speak out like asking questions such as "Do you find the text confusing?"