Monkies, Bananas and all things Deep Space

What a wonderful world… a must watch/listen imo http://www.youtube.com/watch?v=B8WHKRzkCOY

Updates on Occupy Sydney, the Powerhouse Museum, misc photos and the Christmas Bus on YankyDownUnder: http://yankydownunder.tumblr.com/

ThinkingSphinx and it’s where clause

Ran into a problem today.  rake ts:rebuild was taking FOREVER to index 500,000 posts on http://deucescracked.com

In our define_index call in our Post model we had:

 where "`forums`.forum_permissions is null or `forums`.forum_permissions = 0"

Innocent looking enough. We don’t want private forums showing up in search.

Well - that generates a very ugly where clause. It’s not a bad idea to check this stuff on dev before deploying. After doing a rake ts:rebuild take a look at your config/development.sphinx.conf file. The sql_query is what you’re looking for in the model. Ours had this as a where clause:

WHERE 
  `posts`.`id` >= $start 
  AND `posts`.`id` <= $end 
  AND `posts`.`delta` = 0 
  AND forums.forum_permissions is null or forums.forum_permissions = 0 

I’m not 100% sure what the values are for $start and $end when it indexes but, the “or” statement in that last line needs to be in parens. You’d kind of think ThinkingSphinx would do this for you (like ActiveRecord does) but, it doesn’t.

So - we changed to:

 where "(`forums`.forum_permissions is null or `forums`.forum_permissions = 0)"

And everything was right as rain.

Be careful with reference assignments

Say you have this code:

class Bob
  PET = "Dog"
  def self.say_pet
    res = PET
    res << " is my pet"
    return res
  end
end

10.times do |i| 
  puts "#{i} #{Bob.say_pet}"
end 

You’re going to get:

0 Dog is my pet
1 Dog is my pet is my pet
2 Dog is my pet is my pet is my pet
3 Dog is my pet is my pet is my pet is my pet
4 Dog is my pet is my pet is my pet is my pet is my pet
5 Dog is my pet is my pet is my pet is my pet is my pet is my pet
6 Dog is my pet is my pet is my pet is my pet is my pet is my pet is my pet
7 Dog is my pet is my pet is my pet is my pet is my pet is my pet is my pet is my pet
8 Dog is my pet is my pet is my pet is my pet is my pet is my pet is my pet is my pet is my pet
9 Dog is my pet is my pet is my pet is my pet is my pet is my pet is my pet is my pet is my pet is my pet

Probably not what you wanted. I’m not 100% sure why this is allowed (as I thought contants where static) but, it’d because of

res = PET

Change that line to

res = "#{PET}"

And you’re all set.

:default_scope and update_all don’t play nicely together

Rails 2 only

If you have 

  class Topic < ActiveRecord::Base
    default_scope :conditions => "forums.preferences > 1", :include => [:forum]
  end

and you do a 

  Topic.update_all(...)

it’ll fail with 

  Mysql::Error: Unknown column 'forums.preferences' in 'where clause'

The work around for this is:

  Topic.send(:with_exclusive_scope) { Topic.update_all(...) }

You can monkey patch this using this code (and requiring it in environment.rb or else where)

  module ActiveRecordMixins
    class ActiveRecord::Base
      def self.update_all!(*args)
        self.send(:with_exclusive_scope) { self.update_all(*args) }
      end
      def self.delete_all!(*args)
        self.send(:with_exclusive_scope) { self.delete_all(*args) }
      end
    end
  end

  end

Then just you update_all! or delete_all! when it has a default scope.

In other news - :default_scope is beyond ghetto IMO

Really Ruby…. ?? literally

name = “bob?”

puts name[-1] == ??

true

?? is seriously ??

/usr/sbin/apachectl: line 82: ulimit: open files: cannot modify limit: Invalid argument

If you get a

/usr/sbin/apachectl: line 82: ulimit: open files: cannot modify limit: Invalid argument

when you sudo apachectl restart

Simply edit /usr/sbin/apachectl and change

ULIMIT_MAX_FILES=”ulimit -S -n `ulimit -H -n`”
to
ULIMIT_MAX_FILES=”“

WillPaginate with Redis on Rails 3

Just a reminder - this is Rails 3 code - Rails 2 code would be slightly different.

I had the need to do a bit of pagination using will_paginate with Redis on http://txt141.com today.  Pretty straight forward if you dig into their Array code.

First, a brief overview of how to do paging in Redis.  Unless you want to get all keys (in a set, hset, etc.) you need to use ordered sets (zsets) to do pagination.  In this example I have a hset that has the data I want to paginate.  It’s key is data.  I need to setup a zset that has it’s keys… we’ll call the zset data-zset.  You can do a simple migration to populate it:

  REDIS.hkeys("data").each do |key|
    REDIS.zadd("data-zset", Time.now.to_i, key)
    sleep 0.01
  end

You might be wondering about the “sleep 0.01”.  Redis is so fast that if you don’t put that in all the items in your ordered set will have the exact same time… making it not-so-ordered.  

Paging through a ordered set (zset)

Now that you have an ordered set you can grab “pages” of it like so…

  page1 = REDIS.zrange "data-zset", 0, 9
  page2 = REDIS.zrange "data-zset", 10, 20
  page3 = REDIS.zrange "data-zset", 21, 31

You can find the count of data-zset by doing:

  REDIS.zcount "data-zset", "-inf", "+inf"

Putting it together in the controller

will_paginate already supports paginating arrays… but it assumes the array has all the data and not just the current page.  So we need to do a bit more work.

  def index
    @page = (params[:page] || 1).to_i
    @per_page = 10
    # REDIS' zcount is 0 based
    cnt = (@page - 1) * @per_page
    data = REDIS.zrange "data-zset", cnt, (cnt + @per_page - 1)
    data_count = REDIS.zcount "data-zset", "-inf", "+inf"
    # load the data we want from the hset
    @results = data.collect {|key| REDIS.hget("data",key)}}

    # use WillPaginate::Collection to populate the array with the info will_paginate neets
    @results = WillPaginate::Collection.create @page, @per_page, data_count do |pager|
      pager.replace @txts
    end    
  end

Then - in your view you can just do something simple like:

<% @results.each do |r| %>
	<p><%=r%></p>
<% end %>
<%= will_paginate @results %>

Just a reminder - this is Rails 3 code - Rails 2 code would be slightly different.

HDR Thoughts

I’ve been playing around with HDR lately… mostly just on my point and shoot.

Texas State Capital Dome - Sony Cyber Shot G - auto bracket 3 shots from -1 EV to +1 EV, Photomatrix Pro processing

This is done by exposure bracketing.  You set your camera to take 3 shots at -1 EV, 0 EV, and +1 EV (or, on DSLR’s you could do five shots ranging from -2 to +2).  While the photos are all taken within a second you can’t hand hold this and get good results.  You need to use a trip.

Some people have been doing time-lapse HDR

Wouldn’t be cool if you could shoot real-time video this way though?  As far as I can tell this isn’t possible yet.  The closest someone has come was achieved via beam splitting to two Canon 5D Mark II’s together.  Very impressive results. But, still, this is just 2 images.  For optimal results you need 3 or more (preferably 5 shots).  

It’d be very cool if someone would either design a chip that could simultaneously capture multiple exposures values or had multiple sensors to record to.  Kind of like the double clutch VW’s.

Looks like the impossibly priced Red video system is working on exactly this: 

http://gizmodo.com/5643554/this-is-the-first-hdr-video-shot-from-a-red-epic-camera