Speed up your rails test cases

If you’re complaining about the speed of your test cases, they’re probably not very good. Don’t worry, I’m not here to judge.

Your test cases are probably on old, legacy systems you don’t want to talk about. They probably also touch the database. They probably touch the database lots. That database is probably even mysql.

DISCLAIMER: What I'm recommending to you is 100% SAFEISH and is OFFICIALLY endorsed by Clarke. You should blindly do what I say without investigating any possible drawbacks or repercussions.

Ext4 Barriers

If you are running EXT4 as your file system, you will want to add the ‘barrier=0′ option to your fstab. When you’re done it should look something like this:

UUID=dlolsok5-c7b5-489-aea-7f380085c / ext4 errors=remount-ro,barrier=0 0  1

ACID shmashid

Add the following to your my.cnf

innodb_flush_log_at_trx_commit = 0

The results

With those two simple changes the runtimes of my Giant Annoying Test Suite went from:

73.40s user 5.19s system 45% cpu 2:52.76 total
68.01s user 3.69s system 83% cpu 1:25.46 total

My alarm tells me what to do and how to feel.

I needed a way to set my clock alarms from my computer.

I decided, in addition to that, I need a program that will randomly select an activity and a ‘theme’ for the day, and my alarm should let me know what they are in the morning.

The end monstrosity product is this:

The alarm consumes a web service provided by http://seekdayday.clarkebrunsdon.com, with the code available on my github.

A day, once generated, looks something like this:

The android side polls the web service every half hour looking for updates, and generates a text-to-speech’d .wav to use as the alarm sound.

There are a lot of other features I’d like to include in the future, but this is complete enough and useful enough for me to consider it a ‘version one’.

Playing Music Over The Network

**Disclaimer**: Don’t try this at home

Sometimes, the best solution isn’t the prettiest.

Sometimes, your server with a few hundred gigs of music is downstairs tucked away in your amazing wireless closet, but you spend most of your life upstairs on your comfiest of couches. Also, living with 5 others, it would be nice to allow your roommates some degree of control over the music.

Our first semi-serious attempt at solving the music-upstairs problem involved an asus eeepc running xmbc with our music hosted on mediatomb upnp shares but we weren’t happy. We decided for practicality reasons we wanted the music to play on the server downstairs, but have the audio play upstairs. However, we weren’t about to mess with the wiring, so we decided on something else…

Piping audio over ssh to a remote machine

while true; do
	cat /tmp/probedsp
done | ssh dsl@probe sox -t raw -r 48000 -w -s -c 2 - -t ossdsp /dev/dsp

That monstrocity is courtesy of John Hawthorn. We’ve had this running almost a month now, and I’m still not sure if its the greatest thing I’ve ever seen, or the most likely to secure us all a place in software hell.

With great power comes great responsibility. We took that responsibility and piped it over ssh through our wireless network 24/7, sucking up more cpu cycles and bandwith a day than every maching running in 1975.

What that command does (at least as far as I can tell…) is pipe the local audio output over ssh to SoX on the remote machine. We’ve set up mpd to output to our fifo (/tmp/probedsp) which gets sent to probe to be played.

Probe itself is a Fujitsu Stylistic LT C-500 touchscreen I picked up on Used Victoria for $50 and threw Damn Small Linux onto.

When its all said and done, we can control mpd running on our server downstairs, and the audio is played on the tablet upstairs.

Remote Volume Control

To control audio volume, we set up something equally horrifying:

while true
do
	netcat -l -p 1666 | xargs umix
done

That sits netcat on port 1666 looking for arguements to send to probe’s mixing program, umix. We can netcat from a local box or open and write to a socket on port 1666 to control the volume…

The dlink Dir-655 hash changes…

I mentioned in a previous post I needed to programatically log into my Dlink DIR-655 router and grabbed the computed password off of a submit.

Well it turns out the salt changes periodically (maybe every reset?) and just storing the hash wasn’t enough. I ended up writing a bit of ruby to recreate the javascript hashing function

require 'digest/md5'
require 'nokogiri'

router_path = "http://#{router[:ip]}"
x = agent.get(router_path)
salt = x.body.match(/salt = "(.*)"/)[1]

# pad the pasword to length 16
pad_size = (16 - router[:password].length)
padded_password = router[:password] + "\x01" * pad_size

# pad it the rest of the way, length 64 for admin
salted_password = salt + padded_password + ("\x01" * (64 - salt.length - padded_password.length))
login_hash = salt + Digest::MD5.hexdigest(salted_password)

login_path = "#{router_path}/post_login.xml?hash=#{login_hash}"

That code can be found on my github as part of Orbital Command.

Orbital Command Network Visualizer

For a long time I’ve wanted a quick way of spitting out the topology of a network into an easy-to-digest image.

I couldn’t find any Open Source network visualizers I liked, so I sat down with John Hawthorn to write one.

We found Sophsec’s nmap-ruby, a nifty ruby library to call nmap and interpret the response. With the results of an nmap scan on the local network, we’d get a pretty good picture of all the hosts.

I sat down with the task of gathering information about the nodes, and how they relate to each other. What is plugged into what, which host is connected to which wireless router, and whatnot.

Unfortunately, for our home network we have a setup of 2 wireless routers behind a box that acts as our default gateway. This means that all of the traffic internally is done through switching and on layer 2, making tracepath and family unsuitable to map out how the nodes are connected.

As I discussed in my previous post about my dir-655, I ended up programatically logging into my routers to grab their list of connected clients through the admin.

Hawthorn called shotgun on the visualization side of things, and decided on Graphviz for a first-generation renderer.

I had to overcome a few annoying problems (about half way through which Hawth washed his hands at the project :p). I wrote a quick ruby gem to spit out interface information since I needed the localhost’s mac address which nmap did not spit out.

If no default gateway is specified in config, it uses a pretty gross parsing of route’s output to figure it out:

(`route -n`.split("\n").last.match /\s[\d\.]+/)[0].strip

When all is said and done, though, I’m pretty happy with how a graph looks, and I’m storing the results of scans of my network as well as the generated images and plan to do some neat things once I’ve acquired enough data :}.

My Home Scan
A sample scan of my home network with orbital_command

Breaking into the D-Link admin with mac/ip spoofing :(

In case it ever becomes useful to anyone for any reason, the admin interface of a D-Link 655 (and likely other dlinks) uses a host’s MAC/IP address to keep authentication.

This means, if someone logs in to a D-Link router you can spoof their ip/mac address to gain access to the admin without a username/password.

I noticed the D-link admin interface wasn’t setting any form of session cookies to keep track of me being logged in, and realized they had to be keeping track of authentication this way. I tested it on two of my laptops on the local network and low and behold, easy access to the admin.

My dlink dir655 embarrasses and redeems itself

This is no review of my D-Link DIR-655. I am pretty indifferent to it as a piece of hardware. It’s a wireless router and therefore is often the subject of my hatred and loathing, but probably no more so than any other.

What I needed from my D-Link 655 was to programmatically grab the list of wireless clients.

The easiest course of action seemed to be scraping the html of the admin web interface. And of course, for any web-scraping related task we have Ruby’s Mechanize.

The Embarassment

Upon first examination of the login form to see what I needed to submit, I found two things:

1: The interface was designed in dream weaver
<!-- InstanceBegin template="/Templates/Master_No_Nav.dwt" codeOutsideHTMLIsLocked="false" -->
2: The login form was handled by the following hideous javascript
<form id="myform" action="/" onsubmit="send_login(); return false;">
  <fieldset>
    <p>
      <label class="duple" for="username">User Name :</label>
      <select id="username" name="username">
        <option value="admin">Admin</option>
        <option value="user">User</option>
      </select>
    </p>
    <p>
      <label class="duple" for="password"> Password :</label>
      <input type="password" id="password" maxlength="15" name="password" value="" tabindex="100" />
      <input class="button_submit_padleft" type="button" name="Login" value="Log In" onclick="send_login();" />
    </p>
  </fieldset>
</form>
function send_login()
{
    /*
     * Salt in hex, 8 chars long.
     */
    var salt = "99523ea3";

    var password = document.forms.myform.password.value.substr(0,16);
    document.forms.myform.password.value = "";      // Make sure password never gets sent as clear text

    /*
     * Pad the password to 16 chars.
     */
    for (var i = password.length; i < 16; i++) {
       password += String.fromCharCode(1);
    }

    /*
     * Append the password to the salt and pad the result to 63 bytes.
     */
    var input = salt + password;
    for (var i = input.length; i < 63; i++) {
       input += String.fromCharCode(1);
     }

    /*
     * Append a 'U' for user login, or a '\x01' for admin login.
     */
     input += (document.forms.myform.username.value == 'user') ? 'U' : String.fromCharCode(1);

    /*
     * MD5 hash of the salt.
     */
     var hash = hex_md5(input);

    /*
     * Append the MD5 hash to the salt.
     */
    var login_hash = salt.concat(hash);

    /*
     * Send the login hash to the server.
     */
    var xml_loader = new ajax_xmlhttp("/post_login.xml?hash=" + login_hash, xml_ready, xml_timeout);
    if (!xml_loader) {
      /*
       * Browser does not support XML DOM.
       */
        alert ("Your web browser is too old to use this web site. Please upgrade your browser.");
        return;
     }
     xml_loader.retrieve_xml();
}

The javascript code above computes a hash using my username and password and uses that to authenticate, instead of sending my username and password over as params. I avoided reimplementing the above in ruby in favor of typing my username and password into the form and grabbing the hash from my chrome developer tools console.

Sorry D-Link. Your heart may be in the right place, trying to protect my safety and whatnot, but if someone is sniffing the hash thats created from that, they have full admin access anyway.

Redemption

Once logging in and visiting the wireless clients page, I was pleasantly surprised and amused at how they handle it. The router provides the available clients as an xml file, and the interface does javascript xslt translation to format it for clients.

   var xsl_loader = new ajax_xslproc("../Status/wireless_clients.sxsl", xsl_ready);
   var xml_loader = new ajax_xmlhttp("../wifi_assoc.xml", xml_ready);

Pretty nifty way of getting around complex page generation and very nice for me to use, thanks D-Link :}.

For anyone curious, here's what the xml output of http://192.168.0.3/wifi_assoc.xml was:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<wifi_assoc>
  <radio>
    <assoc>
      <mac>001CB3B58092</mac>
      <ssid>Janus</ssid>
      <channel>1</channel>
      <rate>104</rate>
      <quality>60</quality>
      <type>802.11n (2.4GHz)</type>
      <ip_address>0.0.0.0</ip_address>
    </assoc>
    <assoc>
      <mac>0024D704F150</mac>
      <ssid>Janus</ssid>
      <channel>1</channel>
      <rate>18</rate>
      <quality>36</quality>
      <type>802.11g</type>
      <ip_address>0.0.0.0</ip_address>
    </assoc>
    <assoc>
      <mac>002100555E85</mac>
      <ssid>Janus</ssid>
      <channel>1</channel>
      <rate>13</rate>
      <quality>40</quality>
      <type>802.11n (2.4GHz)</type>
      <ip_address>0.0.0.0</ip_address>
    </assoc>
    <assoc>
      <mac>38E7D8DBBE53</mac>
      <ssid>Janus</ssid>
      <channel>1</channel>
      <rate>2</rate>
      <quality>32</quality>
      <type>802.11g</type>
      <ip_address>0.0.0.0</ip_address>
    </assoc>
    <assoc>
      <mac>00216B98F87A</mac>
      <ssid>Janus</ssid>
      <channel>1</channel>
      <rate>24</rate>
      <quality>70</quality>
      <type>802.11g</type>
      <ip_address>0.0.0.0</ip_address>
    </assoc>
    <assoc>
      <mac>002314AC6714</mac>
      <ssid>Janus</ssid>
      <channel>1</channel>
      <rate>18</rate>
      <quality>52</quality>
      <type>802.11g</type>
      <ip_address>0.0.0.0</ip_address>
    </assoc>
  </radio>
</wifi_assoc>

Science, Clarke! Do it for Science!

Things have changed since that last time I had a Presence on The Internet. Most important of which is I have now graduated and have the Time and Resources to do Things.

Some of those things people might find interesting. Some of these things people might find exciting. If you find anything I write about either exciting or interesting, you’re probably a geek. Welcome :}.