holy moly

pangratz prattles

IRC Log Viewer Using Hubot, CouchDB and Ember.js

I am very excited about Ember.js, an awesome JavaScript library with some neat features like Bindings, Computed Properties, Templates, … you should go check it out on GitHub.

If you need some help you can simple ask your question at the #emberjs IRC channel. There is just one problem: you cannot read through older conversations because this channel is not logged - yet. So I decided to hack something together to end this.

The result - which is hosted here http://emberjs.iriscouch.com/irc/_design/viewer/index.html - is a Hubot hanging around the IRC channel, storing the messages in a CouchDB. Additionally there is a viewer written in - you guessed it - ember.js which let you browse through the transcripts. This blog post covers the basics how I implemented all this.

CouchDB - Time to relax

CouchDB is an open source, document oriented database with a HTTP REST interface. It lets you store JSON documents and do powerful map/reduce queries on the data. I don’t own a server where I can install CouchDB so I’ve created a free account with the name emberjs at IrisCouch.

Next I created a database irc with a simple

1

curl -XPUT http://emberjs.iriscouch.com/irc

The IRC logs should be read only so I setup an administrator account and secured the database as described in this excellent blog post by Liz Conlan. So now there is a secured CouchDB instance hosted at emberjs.iriscouch.com with a database named irc which can be read by everybody and to store data you have to be an administrator.

Hubot meets Heroku

Hubot is a bot written in Node.js by sexy GitHub. It is originally written for Campfire but an adapter for IRC exists. Heroku is a hosting platform with a git interface and hence very easy to use. Compared to Google App Engine installing of Heroku is a charm by doing a gem install heroku.

I downloaded the version of Hubot which is designed for Heroku. Inside the scripts folder are some scripts which are available to Hubot. More scripts can be made available by writing one or by downloading them from the hubot-scripts repository. Luckily there is already a script originally written by Patrik Votoček which stores all messages into a CouchDB. Because the Hubot should not make too much noise inside the IRC channel all pre installed scripts are removed and only the essential ones are left inside the scripts folder: store-messages-couchdb.coffee and pugme.coffee.

The downloaded Hubot instance does not come with the ability to connect to IRC. This is easy fixed via adding the IRC adapter hubot-irc. The final dependencies section of package.json looks like:

package.json 
1
2
3
4
5
6
7

"dependencies": {
    "hubot": "2.1.3",
    "hubot-scripts": ">=2.0.4",
    "optparse": "1.0.3",
    "hubot-irc": "0.0.6",
    "cradle": "0.5.8"
}

Before Hubot is deployed to Heroku I’ve changed the content of Procfile so Heroku knows to start Hubot with IRC adapter:

Procfile 
1

app: bin/hubot -a irc

The next step is the actual deployment. First a new heroku application on the cedar stack is created and afterwards a simple git push deploys it on Heroku. It’s that easy.

1
2

heroku create emberjs-hubot --stack cedar
git push heroku master

Afterwards Hubot needs to be configured so he knows where to hang out and where to save the messages. This is simple:

1
2
3
4

heroku config:add HUBOT_IRC_NICK="emberjs-hubot"
heroku config:add HUBOT_IRC_ROOMS="#emberjs"
heroku config:add HUBOT_IRC_SERVER="irc.freenode.net"
heroku config:add HUBOT_COUCHDB_URL="http://USERNAME:[email protected]:5984/irc"

So now there is a Hubot instance running on Heroku and it’s configured to hang out in the #emberjs channel and save all messages into the previously created CouchDB irc database. Perfect. The next step is to create a front end, which is captured in the following part of this post.

Viewer

Because CouchDB is awesome and has an HTTP interface, there are Couchapps which are basically HTML applications hosted inside a CouchDB database. There is a convenient tool available which simplifies the process of writing a Couchapp which suprisingly is named couchapp.

To start I created a new app via couchapp generate app viewer command. This generates a folder viewer with the following structure:

viewer 
1
2
3
4
5
6
7

_attachments/
_id
couchapp.json
language
filters/
lists/
views/

The hosted application itself will be located inside _attachments folder. I removed all other folders except views because those Couchapp related functionalities are not yet needed. A view is used to make data inside the CouchDB available and query-able. The basic structure of an IRC message document inside the irc database looks like:

1
2
3
4
5
6
7
8
9
10
11

{
   "_id": "12761876127833",
   "_rev": "1-das12761876127833",
   "user": {
       "id": "1329740614286",
       "name": "buster",
       "room": "#emberjs"
   },
   "text": "hey brother!",
   "date": "2012-02-21T16:00:27.123Z"
}

To get a sorted list of messages for a specific period of time, I created a CouchDB view via executing couchapp generate view messages. This creates two files named map.js and reduce.js inside views/messages. The messages view implements functionality to list all messages sorted by date:

view/messages/map.js 
1
2
3
4
5
6
7
8
9
10
11
12
13
14

function(doc) {
    if (doc.date) {
        var d = new Date(doc.date),
        Y = d.getUTCFullYear(),
        M = d.getUTCMonth() + 1,
        D = d.getUTCDate(),
        h = d.getUTCHours(),
        m = d.getUTCMinutes(),
        s = d.getUTCSeconds(),
        ms = d.getUTCMilliseconds();

        emit([Y, M, D, h, m, s, ms], 1);
    }
};
view/messages/reduce.js 
1
2
3

function(values, rereduce) {
    return count(values);
}

Deploying the couchapp into the irc database via a couchapp push makes the view available and it is now accessible at http://emberjs.iriscouch.com/irc/design/viewer/view/messages?startkey=[2012,1,1]&endkey=[2012,1,7]&reduce=false&include_docs=true

The query on the view returns the results with following structure:

first 10 message of January 1st, 2012link
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

{
    "total_rows": 19211,
    "offset": 3389,
    "rows": [
    {
        "id": "2c2b1b4d889f994c00cf001b189598cd",
        "key": [2012, 1, 1, 1, 29, 31, 0],
        "value": 1,
        "doc": {
            "_id": "2c2ba2bd239f494380cf101b189598cd",
            "_rev": "1-cfd1963d905b0517a8c208a6e00269b1",
            "date": "2012-01-01T01:29:31.000Z",
            "text": "Come on!",
            "user": {
                "name": "GOB",
                "id": "1234",
                "room": "#emberjs"
            }
        }
    },
    ...
    ]
}

So now there is a view named messages which returns the messages sorted by date. The view can be queried to get all messages for a specific period and in combination with the CouchDB reduce functionality the count of messages for given period. Finally everything is set up for developing the basic HTML application for the IRC log viewer.

BPM

I use BPM for building the application and handling dependencies. BPM stands for Browser Package Manager and is the browser equivalent to NPM, the package manager for Node.js. It seems that it’s no more developed actively, but the current version is stable.

To create a BPM application I executed bpm init irc inside viewer and afterwards I renamed the generated folder irc to _attachaments (this is where Couchapp looks for static files). bpm init creates the following structure:

viewer/_attachments 
1
2
3
4
5
6

index.html
irc.json
css/
    main.css
lib/
    main.js

Next step is to add the needed dependencies via bpm add ember and bpm add spade inside the _attachments folder. I modified the index.html so spade is used to load the application. Spade is a JavaScript Module Loader similar to RequireJS. To interact with CouchDB I’ve included the jquery.couch.js file which is hosted inside a Couchapp’s _utils/script/ folder:

viewer/_attachments/index.html 
1
2
3
4
5
6
7
8
9
10
11
12
13
14

<!DOCTYPE html>
<html>
  <head>
    <link rel="stylesheet" href="assets/bpm_styles.css" type="text/css" media="screen" charset="utf-8" />
    <title>irc</title>
  </head>
  <body>
  </body>
  <script type="text/javascript" src="assets/bpm_libs.js"></script>
  <script src="/_utils/script/jquery.couch.js"></script>
  <script type="text/javascript" charset="utf-8">
    spade.require('irc');
  </script>
</html>

Test setup

Before I’ve added any functionality to the application I created a basic test setup using qunit by adding it as a dependency via bpm add qunit --development. All tests will be located inside _attachments/tests. To create a separate file irc/bpm_tests.js just containing the files inside the tests folder, I modified irc.json:

viewer/_attachments/irc.json 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

{
    ...
    "directories": {
        "lib": ["lib", "tests"]
    },
    "bpm:build": {
        ...
        "irc/bpm_tests.js": {
            "files": [
            "tests"
            ]
        }
    },
    ...
    "dependencies:development": {
        "qunit": ">= 0"
    }
}

Afterwards I created an empty file tests/tests.js which will be the entry for the tests, as well as a tests.html which follows the basic layout as documented in Using Qunit:

viewer/_attachments/tests.html 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

<html>
    <head>
        <link rel="stylesheet" href="assets/qunit/qunit_styles.css" type="text/css" media="screen" />
        <script type="text/javascript" src="assets/qunit/qunit_libs.js"></script>
        <script type="text/javascript" src="assets/bpm_libs.js"></script>
        <script type="text/javascript" src="assets/irc/bpm_tests.js"></script>
        <script type="text/javascript">
            spade.require('irc/tests');
        </script>
    </head>
    <body>
        <h1 id="qunit-header">IRC tests</h1>
        <h2 id="qunit-banner"></h2>
        <div id="qunit-testrunner-toolbar"></div>
        <h2 id="qunit-userAgent"></h2>
        <ol id="qunit-tests"></ol>
        <div id="qunit-fixture">test markup, will be hidden</div>
    </body>
</html>

Running the tests is done via executing bpm preview and navigating to http://localhost:4020/tests.html

Create IRC namespace

The core of the IRC viewer application is located in the lib/core.js file. It should import ember and define the IRC application namespace. So I first created a test:

viewer/_attachments/tests/application_test.js 
1
2
3
4
5
6

module('IRC');

test('exists', 2, function() {
    ok(IRC, 'IRC exists');
    ok(Ember.Application.detectInstance(IRC), 'IRC is an instance of Ember.Application');
});

and modified tests/tests.js to include the new application test:

viewer/_attachments/tests/tests.js 
1
2
3
4

// include core
require('irc/core');

requrie('./application_test');

Accessing localhost:4020/tests.html throws an error in the Web Inspector: Uncaught Error: Module irc/core not found. So I’ve created a file lib/core.js and rerun the tests. Now the module is found but the tests fail. Seems legit since I haven’t created the IRC namespace yet. This is changed by modifying lib/core.js:

viewer/_attachments/lib/core.js 
1
2
3
4

// include ember
require('ember');

IRC = Ember.Application.create();

Because the viewer has to deal with dates I use the sproutcore-datetime library. This library is available as bpm dependency, but the version hosted on GetBPM.org is not compatible with the used ember version 0.9.5. So I downloaded datetime.js into the lib folder and required it in lib/core.js via require('./datetime');:

viewer/_attachments/lib/core.js 
1
2
3
4
5
6

...

/* YES and NO globals needed in datetime */
window.YES = true;
window.NO = false;
require('./datetime');

I then created two utility methods inside the IRC application and added corresponding tests:

viewer/_attachments/lib/core.js 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

...
IRC = Ember.Application.create({

    /**
      Create a Ember.DateTime instance with timezone = 0;
    */
    createDateTime: function(date) {
        if (arguments.length === 0) {
            return Ember.DateTime.create().adjust({
                timezone: 0
            });
        }
        if (date && Ember.DateTime.detectInstance(date)) {
            return date.adjust({
                timezone: 0
            });
        }

        var dateObj = (Ember.typeOf(date) === 'string') ? new Date(date) : date;
        var time = dateObj.getTime();
        return Ember.DateTime.create(time).adjust({
            timezone: 0
        });
    },

    /**
      Get an array of date properties.
      For example: IRC.getDateArray(date, 'year', 'month', 'day') returns [2012, 3, 4]
    */
    getDateArray: function() {
        var date = arguments[0];
        var a = [];
        for (var i = 1; i < arguments.length; i++) {
            a.push(date.get(arguments[i]));
        }
        return a;
    }

}

Implement controllers

The IRC viewer basically has two controllers: one for the messages and one for all available days. Both controllers are located in lib/controller.js and the corresponding tests are defined in tests/controller_test.js. The controllers are basically Ember.ArrayProxy’s with specific methods addMessage and addDay.

viewer/_attachments/lib/controller.js 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

IRC.set('MessagesController', Ember.ArrayProxy.extend({
    content: [],
    loading: true,

    addMessage: function(msg) {
        var obj = Ember.Object.create({
            id: msg.id,
            username: msg.user.name,
            text: msg.text,
            date: IRC.createDateTime(msg.date)
        });
        this.pushObject(obj);
    },

    clear: function() {
        this.set('content', []);
    }
}));

IRC.set('DaysController', Ember.ArrayProxy.extend({
    content: [],
    loading: true,

    addDay: function(day) {
        this.pushObject(Ember.Object.create({
            date: IRC.createDateTime(day.date),
            count: day.count
        }));
    },

    clear: function() {
        this.set('content', []);
    }
}));

Create the templates

The views are described via Handlebars templates which are located inside lib/templates folder. To format dates and parse possible URL’s inside a IRC message, I wrote two Handlebars helpers parse and format which are located in lib/templates.js. parse parses a String for URL’s and wraps them in <a> tags. format formats a date with a given format (enough use of ‘format’ for now).

viewer/_attachments/lib/templates.js 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

require('./core');

String.prototype.parseURL = function() {
    return this.replace(/[A-Za-z]+:\/\/[A-Za-z0-9-_]+\.[A-Za-z0-9-_:%&#~\?\/.=]+/g,
    function(url) {
        return url.link(url);
    });
};

Handlebars.registerHelper('parse', function(property) {
    var value = Ember.getPath(this, property);
    value = Handlebars.Utils.escapeExpression(value);
    value = value.parseURL();
    return new Handlebars.SafeString(value);
});

Handlebars.registerHelper('format', function(property, format) {
    var dateFormat = Ember.DATETIME_ISO8601;
    if (Ember.typeOf(format) === 'string') {
        dateFormat = format;
    }
    var value = Ember.getPath(this, property);
    if (value && Ember.DateTime.detectInstance(value)) {
        value = value.toFormattedString(dateFormat);
    } else {
        value = undefined;
    }
    return new Handlebars.SafeString(value);
});

The templates itself are simple text files with the extension .handlebars located in lib/templates/:

lib/templates/messages.handlebars (messages.handlebars) download
1
2
3
4
5
6
7
8
9
10
11
12

{{#if loading}}
    loading messages ...
{{else}}
    <h3>{{format date "%Y-%m-%d"}}</h3>
    <ul class="unstyled" >
        {{#each messages}}
            <li>
                {{format date "%H:%M"}} <strong>{{unbound username}}</strong> {{parse text}}
            </li>
        {{/each}}
    </ul>
{{/if}}
lib/templates/days.handlebars (days.handlebars) download
1
2
3
4
5
6
7
8
9
10
11
12
13

{{#if loading}}
    loading days ...
{{else}}
    <ul class="unstyled" >
        {{#each days}}
            <li>
                <a href="#" {{action "loadDate" target="IRC" }} >
                    <strong>{{format date "%Y-%m-%d"}}</strong>
                </a> ({{count}})
            </li>
        {{/each}}
    </ul>
{{/if}}

The registered action in the days.handlebars template invokes the IRC#loadDate method. It gets the date of the clicked day and tells the dataSource to load the specific date. The dataSource itself is defined in lib/main.js, which is discussed in the next section:

viewer/_attachments/lib/core.js 
1
2
3
4
5
6
7
8
9
10
11

IRC = Ember.Application.create({
    ...
    loadDate: function(view, event, day) {
        var date = Ember.getPath(day, 'date');
        var dataSource = Ember.getPath(this, 'dataSource');
        if (dataSource) {
            dataSource.loadDay(date);
        }
    }
    ...
});

To load the templates a require is needed. I extended String prototype to add a tmpl function which handles this:

viewer/_attachments/lib/templates.js 
1
2
3
4
5
6
7
8

...

String.prototype.tmpl = function() {
    if (!Ember.TEMPLATES[this]) {
        Ember.TEMPLATES[this] = require('./templates/' + this);
    }
    return this;
};

So now a template can be loaded simply via 'TEMPLATE_NAME'.tmpl().

I also downloaded Twitter’s Bootstrap to _attachments/css/bootstrap.css to add a basic style. Additionally I modified the index.html’s <body> so it looks like:

viewer/_attachments/index.html 
1
2
3
4
5
6
7
8
9
10
11
12
13

    <!DOCTYPE html>
    <html>
      ...
      <body>
        <div class="container" >
            <div class="row" >
                <div class="span2" id="days" ></div>
                <div class="span10" id="messages" ></div>
            </div>
        </div>
      </body>
      ...
    </html>

DataSource

As mentioned when the templates has been discussed, the IRC application has a dataSource. This DataSource is capable of loading messages and days and setting the data on the controllers. The DataSource is defined in lib/couchdb_datasource.js:

viewer/_attachments/lib/couchdb_datasource.js 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

require('./core');

IRC.CouchDBDataSource = Ember.Object.extend({

    loadDay: function(day) {
        var messagesController = this.get('messagesController');
        // clear all messages
        messagesController.clear();
        // set loading state to true
        messagesController.set('loading', true);
        // create date range
        var from = day || IRC.createDateTime();
        var to = from.advance({
            day: 1
        });
        messagesController.set('date', from);
        // get data from CouchDB view 'messages'
        $.couch.db('irc').view('viewer/messages', {
            success: function(data) {
                if (data && data.rows && data.rows.length > 0) {
                    data.rows.forEach(function(row) {
                        messagesController.addMessage(row.doc);
                    });
                }
                messagesController.set('loading', false);
            },
            include_docs: true, // include documents
            reduce: false,
            startkey: IRC.getDateArray(from, 'year', 'month', 'day'),
            endkey: IRC.getDateArray(to, 'year', 'month', 'day')
        });
    },

    loadDays: function() {
        var daysController = this.get('daysController');
        // get data from CouchDB view 'messages'
        $.couch.db('irc').view('viewer/messages', {
            success: function(data) {
                if (data && data.rows && data.rows.length > 0) {
                  // iterate over each result
                    data.rows.forEach(function(doc) {
                        var key = doc.key;
                        // create Ember.DateTime from key, which looks like: [2012, 3, 21]
                        var date = Ember.DateTime.create().adjust({
                            year: key[0],
                            month: key[1],
                            day: key[2],
                            hour: 0,
                            timezone: 0
                        });
                        daysController.addDay({
                            date: date,
                            count: doc.value
                        });
                    });
                }
                daysController.set('loading', false);
            },
            group_level: 3, // group messages by year, month, day
            descending: true // reverse sort order
        });
    }

});

Assembly

Everything is defined now in its own place and its time to assembly everything together. The lib/main.js - which is loaded in the index.html - is therefore defined as follows:

viewer/_attachments/lib/main.js 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

// import all modules
require('./core');
require('./controller');
require('./templates');
require('./couchdb_datasource');

// create controller instances
IRC.daysController = IRC.DaysController.create();
IRC.messagesController = IRC.MessagesController.create();

// create dataSource and set constrollers
IRC.dataSource = IRC.CouchDBDataSource.create({
    messagesController: IRC.messagesController,
    daysController: IRC.daysController
});

// create views and append to DOM
Ember.View.create({
    templateName: 'messages'.tmpl(),
    messagesBinding: 'IRC.messagesController',
    loadingBinding: 'IRC.messagesController.loading',
    dateBinding: 'IRC.messagesController.date'
}).appendTo('#messages');

Ember.View.create({
    templateName: 'days'.tmpl(),
    daysBinding: 'IRC.daysController',
    loadingBinding: 'IRC.daysController.loading'
}).appendTo('#days');

// initially load messages of current day as well as a list of all days
Ember.run(function() {
    IRC.dataSource.loadDay(IRC.createDateTime().adjust({
        hour: 0,
        timezone: 0
    }));
    IRC.dataSource.loadDays();
});

To deploy the app I first executed a bpm rebuild inside the _attachments folder so the assets are freshly generated and afterwards pushed the Couchapp via couchapp push http://USERNAME:[email protected]/irc to the final location and so it is available at http://emberjs.iriscouch.com/irc/_design/viewer/index.html.

Summary

So now there is an IRC viewer build with BPM and using spade. The application itself is written in Ember.js and deployed as a Couchapp to a CouchDB instance which itself hosts all the IRC messages. A Hubot instance hanging around the IRC channel stores the messages into this very CouchDB.

The code for the viewer is hosted at pangratz/irc-log-viewer. Feel free to contribute! Just open an Issue or a Pull Request. I’m planning to blog about how this can be used to create an IRC logger for any IRC channel.

Outlook

During my development I wrote down some notes and future ideas:

  • When Hubot is down, no messages are logged: this is a problem. A possible solution would be to deploy a second Hubot which listens on the event when the primary Hubot leaves the room and takes over. The same problem is on the side of CouchDB: when the CouchDB is down, no messages can be saved. A backup CouchDB - maybe on a different hosting platform - would be a good idea I guess.
  • Export the transcript as Colloquy logs (using the CouchDB’s show / list functionality).
  • Searching the IRC logs would be very cool, for example via Apache Solr. I haven’t found any free hosting service. Anyone knows one? There is also the possibility to use couchdb-lucene but as far as I know iriscouch.com does not offer this…
  • Add realtime updates of incoming IRC messages by using the _changes feed of CouchDB
  • Show some statistics: when are the most messages? On which weekday? When is a user active the most?
  • Parse gist/pastie/… urls and show preview of it

Comments