Spreading my ignorange

From the TC-4310 cable modem manual:

The value is provided in Hertz. So, for 333 MHz, you must type: 333000000.

Go CBOR encoder: Episode 8, structs

This is a tutorial on how to write a CBOR encoder in Go, where we’ll learn more about reflection and type introspection.

Read the previous episodes, each episode builds on the previous one:

To encode structs we’ll mimic what the standard JSON encoder does and encode structs into maps of strings to values. For example if we pass the following to the JSON encoder:

struct {
    a int
    b string
    a: 1,
    b: "hello",

It outputs:

{"a": 1, "b": "hello"}

The struct kind is different from the map kind we implemented in the previous episode: with struct the fields’ are ordered and the keys are always strings. Because struct’s keys are strings, we can’t use all the examples from the spec like we did with maps, we can only use example with string-only keys. This leaves us with these three test cases:

{"a": 1, "b": [2, 3]}
{"a": "A", "b": "B", "c": "C", "d": "D", "e": "E"}

On the flip side because the keys are ordered we don’t have to look for each individual pair in the output like we did with maps. We can use the function testEncoder as it is for our test. Let’s add TestStruct to cbor_test.go:

func TestStruct(t *testing.T) {
    var cases = []struct {
        Value    interface{}
        Expected []byte
        {Value: struct{}{}, Expected: []byte{0xa0}},
            Value: struct {
                a int
                b []int
            }{a: 1, b: []int{2, 3}},
            Expected: []byte{
                0xa2, 0x61, 0x61, 0x01, 0x61, 0x62, 0x82, 0x02, 0x03,
            Value: struct {
                a string
                b string
                c string
                d string
                e string
            }{"A", "B", "C", "D", "E"},
            Expected: []byte{
                0xa5, 0x61, 0x61, 0x61, 0x41, 0x61, 0x62, 0x61, 0x42, 0x61,
                0x63, 0x61, 0x43, 0x61, 0x64, 0x61, 0x44, 0x61, 0x65, 0x61,

    for _, c := range cases {
        t.Run(fmt.Sprintf("%v", c.Value), func(t *testing.T) {
            testEncoder(t, c.Value, c.Expected)

To encode struct we’ll iterate over the fields of the struct with an index using Value.NumField and Value.Field, like this:

var v = reflect.ValueOf(struct {
	AKey string
	BKey string
}{AKey: "a value", BKey: "b value"})

for i := 0; i < v.NumField(); i++ {

This prints:

a value
b value

We have the fields’ values, we still need their names to write the map. The fields’ names aren’t stored in the value itself, they are stored in its type. We’ll use v.Type().Field() to get a StructField with the name of this particular field. For instance if we added the following at the end of the listing above:

for i := 0; i < v.NumField(); i++ {

We’d get the names of each field printed at the end:


Let’s assemble all that into a new function writeStruct in cbor.go. writeUnicodeString writes the keys, and we encode the value recursively with the encode() method:

func (e *Encoder) writeStruct(v reflect.Value) error {
    if err := e.writeInteger(majorMap, uint64(v.NumField())); err != nil {
        return err
    // Iterate over each field and write its key & value
    for i := 0; i < v.NumField(); i++ {
        if err := e.writeUnicodeString(v.Type().Field(i).Name); err != nil {
            return err
        if err := e.encode(v.Field(i)); err != nil {
            return err
    return nil

We add a call to writeStruct in the main switch statement:

case reflect.Struct:
	return e.writeStruct(x)

A quick run of go test confirms everything works as intended:

$ go test -v
--- PASS: TestStruct (0.00s)
    --- PASS: TestStruct/{} (0.00s)
    --- PASS: TestStruct/{1_[2_3]} (0.00s)
    --- PASS: TestStruct/{A_B_C_D_E} (0.00s)

Basic structs work, but we aren’t done yet. We’ll extend support for structs by mimicking the standard JSON encoder and add support for struct tagging. Here’s a summary of what option the encoder supports:

// Field appears in JSON as key "myName".
Field int `json:"myName"`

// Field appears in JSON as key "myName" and
// the field is omitted from the object if its value is empty[...]
Field int `json:"myName,omitempty"`

// Field appears in JSON as key "Field" (the default), but
// the field is skipped if empty.
// Note the leading comma.
Field int `json:",omitempty"`

// Field is ignored by this package.
Field int `json:"-"`

// Field appears in JSON as key "-".
Field int `json:"-,"`

We’ll implement these features with the cbor tag instead of json, like this:

Field int `cbor:"name,omitempty"`

Let’s write a test with the feature we want to verify, we’ll re-use this example from the CBOR spec:

{"a": 1, "b": [2, 3]}

In TestStructTag we call testEncoder with a tagged struct and checks the output. AField & BField have the name a & b respectively, while all the other fields must be ignored:

func TestStructTag(t *testing.T) {
        struct {
            AField int   `cbor:"a"`
            BField []int `cbor:"b"`
            Omit1  int   `cbor:"c,omitempty"`
            Omit2  int   `cbor:",omitempty"`
            Ignore int   `cbor:"-"`
        }{AField: 1, BField: []int{2, 3}, Ignore: 12345},
        []byte{0xa2, 0x61, 0x61, 0x01, 0x61, 0x62, 0x82, 0x02, 0x03},

If we run TestStructTag now the struct won’t be encoded correctly: every field will be in the output and the first two fields won’t have the right key.

The encoding/json package implements best-in-class tagging: we are going to steal some of its code to save time. Why write something new when we have some battle-tested code available?

We’ll copy encoding/json/tags.go into our project and we’ll add the function isEmptyValue from encoding/json/encode.go to it. We’ll replace package json with package cbor at the top to import the new code into our package. The new file tags.go looks like this:

// Source: https://golang.org/src/encoding/json/tags.go
// Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package cbor

import (

// tagOptions is the string following a comma in a struct field's "json"
// tag, or the empty string. It does not include the leading comma.
type tagOptions string

// parseTag splits a struct field's json tag into its name and
// comma-separated options.
func parseTag(tag string) (string, tagOptions) {
    if idx := strings.Index(tag, ","); idx != -1 {
        return tag[:idx], tagOptions(tag[idx+1:])
    return tag, tagOptions("")

// Contains reports whether a comma-separated list of options
// contains a particular substr flag. substr must be surrounded by a
// string boundary or commas.
func (o tagOptions) Contains(optionName string) bool {
    if len(o) == 0 {
        return false
    s := string(o)
    for s != "" {
        var next string
        i := strings.Index(s, ",")
        if i >= 0 {
            s, next = s[:i], s[i+1:]
        if s == optionName {
            return true
        s = next
    return false

// Source for isEmptyValue:
// https://golang.org/src/encoding/json/encode.go
func isEmptyValue(v reflect.Value) bool {
    switch v.Kind() {
    case reflect.Array, reflect.Map, reflect.Slice, reflect.String:
        return v.Len() == 0
    case reflect.Bool:
        return !v.Bool()
    case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
        return v.Int() == 0
    case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
        return v.Uint() == 0
    case reflect.Float32, reflect.Float64:
        return v.Float() == 0
    case reflect.Interface, reflect.Ptr:
        return v.IsNil()
    return false

Copying code like this may be bad for long-term maintenance, if the Golang developers fix something in the upstream code we won’t get the fix until we copy it ourselves. It’s OK to do that with this exercise because we’re here to learn, not to ship! Here’s what each function does:

Let’s refactor writeStruct to handle tagging. We need to know how many elements are in our map before we write the header. For example if we had a struct with 3 fields but one of them had the tag cbor:"-" indicating the field must be ignored, the encoded map would only have 2 key-value pairs. Instead of iterating and writing key-values on the fly, we’ll parse the fields first, and write the result to the output second. We’ll build the list of fields to encode and then write the encoded map from that list.

We define a new type fieldKeyValue to hold our key-value pairs, and iterate over each field in the struct and skip the fields marked with a tag to ignore it. Then we write the list of fields to the output, the new writeStruct function looks like:

func (e *Encoder) writeStruct(v reflect.Value) error {
    type fieldKeyValue struct {
        Name  string
        Value reflect.Value
    var fields []fieldKeyValue
    // Iterate over each field and add its key & value to fields
    for i := 0; i < v.NumField(); i++ {
        var fType = v.Type().Field(i)
        var fValue = v.Field(i)
        var tag = fType.Tag.Get("cbor")
        if tag == "-" {
        name, opts := parseTag(tag)
        // with the option omitempty skip the value if it's empty
        if opts.Contains("omitempty") && isEmptyValue(fValue) {
        if name == "" {
            name = fType.Name
        fields = append(fields, fieldKeyValue{Name: name, Value: fValue})
    // write map from fields
    if err := e.writeInteger(majorMap, uint64(len(fields))); err != nil {
        return err
    for _, kv := range fields {
        if err := e.writeUnicodeString(kv.Name); err != nil {
            return err
        if err := e.encode(kv.Value); err != nil {
            return err
    return nil

As you can see we get the information about each field’s tag via fType.Tag.Get("cbor"). We skip the field if its tag is “-”; or has an empty value with the “omitempty” option.

go test runs and confirms that struct tagging is implemented correctly. structs are done and our encoder is getting closer to be usable by a third party. We only have a few reflect.Kind’s that needs to handled:

We’ll implement floating and complex numbers, and ignore the UnsafePointer kind since we can’t reliably encode it. We’ll cover floating point numbers in the next episode.

Check out the repository with the full code for this episode.

Go CBOR encoder: Episode 7, maps

This is a tutorial on how to write a CBOR encoder in Go, where we’ll learn more about reflection and type introspection in Go.

Read the previous episodes, each episode builds on the previous one:

CBOR has a object or map type like JSON: it’s an ordered list of key/value pairs. We’ll use it to encode two different kinds of Go types: maps and structs. We’ll implement maps first and add support for structs in the next episode.

Here’s what the spec says about maps:

Major type 5: a map of pairs of data items. Maps are also called tables, dictionaries, hashes, or objects (in JSON). A map is comprised of pairs of data items, each pair consisting of a key that is immediately followed by a value. The map’s length follows the rules for byte strings (major type 2), except that the length denotes the number of pairs, not the length in bytes that the map takes up. For example, a map that contains 9 pairs would have an initial byte of 0b101_01001 (major type of 5, additional information of 9 for the number of pairs) followed by the 18 remaining items. The first item is the first key, the second item is the first value, the third item is the second key, and so on. […]

As usual we’ll use the examples from the CBOR RFC to write the tests. Maps are challenging to test because CBOR maps are ordered while Go maps aren’t. The order in which Go maps keys are returned is unspecified according to Value.MapKeys’s documentation:

MapKeys returns a slice containing all the keys present in the map, in unspecified order.

It means testEncoder cannot verify maps with more than one key/value pair in it, because it expects a unique result. Consider this map:

{1: 2, 3: 4}

They are multiple valid CBOR encoding for this map, because Go maps’ items can be in any order. With the example above the first key could either be 1 or 3.

We’ll have to check the different possibilities in the tests. For example from the CBOR spec we see that:

{1: 2, 3: 4}

Turns into:


Here’s the breakdown of the output:

0xa2            → header for a map of 2 pairs
0x01            → first key: 1
0x02            → first value: 2
0x03            → second key: 3
0x04            → second value: 4

Because the map has two elements there’s another valid CBOR encoding for it with 3 as the first key and 1 as the second key like this:

0xa2            → header for a map of 2 pairs
0x03            → first key: 3
0x04            → first value: 4
0x01            → second key: 1
0x02            → second value: 2

Our tests will have to handle unordered keys in the output. To verify the results we will search for every individual key/value pair in the encoded map to ensure all the values are here regardless of the order.

We don’t need to worry about this for maps with less than two entries, we have two examples from the CBOR spec like that:

We’ll use testEncoder for those two cases. Let’s add a new test function TestMap in cbor_test.go with two subtests for the easy examples:

func TestMap(t *testing.T) {
    // {}
    t.Run("{}", func(t *testing.T) {
        testEncoder(t, map[struct{}]struct{}{}, nil, []byte{0xa0})
    // ["a", {"b": "c"}]
    t.Run("[\"a\", {\"b\": \"c\"]", func(t *testing.T) {
            []interface{}{"a", map[string]string{"b": "c"}},
            []byte{0x82, 0x61, 0x61, 0xa1, 0x61, 0x62, 0x61, 0x63},

Now we’ll add what’s needed for multi-item maps: we verify the header’s major type, and the map length, then search all key-value pairs in the output. The tests cases we’ll use to test maps are:

{1: 2, 3: 4}
{"a": 1, "b": [2, 3]}
{"a": "A", "b": "B", "c": "C", "d": "D", "e": "E"}

To verify unordered maps the test needs the list of encoded key-value pairs. In our previous tests the test cases where stored in a structure like this:

struct {
    Value    interface{}
    Expected []byte

We’ll change it to hold what we need to verify the map, we’ll turn Expected from a slice of bytes into a slice of slice of bytes. The length of Expected is the size of the map. Items in Expected are encoded key-value pairs to look-up in the result:

struct {
    Value    interface{}
    Expected [][]byte

We add the new test cases and the code to check the result in the TestMap function:

var cases = []struct {
	Value    interface{}
	Expected [][]byte
		Value: map[int]int{1: 2, 3: 4},
		Expected: [][]byte{
			[]byte{0x01, 0x02}, // {1: 2}
			[]byte{0x03, 0x04}, // {3: 4}
		Value: map[string]interface{}{"a": 1, "b": []int{2, 3}},
		Expected: [][]byte{
			[]byte{0x61, 0x61, 0x01},             // {"a": 1}
			[]byte{0x61, 0x62, 0x82, 0x02, 0x03}, // {"b": [2, 3]}
		Value: map[string]string{
			"a": "A", "b": "B", "c": "C", "d": "D", "e": "E",
		Expected: [][]byte{
			[]byte{0x61, 0x61, 0x61, 0x41}, // {"a": "A"}
			[]byte{0x61, 0x62, 0x61, 0x42}, // {"b": "B"}
			[]byte{0x61, 0x63, 0x61, 0x43}, // {"c": "C"}
			[]byte{0x61, 0x64, 0x61, 0x44}, // {"d": "D"}
			[]byte{0x61, 0x65, 0x61, 0x45}, // {"e": "E"}

Each case the test will extract the major type and length of the map from the header using a bit mask, then verify their values. Our test cases have less than 23 elements in them, so the header will be only one byte. If we had a case with more than 23 elements we would have to change the code accordingly.

Let’s add the loop to iterate over the cases and verify what’s in the header:

for _, c := range cases {
    t.Run(fmt.Sprintf("%v", c.Value), func(t *testing.T) {
        var buffer bytes.Buffer

        if err := NewEncoder(&buffer).Encode(c.Value); err != nil {
            t.Fatalf("err: %#v != nil with %#v", err, c.Value)

        var (
            header     = buffer.Bytes()[0]
            result     = buffer.Bytes()[1:]
            lengthMask = ^uint8(0) >> 3 // bit mask to extract the length
            length     = header & lengthMask
        if header>>5 != majorMap {
            t.Fatalf("invalid major type: %#v", header)

        if int(length) != len(c.Expected) {
            t.Fatalf("invalid length: %#v != %#v", length, len(c.Expected))

We haven’t verified the map’s content yet, let’s add it: we search for each pair in the encoder’s output, then remove it from the output. Once we’re done verifying all the key-values, we check if the slice is empty to ensure there’s nothing left-over in the output. We add that code is at the end of the loop:

for _, c := range cases {
    t.Run(fmt.Sprintf("%v", c.Value), func(t *testing.T) {

        // Iterate over the key/values we expect in the map
        for _, kv := range c.Expected {
            if !bytes.Contains(result, kv) {
                t.Fatalf("key/value %#v not found in result", kv)
            // remove the value from the result
            result = bytes.Replace(result, kv, []byte{}, 1)

        // ensure we got everything is the map
        if len(result) > 0 {
            t.Fatalf("leftover in result: %#v", result)

Tests are done, now let’s get them working. To encode the map we’ll write its size in the header, then recursively encode each key followed by its Then we’ll add a case clause matching reflect.Map in the encode’s switch statement and call writeMap from it:

const majorMap = 5

func (e *Encoder) writeMap(v reflect.Value) error {
    if err := e.writeInteger(majorMap, uint64(v.Len())); err != nil {
        return err

    for _, key := range v.MapKeys() {
    return nil

func (e *Encoder) encode(x reflect.Value) error {
    switch x.Kind() {
    case reflect.Map:
        return e.writeMap(x)

As you can see we didn’t have to add much code to encode maps. The real challenge was the tests. Implemention was easy this time, but it won’t be next time: we’ll work with structs in the next episode and it’ll be a big one.

Check out the repository with the full code for this episode.

Go CBOR encoder: Episode 6, negative integers and arrays

This is tutorial on how to write a CBOR encoder in Go. Its goal is to teach reflection and type introspection. I recommend you read the previous episodes before jumping into this one:

Our CBOR encoder only accepts unsigned integers at the moment, to support all integer types we have to handle negative numbers. Negative number encoding is similar to positive number with a different major type. The spec says:

Major type 1: a negative integer. The encoding follows the rules for unsigned integers (major type 0), except that the value is then -1 minus the encoded unsigned integer. For example, the integer -500 would be 0b001_11001 (major type 1, additional information 25) followed by the two bytes 0x01f3, which is 499 in decimal.

As usual the tests come first, we re-use the examples from the CBOR specification to add TestNegativeIntegers in cbor_test.go:

import "math"  // for math.MinInt64
func TestNegativeIntegers(t *testing.T) {
    var cases = []struct {
        Value    int64
        Expected []byte
        {Value: -1, Expected: []byte{0x20}},
        {Value: -10, Expected: []byte{0x29}},
        {Value: -100, Expected: []byte{0x38, 0x63}},
        {Value: -1000, Expected: []byte{0x39, 0x03, 0xe7}},
            Value: math.MinInt64,
            Expected: []byte{
                0x3b, 0x7f, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,

    for _, c := range cases {
        t.Run(fmt.Sprintf("%d", c.Value), func(t *testing.T) {
            testEncoder(t, c.Value, nil, c.Expected)

For the encoder to recognize all integers types we add a new case clause in Encode()’s switch statement with the additional integer kinds like reflect.Int. It checks the sign of the integer: If the integer is positive we write it as a positive number, if it’s negative we turn it into an unsigned integer using the formula -(x+1), and we write that number to the output:

const majorNegativeInteger = 1


func (e *Encoder) encode(x reflect.Value) error {
    case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
        var i = x.Int()
        if i < 0 {
            return e.writeInteger(majorNegativeInteger, uint64(-(i + 1)))
        } else {
            return e.writeInteger(majorPositiveInteger, uint64(i))

8 lines of code was all we needed to support all integer types. That was easy, now we move onto something harder: arrays.

Arrays are the first composite type we add to the encoder. An array is a list of objects, it can contain any type of object like a JSON array:

[null, true, 1, "hello"]

Arrays have their own major type according to the spec:

Major type 4: an array of data items. Arrays are also called lists, sequences, or tuples. The array’s length follows the rules for byte strings (major type 2), except that the length denotes the number of data items, not the length in bytes that the array takes up. Items in an array do not need to all be of the same type. For example, an array that contains 10 items of any type would have an initial byte of 0b100_01010 (major type of 4, additional information of 10 for the length) followed by the 10 remaining items.

Because arrays can contain any type we’ll have to recursively encode objects, like we did in episode 4 with pointers.

Before we get started we’ll refactor how we recursively encode objects. Our encoder works with reflect.Value but the Encode() method takes an interface{} not a reflect.Value. When we call Encode() recursively we convert the reflect.Value into an interface which is then converted back into a reflect.Value. Those conversions aren’t efficient, so we’ll move all the code in the Encode() method into a new method called encode() —all lowercase— that takes a reflect.Value as parameter. Encode() is now just a call to this new method:

func (e *Encoder) Encode(v interface{}) error {
    return e.encode(reflect.ValueOf(v))

func (e *Encoder) encode(x reflect.Value) error {
    switch x.Kind() {
    case reflect.Ptr:
        if x.IsNil() {
            return e.writeHeader(majorSimpleValue, simpleValueNil)
        } else {
            // this replaces e.Encode(reflect.Indirect(x).Interface())
            return e.encode(reflect.Indirect(x))
    return ErrNotImplemented

With this small refactoring done let’s add tests based on the examples from the CBOR specification to our test suite, we have five cases:

[1, 2, 3]
[1, [2, 3], [4, 5]]
[1, 2, 3, ... 25]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25]

We add TestArray to cbor_test.go that runs a subtest for each of the cases above:

func TestArray(t *testing.T) {
    var cases = []struct {
        Value    []interface{}
        Expected []byte
        {Value: []interface{}{}, Expected: []byte{0x80}},
        {Value: []interface{}{1, 2, 3}, Expected: []byte{0x83, 0x1, 0x2, 0x3}},
            Value:    []interface{}{1, []interface{}{2, 3}, []interface{}{4, 5}},
            Expected: []byte{0x83, 0x01, 0x82, 0x02, 0x03, 0x82, 0x04, 0x05},
            Value: []interface{}{
                1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
                19, 20, 21, 22, 23, 24, 25,
            Expected: []byte{
                0x98, 0x19, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
                0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12,
                0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x18, 0x18, 0x19,

    for _, c := range cases {
        t.Run(fmt.Sprintf("%v", c.Value), func(t *testing.T) {
            testEncoder(t, c.Value, nil, c.Expected)

To get the tests to pass we have to match all array and slice types, except byte array and byte slice. We already match arrays and slices in the previous episode when we implemented byte strings.

When we have an array-like object to encode, we pass it to a new method writeArray. It writes the header with the length of the array, then iterates over the array’s elements and encode them recursively. To iterate over the array all we need are the methods reflect.Value.Len and reflect.Value.Index, we write a simple for loop and retrieve each item with v.Index(i):

majorArray           = 4
func (e *Encoder) writeArray(v reflect.Value) error {
    if err := e.writeInteger(majorArray, uint64(v.Len())); err != nil {
        return err
    for i := 0; i < v.Len(); i++ {
        if err := e.encode(v.Index(i)); err != nil {
            return err
    return nil

Let’s hook up writeArray to the main switch statement in encode(). We want to match array and slice not made of bytes. To achieve this we just need to add a call to writeArray after the if statement to check if we have a byte string in the reflect.Slice’s case clause. We literally add a single line to cbor.go:

func (e *Encoder) encode(x reflect.Value) error {
    switch x.Kind() {
    case reflect.Array:
        // Create slice from array
        var n = reflect.New(x.Type())
        x = reflect.Indirect(n).Slice(0, x.Len())
    case reflect.Slice:
        if x.Type().Elem().Kind() == reflect.Uint8 {
            return e.writeByteString(x.Bytes())
        // We don’t have a byte string therefor we have an array
        return e.writeArray(x)
    return ErrNotImplemented

TestArray successfully runs, we are done with arrays. Check out the repository with the full code for this episode.

With the addition of arrays our encoder can now encode complex data structures. We’re about to make it even better and dive deeper into Go reflection with the next major type: maps. See you in the next episode.

Go CBOR encoder: Episode 5, strings: bytes & unicode characters

CBOR strings are more complex than the types we already implemented, they come in two flavors: byte string, and unicode string. Byte strings are meant to encode binary content like images, while Unicode strings are for human-readable text.

We’ll start with byte string, here’s what the spec says:

Major type 2: a byte string. The string’s length in bytes is represented following the rules for positive integers (major type 0). For example, a byte string whose length is 5 would have an initial byte of 0b010_00101 (major type 2, additional information 5 for the length), followed by 5 bytes of binary content. A byte string whose length is 500 would have 3 initial bytes of 0b010_11001 (major type 2, additional information 25 to indicate a two-byte length) followed by the two bytes 0x01f4 for a length of 500, followed by 500 bytes of binary content.

If we encoded the 5 bytes “hello” as a CBOR byte string we’d have something like that:

0x45    // header for byte string of size 5: (2 << 5) | 5 → 0x45
0x68 0x65 0x6C 0x6C 0x6f   // The 5 byte string hello

To encode byte strings we’ll encode a regular CBOR integer with major type 2, and then we’ll write the byte string itself right after. The header has the type and the size of the string as a positive integer —we implemented this in episode 3—; and the data sized by the integer in the header. Before we can write the special header we’ll change the function writeInteger we wrote in episode 3 to add a parameter for the major type so it is now configurable by the caller and we modify the call to writeInteger() in Encode() to work with the new call:

func (e *Encoder) writeInteger(major byte, i uint64) error {
    switch {
    case i <= 23:
        return e.writeHeader(major, byte(i))
    case i <= 0xff:
        return e.writeHeaderInteger(major, minorPositiveInt8, uint8(i))
    case i <= 0xffff:
        return e.writeHeaderInteger(major, minorPositiveInt16, uint16(i))
    case i <= 0xffffffff:
        return e.writeHeaderInteger(major, minorPositiveInt32, uint32(i))
        return e.writeHeaderInteger(major, minorPositiveInt64, uint64(i))

case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
    // we pass major type to writeInteger
    return e.writeInteger(majorPositiveInteger, x.Uint())

This change will be useful later when we implement more complex types: it’s common to write an integer with the header for variable sized CBOR types.

Now we have to figure out how to match byte string types with reflect. Go has two distict types that match CBOR byte strings: byte slices, and byte arrays. If you don’t know the different between a slice and an array, I recommend the splendid article from the Golang blog: Go Slices: usage and internals. We’ll focus on slices first then arrays.

We start by adding tests based on the examples from the CBOR spec:

func TestByteString(t *testing.T) {
    var cases = []struct {
        Value    []byte
        Expected []byte
        {Value: []byte{}, Expected: []byte{0x40}},
        {Value: []byte{1, 2, 3, 4}, Expected: []byte{0x44, 0x01, 0x02, 0x03, 0x04}},
            Value:    []byte("hello"),
            Expected: []byte{0x45, 0x68, 0x65, 0x6c, 0x6c, 0x6f},

    for _, c := range cases {
        t.Run(fmt.Sprintf("%v", c.Value), func(t *testing.T) {
            testEncoder(t, c.Value, nil, c.Expected)

Slices have their own reflect kind: reflect.Slice. We only handle slices of bytes, so we’ll have to check the slice elements’ type like this:

var exampleSlice = reflect.ValueOf([]byte{1, 2, 3})

if exampleSlice.Type().Elem().Kind() == reflect.Uint8 {
    fmt.Println("Slice of bytes")

We use reflect.Uint8 in the if clause, because the byte type is an alias to uint8 in Go.

We add another case clause in Encode’s switch statement for slices and we check the slice’s elements’ type like this:

case reflect.Slice:
    if x.Type().Elem().Kind() == reflect.Uint8 {
        // byte string

Now all we have left to do is write the header and the byte string into the output, we’ll add the writeByteString method to tuck all the boilerplate code away from our main switch statement:

// we add the major type for byte string
majorByteString      = 2


func (e *Encoder) writeByteString(s []byte) error {
    if err := e.writeInteger(majorByteString, uint64(len(s))); err != nil {
        return err
    _, err := e.w.Write(s)
    return err

... In Encode() ...

case reflect.Slice:
    if x.Type().Elem().Kind() == reflect.Uint8 {
        return e.writeByteString(x.Bytes())

A quick run of go test confirms byte slices work, but we’re not done with byte strings yet, we still have to handle arrays. It’s easier to work with slices in general, so we’ll convert arrays to slices to avoid writing array specific code and re-use what we just wrote. We add the following code to our existing test TestByteString:

// for arrays
t.Run("array", func(t *testing.T) {
	a := [...]byte{1, 2}
	testEncoder(t, &a, nil, []byte{0x42, 1, 2})

Let’s add another case clause right before the case clause matching reflect.Slice:

case reflect.Array:
	// turn x into a slice
    x = x.Slice(0, x.Len())
case reflect.Slice:

We create a slice from our backing array with Value.Slice(), then we run the tests and we get a surprise:

$ go test -v .
=== RUN   TestByteString/array
panic: reflect.Value.Slice: slice of unaddressable array [recovered]
    panic: reflect.Value.Slice: slice of unaddressable array

It turns out we have an “unaddressable” array, and we cannot create a slice on it with Value.Slice() according to the doc. How are we going to get out of this? reflect doesn’t let us reference the array directly, we need to turn the array into something addressable: a pointer to the array. We create a pointer to it with reflect.New, then we use the pointer with reflect.Indirect to create our slice:

case reflect.Array:
    // Create slice from array
    var n = reflect.New(x.Type())
    x = reflect.Indirect(n).Slice(0, x.Len())
case reflect.Slice:

A quick run of go test confirms this solved our issue with the unaddressable array. All TestByteString tests now pass! We’re done with byte strings, unicode strings are next.

Text strings are like byte strings with a different major type. We have the header with the length of the string in bytes, and the data at the end. Text data is encoded in UTF-8 —Go’s native string encoding— so there’s no need to re-encode it: we can just write the string to the output as it is. Like we did for byte strings we add examples from the CBOR spec in a new test called TestUnicodeString:

func TestUnicodeString(t *testing.T) {
    var cases = []struct {
        Value    string
        Expected []byte
        {Value: "", Expected: []byte{0x60}},
        {Value: "IETF", Expected: []byte{0x64, 0x49, 0x45, 0x54, 0x46}},
        {Value: "\"\\", Expected: []byte{0x62, 0x22, 0x5c}},
        {Value: "\u00fc", Expected: []byte{0x62, 0xc3, 0xbc}},
        {Value: "\u6c34", Expected: []byte{0x63, 0xe6, 0xb0, 0xb4}},

    for _, c := range cases {
        t.Run(fmt.Sprintf("%s", c.Value), func(t *testing.T) {
            testEncoder(t, c.Value, nil, c.Expected)

We add a case clause for the kind reflect.String, then we write the header with the size of our string, and finally we write the string to the output:

majorUnicodeString   = 3
func (e *Encoder) writeUnicodeString(s string) error {
    if err := e.writeInteger(majorUnicodeString, uint64(len(s))); err != nil {
        return err
    _, err := io.WriteString(e.w, s)
    return err
case reflect.String:
    return e.writeUnicodeString(x.String())

And we are done with CBOR strings. Check out the code for this episode.

In the next episode we’ll implement signed integers, and our first composite type: array.

Go CBOR encoder: Episode 4, reflect and pointers

In the previous episode we encoded positive integers and learned how to write a CBOR item with a variable size. Our CBOR encoder can now encode nil, true, false, and unsigned integers. cbor.Encoder has grown strong, but type switches have their limits, we need more powerful weapons for the battles ahead: we’re about to take on pointers, and reflect will be our sword.

In the first episode of the series we encoded of the nil value since it was the easiest value to start with, but we aren’t finished with the nil we still got work to do to cover all cases. That’s because our encoder only handles the “naked” nil value, but not typed pointers that are nil. Whaaat? There are two kinds of nil pointers? Yep, that’s because nil by itself is special. Consider the following code:

var p *int = nil
var v interface{} = p
switch v.(type) {
case nil:
case *int:
    fmt.Println("int pointer")

The example above prints “int pointer”, because v isn’t a regular value but an interface that points to a int pointer value. Go interfaces are pairs of 32 or 64bits addresses: one for the type and one for the value. So in the type switch above we match the *int case because p’s type is *int. If we replaced the v definition with var v interface{} = nil, the program would print “nil”. That’s because the type of a nil value is itself nil, but typed-pointers’ type aren’t. Russ Cox’s article Go Data Structures: Interfaces is a superb introduction to how Go interfaces work if you’d like to learn more.

Let’s exhibit the problem in our code and add a test for typed nil pointers:

func TestNilTyped(t *testing.T) {
    var i *int = nil
    testEncoder(t, i, nil, []byte{0xf6})

    var v interface{} = nil
    testEncoder(t, v, nil, []byte{0xf6})

And run our tests with go test to see what happens:

--- FAIL: TestNilTyped (0.00s)
    cbor_test.go:18: err: &errors.errorString{s:"Not Implemented"} != <nil> with (*int)(nil)

The *int(nil) value isn’t recognized. So why did plain nil worked? Because it’s special: both its type and its value are nil. The Encode function matches the naked nil with the case nil statement in the type switch, this means only interfaces with a nil type will be matched. Therefor the code only works with the naked nil value, but not with typed pointers.

It turns out there’s a package to address that: reflect introspects the type system and let us match pointer types individually. The Law of reflection is a great introduction to reflection and the use of this package.

So we want to know if a value is a pointer. How does reflect help us? Consider this snippet:

var i *int = nil

It prints:


What happens here? First we convert each Go value to a reflect.Value, then we query its type with the method Kind that returns a reflect.Kind enumeration. reflect.Kind represents the specific kind of type that a Type represents. Kinds are families of types. For example there is a kind for structs —reflect.Struct—, for functions —reflect.Func—, and for pointers —reflect.Pointer.

We see above that the naked nil value and a nil pointer to integer have different kinds: invalid, and ptr. We’ll have to handle the two cases separately.

Refactoring time! We replace the type switch with a switch statement on the Kind of our value. In the example below x.Kind() allows us to distinguish types the same way the type switch x.(type) did:

func (e *Encoder) Encode(v interface{}) error {
    x := reflect.ValueOf(v)
    switch x.Kind() {
    case reflect.Invalid:
        // naked nil value == invalid type
        return e.writeHeader(majorSimpleValue, simpleValueNil)
    case reflect.Bool:
        var minor byte
        if x.Bool() {
            minor = simpleValueTrue
        } else {
            minor = simpleValueFalse
        return e.writeHeader(majorSimpleValue, minor)
    case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
        return e.writeInteger(x.Uint())
    return ErrNotImplemented

To identify pointer types reflect has a Kind called reflect.Ptr. We add a another case clause for reflect.Ptr, then if the pointer is nil we write the encoded nil value to the output:

case reflect.Ptr:
	if x.IsNil() {
		return e.writeHeader(majorSimpleValue, simpleValueNil)

After we add that, a quick run of go test confirms that TestNilTyped works.

Splendid! We solved nil pointers. How about non-nil pointers? They are relatively easy to handle: if we detect a pointer we can fetch the value it refers to via reflect.Indirect. So when we get a pointer we get the value it references instead of the memory address. Here’s an example of how reflect.Indirect works:

var i = 1
var p = &i
var reflectValue = reflect.ValueOf(p)

It prints:


When we find a non-nil pointer type, we call the Indirect function to retrieve the pointed value and we recursively call the Encode method on that value. We add a new test: TestPointer that verifies pointer referencing works as intended:

func TestPointer(t *testing.T) {
    i := uint(10)
    pi := &i  // pi is a *uint

    // should output the number 10
    testEncoder(t, pi, nil, []byte{0x0a})

With our test written let’s add the code necessary to handle valid pointers in our case clause:

case reflect.Ptr:
	if x.IsNil() {
		return e.writeHeader(majorSimpleValue, simpleValueNil)
	} else {
		return e.Encode(reflect.Indirect(x).Interface())

reflect.Indirect(x).Interface()) retrieves an interface to x’s underlying value, we pass it recursively to Encode and return the result. So if we passed a pointer to pointer to pointer to integer (***int) we’d have 3 recursive calls to Encode. TestPointer now passes, we are done with pointers!

There’s a repository with the code for this episode.

The reflect package will help us to handle more complex types in subsequent episodes. Next time we will encode string types: byte string, and Unicode strings.

kgpdemux is a TCP demultiplexer that uses the KGP protocol. I wrote it about a year ago as an experiment to use with Sauce Connect Proxy. It’s my best example of how to uses channels to implement complex control flows in Go efficiently. Sources are on Bitbucket: https://bitbucket.org/henry/kgp/src

The juicy part is kgp.go, where most of the concurrency is implemented.

That is all.

Go CBOR encoder: Episode 3, positive integers

In the previous episode we wrote a CBOR encoder that can handle the values nil, true, and false. Next we’ll focus on positive integers.

To proceed we have to learn more about how values are encoded. A CBOR object’s type is determined by the 3 first bits of its first byte. The first byte is called the header: it describes the data type and tells the decoder how to decode what follows, sometime the header contains data about the value in the additional 5 bits leftover but most of the time it contains information about the type.

For example: the encoded nil value is a single byte with the value 246, in binary that’s: 0b11110110. The first 3 bits are all 1’s, that’s 7 in decimal. The nil value’s major type is 7, which correspond to the “simple values” major type. The last 5 bits are 0b10110 or 22 in decimal, that’s the additional value with the type of the value, in our case it’s nil. To summarize the nil value’s major type is 7, and the additional value 22 identifies it as type nil. Here’s how you’d reconstruct the header for nil from the major type and the additional value:

byte(majorType << 5) | additionalValue

The booleans true and false have the same major type as nil: 7 and their additional values are 20 and 21 respectively. We’d build booleans from their major type and additional value like this:

fmt.Printf("%x\n", byte(7 << 5) | 20)   // prints f4
fmt.Printf("%x\n", byte(7 << 5) | 21)   // prints f5

Positive integers have their own major type: 0. With only 5 bits in the header that’s not enough to encode values higher than 32, therefor integers’ encoding in more complex than booleans and nil. The first 24 values are reserved for integers from 0 to 23, for integers bigger than 23 we have to write extra bytes to the output to encode them. To indicate how much data is needed to decode the integer we have the special additional values 24, 25, 26, and 27, they correspond to 8, 16, 32, and 64 bits integers respectively.

For example to encode 500 we need to use at least a 2 bytes integer, because 500 is too much to be represented as a single byte. So the first byte would be major type 0 and additional value 25 to tell the decoder: “hey, what follows is a two byte positive integer”. The header would look like this: 0b000_11001, followed by two byte 0x01 0xf4, that’s 500 encoded as a 16 bits big-endian integer.

Start with the easy case: integers from 0 to 23. We add a method called writeHeader to cbor.go that writes the single byte header to the output. To avoid using magic numbers all over our code we’ll also set some constants for the types we can encode thus far. We add the following to cbor.go:

const (
    // major types
    majorPositiveInteger = 0
    majorSimpleValue     = 7

    // simple values == major type 7
    simpleValueFalse = 20
    simpleValueTrue  = 21
    simpleValueNil   = 22

func (e *Encoder) writeHeader(major, minor byte) error {
    h := byte((major << 5) | minor)
    _, err := e.w.Write([]byte{h})
    return err

We use writeHeader to clear the magic numbers we put in the Encode method from the previous episodes. Our Encode method looks tighter now:

func (e *Encoder) Encode(v interface{}) error {
    switch v.(type) {
    case nil:
        return e.writeHeader(majorSimpleValue, simpleValueNil)
    case bool:
        var minor byte
        if v.(bool) {
            minor = simpleValueTrue
        } else {
            minor = simpleValueFalse
        return e.writeHeader(majorSimpleValue, minor)
    return ErrNotImplemented

Our mini-refactoring is done, we check everything is still working with go test and it does still work. Now that we cleaned that up and verified it works we add tests for the small integers in cbor_test.go:

func TestIntSmall(t *testing.T) {
    for i := 0; i <= 23; i++ {
        testEncoder(t, uint64(i), nil, []byte{i})

We loop from 0 to 23, we build our expected return value and check it corresponds to what the encoder gives us. In this case a single byte with the major type 0, and our value i.

Some of you may have noticed that we turn our value i into an uint64 when we pass it to testEncoder instead of a plain int. That’s because Go has different integers types like uint64, and int16, and plain int, unfortunately all those types are different for the Go type system and require adding extra code to work. We will handle the other integers later for now we’ll stick to uint64.

Small integers are easy to implement: in Encode switch’s statement we add a case uint64: clause, and if the integer is between 0 and 23 we output the header with the right additional value and that’s all:

case uint64:
	var i = v.(uint64)
    if i <= 23 {
        return e.writeHeader(majorPositiveInteger, byte(i))

A quick run with go test confirms TestIntSmall works. Time to work on the extended integers: as usual we’ll write the tests first. To get good coverage, we’re going to copy the examples given in the appendix of the CBOR spec for our tests.

We’ll use subtests to make it easier to track what test fails, subtests allows you to define multiple sub-tests with different names inside a single test function. Our subtests’ names will be the numbers we’re checking, for example to test the integer 10 we’d do something like this:

func TestExample(t *testing.T) {
        "10",                 // name of the subtest
        func(t *testing.T) {  // function to execute
            testEncoder(t, uint64(10), nil, byte{0x0a})

When we run go test with this example we’ll have a test named “TestExample/10”, we could add another call to t.Run() with the string “foo” as name to create another subtest named “TestExample/foo”.

Let’s replace this example with real tests. We’ll use a table to store our test cases, iterate over it, and verify each results. Our tests values and expected outputs are taken from the CBOR spec examples:

func TestIntBig(t *testing.T) {
    var cases = []struct {
        Value    uint64
        Expected []byte
        {Value: 0, Expected: []byte{0x00}},
        {Value: 1, Expected: []byte{0x01}},
        {Value: 10, Expected: []byte{0x0a}},
        {Value: 23, Expected: []byte{0x17}},
        {Value: 24, Expected: []byte{0x18, 0x18}},
        {Value: 25, Expected: []byte{0x18, 0x19}},
        {Value: 100, Expected: []byte{0x18, 0x64}},
        {Value: 1000, Expected: []byte{0x19, 0x03, 0xe8}},
        {Value: 1000000, Expected: []byte{0x1a, 0x00, 0x0f, 0x42, 0x40}},
            Value: 1000000000000,
            Expected: []byte{
                0x1b, 0x00, 0x00, 0x00, 0xe8, 0xd4, 0xa5, 0x10, 0x00,
            Value: 18446744073709551615,
            Expected: []byte{
                0x1b, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,

    for _, c := range cases {
        t.Run(fmt.Sprintf("%d", c.Value), func(t *testing.T) {
            testEncoder(t, uint64(c.Value), nil, c.Expected)

If we run the tests as they are now, the ones with numbers less than 24 will pass, but all the bigger numbers will fail with a not implemented error:

--- PASS: TestIntBig/0 (0.00s)
--- PASS: TestIntBig/1 (0.00s)
--- PASS: TestIntBig/10 (0.00s)
--- PASS: TestIntBig/23 (0.00s)
--- FAIL: TestIntBig/24 (0.00s)
	cbor_test.go:18: err: &errors.errorString{s:"Not Implemented"} != <nil> with 0x18
--- FAIL: TestIntBig/25 (0.00s)
	cbor_test.go:18: err: &errors.errorString{s:"Not Implemented"} != <nil> with 0x19
--- FAIL: TestIntBig/100 (0.00s)
	cbor_test.go:18: err: &errors.errorString{s:"Not Implemented"} != <nil> with 0x64

Big CBOR integers have 2 parts: a header to determine the type, followed by the value encoded as a big endian integer. For example 25 is encoded as 0x1819, that’s 2 bytes: the header is 0x18 or 24 in decimal, that corresponds to a 8 bit integer type. The second byte after the header is 0x19 or 25 in decimal the integer we encoded. To re-iterate: the header gives us the type of the value and the bytes following the header is the value being encoded.

The first thing we’ll do is add a helper function to write our native integers as big endian integers. It takes an interface{} as parameter instead of an integer because the package encoding/binary uses the type of the value it writes to determine how much data to write. For example passing the value 1 typed as a uint16 to binary.Write will output 2 bytes: 0x0001. This allows us to cast our integer to the right type to encode our the correct sized integer with binary.Write:

// writeHeaderInteger writes out a header created from major and minor magic
// numbers and write the value v as a big endian value
func (e *Encoder) writeHeaderInteger(major, minor byte, v interface{}) error {
    if err := e.writeHeader(major, minor); err != nil {
        return err
    return binary.Write(e.w, binary.BigEndian, v)

We don’t want the big switch statement in the Encode method to become messy as we’re adding more code, so we create a new method for our encoder: writeInteger where we’ll put all the code to encode integers.

The writeInteger method encodes our single integer value and casts it to the smallest integer type that can hold its value:

func (e *Encoder) writeInteger(i uint64) error {
    switch {
    case i <= 23:
        return e.writeHeader(majorPositiveInteger, byte(i))
    case i <= 0xff:
        return e.writeHeaderInteger(
            majorPositiveInteger, minorPositiveInt8, uint8(i),
    case i <= 0xffff:
        return e.writeHeaderInteger(
            majorPositiveInteger, minorPositiveInt16, uint16(i),
    case i <= 0xffffffff:
        return e.writeHeaderInteger(
            majorPositiveInteger, minorPositiveInt32, uint32(i),
        return e.writeHeaderInteger(
            majorPositiveInteger, minorPositiveInt64, uint64(i),

As you can see we cast the value i into different integer types depending on how big it is to minimize the size of what we write to the output. The less bytes we use the better.

Encode now looks like this:

func (e *Encoder) Encode(v interface{}) error {
    switch v.(type) {
    case nil:
        return e.writeHeader(majorSimpleValue, simpleValueNil)
    case bool:
        var minor byte
        if v.(bool) {
            minor = simpleValueTrue
        } else {
            minor = simpleValueFalse
        return e.writeHeader(majorSimpleValue, minor)
    case uint64,:
        return e.writeInteger(v.(uint64))
    return ErrNotImplemented

Once we add this little bit of code our integer tests will pass:

--- PASS: TestIntBig (0.00s)
    --- PASS: TestIntBig/0 (0.00s)
    --- PASS: TestIntBig/1 (0.00s)
    --- PASS: TestIntBig/10 (0.00s)
    --- PASS: TestIntBig/23 (0.00s)
    --- PASS: TestIntBig/24 (0.00s)
    --- PASS: TestIntBig/25 (0.00s)
    --- PASS: TestIntBig/100 (0.00s)
    --- PASS: TestIntBig/1000 (0.00s)
    --- PASS: TestIntBig/1000000 (0.00s)
    --- PASS: TestIntBig/1000000000000 (0.00s)
    --- PASS: TestIntBig/18446744073709551615 (0.00s)

Let’s add the integer types we ignored thus far to be more exhaustive with what our encoder supports:

case uint, uint8, uint16, uint32, uint64, int, int8, int16, int32, int64:
	if v.(uint64) >= 0 {
		return e.writeInteger(v.(uint64))

Now we can pass a positive int, int8, int16, int32, or int64 and it will work. We can’t handle negative number yet.

That’ll all for now. There’s a repository with the code for this episode. In the next episode we’ll introduce the reflect package to care of pointers.

Go CBOR encoder: Episode 2, booleans

In the previous episode, we learned how to encode the nil value. Now we’ll do booleans. According to the CBOR specification, booleans are represented by a single byte: 0xf4 for false, and 0xf5 for true.

We’ll write the tests first, but before we do that let’s write a helper function for our encoder tests. We want to avoid copy-pasting the same code all over our tests. Looking at the test we wrote in the previous episode, that’s how all of our future tests will look like:

func TestNil(t *testing.T) {
    var buffer = bytes.Buffer{}
    var err = NewEncoder(&buffer).Encode(nil)

    if !(err == nil && bytes.Equal(buffer.Bytes(), []byte{0xf6})) {
            "%#v != %#v or %#v != %#v",
            err, nil, buffer.Bytes(), []byte{0xf6},

We test something with a well defined interface: the encoder gets a value, returns an error, and outputs an array of bytes. This means we can use this to factor out most of the code into a single helper function named testEncoder, we add this to our test file:

// testEncoder test the CBOR encoder with the value v, and verify that err, and
// expected match what's returned and written by the encoder.
func testEncoder(t *testing.T, v interface{}, err error, expected []byte) {
    // buffer is where we write the CBOR encoded values
    var buffer = bytes.Buffer{}
    // create a new encoder writing to buffer, and encode v with it
    var e = NewEncoder(&buffer).Encode(v)

    if e != err {
        t.Fatalf("err: %#v != %#v with %#v", e, err, v)

    if !bytes.Equal(buffer.Bytes(), expected) {
            "(%#v) %#v != %#v", v, buffer.Bytes(), expected,

testEncoder will save quite a bit of typing. TestNil turns into a single line —saving 8 lines— with testEncoder doing all the work:

func TestNil(t *testing.T) {
    testEncoder(t, nil, nil, []byte{0xf6})

Let’s write the tests for booleans now. We only need to test true and false, so TestBool is a terse two lines:

func TestBool(t *testing.T) {
    testEncoder(t, false, nil, []byte{0xf4})
    testEncoder(t, true, nil, []byte{0xf5})

With our current encoder only able to encode nil right now, if we run the tests we’ll get a not implemented error thrown at us:

$ go test -v .
=== RUN   TestNil
--- PASS: TestNil (0.00s)
=== RUN   TestBool
--- FAIL: TestBool (0.00s)
        cbor_test.go:19: err: &errors.errorString{s:"Not Implemented"} != <nil> with false
FAIL    _/home/henry/cbor   0.003s

Now we’ll implement booleans encoding and get those tests passing. From the previous episode our Encode function looked like this:

var ErrNotImplemented = errors.New("Not Implemented")

// Can only encode nil
func (enc *Encoder) Encode(v interface{}) error {
    switch v.(type) {
    case nil:
        var _, err = enc.w.Write([]byte{0xf6})
        return err
    return ErrNotImplemented

We need to add another case to the switch block to know if we have a boolean, and then we’ll have to turn the generic interface{} named v into a boolean value to know what the encode will output into its writer:

// Can only encode nil, false, and true
func (enc *Encoder) Encode(v interface{}) error {
    switch v.(type) {
    case nil:
        var _, err = enc.w.Write([]byte{0xf6})
        return err
    case bool:
        var err error
        if v.(bool) {
            _, err = enc.w.Write([]byte{0xf5}) // true
        } else {
            _, err = enc.w.Write([]byte{0xf4}) // false
        return err
    return ErrNotImplemented

The tricky part here is v.(bool): this turns the non-typed interface v into a boolean value using type assertion.

Encode now works with booleans and our tests pass:

$ go test -v .
=== RUN   TestNil
--- PASS: TestNil (0.00s)
=== RUN   TestBool
--- PASS: TestBool (0.00s)
ok  	_/home/henry/cbor	0.003s

This wraps up the 2nd episode. Next we’ll encode a type that’s more than a single byte of output: positive integers.

I created a public repository with the code for this episode.

Go CBOR encoder: Episode 1, getting started

Let’s write a CBOR encoder in Go. We’ll learn more about type switching and type manipulation with the reflect package. This is going to be a series of posts, each building on the previous one. It requires a good understanding of the Golang syntax.

CBOR is a data format described in RFC 7049, it’s like JSON but binary instead of text. Its design goals are:

We use an interface similar to the encoding/json package. If you are unfamiliar with this encoding sub-packages, I recommend you read the JSON and Go article.

To start our empty package we’ll create a file named cbor.go like this:

// Implements CBOR encoding:
//   https://tools.ietf.org/html/rfc7049
package cbor

import (

type Encoder struct {
        w   io.Writer

func NewEncoder(w io.Writer) *Encoder {
        return &Encoder{w: w}

func (enc *Encoder)Encode(v interface{}) error {
        return nil

Here’s how we use the encoder:

var output = bytes.Buffer{}
var encoder = NewEncoder(&output)
var myvalue = 1234
// write the integer 1234 CBOR encoded into output
if err := encoder.Encode(&myvalue); err != nil {

We have our basic structure we can now start working on the encoder’s implementation. In the previous example we encoded the integer 1234, but we won’t start with integers, instead we will encode the value nil to start because it’s the easiest value to encode.

According the CBOR specification the nil value is represented by a single byte: 0xf6

Let’s write a test with the testing package, we’ll verify the encoder outputs the single byte 0xf6 into the result buffer when we pass nil. We create a new file cbor_test.go beside cbor.go for our tests:

package cbor

import (

func TestNil(t *testing.T) {
    var buffer = bytes.Buffer{}
    var err = NewEncoder(&buffer).Encode(nil)

    if !(err == nil && bytes.Equal(result.Bytes(), []byte{0xf6})) {
            "%#v != %#v or %#v != %#v",
            err, nil, result, []byte{0xf6},

If we run the test with what we currently have we’ll get an error. No surprise: we haven’t implemented anything yet, so the encoder won’t write to the output buffer:

$ go test .
--- FAIL: TestNil (0.00 seconds)
    cbor_test.go:15: <nil> != nil or []byte{} != []bytes{0xf6}
FAIL    _/home/henry/essays/cbor    0.011s

To implement the nil value encoding: we write the byte in the result when calling Encode() and we’ll return an error if the value isn’t nil since we haven’t implemented anything else yet:

var ErrNotImplemented = errors.New("Not Implemented")

// Can only encode nil
func (enc *Encoder) Encode(v interface{}) error {
    switch v.(type) {
    case nil:
        var _, err = enc.w.Write([]byte{0xf6})
        return err
    return ErrNotImplemented

Here we’re using a type switch to determine the type of the value we got. There’s only one case for now: nil, where we write the 0xf6 value in the output and return the error to the caller.

And now the test succeeds:

=== RUN   TestNil
--- PASS: TestNil (0.00s)
ok  	_/home/henry/essays/cbor	0.027s

We created the initial encoder that can encode a single value successfully. In the next episode we’ll implement CBOR encoding for more basic Go types.

Boot OpenBSD with EFI for full resolution display

I got a new radeon graphic card for gaming. Unfortunately it’s not supported yet by the radeon(4) driver on OpenBSD. Luckily there’s a workaround: booting with UEFI. UEFI does all the talking with the graphics card, and this allows OpenBSD to use the screen’s full resolution on unsupported cards: it should work better than the vesa(4) driver. EFI boots the operating system differently than old school BIOS: it uses a special partition for the operating system’s bootloader.

First I needed to enable UEFI. It wasn’t called UEFI or EFI in the BIOS setup but something like “Windows 8 / 10 boot method”, I picked the regular version not the WHQL.

Second I created the EFI system partition to store the EFI bootloaders with gparted: I made a 100 MB partition formatted as FAT32 at the beginning of the disk, then set the flags “boot” & “esp” on it.

Third I created the system’s partition and did the install via a USB stick a usual. Once the install was done and before rebooting, I copied the OpenBSD’s EFI bootloader to the EFI system partition like this:

# mount /dev/sd0i /mnt2
# mkdir -p /mnt2/efi/boot
# cp /mnt/usr/mdec/BOOTX64.EFI /mnt2/efi/boot

I rebooted and it worked out of the box: OpenBSD can now use my Radeon RX 570 and my 2560×1440 screen at full resolution.


How to set the timezone on an OpenBSD system:

# ln -sf /usr/share/zoneinfo/America/Vancouver /etc/localtime

Or if one feels lazier:

# zic -l America/Vancouver

Updated reminder for later: pkg.conf doesn’t exist anymore on OpenBSD, now it’s installurl(5) you use to setup the mirror to install packages.

I work at a company that does Selenium stuff. So I build Selenium load testing tools on the side, because I think it’s cool. Maybe someone will be impressed:

https://bitbucket.org/henry/selenium-surfer https://bitbucket.org/henry/selenium-stresser

I’m a programmer.

I’m always a byte away from disaster.

Vim is my day to day editor: I use it for coding and writing prose. Vim was made for programmers, not really for writers. Here are a few plugins I use when I write prose with Vim.

  1. Goyo offers distraction free mode for Vim. It lets me to resize the writing surface in the editor’s window however I want.

  2. I like soft line wraps when I write prose. Soft line wrap is when the text reaches the end of the window it gets wrap back to the beginning of the next line without inserting a new-line. Vim Pencil lets you do that.

  3. vim-textobj-quote offers support for smart quoting as you type like Unicycle but this plugin is still maintained.

I have this snippet in my .vimrc to get in and out of the writer mode:

" I use a 80 columns wide editing window
function s:WriteOn()
    call pencil#init({'wrap': 'soft', 'textwidth': 80})
    Goyo 80x100%

function s:WriteOff()

command WriteOn call s:WriteOn()
command WriteOff call s:WriteOff()

Reminder for later, how to selectively rollback a file to the specified version:

git reset -p '<hash>' '<filename>'

The function ScanLines splits lines from a io.Reader, it returns a channel where the results are written. Handy when you want to read things on the fly line by line:

// Read input line by line and send it to the returned channel. Once there's
// nothing left to read closes the channel.
func ScanLines(input io.Reader) <-chan string {
        var output = make(chan string)

        go func() {
                var scanner = bufio.NewScanner(input)

                for scanner.Scan() {
                        output <- scanner.Text()

                if err := scanner.Err(); err != nil {
                        fmt.Fprintln(os.Stderr, "reading input:", err)

        return output

func main() {
        var input = ScanLines(os.Stdin)

        for x := range input {
                fmt.Printf("%#v\n", x)

Reminder for later, OpenBSD’s pkg_delete utility can remove unused dependencies automatically:

# pkg_delete -a

How to fill a PDF form with pdftk

I had a rather lenghty PDF form to fill, it took me 2 hours to do it becuase copy-pasting didn’t work in with my PDF editor.

After I saved the file I realized that I clicked on a radio button I shouldn’t have clicked on: Kids. I do not have kid, and the radio selection didn’t contain the zero option, only one and more. After trying to get rid of that radio selection for 5 minutes, it looked like there was no way to undo this: I had selected something I couldn’t unselect.

I didn’t want to waste another 2 hours to fill out the form, I needed to fix this by editing the PDF.

After a bit of googling I found pdftk, a command-line toolkit that can fill & extract information out of PDF forms.

To unselect the radio box, I had to extract the form data. Pdftk can extract the information into a text file that you can edit with a text editor.

pdftk input.pdf generate_fdf output form_data.fdf

Here it will generate form_data.fdf from input.pdf’s form values. After that I had to modify the fdf file to get rid of my selection. In my case, I wanted to reset the selection for the Kids radio selection.

/Kids [
/V (1)
/T (RadioButtonList[0])

I changed it from “1 kid” to “nothing selected”.

/Kids [
/V /Off
/T (RadioButtonList[0])

Then I had to re-enter the information from the FDF file into the PDF.

pdftk input.pdf fill_form form_data.fdf output output.pdf

It took me around an hour to do all this, so pdftk saved me time. I liked it, you can check out pdftk’s own examples to learne more, the documentation is terse and complete.

Still bitmap after all those years

Bitmap fonts are pixel-art fonts. Unlike outline fonts they cannot be automatically scaled with good results, to create a multi-size bitmap font you have to create a different version for each size. They can’t be anti-aliased so they tend to look blocky compared to outline fonts.

Outline fonts use Bézier curves, they are scallable, and their edges can be anti-aliased to make them look nicer. Today everybody is running an operating system that can render outline fonts decently, and can use those smooth looking beauties with superior results compared to bitmap fonts.

Bitmap fonts are a thing of the past.

Yet, I still use a bitmap font for my day to day programming tasks. I transitionned to an outline font for a while, but ultimately switched back after a few months because the outline font didn’t seem as sharp.

It may be silly, but nothing looks as sharp as a bitmap font to me. I’m talking what it looks like on a computer screen in 2016 with a dot pitch of 0.27 mm. Because each pixel is either black or white and nothing smooths out the edges, it’s sharp.

I salivate like everybody on those screenshots of multi-colored terminal window with a fancy outline font that support ligatures, has cool emoji’s icons, and rainbows of bright pastel colors. I’m sure it’s great to feel like you’re on acid while you write code, but I like my bitmap font and my bland terminal colors. It gets the job done and it’s easy on my eyes.

I’ll switch to outline fonts when I get a screen with a high pixel density for my workstation, but for now I’ll use my bitmap font, it doesn’t look so bad with today’s fat pixels.

Reminder for later, how to setup pkg.conf after a fresh OpenBSD install:


You may have to replace %c with snapshots.

Reminder for later: how to mount MTP device on Ubuntu with jmtpfs.

  1. Your user has to be part of plugdev group: usermod -a -G plugdev $(whoami)
  2. user_allow_other in /etc/fuse.conf
  3. jmtpfs path/to/mount/point will mount the 1st MTP device on path/to/mount/point

When I need to scrap data online, I use Python with requests, and lxml, two libraries taht make it easy to extract data without going crazy.

Often I come accross HTML tables with data formatted like this:

    <a href='/data1'><strong>data1</strong></a>

In that case we’d just like to extract the list data1, data2, & data3 from the table. With the different markup in each cell it would take quite a bit of elbow grease to clean it up. lxml has a special method that makes all that easy: text_content. Here’s what the documentation says about it:

Returns the text content of the element, including the text content of its children, with no markup.

For the previous HTML snippet we’d extract the data like this:

>>> from lxml import html
>>> root = html.fromstring('''    <td>
...         <a href='...'><strong>data1</strong></a>
...     </td>
...     <td>
...         data2
...     </td>
...     <td>
...         data<em>3</em>
...     </td>
... ''')
>>> [i.text_content().strip() for i in root.xpath('//td')]
['data1', 'data2', 'data3']

I got new speakers with a built-in USB-DAC for my home computer.

Once plugged OpenBSD recognized it as an USB-audio device, so far so good. Unfortunatly I couldn’t get any sound out of it, but my computer’s sound card —which is recognised as a separate device— worked.

It turns out that by default sndiod —the system audio mixer— uses the first audio device it finds and ignores the others. To get it to use other devices you must specify them in rc.conf.local like this:

sndiod_flags="-f rsnd/0 -f rsnd/1"

I restarted sndiod like this sudo /etc/rc.d/sndiod restart, and everything now works nicely.

I keep diaries on different subjects. When I add a new entry I start by inserting a timestamp at the top of the entry. I used to do it ‘manually’ –by copying the current date and pasting in into Vim–, and yesterday I decided to write a vim function to automate that.

There’s nothing especially hard about that, but it took me a while before I figured out how to insert the timestamp at the current cursor position. It didn’t look like there was any built-in Vim function to do it, and most solution I found online seems overly complicated.

It turns out that all I needed was an execute statetement like this: execute ":normal itext to insert", this will insert the string “text to insert” at the current cursor position.

I this added to my vimrc:

function s:InsertISODate()
    let timestamp = strftime('%Y-%m-%d')
    execute ":normal i" . timestamp
    echo 'New time: ' . timestamp

function s:InsertISODatetime()
    let timestamp = strftime('%Y-%m-%d %H:%M:%S')
    execute ":normal i" . timestamp
    echo 'New time: ' . timestamp

command Today	call s:InsertISODate()
command Now     call s:InsertISODatetime()

Reminder for laster, reset a branch to what its remote branch is:

git checkout -B master origin/master

I looked for a decent weather app on Android for a while. I tried many, they tended to be cluttered and overly complicated for what they were doing. I’m now using Weather Timeline, it’s clear, fast, and simple. I check it every morning, it gives me a quick and clean overview of the forecast, no need to dig the information in sub-menus, there’s no ad, and it’s just $1.

Sometime you need an up-to-date virtualenv for your Python project, but the one installed is an old version. I read virtualenv’s installation manual, but I didn’t like much that you have to use sudo to bootstrap it. I came with an alternative way of installing an up-to-date virtualenv as long as you have an old version. In my case it was an Ubuntu 12.04 which ships virtualenv 1.7.

1st install the outdated version of virtualenv:

$ sudo apt-get install python-virtualenv

Then setup a temporary environment:

$ virtualenv $HOME/tmpenv

Finally use the environment created before to bootstrap an up-to-date one:

$ "$HOME/tmpenv/bin/pip" install virtualenv

$ "$HOME/tmpenv/bin/virtualenv" $HOME/env
$ rm -rf "$HOME/tmpenv"  # Delete the old one if needed

I love Rob Pike’s talks: face-paced and intense. It’s a nice change from the typical talks about programming: slow, and often more about self-promotion than teaching.

I usually feel I miss a few things here and there when I listen to Rob, he’s smart and expect you to be smart, he doesn’t talk down to you. I rarely understand everything perfectly: this is good, it means I don’t fully master the subject, it means I’m learning, it means he’s making a good use of my time.

This talk about implementing a bignum calculator is the perfect example, Rob doesn’t spend much time reading and explaining the code or the examples: he assumes that his audience is smart enough to understand the most of the details; he focuses on the big picture, and the hard details.

I needed a set datastructure for a Go program, after a quick search on the interweb I saw this reddit thread about sets, queues, etc…

Short answer: for sets use maps, specifically map[<element>]struct{}. My first intuition was to use map[<element>]interface{}, but it turns out that an empty interface takes 8 bytes: 4 bytes for the type, and 4 bytes for the value which is always nil, while an empty structure doesn’t use any space.

There weren’t many details on how to do to it. So I just gave it a try, it was pretty easy to figure out the implementation, as long as operations like union, intersection arent’ needed.

That’s how I would implement an integer set:

type set map[int]struct{}

var myset = make(set)  // Allocate the map

// Add an element to the set by adding an empty structure for the key 1
myset[1] = struct{}{}

// Check if we have 1 in our set
if _, ok := myset[1]; ok {
    println("1 in myset")
} else {
    println("1 not in myset")

// Remove the element from the set
delete(myset, 1)

REST toolbox for the command-line hacker

I work with lots of REST services those days. RESTful services are easy to access and use because they’re based on well-known tech, this eliminates half of the tedious work. Unfortunatly the other tedious half is still here: interfacing. We still need to get and convert the data from the original format to the format we want. Lately I found two tools that help a great deal with HTTP and JSON: HTTPie and Jq. Today I’ll talk about HTTPie.

I used cURL for almost a decade to deal with HTTP from the command-line, as few month ago I heard about a command line client called HTTPie, that has a nice interface that totally makes sense:

$ http --form POST localhost:8000/handler foo=1 bar=hello

What does it do? It does a HTTP POST on localhost:8000/handler with the following body:

POST /handler HTTP/1.1
Host: localhost:8000
Content-Length: 15
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: HTTPie/0.8.0
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded; charset=utf-8


It’s exactly the kind of stuff I want. I often automate the common stuff away with a function, like this:

http() {
    # 1st parameter is the path, we pop it out of the parameter list
    local urlpath = "$1"; shift

    # since we use http has our function name we have to use `command' to
    # call the executable http and not the function
    command http --form POST "example.com$urlpath" "$*"

# Do a POST with the following parameters: foo=1&bar=2
http /test foo=1 bar=2

If you’d rather submit JSON instead of an url-encoded form, replace the --form option with --json.

Give HTTPie a shot next time you want to talk with an HTTP service from the command line: it may take you less time to learn it from scratch than remember how to use cURL.

One day at work the Internet went out around 4pm, most of my co-workers couldn’t work: most of the information they needed to work were online, and they didn’t have local copies. If I was writing Python at the time, being offline would have been a problem, I rely on the information on docs.python.org, frameworks, and libraries’ documentation: all of which are online.

With Go, it’s less of a problem: if you have godoc installed you can access the installed packages’ documentation using a local HTTP server:

$ "$GOPATH/bin/godoc" -http=:8080

Point your browser to localhost:8080 and here you have it: the documentation for all your installed packages.

A few tips & tricks for properly managing views with Postgres:

  1. Name the return values with AS
  2. Type constant values by prefixing them with their type

For example consider the following:

$ CREATE VIEW myview AS SELECT 'bar';
WARNING:  column "?column?" has type "unknown"
DETAIL:  Proceeding with relation creation anyway.

Here’s what the Postgres documentation says about it:

Be careful that the names and types of the view’s columns will be assigned the way you want. For example:

  CREATE VIEW vista AS SELECT 'Hello World';

is bad form in two ways: the column name defaults to ?column?, and the column data type defaults to unknown. If you want a string literal in a view’s result, use something like:

  CREATE VIEW vista AS SELECT text 'Hello World' AS hello;

First we’ll name our string to get rid of the “?column?” name:

$ CREATE VIEW myview AS SELECT 'bar' AS bar;
WARNING:  column "bar" has type "unknown"
DETAIL:  Proceeding with relation creation anyway.

Second we set the type of our return value by prefixing with TEXT:

$ CREATE VIEW myview AS SELECT TEXT 'bar' AS bar;

That is all.

I wanted to upgrade to Go 1.3 on my Desktop at work. The version of Go shipped with Ubuntu 14.10 is 1.2. I found this article talking about godeb a go program to package Go into a .deb file installable on Ubuntu.

You still need Go 1.0+ installed, otherwise the installation is straightforward:

$ go get gopkg.in/niemeyer/godeb.v1/cmd/godeb

And it’s easy to use:

$ $GOPATH/bin/godeb list

$ sudo $GOPATH/bin/godeb install 1.3.3

You may to rebuild you Go packages after the install, the easiest is to delete the old version and let Go rebuild them when they are needed:

$ rm -rf $GOPATH/pkg/*

Reminder for later, how to setup an SSH tunnel to a remote PostgreSQL database.

Create the script postgres_tunnel.sh:


: ${REMOTEHOST:=example.com}
: ${LOCALPORT:=12345}

# Assuming postgres listens on localhost:5432
ssh "$REMOTEHOST" -L "$LOCALPORT":localhost:5432 -N

Execute it, and connects to the remote database like this:

$ psql 'postgres://username:password@localhost:12345/mydb'

I work in open spaces a lot, my current job is in a shared open office. Open spaces are known to imped workers productivity. It’s one of the worst office arrangement, yet they are popular in IT. They offers a short-term gain: cheaper space, for an invisible –but real– price: less productive and satisfied workers. Noise and lack of privacy are the main causes of disatisfaction, and while it’s difficult to address the lack of privacy, something can be done about the noise. I deal with it via a two-pronged attack:

  1. Earplugs, I use Howard Leight Max, and there are lots of other good earplugs around with different shape and foam-type.
  2. Headphones with a Pink noise playing in a loop

This way I get a good isolation from the environment, and it makes interruptions awkward for the interrupter: he has to wait for me to take off my headphones and earplugs. This makes interrupting me more costly, which is a nice side-effect.

I have regular headphones; I wonder how good the combo earplugs & noise-cancelling would be.

I used Swiftkey as my Android keyboard: I found it worked better than the default Android keyboard. I switch between English & French often, Swiftkey just works without selecting a language while the default Android keyboard needs to be switched between the 2 languages to work properly.

I was hanging-out on security.google.com, and saw that Swiftkey Cloud had full-access to my Email: read & write access! I didn’t remember giving them any of these permissions, I did at some point, but I don’t know when.

Reading one’s emails is a great way to improve her predictive typing, but it wasn’t clear to me that they’d read my Emails. I’m almost certain there was no big-fat pop-up saying so.

That kind of thing really annoys me: Emails are sacred, you don’t mess-up with it unless you’re a company with no moral or ethic like Linkedin or Facebook… I made fun of people that gave their Email and password to 3rd parties, but I kind of did the same…

I revoked the access, deleted my Swiftkey Cloud account, removed the Switfkey app from my phone, and switched back to Google keyboard, it came a long way since I replaced it with Swiftkey a year ago.

I started a project in Go, when I got started everything was in a single file. Now this file is too big for my own taste, so I split it into 2 separate files, let’s call them main.go & util.go. In main.go I have the main() function, in util.go I have functions used by main.go.

When I tried to run main.go directly I got this error:

$ go run main.go
# command-line-arguments
./main.go:150: undefined: SomeFunction

I didn’t want to create a package just for util.go, sometime source files really are specific to a program and aren’t reusable.

My search for a solution on the web didn’t yield anything useful. I knew it was possible, I saw programs like godeb do it. After a while I built the program with go build to see if the error would be different, and it worked this time. Weird… What’s going on?

Everything was the same except I didn’t specify what to build, in that case main.go. Go just built every go source files in the directory, I got the same error with go build with only main.go specified.

$ go build main.go
# command-line-arguments
./main.go:150: undefined: SomeFunction

That’s when it hit me, I just needed to list all the files necessary to run the program on the command line:

$ go run main.go util.go

Here it is: go run needs the complete list of files to execute in the main package. I’ll know it for next time!

I’m a hater, especially when it comes to programming languages. I approach most of them with pessimism, I rarely look at new language and think it’s great right away.


I started programming in high-school with QBasic. I made a small choose-your-own-adventure style game in text-mode, then moved on to C. I didn’t like C much: it didn’t feel expressive enough, and was too hard to use for the noob I was. I started programming seriously after high-school and discovered C++ during my 2nd year studying computer science. I became instantly a fan, C++ had so many features, the language felt more expressive, more powerful. I though I could master it within a few years. It took me a good 5 years to realize that C++ was too big: It seemed baroque and overly complex after all this time. After 5 years I still didn’t master most of the language; like everybody I just used a subset. I went back to C and saw what I wasn’t able to see then: C was expressive and simple. I took me years of struggle with the seemingly cool features of C++ to realize C was the best part of C++.


I was a Javascript hater for a long time, the language seemed so absurdly hard to work with, there were traps and gotcha. If you asked me 5 years ago: PHP or Javascript I’d reply: “PHP of course! Javascript is terrible.” Then I learned more about it thanks to Douglas Croford’s videos. While Javascript is not my favorite language I came to appreciate it, today I’d pick it over PHP if I had to start a new project.


Python looked a bit ridiculous when I first used it. I didn’t like the indentation to define blocks, or that the language was interpreted, I didn’t get that a dynamic language opens up a realm of new possibilities. At the beginning Python felt like slow, and dumbed down C++. It took time writing Python everyday to fall in love with it, but after a year it was my favorite language. I’ve been writing Python personally and professionally for 10 years now.


My first impression of Go was: it’s kind of like a cleaned-up C. My main problem was that concurrency was part of the language like Erlang, I though it’d be better if the tools for concurrency were contained in a library like multiprocessing in Python. Also there were a few things that really bothered me with it like the semi-colon insertion, a known Javascript gotcha.

Then I heard about goroutines, channels, & Go’s select statement, after that it all made sense. Go has an elegant solution to a fundamental problem of modern computing: concurrency.

The semi-colon insertion turned out to be a convenient quirk.

Go became my new toy a month ago, it’s now on track to replace Python as my favorite programming language.

1.2 GB is easier to understand than 1234567890 Bytes, at least for humans. I write functions to ‘humanize’ numbers often, but it never seems worth keeping those functions around since they are generally quick and easy to write. Today I decided to finally stop rewriting the same thing over and over, and headed to PyPi –the Python module repository–, and of course there’s a module to do that on it:

>>> import humanize
>>> humanize.naturalsize(1234567890)
'1.2 GB'

The Oak Island Money Pit is the story of a 2 centuries long treasure hunt on a small Island near Nova Scotia, Canada. What makes this treasure hunt special is how must resource was sunk into it. For 200 years adventurers lost their time, money, and sometime life trying to find the elusive gold & jewels.

Long story short: In 1795 after seeing lights in the island three teenagers found a mysterious depression on the ground of the Island, and started treasure hunting by digging in the ground. Clues, and signs of treasures were found, fortunes were wasted on fruitless digging, and 6 people died. To this day nothing of value was found. Speculations & theories about the origin of the supposed treasure in the money pit abound.

Something that wasn’t addressed in the article: How the hell did all that stuff got so deep into the ground? If digging deep enough was still a problem in the 1960’s how did 17th century men managed to dig a hole 100 feet deep, along with booty traps, and flooding tunnels? Given the numerous difficulties the treasure hunters went through for the past 200 years, it would have been a great engineering feat. All of that without being detected by the locals, and keeping it secret for 200 years.

Most adventurers probably though about it, and they gave their imaginary enemy —the treasure digger— too much credit, and didn’t give their predecessors enough credit.

The Oak Island Money Pit is a great story because it’s a great tragedy. The only treasure on the island is the memories of this great human adventure.

I use xdm(1) as my login manager under OpenBSD. After I loging, it starts xconsole(1). It’s not a big deal, but I’d rather not have it happen.

To stop xdm from starting a new xconsole for every session, edit /etc/X11/xdm/xdm-config, remove or comment out the following lines:

DisplayManager._0.setup:      /etc/X11/xdm/Xsetup_0
DisplayManager._0.startup:    /etc/X11/xdm/GiveConsole
DisplayManager._0.reset:      /etc/X11/xdm/TakeConsole

I’m not a fan of 37signals, but I must admit their DNS service xip.io is handy. I’m setting up some web servers right now, and I needed a domain to test my configuration. The whole DNS dance is a bit time-consuming: add record to zone file, & wait for my DNS to pick it up. With xip.io there’s no need to wait: prepend your host’s IP address before .xip.io and the domain will resolve to your own IP.

For example will resolve to

They are other services like this like ipq.co or localtest.me, but as far as I know they don’t work out of the box: you have to register your subdomain first or can only use it with localhost.

How to run PostgreSQL as a non-privileged user

The quick and dirty guide to setup a postgres database without root access.

Create a directory where your data will live:

$ postgres_dir="$HOME/postgres"
$ mkdir -p "$postgres_dir"
$ initdb -D "$postgres_dir"
[lots of output...]

Then run postgres:

postgres -D "$postgres_dir"

Create and database for yourself:

$ createdb $(whoami)
$ psql

To stop the server type Ctrl-C, or you can use pg_ctl if postgres runs in the background:

pg_ctl stop -D "$postgres_dir"
$ sudo apt-get install zsh-doc
$ man zsh
No manual entry for zsh
See 'man 7 undocumented' for help when manual pages are not available.
$ man 7 undocumented
       undocumented - No manpage for this program, utility or function

Damn you Ubuntu/Debian/whoever decided that man pages were ‘too big’ to be part of documentation packages!

Copy a branch between Git repositories

Git is tricky to use: after 4 years I still have a hard time figuring out how to do simple operations with it. I just spent 30 minutes on that one:

Say you have 2 copies of the same repository repo1 & repo2. In repo1 there’s a branch called copybranch that you want to copy to repo2 without merging it: just copy the branch. git pull repo1 copybranch from repo2 doesn’t work because it will try to merge the copybranch into the current branch: no good.

It looks like git fetch repo1 copybranch would be the way to go, but when I did it, here’s what I saw:

From repo1
 * branch            copybranch -> FETCH_HEAD

After that a quick look at the logs doesn’t show copybranch, FETCH_HEAD, or any of the commits from copybranch. What happenned? Git copied the content of copybranch, but instead of creating another branch named copybranch it creates a temporary reference called FETCH_HEAD, but FETCH_HEAD doesn’t appear in the logs. In summary: Git copied the branch, & made it invisible, because you know… it makes perfect sense to hide what you just copied.

So how do you copy the branch, and create a branch with the same name referencing the commits? Here it is:

git fetch repo1 copybranch:copybranch

I use VMWare Player to run my OpenBSD under Windows with my laptop –a Thinkpad X1 Carbon–, its newer hardware wasn’t fully supported by OpenBSD when I got it. I had issues with VirtualBox, it was slow: 50%+ of the CPU time was spend on interupts, and I couldn’t find a solution on the Internets. After reading Ted unangst’s blog where he describes his setup I decided to switch to VMWare Player.

I use Putty to connect to the VM, and while VMWare worked well, sometime the dynamically assigned IP changed. I had to reopen Putty to change the IP from time to time, and it was getting annoying. It turns out that you can use static IPs. VMWare uses a netmask, and it reserves the 3-127 range for static IPs. I put this in my /etc/hostname.em0:


It didn’t work right away. It turns out that the gateway was at I put the following in /etc/mygate:

And things are now working nicely.

CSS3 Quickies

I did some web-design for a friend this January. I didn’t use HTML & CSS for a while, I did quite a bit of tinkering with the graphic design of scratchpad 6 months ago, using a custom font and trying a few ‘newish’ features like media queries. It was a good oportunity to discover & use the new HTML5 & CSS3 features.

My friend had a Joomla template he wanted to use for his site. His needs were limited: 5 pages and maybe a contact form. Hosting this with Joomla seemed a bit overkill, so I decided to ‘rip-off’ the template and create a clean HTML skeleton for him to us.

First we tried to work from the source of the template, but the template’s HTML & CSS were very hairy, I couldn’t wrap my head around it, so I decided to rewrite it from scratch. Who doesn’t love to reinvent the wheel? :)

I used purecss on the 1st version, but I wasn’t satisfied with the way it worked. I like to minimize HTML markup. I really dislike it there are 5 divs to size what should be a single box, when all you need to do is use CSS correctly. Unfortunately purecss works this way, you need to get your boxes inside other boxes to get things to work correctly. It’s understandable why they do that: it’s a CSS framework, the CSS directs the way the DOM is structured. CSS is complicated to get to work without a few intermediate steps. Since I was here to learn more about CSS, I dropped purecss, started using what I learned studying it for the new template.

Here are the few things I tried while working on the site:


box-sizing: border-box is handy: it includes the border & the padding in the box’s size. If you have 2px borders and a 1em padding in a 200px box, the box will be 200px with 2px off for the border and 200px - 2 × 2px = 196px of usable space. It simplify box’s placement, no more: My borders are 4px, my box is 200px so that’s 4 × 2 + 200 = 208px… It’s only supported by IE9+, and it needs a prefix on some browsers like Firefox. I used it when developing the site, at the end of the design process I removed it, I had to make a few adjustments here and there, but it was easy to do. border-box was neat though: no more pointless tinkering. I’ll use it again for sure.

Media queries

Media queries are the basis of responsive design. Instead of using pixels as a unit like most do, I use ems, the typographic unit of measure. That made many things simpler, like re-calculating the size of the grid when adjusting the font size.

While media queries aren’t that easy to use & lack expressive power there weren’t too bad and I managed to do what I wanted without too much thinkering.


display: inline-block; allows you to simplify box packing: designing layouts requires less tweaks and hacks. inline-blocks are well supported by all modern browsers. IE6 supports it –short-of–, and it even works correctly on IE7! I’m kind of late to the party, better late than never.

CSS3 transition

Fancy, but meh. It’s all eye-candy, and I don’t think it improves usability / readability one bit. I’ll still used them there and there to fade in and out bits of interface.

I was trying to get Tornado’s AsyncHTTPTestCase to work with Motor, but the tests were blocking as soon as there was a call to Motor. It turns out that Motor wasn’t hooked to the tests’ IO loop, therefor the callbacks were never called. I found the solution after looking at Motor’s own tests:

motor.MotorClient(host, port, io_loop=self.io_loop)

In HTML whitespaces aren’t significant, but single space between tags can wreak havok with your CSS layout. Consider the following:

<div></div> <div></div>



The space between the 2 div tags will insert a single space between the 2 divs. If the 2 divs width were 50% of the parent, they wouldn’t fit in it because of the added space. To fix this you have to remove the spaces:


It looks kind of ugly to have everything on the same line. My favorite way to deal with this is to move the final tag’s chevron at right before the next tag, like so:


It doesn’t look super nice, but it’s better than having everything glued on the same line.

3rd party sharing & social-media buttons are a waste of your and your reader’s time: Sweep the Sleaze


Random C fact of the day: I though that size_t was a built-in type, because that’s what the operator sizeof is supposed to return. Roman Ligasor –a co-worker– proved me wrong. It turns out that it’s defined in the header stddef.h. C is a minimalist language, why define a built-in type that will just be an alias to another built-in type?

Without transition: According to the CERT Secure Coding Standards, one should use size_t instead of integer types for object sizes.


int copy(void *dst, const void *src, int size);


size_t copy(void *dst, const void *src, size_t size);

I use DuckDuckGo those days, one its best feature is the bang, a smart shortcuts to other websites. I use !man & !posix all the time: it’s give you direct access to the POSIX standard manuals & specification. That’s better than relying on Linux manuals, as I have to at work.

I started drinking coffee 15 years ago, when I was a student. Like many students my sleep schedule was messed-up: I was working late, and getting up late. I loved working at night: it’s quiet, there’s almost no distraction. To compensate for my lack of sleep during the day I drank coffee, sodas, and occasionally tea. After graduating, I stopped drinking coffee and soda for a while. I switched to tea, 2 to 4 cups of tea every weekday for 10 years.

I started drinking coffee again 2 years ago when I started my new job. I drank between 1 to 3 cups of coffee at the start of the day, and 2 to 4 cans of Diet Soda during the day on top of that. I ingested 150mg to 400mg of caffeine everyday. I though that coffee was by far the biggest source of caffeine, but it turns out that sodas, and tea also contain a significant amount. A cup of coffee contains around 100mg, a can of Diet Pepsi has 35mg, while a cup of tea is around 40mg.

How much caffeine is too much? According to Wikipedia, 100mg per day is enough to get you addicted:

[…] people who take in a minimum of 100 mg of caffeine per day (about the amount in one cup of coffee) can acquire a physical dependence that would trigger withdrawal symptoms that include headaches, muscle pain and stiffness, lethargy, nausea, vomiting, depressed mood, and marked irritability.

The mayo clinic recommends cutting back for those who get more than 500mg everyday, I suspect this limit is lower for me.

I had my last coffee Sunday morning, almost 4 days ago. I’ve experience most of the withdrawal symptoms, it’s getting better, but I think I have another day or two before I can feel normal again. I didn’t even consume that much caffeine. It must be awful to be nauseous or vomit on top of the other symptoms. I imagine only big consumers get these problems, but this tells you a lot about how strong the addiction can be. The headaches are especially annoying, they’re caused by an increase of blood flow in the head, compressing the brain. I usually exercise when I want to get my mind off something or try to get back into a healthy routine, but In the case of caffeine withdrawal, exercise seems to make the headaches even worse. Aspirin works well, but it still hurts quite a bit. The worse part is how irritable I am right now, I tend to go crazy when I’m on my own, and idle. I get restless and my mind wanders, thinking of past personal injustices, and how I’ll get revenge: I get angry for noting. I can’t even focus on a book for more than 10 minutes without my mind wandering.

The good news is: it’s almost over.

[…] withdrawals occurred within 12 to 24 hours after stopping caffeine intake and could last as long as nine days.

There were positive side effect: I used to go pee 3 to 5 times a day, not anymore. My sleep seems to improve. Sleep is why I stopped caffeine consumption. I don’t sleep well most nights, waking up tired but not sleepy.

Like most things, caffeine isn’t bad, but it has to be consumed in moderation. I don’t plan to ban caffeine from my life, but I do need to reduce my consumption, and take a break from time to time.

I always forget about the HTTP server in Python. I’ve been using a quick’n dirty shell script with netcat to quickly serve a single file over HTTP for a while, but this is easier, and works better:

python -m SimpleHTTPServer [port number]

It will serve the content of the current directory.

I’ve redesigned this space after reading the excellent Practical typography by Matthew Butterick. I picked Charter as the font for the body text. Charter is recommended in the appendix of the aforementioned book as one the bests free-fonts by far. I tried Vollkorn from Google web fonts for a while before switching to Charter. While Vollkorn looked fine to me, Charter looks even better it feels crisper.

I picked fonts from the Source Pro family by Adobe as my sans & mono-spaced fonts, but I may switch to one of the DejaVu’s fonts if I find one that I like better.

I had problems with KiTTY: the session management doesn’t seem to work with multiple sessions. I looked for an alternative and found this one: http://jakub.kotrla.net/putty/. It’s basically a normal PuTTY with a patch to store Sessions on disk. It lacks the URL click-to-open feature, but I think I can live without it. I’ve been using it for 2 weeks now, and I’m happy with it.

curl is a useful tool if you’re working with HTTP. I’m fond of the -w option: it prints all kind of information about the transfert, including timing:

$ curl -s -o /dev/null -w '
url_effective: %{url_effective}
http_code: %{http_code}
time_total: %{time_total}
time_namelookup: %{time_namelookup}
time_connect: %{time_connect}
time_pretransfer: %{time_pretransfer}
time_starttransfer: %{time_starttransfer}
size_download: %{size_download}
size_upload: %{size_upload}
size_header: %{size_header}
size_request: %{size_request}
speed_download: %{speed_download}
speed_upload: %{speed_upload}
content_type: %{content_type}' http://google.ca/

url_effective: http://google.ca/
http_code: 301
time_total: 0.062
time_namelookup: 0.038
time_connect: 0.045
time_pretransfer: 0.045
time_starttransfer: 0.062
size_download: 218
size_upload: 0
size_header: 320
size_request: 153
speed_download: 3504.000
speed_upload: 0.000

My C is rusty. Here are a few tricks I forgot and had to rediscover:

int array[42];
int *pointer = array + 8;

// This will be 8, not 8 * sizeof int
size_t x = pointer - array;

Number of elements in an array:

int array[42];

// x == 42 * sizeof int. Not what we want
size_t x = sizeof array;

// The right way to do it: y == 42
size_t y = sizeof array / sizeof array[0];

That is all.

Reminder for later. Put a file-descriptor in non-blocking mode with fcntl:

static int
set_non_blocking(int fd)
        return fcntl(fd, F_SETFL, fcntl(fd, F_GETFL)|O_NONBLOCK);

I’ve tried to write regularly for more than 5 years. Yet I still struggle to publish content monthly, let alone write something every week. It’s not really a time problem: every week I waste many hours slacking on Internet, at work or at home. I could probably turn one of those wasted hours into a semi-productive writing session. I realize that ‘wasting’ time is a necessary evil, nobody can be productive all the time: sometime you just need to turn the brain off to recharge.

I’m not posting high quality content. Most of my posts in my Blog took 2 to 4 hours of focused effert to research and write, not including the time it takes for the idea to mature. I want to post less, but post better articles.

Most of my writing happen in short bursts over a few days. I’m writing a lot at the moment because I have a shiny new bitbucket repository with all my essays in progress. Unfortunately once the novelty wears out, I don’t think I’ll write at the same rate…

Scratchpad made me write more. It was inspired by Steven Johnson’s Spark file. I think this new repository with short essays in progress is a good complement to it. I’ll see how things turn out…

Cal Newport argues that the best way to write for wannabe writers like me is to have an adaptable writing schedule every week. I wonder if I should reserve a time-slot during the week for writing. 1 hour focused on writing may turn out to be the trigger I need to think and publish valuable articles.

I used a bitmap font for many years for all my development related applications. Bitmap fonts aren’t scalable, but they usually look sharper and clearer than the scalable variety, because they fit perfectly to the pixels on the screen.

My font used to be Terminus, it’s sharp and I like its shape, but it’s missing quite a few glyphs and it’s becoming harder to use as screen resolution and DPI increase. I looked for new fonts to try the past few weeks. There are many font for programming, here’s my selection of monospaced scalable fonts:

Comparison of monospaced fonts: What Are The Best Programming Fonts?

It took me a while to figure out how to dump the content of UDP packets with tcpdump/Wireshark. Here’s how you do it:

# Dump all the UDP traffic going to port 1234 into dump.pcap
$ sudo tcpdump -i lo -s 0 -w dump.pcap udp and port 1234
# Then we print the data on stdout
$ tshark -r dump.pcap -T fields -e data

This will print all the data in hex-encoded form, 1 packet per line. You’ll have to decode it to get the data in binary form. The following Python program does that:

import sys

for line in sys.stdin:
    # Binary data
    data = line.rstrip('\n').decode('hex')
    print repr(data)

Last week I installed Bootstrap as scratchpad’s CSS, before I had a minimal CSS with normalize.css, but I couldn’t get it to display the feed “correctly” on my Smartphone: the article block occupied only half the screen. I had to readjust the zoom-level after loading the page to get it ‘fullscreen’.

I’m unhappy to add a dependency like that to scratchpad, but I can’t be bothered with CSS anymore. The folks from Twitter like to deal with that ridiculous shit, so I figured I should use the thing that just works, no matter how much I dislike the idea. At least the typography is a nicer and the color are prettier.

I had a few problems with KiTTY a clone a PuTTY, and tmux, a terminal multiplexer: pane separators where displayed as ‘q’ or ‘x’ instead of lines. It turns out it’s a PuTTY problem, according to tmux FAQ:

PuTTY is using a character set translation that doesn’t support ACS line drawing.

I had this problem for a while, and I didn’t manage to solve the problem with my old bitmap font: Terminus. Maybe because the font is missing the glythes to draw lines. There were actually a few problems:

  1. My old font didn’t have line-drawing glyth (?)
  2. PuTTY character translation problem
  3. Encoding wasn’t properly detected by tmux

I recently switched to a new font: Source Code Pro by Adobe, which allowed me to fix the problem with the missing glyth. I also had to tweak tmux & KiTTY a little bit. In KiTTY’s settings, in the Windows > Translation Category: set the remote character set to ‘UTF8’, and select the unicode line drawing code points. Make sure your font support line-drawing code point. Start tmux with the -u option to tell it to use UTF-8, and you should be good to go.

I instantly became a fan of micro-credits when I first heard about the idea. It seemed to be a perfect way to get people out of poverty: loans to small businesses in under-developed part of the world. Business, exchange, trade, and the ability to sustain oneself is what lift people out of poverty: not charities, and government hand-outs. Micro-credits fit nicely my view of the world: individuals stepping up to raise their standard of living with help from non-governmental bodies.

Planet money posted an article about micro-credit, or more precisely about studies to determine how effective micro-loans are. It turns out it was a lot of hype, and not a lot of result. It doesn’t look like micro-loans improve much the standard of living of those who benefit from it.

It’s disappointing, but after thinking about it for a while it seemed foolish to think that small sums of money here and there could have a significant impact on the lives of those who live in poverty.

With Unix becoming more and more ubiquitous, the POSIX Shell is the de-facto scripting language on most computers. Unfortunately it’s difficult to write proper scripts with it: few really know it, and there’s no much good documentation available: Rick’s sh tricks is one of the best resource I found about pure POSIX Shell scripting.

After reading The Best Water Bottles on The Wirecutter –a review website–, I got myself the 800ml Klean Kanteen Classic. It’s exactly what I wanted: a water bottle with a mouth opening wide enough for ice cubes. That should help me get over my diet soda addiction: I made myself lemon water this afternoon, plain water has a difficult time competing with diet soda, but lemon water is another story… :)

I’m in relatively good shape for a programmer. I’m not overweight, but I have a little bit of extra fat, I usually shave it off during summer. As I got older, I noticed that it’s getting harder to get back in shape. After reading Hacking strength: Gaining muscle with least resistance & Least resistance weight loss by the excellent Matt Might, I started using MyFitnessPal, a food diary application. It has great ratings on Google Play Store with millions of downloads, and I managed to log a full week with it without going crazy.

Getting up early in the morning has always been a problem for me. It got better last year mostly because we have scrum stand-up meetings at 9:45am every morning. I’d like to start work earlier, especially with summer coming up: I want to get out by 5:00pm to enjoy the sun and the outdoors.

I started tracking my time of arrival at work using joe’s goals, a simple log book. It’s quick and easy to get started and to use, I like it.

I just had a weird problem with sh’s syntax, the following function didn’t parse:

f() {
    return { false; } | true

f should return 0: a successful exit value. The problem is that using return like that is invalid. Return expects a normal shell variable, not a list or compound command according to the POSIX spec. The solution is simply to do something like that:

f() {
    { false; } | true

This will return the last command’s exit code, in that case it’s true: so the value is zero. It’s still difficult to find good information about shell-scripting on the net: that I though I’d throw that here.

Google announced that they will shutdown Google Reader, their RSS feed reader this summer. I’ve been using it for many years now, feeds and Blogs aren’t trendy anymore: the cool places to post content are social networks. I though about letting Reader go without finding a replacement. RSS feeds are a better source of quality content than social networks, but checking Reader every day takes between 10 to 20 minutes.

I still decided to look for an alternative, I briefly tried Newsblur demo. I didn’t like the interface: it felt a little bit too cluttered, and the site wasn’t working well when I tried it. Then I tried Feedly for a week, and so far I’m impressed by it. The interface is clean and minimal, while still providing all the things I had in Reader and more.

The shell job control can be good alternative to terminal multiplexer like screen or tmux. Here’s a small example:

$ vi file

[1]+  Stopped(SIGTSTP)        vi file
$ jobs
[1]+  Stopped(SIGTSTP)        vi file
$ vi other

[2]+  Stopped(SIGTSTP)        vi other
$ jobs
[1]-  Stopped(SIGTSTP)        vi file
[2]+  Stopped(SIGTSTP)        vi other
$ fg %1  # resume 'vi file'
vi file

[1]+  Stopped(SIGTSTP)        vi file
$ jobs
[1]+  Stopped(SIGTSTP)        vi file
[2]-  Stopped(SIGTSTP)        vi other
$ fg %2  # resume 'vi other'

I used Todoist a few years to manage my TODO list. Then I gave up on electrical TODO lists and Todoist, it just didn’t seem to work very well because I needed to be near a computer at all time. I switched to paper, but it didn’t work either, at least for my personal use. I forgot the notebook constantly, and sometime didn’t check it for weeks. Now when I get home after work I write my tasks for the evening on a small piece of paper: this works rather well, but it’s only good for short-term tasks.

Since I got a Nexus 4, I can run modern apps without too much frustration. I’m trying a few TODO list apps to see if any of them is any good. I tried Todoist again: the web app, and the Android app. I didn’t like it: I remember Todoist being more keyboard driven, all the shortcuts I learned back then seem to be gone or not working. A lot of features are reserved for paid users, I didn’t like it enough to pay those $30. Moving on.

I looked for a better alternative and found Wunderlist. It looked nice and simple. After a week of usage, I’m not really convinced. It doesn’t do nested tasks, or more precisely you can only have 2 levels of nesting, and adding subtasks isn’t super intuitive. I use nested tasks quite heavily, that may be a deal breaker. On the other hand the interface seems to be more or less the same everywhere, which is a nice plus. I’ll keep using it for a little while, and probably get rid of it, but I want to give it an honest try.

I liked Emacs Org-mode back in the days when I used Emacs. It could be an interesting option.

Last Christmas I got the book Breakthrough Rapid Reading. I haven’t read it yet, but after listening to Skeptoid’s podcast on speed-reading I may not read it at all.

According to the Podcast the only good way to improve reading speed is… Wait for it:

To read faster, concentrate on reading slower, and read more often.

All those things they teach you with rapid reading are gimmicky: you just trade comprehension for speed. Speed reading doesn’t look that attractive when you know that… I’ve put the book at the bottom of my reading pile: I’ll probably never read it.

My biggest problem when reading is distraction. I read a few paragraphs, I get bored, and I look away for a few minutes. When I come back to the book, I’m out of it, and I have to re-read those paragraphs to get back into it. When that happens I estimate it takes me triple the time to read and comprehend this chunk of text.

Meditation is supposed to help focus, but I think to thing I need the most is sleep. I’m still pretty bad at going to bed at a reasonable time.

Reminder for later: How to manage FreeBSD ports.

FreeBSD handbook:


Man page:


Configure port:

$ make config
$ make rmconfig  # Delete old config
$ make config-recursive  # Configure port & all its dependencies
$ make config-conditional  # Configure port & deps without options

I love Nick Johnsonz’s Damn cool algorithms series, where he writes about new or unusual algorithms. I just finished the post about Homomorphic hash. It’s a cool idea, but it’s based on modular artithmetic like RSA, which is rather slow even on modern computers. I wonder if an algorithm based on Elliptic curve cryptography would be more practical.

Idea for later: A web based carcasonne like game with a bunch of “credible” AI playing it to get it started. this would “solve” the chicken-eff problem that most multiplayers game have: to attract users you need to already have a bunch of them. The challenge would be to have an AI good enough to be midly challenging to human player.

Apparently Disqus –an online commenting service for website– decided unilaterally to put Ads on their users/customers websites:


I liked Disqus, but this is a bad move on their part. I’m not going to use their service after that…

Note to self: apparently boiled eggs are easier to peel when you add baking soda to the water. I should try that.

The W3C doesn’t seem to be a big fan of Twitter. From the HTML5 spec:

The following extract shows how a messaging client’s text entry could be arbitrarily restricted to a fixed number of characters, thus forcing any conversation through this medium to be terse and discouraging intelligent discourse.

<label>What are you doing? <input name=status maxlength=140></label>

On asserts

It’s common to see C/C++ projects disable asserts when building releases. The book Cryptography engineering argues that it’s a mistake: production code is exactly the place where assertions are most needed. That’s where things should never go wrong, and if they do we shouldn’t sweep the problem under the rug.

Patrick Wyatt an ex-Blizzard developer who worked on the early Warcraft, Diablo, and StarCraft came to the same conclusion after working on Guild Wars: it’s OK to “waste” a little bit of CPU to make sure production code runs correctly.

Assertions aren’t that expensive, we really shouldn’t remove them in production. These days speed is rarely an issue while correctness is always an issue.

Do-It-Yourself Dropbox based on Mercurial:


set -o errexit  # Exit as soon as a command returns an error

hg pull --update
hg commit --addremove --message "Update from $(hostname)" $@
hg push

How to use it:

$ hg clone <remote repo> ./shared
$ cd shared
$ cp ..../sync.sh .  # sync.sh is the script above
$ touch file1 file2
$ ./sync.sh  # This will add file1 & file2
$ rm file2
$ ./sync.sh  # This will delete file2
$ touch file3 file4
$ ./sync.sh file3  # This will add file3 but not file4

I also have a script update.sh that doesn’t synchronize remotely:


hg commit --addremove --message "Update from $(hostname)" $@

If you’re using an editor that writes temporary files in the directory like Vim or Emacs don’t forget to add the relevant regex in the directory’s .hgignore.


If you have difficulties sleeping at night because of the noise, or if you work in an open-space and can’t focus for very long: you should give earplugs a try. They require a bit of adaptation, but after around 10 hours of use the initial discomfort almost vanish.

I tried 4 different types of foam earplugs those past 10 years. Initially I used EAR foam earplugs for many years. I tried the classic, and the neon yellow. The classic weren’t great: they’re a bit rough against the skin, which make them rather difficult to wear at night and for long periods of time. The neon yellow were a bit softer and isolated better, I used them for 4 years after I ditched the classic.

6 months ago, I decided to try some new ones. I ordered 20 pair of Howard Leight MAX & Hearos Ultimate Softness Series after reading reviews on the web (links below).

The Howard Leight MAX are great for work. They fit snugly, and isolate well. They are a notch above the EAR neon yellow in comfort and isolation. They aren’t that great for sleeping: if you wear them for more than 2 hours, they start hurting you ear canal a bit. For sleeping the Hearos Ultimate Softness are great: they don’t isolate as well as the others, but when you’re sleeping this isn’t as important. What’s important when you’re sleeping is comfort, and the Ultimate Softness are the most comfortable earplugs I ever tried. After a night of sleep your ears wont hurt a bit. I’m planning to order 100 pairs of those new ones: focus and sleep are 2 things I can’t afford to lose: I need all the help I can get in life.

Reviews links:

New year resolutions were trendy last week, I like to be fashionably late at parties… I wont set ambitious goals for 2013, like winning an Olympic gold medal, or having sex with Twin Swedish super-models. I’ll go for small manageable goals for those first 3 months:

Ideas for the rest of the year (1 per month):

Great blog post about Content centric networking or CCN:


I often wonder what will come after the current “Internet”. CCN is a good candidate to replace the whole or parts of the IP/TCP/HTTP stack, and it can run on top of the existing stuff: IP, IP/TCP, etc… unlike say IPv6.

Awesome isn’t anymore. It’s overused, everybody knows it, as of today the first result from Google for Awesome is the Urban dictionary’s definition:

An overused adjective intended to denote something as “cool” or “great” but instead winds up meaning “lame.”

Instead of awesome, use great: Alexander the Great is better than Alexander the Awesome. The worst with the overuse of the word awesome everywhere: it’s almost always used inappropriately. It comes from awe: a feeling of fear and wonder.

Article about memory corruption, and how to detect if it’s a hardware or not from Antirez, the guy who started Redis:


I love his final idea: Get the kernel to check the memory incrementally.

I tried a few new recipes lately. Fall is great time for some soup:

The white bean / chicken chili tasted OK, but it’s a bit long to cook I find.

I really like the black bean soup, it’s faster to make and tasted better. I spiced it up with some red thai chili in the relish. I’m not big on spicy food, but most soup are bland, and they often need a bit more punch.

Japanese toilets:


I have to try one.

Procrastination: I’ve finally filled the paperwork for a new savings account. I got the form in April, I looked at it, put it on a shelf, and left it there for more than 6 months. I didn’t forget about this form, I just left it, while it was at the back my mind during those 6 months. It took me literally 2 minutes to fill it out. Now I have to send the letter, and if I don’t do it right away I may leave it in my desk for weeks.

I’m like everybody: an every day life underachiever.

Frank and Oak is an online menswear maker. They release a new collection every month, and sell it exclusively on their website. They target guys with slimmer bodies. I ordered a shirt and a blazer, it was quickly shipped and delivered.

The shirt was a good surprise: it fits well out-of-box. I’m a skinny guy, the shirts that fit me best are the ultra-slim 1MX from Express. Frank and Oak’s shirt isn’t as snug, but it’s still one the best fitting shirt I had. For $45 you’ll get a decent shirt, and with the money you saved: get it tailored just right.

The blazer is okay. It’s a little bit narrow around the shoulders, but overall it’s pretty good considering it was only $50. It took me a while to realize the side-pockets were sewed, they looked like those fake pockets you get on low-end blazers.

I’ll probably order more from them.

I used to have a Wrap –a tiny x86 computer– as a router. It wasn’t doing much routing though, since it only had my desktop connected to it. I messed up with it 1 year ago while flashing the firmware, broke it, and never managed to get it to work again.

I just ordered an Alix 2d13 as a replacement. It’s nice upgrade, with a USB port, and an IDE connector. I’m planning to install OpenBSD 5.2 on it. It will be released tomorrow, right before I get the new hardware.

It’s an expensive toy: $300+ not including shipping, but I have an open platform I can hack and play with. I tried to use an old wireless router with OpenWRT, but the wireless signal was pretty bad.

Creating a queue in MySQL isn’t a great idea as highlighted in this article: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you. Yet it’s possible to create a relatively-efficient queue as long as you avoid SELECT FOR UPDATE. I had to create one for work a little while ago.

Here’s the schema for such queue:


The table queue is only used to lock items, and mark them as them as done. You can store data in the queue table, but I’d recommend to store it in a separate table to keep the queue table relatively small.

To lock the next item in the queue:

UPDATE queue SET id = @last_queue_id := id, available = FALSE
    WHERE available = TRUE ORDER BY id LIMIT 1

The key part is id = @last_queue_id := id: this will mark the next item with available = FALSE and set the user variable @last_queue_id to its ID. You can then get it with:

SELECT @last_queue_id

Once you’re done with the item, you delete it from the queue:

DELETE FROM queue WHERE id = @last_queue_id AND available = FALSE

The available = FALSE clause isn’t necessary, but I like to keep it just to be extra safe.

Vancouver is expensive, save money on booze: http://www.vancitydrinkspecials.ca/

Last night I ironed 5 shirts & 1 polo in less than 1 hour. I followed the instructions from the videos I posted yesterday. That’s pretty good considering that I haven’t ironed for 5+ years.

That’d work out as 1h30m weekly if I include pants. It’s too long, I’ll try to lower that to 45m with the same “quality”.

My life is very exciting.

I try to look sharper those days, that’s why I started regularly ironing shirts and pants. I used to iron weekly, but stopped after I moved to Vancouver. Back then I never really tried to improve my ironing technique, it was a mindless chore not worth perfecting.

That was a mistake, I should try to perfect my ironing technique BECAUSE ironing is boring and time-consuming.

There are quite a few videos on how to iron stuff on Youtube. Those 2 are the best I’ve found so far:

Impressive video presentation of acme(1), a “text editor” written by Rob Pike. If you saw the introducing Xiki, it may look familiar. Many ideas from Xiki are implemented in Acme. The idea that text IS the interface is pushed quite far in Acme.

I’m a Vim user, I like to use the keyboard only to drive my computer. Acme takes a completely different approach: The mouse is used everywhere, the mouse does more than in regular programs: Button 2 execute the selected command for example. I never used it, but I’m very tempted to look into it. My main grief with Vim is that is difficult to script: VimScript is yet another language you got to learn. The programming interface in Vim seems to be kind of ad-hoc and looks difficult to learn and use.

Acme seems to be easier to interact with programmatically. It may very well be my next text editor.

I posted on scratchpad a TED talk about aging, and how living past 100 years may be common in a not-so-distant future. I wasn’t convinced, but the talk was cool and informative nonetheless.

It turns out that some guy named Edward Marks did his homework and looked at life expectancy numbers and where we’re headed if we keep going at the same rate. It looks like the dream that average people could live past 100 years is indeed a just a dream. We die less, but progress is slowing down.

Cooking tip: Apparently lemon is a cooking silver-bullet

Steven Johnson is the author of 2 books about creativity: Where Good Ideas Come From, and The Invention of Air. He wrote an article about what he calls a Sparkfile, which is just a list of ideas.

He writes down every idea he has in this Sparkfile, and once in a while revisit it to see if there’s anything of value, or ideas he could combine.

That was more or less what my scratchpad was for. Blog & note taking kind of thing. I need to post more ideas in here.

Another great post by Rob Pike about UTF-8’s creation.

I posted a while ago links to a few python projects worth checking out.

I tried them all for a bit, and I really like PyRepl: an alternative to the standard Python interpreter readline-interface.

It crashes less than bpython, it’s lighter than IPython, and it’s written 100% in Python. What’s not to like? Well it crashes from time to time when I use the arrow keys, kind of like bpython. I haven’t managed to reproduce the problem consistently yet, mostly because it’s rare.

You should use PyRepl for its better completion and coloring, it’s a pretty good alternative to the bloated IPython.

Another great talk from Rich Hickey: Deconstructing the Database

He talks about Datomic a new database with an innovative architecture.

I’m making pizzas for dinner those days.

I used to coat the top of my pizza with a little bit of generic tomato sauce. I saw a a pizza sauce recipe on my favorite cooking blog last week, and decided to try it. It changes everything: because the previous sauces weren’t thick enough I had to put a ton of toppings on my pizzas. This sauce is thicker, and much tastier than what I used. Result: less topping, less work, and yet the pizzas taste better.

The Economist had a good editorial about Money last week.

I found it fascinating that money gets routinely reinvented by people who don’t have access to “regular” currency. Bartering creates such a big transaction cost that we’re almost hardwired to come up with something better.

Another surprising part is that metal money (coins) are likely a state invention. It turns out that the private sector naturally uses things with real “value”, like rice.

I tried DuckDuckGo a few times since it launched. Every time I wanted to like it, but it had a few shortcomings: the results weren’t quite as good as Google, or it was a little bit too unfamiliar.

I tried it once again this week, and I think it’s now good enough for me to switch to it permanently. Google is full of spam right now, it looks like Google refuses to ban content farms like Demand Media which operate eHow & Cracked. This switch to DuckDuckGo has more to do with Google Search deteriorating than DuckDuckGo giving me something new.

Yesterday I tried this recipe:


It was easy enough to prepare, but I made a big mistake: I didn’t taste the pepper before integrating it into the vegetable mixture. It was spicy, too spicy. Otherwise I recommend this recipe.

Side note: Fish is easy to cook, healthy, tasty, and often better for the environment than meat.

Simplicity Matters by Rich Hickey is a great talk about simplicity.

I’m glad I took 40 minutes to watch it, I’ve learned and understood so much.

I started programming in C 15 years ago, but I don’t consider myself an expert or anything like that. There are just too many aspects of C I’m not comfortable with. One of those is the Preprocessor.

Something I never fully understood is the STRINGIFY macro hack. The preprocessor has a “#” operator that turns its argument into a string.

#define STRINGIFY(x) #x
#define HELLO(x) "Hello " #x

STRINGIFY(foo) is expanded to “foo”, HELLO(world) is expanded to “Hello ” “world”. So far, so good, but when you try to stringify another macro it doesn’t work as expected:

#define STRINGIFY(x) #x
#define FOO bar
STRINGIFY(FOO) /* This will output ... "FOO", not "bar" */
STRINGIFY(__LINE__) /* This will output "__LINE__", not "4" */

If you look for a solution on the interweb, the answer is usually to use another auxiliary macro, and it “magically” works:

#define STRINGIFY2(x) #x
#define FOO bar
STRINGIFY(FOO) /* This wil output "bar", why?!?!? */

Why does that work? Because the preprocessor doesn’t expand macro’s arguments, but the result of the expansion can be expanded afterward. Here’s what happen:

  1. STRINGIFY(FOO) is expanded to STRINGIFY2(FOO) because of #define STRINGIFY(x) STRINGIFY2(x)
  2. FOO is expanded to bar using #define FOO bar, we now have STRINGIFY2(bar)
  3. STRINGIFY2(bar) is expanded to “bar”

Rob Pike is a pretty good photographer.

I’m exited by 2 new languages at the moment: Go and Rust.

Rust is still pretty much in development, but Go is already stable. There are a few introduction to Go floating around, but the latest one by Russ Cox just shows you the good stuff:

A tour of Go

I’m trying to get over a Diet Coke addiction. I drink 3 or 4 cans of Diet Coke each day. There’s salt and caffeine in Diet Coke to make people pee, feel thirsty, and drink more.

Going to the bathroom every 2 hours is not pleasant and may have armful health effect in the long run.

There’s 45mg of caffeine, and 35mg of salt in a Diet Coke can. A can of Diet Pepsi contains only 36mg of caffeine, and the same amount of salt. From now on I’ll drink Diet Pepsi: 9mg isn’t a big reduction of caffeine, but it’s a step in the right direction.

I like cycling caps, I like the snug fit. The short visor pointing down protects well against the sun, the wind and the rain. I have a wool one from Walz Caps, I had it for 3 months now and it instantly became my favorite hat. I’ll get rid of my regular caps, and get 2 more of those.

Cycling caps are kind of hipster’s hats, but I can deal with that.

Try one once, you may like it ;)

Finding pants that fit was a problem for me. It takes a while to find one that fits just right: each brand has a slightly different cut, and the right combination of hip / inseam size is not always available. I usually ended buying pants that were slightly too long or too short. There’s an easy solution to this problem: tailors. For $10 or less you can get your pants shortened just right.

I’m embarrassed I never went to a tailor before: in retrospect it seems like such an obvious thing to do.

Captain Clueless, signing-off.

Small python script to calculate a file’s entropy. It reads its input from stdin.

#!/usr/bin/env python

from collections import defaultdict
from math import log

def entropy(input):
    alphabet = defaultdict(int)
    total = 0 # How many bytes we have in the buffer
    buf = True
    while buf:
        buf = input.read(1024 * 64)

        total += len(buf)
        for c in buf:
            alphabet[c] += 1

    if total == 0 or len(alphabet) < 2:
        return 0.0

    entropy = 0.0
    for c in alphabet.values():
        x = float(c) / total
        if x != 0:
            entropy += x * log(x, len(alphabet))
    return -entropy

if __name__ == '__main__':
    import sys
    print entropy(sys.stdin)

If you’re looking for narcissist people, there’s a new site that’s referencing them:


Cool infographic detailing bike parts.

I planned to write a funny essay about how I’m a nerd who can’t dress, that I finally realized the obvious, and would take better care of my look from now on. It turns out that writing funny essays takes a long time, and it wouldn’t be that funny anyway.

In May I decided to get into that “style” thing. I grew a light beard back in March, and it altered the way people perceive me more than I expected. No need to show ID at the liquor store anymore, at 32 it was about time…

Others look at you a lot, way more than they listen to you. Look is a quick and relatively reliable way to weight someone. A guy wearing a nicely cut suite is not a homeless guy to anyone, someone wearing skinny torn jeans and a dirty T-shirt ‘Punk not dead’ is probably not a banker.

I want to get better at looking good, and I want to talk about it. I’ll post more later.

Too few updates lately… I have been pretty busy at work. Here’s something I’ve learned today. In C, when I wanted to initialize an array to all zeros I used memset. But there’s a simpler way:

int array[3] = {0, 0, 0};

OK, that’s nice. But what if the array is really big? There’s a shorter version with the same effect:

int array[3] = {};

When you omit a parameter in an initializer, it automatically defaults to the type’s zero value. If you want to initialize an array without specifying all the elements, you can do something like that:

int array[10] = {1, [3] = 3, [8] = 2};

This will produce an array like that:

[1, 0, 0, 3, 0, 0, 0, 0, 2, 0]

See GCC’s doc about Designated Initializers.

I bough a new bike a few months ago: a fixed gear. After a week of getting used to it, it feels great. It feels like being connected to the ground: it’s easy to adjust your speed, most of the time there’s no-need for brakes: you can slow down with just the pedals. I’m going to keep the brakes on for a little while, better be safe than sorry ;)

I still have my old bike. It’s going to be my “rainy days” bike. I plan on converting it to fixed gear sometime after this summer.

I’ve been looking for a cheap track frame to replace my old road frame. I found this one for $169.

libtomcrypt is a pleasure to work with: the code is clean, readable, and things are well laid-out. One the few things I disliked was how ECC keys are represented internally:

typedef struct {
    void *x, *y, *z;
} ecc_point;

/** An ECC key */
typedef struct {
    /** Type of key, PK_PRIVATE or PK_PUBLIC */
    int type;


    /** The public key */
    ecc_point pubkey;

    /** The private key */
    void *k;
} ecc_key;

If type is PK_PUBLIC, the private component of the key should probably be NULL. I think this is suboptimal and potentially confusing. It seems to me that the following would be better:

struct ecc_public_key {
    void *x, *y, *z;

struct ecc_private_key {
    struct ecc_public_key public;
    void* k;

Introducing 2 different types for public and private keys allows us to be more specific with our type requirements. For example the function ecc_shared_secret looks like that:

int  ecc_shared_secret(ecc_key *private_key, ecc_key *public_key,
                       unsigned char *out, unsigned long *outlen);

A new API could enforce the key’s type more easily:

int  ecc_shared_secret(const struct ecc_private_key *private_key,
                       const struct ecc_public_key *public_key,
                       unsigned char *out, unsigned long *outlen);

This way you can get rid of checks at the beginning of some functions, like this one:

/* type valid? */
if (private_key->type != PK_PRIVATE) {

Now the key type is explicit, private keys will only be struct ecc_private_key, public ones struct ecc_public_key. If you want to be able to pass both keys types, you can do something like this:

int ecc_some_function(struct ecc_public_key* public, void* private, ...);

And pass the private component of the key manually.

Designing a good API is hard. Even the little choices can be difficult. Let’s take a function to decrypt AES blocks for example, this function will consume a buffer 16 bytes at a time. Here’s what such function would look like:

void decrypt(const void* input, size_t nbytes);

(There’s no output parameter, we’re just looking at the input here)

input is a pointer to the buffer we’re working with, nbytes is how many bytes to read from the buffer.

The function consumes blocks of 16 bytes, so what happens when nbytes is not a multiple of 16? Should we silently ignore the few extra bytes? Should we have an assert(nbytes % 16 == 0)? Maybe we could specify how many blocks to consume? But then the API’s user would have to remember to divide the buffer size by 16.

I don’t know what the good answer is there.

Reading or sending data on a TCP socket looks simple, but it can be tricky. read(2) & write(2) don’t have to consume the whole data they get. If you call read on a socket requesting a billion bytes, you’ll probably get less.

If you need to read a set number of bytes, you’ll have to repeat calls until you have all the data you want. Here’s an example of such loop in Python:

def readall(sock, count):
    r = bytes()
    while count:
        x = sock.recv(count)

        if x == '': # End of file
            raise ...

        r += x
        count -= len(x)
    return r

In Python sockets have a sendall method which does more or less that, but there’s no recvall method. There’s another method: the socket.makefile() function creates a file-like object for the socket. With this file-like object there’s no need to loop to get the whole buffer back:

sock = socket.create_connection(...)

f = sock.makefile()
x = f.read(1234) # Will return a 1234 characters long string

You must make sure buffering is enabled for this to work.

PS: NOTE that calling read/write on a socket is basically the same a calling it with recv/send without flags.

Reminder to self.

The Open Group Base Specifications Issue 7, also known as IEEE Std 1003.1-2008, also known as POSIX.1, is publicly available here:


tmux, the terminal multiplexer from heaven, can copy-paste!

Here’s how to use it:

  1. C-b [ start the copy
  2. Move around a stop on the character where you want the selection to start
  3. Press Space to start selection
  4. Press Enter to finish selection
  5. C-b ] paste the copied text

I finished the Kobold Guide to Board Game Design last night. I never tried to create or modify a board game, but I think what makes board games great also applies to other kind of games, especially online browser games.

The book is a collection of articles by various game designers sharing their ideas and stories. The authors include: Richard Garfield (Magic: The Gathering), Steve Jackson (Munchkin), and Dale Yu (Dominion). The book is easy to read and mostly jargon-free.

I expected the book’s content to be more analytical than it is. Most articles turned out to be practical and concrete, and it’s great. I wish there was a little bit of ‘theory’, but board game designers don’t seem to use math & statistics to balance and design their game.

Great read overall: highly recommended, five stars, and all.

Rob Pike: The byte order fallacy:



Don’t just make random changes. There really are only two acceptable models of development: “think and analyze” or “years and years of testing on thousands of machines”. Those two really do work.

If you can ignore the irritating InfoQ video/presentation player, this Rich Hickey’s interview is full of fresh and innovative ideas:


John Carmack posted on Twitter:

I can send an IP packet to Europe faster than I can send a pixel to the screen. How f’d up is that?

Somebody asked about that on superuser, John Carmack went on to explain why it was like that. He didn’t just cross reference a few sources to come to his conclusion: he measured, he checked his assumptions with experiments. He acted like a scientist, adapting his mental model to the world.

While hacking on my dwm I noticed this line in dwm.c:

while(running && !XNextEvent(dpy, &ev))

Notice the ‘not’ before XNextEvent. I wonder why it’s here, as far as I can remember XNextEvent isn’t supposed to return something. A quick look at the manual helped, but didn’t solve the problem:

int XNextEvent(Display *display, XEvent *event_return);

The function returns an int but there’s no explanation on what this return value is. In X.org’s libX11’s source code XNextEvent always returns 0.

I imagine that somewhere there’s a version of the Xlibs that return a non-zero value when there’s an error, maybe there’s a doc somewhere explaining what this means, but I couldn’t find it. Or maybe it’s simply an undefined behavior, an error…

When I connect to a remote host via SSH, I like to start a new Tmux session or re-attach to an existing one. Before that’s how I was doing it:

if tmux has-session
    tmux attach-session
    tmux new-session

There’s something much simpler: tmux attach || tmux new

Things that would be nice to fix with URLs.

  1. Drop the //, http:google.com is shorter and easier to read. Tim Berners-Lee regretted putting those 2 extra characters when he started the web.

  2. Inverse domain names. Example: sub.domain.com, com is the top level, and sub is the bottom level: domain names put the top level last. Path in URLs do the opposite, the top level is first:


With those principles in mind, new URLs would look like that:


Create chaos in your C program with just 1 line:

#define struct union

Aubrey de Grey: Why we age and how we can avoid it

I don’t buy it, pushing death back by 30+ years sounds too good to be true. It’s nonetheless one the best TED talks I’ve seen.

Are Bananas Really as Bad for you as Cookies?

I’ll certainly not switch from bananas to cookies for my mid-afternoon snack, but switching from bananas to apples might be good. Starting today :)

A long, but good and detailed article about RPython: Fast Enough VMs in Fast Enough Time.

Alice is a librarian: she has a house full of books. Bob likes books, he wants to read as many as he can, and is willing to pay. Alice –being of savvy businesswoman– wants to open her library to the public for a fee. She wants to maximize profits; Bob wants to maximize the number of books he gets, up to the limit of what he can read, and he wants to minimize the money he spends.

The books are a limited resource, if Bob takes out all the copies out of the library Alice can’t have more clients, she needs to manage her library to make sure customers don’t abuse the system.

We’ll consider everything else to be equal. All books have the same value, and they all take the same time to read.

Let’s consider 2 different business models:


Alice gets $20 from Bob, and he gets free access to the library for the rest of the month.

To increase her profits Alice needs more customers, that’s the only way; she can’t charge Bob more. Since the number of books is limited, more customers means less books per customer. Every time a customer takes out a copy from the library it reduces Alice’s potential profit. To maximize her profit Alice should minimize its resource usage: limit the number of customers, or how many books a customer can take each month. Bob tries read as many as he can: one extra book doesn’t cost him anything, without any quota he can get more than his fair share.

Alice’s goal is not aligned with Bob’s goal. Alice wants to reduce the number of books Bob reads, Bob wants to maximize it.


With a price of $1 per book, Alice doesn’t really care how many customers she has. She wants to rent as many books as possible. 1 or 100 customers doesn’t really make a difference to her bottom-line.

Here our 2 characters’ goals are aligned, Alice wants Bob to read as much as he can. Bob can choose how much he reads and spends.

So What?

It’s clear that the pay-per-use formula works better than all-you-can-read.

So why do we use the worst model for our Internet access and phone? Why don’t we try to align the goals of carriers and customers?

Getting connected is not all about bandwidth; there are fixed costs than can’t be easily covered by a pay-per-use model. A price per Gigabyte with a minimum price per month could lower our bills while holding back the freeloaders.

That’s not going to happen anything soon though. Many have interests in keeping the old fixed price per month going. Internet and phone carriers like it, because it’s an easy way to maintain and increase their profits. Customers will pick up plans that exceed their needs, and pay more than they should as a result. The price of bandwidth tends to fall steadily over time, but carriers don’t lower their price very often. Freeloaders also want to keep the system the way it is, customers who use most of their quota are the ones getting the best value from their broadband access.

If we want broadband to be more ubiquitous and cheaper, we need to treat it as a real commodity, like water or electricity.

I’m reading http://0x10c.com/doc/dcpu-16.txt, The DCPU-16 Specification. A small computer for Notch’s next game. Notch is the guy who built Minecraft.

Instructions are simple 16 bits words with the following format (bits):


oooo is a 4 bits op code; a’s, and b’s are two 6 bits operand. The instruction format is a little more complicated than that, but that’s roughly it.

To me this looks like a pretty good candidate for a perfect hash function like the ones created by Gperf. What kind of tree do we have?

The 4 bits values spawn 16 branches. 6 bits value spawn –according to my quick glance at the spec– 12 branches. That’s 16 * 12 * 12 = 2304. Quite a bit more than I expected. Gperf might not be such a good idea after all.

Cooperation and Engagement: What can board games teach us?

Google just announced Account Activity, a new feature that let you see what you’ve done on its services every month. Big companies like Google and Facebook know how valuable personal data is: it helps get more customer, and it can be sold more or less directly to advertisers.

It’s also useful to their customers. I suspect my personal data is most valuable to me, I’m sure I can get more use of it than Google or advertisers. This extra data is another reason to use Google’s services. Well played Google, well played.

Linux 3.3: Finally a little good news for bufferbloat

From 7 Years of YouTube Scalability Lessons in 30 Minutes:

You have to measure. Vitess swapped out one its protocols for an HTTP implementation. Even though it was in C it was slow. So they ripped out HTTP and did a direct socket call using python and that was 8% cheaper on global CPU. The enveloping for HTTP is really expensive.

Not a surprise for me, but I guess it would be for most people. HTTP is not a good general purpose protocol, it’s not even good at doing what it was designed for. I try to avoid HTTP like the plague, but it’s difficult to go against the herd of web-developers who think “HTTP is th3 4w3s0me!”

I haven’t maintained my bike for a year, last week-end it got a long overdue tune-up.

Saturday I went to the Pedal Depot a small bike workshop, where you can rent stands and tools to fix your bike, with friendly employees to help you when you’re stuck or have a question, they also sell second hand parts.

There’s another bigger workshop nearby on Main St. I went there 3 years ago, I was able to buy and assemble my current bike for around $200. It took me almost 10 hours over 2 days to get it all done, it was long, but it was fun, and I learned a lot.

The Green Biscuit seems like a good way to improve ice hockey stick handling during summer. I play roller hockey with a ball during summer, when I go back to ice hockey with a regular puck it takes a few weeks to get used to the puck again.

Something to look at again in a few months.

I realized this week-end that I had an artificial scarcity problem with bus tickets.

My old process was:

  1. Realize I’m out of tickets
  2. Go to the store and buy 1 booklet of 10
  3. Repeat after a few weeks.

The new one is:

  1. Realize I’m out of tickets
  2. Go to the store and buy 10 booklets
  3. Chill for 6 months because I now have 100 tickets

I don’t know if the post about nonces was really clear.

To encrypt the counter, you need to put it in a buffer the size of a block. For example to encrypt the counter 1234567890, you’d have something like that (notation little-endian):

0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0x00 0x00 0x00 0x00 0x49 0x96 0x02 0xd2
                    ^^^^^^^^^^^^^^^^^^^ 1234567890

You encrypt that block, then you use the resulting encrypted block as initialization vector to decrypt the first block of the encrypted message.

I wanted to use only unixy tools to build scratchpad, but it turned out to be cumbersome. The HTML and Atom generators are written in Python: 103 lines of code as of today.

I tried to use rc and sh, but it was quite inconvenient to use. After a while I decided to fall back to Python.

These days I read Cryptography Engineering. I just started the part about block cipher modes, that’s where I learned about Nonces.

Using nonces with block ciphers is a good way to minimize the space taken by Initializations Vectors or IV. Instead of sending an additional block with the IV, you associate a number (counter) with each messages. The counter doesn’t necessarily need to be transmitted with each messages, it can be implicit: for example the first message could have 0 as nonce, the second 1, etc…

Then you encrypt the counter with the raw block cipher and use the result as the IV for the 1st block. Simple and elegant, I really like this crypto ‘trick’.

From I’ve read so far I highly recommend Cryptography Engineering. It’s a pleasure to read, and you might learn a thing or two.

I just played another game of Dominion. I’m just getting started with it, but I can already understand why it’s such a popular game:

Looking forward to play more Dominion! :)

I usually clone Virtualenv’s repository to use it. Here’s a quicker solution:

curl -s https://raw.github.com/pypa/virtualenv/master/virtualenv.py | python - .env

Cool Python projects worth checking out:

Possible framesets for a future fixed gear:

The Levi’s 511 Commuter looks nice:

I don’t like it when Mercurial opens vimdiff or other external tools during merges. So I added this to my hgrc:

merge = internal:merge

I believed that passphrases were pretty strong. I was probably wrong:

by our metrics, even 5-word phrases would be highly insecure against offline attacks, with fewer than 30 bits of work compromising over half of users.


I just played Dominion with co-workers. I think it was the 1st time I played a 4 players game of Dominion. It was a surprisingly fast, less than 40 minutes including setup time.

Looking forward to play some more Dominion.

I’ve added Google Analytics’ tracker on Scratchpad, because I can.

Scratchpad’s database is a simple log file like that:


First post

Second post

Just the time, and the content, and ^L –the ascii character 12 or form feed– as separator.

This gives me ‘super easy simple’ back-ups. On another server’s crontab I just add:

curl http://henry.precheur.org/scratchpad/log > backup/$(date)

People get that long passwords with many different characters are safer. Short password = bad, long password = good.

Humans are good at making analogies, a lot of us think that long keys are better than short ones. A crypto-system with 1024 bits keys is safer than one with 256 bits keys, right?

YES! Yes, if “everything else is equal”. If you know a little bit of cryptography, you know there’s a lot more in a crypto-system’s security than its key length. Passwords are much weaker than keys, most passwords are problably weaker than a 32 bits random key.

Marketers uses our –correct– assumption that more bits in the key add much security to sell us insecure products with very long keys. Something true makes us believe something wrong. 10 years ago I believed that long keys significantly improved security. I should be more wary of all those areas I don’t know much about. If I make an analogy with cryptography: my lack of knowledge will make me believe wrong things :)

Quick’n dirty shell script to serve a single file over HTTP:


trap 'echo stop; exit' SIGHUP SIGINT SIGTERM

while true
    (echo -e "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r"; cat $*) | \
        nc -l 8000 > /dev/null

You’ll need netcat (nc) to run it, it will listen on port 8000.

I’m writing the HTML and Atom generation scripts in Python. I tried to use Awk, but it was kind of inconvenient. Lua seemed to be promising, but the lacks idioms I got used to with Python made it a bit frustrating.

I wanted to finish a first version of Scratchpad before the end of January, it’s late-February now: I better get on it with what I know. So Python it is.

Why are score pages so dawn awful?

I follow the NHL, I haven’t found any score page which does a decent job of showing the information I want efficiently.

It might be a good project idea for later :)

/usr/bin shouldn’t exist, /usr was what /home used to be, and today’s Unix hierarchy doesn’t make sense:


I just finished reading “The Development of the C Language” by Dennis Ritchie: a great read for every C programmer.

I find it remarkable that the early versions of C were developed and used in a very constrained environment. Memory and CPU time were scarce, the language had to be simple to implement. I’m convinced that constraints and limitations fuel creativity, not restrain it. It’s easier to find a great solution to a problem when the set of solutions is limited.

From “The Development of the C Language”:

Thompson’s PDP-7 assembler outdid even DEC’s in simplicity; it evaluated expressions and emitted the corresponding bits. There were no libraries, no loader or link editor: the entire source of a program was presented to the assembler, and the output file.with a fixed name.that emerged was directly executable. (This name, a.out, explains a bit of Unix etymology; it is the output of the assembler. Even after the system gained a linker and a means of specifying another name explicitly, it was retained as the default executable result of a compilation.)

I’ve read the Rust’s tutorial last night.

So far I’m pleased by its design choice. I like that everything is constant by default. If you want to modify something, you have to declare it mutable since the beginning.

The thing I’m not a big fan of are vectors. Vectors are Rust’s version of arrays, they are ‘uniquely’ allocated in the heap. I like to have arrays on the stack in C, I’m not sure it’s OK to have ‘everything’ on the heap in practice.

I just looked at Rust from Mozilla. It’s more or less a Go-like language. After looking at it for 10 minutes, I think I like it better than Go.

Turns out that using multiple characters for RS is a GNU extension.

So my reverse example doesn’t work with other implementations of Awk. I should know better, every time I’ve used GNU Unix programs I’ve run into portability problems.

For scratch-pad I need to reverse the order of the records in my log file. The tricky thing is that each record is one or more lines separated by a form feed or page break: “\f” or ^L.

Here’s my 1st version using getline. getline allows us to consume lines without exiting the current rule:

BEGIN { i = 0 }
      do {
              sort[i] = sort[i] $0 "\n";
              getline; # Where the magic happens
      } while ($0 != "\f");
      sort[i] = sort[i] $0;
      i += 1;
      for (x = i - 1; x > -1; x--) {
             print sort[x];

It worked well enough, but there’s something much simpler. In Awk the record separator can be any string, it doesn’t have to be a carriage return. We can change to record separator at runtime using the RS variable. This simplifies things:

BEGIN { RS="\f\n"; i = 0 }
      sort[++i] = $0
      for (x = i; x > 0; x--) {
             print sort[x] "\f";

Played 2 games today. Both were 3 Rax all-ins.

1st was against a Protoss on Shattered temple. I fucked up, lost a SCV, forgot to put one back to build the Baracks. I left right away, I don’t think it’s smart to stay in a game if you screw up early. Better start over and get a ‘real’ game going.

2nd, against another Protoss was a win. He didn’t manage to scout me before my 3 Rax completed. Easy win. 3 Rax all-in is pretty strong at low levels.

I’m learning rc. This one took me a while to figure out:

; for (i in 1 100 10) { echo $i } | sort

It turns out that this is not what I expected: I though that the 3 numbers would be sorted. Here’s how rc interprets it:

for (i in 1 100 10) { echo $i | sort }

What I wanted was:

{ for (i in 1 100 10) echo $i } | sort

It took me a long time to figure that one out:

    $ awk '/\.\./ { print "2. " $0; next }
           /\./ { print "1. " $0; next }
           { print }' <<EOF
    > foo
    > .foo
    > ..foo
    > EOF
    1. .foo
    2. ..foo

I didn’t know that command next which allows you to jump to the next line without executing the other rules.

I forgot to mention that warning that Mercurial gives me:

warning: filename contains ':', which is reserved on Windows: 'scratchpad/2012-01-19T18:35:19-08:00'

Using RFC 3339 times as filenames was not a good idea.

I think I’m going to change the architecture of my scratch pad. Instead of a collection of files in mercurial, I’ll use a simple log file. It’ll give me version control for ‘free’, but I’ll lose the completion from the shell. On the other end I wont use Mercurial, that’s one less tool to worry about.

Not sure if that’ll work. I’ll write a prototype and see how it goes.

I played 2 games of Starcraft during my lunch time break today. I lost against 2 silver players. I was way too passive during both games. I need to be more aggressive, I should go 2 Rax pressure every time.

3rd day of practice. I’ll try to play more tonight.

I finally did a biggish blog post today. It’s been a while since I last blogged. Time to get back into it seriously: 1 article per month. It doesn’t matter if it’s bad. Maybe I’ll take stuff from this scratchpad. That’s what it’s for ;-)

There’s going to be a Starcraft 2 tournament at work next month. I’m planning to participate, but I didn’t play Starcraft for a while now. It’s time to get back into it.

1st thing: pratice a set build order tonight and play a ladder game.

I lost my rear bike light today. I went to Canadian Tire to get a new one, I was underwhelm by the choice and the price. A crappy looking rear light was $9; batteries not-included. I’m cheap, I didn’t buy it.

I went to MEC instead. There for the same price I got a rear light and a small front light, batteries included. A friendly service was also included.

I should know better, specialist store have almost always cheaper and/or have more choice than generalist.

After working at Image-Engine, I realized how important color temperature is. When you take a picture indoor and it looks all yellowish, that’s because the ambient color temperature is ‘low’. Most cheapish light bulbs have a color temperature around 3000K. When you are outside, the color temperature is usually between 4000K and 6500K.

Today I decided to replace the light bulb over my desk at home with a 5000K+ light. Since almost nobody knows about color temperature, it is mostly absent from the technical specifications of cheapish light bulbs. It’s hard to know which one to buy. Luckily the Energy star website has a complete list of its certified light bulbs, with their respective color temperature.

I’ll probably get a Philips ‘Mini Twister Daylight Compact Fluorescent Bulb’, it is supposed to be ‘daylight’-like, with a color temperature of 6500K and a good luminance.

Video is full of stuff like that:


I’m about to write the html generator for my scratchpad. I’ll probably use rc and shell tools, there’s no need for something more complicated using Python and Jinja for example.

Socks are a problem for me. I never have enough of them. I have plenty of T-shirts, but only 10 pairs of socks at any given time.

It is time to end this, and buy 20 pairs in one go!


I’ve just recorded my first test screencast. I’ll need some more practice before posting on Youtube…

Here’s the command I use under OpenBSD to record:

ffmpeg -f sndio -i rsnd/0 \
    -f x11grab -r 10 -s 1284x772 -i :0.0+1,16 \
    -acodec pcm_s16le \
    -vcodec libx264 -vpre lossless_ultrafast -threads 0 \
    -y "./$(date +%FT%T).mkv"

I wanted to generate all the HTML for the scratchpad using Python, but I’ll try to use rc instead.

I just wrote a small pipe editor inspired by vipe. It’s a short shell script:


case $1 in
      cat <<EOF
${0} [command] [arguments...]

${0} is a pipe editor. It read its standard input, opens in a text editor,
and write the result to its standard output.


Edit a file a write it into another file:
  cat input | ${0} > output
  ${0} < input > output

Edit a file using gvim --nofork, a pipe the result into wc:
  ${0} gvim --nofork < input | wc

To call ${0} without any input, use /dev/null:
  ${0} < /dev/null


function edit {
  ${args:-${VISUAL:-${EDITOR:-vi}}} $*

cat > $tmp
edit $tmp < /dev/tty > /dev/tty
cat < $tmp
rm $tmp

Yay, I’m finally done with the first version of my little scratchpad.

I’ll tell you some more later, when this published ;-)

Let’s talk about the buildin fc in zsh. fc allows you to edit the last command in your editor. It might sound kind of pointless, but it makes the command line that much more powerfull. You can have ‘real’ programs doing something useful in 1 minute right from the shell.

This makes languages like Awk that much more interesting.