Scott Graham's blog at h4ck3r.net

Gmail Backup via IMAP using offlineimap

2011-03-13T00:00:00-08:00

Every so often I have a minor freakout when I realize the amount of data that's stored in my Gmail account. The first thing I do is make my Google account password longer and add more obscure characters. Then, I feel slightly nervous that it's stored in the US and subject to who-knows-what laws.

Once those feelings pass, I'm mostly concerned that due to <unforseen circumstance> I might lose access to everything that's in there. It would be a massive hassle.

Fortunately, Gmail supports IMAP access to the mail so it should be relatively straightforward to back up everything. I finally got around to doing that today. I ended up using "offlineimap" which does what its name implies.

First, install offlineimap. On Ubuntu, this is just:

sudo apt-get install offlineimap

If you know how for other OSs, please feel free to leave a comment.

Then, I made a configuration file that looks like this:

    [general]
    accounts = Gmail
    maxsyncaccounts = 3

    [Account Gmail]
    localrepository = Local
    remoterepository = Remote

    [Repository Local]
    type = Maildir
    localfolders = /backup/USERNAME/mail

    [Repository Remote]
    type = IMAP
    remotehost = imap.gmail.com
    remoteuser = USERNAME@gmail.com
    remotepass = PASSWORD
    ssl = yes
    maxconnections = 1
    realdelete = no

Of course, you'll need to replace "USERNAME", "PASSWORD", and probably change "/backup/USERNAME/mail" to where you want the backup to live.

I saved this configuration into /backup/USERNAME/USERNAME.imaprc.

Then, try running it with:

offlineimap -c USERNAME.imaprc -u Noninteractive.Basic

The -c argument points it at our config file rather than ~/.something and -u changes the UI to a standard one rather than a heavy colourful curses default (we're going to use it as a cron job in a second so we want a log-style).

If that works, it will probably take a while, but all your mail should get synced into that folder. You can confirm that everything worked by (e.g.)

mutt -f /backup/USERNAME/mail/INBOX

which will use "mutt" to view the data saved there (you might need to do sudo apt-get install mutt for that too). If you look in the mail/ directory, you should see all your labels, as well as the standard Gmail ones. And, though the files in the directories have ugly names, each one corresponds to one mail message stored in text format, so in case of emergency you could always go grubbing through the files to find important information.

Now, if everything looks OK, you'll probably want to schedule this to run every so often. To accomplish this, I added this script into /etc/cron.daily

#/bin/sh
offlineimap -c /backup/USERNAME/USERNAME.imaprc -u Noninteractive.Basic

Don't forget to chmod +x that script.

And now I can sleep a little easier. I'd still be a Sad Panda if Gmail goes away, but I be much less irritated with a backup copy.

As you might glean from the name of the target directory (/backup) this folder is also backed up via rsync. This is probably not necessary (assuming both Gmail and your copy don't disappear at the same time) but it was just as easy for me.

Another possibility might be to sync Gmail directly into a large Dropbox folder. While it seems a little silly, it would then be almost impossible to lose all the copies of your mail that would be scattered around your various computers.

What can you do in 2k lines of C?

2011-02-02T00:00:00-08:00

Plain Old Awesome C

A few weeks ago I ran across some of Sean Barrett's awesome C code.

It started with stb_image.c because it's just crazy useful and avoids so much hassle in small demo programs. But that was just the gateway...

Check out stb_truetype.h too: TrueType font rendering in one ~1200 line header file. Do you remember how much of a pain in the ass it was the last time you had to try to integrate FreeType 2? Or <shudder> FontFusion? If you think about it for a second, you realize that TrueType rasterization can't be that hard because printers were doing it long ago on crappy little embedded processors, but the default is just to fall back on the big ugly library, and then wrap it and pretend it's not there. How about instead, just write some good code?

There's also stb_vorbis.c (a Vorbis decoder) which I haven't tried yet, but I fully expect it to Do What I Want, if someday I need it.

And, on top of that Sean's released these as public domain (partially thanks to RAD it seems). Not MIT or BSD, and certainly not LGPL, GPL, or some other pain in the ass, but straight up public domain. So much easier when you're embedded in Big Corp, but even if you're not.

So, #1, use them. And #2, let's all copy Sean.

So, what can you do in 1-2k?

Those libraries are standalone. Zero dependencies other than the C standard library. There's no build files to screw with. There's no licenses to get approved by your legal department*. There's no attribution clauses to add to your "About" box. Just all buttery working-out-of-the-box convenient awesomeness.

I got to thinking: What else could be stuffed in 1-2k lines of plain C?

An JSON / XML / YAML parser?
Shading?
A database?
A GUI library?
A web server?
Rendering?

Since I've been language-obsessed the last few years, my first 1-2k LOC project is a programming language, appropriately named "twok". Calling it something that sounds like poop dropping into a toilet probably means that no one will want to use it, but that's probably for the best anyway; there's lots of languages out there.

The question is, how much functionality can you get into 2k lines of code? So far, twok's features are:

Python-style syntax
The "basics": arithmetic, bit ops, logic ops, functions, if, for, etc.
Native compilation to x64 (and supporting both Windows and SysV Mac/Linux ABIs) so you can call back and forth with C code
List syntax/manipulation
Simple zone-based GC
Varargs functions
Basic structs

And, it's just a hair over 1k LOC, and of course, all in one easy-to-use header. My plan for the second half of the code:

Some sort of macros
More standard library functions
List comprehension syntax
Integration with the new GDB debugging-support-for-JITted-languages
A backend for ARM, and maybe an interpreter backend for consoles

... Or other fun suggestions. I'm not sure if all of that will fit in 2k, but I think it should be close.

Certainly this is will not be the next C# or Ruby. And it's not as practical as image loading, font rendering, or audio decoding. But, I think it's interesting how much a hard constraint on size and a clear goal helps focus and at the same time find a way to stuff in the functionality you want.

So, what awesome library could you write in 1-2k lines of C?

Do it!

It's fun!

And release it as public domain.

Programmers everywhere will love you.

_{*: yeah, I know that's probably still not true, unfortunately.}

Data Oriented C

2011-01-07T00:00:00-08:00

There's been a lot of noise (and some light) up in The Twitters about Data Oriented Design. Noel Llopis, Steve Anichini, Mike Acton e.g., Pål-Kristian Engstad, Niklas Frykholm. Christer Ericson, Stefan Boberg, and many others have been having at it.

The points under discussion are well described in those, so I'll defer to them. Read Pål-Kristian Engstad's first if you want to get up to speed quickly.

Most often, I find myself hitting one of the extremes. I either want to use something very high-level and script-y (Python, Ruby), or something very low-level (Assembler, C, ???).

The benefit of the high-level of course, is that you might have the functionality magically accomplished for you by a library or language feature. If you're forced to parse some XML, then generate an image, and POST the result to a server, well, you're probably going to get it done orders of magnitude sooner in a scripting language.

For in-game where you're working on code that's run every frame though, it's likely that you're going to need to carefully design the layout of your data.

In C this means focusing on getting all the data to accomplish a particular task together into a (preferably cache-line-sized) chunk, and grouping a bunch of those together. In C++ it naturally means the same thing, but it also means staying true to your inner C programmer and not getting lured by the Sirens' Song of virtuals up the wazoo.

My thinking is that it would be useful to make a small extension to C (or C++?) that supports this method of thinking about problems. Call it Data Oriented C, a strict superset of C. The goals would be to:

encourage a more stream-centric programming model
encourage grouping the data for a problem into little cache-sized packets
auto-instrument to make performance and data flow easily understood.
support distribution of work across processors
not be too hostile to IDEs, editors, and debuggers

What other goals do you have?

A → A′

The general idea (as espoused by Mike Acton) is that rather than thinking of your program as manipulating a bunch of objects, think of it as transforming data from one form to another, i.e. from A to A′.

I like to think of this as the Unix pipes model, e.g.

cat data | sort | uniq

This is easy to write, easy to test, and easy to understand.

Perhaps we can encourage that by supporting it more directly in the language (off-the-cuff arbitrary syntax):

transform ToWorld
{
    input
    {
        mat44 local;
        int parent;
    }

    output
    {
        mat44 world;
    }

    operation(int N)
    {
        for (int i = 0; i < N; ++i)
        {
            if (input[i].parent == -1)
                output[i] = input[i].local;
            else
                output[i] = output[input[i].parent] * input[i];
        }
    }
}

(where transform, input, output, and operation are new keywords.)

Of course, we could extend the syntax or use helper libraries to go wide in the body of the operation. We could also deviate farther from C and add more specialized operation styles eventually. e.g. a map could elide the for loop, and remove the array subscripting replacing input[i] with just in and output[i] with out. That sort of sugar probably doesn't belong in an early version though.

Piping

It might also be useful helpful to support a high-level syntax for composition à la Unix pipes. Plain | is already taken, so perhaps following F# for a |> operator:

SceneNodeCollection nodes;
// ...
nodes |> ToWorld |> Cull |> PrepareForRender;

where ToWorld, Cull, and PrepareForRender are transforms as defined above.

Is this necessary? Or do people generally want to hardcode the transformation compositions, or set them up at runtime based on configuration data?

Having the composition hardcoded allows for some improvements:

It would allow intelligent overlapping of input and outputs to either reduce or disallow copies.
It would also be a good place to break into packets for DMA to processors that don't share memory.
It could also make for an nice place to add instrumentation for gathering timing data.

Implementation

Doing Data Oriented C as an extension of a plain C parser wouldn't too hard, it could probably just be written by hand.

If it's to be an extension of C++ (probably more palatable for most people), it doesn't seem like it would be too hard to extend clang to do a source-to-source transformation from Data Oriented C(++) to plain C(++). Realistically, that's probably a better option for plain C too.

Thoughts

Data Oriented C doesn't fundamentally do anything magical that you couldn't do in C (naturally). It's all about making the default decision be the correct one. It could also make it easier to help out the C compiler, for example by correctly adding __restrict annotations when appropriate.

It seems like this would only be useful if it was designed by and used across a few studios. So, what do you think? What's the feature you'd like to see removed or added? Is the whole thing even necessary? How would you change it? Twitter, email, or comment below.

Improve your App Launch in 4 hours

2010-11-28T00:00:00-08:00

Picture a customer looking to solve her particular business problem. Very occasionally, she might be looking to solve a problem in 5 minutes or less, and simply be willing to choose the first thing that sort of works that she runs across.

More likely though, she wants the best solution for her.

From experience, she knows that choosing the first half-assed solution isn't going make for a long-term victory. Inevitably, it's going to mean more hassle later on, because something won't work, or there will be a migration cost, or the company will just be a pain to deal with.

Evaluation

When a customer is performing evaluation of specific solutions, they can't possibly have complete information. They're not going to have used all the software for extended periods. Reviews and recommendations are one method of pruning the search, but a review describes how well the product fits the reviewer, not how well it will fit another specific person.

What does this mean to you?

Lacking complete information, a customer is looking for signalling that tells them that everything's going to be OK if they go with you.

It's going solve their problem.
They're not going to get hassled later.
They're going to get good value for their money.

If you're reading this, you're clearly someone who cares about your business, your application, and your customers. You are hungry and eager. You are the polar opposite of the other half-assed company that out-sourced development, design, and UX. You are the opposite of the huge faceless corporation that can't really be bothered with smaller customers, or solving their "trivial" problems.

That you care deeply is what you must demonstrate to your customers.

Focusing resources

Of course, you're exhausted already. You're trying to launch here! You're trying to cram all your launch features in, you're scrambling to fix the blasted thing in IE. How are you going find time to actively demonstrate that you care while you're too busy actually caring?

Slice of Caring

The practical solution to this problem is what I awkwardly call a "Slice of Caring".

Instead of trying to show off that you care in every aspect across your whole business, you choose one particular slice to use as a demonstration of your commitment.

Of course, you do actually care about the entire business, but the effort involved in demonstrating this is huge, so pick something more manageable.

In particular, you probably want to pick something flashy because that's where users will look first. Flashy depends on your application and problem domain. It might mean supporting the newest gadget, or it might mean integrating with Microsoft Outlook, or it might mean streamlining their Basecamp workflow, or it might be some other customization for a vertical-within-a-vertical-within-a-niche that a specific type of lawyer will love.

I don't know what it will be for you, but you can definitely find something.

In my case, I was launching DropPic, an application for designers to create galleries of comps and mockups. A few designers had mentioned that their clients were iPad-obsessed, and would really like to review designs on the iPad.

So, I added a review mode, customized for the iPad. You can try it here if you're on an iPad.

(On a WebKit-based desktop browser, you can add ?forcemob=1 to the preview page to see it. It doesn't work properly, but you can get the general idea.)

My customers are thrilled because they know their clients will love it, and what they want is happy, paying clients.

This took me 4 hours, from deciding to do it, to figuring out that Sencha Touch was probably the fastest way to get it done, to implementation. It involved no changes to the backend, and is only about 200 lines of code. This is the sort of flashy feature you're looking for.

It's important that you not dig yourself a maintenance hole. Pick something that's both a slice to show you care, and also a standalone column of functionality that isn't going to break every time you change your application.

In my case, I'm using the same API for communication, so it's strictly a different UI. If you were, for example, pulling contacts from Outlook, you'd want to make sure that the "slice" was tool that talked to Outlook, got data, and stuffed it through your regular contact-importing-procedure, whatever that might be. Don't create exceptions and special cases for extra features, because that'll definitely come back to haunt you.

Do it now

This is of key importance.

Demonstrating that you care is difficult. Caring is one thing, but before they've used your great product, or experienced your outstanding customer support, prospective customers won't really know if you care. They therefore won't know whether they're going to be happy with your product.

You must expend additional effort to show them that you care about the big picture, the little details, and everything in between.

Without the extra effort, they're never going to experience the excellence you have to offer.

But you can find something that will only take you 4 hours, be suitable for signalling, and will improve your launch.

Hackage for Android and Vonage voicemail .wav's

2010-09-08T00:00:00-07:00

Froyo (Android 2.2) broke the playing of Vonage's attached voicemails. I think it's because they're not really .wav's or something, but whatever the reason, it's a total pain in the ass.

Presumably it'll be fixed in Gingerbread, but it was driving me crazy in the meantime.

I hacked together some of the worst code ever. I honestly felt like I was programming like an AI would: Google for something, paste it, run it, then read it to see why it didn't work, and repeat.

Anyway, it took me almost an hour, so here it is case it helps someone else.

Put it in wavconv.py (or whatever) and run it as:

python wavconv.py your_user_id@gmail.com your_gmail_password

It uses sox to convert the wav's to .ogg which seem to play OK on my Nexus One. To install sox on Ubuntu, just do:

sudo apt-get install sox

Please feel free to leave a comment if you know how to do it for another platform.

It sits in a loop and checks for messages every 2 minutes, converting them if it sees a new one. So, you'll probably want to run it on a server in a screen session. Or maybe it'd be better in cron and delete the loop/sleep.

Contents of the above link, below.

import email, getpass, imaplib, os, smtplib, sys, time
from email.mime.multipart import MIMEMultipart
from email.MIMEBase import MIMEBase
from email.mime.text import MIMEText
from email.mime.message import MIMEMessage
from email import Encoders

if len(sys.argv) != 3:
    print "usage: wavconv.py user@gmail.com your_password"
    raise SystemExit()

user = sys.argv[1]
passwd = sys.argv[2]

while True:
    m = imaplib.IMAP4_SSL("imap.gmail.com")
    m.login(user, passwd)
    m.select("INBOX")

    resp, items = m.search(None, "(SUBJECT \"new voicemail\")")
    items = items[0].split()

    for emailid in items:
        resp, data = m.fetch(emailid, "(RFC822)")
        email_body = data[0][1]
        mail = email.message_from_string(email_body)

        if mail.get_content_maintype() != 'multipart':
            continue

        #print "["+mail["From"]+"]: " + mail["Subject"]

        for part in mail.walk():
            if part.get_content_maintype() == 'multipart':
                continue

            if part["Content-Transfer-Encoding"] != "base64":
                continue

            filename = mail["Date"]
            filename = filename.replace(" ", "_").replace(":", "_").replace(",", "_")
            # hacky non-dupe!
            if os.path.exists(filename) or mail["From"] == user:
                break

            fp = open(filename, "wb")
            fp.write(part.get_payload(decode=True))
            fp.close()
            os.system("sox " + filename + " conv.ogg")

            # build a response and send it (ugh!)
            msg = MIMEMultipart()
            msg['Subject'] = mail["From"] + ", " + mail["Subject"]
            msg['From'] = user
            msg['To'] = user
            attach = MIMEBase('audio', 'ogg')
            part.set_payload(open("conv.ogg", 'rb').read())
            Encoders.encode_base64(part)
            part.add_header('Content-Disposition', 'attachment; filename="conv.ogg"')
            fp.close()
            msg.attach(part)

            s = smtplib.SMTP('smtp.gmail.com', 587)
            s.ehlo()
            s.starttls()
            s.ehlo()
            s.login(user, passwd)
            s.sendmail(msg["From"], [msg["To"]], msg.as_string())
            s.quit()

            print "converted", filename

    time.sleep(120)

Skulpt Parsing Revisited

2010-08-19T00:00:00-07:00

Over and over, I "rediscover" that the easiest and most reliable way to do something is to have an exact description of what needs to be done.

That sounds like a tautology, but when applied to code it can be very powerful.

My latest reinforcement and example of this idea was in my renewed attempt to get Skulpt back on the rails to something that I could maintain and extend more easily.

The first time around I was a bit overwhelmed: it turns out that implementing a full version of Python is quite a large undertaking (who knew! <ahem>).

But, the thing is, there's plenty of help available. There's of course the source code to CPython. I have referred to that quite frequently, because when it comes down to it, that's unfortunately the only "spec" for Python.

However, it doesn't turn out to be the most useful thing. The most useful thing, by far, is the ability to generate tons of test data that's known-good.

The first time around on trying to write a compiler, I had done the obvious thing of passing python code to CPython and also to my compiler, and then making sure that running them both resulted in identical output.

However, the crucial part that I didn't do was to mirror CPython at other levels of implementation.

As one example, there's any number of ways you could choose to implement scoping and binding of names in a programming language. By only comparing the run output between CPython and Skulpt I was effectively reverse engineering a lot of that knowledge. Of course, for the common cases, everyone knows how variables are bound, shadowed, closed over, deleted, etc. But it's the obscure cases that need to be correct so that everything is rock-solid, and that's where you never feel sure that you've considered all cases in tests.

So, the magic step this time around: I made sure there was two more levels where I could dump internal state from CPython and make sure that it matched Skulpt's exactly. Both the AST nodes (using the ast module) and the symbol table (using the symtable module) now match 1:1 between Skulpt and CPython. This includes many little details that are important but easily overlooked: line numbers and column offsets being identical in the AST for error messages, all variables being tagged as free or cell identically, and so on.

This was huge for my confidence in correctness. I now have the ability to generate as much test data as I want for the AST and symbol table. I could, for example feed all of Python/Lib/*.py to both Skulpt and CPython and compare a pretty-print of the ASTs and symbol tables.

Or, in other words, I have an exact description of what needs to be done in test-data form. How hard can it be at that point?

The AST and symbol table are two big chunks of the implementation of the compiler. With confidence that they contain a correct version of all the information required, I now have a solid foundation to build the rest.

The "rest" is compilation of the AST to .js, runtime support, and the object model. More on their progress next time.

Opcode Design for a Lisp processor

2010-06-14T00:00:00-07:00

I've been thinking about trying to design and build a TTL-based processor for the last few weeks, and specifically trying to make it something that supports Lisp as its standard language.

Going all the way down to the level of electricity definitely opens up the design options, perhaps way too far! It's very difficult to decide where best to solve a problem when you're able to change everything from word size and available hardware, all the way up through to compiler implementation.

I've gone through about 10 each of completely different architectures, instruction set styles, opcode encodings, word sizes, and so on.

I started with something based on Sussman and Steele's SIMPLE, then veered off into a SECD-style machine, then explored things similar to the PDP-11 instruction set.

Many of my attempted designs were pretty "top-down" in the sense that I started writing assemblers and simulators for them, and got carried away designing fancy instruction sets that were either too non-orthogonal, or just simply way too complicated for me ever to have a chance of implementing them in basic TTL logic. Others were just plain silly or broken.

Here's a random brain dump of what I'm thinking now. Given my current rate of starting over, it's not really likely that this one will be the one I actually build, but this is where I'm at now.

RAM is 32k, divided into 2 half spaces. Pointers are 13 bits long and address 16bit words, making for 8k addressable words. 16 registers are mapped into the low 16 words.

Out-of-memory is sort of like a "fault" and jumps to a system provided (or user-written :)) GC routine.

Embedded description of opcodes and encoding (or external if that sucks).

It's pretty PDP-11-y, but simpler, and the modified to my liking. cons is a basic instruction, and car, cdr, rplaca, rplacd have one instruction implementations too.

I never got around to properly writing up older designs, but some of the code is here along with some mumbly text files about design for various versions.

main.lisp simulates one design (based on SIMPLE), which uses Lisp tree structures as the basis for evaluation, as opposed to the usual linear stream of instructions. Type bits in the pointers are things like if, lambda, etc.

opcodes.lisp & as.py simulate a different custom design. It's sort of an accumulator design, but most operations can be accumulated to A or D, and then in some cases, A and D are treated as one register for simple cons-cell handling.

I haven't written a simulator for the version described above in the spreadsheet, but that's next, assuming it survives the next round of pondering.

The hardware side of a Lisp Processing Unit

2010-05-31T00:00:00-07:00

A couple months ago, I was thinking about trying to build a processor based on Lisp.

I got some time and motivation again to work on this (who knows where it comes from some times). I think perhaps it was finding the excellently-crazy Relay Computer that Harry Porter built. Bill Buzbee made an amazing and more functional computer out of mere discrete logic TTL chips, also very impressive. (And I expect that's the first time it's ever been described as a more functional computer.)

A relay is a very big clunky version of a transistor. When power is supplied to a control line, it activates an electromagnet, which causes a switch on a spring to be shoved from touching one output line to touching the other. In this way, you can cause incoming electricity go one way or the other based on the switch.

But that's it. That's all they do.

Building a computer out of these large things captures the magic of creating a computation machine out of thin air. Everyone can understand the description of a relay, and has probably played with magnets and electromagnets. When you string a whole lot of these switches together and attach them in a specific way, you suddenly get a computer? How? <insert-Zen-moment-here>

Somehow in the TTL version (which of course is really very similar, except faster) it feels like the computation might be "hiding" in those chips somewhere. Of course they're really just switches too, and they switch much more efficiently.

Maybe it's just some silly human thing to enjoy the clicking noises that make it feel like "It's really doing something in there."

Onwards

I currently know roughly nil about electricity, but was inspired enough to think it might help things if I tried to burn my fingertips with a soldering iron: I ordered Make: Electronics.

Experienced electronics dudes will probably sneer at the fact that I also ordered their kit that includes all the bits needed for the first 12 experiments. I'm sure it's overpriced for the raw parts I'm getting, but hopefully will flatten the learning and frustration curve, not to mention me wasting money buying all kinds of not-quite-right bits.

How long can it possibly take before I'm prepped to make a relay computer, right? :)

Disable STYLE-WARNING on sbcl

2010-05-30T00:00:00-07:00

When sbcl wants to tell you that it's unable to generate the best code, or at least unambiguously compile the code you give it, it emits a STYLE-WARNING. Generally, the warnings are things like functions that haven't been defined, and so are things you do want to know about.

But, when you're working interactively, it spits out a warning that I find irritating, telling you that you're redefining a function. Well, duh, I know.

STYLE-WARNING: redefining HOST-GC in DEFUN

I'm sure there's some sensible reason for this, but I'd still rather not see it.

Also, in using someone else's code (perhaps written on a different Common Lisp) there's often a spew of STYLE-WARNINGs too. For example, I'm using lisp-unit, a simple testing thingy. It works nicely enough, but sbcl spits out lots of warnings as it loads it.

I had to hunt around for a while to find the way to silence these, as it wasn't obvious from the sbcl docs, but this is the way to nuke the warning while loading a particular file. For example, with lisp-unit:

(declaim #+sbcl(sb-ext:muffle-conditions style-warning))
(load "lisp-unit.lisp")
(declaim #+sbcl(sb-ext:unmuffle-conditions style-warning))

The #+sbcl makes it conditional for sbcl. That bit of code would fail on other CLs because of the sb-ext. I occasionally run on ClozureCL too, but it doesn't cause warnings in the same way, and I haven't had a need to silence warnings there.

You may or may not want to wrap that declaim around your interactive bits, weighing that you might miss more important warnings of course.

Fix Ubuntu 10.04 Lucid minimize/maximize/close button position

2010-05-29T00:00:00-07:00

I tweeted this on first install, but that's like throwing it into wind if you ever want to remember it (i.e. when I install on a new PC).

This leftist conspiracy must be fought people! Here's the one liner you're looking for:

gconftool -s /apps/metacity/general/button_layout -t string "menu:minimize,maximize,close"

That is all. Fight The Power! Represent The People! Other Nonsensical Slogan!

Copying syntax highlighted code from Vim

2010-05-27T00:00:00-07:00

Occasionally, I need to email code to a co-worker, and in general, Outlook seems to enjoy destroying anything you write in its editor.

It always wants to convert '-' to an endash, '--' to an emdash, replaces quotes with smart quotes, capitalizes random words, and so on. Those are great things for your average dude writing some English, but man does it suck for programmers. It's terrible for command lines too.

If the code's longer than about a line, it's also pretty irritating when you copy code from Vim, paste into Outlook, and you get plaintext. Some people might prefer it that way, but everyone in my office uses rtf/html formatted emails anyway, so I figured I might as well get some benefit out of it at least.

I whacked up a Vim script "cliphtml" that copies from Vim, using your current colorscheme and stuffs it in the clipboard in html format. Current requirements are Windows, Vim with +python, and pywin32 installed in your system python. Windows Vim loads the python dll from the system location (rather than statically linking unfortunately) so you will have to install python2.4 (at least for Vim 7.2) and pywin32 if you're not already using Python in your Vim.

To install the script, just save cliphtml.vim to your plugins directory. The command is

:ClipHtml

and then paste into Outlook. You should get something like this:

It also works with ranges or visual ranges of course, so you can select with 'V' and then :ClipHtml a snippet.

Mercurial (hg) with Dropbox

2010-05-11T00:00:00-07:00

I use Dropbox and Mercurial together so often now, and it's so freakin' awesome.

But just as I was setting up another repo, I realized that there was a slim possibility that there's programmers out there who don't use this trick. So they're still doing something crazy for source control, like trying to manage an SVN server, or a P4 server, or dealing with "only 1 private repo" restrictions, etc.

So... here it is, easy as pie. Or something that's easy, that isn't pie. Your Mom, etc.

First, sign up for Dropbox. (That's a referral link, you'll get an extra 250 meg on top of your free 2 gig if you use that instead of just going to the website).

Now, use Mercurial as normal. HgInit is a decent basic tutorial. You'll probably want to install hgtk on Linux, or TortoiseHg on Windows. But, just for example, here's the command line version:

~/myproj$ hg init
~/myproj$ hg addremove
~/myproj$ hg commit -m "initial commit"

Now you have revisions locally, which is nice enough, but that's just Mercurial. If your hard drive dies, you're hooped.

On the other hand, you didn't have to worry about servers or making depots, or backup, or other mucking around until you've spent at least a few hours on the exciting new "myproj" and you decide you'd rather not lose it.

So, what do you do? First, clone your repo into your Dropbox:

~$ cd ~/Dropbox
~/Dropbox$ hg clone ~/myproj ~/Dropbox/myproj-hg --noupdate

The syntax is hg clone <from> <to>. The --noupdate flag tell Mercurial to just store the repository data, not to check out a working copy. You're never going to work directly in the Dropbox directory, but instead use it as a "remote" where you push and pull to and from it. The only thing that will ever be in that directory is the .hg directory, and there's no user-serviceable parts in that particular one. That's why I renamed myproj to myproj-hg in the clone step, just so I remember that it's a Mercurial directory, and it's supposed to be "empty" (because the .hg will be hidden on Linux).

Second step, back in your original directory, edit your paths to point at the Dropbox backup of your repository:

$ vim ~/myproj/.hg/hgrc

(or your editor of course). Paste something like this (the file will be new/empty if you haven't added anything else yourself:

[paths]
default = /home/sgraham/Dropbox/myproj-hg/

Replacing "sgraham" and "myproj" naturally.

Now,

$ hg push
$ hg pull

will work as expected in ~/myproj, and every time you push, you'll have a secure, private, offsite backup of your code.

Step Three: Well, that's it. There's no more steps.

When you go to another machine, say a Windows machine, all you need to do to get access to your repository is to install Dropbox and then clone from it:

C:\code> hg clone "C:\Users\sgraham\My Dropbox\myproj-hg" myproj

And that's it. hg push and hg pull && hg update will already work properly.

And there you have it, Dropbox is the shizzle. Along with Mercurial, you've got two great tastes that taste better together.

Step Over in Eclipse under Ubuntu using F10

2010-04-07T00:00:00-07:00

And another for myself and search engine stumblers...

If you use Eclipse on Ubuntu coming from 15-20 years of Visual Studio you'll soon be remapping the debugging shortcut keys to match Visual Studio's before you start swearing up a storm.

However, F10 is mapped to menubar_accel by something in Ubuntu. Gnome? Metacity? Not sure what exact thing it is that steals it, but the key doesn't make it to Eclipse anyway, so pressing F10 just opens the File menu in Eclipse which certainly isn't what you want.

The fix

Run gconf-editor, browse to /desktop/gnome/interface then scroll down to menubar_accel and delete the F10 value.

You could probably reassign it instead, but if you're like me, you've never used that functionality on purpose so don't bother.

Android ADT + eclim on Ubuntu 9.10

2010-04-06T00:00:00-07:00

Stuffing this here so I remember for next time and for search engine stumblers...

On Ubuntu (9.10) I installed Eclipse (3.5) via apt-get, then the Android ADT per Google instructions, and then eclim via sudo ./eclim_1.5.6.sh. After installing eclim, the Android functionality in Eclipse disappeared.

I'm not sure what happened, the Android software was still listed as being installed, but the functionality just didn't seem to be available.

If instead I just install eclim into local user i.e. ./eclim_1.5.6.sh, all seems well.

If you got yourself into the same situation, the easiest thing I found to do was to "Completely Remove" eclipse and related packages via Synaptic, then sudo rm -rf /usr/lib/eclipse and sudo rm -rf ~/.vim/eclim (or wherever you have your vimfiles), and then install again, making sure not to install eclim as root.

As a side note, I didn't know about eclim until the other day. It seems pretty sweet. You can run Eclipse headless (or not) and then most/all your project maintainance and editing in the sanity and comfort of Vim. Specifically, this includes proper completion via ^X^U (imapd to something nicer, natch) as well as error annotations on write.

"Balanced" Reporting

2010-03-03T00:00:00-08:00

Ranting on this story: Grammar guide an 'education disaster'

Summary: The English Teachers' Assocation of Queensland¹ published an error-filled grammar guide, a Professor of Linguistics tried to suggest corrections, disaster resulted.

It's one of those increasingly common articles where one party states facts about a topic and another party claims an opposing viewpoint, citing a "difference of opinion". But in reality, the story is not about two different viewpoints, it's a case of sanity versus insanity.

There are objective truths. There are things not subject to one's opinion.

Anyhow, that wasn't what I was irritated about before I started typing. The concluding paragraph of the article:

Another example is the phrase "set of", as in "a set of bowls", being described as an adjective, which Professor Huddleston says is not a grammatical unit but a noun followed by a preposition.

No! Well, yes! But what the hell is with subordinating simple facts into a quote to appear balanced? "set of" as in "a set of bowls" isn't a grammatical unit. "set" is a noun. It is so. It does not require an authority opinion from dear Professor Huddleston.

I should be more concerned about the subject of the article, but Mr. Graham says "Meh".

...

And because this occasion happens to be a passable reason to offer this link: some classic Dave Barry.

Meta-incorrectly punctuated in the article!

A New Old Processor

2010-03-02T00:00:00-08:00

aka "Lisp all the way down"

Last time I was talking about the my new-found excitement for designing a processor from scratch.

Clearly there's a lot of previous research and art to draw on in this area. I started going through some ancient, and some newer.

ZPU

There's a few decent processors on OpenCores: cycle accurate version of Z80s, 6502s, simple x86s, etc.

There's also a neat processor called ZPU that's entirely stack based. It's sort of assuming a pre-2000s model of computing in that RAM access isn't (relatively) abysmally slow. It's got a very nice opcode encoding where every opcode is 8 bits, including the immediate instruction. The immediate opcode is "high bit set" and that pushes the other 7 bits as a signed value. If you put 2+ immediates in the opcode stream, then the previous value on the stack is shifted left and the new 7 bits are ORd in, rather than creating a new stack value. The data bus is 32 bits, and the rest of the 128 available opcodes are assigned to the various standard operations, mostly operating on the stack items, but a few include a small immediate value (again, only in the 8 bits) to allow a displacement. This is used in the SP-relative indexers to allow local variables to be accessed without a lot of pain. I poked around this design for a while, and it's really nifty. I might try something like this one, though it's kind of already "been done" so I'm not sure how interesting it'd be just to reimplement that design.

Lisp Machines and SIMPLE

I didn't get too far in my pseudo-literature review before I started looking at Lisp Machines. It turns out the actual Lisp Machines were not really fundamentally anything that was specifically designed to run Lisp. They did have custom microcode, and extra pointer bits for helping on type dispatch, or helping the GC, but fundamentally, they were "regular" machines that ran simple microcode, had flags, PC, stacks, registers, conditional branching, etc.

An interesting exception to this is SIMPLE which at the very lowest level is actually an interpreter for a Lisp. car, cdr, cons, etc. are all primitive opcodes, and computation is modelled on eval/apply'ing a recursive tree structure.

That paper is very interesting. Looking back with 35 years future vantage point on someone's University project is pretty humbling (well, it's not just anyone's University project, but still). This is a piece of hardware that was designed with hardware garbage collection, and evaluating properly lexically scoped closures as a computation model. The average environment circa 1979 would have been a machine language monitor on a 6502, or an assembler on a Z80, or a primitive BASIC implementation. Compared to that sort of code-writing-experience, SIMPLE blows me away. Hell, I had to write a singly-linked list #define "class" in plain C the other day (...which is of course just plain nuts, but that's another story).

Of course, there are some limitations to SIMPLE. Looking at it now, limiting all memory words to homogeneous two-word cons pairs (with no vectors/arrays) seems very constrained. This was hand-waved away by saying that this processor could be paired with an APL processor to handle array-based math. That it would today just be a similar NV GPU instead doesn't say much for the advancement of our discipline.

The non-realtime GC also sticks out, though there was a paper the next year on how to make it realtime, for some definition of realtime. Unfortunately, that paper is only available via ~~HellSpawn~~, sorry, ACM Digital HELLSPAWN Library, so I haven't read it yet. Having a process or thread block for a GC is one thing, having the whole processor block seems very foreign.

There's also no ALU! It's not really needed for basic evaluation, but I think its lack is more due to wanting to demonstrate that it wasn't necessary and saving on die space, rather than believing it didn't belong there.

All the way...

So, SIMPLE is the basis of my plan for a new (but very old) processor. Lisp all the way down. If all goes well, phase 2 will be trying to lift some of those limitations if they actually turn out to be limiting.

Designing Hardware

2010-02-28T00:00:00-08:00

Dave and I both did the NAND-to-Tetris course a while back. Actually, I believe Dave did the whole course, but I stopped around the "compiler" assignment because it was too close to what I was doing at work at the time to be entertaining.

The course was geared towards very beginner CS students which is great. It gives a nice vertical slice of all the bits and pieces that go into getting your average piece of code to run, and did it all in a way that was simple to follow along with. The assignments and test cases given were excellent in the way they walked you down the garden path very clearly, but you still felt like you were "figuring it out" yourself.

I work firmly in the realm of software, but that course was enough to catch a bit of the hardware bug and have me wanting to venture below machine code.

I found a few nifty/slightly crazy attempts to recreate pre-Z80-era micros (like Magic-1 and Mark 1 FORTH). Those two are particularly crazy in that they don't use any microprocessor. The largest IC they use is a 4-bit adder, and everything is built up from TTL 74xx series chips (basically just and/or/flip-flops/etc.). I was all amped up to try to build something similar, but looking at some wire wrap pictures and then watching Bill Buzbee's bring-up attempts of his CPU pretty much scared me off of that idea. They look extremely cool, and I'm duly impressed, but that's a whole lot of tedium I'm not quite prepared for yet.

So, I started looking into FPGAs. Doing a CPU on FPGA isn't nearly as cool as having your own wirewrapped CPU. Mostly because it'll just look like some generic standard board, so when the victorious moment arrives when it finally prints "Hello, World" or the answer to fib(7) over a serial cable, onlookers may not be as impressed as they ought to be.

However, it does turn hardware into a (relatively) non-tedious/not-too-expensive project, so I think I'll go that way. My other excuses are that I don't have a serial port on my computer, and TTL chips are hard to find nowadays. This FPGA beginner dev board has a bunch of nifty connectors, and isn't too expensive. They have others that are cheaper too, with less connectivity.

Next time, my take on trying to come up with an interesting processor that isn't just a generic RISC clone.

Back from the holidays — brain-reboot time...

2010-01-23T00:00:00-08:00

Time to wade out of the eggnog haze, and get that old stupid brain in gear again after a vacation.

SICP revisited? Smalltalk-80? A CL project?

For the first time in a while, I feel like working on something that's explicitly focused on an incremental revenue stream.

I'm pretty confident that "clickware for the masses" ala Business of Software is never going to make retirement levels of income, or even daily-job-replacement levels of income, so a minor income supplement is the goal.

But in fact, I have no idea if my first idea will be saleable. So, digging deeper, the goal isn't even really a income supplement, but instead to try to learn whether or not there's any market for this style of software. And assuming there is a market, whether I'm willing and able to learn the skills required to find a sales niche in this market.

The first actual product idea that came to mind in an evening of brainstorming with my wife was a decent kid-safe browser. The plan in the sense of (Hopefully) Minimum Viable Product is a whitelist built into an existing browser. I think there's a lot of room for expansion of features and capabilities, if people want such a thing.

So, off I go. In writing down the things to be done, the most obvious thing that jumps out is that somewhere on the order of 80% of the things to be done are not "what I do". The work involved is writing sales copy, making web pages and CSS, setting up payment processors, and so on. There's a bit of technical evaluation, a smattering of C++ to write, but there's not a lot to that side of things.

I have absolutely no idea how to sell software, despite having been making quite a good living for the last 10 years ostensibly selling software.

It'd sure be a lot easier and more comfortable to jump back into that big ol' rut, and write a library that does something neat, preferably in an obscure language to avoid getting too many users, and then "ship it" by pushing to github.

So going a bit deeper still: the real meta goal is to do a brain-reboot. Ctrl-Alt-Del me please, but maintain all of the good previous state information.

New Tab Page As Tasks

2009-12-13T00:00:00-08:00

For a while I used the little Google Tasks window in Gmail, but then for some reason I stopped.

At first I wasn't sure why I stopped using it, but then I realized that it was because it closes itself frequently (always?), so I don't see it enough.

So, in the vein of random tweaks, another Chrome extension.

This one's very simple: it just sets the Chrome New Tab page to be Tasks. Hopefully if I see them more often, I'll be more inclined to do some of 'em.

Background Open For Reader In Chrome

2009-12-09T00:00:00-08:00

Chrome just learned how to do extensions thankfully.

I was already using Chrome as my main browser, mostly because it's all-around smokin' fast. There's minor irritations, but the Chrome team sticks to a minimalistic and sleek world view, so the toggles I'm looking for are never going to show up as mainline options.

In addition to adding buttons and processing to Chrome, the extensions layer enables Greasemonkey-ish "content scripts". In Google Reader, the default behaviour for the 'V' shortcut key is to open the RSS item in a new tab. Unfortunately, it opens it in a foreground tab rather than a background tab.

I like to burn through all the unread items, V-ing the ones that seem interesting, and then later go back and read all of them. This is quite cumbersome in Chrome when combined with non-MRU tab switching behaviour, and tab pinning. When Reader is pinned on the left, the number of keystrokes to get back to Reader isn't "one", and it isn't even constant. So, you end up needing to grab the mouse to get back, which is irritating.

So!

I made an extension for Chrome that adds "Shift-V" to Reader that opens the selected item in a background tab.

A couple slight hacks:

Shift-V instead of V: unfortunately, there doesn't appear to be a way to stop propagation of the event to Reader in the API exposed to extensions yet, so the background open can't replace the default open. Instead it has to go on a different key binding. Occasionally, I do actually want to read the item right away though, so I guess it handles that case. (I'd actually rather that they were swapped so that foreground open was Shift-V, but anyway...)
The chrome.tabs.create API doesn't have an option to open the tab in the background, so instead I have to save the current (Reader) tab, open the new tab, and then switch back to the Reader tab. On some machines (versions of Chrome?) this causes a visible flicker, but on the current as of right-this-minute, it doesn't seem to, so it'll do for now.

Yes, I've spent longer API spelunking and writing this post than I ever would have spent doing Ctrl-Shift-Tab/Ctrl-PgDn.

But, it's just one of those things that reduces friction and irritation and makes me happy. Or at least sates the OCD beast for a while.

Mini Jpeg Decoder

2009-12-02T00:00:00-08:00

Occasionally, I have the need to load some image data. Typically, the data can be relatively easily converted offline to raw RGB or .TGA using ImageMagick. By converting ahead of time to something simple, there's generally not much need to write $random_format_loader.

But, for some stuff I've been doing recently, the data's coming over the network, or is embedded in a larger file, so it's cumbersome to modify it as a preprocess.

So, I'd like to be able to load JPEG, PNG, and DDS (which is basically a wrapper around DXTn).

Tackling JPEG first, I was a little frightened of what I might be getting into. Most places point to libjpeg which is quite a scary huge beast. I'm sure it's mostly huge and scary for legitimate reasons (encoding support, various obscure variants, optimizations, etc.) but it's not something I'm interested in having a dependency on to just load a picture.

A little hunting turned up Tiny Jpeg Decoder by Luc Saillard, and NanoJPEG by Martin Fiedler. I found NanoJPEG nicer, and it's a very pleasant ~900 lines of straight C. The resulting image from NanoJPEG looked a bit better too.

I crammed Martin's implementation into one C++ .h file that doesn't require any supporting .cpp, and also hacked it up to be to my liking.

My mashed up version doesn't use any global/static data, so I guess there's a small benefit there if you were using it in multiple threads, but I haven't actually exercised that.

Martin's original version supports some configuration #defines for controlling different methods of upsampling, and avoiding using use of stdlib, and so is probably better for most people. But, here's mine anyway in case someone finds it useful.

Lightly tested on cl vs08 win64 and gcc 4.4.1 x86 ubuntu.

... PNG and DDS solutions to follow, once they're sorted out in similar fashion.

MinGL texturing

2009-11-17T00:00:00-08:00

In doing an old-school software triangle texture mapper, I've been trolling through some old parts of the internet.

I've found a huge amount of pages (like the semi-reborn venerable 3D Engines List) with pages that have fully 100% dead links.

There's tons of references to flipcode.com and x2ftp.oulu.fi both of which are long gone but well-remembered in some circles. The best reference I remembered was Chris Hecker's articles in GDMag, and happily he's got them online now because I'm pretty sure my dead-tree versions are long since recycled.

(As a side note, it seems like pure Institutional Retardation that Chris Hecker got laid off from Maxis.)

Anyway, today Jaap (apparently a dead link also right now), was extolling the virtues of sub-texel and sub-pixel accuracy with a righteousness that only an ex-flipcode'r could. :) Apparently the classic test of whether you've got it right is to do a 1:1 mapping and then rotate in 2D, parallel to the near plane (i.e. around Z).

So, here's MinGL doing that. I believe it's right, though the test texture causes some bad moiré. This is also quite possibly the most boring video ever.

For a long while last week, I was convinced that I had a bug in texel sampling. This is a little tricky to explain, but here's the situation.

First, a few definitions. Texel coordinates on a 64x64 texture range from (-0.5, -0.5) at the top left, to (63.5, 63.5) at the bottom right. Those values are the extents of the edge of texels, so the very centre of the top left texel is (0, 0) and the centre of the bottom right is (63, 63).

Similarly, the screen space coordinates range from (-0.5, -0.5) to (screenWidth - 0.5, screenHeight - 0.5) with the centre of the pixels being integral coordinates.

Now, if want to do a bitblt, i.e. have the texture drawn at 1:1 as if it was plain old 2D, there should be vertex positions where that will happen. And of course there is: with the vertex and texture coordinates at (-0.5, -0.5) to (63.5, 63.5), we get a perfect 1:1 mapping. Here's a screenshot of it, blown up after the fact:

Not very exciting, but it is perfectly 1:1, which is the sort of thing 2D hud artists will get all worked up about.

However, because we're just using point-sampling, it seems it should be fine to put the geometry anywhere. As long as the size is 64x64, then there should always be one texel per pixel regardless of whether it's offset by the same amount. However, when I put the vertices from (0, 0) to (64, 64), I get this artifact:

A brief digression into texture sampling...

Because we can only either light or not light individual pixels, and not anything smaller, we want to be sure we get as close to the right colour as possible for each pixel. Specifically, we want to sample the texture at the texel location that represents the centre of the pixel.

The triangle rasterization calculates gradients across the triangle. One of these is how much to step in the texture when moving across a scan line. So, to make sure that we always sample from what would be the centre of the pixel, at the beginning of the rasterization for each line, we just make sure to move the running sum of deltas so that it lines up with the centre of the pixel.

So, in the case where the vertex is also at a -0.5 offset and we're drawing 1:1, this prestep will be 0.5. If the vertex offset is -0.25 then the prestep will be only 0.25.

However, when the geometry is exactly aligned on integral coordinates, then the prestep is 0. This is the "edge" (har-har!) case, where the vertex lies exactly on the pixel centre, so there's no need to move at the start to align. Because the texture's being drawn 1:1, we're advancing the texture coordinates by 1 for every pixel, so we simply step along by 1, starting at 0.

It seems like the simplest possible case.

But, when actually looking up texture data the texel coordinates need to be converted to integers, because we're just point sampling, not blending. So, the texel coordinates have to be ceild or floord somewhere. So, when we're stepping starting at zero, and adding one, we might get sample data like:

0.000000, 1.000000, 2.000000, 3.000000, 4.000000, 4.999999988899, 5.9999999, ...

And, in using floor or truncate or ceil, or anything else we've just sampled the same texel twice when we really didn't want to. Of course, adding a bias just masks the problem in this case, and move it to some other offset of vertices relative to texels.

This probably woudldn't be very noticable in a fixed-point rasterizer because then the +1 step would be stable. The problem still exists for some other fractional value that doesn't-quite-make-it, but it's not when trying to do 1:1, so it probably goes undetected.

It was actually very difficult to get that screenshot of the artifact too, I had to disable the entire transformation pipeline, and pass screen coordinates into the texture mapper, and then try various pixel offsets.

Anyway, it's not really a practical problem because 1:1 geometry should just be aligned properly in 2D, but I thought it was an interesting investigation.

MinGL Basic Rendering Working

2009-11-15T00:00:00-08:00

Previously, I described my motivation for doing something as silly as writing an underfeatured software rasterizer in this day and age.

Well, I printed out the OpenGL ES 1.x specification, and I've been doing a little banging away. Some usual suspects that are evoking rage:

SSE sucks. As Max said on Friday, by SSE4+, they've gotten all the way to 75% of a good vector instruction set!
I wish there were better debugging environments on Ubuntu. I haven't tried the native interface on MonoDevelop 2.2 yet, perhaps that'll save me.
And of course the usual stupid C++ crap, but hey, "There are only two kinds of languages: the ones people complain about and the ones nobody uses." so I'll skip that for today.

It turns out that OpenGL is not greatly suited to (non-jitting) software implementations. Even in the pared-down, less-featureful, fixed-function version that is ES 1.0, there's an ungodly amount of insertion points where the pipeline can be reconfigured. To avoid runtime switching in the lowest level part of rasterization (i.e. drawing a scanline), there'd need to be hundreds of different "shaders" for the fixed function. For example:

vertex colour (bool)
texture0 (bool)
texture1 (bool)
fog (4 possible equations)
texture sampling method (6 different methods * 2 texture units)
colour mask (bool)
depth mask (bool)
stencil mask (bool)
all the fragment ops:
- pixel ownership (bool)
- scissor (bool)
- alpha test (bool)
- stencil test (bool)
- depth test (bool)
- blending (11 blend functions)
- logic ops (16 logic ops)

This doesn't include any vertex level operations, just the fragment level, and I'm sure I'm still missing some. That's also only for two texture units, which is the minimum the specification allows. You'd have to do a couple more x2's for each additional texture unit. Permuting out all the combinations, just with what's there though is already 17,301,504 shaders! So, clearly that's not going to work.

It's reasonable (I guess) to make all those part of a hardware pipeline for little additional overhead, but there's a crapload of checking, testing, and jumping going on to do that at a software level.

Another alternative would be runtime generation via JITing. This falls outside of what MinGL is going to do. The targetted platforms include x86, x64, PPC32, and PPC64 at a minimum which is way too much code to cram into a header. It's also not possible to JIT on all platforms I want it to run on, specifically Xbox 360 and PS3.

So, I guess I'll have to do something like picking 10-100 of these "shaders" that seem like average settings. e.g. tex0 + depth test + noblending, or colour + no depth test + one_minus_src_alpha, and so on. These will become a fast(er) path that's hopefully used 95%+ of the time, and other settings will have to fall back on the uber-do-all-tests-at-runtime shader.

Progress

Lest it seem like all bad news, I have sorted my way through a good chunk of the vertex pipeline, and have written a few of those fast-path shaders. Here's the always exciting gouraud + unlit in this case, doing what's probably the 3D version of "Hello, world!": a primary coloured rotating cube.

I also stuck the code up on code.google, on the off chance you want to trick out an SSE vector math library, or show off your mad texture mapping optimization skillZ, have at it.

mingl Very Early Progress

2009-11-08T00:00:00-08:00

On Friday at work, I was very irritated with the state of rendering for demo/sample purposes. Of course at EA, there's a vast array of snazzy engines that do asset management, post processing effects, particles, etc., etc., and are heavily optimized for the consoles and DX9+ platforms.

Unfortunately, all of this nifty functionality means that these quite-large libraries need to interoperate and depend on other subsystems so that they can be shared across a variety of game titles. As a result they have a huge number of dependencies, and an enormous footprint in conceptual API to master. The large number of dependencies means a lot of adapters to write or find and then integrate.

In practice what this means is that setting up a new test application is a lot of work. This in turn means that someone is more likely to slap the code for their mostly-separable library into an already-existing game, rather than keeping it a functionally separate library as soon as a demo or example program requires any rendering functionality.

Which is where we get to the point of this post. On the way home I was thinking "How hard can it be to write a simple, no-dependency rendering library?".

What I came up with was "mingl". Its goal is to be the SQLite or Lua of rendering very simple scenes. I want it to work on "all" platforms, for some definition of all, not require any configuration at all on any of those, and be simple to understand, maintain, and integrate. Continuing the parallel with SQLite and Lua, it certainly isn't going to be the fastest or most featureful rendering library. No one will be shipping a game with this library.

What I've started with is a software-only implementation of OpenGL ES 1.0. It might seem like madness, and perhaps it is, but if it's distributable as one .cpp and one .h that you slap into your project to render some triangles, I'll be a happy guy. (In fact, I might try to cram it all into inline classes to get it down to just a .h).

On that note, I've poked around for two evenings now and I've got enough of the API roughed in, along with a texture mapper that would make John Carmack, Michael Abrash, Chris Hecker, et al. cringe. If you don't recognize those names from the days of yore... well, let's just say it ain't going to win any speed records. But hey, it's (just barely) working:

Performance Of Closure Compiler On Skulpt

2009-11-05T00:00:00-08:00

As is flying around today (at least in some very specific circles), Google released the unfortunately named Closure Tools.

There's 3 components:

a template system,
a "standard" library for JS (they're comparing it to STL or the JDK),
a JS-to-JS compiler.

I can't say I care much about the template system, as I haven't had much occasion to use a templating client-side. But, I need to look at it more to understand what its intended uses are.

The standard library looks pretty decent. I think the most interesting part is definitely goog.ui.* which appears to be most of the UI widgets that are used in the building of Gmail, Maps, etc. There's some demos via their SVN. These aren't flashy demos, but there's broad functionality, and (I assume) broadly cross-browser tested support here, which is a huge time saver.

And the compiler. I've been using the popular YUI compressor on Skulpt. It's easy to use, robust, and gives good compression results.

The Skulpt code seemed like a good test for the Closure compiler. Below, "selfcomp.js" is a debug build of Skulpt (written in Python) compiling itself to JavaScript. This results in a pretty big JS file, weighing in at about a meg. Not unreasonable for a full application though, given that the debug build is heavily commented, and we're talking pre-minification.

Below is the transcript of a quick Closure compiler test.

~/skulpt$ ls -l support/tmp/selfcomp.js 
-rw-r--r-- 1 sgraham sgraham 965008 2009-11-05 00:21 support/tmp/selfcomp.js

~/skulpt$ time java -jar support/yui/yuicompressor-2.4.2.jar support/tmp/selfcomp.js \
    -o yui.js
real    0m6.340s
user    0m8.393s
sys 0m0.172s

~/skulpt$ time java -jar support/closure-compiler/compiler.jar --js support/tmp/selfcomp.js \
    --js_output_file closure.js
real    4m21.287s
user    4m27.177s
sys 0m1.660s

~/skulpt$ time java -jar support/closure-compiler/compiler.jar --js support/tmp/selfcomp.js \
    --compilation_level ADVANCED_OPTIMIZATIONS --js_output_file closure_adv.js
java.lang.RuntimeException: java.lang.RuntimeException: INTERNAL COMPILER ERROR.
...

~/skulpt$ gzip closure.js 
~/skulpt$ gzip yui.js

~/skulpt$ ls -l *.gz
-rw-r--r-- 1 sgraham sgraham 24459 2009-11-05 19:11 closure.js.gz
-rw-r--r-- 1 sgraham sgraham 40814 2009-11-05 18:31 yui.js.gz

Yeah, almost four and half minutes to compile vs. 6 seconds for the YUI compressor. I mean, that's fine, it's not like you're doing it on every change, but it's still long enough to be irritating. It's as bad as linking on the PS3 or 360!

I haven't tracked down why the ADVANCED_OPTIMIZATIONS version doesn't work yet. I'm sure that's a fixable problem and/or something in the Skulpt-generated code that violates the assumptions that closure-compiler uses. Not too worried about that.

Speed improvements however, would definitely be welcomed.

But, even without the "advanced" optimizations, the end result post-gzipping is pretty awesome. In this case it's only 60% of the size of the YUI version which is damn impressive.

Update

I was missing some extern declarations that caused the ICE. With those, the advanced version completes successfully, and I'm getting damn near close to a 50% reduction compared to YUI.

-rw-r--r-- 1 sgraham sgraham    21091 2009-11-05 20:48 closure_adv.js.gz

Pap Music Piping

2009-11-04T00:00:00-08:00

There's this horrible, over-produced, top 40, shit music that's piped through the office. Thankfully it's only in the stairwells, elevator area, bathroom, etc. and not audible from anywhere near where I sit. Though I suspect there's some poor bastard that has the Worst Cubicle Ever that can hear it all day.

Anyway, it's been like this since that phase of the building opened 3-4 years ago. So this isn't a new thing, it's been the same insipid music ever since someone made the mistake of wiring in the possibility.

The last two days though, I was sitting taking a crap and realized just how much it sucks and irritates me.

Taking a huge dump is one of the best times for a complicated technical issue to crystalize, or for the cause of a particularly nasty bug to become obvious. It's second perhaps only to the supremely productive meditative time during a morning shower.

But no.

Instead, I'm sitting listening to Miley Cyrus or Hilary Duff or Someone-Who-Just-Had-a-Photo-Shoot-in-Teen-With-This-Month's-Hottest-Vampire!!

I have visions of smashing a screwdriver into the metal speaker grating. I expect it'd be very satisfying.

Google Web Toolkit (GWT) on Karmic

2009-10-31T00:00:00-07:00

I wanted to try out a more recent version of GWT, it's supposed to have support for mobile browsers as well as some other shiny stuff.

I installed Eclipse on Ubuntu 9.10 (Karmic), which is the most recent version (3.5.1). Google's got a nice plugin that you can install in one place and it sets up templates, and installs the AppEngine SDK and the GWT SDK.

So far so good.

Unfortunately, when I try to run any application in Eclipse, I get this lovely message:

** Unable to load Mozilla for hosted mode **
java.lang.UnsatisfiedLinkError: /home/sgraham/.eclipse/org.eclipse.platform_3.5.0_155965261/
plugins/com.google.gwt.eclipse.sdkbundle.linux_1.7.1.v200909221731/gwt-linux-1.7.1/
mozilla-1.7.12/libxpcom.so:
libstdc++.so.5: cannot open shared object file: No such file or directory

I guess there's some binary version of Mozilla they're using internally, but it's built against an old shared stdc++ library that Karmic doesn't ship anymore. libstdc++5 isn't installable from the repos, but I just grabbed the .deb from the Jaunty page here and installed it.

And... all seems well now. I don't really know if that has any adverse effects, but I haven't noticed anything yet.

New Software

2009-10-26T00:00:00-07:00

I've tried Wordpress, Blogger, and Tumblr (most recently), but they're a sack of ass. The text editors are horrible and butcher the posts, and it's just about damn impossible to paste/type code into a post and expect it to turn out reasonable. I thought I could trade the asseyness of them for the less hassle that a hosted service should offer, but apparently I can't.

So, I sucked it up and configured Jekyll. It's pretty manual to set up, but it's just a static site generator from a simpler-than-html-language (markdown et al.) and spits out some stuff.

I went "fancy" (OK, not really) and used Cufón on the headings too. It's a pretty neat JS library+tool. It renders a font to VML + another format as data for a JS script, so then can incrementally replace text with your custom one, either by using VML on IE, or by using <canvas> on all other browsers. Et voila, The New Yorker-ish heading font.

Anyway, it's easier to post now, and I can edit in Vim, but I probably won't still, because, well, 140 characters about lunch is a lot less work than writing thoughtful essays.

Team Composition

2009-10-25T00:00:00-07:00

Team makeup

Pragmatist, Perfectionist, Crazy Smart

http://heroku.com/about

Interactions

A quote to start: This quote is the designers’ reflections on designing a programming language (SIMULA).

In the spring of 1967 a new employee at the NCC in a very shocked voice told the switchboard operator: “Two men are fighting violently in front of the blackboard in the upstairs corridor. What shall we do?” The operator came out of her office, listened for a few seconds and then said: “Relax, it’s only Dahl and Nygaard discussing SIMULA”.

The story is true. The SIMULA 67 language was the outcome of ten months of an almost continuous sequence of battles and cooperation in front of that blackboard - interrupted by intervals when we worked in our neighbouring offices (communicating by shouting through the wall if necessary) or at home. (The people arranging this conference asked us to provide material relating to the development of our respective languages. We felt that the best thing we could have done was to bring along that blackboard. But we did not know for certain whether we would be flying a wide-body aircraft.)

In some research teams a new idea is treated with loving care: "How interesting", "Beautiful". This was not the case in the SIMULA development. When one of us announced that he had a new idea, the other would brighten up and do his best to kill it off. Assuming that the person who got the idea is willing to fight, this is a far better mode of work than the mode of mutual admiration. We think it was useful for us, and we succeeded in discarding a very large number of proposals.

(from http://www.oberon2005.ru/paper/kn1978-01e.pdf via ltu)

Isn’t that fantastic? You know that the "idea-defender" here would need to quickly detach himself from any silly latent emotional attachment to an idea. Any idea in that situation is going have to live on its own merits.

If you’ve got someone who you know will rip into your shitty ideas with pointed criticism, then:

you’re not going to do dumb stuff; and
you’re going to know that you've got a winner when you get them on board.

That kind of relationship is a huge asset. It’s one thing that turns you into (the very cliched) “more than the sum of your parts”. If you can do this, you’re really combining your 2+ brains effectively.

Drop your ego, cultivate these relationships, and then hang on to them for dear life.

Really? REALLY? WTF.

2009-07-25T00:00:00-07:00

js> x = [];
js> x[0] = "num0";
num0
js> x[1] = "num1";
num1
js> x[2] = "num2";
num2
js> x
num0,num1,num2
js> x["1"] = "str";
str
js> x
num0,str,num2
js>

Someone please tell me there's a way to avoid this idiotic behaviour.

Imgburn Rocks The Shizzle

2009-02-09T00:00:00-08:00

Huzzah!

2009-01-28T00:00:00-08:00

First Clojure Program

2008-11-15T00:00:00-08:00

So, I watched a few of Rich’s videos on Clojure a couple months back, but I finally got an afternoon to spend trying it out.

I skimmed through Ruby Quiz to find an interesting but simple problem, and I decide to write a Sokoban clone.

My initial thought was to do it as a gui app, but I haven’t written any Java code since JDK 1.0-alpha in late 1995, and honestly, I didn’t really feel like putting effort into learning the Java GUI API only to end up with a bizarrely emulated, slightly broken GUI.

So, I grabbed Compojure off of git, and made the UI a hack webapp. (Yes, it’s bizarrely emulated and slightly broken still, but at least it was easy to write).

First, have a look at the result: Clojban!

If you care, the (probably horrible and un-idiomatic) code follows.

(ns clojban
  (:use (compojure html
                   http
                   file-utils)
        (clojure.contrib str-utils
                         seq-utils)
        (clojure set))
  (:import (java.util.regex Pattern)))

(defstruct pos :x :y)
(defstruct level
           :player ; pos
           :boxes  ; set-of-pos
           :goals  ; set-of-pos
           :walls  ; set-of-pos
           )

(defn explode
  "return a pos/type seq for each char in the line"
  [y line]
  (map (fn [x y type] (vector (struct pos x y) type))
       (iterate inc 0) (repeat y) line))

(defn line-explode
  "call explode on each line with the appropriate y"
  [lines]
  (let [linesplit (. Pattern compile "\n" (. Pattern MULTILINE))] ; must be a better way?
    (reduce into
            (map explode
                 (iterate inc 0)
                 (. linesplit split lines)))))

(defn filter-level
  "extracts a set of the pos's that match types"
  [expl-level type1 type2 &amp; type3]
  (set (map first
            (filter (fn [item]
                      (or (= (second item) type1)
                          (= (second item) type2)
                          (= (second item) type3)))
                    expl-level))))

(defn load-levels
  "load and return seq of level's"
  [filename]
  (let [contents (slurp filename)
        levelsplit (. Pattern compile "^$\n" (. Pattern MULTILINE)) ; each level separated by blank line
        levels (seq (. levelsplit split contents))]

    (map (fn [levelstr]
           (let [expl-level (line-explode levelstr)]
             (struct-map level
                         :player (first (filter-level expl-level \@ \+))
                         :boxes (filter-level expl-level \o \*)
                         :goals (filter-level expl-level \. \* \+)
                         :walls (filter-level expl-level \# \#))))
         levels)))

(def Levels (load-levels "sokoban_levels.txt"))
(def LvlWidth 19)
(def LvlHeight 16)
(def PlayerRenderOpen "♦")
(def PlayerRenderGoal "♦")
(def BoxRenderOpen "▨")
(def BoxRenderGoal "▨")
(def WallRender "█")
(def GoalRender "░")

(defn level-index [x y] (+ (* y LvlWidth) x))



(defn render-level-layer
  "my, what an insane level representation i have here"
  [target items if-empty if-full]
  (loop [target target
         remain (seq items)]
    (if remain
      (let [x (:x (first remain))
            y (:y (first remain))
            i (level-index x y)
            at (target i)]
        (recur (assoc target i (if (= at \space) if-empty if-full))
               (rest remain)))
      target)))

(defn render-level
  [lvl]
  (let [finallevel (reduce (fn [lvl data] (render-level-layer lvl (nth data 0) (nth data 1) (nth data 2))) ; todo how to splat `data'
                           (vec (replicate (* LvlWidth LvlHeight) \space))
                           `((~(:walls lvl) ~WallRender ~WallRender)
                             (~(:goals lvl) ~GoalRender ~GoalRender)
                             (~(:boxes lvl) ~BoxRenderOpen ~BoxRenderGoal)
                             (~(set (list (:player lvl))) ~PlayerRenderOpen ~PlayerRenderGoal)))]
    (dotimes y LvlHeight
      (print "")
      (dotimes x LvlWidth
        (print "")
        (print (finallevel (level-index x y)))
        (print ""))
      (print ""))))

(def AsciiToDx {72 -1,
                76  1,
                75  0,
                74  0})
(def AsciiToDy {72  0,
                74  1,
                75  -1,
                76  0})

(defn do-player-move
  "handle player movement, input is int ascii code for HJKL."
  [level input]
  (let [dx (get AsciiToDx input)
        dy (get AsciiToDy input)
        curpos (:player level)
        candidatepos (struct pos (+ (:x curpos) dx) (+ (:y curpos) dy))
        pastcandidatepos (struct pos (+ (:x curpos) dx dx) (+ (:y curpos) dy dy))
        walls (:walls level)
        boxes (:boxes level)]
    (if (walls candidatepos)
      level
      (if (boxes candidatepos)
        (if (or (walls pastcandidatepos) (boxes pastcandidatepos))
          level
          (assoc
            (assoc level :boxes (conj (disj boxes candidatepos) pastcandidatepos)) ; move box
            :player candidatepos)) ; and player
        (assoc level :player candidatepos))))) ; only player

(def JSCode "
function postwith (p) {
  var myForm = document.createElement('form');
  myForm.method='post';
  myForm.action='/';
  for (var k in p) {
    var myInput = document.createElement('input');
    myInput.setAttribute('name', k);
    myInput.setAttribute('value', p[k]);
    myForm.appendChild(myInput);
  }
  document.body.appendChild(myForm) ;
  myForm.submit() ;
  document.body.removeChild(myForm) ;
}

function handlekey(e) {
  if (!e) var e = window.event
  if (e.keyCode) code = e.keyCode;
  else if (e.which) code = e.which;
  if (code == 37) code = 72;
  if (code == 38) code = 75;
  if (code == 39) code = 76;
  if (code == 40) code = 74;
  if (code == 72 || code == 74 || code == 75 || code == 76 || code == 82 || code == 65 || code == 90) postwith({'code': code});
}
")

(defn restart-level [session]
  (alter session assoc :curlevel (or (session :curlevel) 0))
  (alter session assoc :complete (or (session :complete) (set nil)))
  (alter session assoc :nummoves 0)
  (alter session assoc :level (nth Levels (or (session :curlevel) 0))))

(defn next-level [session]
  (alter session assoc :curlevel (min
                                   (inc (session :curlevel))
                                   (- (count Levels) 1)))
  (restart-level session))

(defn prev-level [session]
  (alter session assoc :curlevel (max (dec (session :curlevel)) 0))
  (restart-level session))

(defservlet clojban-servlet
  (POST "/"
        (dosync
          (let [prevlevel (session :level)
                keycode (. Integer parseInt (params :code))]
            (if (= keycode 82)
              (restart-level session)
              (if (= keycode 65)
                (prev-level session)
                (if (= keycode 90)
                  (next-level session)
                  (let [moveresult (do-player-move prevlevel keycode)]
                    (if (= 0 (count (difference (:boxes moveresult) (:goals moveresult))))
                      (do
                        (alter session assoc :complete (conj (session :complete) (session :curlevel)))
                        (next-level session))
                      (do
                        (alter session assoc :nummoves (inc (session :nummoves)))
                        (alter session assoc :level moveresult)))))))))
        (redirect-to "/"))
  (GET "/"
       (let [levelstate (or (session :level) (dosync (restart-level session) (session :level)))]
         [{"Content-Type" "text/html"}
          (html
            (doctype :xhtml-transitional)
            [:html {:xmlns "http://www.w3.org/1999/xhtml" :lang "en"}
             [:head
              [:title "Clojban!"]
              [:meta {:http-equiv "Content-Type", :content "text/html; charset=utf-8"}]
              [:script {:type "text/javascript"} JSCode]]
             [:body {:onkeydown "handlekey(event);" :style "font-family: arial; font-size: 13px;"}
              [:h1 {:style "font-family: arial;"} "Clojban"]
              [:p "This is a silly <i>Sokoban</i> implemented in " (link-to "http://clojure.org" "Clojure") ". It's hitting the server every time you move (rather than Javascript) so it might not respond too quickly. The unicode box drawing craziness looks OK in FF, and reasonable in IE, but not so hot in Chrome/Webkit. Sorry."]
              [:table {:style "border-style: none; font-size: 36px; font-family: courier; padding: 0px 0px 0px 0px; margin:0px 0px 0px 0px; line-height: 1em;"
                       :border "0"
                       :cellpadding "0"
                       :cellspacing "0"}
               (with-out-str (render-level levelstate))]
              [:p "Use arrow keys or hjkl to move all the boxes into the goals. Press r to restart level, or a/z to give up and skip through levels."]
              [:p "Level: " (session :curlevel)]
              [:p "Moves: " (session :nummoves)]
              [:p "Completed: " (str-join " " (map (fn [i]
                                                     (let [complete (session :complete)]
                                                       (format "%d"
                                                               (if (complete i) "green" "#ccc")
                                                               i)))
                                                   (range (count Levels))))]
              ]])]))
  (ANY "*"
       (page-not-found)))

Configuring Vim Some More

2008-11-10T00:00:00-08:00

As an addendum to http://items.sjbach.com/319/configuring-vim-right

Here's a few more that I think are un-live-able-without:

" ease of use keyboard mappings (why do I care about top/bottom of screen?)
map H ^
map L $

" buffer switching/management, might as well use those keys for something useful
map <Right> :bnext<CR>
imap <Right> <ESC>:bnext<CR>
map <Left> :bprev<CR>
imap <Left> <ESC>:bprev<CR>
map <Del> :bd<CR>

" get rid of stupid scrollbar/menu/tabs/etc
set guioptions=a

" don't need /g after :s or :g
set gdefault

" i prefer this to visualbell
set noerrorbells

" Hide the mouse pointer while typing
set mousehide

I've used all of these for longer than I can remember, probably time to troll through recent help files and vim.org to find new and exciting juicy settings.

EDIT: forgot the one I miss the most when I don't have my .vimrc:

cab o find

so that :o does something useful. I'd like one of the emacs-style smart-fuzz-file-opener command line thing, I should probably hunt around for a plugin to do that.

Bill C-61

2008-08-24T00:00:00-07:00

Canada’s Bill C-61 is an even-worse version of the United States’ DMCA.

For more information, http://www.michaelgeist.ca/ is a reasonable starting point. I rolled up my sleeves and had a go at http://www2.parl.gc.ca/HousePublications/Publication.aspx?Docid=3570473&file=4 which is relatively readable.

Here’s the letter I just posted (yes, on actual paper, in an envelope!) to my MP.

If you’re Canadian, please read the above links and if you don’t agree with this Bill, as I’m confident you won’t, phone or write to your MP (especially if your MP is a Conservative Party member!)

Office of the Honourable Dr. Hedy Fry, P.C., M.P.
Confederation Building House of Commons
Ottawa, Ontario
K1A 0A6

Hello Dr. Fry

I am a resident of Vancouver Centre, and I'm writing to you to express my
absolute horror at the current state of Bill C-61.

I am employed as a software engineer in the entertainment industry, and as
such, fully respect and understand the need for copyright reform in
Canada.

However, from reading this bill (at
http://www2.parl.gc.ca/HousePublications/Publication.aspx?Docid=3570473,
the reproduction rights afforded to Canadians are absolutely useless
because of section 29.21(c). This section subordinates any time-shifting
and media-shifting rights that should be allowed, merely by having any
kind of digital lock mechanism on the content. But, virtually all media
sold in Canada now contains a digital lock of some sort, so it's almost
pointless to "grant" these rights.

Inevitably, the devices or internet services which control the digital
lock access fail, go out of business, or otherwise are not maintained. If
we are unable to legally shift to other formats, and the media become
un-unlockable, they are effectively gone. Will all of the companies and
individuals controlling the locks exist in 5 years? Maybe. 50 years?
Unlikely.

I would also like to especially point out that this does not solely
concern things that could perhaps be considered "frivolous" like
pop-consumer-ish music and video. Over the next 10-15 years, it is all but
guaranteed that most books, and crucially text books, will be delivered
and consumed in electronic form. My wife is a high school teacher employed
by the Vancouver School Board, and I would be horrified to find that our
already underfunded education system had lost access to teaching
resources, simply because the company controlling a digital lock had
failed.

I sincerely hope you will make sure that this Bill is heavily modified, or
defeated, when the time comes.

Thank you for your time,

Scott Graham
*my address removed*

Braid

2008-08-09T00:00:00-07:00

Braid is very cool. I often enjoyed reading The Inner Product in GDMag, but always got the impression Jonathan was a little “ivory tower”. Turns out, I don’t know shit. Congrats to Mr Blow! (just grabbed the full 360 version: it’s fantastic).

Floating Point

2008-08-06T00:00:00-07:00

Floating-point tends to chafe my ass. I was thinking perhaps I’d try to really understand it. These look interesting:

Only way to understand something is to implement it though…

Update: even better by the looks of it: http://docs.sun.com/source/806-3568/ncgTOC.html.

Second Das Keyboard

2008-07-29T00:00:00-07:00

So, I just got my second http://www.daskeyboard.com/ in the mail. The first one I had was the “original”, and I loved and continue to love it. I’ve been using it as my work keyboard (where I do the majority of my coding) for the last 3 years or so.

I just got a second one, but it’s the “new” version (the only one they offer now). It’s cleaner design-wise, but there’s some things that I think they’ve made worse.

The first is that the keyboard is just too thick. It’s gotten half an inch (or so) thicker. It feels more solid now, and is quite a bit heavier (so it doesn’t move around at all, which is nice), but the angle of my forearms is slightly higher, which inevitably means my wrists are more crooked when I get more tired and drop my arms, and slouch, etc. Don’t like that very much, especially as I’ve had some wrist and arm pain recently.

They also seem to have killed the variable resistance on keys (or at least tuned them differently). The only key I notice that’s harder resistance is the backspace key, whereas the space key used to have a lot more resistance, and doesn’t all now. Perhaps the resistance has changed over time on my original one, so it’s just a matter of getting used to this one, or breaking it in more. Maybe it’s more expensive to do the bands of keys with different resistances, but I’m pretty sad about it, especially since this one ended up being about 30% more expensive than the original.

Oh, and the F and J have icky bumps instead of being nicely scooped. :( I’m guessing that one was a concession to people who weren’t very good typists, and couldn’t find the home row, but I prefer the scoops, since we are going for “stealth” here, after all. They just felt nicer on your finger tips too.

The sound of the switches has changed too, but that’s just… different, not really better or worse.

I’m going to guard my old one even more carefully now with the knowledge that they’re not really replaceable. Hopefully I’ll grow to love the new one as much, but the jury’s still out.

Problems With Vinagre

2008-07-26T00:00:00-07:00

So, the default remote desktop app in Ubuntu changed from, um, I don’t know what actually, to a new app called “Vinagre”.

You’d think that’d be something I wouldn’t give a patooty about. The first thing you notice is that it has a list on the side that lets you keep track of servers you connect to which seemed nice enough.

I didn’t use it much at first, so I thought nothing more about it.

Today, I started using it. Dear god:

Tried to connect to a fully up-to-date Fedora box (from a fully up-to-date Ubuntu). vino-server crashes on the Fedora machine. I don’t know who’s fault it is, but I don’t really care. Working around that by using XDMCP for the time being, but that’s pretty irritating, interface-wise.
While repeatedly trying to connect to the Fedora box (it silently fails on the Ubuntu machine), Vinagre doesn’t save the last-entered machine name, so I have to keep entering “192.168.0….” every time. Irritating.
Insanely hard to send Ctrl-Alt-Del to log into Windows machines. For same reason, Ctrl-Alt was chosen as the “Capture/Release” input. So, in order to send Ctrl-Alt-Del to login to a Windows box, you have to first focus the Viagre app (click/whatever), then make sure you’ve uncaptured the input by doing Ctrl-Alt, then finally, recapture the input by doing Ctrl-Alt again, and then without letting go of Ctrl-Alt, press Del to get the final combo. If you just push that combo of course, the Ctrl-Alt first removes capture, and then sends Ctrl-Alt-Del to the local machine. Not very intuitive, especially since there’s very little (no?) indication of whether input is captured.
Related, there seems to be absolutely no way to send F11 to the client machine. The general hotkey behaviour is irritating enough as it is: Alt-F4 closes either all of Vinagre, or an app on the client, depending on whether you’ve pressed Ctrl-Alt recently, with no indication of which “mode” you’re in). But, it seems that F11 is even worse. If you’re in captured mode, then most things seem to be sent to the target machien, but apparently F11 was deemed too important to be handled normally, and there’s no configuration mechanism. Fine, until I try to debug something and naturally hit F11 to “Step Into” a function.

Update When input is captured, there’s no way to scroll the virtual desktop if it’s bigger than your current monitor. WTF?

Basically, it just seems like this app was pushed out way too quick to the main/default user stream. Please give me back my old VNC/RDP viewer. :(

Windows iTunes

2008-07-12T00:00:00-07:00

I genuinely don’t understand.

I’m not trying to be difficult. I really tried to use iTunes this time. It seemed like the march of inevitability if I wanted to get iPhone apps, and I thought perhaps it would be nice to use. I grabbed the new 7.7 that has iPhone apps, since I wanted to have a look at what was available. It was a little large at 60meg, but hey, whatever, there’s lots of bandwidth to go around these days.

First of all, I had to reboot after installing it. Insane. Strike one. If that was it, well fine. But it’s completely unusably slow on my laptop. The laptop is about 2 years old. It was Dell’s top-of-the-line (XPS M170) when I bought it, so it’s not crazy fast, but it’s not so slow either. I’m pretty sure that it’s still well above the current median machine in performance.

I added my music collection (largely stored on a network drive). It’s about 65 gig, comprised mostly of mp3s and flacs, both ripped from CDs and downloaded. Maybe a little larger than some collections, but certainly not unreasonably large.

I started the “Import” last night around 8pm. It’s now 2pm the next day, and it’s still trying to download album artwork. That’d be fine, except that apparently while it’s doing that, it’s impossible for the UI to be responsive. WTF! Is this goddamn amateur hour? It only refreshes the UI at the end of each attempt to download an album cover?!

Pausing or unpausing music, or changing the volume (since apparently it feels the need to steal the functionality of the hardware media keys) takes over 10 seconds. That is not hyperbole! It actually takes longer than 10 seconds for anything to happen. On top of that, the mouse event handling also appears to be “sampling” rather than using the event queue properly, because if I just click the play/pause button nothing happens. I have to hold the fucking mouse button down until the UI responds (10 to 15 seconds, remember!), putting the button into the down state, and then release it.

On a side note, iTunes.exe currently owns roughly 300 threads, and has had a continuous IO delta of around 500k/sec for nearly 24 hours now. Jumped up Jesus Christ. You’re not controlling air traffic, you’re cataloging and playing some fucking music.

If this is Apple’s shitty attempt to create some sort of Mac “halo”, they’ve failed miserably. With 100% certainty, I will never, ever, by a Mac with this as the demo of the user experience.

Project Euler, Problem 10 in F#

2008-07-12T00:00:00-07:00

Problem’s quite simply stated this time:

The sum of the primes below 10 is 2 + 3 + 5 + 7 = 17.

Find the sum of all the primes below two million.

First typed in solution was simply:

[2..1999999]
|> Seq.filter(isprime)
|> Seq.fold1(+)

Which got me a number pretty quickly that looked lovely, but was completely wrong. I ran it again, apparently hoping that’d it magically be right the second time. Then I tried for those below 1000000 instead and the sum was negative. Oops! So, the quick fix and working version (changing the “int” range to “bigint” range):

[2I..1999999I]
|> Seq.filter(fun x -> isprime(int x))
|> Seq.fold1(+)

I wonder if there’s a checked-overflow-arithmetic version of F#? Might be useful for some of these questions, since they’re so computationally light anyway.

FSharp, Euler 9

2008-05-24T00:00:00-07:00

The problem statement:

A Pythagorean triplet is a set of three natural numbers,

a² + b² = c²

For example, 3² + 4² = 9 + 16 = 25 = 5².

There exists exactly one Pythagorean triplet for which a + b + c = 1000.

Find the product abc.

This is another one that’s probably really more “math”-y than programming-y, but I’m enjoying using my new F# hammer to solve these, so might as well keep at it.

It seems like another nice make-a-sequence-and-then-filter-it, but just doing 1..1000 for a, b, and c, results in an awfully big sequence (a billion long). So, a couple simple observations to make the numbers a lot smaller. The main one is that you can just make c = 1000 - a - b, since we know that all answers have to have that form anyway.

seq { for a in 1..1000 do
        for b in a..1000 do
            let c = 1000 - a - b
            yield (a,b,c) }
|> Seq.first(fun (a,b,c) -> if a*a + b*b = c*c then Some(a,b,c) else None)

Voila!

Euler 8 in FSharp

2008-05-19T00:00:00-07:00

The problem:

Find the greatest product of five consecutive digits in the 1000-digit number.

73167176531330624919225119674426574742355349194934 96983520312774506326239578318016984801869478851843 85861560789112949495459501737958331952853208805511 12540698747158523863050715693290963295227443043557 66896648950445244523161731856403098711121722383113 62229893423380308135336276614282806444486645238749 30358907296290491560440772390713810515859307960866 70172427121883998797908792274921901699720888093776 65727333001053367881220235421809751254540594752243 52584907711670556013604839586446706324415722155397 53697817977846174064955149290862569321978468622482 83972241375657056057490261407972968652414535100474 82166370484403199890008895243450658541227588666881 16427171479924442928230863465674813919123162824586 17866458359124566529476545682848912883142607690042 24219022671055626321111109370544217506941658960408 07198403850962455444362981230987879927244284909188 84580156166097919133875499200524063689912560717606 05886116467109405077541002256983155200055935729725 71636269561882670428252483600823257530420752963450

Treating it as a number is probably not going to be too pleasant.

let prob8num = "7316 ... 52963450"
let chars = prob8num.ToCharArray() |> Array.map(fun x -> Char.code(x) - Char.code('0'))
let rec prob8 (rest : array) =
    if rest.Length     else
        let tl = Array.sub rest 1 (rest.Length - 1)
        let curProd = Array.sub rest 0 5 |> Array.reduce_left( * )
        Math.Max(prob8(tl), curProd)
prob8 chars

So, store it as a string, but then convert it to a character array, and map to the actual numerical values (assuming ASCII I guess).

From that, just a simple recursive solution that does way too much Array.sub’ing. It would make much more sense to use List’s instead of Array’s for all of those .subs, but, there was no sub extraction built into List so I just did it that way instead.

I wrestled a little with the syntax of F# again in this question. I tried putting the “let tl” as inline into the body of the call to Math.Max, but I couldn’t get it to work correctly. Not sure what the required syntax is, maybe I need some of the keywords that I don’t know from non-#light yet?

Also, my copy of Expert F# has arrived, so I’m just starting to peruse it now. Hopefully my code will become more idiomatic soon enough.

And man, do I hate blog software’s handling of code. You’d think that maybe if I typed a less-than sign they could figure out how to escape it moderately properly, but apparently that’s way too complicated. Idiotic.

Project Euler, Problem 7 in FSharp

2008-05-18T00:00:00-07:00

The problem is:

By listing the first six prime numbers: 2, 3, 5, 7, 11, and 13, we can see that the 6^th prime is 13.

What is the 10001^st prime number?

Well, this sort of looks like another one that’s intended more for pencil-and-paper solvers than programming-language solvers. But, maybe prime #10001 is very big, so it becomes a problem for code too. My first thought is just to loop with a counter until we find the 10001st number, but that involves variables which I’m apparently trying to avoid in these problems. I didn’t get to use infinite lists in the last question, so let’s try again:

let rec getprime(n) =
    if n = 1 then 2
    else
        let prev = getprime(n - 1)
        Seq.init_infinite(fun x -> x + prev + 1)
        |&gt; Seq.find(isprime)getprime(10001)

First, we assume we have the previous prime (say it’s “11”). Then, we generate an infinite list starting at the next number (i.e. [12..inf]). Then, find the first number in that list that’s prime, which is the prime after the previous one, which is the one we’re looking for. Finally, take that functionality and wrap it in a recursion stopping at prime #1 = 2 and we’re done.

When I first typed this code in, I messed up the initial value for the start of the sequence as (x + prev), which of course meant it kept finding “2” as the answer. Just a dumb mistake, but the interesting part is that I’d forgotten I even had a debugger since I was getting so used to using FSI to run bits of code. In the end I found the mistake by replacing Seq.find with Seq.nth(0), and adding a couple Console.WriteLine()s.

It’d be really nice if there was a way to put focus onto the FSI tab window though. I find myself defining functions by using Alt-Enter on the body of the function in the text editor, and then wanting to switch to the window to try passing it a few values to see how it works. Without a shortcut, I have to grab the mouse, click in the right place, sometimes Ctrl-End to find the prompt, etc. Maybe I just haven’t found the key yet, not sure.

Euler Project in F# Problem 6

2008-05-15T00:00:00-07:00

Kick it, Euler:

The sum of the squares of the first ten natural numbers is,

1² + 2² + … + 10² = 385

The square of the sum of the first ten natural numbers is,

(1 + 2 + … + 10)² = 55² = 3025

Hence the difference between the sum of the squares of the first ten natural numbers and the square of the sum is 3025 - 385 = 2640.

Find the difference between the sum of the squares of the first one hundred natural numbers and the square of the sum.

Not much to this one. I learned about the exponentiation operator which seems to be a rewrite to calling a Pow method (it doesn’t seem to work on int). Here’s a “my brain is currently warped into thinking about everything as a sequence and operations on sequences”-solution:

([1.0..100.0] |> Seq.fold (+) 0.0) ** 2.0
- ([1.0..100.0] |> Seq.map(fun x -> x*x) |> Seq.fold(+) 0.0)

See y’all next time.

Dynamic Languages

2008-05-14T00:00:00-07:00

...

Because we couldn’t see how the system worked anymore!

Small systems are not only easier to optimize, they’re possible to optimize. And I mean globally optimize.

So when we talk about performance, it’s all crap. The most important thing is that you have a small system. And then the performance will just fall out of it naturally.

http://steve-yegge.blogspot.com/2008/05/dynamic-languages-strike-back.html

Euler is streaking in F#, problem 5

2008-05-13T00:00:00-07:00

Problem #5:

2520 is the smallest number that can be divided by each of the numbers from 1 to 10 without any remainder.

What is the smallest number that is evenly divisible by all of the numbers from 1 to 20?

Speaking of infinite sequences, here’s the solution that seemed obvious to me at first.

open Microsoft.FSharp.Math
Seq.init_infinite (fun x > BigInt.FromInt32 x)
|> Seq.find(fun (x:BigInt) ->
    x > 1I && Seq.for_all (fun (n:BigInt) -> x % n = 0I) [1I..20I])

Nice and simple, just iterate through all numbers, and find the first ne that’s evenly divisible by all the numbers from 1 to 20. Unfortunately let it run for an hour (!) while I had dinner, and it hadn’t completed. I just assumed was doing something dumb, but it seems to give sensible answers for the numbers in the range of 1..10 and 1..16. Apparently, at 1..17 or more though the answer becomes “quite” large. The first version just used int32’s rather than BigInt; I thought perhaps it was getting past 2^32 and so never finding the answer, but it’s hard to say since I wasn’t prepared to wait any longer for the BigInt version to finish.

As a side note, the bigint stuff is a bit ugly in this example, I guess an unfortunate side effect of .NET showing through where the numerical stack is fractured (sensibly and everything, just a little unfortunate here).

In any case, a more mathematical and less brute-force algorithmic approach seems to be required for this problem.

Here’s one that takes only milliseconds to run based on:
http://en.wikipedia.org/wiki/Least_common_multiple#Alternative_method

let rec numTimes(x, p) =
    if x % p = 0 then numTimes(x / p, p) + 1
    else 0
let maxNumTimes p =
    let num = float([1..20] |> Seq.map(fun x -> numTimes(x, p)) |> Seq.fold(max) 0)
    int(Math.Pow(float(p), num))
[1..20]
|> Seq.filter(isprime)
|> Seq.map(maxNumTimes)
|> Seq.fold1( * )

For all primes in the range, figure out the maximum number of times that prime appears in the prime factorization of any of the numbers, and find the product of those primes raised to the maximum number of powers. I learned a little F# in this one, but as far as the math, it would have taken me a long while to recall or figure that out, so it was basically just implementing what Wikipedia described. Oh well.

My “fold” got one better again, instead of passing a pointless anon function and a pointless initial value, last time I got rid of the function. This time I found “fold1” and got rid of the non-useful initial value too! :)

Some More Euler Project, Problem 4

2008-05-12T00:00:00-07:00

This one was pretty straightforward.

A palindromic number reads the same both ways. The largest palindrome made from the product of two 2-digit numbers is 9009 = 91 × 99.

Find the largest palindrome made from the product of two 3-digit numbers.

The hardest part was just reversing the number to be able to tell if it was a palindrome. Unfortunately, when I searched for “reverse string f#” I got http://basildoncoder.com/blog/2008/04/21/project-euler-problem-4/ which while certainly a good search result, could have ruined the fun. Anyhow: the solution to reverse (why the heck isn’t there a .Reverse on the .net string anyway?) is the obvious thing. The only thing notable thing here is the “:string” on the argument to the function which is the first time I’ve had to write a type other than the type coercions from float <-> int. It reminds me of the irritating problem with C# generics where it’s often not possible to write the generic function because it has to be completely generic, or has to implement an interface so the type can be where-constrained (or in other words, generics are generics, not templates, I guess. The C++ approach definitely has benefits, sometimes, though.)

let rev (s:string) = new string(Array.rev(s.ToCharArray()))

{ for i in [100..999] do
    for j in [100..999] do
        let x = i * j
        if x.ToString() = rev(x.ToString()) then yield x }
|> Seq.fold(max) 0

After that, it’s pretty simple. I just make a list of all the palindromes created from 3 digit multiplicands, and then fold out the maximum of those. I noticed something very dumb in the last solution, I was folding an anonymous function that just directly passed its arguments to max, which is of course silly. I’m finding F#’s syntax a little weird, but getting better.

It occurs to me after reading Mr. Basildon’s solution that I don’t know the difference between Seq, List, etc. and when it matters. Maybe it’s just that seq can be infinite, vs. List is strictly an actual memory block?

More FSharp thoughts

2008-05-11T00:00:00-07:00

It’s pretty irritating so far, but I can’t tell if that’s because I don’t know what I’m doing or because I don’t have some key insight into the language structure yet, or what. I find I’m poking around a lot trying to figure out if I need a terminator or a keyword in random places, and because of the structure of the language there’s not as many opportunities as in C# for the IDE to tell you what to type next.

There’s also the #light thing which is silly. The option should clearly have been #dark (or whatever the opposite of #light is in this case) or a command line option to make it compatible with whatever Caml dialect is non-#light.

I feel like there’s things missing from Seq. and so on, but that’s more likely that I just don’t know what they’re called yet, or I’m expressing things incorrectly I’d guess.

At the same time, it’s very pretty in a lot of ways. The interop with .net (though I haven’t used it yet), immumtability by default, FSI-in-VS, and multicore-niceness are obvious wins. I don’t know if it’s applicable to games yet: there’d be an inordinate amount of panic if extra computation was done unnecessarily of course. Anyway, I like it enough apparently, I just got Amazon to fire me a copy of “Expert F#”, which I hope is teh awesome.

Euler Problem 2 Redux

2008-05-11T00:00:00-07:00

This is a similar attempt, not sure if it's better or worse really, but uses comprehensions instead.

{ for i in [1..33] do
      let x = fib(i)
      if x < 4000000 && x % 2 = 0 then yield x }
|> Seq.sumByInt(fun x -> x)

Euler 3

2008-05-11T00:00:00-07:00

The problem statement:

The prime factors of 13195 are 5, 7, 13 and 29. What is the largest prime factor of the number 600851475143?

The first thing we need to do likely is know if a number is prime:

let isprime x =
    x > 1 &&
    not(Seq.exists (fun n -> x % n = 0) {2..int(sqrt (float x))})

This is a little sneaky. First, it generates a list of numbers from 2 to the square root of the number to be tested (square root because the largest possible factor). Then, it uses an existence test to see if any of those numbers can divide into the candidate number. If they can, we know it’s not prime.

With that out of the way, we can attack the actual problem. My first attempt was very C-like, to whit:

let prob3 num = 
    for i = int(sqrt(float num)) to 2 do
        if num % int64 i = 0L && isprime(i) then return i
prob3 600851475143L

But, that doesn’t work. ‘return’ doesn’t exist, and the type of ‘for’ is unit, so there’s no way to get the result out. I was trying to avoid computing too many results, so going backwards was the goal (rather than making a list of answers and filtering them, as in the previous problems).

I didn’t exactly figure out how to do that, but I did come up with a reasonable solution (at least to my non-F# eyes):

let num = 600851475143L
{ for i in [2..int(sqrt(float num))] do
    if num % int64 i = 0L && isprime(i) then yield i }
|> Seq.fold(fun a x -> max a x) 0

First, we make a list of all the factors of the target number that are also prime (the comprehension). Then, fold the maximum value out of those. It’s definitely doing a lot more work than necessary, but it works OK.

There’s also a little bit of mucking around in this problem because the number was bigger than 2^32-1. F# handles it reasonably cleanly though, at least up to int64 range, by just adding some float/int/int64 calls to convert things to the right types explicitly. Not sure what bignum looks like yet though.

FSharp and Project Euler

2008-05-10T00:00:00-07:00

I was vaguely looking for something that might make me smarter, wouldn’t be at all practical, and also wouldn’t be a huge time investment.

I started by working through the exercises of “Introduction to Algorithms” by C/L/R, but I didn’t make it too far. Maybe someday.

Project Euler is some math/algorithmic problems that seem to fit the bill. Since I was thinking I should learn an ML family language, I thought I’d try to solve them in F# and see how far I got.

Without further ado, here’s problem 1:

If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23.

Find the sum of all the multiples of 3 or 5 below 1000.

and my probably not so amazing solution:

[1 .. 999]
|> Seq.sumByInt(fun x ->
    if x % 3 = 0 || x % 5 = 0 then x
    else 0)

Not too hairy, but some neat bits:

> is a pipelining operator that just changes the order of the arguments. Rather than nesting the [1..999] at the end, it goes at the beginning which makes more sense to read.

And, of course, first class functions.

I really hate “then” for whatever repressed-loathing-of-some-crappy-language-reason, but I’ll survive. I guess.

Euler Problem 2

2008-05-10T00:00:00-07:00

(only 191 problems to go, currently)

Problem #2 was stated as:

Each new term in the Fibonacci sequence is generated by adding the previous two terms. By starting with 1 and 2, the first 10 terms will be:

1, 2, 3, 5, 8, 13, 21, 34, 55, 89, ...

Find the sum of all the even-valued terms in the sequence which do not exceed four million.

Pretty straightforward I guess.

let rec fib n =

if n     else fib(n-1) + fib(n-2)

[1..40]
|> Seq.map(fib)
|> Seq.filter(fun x -> x < 4000000 && x % 2 = 0)
|> Seq.sumByInt(fun x -> x)

I was wondering if the 4,000,000 was supposed to try to make me do something “smart” because it was intended to take a long time. But, I didn’t and it took about a second on my computer, even with the very dumb fib.

The weakest part is the “40”, but I feel like F# earned me that because after I typed in the definition of fib, I used the awesome FSI (interactive F# window in Visual Studio) to quickly test what numbers would make sure I got up to at least 4M (it was 33, but 40 seemed less sneaky).

There’s probably a better way to do the last line too, I guess fold over + would work. I was sort of hoping for a default Seq.sum without arguments, but I guess not. Maybe there’s a shorthand way of writing the identity function?

Where's My Monitoring?

2008-03-06T00:00:00-08:00

Jenn and I were recently arguing about taking vitamin and other dietary supplements. Historically, I’d been rather dismissive towards them because they always felt very pseudo-scientific, but I’d never really thought about why it was that I felt that way.

Obviously it’s possible that they’re filling a very real need, particularly for people who have unusual diets (we’re vegetarian for example, which isn’t particularly unusual these days, but might be considered unusual on an evolved-from-monkeys-timescale). But (and it’s a big but), there’s really no practical way to know what effect of taking these things is. You could go for a weekly battery of blood/skin/whatever tests, but that’s both costly and time-consuming. Other than that, it falls to either correlation with secondary effects, or “how I feel” both of which are way too heavily influenced by other things going on in life and placebo effects.

Visiting any supplement or “natural”-something-or-other store reveals thousands of random options. The thing that really bothers me, is that if it they turn out not to be helping, at best you’re wasting a lot of money (if they’re doing nothing), but at worst, you could be creating health problems by ingesting mixes of things that really aren’t good for you.

So, getting to the point of all this, why can’t I monitor myself yet? I think this would pretty clearly be a game-changing startup: I stick my finger in a little machine before I get in the shower in the morning and it grabs some blood and skin (and maybe occasionally I give it some hair as a little treat). Then, it chugs away extracting as much useful information as is possible from the samples (lots and lots and lots of info in there!). Add a WiFi connection to it, and it doesn’t need any interface at all, just the ability to upload data to either a local machine, or preferably to a suitably encrypted and secured web site. The web site can give me a history of changes, monitor for flucuations, highlight defiencies (and maybe suggest supplements or diet modifications), or tell me that now would be a damn good idea to go see an MD or nutritionist.

So, where the heck is my simple desktop monitoring system?

Foo

2008-03-02T00:00:00-08:00

Heard the new Foo Fighters on the radio (I assume the latest single), and all I could think was that if Kurt had heard it, he'd have shot himself in the face.

...

But hey, they won a Grammy or whatever. Maybe it'll be for best Adult Contemporary next time.

bash .inputrc tab completion incantations

2008-03-01T00:00:00-08:00

I can’t stand hitting TAB and having it do nothing because there’s two matches. Cycling seems vastly more useful to me. And, history-search (F8 in Windows), but both directions using PgUp/PgDown. I can never remember this magic so here it is for posterity (goes in ~/.inputrc):

TAB: menu-complete
"\e[Z": "\e-1\C-i"
"\e[5~": history-search-backward
"\e[6~": history-search-forward

Then, Tab and Shift-Tab cycle, and PgUp/PgDown cycle through history that matches. Much better!

Emacs For Vim

2008-02-26T00:00:00-08:00

I’m having a whack at trying to learn a reasonable amount of Emacs so I can attempt a project more reasonably in sbcl (and Weblocks). I’ve been programming in Vim for about 12 years and my brain/hands are very, very angry at this editor transgression.

Mostly because I know I’ll give up, and then try again much later, here’s some stuff that is permanently ingrained in my Vim brain that is the closest I could find in Emacs (so far at least). To be edited as I find more things. Vim on the left, Emacs on the right:

Movement

C-f, C-b = C-v, M-v
hjkl = C-b, C-n, C-p, C-f
HL (or ^$ for most people) = C-a, C-e
wb = M-f, M-b
O, o = ? (C-o does some stupid thing)
u, C-r = ?

Change/Deleting

das = ?
cab = ?
gqap = ?
dd = ?

Clipboard

"+y = ?
"+P = ?

The page movement commands are probably my biggest problem right now: for one they’re the dumb ones like Vim’s C-u/C-d that only move half a page (haven’t figured out how to make them only leave one line instead yet), and for two, it’s just kind of painful because you have to move your pinky and then repress the v, rather than just hold down control and toggle between f/b. Anyway, whatever. I’ll suck and use PgUp/PgDn for now.

No hjkl of course kills me, but everyone else too.

I keep hitting Esc when I’m done entering text so then the next Ctrl/Meta command doesn’t work because it’s been prefixed by an Escape. Ack. Pffft.

The most amusing thing so far is that I couldn’t figure out enough to edit my .emacs in Emacs so the first few edits were:

$ vim ~/.emacs

Hrm.

Javascript Framework

2008-02-22T00:00:00-08:00

I thought I liked jQuery, but it’s just a bit too weird after using it for a bit. I kept having the feeling that it was too much magic and not enough JavaScript. So, I’m using mootools now. It’s very similar, but one or both of the design and documentation is better, and as a result I’m much happier with it.

N Plus

2008-02-21T00:00:00-08:00

N+

Nick and Kees over at Slick Entertainment just had their 360 XBLA version of “N” (a cool platformer) released today. Just grabbed the full version and it’s a heck of a lot of fun.

Arcish Web Programming In Python

2008-02-02T00:00:00-08:00

There seems to be little more sanity coming to the Arc discussion. Some people are still paniced about HTML generation, but I’m not sure that you really have to use that if you don’t like it. Others are just disappointed because they were hoping for … I don’t know what, some sort of Holy Grail, but it’s, y'know, just a programming language.

The terseness it brings to common web expressions is quite pleasant though, and I wanted to be able to have a similar level of power and terseness in Python if possible. I spent a little time putting together “pyflow” (named for both the nicer control-flow primitives and, hopefully, a state of mind for its users). The “Arc Challenge” code looks like:

@opapp
def said(req):
    def result(foo):
        cede(link(text("you said: " + foo),
                  "click here"))
    cede(aform(result, input("foo")))

It’s 19 tokens (which is 4 more than the Arc version), but I still find it nice to read. I prefer “foo” being passed as a real parameter (vs. (arg _ “foo”) in Arc), but the cede()’s are somewhat less pleasant on the Python side.

Moving to a reimplementation of blog.arc, pyflow fares better.

Posts = getstorelist("posts")
BlogTitle = "A Blog"

def blogPage(body):
    return defpage(fixed600(
                    tag("h1", link("blog", BlogTitle)),
                    body,
                    withsepbull(link("archive"), link("newpost", "new post"))),
                   title=BlogTitle)

def permalink(p): return "viewpost?id=%d" % p["id"]
def notfound(): return blogPage(div("spacedp", "No such post."))

def displayPost(p, user):
    head = tag("h2", link(permalink(p), p["title"]))
    user = " " + link("editpost?id=" + str(p["id"]), "[edit]") if user else ""
    return head + user + div("spacedp", markdown(p["body"]))
    
@opapp
def blog(req):
    user = getUser(req)
    return blogPage(flat([displayPost(p, user) for p in reversed(Posts[-5:])]))
    
@op
def archive(req):
    links = []
    for p in reversed(Posts):
        links.append(tag("li", link(permalink(p), p["title"])))
    return blogPage(ul(*links))

@op
def viewpost(req):
    id = req.getint('id')
    user = getUser(req)
    if id and id     else: return notfound()

@oplogin
def newpost(req):
    def res(t, b):
        p = {'id': len(Posts), 'title': t, 'body': b}
        Posts.append(p)
        commit()
        cede(redir(permalink(p)))
    cede(defpage(
            aform(res, withsepbr(
                input("t", "", 60),
                textarea("b", "", 10, 80) + br())),
            title="new post"))
        
@oplogin
def editpost(req):
    id = req.getint('id')
    def res(t, b):
        Posts[id]['title'] = t
        Posts[id]['body'] = b
        commit()
        cede(redir(permalink(Posts[id])))
    cede(defpage(
        aform(res, withsepbr(
            input("t", Posts[id]["title"], 60),
            textarea("b", Posts[id]["body"], 10, 80) + br())),
        title="edit post"))

Kind of long for Wordpress’s asinine (non-)code handling, but quite short given the functionality that’s in there. (I got lazy on editpost, it should really share code with newpost). Wins over the Arc code include:

Simple persistent list/dict built in. A lot of the fiddling in the Arc code is to load/store/iterate over numbered files in the ‘posts’ directory. pyflow includes a simple transactional dict/list store for simple storage (that also happens to work when there’s concurrent users, which I don’t think blog.arc would do right now).

Python functionality that’s somewhat terser than Arc. For example, the main entry point “/blog” looks like this in blog.arc:

(defop blog req
  (let user (get-user req)
    (blogpage
      (for i 0 4
        (awhen (posts* (- maxid* i))
          (display-post user it)
          (br 3))))))

vs. in pyflow, it looks like:

@opapp
def blog(req):
    user = getUser(req)
    return blogPage(flat([displayPost(p, user) for p in reversed(Posts[-5:])]))

List comprehension work nicely here to keep the number of tokens down.

CSS: Some of the layout-y (e.g. (br 3)) code that ends up being code in Arc is in a default.css which also allows for snazzing things up pretty easily. Although, I got lazy so there’s some formatting in the code in pyflow that could be tidied up and shortened.

I don’t mean to imply the blog code couldn’t be squished down more in Arc (I’m sure it could), or that I don’t like Arc’s approach, but I think it’s worth exploring how similar idioms can be expressed in various languages to try to improve everyone’s end solution. Perhaps when Paul releases “news”, I’ll have a chance to see if Arc is more of a win on larger and more production-quality code bases.

Anyhow, I think the discussion surrounding Arc is the best thing that’s happening right now. So, if you’re contributing, keep it up. Lots of interesting stuff coming out. If anyone has any suggestions on how the pyflow example code could be simplified with new idioms, please fire away.

Better Flow Control

2008-02-01T00:00:00-08:00

Been poking around the Arc site; I did a quick Vim integration (to allow sending files/forms to the running Arc process), and I’ve been reading its code a little, mostly to get a better understanding of the approach it’s taking for html generation.

There’s bad and good, but it’s certainly very terse. Here’s the Arc version of prompt, interstitial, and results:

(defop said req
    (aform [w/link (pr "you said: " (arg _ "foo"))
            (pr "click here")]
            (input "foo")
                  (submit)))

I’d written a Python library that makes things pretty terse (there’s an example on that page), but after seeing the Arc version, I wanted to trim it down some more. It now looks like:

@op
def said(req):
    def result(foo):
        cede(link("you said: " + foo,
                  "click here"))
    cede(aform(result, input("foo")))

There’s a bit of magic going on, but it’s quite nice and tidy I think. Compared to the Arc version, I prefer having foo as a regular argument to the handler function vs (arg _ foo) in Arc, but the simpler closures in Arc allow you to avoid cede and the name result which is a little nicer. It's pretty close to a direct translation which I find interesting. I'll definitely be keeping an eye on Arc as it develops, I found this an expanding experience.