Skilldrick

Well, I’m 27 now. I thought I’d write a little update to my New Year’s 2011 post, to assess where I am and where I’m going. As before, if you’re interested then read on, but don’t expect any life-changing revelations or anything :)

Resolutions or whatever

The following are the headings from my 2011 post, with my thoughts/progress on each point.

Finish SICP

On my way with SICP. It’s a hard slog but enjoyable too. I really hope I can finish it this year, so I can start concentrating on other things. I’ve decided that it’s OK to skip the odd chunk of exercises.

Get a decent online portfolio

Haven’t really got anywhere with this yet, as I’ve been busy working and learning.

Keep blogging

I’ve been fairly successful with the regular blogging schedule – trying to keep to weekly posting. I’ve had some brilliant successes recently, getting onto the front page of Hacker News yesterday (a nice birthday present) AND appearing in JavaScript Weekly today.

Get some more freelance work

This is going well – given the nature of this kind of work I can’t go into great detail, but I’ve got more than enough work to be getting on with right now, so no complaints here.

Do something good in node.js/Learn a functional language

Not yet, but still hoping to find the time.

Stack Overflow

Well, I talked about the page numbers of the list of Stack Overflow numbers, but they’ve changed that now so any target based on that is a bit irrelevant now. The user pages now show a “top x% overall” reputation figure – it would be nice to get that to 1% (I’m on 2% at the moment), but I’m easy.

Nick Morgan, not just Skilldrick

This is going well. This blog comes up on page 1 of google.co.uk when searching for Nick Morgan and page 2 of google.com. Seems that putting my name in the header and sidebar worked! So a solid win there.

Etc.

I’m thinking more about the social side of things now too. I went to an excellent Code Retreat in Winchester (here’s a nice review), have been attending (and beginning to help organise) my local Ruby user group, and went along to the Bristol version of the Stack Overflow Meetup. All of these have been great ways to meet my peers, get to know some really nice people, and find out what other people are up to. The Code Retreat in particular was awesome and I’d thoroughly recommend anyone seek one out and do it. I learnt a huge amount and it’s affecting my coding constantly.

I feel a bit like I’m playing catch-up – most people my age have been doing this for way longer – but I also feel like I’m actually catching up. And I’m finally doing something that I really want to be doing. These four months have passed in no time at all, so I know it won’t be long until the end of 2011 and the start of my full-time freelance career. W00t!

First off, so we’re on the same page, we need to be talking in the same terms. So, what is magic? I’d define it as anything below your current level of abstraction that you don’t understand.

I first started thinking about this topic because of this angst I felt about all this magic I was relying on. There’s so many things that I rely on every day and I have no idea how they work. As far as I’m concerned, they’re magic. As long as I think about these things as magic, they seem unknowable. I think that’s where the angst comes from. What I need to remind myself is that everything I’ve learned about software seemed like magic before I understood it, and there’s nothing that’s unknowable, just things that aren’t known yet.

This post is my attempt to come to an understanding of what this ‘magic’ is, whether it’s a problem, and how best to deal with it.

The ‘Dave’ technique

At University, I learned how to record music. We recorded onto tape, so we had to learn how a tape machine worked. To learn how a tape machine worked, we had understand hysteresis and remanence and modulation and various other things that I’ve mostly forgotten now. We also had to understand how radio worked, so we learned about amplitude and frequency modulation, and how stereo FM was built on top of mono FM with an AM difference signal on top so as to be backwards compatible, and various other technical issues. Then came digital audio with ADCs, DACs, sampling, quantisation, dithering, etc. Each of these three topics was one or two modules, i.e. a semester’s-worth of lectures, so we went pretty deep.

Did that knowledge make me a better recording engineer? Probably not. But it was a helpful basis for the latter things I learned, which did help make me a better recording engineer.

The ‘Tim’ technique

Another lecturer taught us computer audio. This started off with an introduction to computers, starting at the level of AND, OR and NOT gates, then on to NAND and NOR (and how the simpler to understand AND, OR and NOT gates can be implemented in terms of NAND and NOR), then on to the ALU, then the CPU, then adding memory and disk storage and network and MIDI connections, creating a full computer. This was all presented in two or three lectures, so inevitably certain steps had to be left out (e.g. “we put together a load of these gates and we get the ALU”).

The benefit of this approach was that we were given a vague conceptual understanding of the basis on which computer audio is built, without having to spend hours and hours learning the full intricacies.

Bottom-up

If we’re intent on uncovering all the magic, how should we learn – bottom-up or top-down? Bottom-up, right from the bottom, implies first learning enough physics to understand electronics, which you’d need to understand digital circuit design, which you’d need to understand CPU design, which you need to understand assembly language, which you need to understand C, then enough C to understand Ruby (say). You’ll also need to understand how networks work right from the bottom of the TCP/IP stack and beyond; how storage works right from the level of hysteresis in magnetic particles on a hard disk; how RAM works from the level of individual capacitors; etc.

This reminds me of the famous Carl Sagan quote:

If you wish to make an apple pie from scratch, you must first invent the universe.

Top-down

Top-down is essentially the process followed by Structure and Interpretation of Computer Programs, at a micro and macro level. At the micro level, we are taught to solve problems using “wishful thinking”: first represent the problem using large custom building blocks, then build the pieces those blocks are made out of using language primitives. And at the macro level, the book teaches the workings of Scheme through a series of increasingly complex models.

Bottom-up vs top-down

The problem with the bottom-up approach is it would take decades of studying before you were let near an interpreted language. This approach may have worked in the fifties when there wasn’t as much to learn. But these days you won’t get anything useful done.

The problem with the top-down approach is that the levels we work at these days are so large, you can’t just learn all of your current level and then start to look at the lower levels, because you’d never finish learning all the details of the current level.

Leaky abstractions

Joel Spolsky has something to say on this matter in his piece The Law of Leaky Abstractions. This basically says that all abstractions are leaky, meaning that we cannot rely on them to shield us from the levels below our current level. So, to work at the current level we must learn the lower levels. It’s OK to work at the level of the abstraction, and it will save us time working at that level, but when it breaks down we still need to understand why. As Joel says:

the abstractions save us time working, but they don’t save us time learning.

Magic’s alright, yeah

Some of us don’t like the idea of magic. It feels like cheating. But it’s useful as a placeholder, something to learn about later; not now. It’s useful to delve down to lower levels, understanding how the abstractions we work with every day are implemented, but at some point the law of diminishing returns kicks in. (In some fields, we can’t even assume we’ll ever learn how the magic works. As @sermoa said, “we don’t understand exactly how quantum physics works but we accept that it does and make good use of it”.)

What we need to get to, eventually, is an understanding of the whole stack, but at greater and greater granularities as we get further away from our main level. So, if you’re a Ruby on Rails programmer, you need to know most of Ruby (but not all of it). It’s helpful to know a bit of C, and a bit of how the operating system responds to certain commands. Below that level, it’s good to have a vague understanding of how a computer works. And it’s useful to know that electrical signals can only transmit at a finite speed, and that there’s a physical limit to the number of transistors in a processor. As time goes on you can begin to flesh out these lower levels, remembring that the higher levels are generally going to be more beneficial to your day-to-day work.

Before you’ve delved deeper, those lower levels are magic. And that’s fine.

Thanks to @marcgravell, @ataraxia_status, @sermoa, @bytespider, @ghickman, @SamirTalwar and @josmasflores for their input on this post.

Also, thanks to Dave Fisher and Tim Brookes for being the sources of the ‘Dave’ and ‘Tim’ techniques.

Last year I wrote A brief introduction to closures which was meant to help people understand exactly what a closure is and how it works. I’m going to attempt that again, but from a different angle. I think with these kinds of concept, you just need to read as many alternative explanations as you can in order to get a well rounded view.

First-class functions

As I explained in Why JavaScript is AWESOME, one of the most powerful parts of JavaScript is its first-class functions. So, what does “first-class” mean in a programming language? To quote wikipedia, something is first-class if it:

• can be stored in variables and data structures
• can be passed as a parameter to a subroutine
• can be returned as the result of a subroutine
• can be constructed at runtime
• has intrinsic identity (independent of any given name)

So, functions in JavaScript are just like objects. In fact, in JavaScript functions are objects.

The ability to nest functions gives us closures. Which is what I’m going to talk about next…

Nested functions

Here’s a little toy example of nested functions:

The important thing to note here is that there is only one f defined. Each time f is called, a new function g is created, local to that execution of f. When that function g is returned, we can assign its value to a globally defined variable. So, we call f and assign the result to g5, then we call f again and assign the result to g1. g1 and g5 are two different functions. They happen to share the same code, but they were executed in different environments, with different free variables. (As an aside, we don’t need to use a function definition to define g and then return it. Instead, we can use a function expression which allows us to create a function without naming it. These are called ‘anonymous functions’ or lambdas. Here’s a version of the above with g converted to an anonymous function.)

Free variables and scope

A variable is free in any particular scope if it is defined within an enclosing scope. To make that more concrete, in the scope of g, the variable x is free, because it is defined within the scope of f. Any global variables are free within the scopes of f and g.

What do I mean by scope? A scope is an area of code where a variable may be defined, without the enclosing scope knowing about it. JavaScript has ‘function scope’, so each function has its own scope. So, any variable defined within f is invisible outside of f. Scopes can be nested, so in the above example, g has its own scope which is contained within the scope of f, which is contained within the global scope. Whenever a variable is accessed, the JavaScript interpreter first looks within the current scope. If the variable is not found to be defined within the current scope, the interpreter checks the enclosing scope, then the enclosing scope of the enclosing scope, all the way up to the global scope. If the variable is not found, a ReferenceError is thrown.

Closures are functions that retain a reference to their free variables

And this is the meat of the matter. Let’s look at a simplified version of the above example first:

It’s no surprise that when you call f with the argument 5, when g is called it has access to that argument. What’s a bit more surprising is that if you return g from the argument, the returned function still has access to the argument 5 (as shown in the original example). The bit that can be a bit mind-blowing (and I think generally the reason that people have such trouble understanding closures) is that the returned g actually is remembering the variable x that was defined when f was called. That might not make much sense, so here’s another example:

When person is called, the argument name is bound to the value passed. So, the first time person is called, name is bound to ‘Dave’, and the second time, it’s bound to ‘Mary’. person defines two internal functions, get and set. The first time these functions are defined, they have a free variable name which is bound to ‘Dave’. These two functions are returned in an array, which is unpacked on the outside to get two functions getDave and setDave. (If you want to return more than one value from a function, you can either return an object or an array. Using an array here is more verbose, but I didn’t want to confuse the issue by including objects as well. Here’s a version of the above using an object instead, if that makes more sense to you.)

And this is the magic bit. getDave and setDave both remember the same variable name, which was bound to ‘Dave’ originally. When setDave is called, that variable is set to ‘Bob’. Now when getDave is called, it returns ‘Bob’ (Dave never liked the name ‘Dave’ anyway). So getDave and setDave are two functions that remember the same variable. This is what I mean when I say “Closures are functions that retain a reference to their free variables”. getDave and setDave both remember the free variable name. Even though person has now returned, the variable name lives on, because it is referenced by getDave and setDave.

The variable name was bound to ‘Dave’ when person was called the first time. When person is called a second time, a new version of name comes into existence, as well as new versions of get and set. So the functions getMary and setMary are completely different to the functions getDave and setDave. They execute identical code, but in two different environments, with different free variables.

Summary

In summary, a closure is a function called in one context that remembers variables defined in another context – the context in which it was defined. Multiple closures defined within the same context remember that same context, so changes they make are shared between them. Every time a function is called that creates closures, a new, shared context is created for the new closures.

The best way to learn is to play, so I’d recommend editing the fiddles above (by clicking on the + at the top right) and having a fiddle. Enjoy!

Addendum: Lasse Reichstein had some useful comments on this post on the JSMentors mailing list – read them here. The important thing to realise is that a closure actually remembers its environment rather than its free variables, so if you define a new variable in the environment of the closure after the closure’s definition, it will be accessible inside the closure.

When you first start learning to program, the best thing you can do is just write code, and lots of it. Coding is a muscle, and it needs to be exercised.

That’s just a start though. If you enjoy it, and it’s something you want to do professionally, then there are some essential practices that you need to learn, and make part of your routine.

This list is split into two parts. The first items are directly related to coding. They all take some time to setup and master, but once you’ve got to that stage you’ll get nothing but good out of them. Think of them as an investment in your future.

The second bit is about the world beyond coding, and they mostly involve interacting with real people. If that’s not a problem, then great, but if it is, you’re gonna have to get over it, because dealing with people is a surprisingly big part of this job.

Coding

1. You need to be testing your code. You won’t get everything perfect the first time, so at some point you’ll need to refactor. And if you refactor without tests, “you’re not refactoring; you’re just changing shit”.

2. Your code needs to be under version control. I use git, but there are plenty of other options. Code that’s not under some form of version control may as well not exist. And using something with easy branching like git makes it much easier to experiment with big changes to your code that might not work out.

3. Learn to touch-type. It’ll take a bit of effort, but it’s worth it. If you’re serious about being a professional, then you’re going to be spending a lot of your time on a keyboard. Learn to use it properly. Steve Yegge wrote a great blog post on touch-typing. If you want to learn, I don’t know of anything better than GNU Typist.

Everything else

4. Start a blog. Write about code you’ve written, problems you’ve solved, things you’re learning about. It’s not for anyone else, just you. The act of putting your thoughts into words forces your brain to look at issues from a different angle. There may come a point that people start finding your content and making use of it, so it’s worth making it public.

5. Join Twitter. Find local developers and the big names in whatever languages you’re currently learning. Twitter’s like a global virtual watercooler. People talk a lot of shit, but some of the most important conversations happen there as well, as well as job offers, so if you’re not involved you could miss out.

6. Use and participate in Stack Overflow. You can get a lot of useful guidance if you learn how to ask questions sensibly, and you can learn loads by answering questions as well. Also, once you’ve got a good amount of answered questions, Stack Overflow becomes a great showcase of your knowledge and writing.

7. Find a local user group. Google for the name of your town or city, the name of a language you’re interested in, and ‘user group’ (in Ruby that would be "#{city_name} #{language} user group"). There’ll be something out there.

That all looks too much like hard work to me

If you’re just starting out, these lists may seem a bit daunting. You don’t need to do everything at once, but if you’re taking this seriously then at some point you should be doing them all.

Edit: New readers are advised to check the date of posting before taking anything too seriously…

In recent months I’ve grown bored of this whole programming thing. It’s time for me to start focussing on my true passion – linguistic analysis of popular music.

I was first introduced to this fascinating subject by my university lecturer Allan Moore, whose insights into the hidden depths of popular songs were quite enlightening.

My first post on this new subject will be an analysis of Rebecca Black’s latest hit, ‘Friday’.

‘Friday’, or, Roads to Freedom

Rebecca Black, or, the protagonist

‘Friday’ starts with what is ostensibly a description of Black’s morning routine. Obviously, there is much more to the lyrics than what is visible on the surface. The first question we must ask ourselves is whether the protagonist of the song is Black herself, or a persona created purely for the purposes of the song. The ‘truth’ is not important – only our perception counts, so I won’t attempt to answer this question.

The protagonist wakes up at 7 a.m. Is that normal for the youth of America? It’s hard to say, but it is obscenely early, and it looks like Black is highlighting the plight of school children across the United States, under this oppressive regime.

‘Gotta be fresh’: what does this mean? ‘Fresh’ in the sense of clean or awake, or ‘fresh’ in the more hip-hop sense of the word? The use of such an ambiguous word at the start of this song sets the tone for the rest of it.

The protagonist goes to the bus stop, but then sees her friends. We assume that she was planning to take the bus, but was this just a ruse? Is the ‘bus stop’ a metaphor for meeting friends? Here comes the first dilemma of the day – should she sit in the front or the back? Clearly this is a metaphor for the classroom – should she sit at the front of the class or the back? We all know the ramifications of these alternative positions. In a normal classroom there is always the option of the middle-ground, but in a car this is reduced to the stark choice of front or back. Look a bit further, and you’ll find an oddity. Black doesn’t ask ‘which seat should I take’, but rather ‘which seat can I take’. It turns out there is no choice involved here – the protagonist is at the will of the fates (much like her school life).

We then get to the crux of the song – the evocation of ‘Fridayness’. Just like the Crunchie bar is designed to evoke ‘that Friday feeling’, ‘Friday’ is all about the truth and beauty of Friday. To really get the point across, Black repeats the word ‘Friday’ multiple times in the chorus. She also highlights the ambiguity of this strange day of the week. Is it the weekend? Or the precursor to the weekend? With the line ‘Everybody’s lookin’ forward to the weekend’ it’s hard to tell.

Now it’s 7:45. A.m or p.m? Who can tell? Have 12 hours passed already, or is this still the morning? I think what is happening here is some kind of quantum superposition of the two states – the actual time doesn’t matter as much as the day. We are reminded slightly of the post-modernist masterpiece ‘Ferris Bueller’s Day Off’, where school time and play time merge into a state of general ‘beingness’.

‘Cruisin’ so fast, I want time to fly’: we have now moved from quantum physics to general relativity. By including these references so close to each other, Black is almost mocking the current lack of a unified theory of Physics. This is followed by more Physics/Buddhist thought: ‘I got this, you got this’. We don’t know what ‘this’ is, but the fact that Black and her friend both have it at the same time tells us a lot about her holistic view of the world.

Skip forward a bit, and we get to the most genius part of the song, repeated here:

Yesterday was Thursday, Thursday
Today i-is Friday, Friday
We-we-we so excited
We so excited
We gonna have a ball today
Tomorrow is Saturday
And Sunday comes afterwards
I don’t want this weekend to end

Black is truly putting the day of Friday in its context – after Thursday, before Saturday, and two days before Sunday. The lyrics here are split – we hear about Thursday and Friday, then a break, and then Saturday and Sunday. So Friday belongs with Thursday, and Saturday with Sunday. The past is the present, the future the future. This is the kind of depth that is all-too lacking in most modern pop music.

We now move to the secondary protagonist of the song, who I believe to be Black’s alter ego. This character is driving, thus removing the dilemma of front seat/back seat (there can be only one driver’s seat). Then on to ‘It’s Friday, it’s a weekend’, which contrasts with the other lyric ‘Lookin’ forward to the weekend’ – setting up a conceptual dissonance that is never quite resolved.

Conclusion, or, why does my heart feel so bad?

We have only scratched the surface of ‘Friday’ in this analysis, but I hope I’ve given you an idea of the depth of Black’s music. It’s all too easy to see this piece as just another Bieberfied tween pop song with no artistic merit, but as you will now see, this could not be further from the truth. So, go forth and enjoy this beautiful Friday!