Easy functional programming in JavaScript with Underscore.js — part 1

July 27th, 2011 6 comments

So, you’ve been developing in JavaScript for a while, you’re getting on quite well with your for and while loops, when somebody comes along and tells you you shouldn’t be doing that, that JavaScript is a functional programming language and there’s much better ways to solve most problems that with loops. Whaaa…? Well, that’s what I’m here to tell you anyway.

What is functional programming?

Functional programming (FP) is all about the idea of functions as a mapping from one value to another. An FP function just turns one value into another – it doesn’t have side effects. Side effects are anything that happens within the function that changes something outside of the function. For example, changing the value of a variable not defined in the function, or logging output, or showing an alert message … basically anything that happens in the function. The only thing an FP function does is takes in a value, and returns a new value.

Another interesting thing about FP is the use of higher-order functions. These are functions that either take or return functions as values (or both). I’m mainly going to be looking at functions that take other functions as parameters, because there are some interesting things you can do with that.

An underscore, courtesy of Wikipedia

Functional programming in JavaScript is made a lot easier with a suite of functions called Underscore.js all packed into a minified script of just 3kb. Underscore provides a selection of common FP functions for working on collections, like map, each, reduce, select etc. I’ll walk through some of these to demonstrate the power of FP. Note that functional programming isn’t just about working with collections – there are lots of other exciting things you can do – but for now I’ll concentrate on the collections.

Mapping over an array

One of the most widely used Underscore functions is _.map. This takes an array and a transforming function, and creates a new array based on the return value of the transforming function. That’s a bit abstract so I’ll give you an example:

So, _.map is passed an array, and a function. That function is called on each element of the array in turn, and the return value of the function gives the corresponding element in the new array. Note that the original array is left intact. The _.map function takes an array, and returns another array – it’s a mapping from one value to another.

You can do much more interesting things with _.map than just multiplying numbers though. You could transform an array of strings into an array of cats!

I hope you’ll forgive me for using two Underscore functions there. First off, we’ve got _.map. In this case, we’re going through the array of cat names, and creating an array of cats. The second is _.each. This, like _.map, goes through each element of the array, passing them into the function, but it doesn’t care about the return value. It just executes the function and carries on with its life. Now, if you’re concentrating, you may have noticed that each isn’t actually a mapping from one value to another, and you’d be right. _.each is only useful if you allow side effects, for example console.loging. A JavaScript program isn’t much fun without any side effects, because you’d have no idea if anything happened otherwise. The key to functional programming is minimising the side effects.

What’s great about _.map is that once you’ve learned to read it, and understand what it does, it really succinctly explains the meaning of your code. In that cat example, you can see that all the _.map function is doing is turning strings into cats. That’s all you need to know. The old-school JavaScript technique would look like this:

In comparison, isn’t that just damn ugly? I mean, to read that you need to keep track of the array, and the array index, and the length of the array, and you have to make a new blank array to contain the new cats, and … you get the point. It’s mixing up the how with the what. All I care about is that I’m getting an array of cats, from an array of cat names. Don’t bother me with insignificant details like array indices.

Here’s another example: constructing a query string:

This one uses an object, but the same principles apply. From a list of key/value pairs, we’re constructing an array of ‘key=value’ strings. Then all you need to do is join the array with &s. This describes how a query string is formed much more declaratively than the standard

append the key, then an equals, then the value, then an ampersand, then the next key, then the next value, then another ampersand, then – oooh we’re at the end – better get rid of that last ampersand, whoops!

Go forth and map-ify!

I hope that’s given you some ideas as to how functional programming can be used in JavaScript. And I really hope that you can begin to see how much cleaner your code can get when you cast out the for loops and embrace the _.map. If all you’re after is map and each, and you’re using jQuery, you’ll be glad to know both functions have jQuery equivalents: jQuery.map and jQuery.each, albeit with a slightly different API. Next time I’ll be looking at some of Underscore’s other functions, for the times when _.map just won’t cut it.


Part 2 is now up!

Categories: Programming Tags: ,

Test-drive your JavaScript!

July 23rd, 2011 5 comments

Testcard

JavaScript is such an important language today. It’s stopped being a toy scripting language, and become a serious programming language. Unfortunately, the vast majority of JavaScript developers don’t unit test their code.

Testing is a vital part of modern development. Code without tests isn’t code – it’s random scribblings that may or may not be executable. Even if they are executable, the only way you can tell if they all work is by loading your application or website, and trying out every single thing that a user could possibly do.

Test-driven development (TDD) is a really nice way of developing software by writing tests before you write the code. If you’re not used to this way of working then it sounds like a weird way to do things, but hear me out.

Imagine you’re building a web app. You wouldn’t just write the whole thing, then open up a browser and see if it works. You’d write a small bit, open the browser and see if it does the expected thing. If it didn’t, then you’d work out why, then fix it, and look again. Then you’d do the next bit.

TDD is just like that, but easier. With TDD, instead of going to the browser each time and seeing if what you’ve just written works, you write a bit of test code that checks that your code does the expected thing. Once it does, you write another test that checks the next thing you’d like it to do. The advantage of working this way is that every time you run your tests, you’re running all of the previous tests you’ve written. That way you can see if something you’ve written to get the most recent thing working ended up breaking something you did earlier. If you were going to do this the old way, you’d either have to check everything every time you made a change, or risk breaking other things that you’re not concentrating on right now. A good test suite will run in under a second, so you get instant feedback whenever you make a change.

Using TDD also means that you end up thinking about the application in a much more sensible way. It forces you to limit the interactions between different bits of the code, which makes the code much easier to modify at a later date.

A side-effect of TDD is that you end up with a comprehensive test suite that can be used to verify the behaviour of your app.

Ok – I’m sold!

Now, TDD takes some practice. You need to learn the tools and the techniques so you can get to the point where you actually write code faster when you’re test-driving it. So, how do you get there?

Test-Driven JavaScript Development

The best book I know on the subject is Test-Driven JavaScript Development by Christian Johansen. This book works on the assumption that you’ve never done TDD before, and gives an excellent grounding in both TDD in general, and how to do it in JavaScript. This is done through example code that you can write and test as you read the book. By the time you finish the book you’ll know how to do TDD, you’ll have some awesome tools for testing JavaScript, and you’ll have written a Node.js app!

If you’re not sure whether you want to get the book yet, have a look at the author’s introductory testing article on Script Junkie.

Testing tools

Although the author of Test-Driven JavaScript Development advocates the use of Google’s JSTestDriver, I much prefer to use Jasmine. If you work with Ruby you’ve likely used RSpec – Jasmine is basically RSpec for JavaScript, but with some ideas of its own as well. Jasmine is a BDD framework, not TDD. BDD (behaviour-driven development) is like TDD, but with a different emphasis and slightly different philosophy (and terminology). To find out more about BDD have a look at the Wikipedia page, as it’s outside the scope of this article.

Finally

I hope I’ve convinced you by this point that testing is worth it – even in JavaScript. I also hope that you’ll give TDD a go, as I think it’s (a) a great way to develop software, and (b) a brilliant way to create a test suite for your code. It does take a bit of time to set everything up the first time, and to learn the right techniques, but that time will pay for itself over and over again, when you have applications that actually do what they’re supposed to do, and don’t break when you change them. So please, do give it a go, and let me know how you get on!

Categories: Programming Tags: ,

You should learn to type (properly)

June 18th, 2011 13 comments

The Typing of the Dead screenshot

As a developer, my job is to make software. To start off with, this software lives only in my head. Programming is the act of extracting the software out of my head and into the computer. The keyboard is the interface between my brain and my computer. There are times when I’m primarily thinking, and my speed on the keyboard doesn’t matter. But when a thought comes to me, I want to get it out of my brain and into the computer as fast as possible, or I might lose it. I need a high-bandwidth connection.

If you write code and you’re not touch-typing, I don’t think you’re taking it seriously. Touch-typing is one of the easiest things you can do to level-up as a developer – the return on investment is massive.

The keyboard is your primary tool as a developer, and you should wield it like a sword.

Also, if you ever end up in a situation where the world has become infested with zombies, and the only way to kill them is by typing at them, you wanna make sure you’re in with a chance.

How do I learn?

By now, I’m really hoping I’ve convinced you to give touch-typing a go. Now, here’s the good news: it’s easy! It takes a bit of time to master, but nothing like the effort that goes into learning a new language, framework, or pretty much anything else.

GNU Typist

The first thing you should do is download GNU Typist. Complete all the lessons in the first course (Series Q – Quick QWERTY course). Make sure you use the fingers it tells you to use for each key. And whatever you do, don’t look at the keyboard. I’ve heard that some people find this so difficult they put a towel or a piece of paper over their hands so they can’t see. If you’re looking at the keyboard, you’re doing it wrong. Even if there’s one key that you keep missing, the only way to learn is to keep on going until you get it right. Next time you get to that problem key, pause, breathe, and press it (use the Force). If it’s wrong, pause again, breathe again, and press it again. Eventually you’ll have spent so much time thinking about the position of that key that you’ll never have to think about it again.

Now, this is the hardest bit. Once you’ve finished those lessons, touch-type everything. Whether you’re writing emails, programming, working at the command line, googling, chatting, or anything, make sure you’re doing it like GNU Typist taught you. This is hard, because at this stage you’ll be slower than you were before. It’ll be so easy to lapse back into your old habits, so you need to be strict on yourself. Any time you start to look down at the keyboard, stop. Any time you start to use your index finger to type a character that your middle finger should be typing, stop. It takes a few days of this before you start to notice that you’re typing faster than you were before. But that point will come, and from then on, you’ll get faster and faster.

Next steps

This is the perfect time to start playing on TypeRacer. TypeRacer pits you against other typists over the world. It’ll only put you up against others of a similar speed, so you definitely won’t be the slowest there. It’s a lot of fun, and great practice.

The Typing of the Dead box

Another good one is The Typing of the Dead, if you can get your hands on a copy. If you’ve ever seen The House of the Dead, it’s like that, except you walk around with a keyboard killing zombies by typing. Seriously. It’s a great game, but beware – you don’t have to backspace over incorrect keypresses, so you’ll need to be careful or it could adversely affect your accuracy.

Once you’ve got your head around touch-typing, you may want to have a play with Vim or Emacs. They’re both well suited to touch-typists, as all the important commands lie under your fingers without having to move your hands, unlike in other editors (e.g. Page Up/Down, arrow keys, and, god forbid, the mouse).

I really hope you do give this a go, because it really will help you to improve as a developer. And if you don’t want to improve, what’s the point? Good luck, and let me know how it goes.

Categories: Miscellaneous Tags:

The Ruby world is nuts

June 8th, 2011 10 comments

Apparently they started out using God, but switched to Bluepill due to its ability to monitor child processes, such as things spawned by Unicorn.

From Episode 166 of The Ruby Show.

Categories: Programming Tags: ,

JavaScript, JSON and modern web MVC

June 2nd, 2011 7 comments

As web developers we’re working through a transitionary period. New technologies are becoming widespread enough to be usable in real web apps – not just toys. After years of stagnation in JavaScript engines, Google Chrome started an arms race in speed, with all the major browser makers competing to make the fastest JavaScript interpreter. These changes are opening up a new world of web app development that hasn’t been fully explored yet, but this means we may have to rethink some of the best practices we’ve been following for the past decade.

MVC today

In a modern web application framework, MVC is well defined: requests get routed to a controller, which requests data from the model (which is probably using an ORM) and instantiates a templated view with this data. This view is sent as HTML to the browser which creates a DOM from the HTML, and renders the DOM (with CSS and images) as a web page.

JavaScript then operates on the DOM, adding a layer of interactivity to the page. As things advance, the JavaScript layer has taken on more importance, reducing the number of page reloads by selectively reloading parts of the page with Ajax.

As the complexity of JavaScript on the client has increased, it’s been easy to end up with unstructured spaghetti code based around the jQuery model of “find something, do something with it”. As more state has moved to the browser, MVC frameworks for JavaScript have stepped in to add structure to the code. The problem is, now we have two MVC stacks:

Current JavaScript solutions suffer from "Double MVC". You need both server and client-side MVC stacks. Conceptual complexity is very high.
@dhh
DHH

MVC of the future

MVCMVC or MMVVCC aren’t just un-catchy, they’re a rubbish idea. If we’re going to make rich apps that rely on JavaScript (and as I said previously, I think we should), we need to lose some redundancy.

The model will usually be accessing some shared database, so in most cases it makes sense for this component to be on the server (authentication usually lives here as well). The view is the DOM, and the controller is the JavaScript. There’ll also need to be model code in the JavaScript – the extent to which the model logic is split between client and server depends on the application.

All the server needs to do is provide JSON responses when the client asks for more data, ideally using simple RESTful resources.

One benefit of the server being a simple JSON store is that it enables us to easily add redundancy to our apps – if server1 fails to deliver a response, fall back to server2.

No more HTML

The DOM is the means by which we display our app to the user. The browser thinks in terms of the DOM. HTML is just a serialisation of the DOM. Modern web app developers shouldn’t be thinking in terms of HTML. HTML is the assembly language of the web, and we should be working at a higher level of abstraction if we want to get stuff done. Most of the time we should be thinking in terms of layouts, widgets and dialogs, like grown-up UI designers.

The problem at the moment is that HTML is being used as a combined data and presentation format, and it should be obvious that this is a Bad ThingTM.

Of course at some point we’ll need to go down to the level of the HTML, and at that point we should be using templates. jQuery.tmpl is one (see my slides and presentation on the subject), but there are many others too. One interesting development (thanks to James Pearce for putting me onto this) is Model-driven Views (MDV).

MDV links JavaScript objects to the DOM in such a way that any changes to the object are automatically reflected in the DOM, and likewise, any user changes to the DOM are automatically reflected in the model objects. Hopefully this will become standard in modern browsers, as it provides a clean and easy way to keep the DOM up-to-date with the model objects without endless wiring, which again helps to insulate us from having to deal with too much HTML.

I’m not suggesting that HTML will go away, or that developers won’t need to learn it – just that most of the time, most developers won’t need to be worrying about it, just like most of the time, most C developers don’t need to worry about assembly code.

Conclusion

Most web-apps are currently halfway there in terms of moving functionality to the client. The problem is that we have ended up in the worst of both worlds as app developers. Our serverside MVC framework provides a nice template-based abstraction over the HTML, which we then break apart on the client. We’re forced to work at too many different levels of abstraction simultaneously, which is never a good thing.

Things were simple when everything happened on the server. Things will be simple again when everything happens on the client. Over time new tools and best practices will emerge to encourage this new way of working. In the meantime, it’s going to take a bit of extra effort, but it’ll be worth it.

The end of progressive enhancement revisited

May 29th, 2011 25 comments

This is a follow-up post to JavaScript and the end of progressive enhancement - you may want to read that first if you haven’t already. The comments are worth a read, as well as the Reddit thread.

The app/doc divide

What I didn’t make clear in my original post (because I wasn’t fully clear about it at the time) is the distinction between document-based web sites, and web applications. Documents are made to be read, whereas apps are made to be interacted with. James Pearce wrote a great description in the comments:

Actually what we’re maybe observing here is a clash of civilisations: the documentistas vs the applicationistas; web-as-a-medium vs web-as-a-technology-stack; designers vs programmers. It’s no wonder that progressive enhancement is a powerful tenet for the former of each pair to rally around… but not necessarily the most architecturally important consideration of the latter.

[James has since written a great, and highly relevant, post on the subject of URLs in JS apps.]

It goes without saying that apps and documents are two completely different things (even though there is a huge amount of overlap between the two). Trying to treat web design/development as a single discipline is a mistake. Anything that is primarily meant for reading, such as a newspaper website or a blog, exists as a collection of documents. I should be able to link to another blogger’s blog post and expect that in a week or a year that content will still be available at that address. I shouldn’t need to have JavaScript enabled in order to visit that address.

Web apps aren’t the same thing. A web app is an application that happens to use web technologies. Most web apps could be desktop apps – what makes a web app a web app is that it is accessed using a browser. No-one expects to be able to hyperlink to a particular view in Microsoft Excel – what would that even mean? In the same way, I can’t share a link to a particular email in Gmail. Web apps don’t always exist in the same web as everything else – many apps are islands.

Historically, web apps have been built in the document-based paradigm, using links to move to different parts of the app and forms to submit data for processing. The only reason they were made like this was because this was all the technology allowed. Now that technology has advanced, first with Ajax and now with HTML5, we aren’t tied to this old practice.

Amy Hoy summed it up nicely:

"graceful degradation" (and likewise, p.e.) died the moment we moved away from document design and into app design.
@amyhoy
Amy Hoy

The problem is that because everyone building stuff online is using the same technologies, there can be a tendency to think that there is one true way of using those technologies, independent of what they’re being used for. This is clearly false.

The end of progressive enhancement?

Progressive enhancement is still the way to make document-based sites on the web – there’s no denying that. But with applications the advantages are less clear. If we can accept the fact that the document-based paradigm isn’t the best way to make web apps, then we should also accept the fact that progressive enhancement doesn’t apply to apps.

JavaScript and the end of progressive enhancement

May 4th, 2011 53 comments

Progressive enhancement is the Right WayTM to do things in web development. It works like this:

  1. Write the HTML for your content, 100% semantically (i.e. only the necessary tags to explain the meaning of the content).

  2. Style the HTML using CSS. You may need to add hooks to your HTML in the form of classes and ids for the CSS to target.

  3. Add JavaScript enhancements to the interface, but only enhancements.

Parallel to this, the functionality of the site/app is progressively enhanced:

  1. All navigation must happen via links and form submissions – don’t use JavaScript for navigation.

  2. Add JavaScript enhancements to navigation, overriding the links and form submissions, for example to avoid page reloads. Every single link and form element should still work when JavaScript is disabled.

But of course you knew that, because that’s how you roll. I’m not sure this is the way forward though. To explain why I’ll need to start with a bit of (over-simplified) history.

A history lesson

Static pages

Originally, most sites on the web were a collection of static HTML pages saved on a server. The bit of the URL after the domain gave the location of the file on the server. This works fine as long as you’re working with static information, and don’t mind each file having to repeat the header, footer, and any other shared HTML.

Server-side dynamism

In order to make the web more useful, techniques were developed to enable the HTML to be generated on the fly, when a request was received. For a long time, this was how the web worked, and a lot of it still does.

Client-side dynamism

As JavaScript has become more prevalent on the client, developers have started using Ajax to streamline interfaces, replacing page loads with partial reloads. This either works by sending HTML snippets to replace a section of page (thus putting the burden of HTML generation on the server) or by sending raw data in XML or JSON and letting the browser construct the HTML or DOM structure.

Shifting to the client

As JavaScript engines increase in speed, the Ajax bottleneck comes in the transfer time. With a fast JavaScript engine it’s quicker to send the data to the client in the lightest way possible and let the client construct the HTML, rather than constructing the HTML on the server and saving the browser some work.

This raises an issue – we now need rendering code on the client and the server, doing the same thing. This breaks the DRY principle and leads to a maintenance nightmare. Remember, all of the functionality needs to work without JavaScript first, and only be enhanced by JavaScript.

No page reloads

If we’re trying to avoid page reloads, why not take that to its logical conclusion? All the server needs to do is spit out JSON – the client handles all of the rendering. Even if the entire page needs to change, it might be faster to load the new data asynchronously, then render a new template using this new data.

Working this way, a huge amount of functionality would need to be written twice; once on the server and once on the client. This isn’t going work for most people, so we’re left with two options – either abandon the “no reload” approach or abandon progressive enhancement.

The end of progressive enhancement

So, where does this leave things? It depends on how strongly you’re tied to progressive enhancement. Up until now, progressive enhancement was a good way to build websites and web apps – it enforced a clean separation between content, layout, behaviour and navigation. But it could be holding us back now, stopping the web moving forward towards more natural interfaces that aren’t over-burdened by its history as something very different to what it is today.

There are still good reasons to keep using progressive enhancement, but it may be time to accept that JavaScript is an essential technology on today’s web, and stop trying to make everything work in its absence.

Or maybe not

I’m completely torn on this issue. I wrote this post as a way of putting one side of the argument across in the hope of generating some kind of discussion. I don’t know what the solution is right now, but I know it’s worth talking about. So let me know what you think!

Appendix: Tools for this brave new world

Backbone.js is a JavaScript MVC framework, perfectly suited for developing applications that live on the client. You’ll want to use some kind of templating solution as well – I use jQuery.tmpl (I’ve written a presentation on the topic if you’re interested in learning more) but there are lots of others as well.

Sammy.js (suggested by justin TNT) looks like another good client-side framework, definitely influenced by Sinatra.

If anybody has any suggestions for other suitable libraries/frameworks I’ll gladly add them in here.


Edit: I’ve now written a follow-up to this post: The end of progressive enhancement revisited.

27

April 29th, 2011 No comments

Well, I’m 27 now. I thought I’d write a little update to my New Year’s 2011 post, to assess where I am and where I’m going. As before, if you’re interested then read on, but don’t expect any life-changing revelations or anything :)

Resolutions or whatever

The following are the headings from my 2011 post, with my thoughts/progress on each point.

Finish SICP

On my way with SICP. It’s a hard slog but enjoyable too. I really hope I can finish it this year, so I can start concentrating on other things. I’ve decided that it’s OK to skip the odd chunk of exercises.

Get a decent online portfolio

Haven’t really got anywhere with this yet, as I’ve been busy working and learning.

Keep blogging

I’ve been fairly successful with the regular blogging schedule – trying to keep to weekly posting. I’ve had some brilliant successes recently, getting onto the front page of Hacker News yesterday (a nice birthday present) AND appearing in JavaScript Weekly today.

Get some more freelance work

This is going well – given the nature of this kind of work I can’t go into great detail, but I’ve got more than enough work to be getting on with right now, so no complaints here.

Do something good in node.js/Learn a functional language

Not yet, but still hoping to find the time.

Stack Overflow

Well, I talked about the page numbers of the list of Stack Overflow numbers, but they’ve changed that now so any target based on that is a bit irrelevant now. The user pages now show a “top x% overall” reputation figure – it would be nice to get that to 1% (I’m on 2% at the moment), but I’m easy.

Nick Morgan, not just Skilldrick

This is going well. This blog comes up on page 1 of google.co.uk when searching for Nick Morgan and page 2 of google.com. Seems that putting my name in the header and sidebar worked! So a solid win there.

Etc.

I’m thinking more about the social side of things now too. I went to an excellent Code Retreat in Winchester (here’s a nice review), have been attending (and beginning to help organise) my local Ruby user group, and went along to the Bristol version of the Stack Overflow Meetup. All of these have been great ways to meet my peers, get to know some really nice people, and find out what other people are up to. The Code Retreat in particular was awesome and I’d thoroughly recommend anyone seek one out and do it. I learnt a huge amount and it’s affecting my coding constantly.

I feel a bit like I’m playing catch-up – most people my age have been doing this for way longer – but I also feel like I’m actually catching up. And I’m finally doing something that I really want to be doing. These four months have passed in no time at all, so I know it won’t be long until the end of 2011 and the start of my full-time freelance career. W00t!

Categories: Miscellaneous Tags:

Magic in software development

April 24th, 2011 5 comments

First off, so we’re on the same page, we need to be talking in the same terms. So, what is magic? I’d define it as anything below your current level of abstraction that you don’t understand.

SICP cover

I first started thinking about this topic because of this angst I felt about all this magic I was relying on. There’s so many things that I rely on every day and I have no idea how they work. As far as I’m concerned, they’re magic. As long as I think about these things as magic, they seem unknowable. I think that’s where the angst comes from. What I need to remind myself is that everything I’ve learned about software seemed like magic before I understood it, and there’s nothing that’s unknowable, just things that aren’t known yet.

This post is my attempt to come to an understanding of what this ‘magic’ is, whether it’s a problem, and how best to deal with it.

The ‘Dave’ technique

At University, I learned how to record music. We recorded onto tape, so we had to learn how a tape machine worked. To learn how a tape machine worked, we had understand hysteresis and remanence and modulation and various other things that I’ve mostly forgotten now. We also had to understand how radio worked, so we learned about amplitude and frequency modulation, and how stereo FM was built on top of mono FM with an AM difference signal on top so as to be backwards compatible, and various other technical issues. Then came digital audio with ADCs, DACs, sampling, quantisation, dithering, etc. Each of these three topics was one or two modules, i.e. a semester’s-worth of lectures, so we went pretty deep.

Did that knowledge make me a better recording engineer? Probably not. But it was a helpful basis for the latter things I learned, which did help make me a better recording engineer.

The ‘Tim’ technique

Another lecturer taught us computer audio. This started off with an introduction to computers, starting at the level of AND, OR and NOT gates, then on to NAND and NOR (and how the simpler to understand AND, OR and NOT gates can be implemented in terms of NAND and NOR), then on to the ALU, then the CPU, then adding memory and disk storage and network and MIDI connections, creating a full computer. This was all presented in two or three lectures, so inevitably certain steps had to be left out (e.g. “we put together a load of these gates and we get the ALU”).

The benefit of this approach was that we were given a vague conceptual understanding of the basis on which computer audio is built, without having to spend hours and hours learning the full intricacies.

Bottom-up

If we’re intent on uncovering all the magic, how should we learn – bottom-up or top-down? Bottom-up, right from the bottom, implies first learning enough physics to understand electronics, which you’d need to understand digital circuit design, which you’d need to understand CPU design, which you need to understand assembly language, which you need to understand C, then enough C to understand Ruby (say). You’ll also need to understand how networks work right from the bottom of the TCP/IP stack and beyond; how storage works right from the level of hysteresis in magnetic particles on a hard disk; how RAM works from the level of individual capacitors; etc.

This reminds me of the famous Carl Sagan quote:

If you wish to make an apple pie from scratch, you must first invent the universe.

Top-down

Top-down is essentially the process followed by Structure and Interpretation of Computer Programs, at a micro and macro level. At the micro level, we are taught to solve problems using “wishful thinking”: first represent the problem using large custom building blocks, then build the pieces those blocks are made out of using language primitives. And at the macro level, the book teaches the workings of Scheme through a series of increasingly complex models.

Bottom-up vs top-down

The problem with the bottom-up approach is it would take decades of studying before you were let near an interpreted language. This approach may have worked in the fifties when there wasn’t as much to learn. But these days you won’t get anything useful done.

The problem with the top-down approach is that the levels we work at these days are so large, you can’t just learn all of your current level and then start to look at the lower levels, because you’d never finish learning all the details of the current level.

Leaky abstractions

Joel Spolsky has something to say on this matter in his piece The Law of Leaky Abstractions. This basically says that all abstractions are leaky, meaning that we cannot rely on them to shield us from the levels below our current level. So, to work at the current level we must learn the lower levels. It’s OK to work at the level of the abstraction, and it will save us time working at that level, but when it breaks down we still need to understand why. As Joel says:

the abstractions save us time working, but they don’t save us time learning.

Magic’s alright, yeah

Some of us don’t like the idea of magic. It feels like cheating. But it’s useful as a placeholder, something to learn about later; not now. It’s useful to delve down to lower levels, understanding how the abstractions we work with every day are implemented, but at some point the law of diminishing returns kicks in. (In some fields, we can’t even assume we’ll ever learn how the magic works. As @sermoa said, “we don’t understand exactly how quantum physics works but we accept that it does and make good use of it”.)

What we need to get to, eventually, is an understanding of the whole stack, but at greater and greater granularities as we get further away from our main level. So, if you’re a Ruby on Rails programmer, you need to know most of Ruby (but not all of it). It’s helpful to know a bit of C, and a bit of how the operating system responds to certain commands. Below that level, it’s good to have a vague understanding of how a computer works. And it’s useful to know that electrical signals can only transmit at a finite speed, and that there’s a physical limit to the number of transistors in a processor. As time goes on you can begin to flesh out these lower levels, remembring that the higher levels are generally going to be more beneficial to your day-to-day work.

Before you’ve delved deeper, those lower levels are magic. And that’s fine.


Thanks to @marcgravell, @ataraxia_status, @sermoa, @bytespider, @ghickman, @SamirTalwar and @josmasflores for their input on this post.

Also, thanks to Dave Fisher and Tim Brookes for being the sources of the ‘Dave’ and ‘Tim’ techniques.

Categories: Programming Tags: ,

Closures explained with JavaScript

April 22nd, 2011 18 comments

Last year I wrote A brief introduction to closures which was meant to help people understand exactly what a closure is and how it works. I’m going to attempt that again, but from a different angle. I think with these kinds of concept, you just need to read as many alternative explanations as you can in order to get a well rounded view.

First-class functions

As I explained in Why JavaScript is AWESOME, one of the most powerful parts of JavaScript is its first-class functions. So, what does “first-class” mean in a programming language? To quote wikipedia, something is first-class if it:

  • can be stored in variables and data structures
  • can be passed as a parameter to a subroutine
  • can be returned as the result of a subroutine
  • can be constructed at runtime
  • has intrinsic identity (independent of any given name)

So, functions in JavaScript are just like objects. In fact, in JavaScript functions are objects.

The ability to nest functions gives us closures. Which is what I’m going to talk about next…

Nested functions

Here’s a little toy example of nested functions:

The important thing to note here is that there is only one f defined. Each time f is called, a new function g is created, local to that execution of f. When that function g is returned, we can assign its value to a globally defined variable. So, we call f and assign the result to g5, then we call f again and assign the result to g1. g1 and g5 are two different functions. They happen to share the same code, but they were executed in different environments, with different free variables. (As an aside, we don’t need to use a function definition to define g and then return it. Instead, we can use a function expression which allows us to create a function without naming it. These are called ‘anonymous functions’ or lambdas. Here’s a version of the above with g converted to an anonymous function.)

Free variables and scope

A variable is free in any particular scope if it is defined within an enclosing scope. To make that more concrete, in the scope of g, the variable x is free, because it is defined within the scope of f. Any global variables are free within the scopes of f and g.

What do I mean by scope? A scope is an area of code where a variable may be defined, without the enclosing scope knowing about it. JavaScript has ‘function scope’, so each function has its own scope. So, any variable defined within f is invisible outside of f. Scopes can be nested, so in the above example, g has its own scope which is contained within the scope of f, which is contained within the global scope. Whenever a variable is accessed, the JavaScript interpreter first looks within the current scope. If the variable is not found to be defined within the current scope, the interpreter checks the enclosing scope, then the enclosing scope of the enclosing scope, all the way up to the global scope. If the variable is not found, a ReferenceError is thrown.

Closures are functions that retain a reference to their free variables

And this is the meat of the matter. Let’s look at a simplified version of the above example first:

It’s no surprise that when you call f with the argument 5, when g is called it has access to that argument. What’s a bit more surprising is that if you return g from the argument, the returned function still has access to the argument 5 (as shown in the original example). The bit that can be a bit mind-blowing (and I think generally the reason that people have such trouble understanding closures) is that the returned g actually is remembering the variable x that was defined when f was called. That might not make much sense, so here’s another example:

When person is called, the argument name is bound to the value passed. So, the first time person is called, name is bound to ‘Dave’, and the second time, it’s bound to ‘Mary’. person defines two internal functions, get and set. The first time these functions are defined, they have a free variable name which is bound to ‘Dave’. These two functions are returned in an array, which is unpacked on the outside to get two functions getDave and setDave. (If you want to return more than one value from a function, you can either return an object or an array. Using an array here is more verbose, but I didn’t want to confuse the issue by including objects as well. Here’s a version of the above using an object instead, if that makes more sense to you.)

And this is the magic bit. getDave and setDave both remember the same variable name, which was bound to ‘Dave’ originally. When setDave is called, that variable is set to ‘Bob’. Now when getDave is called, it returns ‘Bob’ (Dave never liked the name ‘Dave’ anyway). So getDave and setDave are two functions that remember the same variable. This is what I mean when I say “Closures are functions that retain a reference to their free variables”. getDave and setDave both remember the free variable name. Even though person has now returned, the variable name lives on, because it is referenced by getDave and setDave.

The variable name was bound to ‘Dave’ when person was called the first time. When person is called a second time, a new version of name comes into existence, as well as new versions of get and set. So the functions getMary and setMary are completely different to the functions getDave and setDave. They execute identical code, but in two different environments, with different free variables.

Summary

In summary, a closure is a function called in one context that remembers variables defined in another context – the context in which it was defined. Multiple closures defined within the same context remember that same context, so changes they make are shared between them. Every time a function is called that creates closures, a new, shared context is created for the new closures.

The best way to learn is to play, so I’d recommend editing the fiddles above (by clicking on the + at the top right) and having a fiddle. Enjoy!


Addendum: Lasse Reichstein had some useful comments on this post on the JSMentors mailing list – read them here. The important thing to realise is that a closure actually remembers its environment rather than its free variables, so if you define a new variable in the environment of the closure after the closure’s definition, it will be accessible inside the closure.