Sunday, December 21, 2014

PyTutorial: Working on Larger Software Projects

Working on "larger" software projects entails problems not normally encountered in smaller ones.
But what do I mean by "larger", and what are these problems, and how are they typically overcome?

Larger projects can involve:

  • More than one developer (perhaps dozens?)
  • Lots of complicated code (think hundreds if not thousands of modules.)
  • Long development period (think years)
  • Actual customers (think bugs, priorities, schedules, etc.)
Well, good luck! :)

Thankfully there are a few tools, approaches, and strategies that help deal with these problems...

Integrated Development Environment (IDE)

Dealing with lots of code is hard.  "What was that function called?"  "What were it's parameters?" blah blah blah.  Just moving around the code can be a pain, and a cute little editor like IDLE or Notepad just doesn't cut it.

Here come the IDEs to the rescue!  IDEs do many things to many people, but their main, most critical functionality is helping you navigate tons of code.  Basically, instead of opening a file, they open a project, and let you do fun things such as: opening files within the project, searching files, looking up function / class definitions, renaming functions (refactoring), debugging (pausing a program while its running and snooping around), etc. etc. etc.  As you can imagine, there are LOTS of IDEs out there, some of them free, others not so free.

By far, the most popular and powerful free IDE is Eclipse.  Although initially designed for Java, it can handle other languages, including Python, via extensions.  Python's extension is PyDev.

All IDEs have a learning curve, and make some things easier than others.  I, personally, fell for PyCharm.  It's free for work on open-source projects, and they have various discounts as well.  I don't mind coughing up a little cash for a wonderful tool.

But everyone has an opinion about their favorite / most hated IDEs.  I really don't care, they're all good, as far as I'm concerned (at least much better than no-IDE.)  So, look around, and fine one that you like.

Source Control

A.k.a. "Version Control", "Source Control Management" (SCM), etc. etc. etc.

Large software development projects are iterative.  Add a little feature here, fix a little bug there.  They are also accomplished by quite a few people, each adding some features, and fixing some bugs.  But there needs to be a way to coordinate all of this code-writing.  It's also useful to be able to see the history of some code.  Who changed what, when, and why.  As you have probably guessed, this is where source control comes to the rescue.

The most popular SCM is Git.  It's open-source, and does some incredibly hard things very well.  Another popular free SCM is SVN.  It may not be able to do everything that Git does, but is a lot simpler to work with, and does a great job for "smaller" projects (i.e. projects that don't have multiple development centers around the world.)  I actually really like it.

Which ever one you want to work with, setting it up can be a pain.  Thankfully, there are a quite a few online services that set them up for you, provide backup services, etc.  Some of the popular ones are GitHubBitBucket, and CloudForge.  There are hundreds (if not thousands) other similar services out there.  Find one that you can trust, and go with it!  If you can't trust anyone, then you're going to have to set it up yourself.  That's not impossible, but it takes time to do right.

Both Git & SVN enable you to:

Check-In some code.  That is, upload your changes into the code-server.  You can usually add a comment about the purpose of this code-change.
Check-Out / update your code.  This gets the latest code from the code server.  All the code that other people checked-in will now be on your computer too!  Woohoo!
View a history log:  see which files have changed, when, by whom, and why.
Merge multiple code changes / handle conflicts:  sometimes, more then one person may work on the same file.  In many cases, the SCM will be be able to automagically figure out how all these changes fit together.  But sometimes, this becomes too hard for the SCM, and manual intervention is required.  This usually occurs when you try to check in some code that someone changed before you.  Imagine file  Both you and "Mary" have checked-out the same copy of it.  Great!  Now both you and Mary changed the same line of code.  Perhaps you changed it, while Mary deleted it?  Mary checked-in her changes first.  So far, so good.  But now, when you come to check-in your changes, the SCM throws a complaint:  how can it change a line of code that was just deleted??  You need to get in there and fix this contradiction before you can check-in the code.  Basically, you have to decide between taking your changes (line change), or her changes (line delete), or something else entirely.  After that's done, all is well, and you can check it in and keep working.   An important lesson to be learned for this whole thing is to check-out / update your code regularly, as you want it to incorporate as many of other people's changes as possible, and thus reduce the changes for a manual-merge being required.

Branches:  a branch is basically a copy of the code that is set aside for some particular purpose.  For example, if a feature is big and complicated, work on it may be done on a separate branch.  This way, the "main" branch won't be in a semi-broken state for the foreseeable future.  The price associated with a branch is the later "re-integration" work, which takes all the changes made to that branch, and merges them back into the "main" branch.  This can be error-prone and difficult, especially if the code has changed a lot in these two branches.  Fortunately, any good SCM will help automate this process, and make it as painless as possible.  Most organizations have their own branching policies.

Tag / Snapshot: a tag is basically a snapshot of the code at any given moment in time.  This is helpful for future reference.  Tags don't actually change the code, they just label it. For instance, a tag could be "First release", or "Release 2.0", etc.  One can then compare the code in different tags, or inspect the code of a tag to see where a problem may have first come up.

BTW, most IDEs have built-in integration with most SCMs.  This way, you can update and check-in your code directly from your IDE!  If you don't like that, most SCMs have nice user-interfaces that can be used.  For example, Tortoise SVN integrates SVN commands right into your file "explorer".  Letting you perform SCM tasks as easily as right-clicking on a file or folder.  And of course, there is always the command-line interface, which usually gives you access to the full range of commands, if you need to run something peculiar. 

Python Specifics

On working on larger Python projects, there are a couple of tools / libraries you probably want to be familiar. 

The first is virtualenvwrapper.  It's a simple tool designed to let you create multiple python environments for you to play with.  Each environment can have different packages installed, and thus your different projects won't conflict.  It's handy if you want to play around with some packages without "messing up" your main work environment.  The docs are pretty good, but basically, to get started you need to:
  • Download / install.  I think this only works on Linux & Windows systems.  But I'm not too sure.
  • Set up your system to automatically us this package.  On linux this means adding to your .bashrc file the following environment variables: WORKON_HOME, VIRTUALENVWRAPPER_PYTHON, and VIRTUALENVWRAPPER_VIRTUALENV.  Then, also add "source /usr/local/bin/".  QED.
  • To create a virtual environment, run: mkvirtualenv <env_name>
  • To use a virtual environment, run: workon <env_name>
  • To remove a virtual environment, run: rmvirtualenv <env_name>
  • See all the commands.
  • After running workon, you can add & remove python packages normally using pip.  It will only install them for that particular virtual env.
The other is nose.  This is a very common unit-testing tool.  I'll probably blather more about it in my section on unit-testing.  At the very least you need to be aware of it.

Gimme more tools!

IDEs and SCMs are by far the two most important tools for larger software projects.  But lots of other helper tools exist, and their usage depends largely on tastes and needs...

Bug-Tracking: if you got lots of bugs coming in, and you want to figure out who works on what, when is each fixed, etc., then you're going to want some sort of bug-tracking tool.  Some popular ones are BugzillaTrac, and Mantis.   A nice feature of Trac is that it has an integrated Wiki (and was written in Python).  They are all good.  Some prefer writing bugs on a white-board, or on little cards. :)

Wiki / Documentation: I'm not a HUGE fan of horrible, outdated documentation.  But having a place for people to share ideas is nice.  I would keep it small and simple.

Ok, that's enough! Just get to work already!! :)

But I want more!  Gimme more!!!

Argf! Ok.  Well, larger projects benefit from a sane approach to development. There is not one "right" answer, but very few people seem to realize this. :) 

I think the most important aspect of large project development is unit testing.  It really helps keep things sane.  

But there are also philosophies! Today's fashion include Agile.  From Agile, I mostly like Pair Programming.

Another important aspect is taking naps.  Working too hard reduces productivity.  It also reduces the amount of fun you have in your life.  And that would be a shame now, wouldn't it?

Oh yea, there's also that great classic, The Mythical Man-Month, in which the concept that just adding more people on a project will actually make that project complete faster is thoroughly destroyed. 

Making big software thingis is hard work.  It takes time.  And you can't plan for everything.  Have fun and good luck!  :)

Saturday, December 20, 2014

PyTutorial: Minesweeper, anyone?

So, you've written a text-based Minesweeper game, and now it's time to add some nice little graphics to it.  The big question though, is how? :)

I hope you spent some time looking around for some packages, have you seen anything you liked?
But before delving into specific packages, let's take a moment to think about our requirements:
  • We want to add graphics (images) and sounds
Anything else?  Perhaps:
  • Add some graphic controls, like a menu and a way to chose the size of the board, etc.
As with any programming task, the more you search the web for help, the more you'll learn about the problem itself.  You may find out that in tech-speak, the graphics of a program is called a GUI (Graphical User Interface).  Furthermore, typical GUI controls (like menu items, buttons, check boxes, etc.) are called widgets.

Now, if you think about it, Minesweeper can be made to work entirely with widgets!  Your main board is basically just a bunch of buttons: left-click to see what's inside it, right-click to flag as a mine.  Not much more complicated than that!  Therefore, we may want to refine our requirements to include:
  • Support for widgets
Now, let's take a look at some packages out there:
  • pygame: a cute little package that seems to be designed for game making.  Although no direct support for widgets, there are other packages that provide such functionality (see pygu and pygameui), although their documentations seems to be lacking.
  • pyjs: an interesting tool that lets you automatically convert your python code into javascript, and thus have the game run in a standard web browser.
  • kivy: a sophisticated package designed to have your python code run on smartphones and tablets.
  • TkInter: seems to be the "standard" for Python widget programming.  Although not designed for games, it should be fine for Minesweeper.  Also, it's easy to find tutorials for it.
The above four options (and there are lots more, of course), raise another question: how do we want our game to be played?  Or more precisely, where?  Do we want our game to be played as a regular desktop application (a bit old fashioned, perhaps?)  Or as an online in-browser game?  Or even as a mobile app?  As you probably imagine, each of these options mean not only a a different way for your code to look, but also a different way for your actual game to look!  There's no "right click" on the smartphone, and no "saving high scores to disk" on browser games (they usually save on a remote web server.)  These differences, although may be slight, are still different enough for you to have to consider.

But there's more.  I want you to spend about 15 to 30 minutes studying each of the above four packages. Browse through some tutorials, look at their APIs (Application Programming Interface, or tech speak for the details on how one can use these packages.)  I want you to notice how these packages deeply influence the way you will write your code!

Go ahead and browse through them already.  Go! :)

Back already?  Great!  I hope you noticed that there's much more here than choosing between "urllib2.urlopen()" and "requests.get()" (from the previous lesson) that basically do the same thing in slightly different ways.  Each of the four packages above will dictate how you will even think about writing a Minesweeper game!  They involve completely different APIs (modules, classes and functions), and even use different terminology.  They may overlap in the sense that they let you achieve similar things (in different ways), but also differ in that each will offer you the abilities to do important things that the others just aren't capable of.  They influence your program so deeply that instead of being called mere "packages", they get to be called "frameworks"


It's not always clear when a package becomes a framework.  The rule of thumb is that if a package requires you to organize your code differently, then it's most likely a framework!  Another is that if one package is deeply incompatible with another (that is, you can't use both at the same time, nor would you want to), then again, most likely both of them are frameworks.  If instead of examples, these packages require entire tutorials (perhaps even more than one), then most likely you're dealing with a framework.

Frameworks are entire worlds. They are not simple to learn.  They have many quirks.  You may never feel like you know everything or even most about them.

Then why bother to use them?  The short answer is that time is money, and frameworks can save you lots and lots and lots of valuable time.   More than that, good frameworks teach you a good way (but usually not the only good way) to achieve a very complicated goal.

Just like you would much prefer to use "urllib2.urlopen()" to fetch some data from the web (instead of figuring out how to do that yourself), a good framework will take care of a lot of the dirty details involved in writing complicated applications, leaving you to concentrate on what you want to get done.

Having said all that, how do we chose one framework among many?  There's no simple answer here, and it really depends on your requirements, what the different frameworks offer, as well as which framework best fits your personal programming tastes.  Since framework affect how you think about programming, you may as well find a framework that thinks like you do.  A good framework needs to earn your respect. :)

Unless, of course, you're going through some online programming tutorial.  In that case, whoever wrote that tutorial will simply choose one for you! :)  Very nice.

And the winner is.......

PyGame!!!  I like it best for this tutorial because it doesn't require you to learn any major new technology (such as web or mobile programming).  But also because it's a truly fun package that helps keep the spirit of programming alive!  Minesweeper is a game.  Let's use a framework designed for writing games!

You can get pygame from here, or if you're running Ubuntu, just run:
> sudo apt-get -q -y install python-pygame

I would start by looking over (and typing in, of course), this cheat-sheet that someone (not me) made.  There's also a newbie guide that helps understand some key pygame concepts.  Of course, there's also the main documentation page with lots of tutorials and references.

But how does one really get started with such a package?

  • Start small: make the simplest, dirtiest prototype just to get started
  • Trial and error: assume that you're going to learn things as you go.  So just get started.
This approach is good if you've already made sure that the package will over-all meet your needs. For instance, if you want the game to run on your phone, you don't want to find out that PyGame doesn't support that after you put in all the effort to learn it.  But who cares about mobile?!  Let's get started!

So what's small? The good news about PyGame is that it works with the regular text-console that you're used to work with.  This way, you can still use print for debug, as well as raw_input to get input from the users.  This way, you can add the graphics part of the game incrementally, which is nice when you're learning.

You can start by writing the code just to display the mine-field, with all other user-input happening in the text-console.  The first step to achieving that is to draw a bunch of pictures!  You can use any old graphic editor for the job, including Gimp.  Thing of all the different kind of cells you're going to want to display: unknown cell, flagged, cell, clicked cell with no mine, clicked cell with one neighboring mine, etc.  Draw all of those, and save them as PNG.  Make sure they're all the same size, of course.

Now comes the fun part, use pygame to display a minesweeper grid!

First, you're going to want to init pygame.  You can look at the cheat-sheet to see how to do that.

Then, you're going to want to load the graphic files, you'll want to use:

Btw, if you want to use the more efficient "convert" function (as the newbie guide) recommends, you're going to need to call pygame.display.set_mode() first (see just below.)  While for most animation games this is probably important, in Minesweeper, this is less so.

Next, you'll have to figure out the size of the window that you're going to need.  Basically it's going to be width_of_image_cell * num_cols by height_of_image_cell * num_rows.  Then create such a window using:
display_surface = pygame.display.set_mode((desired_width, desired_height)

Now, there are many ways to draw the cells onto the window.  Minesweeper isn't animation-intense, and just about anything you chose to do will probably work fine.  PyGame works with Surface objects.  For instance, both pygame.image.load() as and pygame.display.set_mode() return Surface objects.  If you want to draw one surface onto another, you use the blit class-method:

Thus you can draw directly on the display surface (as returned by pygame.display.set_mode), or draw on an intermediary surface, and draw that on the display surface.  It's up to you!  I prefer the latter as it gives you a little more flexibility (imagine wanting to display more things besides the minesweeper grid.)  You can also use sprites, but at this point, they may be somewhat of an over-kill.  You can go back to them later. 

Remember that blitting surfaces does not actually make anything get displayed on the screen.  To achieve that, you need to blit on the display surface, and then call update() on the display, like so:

Update is actually a bit more sophisticated than it seems.  Calling it with no parameters tells it to update the entire display.  For something like Minesweeper, this is probably good enough.  That's because you only need to call update() when something changes in the display, and in Minesweeper, this doesn't happen very often (only when someone clicks or flags a cell.)  But imagine a game like Pac-Man or The Legend of Zelda.  These games have lots of little animations, and painting the entire display, 30 times a second (for the animations to look smooth), is very inefficient.  Inefficient enough that it may actually be noticeable.  For those games, you only want to update parts of the display, and pygame.display.update() allows for that as well.  It is these sort of games that PyGame's sprites really make a difference, and if you want to make such a game, then definitely look into them.  Minesweeper, on the other hand, is not that kind of game.  A player may click or flag a cell at most at around 1 a second, and at these rates, updating the entire display will not be a problem.

So there you have it, now you should be able to actually display a minesweeper grid.  But that's not much of a game.  You also want to be able to handle mouse-clicks.  

That's actually fairly straight-forward.

Mouse-clicks are events.  Things that happen out of your control, as the player decides if, when, where, and how (right or left mouse button) to click.  PyGame handles these and other such events in a very simple event-loop.  The cheat-sheet goes into all the details you need, but basically, at any time you want, you can check if an event has occurred calling pygame.event.get() from inside a for loop.  Each event is an object with a type attribute that you can check for, and other attributes as well depending on the type of event that it is.  For example, a MOUSEBUTTONUP event has a pos attribute which is a tuple containing the (x, y) coordinates where the mouse button went up.  

Well... that's pretty much all you need to do to actually have a graphical interface for the game-play!  If you want to go all-out with all user-input being graphical (as a normal game is), then you'll want to find a package for pygame that can do widgets.  I'm not going to go into that here, but if you really want to do this, I recommend you download the pgu package.  It has zero documentation, but lots of nice little examples that you can run to see how it works. :)

Well, there you have it!  By now you should be able to figure out how to write a nice little Minesweeper game!  That's no small feat, and I truly hope you enjoyed the journey.

Of course, you only just began your adventure into programming land!  But now, you have enough wisdom to help guide wherever you may go.

Where to go from here?

As you probably imagine, that largely depends on you.  But here are some random ideas:
  • Make another game! How about checkers or chess?  I haven't taught you how to do artificial intelligence yet (I probably will in one of my advanced lessons), but you can definitely make a two human player game.  But you can also make other kinds of single-player games, like some crazy Pac-Man or something. 
  • If you're into math / programming riddles, check out Project Euler.  It's lots of fun, but hard as all hell.  So good luck, but don't expect to finish it all in a week.
  • Check out my blog, as every now and then I'll add another "advanced" lesson.  I already have on one writing web-apps, check it out!
  • Learn another language!  Nothing helps hone your programming skills by learning some other language.  If nothing else, it'll give you an ever deeper appreciation of Python! :)  You can learn Javascript, which is fairly popular these days.  Java and Swift are also popular.  You can also learn ChucK, a wacky language specifically designed for music.  A completely useless (yet interesting) language to just muck-around with a bit is lisp.  And of-course, there will always be logo.  :)
  • Python has a very wide selection of packages that do some amazing things!  Just search for one that does something you find interesting, and play with it!  
But no matter what you chose to do with your new-found knowledge (and mad skillz), just remember not to work too hard, to take naps, and to make the time to smell the roses.


Friday, December 12, 2014

PyTutorial: writing web applications

This post teaches how to do stuff on the I-N-T-E-R-N-E-T.... woa!

But what is stuff?  And what is the internet?  Very good questions!

Since you are able to reach this blog, which is on the internet, I'm going to assume that you have at least a super-basic knowledge of the internet.  That is, that you can use web browsers to visit web sites.

In this post, I'll teach how to write web-applications.  By that, I mean writing web sites that do some fairly complicated tasks.  Another name for web-applications is dynamic web sites.  Basically, it's a web site that interacts with web user (a.k.a. client.)  This is contrasted with a web site that does not interact with the clients (a.k.a a static web site).  For example, this very web-site is very minimally dynamic: you can rate and add comments at the bottom of the page.  If I were to disable those, this web site would be completely static.  In a dynamic web site, the client can change something (such as add a comment, etc.) on the web site.  In a static site, all the web client can do is get stuff.   An example of a static web site is The Best Page In The Universe, as no one can leave comments or change the content in any way.  An example of dynamic web sites are Google (the page you get depends on the words you typed), Facebook (news feeds, recommendations, ads, what-not, depend on content people gave Facebook), etc. etc. etc.

There is a caveat, however, and to understand it, I'll have to go into how web sites work.

How Web Sites Work

Web sites have two parts:  the part that runs on the servers (the computers that belong to the web site), and the part that runs on the client (the browser running on your computer.)

Clearly, there is some interaction going on there.  To simplify things, it's good to think of what happens as a transaction, that is, the browser asks for a web page from the web server, and the web server responds with that web page.  When the transaction is over, the web server and web browser are free to go on with their business.  They can effectively forget about each other, until the web browser initiates another transaction.  In other words:

  • Only the web browser can initiate a transaction.
  • A transaction happens very quickly.
  • When the transaction is finished, it is finished.
Of course, just because a transaction is finished, it doesn't mean that the browser or server won't remember anything.  They may, but there is nothing "built-in" that forces them to.  For example, a browser can keep a history of places you've visited.  And a web site (especially a dynamic one) may save information from that transaction as well (such as your Twitter tweets.)

Let's "zoom in" on that transaction for a moment.  We said that the browser initiated a transaction.  This may happen by a person typing in "" in their address bar thingi.  The browser goes to Google's web site, and gets something back.  That something is called HTML.  If you're using Chrome, and your right-click on any web site, you should see a menu option called "View page source."  If you choose that, you will see the HTML that you got back.

Your trusted web-browser got that HTML, analyzed it, and made it look all nice and pretty: made it look like whatever it is you're seeing when you're browsing around.   HTML can contain links to other web pages.  If you click on such a link, another transaction is initiated and another HTML document is returned to your browser.  HTML can also contain forms, and if you fill it out and hit "submit" or "search" or "whatever", then another transaction is initiated, the browser sends whatever it is you filled out to the web server, and the web server gives back another HTML doc.  That's the way it works.

Sort of.  Let's zoom in a little more.  We mentioned that the web site responds to a web request with HTML.  But how does it figure out which HTML to respond with?  That greatly depends on the nature of the web site, and there are not hard-and-fast rules on how to do this.  For example, a static web site may just be a bunch of HTML files laying around in a computer folder somewhere.  A browser requests a certain file, and the server responds by giving back the contents of that file.  

A more complicated, or dynamic web site will have some sort of programming running on that server that figures out what to do.  That program can be written in just about any language imaginable!  Python is one of these languages, and lucky for you, Python actually gives you great tools to write sophisticated web sites with relative ease.  But you are not limited to Python.  You can write web sites in C++, Java, and even languages that were specifically invented for writing dynamic web sites (such as PHP and ASP.)  

So, you may naturally ask: hey, I've heard of Javascript; is that just such a language?  Unfortunately, the answer to that question would have to be no.  Javascript is actually a part of HTML.  

Say what?

Yes, the web is a dangerous place. :)  Basically, the world wasn't satisfied with boring old transactions as I've just described.  The world wanted web sites to be cool, to seem even more interactive than some dull forms and links.  The people demanded to be able to play Farmville or whatever it is they play these days.  They wanted Google auto-complete searches!  They wanted the world!!! What they got, unfortunately, was Javascript.  In a sentence, Javascript is the programming language you must write in if you want a program to run in the web browser (as opposed to the web server.)

Ah, Javascript.  Unfortunately, any tutorial on web development would be incomplete with out it.  Basically, Javascript is one of the worse languages ever invented.  It is horrible at just about everything it does.  But to its credit, it does a lot

So what is this demon I speak of?  Let me put it this way.  When you wrote your little Python scripts, you downloaded the Python interpreter, wrote the code, and had it run.  You can have it do anything you want, you can have it completely destroy the computer if you wanted too (I leave writing a quick program that tries to erase every file on the computer as an exercise to the student.)  And that's Python, a fairly advanced language.

Javascript is a way to write code that cannot harm the computer.  You can feel safe about browsing any web site in the world without worrying about your computer getting destroyed.  That's one thing that Javascript does well.  Another thing is that you can visit a web site from a Windows, Mac, or Linux computer, or even from an iPhone or an Android device, and that Javascript will work just as well.  Kind of.  This is something that Javascript tries to do well, but in reality it kind of sucks at it.  But again, to its credit, running on every imaginable device and running well is a pretty tough thing to expect.  And it does this, sort-of kind-of well.  Or at least well-enough.  In essence, it does this at the cost of making the Javascript-developer want to constantly want to throw up.  Or at least that's my experience with it.  :)

The reason that I'm telling you all this is that even though you can write the server-side code in any nice computer language you like (a.k.a. Python), if you want to do cool things on the client-side, you're going to have to do it in Javascript.  

The plus-side of Javascript is that it used to be way worse. :)  Today, you have what I call the new "HTML-Trio": HTML5, CSS3, and modern Javascript that make that whole crappy-mess somewhat less crappy.  Simply put, all three things are basically a part of the HTML standard, but each serves a different purpose:
  • HTML5: that's the basic HTML, what describes the structure of any given web page.  What are its links, where are its images, etc.
  • CSS3: that's describes the looks of a web page.  What fonts should be used, how are things spaced, the colors, etc.
  • Javascript: that describes dynamic aspects of the web page itself.  Things that the user can interact with without having to initiate another transaction with the web server.
To make things more difficult, CSS & Javascript can be embedded in HTML (that is, the code is write there inside the HTML.)  Not only that, Javascript can actually modify HTML on-the-fly.  I don't want to get into the details.  No one does.  On that note, I give you your first exercise:
Exercise 1: go to and play with their HTML, CSS, and Javascript tutorials.  This may take you some time (as in a few days), but get familiar with the technologies.  No need to be an "expert", just have a good sense of how they work together.
Exercise 2: review the HTML cheatsheet and  HTML Form guide to get a good, quick overview of what HTML has to offer.
BTW, this discussion won't be complete with a little tangent.  Have I mentioned that everyone hates Javascript?  As a result, many have tried to replace it (failed) or make work-arounds for it (mostly failed.)  For example, Google came up with a new programming language, Dart, which lets you write in Dart, and then "convert it" to Javascript.  Google also came up with an interesting technology called GWT that attempts to kill two birds with one stone: you write the server-side code in Java (sorry, not Python), and it automatically generates the required HTML, CSS, and Javascript code for you.  And of course, there is also pyjs, which lets you program in Python and have that turn into Javascript.

The reason I say that at best these technologies have mostly failed is that ultimately, they all generate Javascript.  And as such, if there is a problem, it's going to be in Javascript.  What's worse, is that if you want the browser to do things that they didn't think of, you will have to either:

  1. have to write Javascript
  2. cry a lot, and realize that you can't do it.  Then cry some more.  

Still, GWT has its followers, and Dart may be picking up steam.  Pyjs seems to be less perfected, but if you want to write a little Javascript game, it may be what you're looking for.

The last thing that I'm going to say about Javascript before moving on. Just like in Python, in order to do anything useful you are going to want to use packages that others have wrote.  Some popular Javascript ones you may want to be familiar with are jquery,  jquerymobile,  bootstrap,  angular,  dojo,  d3,  and many many more.  You may be tempted to think that with all these packages, writing Javascript should be a breeze.  But in that case, you would be forgetting that Javascript is crap, and thus these wonderful and amazing packages tend to break one anther in mysterious and creative ways.  Enjoy!

BTW, writing CSS can be a bit of a pain as well (but not nearly as bad as Javascript.)  Still, there are some nice tools out there to make authoring nice CSS all the merrier.  Check out SassCompass, and Bourbon.

There are also plenty of HTML authoring tools out there.  They are divided into two major kinds: WYSIWYG (What You See Is What You Get) and those that just help you work with HTML.  WYSIWYG are the most tempting, because, well, you don't need to know any HTML to create HTML.  The problem with them is that after you have something that looks nice, when you see the HTML that it created, well, it will most likely be complete junk.  You'll have 10,000 lines of code for something that can probably be achieved in 100.  This makes changing it, and making it dynamic (with Javascript or with Python) near impossible. This leaves you with editors that just help you work with HTML (syntax highlight & preview), which are cute, but not extremely helpful.  In other words, not much you can do to get around learning HTML.  But then again, you always wanted to learn HTML, so it works out! :)  Of course, you don't need to be an expert at it.  It may be good enough for you to have a basic understanding of it, and then get the help of an expert HTML designer to make it look all spiffy.
Exercise 3: check out Bootstrap's templates that make creating simple, yet beautiful, static web pages easy!  They take advantage of sophisticated JS & CSS libraries, including Font-Awesome, to simply development.  If you need to add some nice pictures, check out Pexels for some images free of charge.  Makes you want to create a web site, doesn't it? :)

Oh No! Another Tangent!!!

Sorry.  But as with any new programming field, there is a lot to learn.  The good news is that I'm close to getting started on showing you how to write dynamic web applications in Python.  The bad news, is that I feel that a little more background information would be useful.  The reason I'm doing this is that although you may not need all the information I will impart you with, you may run into some of the terminology that I'm going to teach you, and I want you to understand what it means.  Nothing here is rocket-science, btw.  Programming in Javascript is a lot harder.

In the previous tangent I discussed the transaction that takes place between a web browser and a web server.  Before going into the details of the server-side in the transaction, I want to give you a little more background about how the internet freaking works!

Say you type "" in your browser and hit go! Many things happen to get that transaction going.

First, the browser needs to find the server.  How can it do that?  Well, is called a URL that describes to the browser the steps in needs to take to initiate the transaction.

In that URL, there is, that's the domain name (or host name) of the web server that it needs to find.  The browser can then, ask a domain name server (DNS) for the IP address that that domain name refers to.  IP what?  Every computer on the internet has a number.  That number is called its IP-address.  Most IP addresses are four numbers separated by a period.  To see what I mean, you can go to host2ip, type in any domain name that you want (,, etc.) and it'll give you its IP address (something like  Every computer has a name for itself called localhost, as well as an IP address for itself:  

Once the browser figures out the IP-address, it can use cool little internet protocols to figure out how to send data to the web server.  How that happens is also very interesting, but for another day (the curious can read about the OSI model.)

It may be shocking to discover that the internet is used for many purposes, and not just web browsing (sending emails is just one of many other things that can be done.)  Each such purpose has an internet protocol that define how two or more computers on the internet interact with one another.  For web browsing, the internet protocol is called HTTP, hence the "http://" at the start of the URL.  Secure (encrypted) web browsing is governed by the HTTPS (HTTP-Secure) protocol.  Luckily, HTTPS is just HTTP with encryption lapped on-top of it.  So once you know HTTP, you don't need to know much more to do HTTPS, just everything that there is to know about security.  

But that's not all.  In the basic Python tutorial you should have learned that computers are dumb.  Therefore if the client wants to talk HTTP, but the server is hearing SMTP (used for emails), as you can probably imagine, at best nothing will happen, and at worse, a big mess will happen.  Most protocols aren't designed to let the computer on the other end know which protocol is being used.  It's not like to do HTTP, the first thing the browser says is "I'm going to do HTTP now."  It would be nice if that how things worked, but it's not.  Instead, another way was devised to enable one server to talk multiple protocols at once.  This is achieved using ports.  

A port is just a number between 0 and 65,535.  The "standard" or "reserved" ports are under 1024.  For example, the standard port for HTTP is 80.  The standard for HTTPS is 443.  The ones for SMTP are 25 and 587.  The significance of these numbers is that now a computer on the internet can be used for many purposes at the same time without confusion about what is happening at any given time.  It's important to remember that these numbers are just standards and recommendations.  No one is stopping you from running an HTTP server on port 8080, or port 25 for that matter.  It's highly advised that you not use a port under 1024 unless it's the standard for that protocol (i.e. 80 for HTTPS).  But anything over 1024 is free for you to do as you please.  For instance, if you wanted to run an HTTP server on port 8888, you can do that, but in order for the web server to know that you it's running on port 8888 instead of 80, it will have to be explicitly told to do so in the URL, like this:

(Note the colon followed by 8888 at the end of the host name in the URL.)
Of course, there is nothing listening on that port for the server, so if you type in that URL in your browser, you'll get a webpage not available message from your friendly browser.

You can also be explicit about port 80 in the URL:

But it's the same as leaving it out, as it is the standard port for HTTP.  In fact, your browser may conveniently "erase it" for you.

You can also be a bastard, and so something like this:

Now you're telling your browser to talk HTTPS (secure) on port 80 (non-secure HTTP).  It'll try, but it'll fail because the server will talk HTTP back.  You should get an SSL connection error of some sorts.  SSL is basically the technology behind the 'S' in HTTPS. :)  While also very interesting, this too shall be for another day (the curious can read about Public-Key Cryptography.)

What's left in the URL is all that stuff after the host. In my example its:

This tells the browser to get the file called "/p/pytutorial.html"  Or more completely, it tells the browser to use the HTTP protocol to ask the server named and which is listening on port 80 (implied) for the file found at path "/p/pytutorial.html".  QED.

(BTW, I don't want to mislead you into thinking that all internet protocols are transactional in the same way that I described in the previous section.  HTTP and HTTPS are, and these are the protocols that you'll be familiar with in web development.)

So is that it?  Are we FINALLY ready to get started on Python programming for the web?

Almost.  I don't want to get into all the details, but there is more to the HTTP protocol than the URL that I've just described (and more to the URL than I have just described too.)  

HTTP also allows for parameters (can be thought of like parameters of a Python function) to the web server, and thus allow more structured data to go through.

HTTP allows the web server to save cookies over at the browser side.  Cookies are similar to a Python dictionary.  For instance, a cookie name can be "last_visit_date" with a value being "2014-12-13", etc.  A web server can use cookies to keep track of returning visitors, or anything else they find useful.  They way they work is that the server, in the response of the transaction, basically tells the browser to save a few key-value pairs (like a dictionary.)  Then, in future transactions that the browser initiates to that same web-server only, it will provide those same key-value pairs to the server.

Another feature in HTTP is that with every response, the web server responds with a status code.  For example, 200 means "all is well", while 500 means "something is seriously broken".  404 means "i can't find what you're looking for."  (and thus a broken link.)

That's pretty much it for now.  There are more online explanations of http and cookies.  If you want the actual protocol reference, you can look up the relevant RFC (see http and cookies).  But at this point (and possibly ever) these RFCs may be too much information.

"Please remind me why I'm reading this post?"

To learn to do web development [using Python.]  "Ahhh, now I remember, yes!"

As I hinted previously, Python has some really cool tools to help you in your web development.  Each has a different approach, with advantages and disadvantages each.

django - a very comprehensive framework.  It has a lot of built-in features that help you build super-complicated stuff super-fast.  The draw back is that it has a somewhat steep learning curve, and making little modifications may be complicated.  You need to understand how it works and work with it, not against it.  It a bunch of powerful reusable apps that you can download an "plug into" your project.  It also has a built-in admin interface that lets you develop sophisticated data-entry websites quickly and easily.  In a sentence: it's a wonderful world, but one that takes getting used to (for help on that, see my post on Django hints & tips.)

flask - in many ways it's the also-powerful opposite of django.  It has a very low learning curve, and comes with some very powerful tools to help you with.  But the simplicity comes with a price: you have the tools, but now you need to work hard to use them in order to build something useful.  If you're doing something very unique, this may be your best bet.  Otherwise, you may be longing for a few of those django reusable apps.

There are many more frameworks, but I believe that (at the time of this writing) these two are the dominant ones.   While very different from another, they both share a few key (and wonderful) features:

ORMs (Object Relational Mappers)

This is a powerful technology that lets you work with databases (huge data stores containing complicated, and related information) without having to know anything about databases!  They let you use object-oriented programming (Python classes) to add, retrieve, update, and delete information.  Django has ORMs built-in, while Flask is very well integrated with sqlalchemy

Clean & Smart Template Model

As mentioned earlier, the final output of your web application will be some sort of mix of HTML, CSS, and Javascript.  Of those three things, the CSS & Javascript tend to be rather static (that is, the CSS & Javascript files are returned to the browser unmodified.)  What actually makes each transaction different is that the HTML is different (which may invoke the CSS & Javascript differently.)  Thus, in essence, the dynamic part of a web application is actually the different HTML that is generated for every request. 

This can be very hard and messy to achieve.  In the PHP & ASP programming language, the code is deeply embedded in the HTML, making trouble-shooting and testing quite difficult.  With both Django & Flask, there is a clean separation between the abstract work that the web application is performing (information processing, etc.) and how it is presented to the end-user (HTML generation.)  Django calls this MTV (or MVT). The Model is the basic information structure.  The View describes which information is then presented (that is, given back to the web browser.)  While the Template describes how this information is to be converted to HTML. 

As is the theme with Django and Flask, Django has this template model built-in, while Flask is very well integrated with jinja2.

Integrated Unit-Testing

Both Django and Flask support unit-testing.  They help generate "test databases" and provide tools to help you make sure your web application is working as expected.

Reverse URLs (creating links)

Building dynamic applications is, well, a dynamic process! :)  As your application changes, you want the links that are generated to be good (that is, not broken.)  Both Django & Flask have built-in tools to help ensure that this never happens.


Having your web application out there begs hackers to try and mess with it.  Both frameworks give you the tools to prevent many common web attacks.  Of course no defense is prefect, and your web application may still be vulnerable some some attacks.  And although very interesting as well, this discussion will be left for another day.  (The curious can look in OWASP's Top Ten.)

Complementary Technologies

Both support integration with many complementary technologies.  More on that in a bit.

And much, much, more....

Both frameworks offer so many other features that I'll be hard pressed to name them all.  Internationalization -- allowing web sites to support multiple languages / localization. Performance optimizations.  Debugging tools. And of course, great online documentation (as well as a few books.)

Complementary Technologies

(See, that was in a bit!)

HTTP is a complicated protocol. And to implement it well, as well as fast, is hard.  Therefore, a division of labor was established in which certain web servers do the HTTP, while application frameworks communicate with the web servers and let you, the developer, write code mostly oblivious to HTTP.  Common (and free) web servers that are used in production environments are apache as well as nginx.  But don't worry about it for now, as both Django & Flask have built-in web servers that are great for development purposes.  And even if you'll want to go to production, you can leverage services that take care of the web servers as well as other complementary technologies for you.  (A quick search got me django and flask)

Another common complementary technology is a database server.  This is the technology that stores all your valuable information.  It is also the technology that the ORM in these frameworks tend to shield you from very effectively.  But even if you're not interacting with them directly, you still need to use them!  A great free such database that is good for development purposes is sqlite.  What's great about it is that it comes built-in with Python, and you don't have to do much to set it up!  Unfortunately it is not so great for many production environments.  In those cases, you may want to use postgresql or mysql (both are free, and work great!)  Again, you don't need to be concerned about this now.  This is just something to keep in mind.

These are the big ones.  Are are more, niche technologies (such as redis), but their use will depend on your specific needs.

Get on already!

Well, now you should be familiar enough with the main aspects relating to web development.  
Exercise 4: go through both the flask tutorial as well as the django tutorial.  Going through both will give you a better feel for what they are like and how they differ.

That's it!

Seriously.  :)  All I can do now is give you some personal recommendations and tid-bits of information.

First, what do I recommend?  Flask or Django?  My honest answer is that if you have the time to learn Django, it's well worth it.  I say this not because I like the learning curve, but because all the re-usable apps and the admin interface as incredibly powerful tools at your disposal.

Saturday, November 8, 2014

Let's make some Hummus!

Hummus is yummy.  It's also fairly easy to make, is very filling, and is considered a healthy alternative to not eating hummus!  (Oh yea, and it's cheap.)

But before showing you how fun and simple it is to make, I have to make a little confession: I, although born in the Middle East, well, how can I say it, grew up thinking that hummus is, well, not so yummy.  Ok! I said it!  I thought hummus was not even good as spread!  I couldn't even imagine eating it as a meal.  Well, until a few years ago, when I went to some hole-in-the-wall restaurant in Israel, and "discovered" what everyone here already knew: hummus is yummy.  Very yummy.

See, eating hummus out of a plastic container that is sold in a grocery store is like eating scrambled eggs that was frozen, then defrosted, frozen again, defrosted again, mixed with preservatives, blended into some paste, and then served cold.  Just because that doesn't sound appetizing doesn't mean scrambled eggs can't be good.  They are just two very different things, that is all.

Fresh hummus is like good pizza.  Everyone has an opinion about what makes it good: fine or coarse, spicy or not, etc. etc.  I make no claims about what is the best "ultimate" hummus.  I'm just going to show you how I make good, fresh, hummus, with the hopes that it will inspire you, dear reader, to try to make it yourself, and see which style of hummus you go all-hummus about.

So here goes.  Years of experience, right here, on a little blog.

First, get some dried chickpeas.  I only ever saw the white (beige?) kind, but apparently there are also green ones.  I have no idea if they're any good.  So if you're doing this for the first time, get the white ones.  They probably sell them in a bag, maybe a pound or two.  Doesn't really matter which size you get, at least I sure as hell don't care.  Just find some dried chickpeas somewhere.  I'm sure you can find them.   Another "exotic" ingredient that you're going to want is tahini.  That is basically ground sesame-seed paste, and is awesome in and of itself.   You're gonna want to see my post on how to make tahini for this.  Don't worry, that's even easier, and keeps in the refrigerator for about a week.

To sum it up, you'll need:

  • A bunch of dried chickpeas
  • Some pure tahini paste
  • Some tahini sauce (see sister recipe)
  • A real lemon
  • An actual garlic (you won't need the whole thing, just 1 or 2 cloves, depending on taste.)
  • A little salt
  • Maybe some olive oil
  • If you have some parsley, all the more power to you.
  • Ah and as decoration (and definitely not a must), some za'atar or sumac.  Not a must, people.

Pre-soaked, frozen chickpeas waiting to be cooked.
Start by putting all (or a good amount) of the chickpeas in a bowl of water overnight.  The chickpeas will expand, so make sure you have a bunch of extra water in there.   The reason I recommend using all the chickpeas is that soaking the chickpeas is annoying, and after you soak them, you can keep them in the freezer for a long time anyways, so may as well get it over with once.  Having soaked chickpeas in the freezer has the additional benefit of letting you be spontaneous.  You can just wake up one morning and say to yourself: I want to have a nice bowl of hot hummus for brunch, and guess what?! You'll be able to have one, because you, in your infinite wisdom, have kept pre-soaked chickpeas in the freezer!  That's right!

Cooking with a pressure cooker
So fast-forward to you having soaked your chickpeas, you can put some (either from the freezer or from last night) in a pressure cooker.  Don't have one?  Sucks for you.  Consider getting one.  It's amazing.  Really, it is.  But don't worry, you can still make hummus without one.  It just takes a little longer, but no big deal.  Take the amount of soaked chickpeas that you want to make, put them in your pot (or pressure cooker), cover with them with about an inch of water above the highest chickpea, and cook them until they're soft. The rest you can keep in the freezer for another day.  But how much to make?  Again, that depends on appetite and how many other things you'll serve with it, but a rough guesstimate is to cook about a cups worth (pre-mashed) per serving.  That seems to work for me.  Better make a little bit more than a little bit less, because if you make a little extra, you can throw them in rice or something else that you make.  Just don't use them to make another batch of hummus.  You can, in theory, but you shouldn't.  It just isn't the way. :)

Magic water.
  • In my pressure cooker, it needs about 10 minutes to reach full heat, 10 minutes of cooking time, and 10 minutes of "cool down" once I turn of the flame.  For you it may be different, it's ok to over-cook it a little, but don't under-cook it.  If the chickpeas aren't soft and edible when done, keep cooking.
  • In a pot, well, I'm not really sure how long it takes to cook, because I'm fortunate enough to use a pressure-cooker.  You can add a half-spoon of baking soda to the boiling water, it speeds up the cooking time for some reason.  I'm guessing that it'll take about an hour for the chickpeas to get soft.  If this is your first time doing this, give yourself an hour and a half. 
Some like it coarse.
While the chickpeas are cooking,  you can go ahead and make some tahini sauce.  I have a little post that shows you how. (You are welcome!)  Tahini sauce not only goes well with hummus, but you can eat it with salad, or as spread on toast!  So, yea... Now is a good time to make some. :)

Now that the chickpeas are nice and soft, do not pour out the water that they cooked in.  It is magical water, that will give that extra-kick to your hummus.  Get a potato-masher, and mush-mush-mush the chickpeas with some of that magic water until you get a nice consistency.  What's a nice consistency?  Good question!  I would say that it depends on taste.  I, personally, like it coarse, and not too runny.  But better too runny than too dry. Keep in mind that mushing the chickpeas will make it absorb some of that magic water, so add water slowly, and to taste.  Now its time to add some flavor to your hummus.  Start by pouring in some 100% pure sesame paste (a.k.a. tahini.)  Stir it in with a spoon, as it takes a little time for the tahini to mix in well.  Add as much as you like.  I recommend to start with about a big spoonful per serving.  Add a dash of salt (just a little goes a long way.) Some minced garlic, and squeeze in a little bit of lemon.  I sometimes put in a little olive oil as well, but I think I do it more out of superstition than anything else. :)
A little tahini paste

Now the main part of hummus is complete!  Serve it as soon as it's ready, because it's order of magnitude better warm.  Plus, if it sits out long, it tends to dry up.  Best remedy for that is to mix in some re-heated (in the microwave) magic water.

The way I like to serve it is by spreading some on the bottom and sides of a semi-deep bowl.  Then, in the middle, put a nice spoonful of tahini-sauce, squirt some lemon and olive oil around the tahini, and garnish with some left-over whole chickpeas.  Finally, color it with a little za'atar or sumac (or both.)
The classic way to serve it is with some pita bread, a hard-boiled egg, and a nice, fresh vegetable salad (that can be dressed with either tahini sauce or lemon & olive oil.)  Pickles and green olives are great sides as well.  Usually, the pita bread is used to scoop some hummus, but large pieces of onion peels can be used as well (not for the faint of heart...)

Bon Apetit!

Let's Make Tahini

Tahini paste
Tahini sauce
Tahini sauce is one of those greatly underestimated things.  It's super-easy to make, healthy, and amazingly delicious (although I must admit that it may be somewhat of an acquired taste.)  It's great as a spread on toast, as a salad dressing, and of course, it goes amazing well with hummus! So, how does one make this mysterious tahini?  Well, the journey starts at buying some, well, how shall I put this?  Tahini? :)  I'm not kidding, to make tahini sauce, one needs to buy tahini paste, which is basically 100% pure sesame seeds, mushed together into a fairly thick paste.  Don't ask me where you can buy this, but I'm sure if you look around a bit, you will find it.  The good news about tahini paste is that it doesn't require refrigeration, and keeps for who-knows how long.  So it's easy to always have around.

Once you have the paste, making the sauce is easy!  All you need is:
Good stuff

  • some water
  • lemon
  • garlic
  • parsley
  • a tad of salt.

Take some tahini paste, put it in a little bowl that you can mix it with stuff.  Make as much or as little as you want, keeping in mind that the tahini sauce keeps in the refrigerator for about a week.  Gradually add some water, you will need about 50% as much water as paste.  You may need some more, you may need some less.  So I recommend adding the water gradually.  When you add the water, you'll notice that the tahini paste changes color!  It actually becomes whiter!  Neat, huh?  You'll want to get it to have a consistency similar to cake batter.  That is, somewhat runny but not too watery.
Good consistency.
After that, add as much lemon as you like (and I like!), a dash of salt, a good amount of minced parsley, and of course some minced garlic.  You really can't put too much parsley in tahini sauce. As for the garlic, I like putting in one or two cloves for the amount you see pictured.  I also like putting in between half and a whole lemon.  Tada!! You have now made awesome tahini sauce!