Adventures in Python 3

As part of my playing with PyBuilder (mentioned in the last post), I decided to convert one of my old tools to use it and to convert to python3 at the same time.  While this was probably foolish (making two changes at once) it was educational in the long run.  I thought I’d share my learnings here.

Simple Changes

The first and most prevalent change was the change of print from a keyword to a function.  Trivial to fix and not worth discussing.

The urllib2 library was modified and required changes to the imports of urlopen and URLError.  The queue import was changed to Queue, but otherwise worked as I was using it.

There were a couple of places where mocks in the unit tests needed to be modified.  Looking at the changes I had to make, which were largely making the mocks specific to the module under test, I’m not sure how they worked in the old code.  The new way certainly looks more ‘correct’.

The final simple change was one I didn’t expect.  Python 3 changed how comparison operators handle None as described here.  I’m sure I’m not the only person that had written code assuming None is always less than any string.

String vs buffer

One change that took me a while to figure out was the buffer interface versus string.  I had read about this and thought I understood it, but I was stumped by why this line was failing.

title = title.replace("'", "''")

It probably shouldn’t have taken that long, but I’ll admit it took me a while to see that the ‘string’, title, was being returned from the feedparser library and was actually not a string at all.  Adding a decode to this to make it a string worked just fine.

title = title.decode('utf8').replace("'", "''")

Note that I’m making a big assumption that the title will actually be encoded in utf8.  For the purposes of this program, that assumption is true, but it’s not in general.  YMMV.


This change caught me by surprise as well.  This is the only one of the issues I ran into that wasn’t listed in the excellent “What’s New in Python 3.0” article.

The argparse module changed behavior between py2 and py3.  This looks like a bug, as shown here.

Apparently subparsers do not have the required property set by default.  The work-around, once you find it, is trivial, after adding a new subparser:

subparsers = parser.add_subparsers(help='commands', dest='command')

you need to set its required property to True:

subparsers.required = True

Next post (which might be a couple of weeks) will look at changes to the project to bring it into pybuilder and to have it take advantage of things I’ve learned since I originally wrote it (requirements.txt, etc).

Until then, happy holidays!

Build Tools

I’m doing a survey for build tools in python.  Now, before any of you say “make”, be warned that I’ve been doing C and C++ programming for decades and even those project don’t even use make anymore if they can help it.  There are way better tools than make, especially in python!


I used paver on a previous project and thought it was OK but fairly verbose.  It was pretty easy to extend it and get it to do exactly what I wanted, however, so it worked just fine for that project. It doesn’t look like there’s too much activity on the project in github currently, but that may just mean it’s stable.

Scons, Waf

I’ve looked into these in previous lives as a build tool for C++ projects.  They both seem a little heavy for what I’m doing here (but I’d entertain arguments to the contrary).  I’m skipping them for now as “too complex”.  Again, this might not be a fair assessment.


This looked promising.  I like design and the documentation was pretty good.  It looks to be under active development.  The reason this isn’t my front runner at this point is, oddly enough, its name.  “Doit” is a french verb, making it cumbersome to do google searches on.  This is just a little extra friction, I’ll admit.  Given a wildly superior tool, I’d definitely put up with it, but this didn’t stand out that much.


This is the one I’m exploring for now.  It has an active develpment community with quick response times to issues.  The documentation is OK, but certainly could use some work.  It is not at all verbose.  Actually, I’d say it errs on the side of being “too magic” where it’s hard to understand the “how” of what’s going on.  But, it seems to do the things I need AND it has a project to do a django plugin.

I haven’t tested the plugin out as of yet, but it looks like it could be a good building block to get a good development system in place with unit tests, coverage reports (so you can play the ‘coverage game’) and some good packaging.

If you’ve got a favorite I skipped or think I missed some important feature of one of the above tools, please let me know in the comments section!

Exploring with Cookiecutter

After reading Two Scoops of Django, I decided to explore the cookiecutter app that they recommend.  If you haven’t used it, cookiecutter is ” A command-line utility that creates projects from cookiecutters (project templates). E.g. Python package projects, jQuery plugin projects.”

In particular I explored some of the Django-specific cookiecutter templates (though I did peek at a few other languages, too).  I started with the cookiecutter-django template as it was written by one of the authors of the book and it was at the top of the list I was looking at.


Cookiecutter-Django is a full-on professional level template for producing enterprise-level Django apps.  As such, it comes with a LOT of bells and whistles.  Celery support?  Yep.  Docker? Yep.  Custom windows and pycharm setups?  Yep.  The list goes on.  This is pretty cool, but, at 134 files, many of which I do not know the use of, this is clearly not aimed at me, the hobbyist learning Django in his spare time.

It does, however, have some interesting project layout ideas that I think ARE useful and sound.  Among these are having separate doc and requirements directories and having the requirements files broken into separate environments (local, production, test) to codify and simplify the set up for the job you’re doing.

Lots of good stuff here to dig into in the future, but this is a bit over my head at the moment. Fortunately, there are a myriad of other templates to play with.  I found a good list of them at readthedocs.


The next one I ventured into was django-crud.  This is different from the others I looked at in that it was not a full django project, rather just the webapp portion of it.

It is pretty cool in that it gives a really detailed example of how to write a simple model (performing the CRUD database operations, of course) with full unit tests and simple templates.  At 16 files, it’s not so overwhelming and has a pretty clear intent.  I’m sure the author uses it as a starting point for webapps, but I’m finding it as useful learning tool and will likely be the basis as I start down the unit testing road (coming soon!).


I also took a quick tour through Django-paas. which is billed as a “Django template ready to use in SAAS platforms like Heroku, OpenShift, etc..”.

While this isn’t likely one I’m ready to use just yet (I’m still hosting locally on my private machine for use inside my network).  It’s definitely got a few features that are worth looking at.  Of note was the use of waitress (instead of gnuicorn) which might be an interesting alternative.  It also provided a procfile for heroku, which I can see being handy in the future.

Simple Django

My final stop of this tour was Simple-Django.  This is a very trimmed down version of the full cookiecutter-django template I started with.  It still has separate requirements and docs directories, but does not pull in nearly the number of third-party packages that its full-blown antecedent does.

This looks like a good start place for my level of development and interest.  It will likely be the basis I use if and when I create a django template for my own use.


While none of these cookie-cutter templates are exactly what I’m looking for at the moment, I found exploring them a good way to learn about different tools, directory layouts and general best practices in the django world.  I’d encourage you to explore some of these, even if they’re not something you need at the moment.

I’d like to thank the authors of each of these templates for the work they put into them and for sharing them with us, and, in particular, Audrey Roy Greenfeld and Danny Roy Greenfeld for creating cookiecutter and the cookiecutter-django template.


“Get back to where you once belonged…”

I recently ran into the situation where I wanted to be able to return from a form to the previous view.  There was not a way to add kids to the system (short of using the admin portal) and there were two obvious places that should have the ability to do that.  Of course, after the kid is added, you want to take the user back to where they were.

After a bit of searching I ended up on this SO page which has a good description of how to make this work.  I’m writing this up here as some of the examples are a bit terse and it seemed like a more detailed example might be useful.

The Idea

The general idea is that all links and referrers to the new form view will contains a GET parameter containing a link back to the referring page.  This parameter needs to get passed through the GET (or non-POST) portion of the view to be part of the form.  Then, when the user POSTs the form back to the view, it is returned there and used by the POST portion of the view to redirect to the starting point.  Got it?  Let’s walk through in code.

The Code

First, let’s start with the referring code.  I’m using simple hrefs in the referring page to bring up this form, so it’s pretty easy.

The link in the referring templates change from

<br/> <a href="{% url 'kid_new' %}">Add New Kid</a>


<br/> <a href="{% url 'kid_new' %}?from=
       {{ request.path|urlencode }}">Add New Kid</a>

The added “?from” portion is what builds the return info.

Moving on to the view, we’ll skip the POST portion and see what the GET portion looks like:

def new_kid(request):
    """ Create a new kid!  Sounds more exciting than it is. """
    if request.method == "POST":
        # skipped
        form = KidForm()
    return render(request, 'tasks/kid_edit.html',
                  {'form': form, 
                   'from': request.GET.get('from', None)})

You can see that it still build the same form, but passes in the ‘from’ field in the context to the template.

The template (kid_edit.html in this case) adds that to the form itself (this is the part that confused me at first).  So the form went from looking like:

    <form method="POST" class="kid-form">


    <form method="POST" class="kid-form"
       action="{% if from %}?next={{ from }}{% endif %}" >
         same stuff skipped ...

The action field is what is going to pass the from field back to the view.

Finally, let’s go back and look at the rest of the view.  Here’s the entire function:

def new_kid(request):
    """ Create a new kid!  Sounds more exciting than it is. """
    if request.method == "POST":
        form = KidForm(request.POST)
        if form.is_valid():
            next = request.GET.get('next', None)
            if next:
                return redirect(next)
            # if for some reason it was not set, default to today
            return redirect(reverse('today'))
        form = KidForm()
    return render(request, 'tasks/kid_edit.html',
                  {'form': form, 'from': request.GET.get('from', None)})

You can see in the POST section that we do the normal form validation, then get the next field out of the POSTed form.  If it exists, we go to the referring page.  In this case, just for safety, if it does not exist we still go back to one of our two main pages.  You could also raise an error here if that was more appropriate.

The form portion of this code is exactly like any other form you’ve used.  Nothing special needs to be added there.

That’s all it takes!

Custom Filters and Ordered Dicts

This week I ran into a minor problem that took a surprising amount of time to resolve. Getting a Django template to produce a dict in sorted order.  While there were answers out there, none seemed to match the environment that I am using (python 3, Django 1.10).  After some experimentation, I finally came up with what I think is a good solution.  Feel free to tell me a better one if you know it.

The Problem

The basic problem was that I have a template which walks through a dict and displays it in a particular format.  As we all know, python dicts are not ordered, so the display would flip around between invocations of the server.  Not a huge deal, but annoying.  The template code looked like:

        {% for kid, days in kids.items %}
            <h1>{{ kid }}</h1>
             ... deleted stuff here ...
        {% endfor %}

StackOverflow to the rescue!

There was a quite good answer on SO which pointed me in the right direction, but it unfortunately did not work for my situation (again, I suspect python 2 vs 3 differences).

After doing some digging, I figured out that invoking the filter with dict.items()

{% for kid, days in kids.items|sort %}

was not resulting in the parameter as a dict, but rather as a ItemsView which is a ‘view’ into a dict.

I played some with using

{% for kid, days in kids|sort %}

which did result in a dict getting passed into the filter, but returning the OrderedDict resulted in some strange iterable being returned (the number of items to unpack changed on each iteration, so I suspect that it was flattening my structure somehow, but I did not pursue that).

I decided to embrace the original form and move forward with the ItemsView object that was getting passed in.  This turned out to be much shorter and easier code than the original, and so I went with that.

My Solution

Here’s where I ended up.  The filter looks like this:

from django import template
from collections import ItemsView

register = template.Library()

def sort(value):
    if isinstance(value, ItemsView) or isinstance(value, list):
        return sorted(value)
        return value

Note that since ItemsView is iterable and I want an iterable result, I can
just used sorted() on it as we did with lists.

To invoke this we call it with:

{% for kid, days in kids.items|sort %}

in the template.

For completeness, here’s the answer I posted on that SO question.

Why kwargs?

I’ve noticed that several of the Django functions, especially filter, take a kwargs-style set of arguments.  I casually wondered about this but didn’t give it much thought until now.  I ran into a very practical use for it, that should have been obvious, but wasn’t to me.

The Setup

In my recent redesign of the KidTasks project, I opted to take the easy route for tasks which repeat on particular weekdays and simply put a BooleanField for each day of the week into the model.  This makes forms a bit easier but does cause some complications when trying to use a filter function to get a list of weekday-task pairs.  I started with the naive (and un-good) method of filtering each weekday independently:

# note: syntax on the filter here is from memory - might be incorrect!
qs = RepeatingTask.objects.filter(kid=self).filter('monday'=True)
tasks.append((day, [task for task in qs]))
qs = RepeatingTask.objects.filter(kid=self).filter('tuesday'=True)
tasks.append((day, [task for task in qs]))

While that will work, I’m sure you’ll agree that it’s ugly.

Kwargs to the rescue!

It turns out that the solution to this problem lies with those mysterious kwargs function signatures.  It turns out that filter takes one. This allows you to create a dict which you can create on the fly.  Which, in turn, allows you to use a variable to define the field on which you’re filtering!

days = [


for day in self.days:
    qs = RepeatingTask.objects.filter(kid=self).filter(**{ day : True })
    tasks.append((day, [task for task in qs]))

Much nicer.  While this is likely obvious to most seasoned Django users, I thought it tied things together nicely.

KidsTasks: Another redesign!

My work on the KidsTasks app has hit a bit of a roadblock.  I’ve been looking at adding forms for the users (i.e. mom) to assign new tasks to kids. As I’ve worked, I’ve realized that I made two important mistakes when starting this project.  The first was unavoidable; not taking the structure of how Django works into account when creating the design.

The second mistake was more avoidable.  True to my education and experience, I focused the original design on the internal data structures of the problem rather than the user interface.

The two mistakes together brought me to the point where, while it’s possible to create the UI I want on the models I currently have, it’s much more cumbersome than I feel it should be.  Therefore it’s time to re-evaluate the design.

User Scenarios

Most of my experience comes from embedded systems where the user interface is trivial if not completely absent.   Focusing on the UI earlier is not something in my normal skill set.  That said, let’s dive in.

I envision two primary user scenarios:

  • After school, the parents sit down with the kids and evaluate what homework and other tasks need to be accomplished.  These get added to the list for the day.  There should be recurring tasks that occur on given days every week and ad-hoc tasks for that day only.
  • The kids, when they have completed a task, can go to the app and indicate that it is completed, having its status changed. (changing color from red to green, strike-through, etc)

User Interface

To satisfy the first user scenario, I envision a form that looks similar to the the following:

Task Name: My New Task

Which Kid:
x kid1     _ kid2

Repeat Every
x Mon     _ Tue    _ Wed   x Thu   x Fri  _ Sat    _ Sun

<save and add another>  <save> <exit>

For the second user scenario, I’m envisioning a horizontal row of colored boxes with the name of each task inside.  Each row would correspond to a particular kid.  Clicking on a box would change its state (from red to green to indicate it was completed).

    Kid1:  | piano | sleep | chew gum | walk |
    Kid2:  | piano | sleep | spelling | talk |

New Models

Considering all this, I threw the old design out and started with a new set of models.  This actually made things much simpler and fits into the Django world view a bit better.

The Kid model changed to just a name.  I was mistaken in my earlier thinking that the Kid needed to “own” the set of tasks.  Once I saw the database modeling underneath the hood, it made sense that I merely needed to create a table of tasks which had a foreign key to the Kid, allowing me to do a simple query to get all the tasks for a given Kid.

I still ended up with two different types of tasks; the first is a RepeatingTask which is in reality a template for a Task.  It stores the idea of a task that is repeated on various days of the week.  Note that I did some searches on the best way to store days of week in a Django model.  There are clearly more efficient ways to store this, but using separate boolean fields makes creating forms for that info much easier.  I chose to stick with “easy” over “efficient” for this project.

The second task is an actual task assigned to a Kid on a particular date.  This is, thankfully, fairly straightforward.

Here’s the code:

""" Define the data models for the KidsTasks app """
import datetime
from django.db import models

class Kid(models.Model):
    """ Defines the kids which have to do the tasks. """
    name = models.CharField(max_length=256)

    def __str__(self):

    class Meta:
        ordering = ['name', ]

class Task(models.Model):
    """ A Task is associated with a kid and a date.  This is the actual thing
    the kid has to do! """
    name = models.CharField(max_length=256)
    completed = models.BooleanField()
    date = models.DateField(
    kid = models.ForeignKey(Kid)

    def __str__(self):

    class Meta:
        ordering = ['name', ]

class RepeatingTask(models.Model):
    """ Defines a repeating task """
    name = models.CharField(max_length=256)
    kid = models.ForeignKey(Kid)  # NOTE: RepeatingTasks are kid specific
    monday = models.BooleanField(default=False)
    tuesday = models.BooleanField(default=False)
    wednesday = models.BooleanField(default=False)
    thursday = models.BooleanField(default=False)
    friday = models.BooleanField(default=False)
    saturday = models.BooleanField(default=False)
    sunday = models.BooleanField(default=False)

    def __str__(self):
        return "{0}:{1}".format(,

    class Meta:
        ordering = ['kid', 'name', ]

That’s where I ended this time.  Hopefully I’ll get back on track next time in creating views and forms for this app.

If you’re interested, this code will be stored in the blog/04-Redesign branch.

thanks for reading!


KidTasks: Simple Class Based Views

For this post I’ve taken some advice from a python talk I watched a few weeks ago and converted some of my function-based-views (FBV) to class-based-views (CBV).  I found this simple and very cool for my simplest view and a bit more complex for another view.  I’ll look at each of these in turn.

Index View

The first view I changed was the simplest.  It merely took a single data type and sent it to a template.

The FBV version of this looked like:

def index(request):
    """ Shows all of the tasks in the system, broken down by kid """
    kids = get_list_or_404(Kid)
    return render(request, 'tasks/index.html', {'kids': kids})

It’s not much of a surprise to see that a simple FBV turns into an even simpler CBV.  Here’s how that ended up:

class FullList(ListView):
    """ Shows all of the tasks in the system, broken down by kid """
    model = Kid

Note that this change required that I move the template “index.html” to kid_list.html.  I also could have specified a template_name in the class, but this seemed like a good time to clean it up.

Today view

The transformation for this view was less pretty.  After a bunch of messing around and reading different examples, I came up with the solution below.  In general it seems the ListView is fantastic at representing a specific model as it exists in the database.  My issue was the need to update the “today” list if needed to match the tasks for the current day and then return only that.

I’ll show what the change looked like and hold off my concerns and ideas until afterward.

Here’s where we started:

def today(request):
    """ Generate the 'today' page showing which tasks are due today """
    day_name ="%A")
    kid_list = get_list_or_404(Kid)

    kids = dict()
    for kid in kid_list:
        tasks = build_today(kid)
        if tasks:
            kids[] = tasks
    return render(request, 'tasks/today.html', {'kids': kids, 'day': day_name})

From which we ended up here:

class TodayList(ListView):
    """ Generate the 'today' page showing which tasks are due today """
    model = Kid
    template_name = 'tasks/today.html'

    def get_context_data(self, **kwargs):
        # get_context_data creates the context
        context = ListView.get_context_data(self, **kwargs)

        day_name ="%A")
        kid_list = get_list_or_404(Kid.objects.order_by('name'))

        kids = { kid.build_today() for kid in kid_list}
        context.update({'kids': kids, 'day': day_name})

        return context

It’s about the same amount of code, but somehow feels less ‘obvious’.  It relies on hooking the CBV pretty late in the processing and substituting the newly created kids dict into the context.  The template for this ONLY uses that part of the context which means that technically I could forego the call to the superclass (I think) and just return the kids dict.

I have two thoughts on how this could be done better.  The first is to split the work here.  I could hook the processing earlier to ensure that the today task is updated and stored in the kid and then modify the template to take the full list of kids, running through them to display the today list for each.  If I were to stay with this overall design, I think I’d go this direction.

However, my other thought is to re-design the models (again) to more closely match what’s happening here.  As I write this, I am also working on adding FormViews to allow easier addition of tasks for a given day.  That will likely result in a model redesign, so I suspect this part will change anyway.

The code for this point in the project is stored in branch blog/04-SimpleCBV.

Thanks for following along!

KidsTasks: Minor Redesign

I’ve been slowed down in my progress this month due to a family vacation and a change of jobs, so I’ve only made a little progress on the KidsTasks app.

This post gets the app up to the ‘almost functional’ stage where the admin portal can be used to create tasks and schedules and the normal app UI can be used to list the day’s tasks, all of the tasks, and to mark tasks complete (or incomplete).  It’s not a great idea to have the admin pages be the main data entry access, but for this early stage it will be sufficient.

All of this functionality needs some more work, but the basics are there, which is always a good time to stop and consider your design.  Especially when I’m learning a new language or technology, I find that iterative design works best for the first several projects.  Once you get your hands dirty, it’s easier to see what the next step should be.  This doesn’t often (some would say ‘rarely’) ends up with an optimal design, but it gives you more room to experiment and learn as you go.  That’s definitely what’s going to happen here, as I already have several changes in mind.

What Changed This Time

While working on the very simple views and templates, I decided that some simplification of the models was also in order.  The models went from two different types of tasks existing in to different types of schedules.  The different schedule types got removed and a new field got added to the Kid model to hold a list of DateTasks.

The basic issue driving this (which would have been obvious to many designers) was the ability to change “abstract tasks” (i.e. ‘practice piano every Tuesday’ into DateTasks (“It’s Tuesday today, get a list of all tasks that are due today and display them.”).   The quick answer to this was to create the new list of DateTasks in the Kid model.  This will store all of the tasks due today.  This is really just a stepping stone to where I want to end up, having a history of completed (and non-completed) tasks.

Once I got this change to the models in place (complete with several rounds of rebuilding the database), I moved on to getting a most basic of user interfaces in place.  To do this I hacked my way through learning the basics of accessing the Django ORM from views and templates.  While this is fairly well documented, it still took some playing with to get where I was going and get the proper syntax.

I ended up doing most of my hacking in the file and that’s where I put the main function to convert the Tasks to DateTasks.  In the subsequent week since I wrote that, I’ve read (in Two Scoops of Django) that putting logic in your views is not the best practice, so the next revision will likely fix this and put that functionality into the model.

The Templates and Views

There’s really not all that much to the new views once you see the code.  One thing that stood out for me was the simplicity of the Update view function.  Being able to do a redirect “by magic” is pretty cool and produces a nice effect in the UI.

As I’ve mentioned before, I’m not a graphic designer, but I do have intentions of creating something a little nicer than it currently stands: a simple HTML list with links to change the state of each task.


On a final note, I had to rebuild my laptop a few months ago and in the process I managed to forget to reinstall flake8!  I finally noticed this and now my vim setup is marking all the errors it found.  These got cleaned up on this round as well.

The next set of work on KidsTasks is to get a little nicer UI and to add some views for creating tasks instead of relying on the clunky admin portal.  This might or might not entail a dive into class-based-views.

For those of you keeping score at home, the code at this state is in branch: blog/03-Simple-Views-Update in this repo:

Thanks for reading!

A Few Interesting Recommendations

I want to pass on a couple of recent recommendations that have come my way.

Weekly Python Chat

Weekly Python Chat is run by Trey Hunner who does a weekly live chat covering python topics.

Trey has been doing Django-related topics recently which have been great learning opportunities for me. I especially like the annotations to the video which allow you to skip to the questions you’re particularly interested in.

This resource led me to the next one.

Two Scoops of Django

This book was brought up in one of the Weekly Python Chats. The current version is for Django 1.8, but so far much of the material has been fairly non-version specific. I’m about 5 chapters in and am not only enjoying it, but also learning some good “common practice” advice that is tough to get when you’re working on your own projects.

From this book, I’ve found the cookiecutter app to generate various types of applications boilerplate structure (proper layout of project, Readme files, etc) in various languages. This might be applicable well beyond my python/Django tasks.

This book also led me to the next topic.


virtualenvwrapper is a handy little tool which extends the functionality of virtualenv and makes it easier to use across multiple projects.

In the first of my KidsTasks posts, I walked through the project setup using virtualenv. I hope for my next project to use cookiecutter and virtualenvwrapper to simplify and “normalize” the layout of the directories.


Finally, a quick tip I pulled from one of Daniel Bader’s code reviews. I did not know that you can use a requirements.txt file to list the dependencies for your project and then use pip to install them. This, married with virtualenvwrapper makes creating virtual envs that match any given project much easier.

The syntax for the requirements.txt file is quite simple:


Running pip is, similarly, quite simple.

$ sudo pip install -r requirements.txt
 Requirement already satisfied (use --upgrade to upgrade): RtmAPI==0.6.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 1))
 Downloading/unpacking Django==1.10.0 (from -r requirements.txt (line 2))
 Downloading Django-1.10-py2.py3-none-any.whl (6.8MB): 6.8MB downloaded
 Installing collected packages: Django
 Successfully installed Django
 Cleaning up...

That’s it for recommendations. Feel free to add your cool tricks and recommendations below!