Python, Snowboarding, and time

It’s mid-snowboard season here in North America, so time for learning, researching and writing is scarce.  I wanted to post to keep the blog alive and to give a shout to a friend who’s starting up a business selling some cool python swag.

I’m not in marketing so I won’t even pretend to give a pitch, other than to say I bought a mug (from coffee import *) and love it!

If you’re looking for something unique to give to a python geek (like yourself!) check it out:

https://nerdlettering.com/collections

OK.  I promise this blog won’t turn into a sales platform AND I promise to be back in the spring with more learning in python and django.  But for now, let it SNOW!!!!

Django Debug Toolbar

Django debug toolbar is a nifty little utility to allow you to examine what’s going on under the hood.  It’s a fairly easy install and gives quite a lot of info.

Installation

I’m not going to waste your time (or mine) with details of how to install the debug toolbar.  The instructions are here.

I will, however, point out that the “tips” page starts with “The toolbar isn’t displayed!“, which helped me get running.  My problem was a lack of <body> </body> tags on my template.  (side note: I’m wondering if something like bootstrap would provide those surrounding tags automatically.)

Using The Toolbar

The use of the toolbar is pretty obvious.  The information is pretty clearly laid out on each of the sections.

The section I found the most interesting was the SQL tab (shown below), which not only shows which queries were done for the given page, but also how long each took.

The page I instrumented has a task which updates several fields in the database the first time it is loaded on any given date.  Using this tab it was clear how much of the page load time was taken up in this update process.

Not only would this be handy for performance troubleshooting, but it’s also instructional to see which python statements turn into queries and how.

Conclusion

As a fan of development tools, Django Debug Toolbar certainly makes me happy not only for its features, but also its simplicity in use and design.  I would definitely recommend it

Extending Templates

Recently at work, I had a quite long code review from a very young, very rushed co-worker.  Let many such reviews (my own included) there were several instances of copy-paste coding, where the same block of code is copied to several locations with only slight modifications.

This was fresh in my mind when I started in on this series of tutorials (which are quite good):  Building Data Products with Python

In the middle of the first page, there is a great description of how I have been doing copy-paste coding in the KidTasks app – navigation links in the templates.

Fortunately for me, they also give a good example of how to fix this by using the extends tag to reference shared template code, similar to using a base class.

The Problem

Two of my templates end with the following four lines of code

         <br/> <a href="{% url 'task_new' %}?from={{ request.path|urlencode }}">Add New Task</a>
         <br/> <a href="{% url 'rep_task_new' %}?from={{ request.path|urlencode }}">Add New Repeating Task</a>
         <br/> <a href="{% url 'kid_new' %}?from={{ request.path|urlencode }}">Add New Kid</a>
         <br/> <a href="{% url 'today' %}">View Today's Tasks</a>

While there are currently only two instances of this, there are likely to be more in the future (or in a real app).

The Fix

Fortunately, the solution was quite simple.  First you create the base template which includes the shared code and some blocks for parts of the template content.  In my case, for such simple pages, there was really only a title block and a content block.  The base template looks like this:

<div>
   <h1>{% block title %}(no title){% endblock %}</h1>

   {% block content %}(no content){% endblock %}

   <nav>
      <div id="navbar">
         <br/> <a href="{% url 'task_new' %}?from={{ request.path|urlencode }}">Add New Task</a>
         <br/> <a href="{% url 'rep_task_new' %}?from={{ request.path|urlencode }}">Add New Repeating Task</a>
         <br/> <a href="{% url 'kid_new' %}?from={{ request.path|urlencode }}">Add New Kid</a>
         <br/> <a href="{% url 'today' %}">View Today's Tasks</a>
      </div>
   </nav>
</div>

Note that this puts the navigation links on the bottom of the page (where they were previously).  It also adds a div with an id to the nav bar which, I suspect, is going to be handy when I start diving in to CSS and making the site look a bit more professional.

Another thing to note is that this will also move the formatting of the title for these pages to the base template rather than having it in each individual template.  This is helpful for maintaining a consistent look throughout your app.

The change to the sub-templates was quite small.  First I had to move the title from inside the content block into its own block:

{% block title %} Today's Tasks {% endblock %}

…and then all I had to do was use the extends tag to reference the base template:

{% extends 'tasks/base.html' %}

NOTE: the extends tag is required to be the first line in the template.  Django will complain loudly if it is not.

One of the full sub-templates ended up looking like this:

{% extends 'tasks/base.html' %}

{% block title %} Today's Tasks {% endblock %}

{% block content %}
{% if kids %}
    {% for name, tasks in kids.items %}
        <h1>{{ name }} on {{ day }}</h1>
        {% if tasks %}
            <ul>
            {% for task in tasks %}
                <li><a href="{% url 'update' task.id %}">
                {% if task.completed %} <strike> {% endif %}
                    {{ task.name }}
                {% if task.completed %} </strike> {% endif %}
                </a></li>
            {% endfor %}
            </ul>
        {% else %}
            <p>No tasks for {{ name }}.</p>
        {% endif %}
    {% endfor %}
{% else %}
    <p>No kids are present.</p>
{% endif %}
{% endblock %}

That’s all there is to it!  Thanks to Jose A Dianes for the well-written tutorial referenced above.

Adventures in Python 3

As part of my playing with PyBuilder (mentioned in the last post), I decided to convert one of my old tools to use it and to convert to python3 at the same time.  While this was probably foolish (making two changes at once) it was educational in the long run.  I thought I’d share my learnings here.

Simple Changes

The first and most prevalent change was the change of print from a keyword to a function.  Trivial to fix and not worth discussing.

The urllib2 library was modified and required changes to the imports of urlopen and URLError.  The queue import was changed to Queue, but otherwise worked as I was using it.

There were a couple of places where mocks in the unit tests needed to be modified.  Looking at the changes I had to make, which were largely making the mocks specific to the module under test, I’m not sure how they worked in the old code.  The new way certainly looks more ‘correct’.

The final simple change was one I didn’t expect.  Python 3 changed how comparison operators handle None as described here.  I’m sure I’m not the only person that had written code assuming None is always less than any string.

String vs buffer

One change that took me a while to figure out was the buffer interface versus string.  I had read about this and thought I understood it, but I was stumped by why this line was failing.

title = title.replace("'", "''")

It probably shouldn’t have taken that long, but I’ll admit it took me a while to see that the ‘string’, title, was being returned from the feedparser library and was actually not a string at all.  Adding a decode to this to make it a string worked just fine.

title = title.decode('utf8').replace("'", "''")

Note that I’m making a big assumption that the title will actually be encoded in utf8.  For the purposes of this program, that assumption is true, but it’s not in general.  YMMV.

Argparse

This change caught me by surprise as well.  This is the only one of the issues I ran into that wasn’t listed in the excellent “What’s New in Python 3.0” article.

The argparse module changed behavior between py2 and py3.  This looks like a bug, as shown here.

Apparently subparsers do not have the required property set by default.  The work-around, once you find it, is trivial, after adding a new subparser:

subparsers = parser.add_subparsers(help='commands', dest='command')

you need to set its required property to True:

subparsers.required = True

Next post (which might be a couple of weeks) will look at changes to the project to bring it into pybuilder and to have it take advantage of things I’ve learned since I originally wrote it (requirements.txt, etc).

Until then, happy holidays!

Build Tools

I’m doing a survey for build tools in python.  Now, before any of you say “make”, be warned that I’ve been doing C and C++ programming for decades and even those project don’t even use make anymore if they can help it.  There are way better tools than make, especially in python!

Paver

I used paver on a previous project and thought it was OK but fairly verbose.  It was pretty easy to extend it and get it to do exactly what I wanted, however, so it worked just fine for that project. It doesn’t look like there’s too much activity on the project in github currently, but that may just mean it’s stable.

Scons, Waf

I’ve looked into these in previous lives as a build tool for C++ projects.  They both seem a little heavy for what I’m doing here (but I’d entertain arguments to the contrary).  I’m skipping them for now as “too complex”.  Again, this might not be a fair assessment.

Doit

This looked promising.  I like design and the documentation was pretty good.  It looks to be under active development.  The reason this isn’t my front runner at this point is, oddly enough, its name.  “Doit” is a french verb, making it cumbersome to do google searches on.  This is just a little extra friction, I’ll admit.  Given a wildly superior tool, I’d definitely put up with it, but this didn’t stand out that much.

Pybuilder

This is the one I’m exploring for now.  It has an active develpment community with quick response times to issues.  The documentation is OK, but certainly could use some work.  It is not at all verbose.  Actually, I’d say it errs on the side of being “too magic” where it’s hard to understand the “how” of what’s going on.  But, it seems to do the things I need AND it has a project to do a django plugin.

I haven’t tested the plugin out as of yet, but it looks like it could be a good building block to get a good development system in place with unit tests, coverage reports (so you can play the ‘coverage game’) and some good packaging.

If you’ve got a favorite I skipped or think I missed some important feature of one of the above tools, please let me know in the comments section!

“Get back to where you once belonged…”

I recently ran into the situation where I wanted to be able to return from a form to the previous view.  There was not a way to add kids to the system (short of using the admin portal) and there were two obvious places that should have the ability to do that.  Of course, after the kid is added, you want to take the user back to where they were.

After a bit of searching I ended up on this SO page which has a good description of how to make this work.  I’m writing this up here as some of the examples are a bit terse and it seemed like a more detailed example might be useful.

The Idea

The general idea is that all links and referrers to the new form view will contains a GET parameter containing a link back to the referring page.  This parameter needs to get passed through the GET (or non-POST) portion of the view to be part of the form.  Then, when the user POSTs the form back to the view, it is returned there and used by the POST portion of the view to redirect to the starting point.  Got it?  Let’s walk through in code.

The Code

First, let’s start with the referring code.  I’m using simple hrefs in the referring page to bring up this form, so it’s pretty easy.

The link in the referring templates change from

<br/> <a href="{% url 'kid_new' %}">Add New Kid</a>

to

<br/> <a href="{% url 'kid_new' %}?from=
       {{ request.path|urlencode }}">Add New Kid</a>

The added “?from” portion is what builds the return info.

Moving on to the view, we’ll skip the POST portion and see what the GET portion looks like:

def new_kid(request):
    """ Create a new kid!  Sounds more exciting than it is. """
    if request.method == "POST":
        # skipped
    else:
        form = KidForm()
    return render(request, 'tasks/kid_edit.html',
                  {'form': form, 
                   'from': request.GET.get('from', None)})

You can see that it still build the same form, but passes in the ‘from’ field in the context to the template.

The template (kid_edit.html in this case) adds that to the form itself (this is the part that confused me at first).  So the form went from looking like:

    <form method="POST" class="kid-form">
        skipped...
    </form>

to:

    <form method="POST" class="kid-form"
       action="{% if from %}?next={{ from }}{% endif %}" >
         same stuff skipped ...
    </form>

The action field is what is going to pass the from field back to the view.

Finally, let’s go back and look at the rest of the view.  Here’s the entire function:

def new_kid(request):
    """ Create a new kid!  Sounds more exciting than it is. """
    if request.method == "POST":
        form = KidForm(request.POST)
        if form.is_valid():
            form.save()
            next = request.GET.get('next', None)
            if next:
                return redirect(next)
            # if for some reason it was not set, default to today
            return redirect(reverse('today'))
    else:
        form = KidForm()
    return render(request, 'tasks/kid_edit.html',
                  {'form': form, 'from': request.GET.get('from', None)})

You can see in the POST section that we do the normal form validation, then get the next field out of the POSTed form.  If it exists, we go to the referring page.  In this case, just for safety, if it does not exist we still go back to one of our two main pages.  You could also raise an error here if that was more appropriate.

The form portion of this code is exactly like any other form you’ve used.  Nothing special needs to be added there.

That’s all it takes!

Custom Filters and Ordered Dicts

This week I ran into a minor problem that took a surprising amount of time to resolve. Getting a Django template to produce a dict in sorted order.  While there were answers out there, none seemed to match the environment that I am using (python 3, Django 1.10).  After some experimentation, I finally came up with what I think is a good solution.  Feel free to tell me a better one if you know it.

The Problem

The basic problem was that I have a template which walks through a dict and displays it in a particular format.  As we all know, python dicts are not ordered, so the display would flip around between invocations of the server.  Not a huge deal, but annoying.  The template code looked like:

        {% for kid, days in kids.items %}
            <h1>{{ kid }}</h1>
             ... deleted stuff here ...
        {% endfor %}

StackOverflow to the rescue!

There was a quite good answer on SO which pointed me in the right direction, but it unfortunately did not work for my situation (again, I suspect python 2 vs 3 differences).

After doing some digging, I figured out that invoking the filter with dict.items()

{% for kid, days in kids.items|sort %}

was not resulting in the parameter as a dict, but rather as a ItemsView which is a ‘view’ into a dict.

I played some with using

{% for kid, days in kids|sort %}

which did result in a dict getting passed into the filter, but returning the OrderedDict resulted in some strange iterable being returned (the number of items to unpack changed on each iteration, so I suspect that it was flattening my structure somehow, but I did not pursue that).

I decided to embrace the original form and move forward with the ItemsView object that was getting passed in.  This turned out to be much shorter and easier code than the original, and so I went with that.

My Solution

Here’s where I ended up.  The filter looks like this:

from django import template
from collections import ItemsView

register = template.Library()


@register.filter
def sort(value):
    if isinstance(value, ItemsView) or isinstance(value, list):
        return sorted(value)
    else:
        return value

Note that since ItemsView is iterable and I want an iterable result, I can
just used sorted() on it as we did with lists.

To invoke this we call it with:

{% for kid, days in kids.items|sort %}

in the template.

For completeness, here’s the answer I posted on that SO question.

Why kwargs?

I’ve noticed that several of the Django functions, especially filter, take a kwargs-style set of arguments.  I casually wondered about this but didn’t give it much thought until now.  I ran into a very practical use for it, that should have been obvious, but wasn’t to me.

The Setup

In my recent redesign of the KidTasks project, I opted to take the easy route for tasks which repeat on particular weekdays and simply put a BooleanField for each day of the week into the model.  This makes forms a bit easier but does cause some complications when trying to use a filter function to get a list of weekday-task pairs.  I started with the naive (and un-good) method of filtering each weekday independently:

# note: syntax on the filter here is from memory - might be incorrect!
qs = RepeatingTask.objects.filter(kid=self).filter('monday'=True)
tasks.append((day, [task for task in qs]))
qs = RepeatingTask.objects.filter(kid=self).filter('tuesday'=True)
tasks.append((day, [task for task in qs]))
    [etc]

While that will work, I’m sure you’ll agree that it’s ugly.

Kwargs to the rescue!

It turns out that the solution to this problem lies with those mysterious kwargs function signatures.  It turns out that filter takes one. This allows you to create a dict which you can create on the fly.  Which, in turn, allows you to use a variable to define the field on which you’re filtering!

days = [
    'monday',
    'tuesday',
    'wednesday',
    'thursday',
    'friday',
    'saturday',
    'sunday'
]

...

for day in self.days:
    qs = RepeatingTask.objects.filter(kid=self).filter(**{ day : True })
    tasks.append((day, [task for task in qs]))

Much nicer.  While this is likely obvious to most seasoned Django users, I thought it tied things together nicely.

KidsTasks: Another redesign!

My work on the KidsTasks app has hit a bit of a roadblock.  I’ve been looking at adding forms for the users (i.e. mom) to assign new tasks to kids. As I’ve worked, I’ve realized that I made two important mistakes when starting this project.  The first was unavoidable; not taking the structure of how Django works into account when creating the design.

The second mistake was more avoidable.  True to my education and experience, I focused the original design on the internal data structures of the problem rather than the user interface.

The two mistakes together brought me to the point where, while it’s possible to create the UI I want on the models I currently have, it’s much more cumbersome than I feel it should be.  Therefore it’s time to re-evaluate the design.

User Scenarios

Most of my experience comes from embedded systems where the user interface is trivial if not completely absent.   Focusing on the UI earlier is not something in my normal skill set.  That said, let’s dive in.

I envision two primary user scenarios:

  • After school, the parents sit down with the kids and evaluate what homework and other tasks need to be accomplished.  These get added to the list for the day.  There should be recurring tasks that occur on given days every week and ad-hoc tasks for that day only.
  • The kids, when they have completed a task, can go to the app and indicate that it is completed, having its status changed. (changing color from red to green, strike-through, etc)

User Interface

To satisfy the first user scenario, I envision a form that looks similar to the the following:

Task Name: My New Task

Which Kid:
x kid1     _ kid2

Repeat Every
x Mon     _ Tue    _ Wed   x Thu   x Fri  _ Sat    _ Sun

<save and add another>  <save> <exit>

For the second user scenario, I’m envisioning a horizontal row of colored boxes with the name of each task inside.  Each row would correspond to a particular kid.  Clicking on a box would change its state (from red to green to indicate it was completed).

           ___________________________________
    Kid1:  | piano | sleep | chew gum | walk |
           ___________________________________
    
           ___________________________________
    Kid2:  | piano | sleep | spelling | talk |
           ___________________________________

New Models

Considering all this, I threw the old design out and started with a new set of models.  This actually made things much simpler and fits into the Django world view a bit better.

The Kid model changed to just a name.  I was mistaken in my earlier thinking that the Kid needed to “own” the set of tasks.  Once I saw the database modeling underneath the hood, it made sense that I merely needed to create a table of tasks which had a foreign key to the Kid, allowing me to do a simple query to get all the tasks for a given Kid.

I still ended up with two different types of tasks; the first is a RepeatingTask which is in reality a template for a Task.  It stores the idea of a task that is repeated on various days of the week.  Note that I did some searches on the best way to store days of week in a Django model.  There are clearly more efficient ways to store this, but using separate boolean fields makes creating forms for that info much easier.  I chose to stick with “easy” over “efficient” for this project.

The second task is an actual task assigned to a Kid on a particular date.  This is, thankfully, fairly straightforward.

Here’s the code:

#tasks/models.py
""" Define the data models for the KidsTasks app """
import datetime
from django.db import models


class Kid(models.Model):
    """ Defines the kids which have to do the tasks. """
    name = models.CharField(max_length=256)

    def __str__(self):
        return self.name

    class Meta:
        ordering = ['name', ]


class Task(models.Model):
    """ A Task is associated with a kid and a date.  This is the actual thing
    the kid has to do! """
    name = models.CharField(max_length=256)
    completed = models.BooleanField()
    date = models.DateField(default=datetime.datetime.now)
    kid = models.ForeignKey(Kid)

    def __str__(self):
        return self.name

    class Meta:
        ordering = ['name', ]


class RepeatingTask(models.Model):
    """ Defines a repeating task """
    name = models.CharField(max_length=256)
    kid = models.ForeignKey(Kid)  # NOTE: RepeatingTasks are kid specific
    monday = models.BooleanField(default=False)
    tuesday = models.BooleanField(default=False)
    wednesday = models.BooleanField(default=False)
    thursday = models.BooleanField(default=False)
    friday = models.BooleanField(default=False)
    saturday = models.BooleanField(default=False)
    sunday = models.BooleanField(default=False)

    def __str__(self):
        return "{0}:{1}".format(self.kid.name, self.name)

    class Meta:
        ordering = ['kid', 'name', ]

That’s where I ended this time.  Hopefully I’ll get back on track next time in creating views and forms for this app.

If you’re interested, this code will be stored in the blog/04-Redesign branch.

thanks for reading!

jima

KidTasks: Simple Class Based Views

For this post I’ve taken some advice from a python talk I watched a few weeks ago and converted some of my function-based-views (FBV) to class-based-views (CBV).  I found this simple and very cool for my simplest view and a bit more complex for another view.  I’ll look at each of these in turn.

Index View

The first view I changed was the simplest.  It merely took a single data type and sent it to a template.

The FBV version of this looked like:

def index(request):
    """ Shows all of the tasks in the system, broken down by kid """
    kids = get_list_or_404(Kid)
    return render(request, 'tasks/index.html', {'kids': kids})

It’s not much of a surprise to see that a simple FBV turns into an even simpler CBV.  Here’s how that ended up:

class FullList(ListView):
    """ Shows all of the tasks in the system, broken down by kid """
    model = Kid

Note that this change required that I move the template “index.html” to kid_list.html.  I also could have specified a template_name in the class, but this seemed like a good time to clean it up.

Today view

The transformation for this view was less pretty.  After a bunch of messing around and reading different examples, I came up with the solution below.  In general it seems the ListView is fantastic at representing a specific model as it exists in the database.  My issue was the need to update the “today” list if needed to match the tasks for the current day and then return only that.

I’ll show what the change looked like and hold off my concerns and ideas until afterward.

Here’s where we started:

def today(request):
    """ Generate the 'today' page showing which tasks are due today """
    day_name = datetime.datetime.now().strftime("%A")
    kid_list = get_list_or_404(Kid)

    kids = dict()
    for kid in kid_list:
        tasks = build_today(kid)
        if tasks:
            kids[kid.name] = tasks
    return render(request, 'tasks/today.html', {'kids': kids, 'day': day_name})

From which we ended up here:

class TodayList(ListView):
    """ Generate the 'today' page showing which tasks are due today """
    model = Kid
    template_name = 'tasks/today.html'

    def get_context_data(self, **kwargs):
        # get_context_data creates the context
        context = ListView.get_context_data(self, **kwargs)

        day_name = datetime.datetime.now().strftime("%A")
        kid_list = get_list_or_404(Kid.objects.order_by('name'))

        kids = {kid.name: kid.build_today() for kid in kid_list}
        context.update({'kids': kids, 'day': day_name})

        return context

It’s about the same amount of code, but somehow feels less ‘obvious’.  It relies on hooking the CBV pretty late in the processing and substituting the newly created kids dict into the context.  The template for this ONLY uses that part of the context which means that technically I could forego the call to the superclass (I think) and just return the kids dict.

I have two thoughts on how this could be done better.  The first is to split the work here.  I could hook the processing earlier to ensure that the today task is updated and stored in the kid and then modify the template to take the full list of kids, running through them to display the today list for each.  If I were to stay with this overall design, I think I’d go this direction.

However, my other thought is to re-design the models (again) to more closely match what’s happening here.  As I write this, I am also working on adding FormViews to allow easier addition of tasks for a given day.  That will likely result in a model redesign, so I suspect this part will change anyway.

The code for this point in the project is stored in branch blog/04-SimpleCBV.

Thanks for following along!