Hello, Heroku!

This week I finally deployed the KidTasks app to Heroku.  While I was very happy with Heroku and how it behaved, I’ll have to admit that many of the shortcuts I’ve taken in developing this first app have come back to bite me during this process.  None of the issues are things I can blame Heroku for.  They were all self-inflicted.

Help!

To get started, I walked through the demo application instructions on Heroku and deployed the demo app they have.  I thought this demo was really well put together and explained things quite well.  Unfortunately, there were several requirements that I hadn’t yet met.  Fortunately, most of those were easily solved.

I had only been running the app locally and using the built-in Django server.  This is great for what it is, but not recommended for an actual deployment.

It’s Easy

The first several changes were fairly trivial.  They recommend using gunicorn; I’ll use gunicorn.  pip install; pip freeze; git add requirements.txt and that’s all set.

Similarly they recommend dj_database_url for reading the database settings out of the environment.  Also easily installed and the code modifications they showed worked without issue.

The White Album

The next suggested package was WhiteNoise which manages serving static content from Django.  I honestly had never thought of this before and had some learning to do.  The install was fine, but I ran into issues with the bootstrap fonts not being present during the collectstatic operation.  It took me a bit of digging but I figured out a solution for this – I put the bootstrap fonts in the static/bootstrap/fonts directory.  I’m not sure if this is the preferred solution, but it works for now and keeps all the files local.

At this point I attempted my first deploy.  Of course, it failed hard.

But the logs were kind enough to point out the problem:

CommandError: You must set settings.ALLOWED_HOSTS if DEBUG is False.

and, doing some searching on the web and the example app, the answer was easy, simply set the field to * in settings.py

# Allow all host headers
ALLOWED_HOSTS = ['*']

It was also at this point that it occurred to me that I had to make other changes to my settings.

Do You Want To Know A Secret?

Setting the debug flag to False was an obvious change in settings, but what to do about that pesky secret key?   Heroku (and the Two Scoops book) both recommend using an environment variable, but several sites on the web point out that environment variables are less secure than a file on the filesystem.  While this may be true in general (“cat /proc//environ” on a linux box will give you the environment variables), it doesn’t appear to be on Heroku’s dynos.

Another idea was to use a dynamic secret key for the app.  This can be done with code (from SO) that mirrors how Django initially creates these keys:

from django.utils.crypto import get_random_string

chars = 'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)'
get_random_string(50, chars)

While this would work for this instance, there are some problems with it and it seems like using “best practices” as general rule would be a good idea, so I followed the directions and set it in the environment.

Mr. Post(gres)Man

Next came the big challenge.  I had been using sqlite3 for my app because, well, it’s easy.  Heroku doesn’t support sqlite3 as explained in the first answer in this SO post.

So, I needed to switch to a new database.  As postgres was the recommended solution and I didn’t have another preference, I went with that.

This conversion actually took a good portion of the entire process, again, largely through self-inflicted problems.

The basic setup for running postgres on Linux I gleaned from this site.

The dj_database_url and heroku’s default database config settings were fine.  There were just a couple of things I needed to learn first.  I needed to set up the postgres database with a full user, grant rights, etc.  Doing this locally, it took me a while to figure out all the the steps.  But, once I had done this, transferring that to Heroku was very straightforward.  You can to

heroku run bash

to get into the dyno, or for this case

heroku pg:psql

to get you directly into the postgres prompt on the production database.

Get Back

The final hurdle turned out to be with the database.  I had forgotten that you need to specify the app name when doing makemigrations and migrate.  I kept getting errors starting the app because it couldn’t find any of the required tables for the app.  It turns out that

heroku run python manage.py makemigrations
heroku run python manage.py migrate

acted like they succeeded but didn’t create the required tables.  After a bit of late-night head scratching, it finally dawned on me to add the name of the app

heroku run python manage.py makemigrations tasks
heroku run python manage.py migrate tasks

Once I got that solved the app was up and running.

Full Speed Ahead, Mr. Boatswain!

Mainly because I will likely find it handy in the future, I’m going to list some of the commands I found useful in the adventure.

  • heroku local web – test the configuration on your local system
  • heroku create kidtasks – create a new heroku app with a name
  • git push heroku master– this triggers a full deploy of the code
  • heroku config:set SECRET_KEY=<my super secret key>– sets environment variables on the app’s dyno
  • heroku run python manage.py [makemigrations tasks | etc] – basic django commands on the deployed app
  • heroku logs –tail – get the logs from the app

The End

While this process took more effort than I expected, I don’t think Heroku is to blame for any of it.  Now that I have a working example (and an idea of how postgres works), the next time will be much smoother.

Next app I start, I will definitely be using this template and see how much time it saves me.

Login, Please!

This week I was gearing up to try to deploy the KidsTask app. My first inclination was to try out Docker, mainly as it would be potentially useful knowledge for some work projects that are coming up.  That idea fell at the first hurdle.  I use an old laptop for my main development machine and Docker is only supported on 64 bit Linux.  While I did see ways to rebuild patched source to get Docker to run on a 32-bit system, that was more than I was interested in this week.

Next up: Heroku.  I’ve heard good things about this as a place to put very small apps and there are many resources for how to deploy a Django app to Heroku.  Putting the app on the external web, however, posed a new issue; KidsTasks didn’t have any form of authentication.  Anyone could go in and modify the data, which seems less-than-ideal in today’s world.

I (mistakenly) thought that adding user access to KidsTasks would be a good chunk of work.  After a little research and messing around, it turned out to be quite easy!  It turns out that the auth module shipped with Django was adequate for my needs and that configuring it was fairly straight forward.

Settings

You need to tell the auth module two pieces of information through the settings.  The first is where to go once a new user has logged in. This is specified by the name you supplied in the urlpatterns, not the url itself.  In my case this was just the “today” page which lists the daily tasks.  (NOTE: both of these setting are stand-alone variables inside the settings file, they are not part of any list or other structure)

LOGIN_REDIRECT_URL = 'today'

The next setting comes into play when you want to prevent users from accessing views without being logged in.  We’ll discuss how to set that up toward the end of this post, but for now, we’ll need to tell the auth module what the url is for the login page.

LOGIN_URL = 'login'

URLS

Next step is to add URLs to the urls.py file.  Most projects will have two or more urls.py files.  I chose the main project urls.py, rather than the one in my app, tasks, as those urls are prefixed by “tasks” and I wanted the login page to be reachable by just “login”.

We’re going to be using the built-in views in the auth module, so we need to import those first:

from django.contrib.auth import views as auth_views

Then we need to add the urls to the urlpatterns list.

urlpatterns = [
...
    url(r'^login/$', auth_views.login, name='login'),
    url(r'^logout/$', auth_views.logout, {'next_page': 'login'}, name='logout'),

Note the next_page parmaeter we’re passing to the logout view.  This is a nice little trick which forces a load of the logout url to log the user out and then redirect to the login page.

You must logout!

I almost forgot that we still need a way to get to the logout page.  I added a button to the navbar which is shown on all of the pages which points to logout.  Simple.

                  <li>
                  <a href="{% url 'logout' %}">Logout</a>
                  </li>

Login, please

The login template was by far the biggest section of work (and it wasn’t that large).  The template is very simple, just providing a form for the login information.  I’m still not happy with how it turned out, however.
I started by using the base.html like all of the other templates.  This, unfortunately put the navbar on the login page, which seemed odd.  So I ended up replicating some of the base.html code to get my css theme applied.  I’m still not happy with this solution, however, as I suspect there should be a better way not repeat code.  I might explore some of these options at a later point.

Another issue with the login template is that it wants to live in the ‘registration’ app by default.  Rather than figuring out how to change the default, I simply added a registration directory under the tasks app’s templates directory.  (I started by creating a top-level registration app which simply had the template in it.   I liked this even less.).  This is another ‘technical-debt’ issue that I hope to clean up in the future.

The template itself is seen below.  Nothing too shocking; basic html wrapper, bootstrap initialization, and a form.

<html>
<div>
   <head>
   {% load bootstrap3 %}
   {% bootstrap_css %}
   {% bootstrap_javascript %}
   {% bootstrap_messages %}

   <title>KidTasks</title>
   </head>
   <body>

   <div class="page-header">
      <h1>{% block title %}KidTasks Login{% endblock %}</h1>
   </div>

   <div>
      <form method="post">
         {% csrf_token %}
         {% bootstrap_form form %}
         {% buttons %}
         <button type="submit" name="save" value=True class="btn btn-success">Login</button>
         {% endbuttons %}
      </form>
   </div>
   </body>
</div>
</html>

Login Required!

Once you’ve done all of the above, you’ll be able to log in to your app.  The app will still allow you to access all of the pages without logging in.  It turns out there is a remarkably clever and simple solution for this: the @login_required decorator.

In your views file, you’ll need to import

from django.contrib.auth.decorators import login_required

and then add this decorator to each view function you want to be restricted.

@login_required

Works like a charm!

While I left a couple of loose ends I’ll eventually have to clean up, the task of adding login restrictions to KidsTasks was much easier than I expected.

Extending Templates

Recently at work, I had a quite long code review from a very young, very rushed co-worker.  Let many such reviews (my own included) there were several instances of copy-paste coding, where the same block of code is copied to several locations with only slight modifications.

This was fresh in my mind when I started in on this series of tutorials (which are quite good):  Building Data Products with Python

In the middle of the first page, there is a great description of how I have been doing copy-paste coding in the KidTasks app – navigation links in the templates.

Fortunately for me, they also give a good example of how to fix this by using the extends tag to reference shared template code, similar to using a base class.

The Problem

Two of my templates end with the following four lines of code

         <br/> <a href="{% url 'task_new' %}?from={{ request.path|urlencode }}">Add New Task</a>
         <br/> <a href="{% url 'rep_task_new' %}?from={{ request.path|urlencode }}">Add New Repeating Task</a>
         <br/> <a href="{% url 'kid_new' %}?from={{ request.path|urlencode }}">Add New Kid</a>
         <br/> <a href="{% url 'today' %}">View Today's Tasks</a>

While there are currently only two instances of this, there are likely to be more in the future (or in a real app).

The Fix

Fortunately, the solution was quite simple.  First you create the base template which includes the shared code and some blocks for parts of the template content.  In my case, for such simple pages, there was really only a title block and a content block.  The base template looks like this:

<div>
   <h1>{% block title %}(no title){% endblock %}</h1>

   {% block content %}(no content){% endblock %}

   <nav>
      <div id="navbar">
         <br/> <a href="{% url 'task_new' %}?from={{ request.path|urlencode }}">Add New Task</a>
         <br/> <a href="{% url 'rep_task_new' %}?from={{ request.path|urlencode }}">Add New Repeating Task</a>
         <br/> <a href="{% url 'kid_new' %}?from={{ request.path|urlencode }}">Add New Kid</a>
         <br/> <a href="{% url 'today' %}">View Today's Tasks</a>
      </div>
   </nav>
</div>

Note that this puts the navigation links on the bottom of the page (where they were previously).  It also adds a div with an id to the nav bar which, I suspect, is going to be handy when I start diving in to CSS and making the site look a bit more professional.

Another thing to note is that this will also move the formatting of the title for these pages to the base template rather than having it in each individual template.  This is helpful for maintaining a consistent look throughout your app.

The change to the sub-templates was quite small.  First I had to move the title from inside the content block into its own block:

{% block title %} Today's Tasks {% endblock %}

…and then all I had to do was use the extends tag to reference the base template:

{% extends 'tasks/base.html' %}

NOTE: the extends tag is required to be the first line in the template.  Django will complain loudly if it is not.

One of the full sub-templates ended up looking like this:

{% extends 'tasks/base.html' %}

{% block title %} Today's Tasks {% endblock %}

{% block content %}
{% if kids %}
    {% for name, tasks in kids.items %}
        <h1>{{ name }} on {{ day }}</h1>
        {% if tasks %}
            <ul>
            {% for task in tasks %}
                <li><a href="{% url 'update' task.id %}">
                {% if task.completed %} <strike> {% endif %}
                    {{ task.name }}
                {% if task.completed %} </strike> {% endif %}
                </a></li>
            {% endfor %}
            </ul>
        {% else %}
            <p>No tasks for {{ name }}.</p>
        {% endif %}
    {% endfor %}
{% else %}
    <p>No kids are present.</p>
{% endif %}
{% endblock %}

That’s all there is to it!  Thanks to Jose A Dianes for the well-written tutorial referenced above.

KidsTasks: Minor Redesign

I’ve been slowed down in my progress this month due to a family vacation and a change of jobs, so I’ve only made a little progress on the KidsTasks app.

This post gets the app up to the ‘almost functional’ stage where the admin portal can be used to create tasks and schedules and the normal app UI can be used to list the day’s tasks, all of the tasks, and to mark tasks complete (or incomplete).  It’s not a great idea to have the admin pages be the main data entry access, but for this early stage it will be sufficient.

All of this functionality needs some more work, but the basics are there, which is always a good time to stop and consider your design.  Especially when I’m learning a new language or technology, I find that iterative design works best for the first several projects.  Once you get your hands dirty, it’s easier to see what the next step should be.  This doesn’t often (some would say ‘rarely’) ends up with an optimal design, but it gives you more room to experiment and learn as you go.  That’s definitely what’s going to happen here, as I already have several changes in mind.

What Changed This Time

While working on the very simple views and templates, I decided that some simplification of the models was also in order.  The models went from two different types of tasks existing in to different types of schedules.  The different schedule types got removed and a new field got added to the Kid model to hold a list of DateTasks.

The basic issue driving this (which would have been obvious to many designers) was the ability to change “abstract tasks” (i.e. ‘practice piano every Tuesday’ into DateTasks (“It’s Tuesday today, get a list of all tasks that are due today and display them.”).   The quick answer to this was to create the new list of DateTasks in the Kid model.  This will store all of the tasks due today.  This is really just a stepping stone to where I want to end up, having a history of completed (and non-completed) tasks.

Once I got this change to the models in place (complete with several rounds of rebuilding the database), I moved on to getting a most basic of user interfaces in place.  To do this I hacked my way through learning the basics of accessing the Django ORM from views and templates.  While this is fairly well documented, it still took some playing with to get where I was going and get the proper syntax.

I ended up doing most of my hacking in the views.py file and that’s where I put the main function to convert the Tasks to DateTasks.  In the subsequent week since I wrote that, I’ve read (in Two Scoops of Django) that putting logic in your views is not the best practice, so the next revision will likely fix this and put that functionality into the model.

The Templates and Views

There’s really not all that much to the new views once you see the code.  One thing that stood out for me was the simplicity of the Update view function.  Being able to do a redirect “by magic” is pretty cool and produces a nice effect in the UI.

As I’ve mentioned before, I’m not a graphic designer, but I do have intentions of creating something a little nicer than it currently stands: a simple HTML list with links to change the state of each task.

Cleanup

On a final note, I had to rebuild my laptop a few months ago and in the process I managed to forget to reinstall flake8!  I finally noticed this and now my vim setup is marking all the errors it found.  These got cleaned up on this round as well.

The next set of work on KidsTasks is to get a little nicer UI and to add some views for creating tasks instead of relying on the clunky admin portal.  This might or might not entail a dive into class-based-views.

For those of you keeping score at home, the code at this state is in branch: blog/03-Simple-Views-Update in this repo: git@github.com:jima80525/KidTasks.git

Thanks for reading!

KidsTasks – Working on Models

This is part two in the KidsTasks series where we’re designing and implementing a django app to manage daily task lists for my kids. See part 1 for details on requirements and goals.

Model Design Revisited

As I started coding up the models and the corresponding admin pages for the design I presented in the last section it became clear that there were several bad assumptions and mistakes in that design. (I plan to write up a post about designs and their mutability in the coming weeks.)

The biggest conceptual problem I had was the difference between “python objects” and “django models”. Django models correspond to database tables and thus do not map easily to things like “I want a list of Tasks in my DayOfWeekSchedule”.

After building up a subset of the models described in part 1, I found that the CountedTask model wasn’t going to work the way I had envisioned. Creating it as a direct subclass of Task caused unexpected (initially, at least) behavior in that all CountedTasks were also Tasks and thus showed up in all lists where Tasks could be added. While this behavior makes sense, it doesn’t fit the model I was working toward. After blundering with a couple of other ideas, it finally occurred to me that the main problem was the fundamental design. If something seems really cumbersome to implement it might be pointing to a design error.

Stepping back, it occurred to me that the idea of a “Counted” task was putting information at the wrong level. An individual task shouldn’t care if it’s one of many similar tasks in a Schedule, nor should it know how many there are. That information should be part of the Schedule models instead.

Changing this took more experimenting than I wanted, largely due to a mismatch in my thinking and how django models work. The key for working through this level of confusion was by trying to figure out how to add multiple Tasks of the same type to a Schedule. That led me to this Stack Overflow question which describes using an intermediate model to relate the two items. This does exactly what I’m looking for, allowing me to say that Kid1 needs to Practice Piano twice on Tuesdays without the need for a CountedTask model.

Changing this created problems for our current admin.py, however. I found ideas for how to clean that up here, which describes how to use inlines as part of the admin pages.

Using inlines and intermediate models, I was able to build up a schedule for a kid in a manner similar to my initial vision.  The next steps will be to work on views for this model and see where the design breaks!

Wrap Up

I’m going to stop this session here but I want to add a few interesting points and tidbits I’ve discovered on the way:

  • If you make big changes to the model and you don’t yet have any significant data, you can wipe out the database easily and start over with the following steps:
$ rm db.sqlite3 tasks/migrations/ -rf
$ ./manage.py makemigrations tasks
$ ./manage.py migrate
$ ./manage.py createsuperuser
$ ./manage.py runserver
  • For models, it’s definitely worthwhile to add a __str__ (note: maybe __unicode__?) method and a Meta class to each one.  The __str__ method controls how the class is described, at least in the admin pages. The Meta class allows you to control the ordering when items of this model are listed. Cool!
  • I found (and forgot to note where) in the official docs an example of using a single char to store the name of the week while displaying the full day name. This looks like this:
    day_of_week_choices = (
        ('M', 'Monday'),
        ('T', 'Tuesday'),
        ('W', 'Wednesday'),
        ('R', 'Thursday'),
        ('F', 'Friday'),
        ('S', 'Saturday'),
        ('N', 'Sunday'),
    )
	...
    day_name = models.CharField(max_length=1, choices=day_of_week_choices)
  • NOTE that we’re going to have to tie the “name” fields in many of these models to the kid to which it’s associated. I’m considering if the kid can be combined into the schedule, but I don’t think that’s quite right. Certainly changes are coming to that part of the design.

That’s it! The state of the code at the point I’m writing this can be found here:

git@github.com:jima80525/KidTasks.git
git checkout blog/02-Models-first-steps

Thanks for reading!

KidsTasks: The first appliciation – Data Models

This post is the first in a series where I will document that process of creating a new django app from the ground up. In this post I’ll talk a bit about the high-level requirements of the app we’ll be doing and I’ll follow-up next with a quick run-down on how to get the first steps of the app running.

Since this is a blog for experienced developers, I’m going to make a few assumptions here:

  • You know how to use git, at least at a rudimentary level
  • You have and know how to use a good editor
  • You’ve already worked through the Django tutorial in the official documentation

Note that I tend to keep diagrams and requirements informal as my goal here is not to teach/learn UML or rigid design practices, but rather to focus on learning full-stack development. I am using graphviz tied to the django-extensions tools to allow me to automatically export my app’s models and diagram them. This is quite handy. The install is here
and the instructions are here

KidsTasks

My first django app will be a replacement for the daily/weekly task list my kids are using currently. They have a paper chart on which they move magnets to indicate that a task is complete. Each day the list is reset for a new day.

Requirements

Here’s a quick enumeration of requirements. I’m sure this isn’t all of them, but it’s sufficient to start doing the data design for the project.

  • The list of tasks for each day consists of three sets:
    1) tasks due on each day of the week (i.e. Tuesday is bath day)
    2) tasks due on a particular date (i.e. get present for sister on 9/3)
    3) tasks due sometime this week, but not necessarily today (i.e. clean room) Weekly tasks can have multiple occurrences.
  • Each individual task can be “required” or not
  • The kids should be easily able to mark a task as completed (or uncompleted in case of mistakes).
  • Mom should be able to also mark individual tasks in addition to being able to add, remove and reschedule tasks.
  • A history of completed tasks should be viewable over different time frames (yesterday, last week, last month, custom date range).
  • Kids should be able to view all tasks, completed and uncompleted as well as filtering only one kind.

Data Model

UML diagram of data model.As you can see from the diagram, the fundamental data object here is a task. There are two types of tasks: Task and CountedTask with the latter sub-classing and adding a number of occurrences needed for the weekly tasks.

Tasks are brought together into DayOfWeekSchedules which allow us to list what tasks are due on Mondays, and DateSchedules, which allow us to list what’s due on the 3rd of October.

CountedTasks form WeekSchedules and HistoricDates (more on history later).

The different Schedules are combined into a Schedule object which is then owned by Kids.

Kids also have Histories which are comprised of HistoricDates which, in turn, are CountedTasks coupled with a date.

This is the basic data structure with which we’ll start our adventure. Next time I’ll do more Django-specific work, talking about how to get from the last stage (basic Django install and app creation) to the point where we can view and manipulate these objects on the admin page.