Adventures in Python 3

As part of my playing with PyBuilder (mentioned in the last post), I decided to convert one of my old tools to use it and to convert to python3 at the same time.  While this was probably foolish (making two changes at once) it was educational in the long run.  I thought I’d share my learnings here.

Simple Changes

The first and most prevalent change was the change of print from a keyword to a function.  Trivial to fix and not worth discussing.

The urllib2 library was modified and required changes to the imports of urlopen and URLError.  The queue import was changed to Queue, but otherwise worked as I was using it.

There were a couple of places where mocks in the unit tests needed to be modified.  Looking at the changes I had to make, which were largely making the mocks specific to the module under test, I’m not sure how they worked in the old code.  The new way certainly looks more ‘correct’.

The final simple change was one I didn’t expect.  Python 3 changed how comparison operators handle None as described here.  I’m sure I’m not the only person that had written code assuming None is always less than any string.

String vs buffer

One change that took me a while to figure out was the buffer interface versus string.  I had read about this and thought I understood it, but I was stumped by why this line was failing.

title = title.replace("'", "''")

It probably shouldn’t have taken that long, but I’ll admit it took me a while to see that the ‘string’, title, was being returned from the feedparser library and was actually not a string at all.  Adding a decode to this to make it a string worked just fine.

title = title.decode('utf8').replace("'", "''")

Note that I’m making a big assumption that the title will actually be encoded in utf8.  For the purposes of this program, that assumption is true, but it’s not in general.  YMMV.

Argparse

This change caught me by surprise as well.  This is the only one of the issues I ran into that wasn’t listed in the excellent “What’s New in Python 3.0” article.

The argparse module changed behavior between py2 and py3.  This looks like a bug, as shown here.

Apparently subparsers do not have the required property set by default.  The work-around, once you find it, is trivial, after adding a new subparser:

subparsers = parser.add_subparsers(help='commands', dest='command')

you need to set its required property to True:

subparsers.required = True

Next post (which might be a couple of weeks) will look at changes to the project to bring it into pybuilder and to have it take advantage of things I’ve learned since I originally wrote it (requirements.txt, etc).

Until then, happy holidays!

Build Tools

I’m doing a survey for build tools in python.  Now, before any of you say “make”, be warned that I’ve been doing C and C++ programming for decades and even those project don’t even use make anymore if they can help it.  There are way better tools than make, especially in python!

Paver

I used paver on a previous project and thought it was OK but fairly verbose.  It was pretty easy to extend it and get it to do exactly what I wanted, however, so it worked just fine for that project. It doesn’t look like there’s too much activity on the project in github currently, but that may just mean it’s stable.

Scons, Waf

I’ve looked into these in previous lives as a build tool for C++ projects.  They both seem a little heavy for what I’m doing here (but I’d entertain arguments to the contrary).  I’m skipping them for now as “too complex”.  Again, this might not be a fair assessment.

Doit

This looked promising.  I like design and the documentation was pretty good.  It looks to be under active development.  The reason this isn’t my front runner at this point is, oddly enough, its name.  “Doit” is a french verb, making it cumbersome to do google searches on.  This is just a little extra friction, I’ll admit.  Given a wildly superior tool, I’d definitely put up with it, but this didn’t stand out that much.

Pybuilder

This is the one I’m exploring for now.  It has an active develpment community with quick response times to issues.  The documentation is OK, but certainly could use some work.  It is not at all verbose.  Actually, I’d say it errs on the side of being “too magic” where it’s hard to understand the “how” of what’s going on.  But, it seems to do the things I need AND it has a project to do a django plugin.

I haven’t tested the plugin out as of yet, but it looks like it could be a good building block to get a good development system in place with unit tests, coverage reports (so you can play the ‘coverage game’) and some good packaging.

If you’ve got a favorite I skipped or think I missed some important feature of one of the above tools, please let me know in the comments section!

Exploring with Cookiecutter

After reading Two Scoops of Django, I decided to explore the cookiecutter app that they recommend.  If you haven’t used it, cookiecutter is ” A command-line utility that creates projects from cookiecutters (project templates). E.g. Python package projects, jQuery plugin projects.”

In particular I explored some of the Django-specific cookiecutter templates (though I did peek at a few other languages, too).  I started with the cookiecutter-django template as it was written by one of the authors of the book and it was at the top of the list I was looking at.

Cookiecutter-Django

Cookiecutter-Django is a full-on professional level template for producing enterprise-level Django apps.  As such, it comes with a LOT of bells and whistles.  Celery support?  Yep.  Docker? Yep.  Custom windows and pycharm setups?  Yep.  The list goes on.  This is pretty cool, but, at 134 files, many of which I do not know the use of, this is clearly not aimed at me, the hobbyist learning Django in his spare time.

It does, however, have some interesting project layout ideas that I think ARE useful and sound.  Among these are having separate doc and requirements directories and having the requirements files broken into separate environments (local, production, test) to codify and simplify the set up for the job you’re doing.

Lots of good stuff here to dig into in the future, but this is a bit over my head at the moment. Fortunately, there are a myriad of other templates to play with.  I found a good list of them at readthedocs.

Django-crud

The next one I ventured into was django-crud.  This is different from the others I looked at in that it was not a full django project, rather just the webapp portion of it.

It is pretty cool in that it gives a really detailed example of how to write a simple model (performing the CRUD database operations, of course) with full unit tests and simple templates.  At 16 files, it’s not so overwhelming and has a pretty clear intent.  I’m sure the author uses it as a starting point for webapps, but I’m finding it as useful learning tool and will likely be the basis as I start down the unit testing road (coming soon!).

Django-paas

I also took a quick tour through Django-paas. which is billed as a “Django template ready to use in SAAS platforms like Heroku, OpenShift, etc..”.

While this isn’t likely one I’m ready to use just yet (I’m still hosting locally on my private machine for use inside my network).  It’s definitely got a few features that are worth looking at.  Of note was the use of waitress (instead of gnuicorn) which might be an interesting alternative.  It also provided a procfile for heroku, which I can see being handy in the future.

Simple Django

My final stop of this tour was Simple-Django.  This is a very trimmed down version of the full cookiecutter-django template I started with.  It still has separate requirements and docs directories, but does not pull in nearly the number of third-party packages that its full-blown antecedent does.

This looks like a good start place for my level of development and interest.  It will likely be the basis I use if and when I create a django template for my own use.

Summary

While none of these cookie-cutter templates are exactly what I’m looking for at the moment, I found exploring them a good way to learn about different tools, directory layouts and general best practices in the django world.  I’d encourage you to explore some of these, even if they’re not something you need at the moment.

I’d like to thank the authors of each of these templates for the work they put into them and for sharing them with us, and, in particular, Audrey Roy Greenfeld and Danny Roy Greenfeld for creating cookiecutter and the cookiecutter-django template.

Thanks!