Return of pylint

Until last fall I was working in python 2 (due to some limitations at work) and was very happy to have the Syntastic module in my Vim configuration to flag error each time I save a python file.  This was great, especially after writing in C/C++ for years where there is no official standard format and really poor tools to enforce coding standards.

Then, last fall when I started on Django, I made the decision to move to Python 3.  I quickly discovered that pylint is very version-dependent and running the python2.7 version of pylint against Python3 code was not going to work.

I wasn’t particularly familiar with virtualenv at the time, so I gave up and moved on with other things at the time.  I finally got back to fixing this and thus getting pylint and flake8 running again on my code.

Syntastic

I won’t cover the details of how to install Syntastic as it depends on how you manage your plugins in Vim and is well documented.  I will only point out here that Syntastic isn’t a checker by itself, it’s merely a plugin to run various checkers for you directly in Vim.  It run checkers for many languages, but I’m only using it for Python currently as the C code I use for work is so ugly that it will never pass.

Switching versions

The key to getting pylint to run against different versions of python is to not install pylint on a global level, but rather to install it in each virtualenv.  This seems obvious now that I’m more familiar with virtualenv, but I’ll admit it wasn’t at the time I first ran into the problem.

The other key to getting this to work is to only initiate Vim from inside the virtualenv.  This hampers my overall workflow a bit, as I tend to have gVim up and running for the long-term and just add files in new tabs as I go.  To get pylint to work properly, I’ll need to restart Vim when I switch python versions (at a minimum).  This shouldn’t be too much of a problem, however, as I’m doing less and less python2x coding these days.

Coding Style Thoughts

As I beat my head against horrible C code on a daily basis at work, I find myself appreciating more-and-more the idea of PEP-8 and having good tools for coding style enforcement.  While I frequently find some of the rules odd (two spaces here, but only one space there?) I really find it comforting to have a tool which runs, and runs quickly, to keep the code looking consistent.  Now if I could only get that kind of tool for C…….

 

Deep dive into manage.py

I don’t know about you, but when I started with Django, I wondered about the “manage” script and what it did.  But there were too many other things to learn, so I accepted that it worked and moved on, using the recipes I found in examples and documentation.

My curiosity has gotten the better of me, so now I’m going to take a break and examine what’s going on under the hood.

Naively, I had assumed that the manage script was some large, static beast with 1000’s of lines of code.  (Too much C and make in my background, I guess).  Of course, this is python, so manage.py is small, reflective, and quite dynamic.

On a brand new project, manage.py weighs in at a whopping 22 lines of code.  Now, that’s not really a fair accounting as all that code is doing is importing a method from a module:

from django.core.management import execute_from_command_line

and then running it:

execute_from_command_line(sys.argv)

Th execute_from_command_line function is located in the __init__.py package in python/site-packages in your virtualenv   (you are using virtualenv, right?).

This app does several bookkeeping things (managing command line args, deciding if it needs to do “help”, etc) and then calls the get_commands function.  This is a very elegant function (the comment block explaining it is much longer than the code) that creates a { command_name : app_name } dictionary for all of the commands it can find.  It starts by adding all of the django core commands by searching the management/commands directory in the core django directory.  It then searches the management/commands subdirectory of each installed app.

It took a little digging, but I managed to figure out a bit more detail here which I found interesting.  Django has a list called django.apps.apps.  These are the apps listed in the INSTALLED_APPS list in your settings.py file.  An entry is made for each app loaded with some details about the app.  For our purposes here, the data that we care about is the path and the name.

Jumping back to the get_commands function and the __init__.py file in the django.core.management module, you can see at the top of the file that it imports this list of apps:

from django.apps import apps.

This allows the get_commands function to walk the list of apps, getting the name of each app and the path in which it’s stored.  It appends the management/commands subdirectory to the path and adds each python file there as a command to the { command_name : app_name }, listed above.  (Note that I’m skipping some details here).  That’s how it figures it out.

So, to complete the loop, we can see that the apps listed in settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

show up in the lib/python3.4/site-packages/ directory structure:

django/
├── core
│   ├── management
│   │   ├── commands
│   │   │   ├── check.py
│   │   │   ├── compilemessages.py
│   │   │   ├── createcachetable.py
│   │   │   └── [several commands removed]
│   │   │   ├── runserver.py
│   │   │   └── [several commands removed]
├── contrib
│   ├── auth
│   │   ├── management
│   │   │   ├── commands
│   │   │   │   ├── changepassword.py
│   │   │   │   ├── createsuperuser.py
│   ├── contenttypes
│   │   ├── management
│   │   │   ├── commands
│   │   │   │   └── remove_stale_contenttypes.py
│   ├── sessions
│   │   ├── management
│   │   │   ├── commands
│   │   │   │   ├── clearsessions.py
│   ├── staticfiles
│   │   ├── management
│   │   │   ├── commands
│   │   │   │   ├── collectstatic.py
│   │   │   │   ├── findstatic.py
│   │   │   │   └── runserver.py

and are represented in the output of ./manage.py –help

Available subcommands:

[auth]
    changepassword
    createsuperuser

[contenttypes]
    remove_stale_contenttypes

[django]
    check
    compilemessages
    createcachetable
    [several commands removed but NOT runserver]

[sessions]
    clearsessions

[staticfiles]
    collectstatic
    findstatic
    runserver

So, when you add django-extensions to your INSTALLED_APPS list, that is how the new commands show up!

Those of you that are really detail-oriented might have noticed that the runserver command is listed in both the django.core app as well as in the staticfiles app.  It is only listed once in the command output.  It turns out that the way the dictionary of commands is constructed, it uses the command name for the key and the app name for the value.  This means that the last one found will be the command listed.  This works out well in this case as the staticfiles version of the command add functionality you very likely want (mainly serving static files) on top of the core runserver command.  In classic DRY style, the staticfiles version of the command makes use of the core version.  It also has the nice effect of providing a runserver command which does NOT serve static files if you do not have the django.contrib.staticfiles app installed.  Sweet!

Building your own extensions

If you follow all of that, the documentation that describes adding your own management commands for your app should seem fairly obvious.  Basically you just create a management/commands directory in your app and populate it with the commands you wish to add.  Note that there are a few other requirements (there needs to be an __init__.py file, the commands must all be subclasses of django.core.management.base.BaseCommand, etc), but the basic mechanism is exactly the same.

Exploring with Cookiecutter

After reading Two Scoops of Django, I decided to explore the cookiecutter app that they recommend.  If you haven’t used it, cookiecutter is ” A command-line utility that creates projects from cookiecutters (project templates). E.g. Python package projects, jQuery plugin projects.”

In particular I explored some of the Django-specific cookiecutter templates (though I did peek at a few other languages, too).  I started with the cookiecutter-django template as it was written by one of the authors of the book and it was at the top of the list I was looking at.

Cookiecutter-Django

Cookiecutter-Django is a full-on professional level template for producing enterprise-level Django apps.  As such, it comes with a LOT of bells and whistles.  Celery support?  Yep.  Docker? Yep.  Custom windows and pycharm setups?  Yep.  The list goes on.  This is pretty cool, but, at 134 files, many of which I do not know the use of, this is clearly not aimed at me, the hobbyist learning Django in his spare time.

It does, however, have some interesting project layout ideas that I think ARE useful and sound.  Among these are having separate doc and requirements directories and having the requirements files broken into separate environments (local, production, test) to codify and simplify the set up for the job you’re doing.

Lots of good stuff here to dig into in the future, but this is a bit over my head at the moment. Fortunately, there are a myriad of other templates to play with.  I found a good list of them at readthedocs.

Django-crud

The next one I ventured into was django-crud.  This is different from the others I looked at in that it was not a full django project, rather just the webapp portion of it.

It is pretty cool in that it gives a really detailed example of how to write a simple model (performing the CRUD database operations, of course) with full unit tests and simple templates.  At 16 files, it’s not so overwhelming and has a pretty clear intent.  I’m sure the author uses it as a starting point for webapps, but I’m finding it as useful learning tool and will likely be the basis as I start down the unit testing road (coming soon!).

Django-paas

I also took a quick tour through Django-paas. which is billed as a “Django template ready to use in SAAS platforms like Heroku, OpenShift, etc..”.

While this isn’t likely one I’m ready to use just yet (I’m still hosting locally on my private machine for use inside my network).  It’s definitely got a few features that are worth looking at.  Of note was the use of waitress (instead of gnuicorn) which might be an interesting alternative.  It also provided a procfile for heroku, which I can see being handy in the future.

Simple Django

My final stop of this tour was Simple-Django.  This is a very trimmed down version of the full cookiecutter-django template I started with.  It still has separate requirements and docs directories, but does not pull in nearly the number of third-party packages that its full-blown antecedent does.

This looks like a good start place for my level of development and interest.  It will likely be the basis I use if and when I create a django template for my own use.

Summary

While none of these cookie-cutter templates are exactly what I’m looking for at the moment, I found exploring them a good way to learn about different tools, directory layouts and general best practices in the django world.  I’d encourage you to explore some of these, even if they’re not something you need at the moment.

I’d like to thank the authors of each of these templates for the work they put into them and for sharing them with us, and, in particular, Audrey Roy Greenfeld and Danny Roy Greenfeld for creating cookiecutter and the cookiecutter-django template.

Thanks!