Posts Tagged ‘django’

Django Testing

November 27, 2008

Like many a geek, I’m a lazy bastard, so it’s only recently I’ve gotten around to writing unit tests for my tiny website (I at least have the excuse of doing this only for jollies; I’m no professional web developer). I had vaguely assumed that writing (and tests) would be more trouble than it’s worth, and that it would be difficult to test Really Important Stuff anyway. I had also vaguely assumed that those vague assumptions were almost certainly wrong, and I was a contemptible fool for not having written the tests up front.

It turns out, no surprise, that the latter vague assumption was correct. Python’s doctest and unittest frameworks are already relatively simple, and the django testing framework makes them simpler still (there’s a bit of annoying boilerplate to figure out in python’s raw unittest, which the django framework thoughtfully hides). The setup for test databases is especially nice.

[Mind you, in my first attempt I did somehow manage to blow away my local copy of my database—not just the test database, but the real one. I never did figure that one out…]

Embarrassingly, one of the first tests I wrote turned up a bug. Not to be wondered at, I suppose.

One note, on the very slim chance someone finds it useful. The django test framework looks for tests in each app’s model and test modules. At first I was annoyed by this: I like to put doctests in other places, as they really belong in the docstrings of the functions and classes they test. But—and you already know this if you’re a better python programmer than I—there’s a trivial way around this. The unittest and doctest modules already coexist awfully well. doctest.DocTestSuite creates a TestCase from a module’s doctests, and this can be amalgamated with unittests by defining a function called suite in the test module, which django will look for. That is, put something like this in tests.py or tests/__init__.py:

def suite():
    # An easy way of finding all the unittests in this module
    suite = unittest.TestLoader().loadTestsFromName(__name__)
    for mod in myapp.views, myapp.forms:
        suite.addTest(doctest.DocTestSuite(mod))
    return suite
Advertisements

A little Django flatpage trick

May 31, 2008

For each general area of my humble little website, I have a base template that takes care of a links bar and breadcrumbs and things. All the individual pages in that area extend that base.html, which in turn extends parent base templates. Nothing unusual there, and all well and good.

Except for flatpages. Flatpages are great, but out of the box they’re a tiny bit rigid. I want a flatpage in a particular area to extend the right base.html, but that’s not quite what the flatpage template setting provides—that’s a whole template, and I just want a base to extend. One could of course have a different flatpage template for each base.html, but that’s gross, and not very DRY.

Maybe there is some simple obvious way to do this built in to Django, but I couldn’t find it. My solution is to extract the appropriate base template from the flatpage itself. I put it in a django comment, which will get stripped out of the content before rendering. So the line

{# Base utilities/base.html #}

goes in the body of the flatpage, and gets extracted with a filter. [It might be better to infer it from the page’s url, if one’s directory structure and url structure always match.] The only problem is that I need the filter loaded before the extends tag, and the extends tag needs to come before anything else. It used to be possible to load before extending, but that was an evil (if useful) loophole, now closed.

Like all problems in computer science, this can be solved with another level of indirection. flatpages/default.html loads the filter and extracts the base template name, and then includes another template to do the actual rendering.

Here’s the code, simple and completely non-robust though it is. In a tempatetags/flatpage_utils.py or whatever you want to call it:

@register.filter
@stringfilter
def stripdjangocomments(text):
    """
    Strip django comments from the text.
    """
    s = re.sub(r'{#.*?#}', '', text)
    return s

@register.filter
@stringfilter
def getbase(text, default = "base.html"):
    """
    Look for a string of the form {# Base foo #} and return foo
    """
    m = re.search(r'{#\s*Base\s*(\S*?)\s*#}', text)
    if m and m.groups()[0]:
        return m.groups()[0]
    else:
        return default

In templates/flatpages/default.html

{% load flatpage_utils %}

{% with flatpage.content|getbase as pagebase %}
{% include "flatpages/flatpagebody.html" %}
{% endwith %}

And in templates/flatpages/flatpagebody.html

{% extends pagebase %}
{% load whatever_else %}

{% block title %}
{{ flatpage.title }}
{% endblock %}

{# maybe other stuff #}

{% block content %}
{# add more filters if you like #}
{{ flatpage.content|stripdjangocomments }}
{% endblock %}

And that’s it.

Some AJAX in Django

May 11, 2008

Months ago I started looking into doing AJAXy things within Django, and (typically for me) never actually did any of them. Finally I’ve started looking at that again. My needs are simple and dull: I just wanted quick and seamless responses to changes in form data in the little utilities I just added to my website.

Now I know very little about Ruby on Rails, but some of what I’ve seen if does look kinda cool. In particular I liked the respond_to gadget, which switches on requested mimetypes to figure out what response to send from a view (or action, or whatever they’re called on Rails). That seems to allow nice code factoring with minimal syntax, in a way that’s concise and clever (typical for Ruby) and clear (not so typical, IMO…).

I’m not convinced this is a truly great idea, for reasons I’ll detail below, but what the hey, when did that ever stop anyone? So I hacked up a python/Django analogue (see the end of the post). I may have course have misunderstood completely what’s up with the Ruby thing, in which case, oh well.

Here’s an example of how you use this thing—a Responder object—in a view:

def index(request):
    data = { 'foo' : 'bar', 'this' : 'that' }
    responder = Responder(request, 'template', data)
            { 'raw' : raw, 'types' : types })

    responder.html

    responder.js

    return responder.response()

This says, more or less “If the request wants HTML, render the data with the template template.html. If it wants javascript, render with the template template.js and the javascript mimetype.” That is, it’s something like

def index(request):
    data = { 'foo' : 'bar', 'this' : 'that' }

    if <wants html>:
        return render_to_response('template.html', data)

    if <wants js>:
        return render_to_response('template.js', data,
            mimetype='text/javascript' )

    return responder.response()

[where that <wants html/javascript> conceals some complexity…]

The render-a-template behavior can be overridden: those hacky ‘html’ and ‘js’ attributes are callable. If one of them is passed a function, it calls it: if the function returns something, that something is used as the response. It can also modify data and return None to proceed with default handling. Here’s an example I used when testing this stuff on different browsers. It prints the contents of the HTTP_ACCEPT header, and provides a button to fire an ajax request to replace that. In this case I built the javascript messily by hand.

def index(request):
    raw = request.META['HTTP_ACCEPT']
    types = parseAccept(request)

    responder = Responder(request, 'index.html',
            { 'raw' : raw, 'types' : types })

    responder.html

    @responder.js
    def jsresp(*args, **kwargs):
        text = raw + '<br><br>' + \
            '<br>'.join('%s %g' %(s, q) for s,q in types)
        js = "$('content').update('%s');" % text
        return HttpResponse(js, mimetype='text/javascript')

    return responder.response()

Here’s the corresponding template (which uses prototype):

<script src="/static/js/scriptaculous-js-1.8.1/lib/prototype.js" type="text/javascript"></script>
<script type="text/javascript">
    function ajaxUpdate () {
        headers = { Accept : 'text/javascript;q=1, */*;q=0.1' };
        if(Prototype.Browser.Opera)
        {
            headers.Accept += ',opera/hack'
        }

        new Ajax.Request('/',
            { method:'get', parameters: {}, requestHeaders : headers } );
    };
</script>

<div id="content">
    {{ raw }}<br><br>
    {% for s in types %}
    {{ s.0 }} {{ s.1 }}<br>
    {% endfor %}
</div>
<div>
    <br>
    <input id="b1" onclick="ajaxUpdate();" type="button" value="Click Me!">
    </input>
</div>

So What Have I Learned From This? Well, it all seems to work, so I’ll keep using it. But I’m not totally sold that this—switching on HTTP_ACCEPT, and my own particular implementation—is the Right Way to do things.

Philosophically, the general idea seems awfully prone to abuse. As I understand RESTful web services (i.e. not very well), different requests correspond to different representations of the same underlying data. But are the original html and the javascript that updates it really different representations of the same thing, or different animals altogether? I think that’s a murky point, at best. And I should think that in real life situations it could get messy. What happens, for example, if there is more than one sort of javascript request (e.g. if there are different forms on a page that do fundamentally different things)?

Rails and REST fans, please set me straight here!

Practically, the HTTP_ACCEPT thing seems delicate. I had to futz around a bit to get it to work in a way I felt at all confident of. Browsers seem to have different opinions about what they should ask for. Oddly, the browser that caused me the most problems was Opera—despite what I told prototype’s AJAX request, Opera insisted on concatenating the ACCEPTed mimetypes with the original request’s mimetypes. I hacked around that by throwing in a fake mimetype to separate the requests I wanted from those Opera wants; see the template above and the code below.

So anyway, maybe it would be better, or at least more Django, to be explicit about these AJAX requests, and either give them different URLs (and factor common code out of the various views) or add a piece of get/post data, as here. For now I’ll keep doing what I’m doing, and see if I run into problems.

Here’s the Responder code. It has numerous shortcomings, so use at your own risk. It is completely non-bulletproof (and non-debugged), and won’t work if you don’t use it just like I wanted to use it (e.g. you’d better give it a template name). It obviously needs more mimetype knowledge—it falls back on python’s mimetype library, but that seems seriously unacceptable here. And I’m very lame about how I parse the HTTP_ACCEPT strings.

import sys
import re
import os, os.path, mimetypes
import django
from django.http import HttpResponse
from django.shortcuts import render_to_response
from django.template import RequestContext

class _ResponseHelper(object):
    def __init__(self, ext, mimetypes, responder):
        self.responder = responder
        self.ext = ext
        self.mimetypes = mimetypes
        self.fn = None

    def __call__(self, fn=None):
        self.fn = fn
        return fn

class Responder(object):
    """
    Utility for 'RESTful' responses based on requested mimitypes,
    in the request's HTTP_ACCEPT field, a la Reils' respond_to.

    To use, create a responder object.  Pass it the request object
    and the same arguments you would pass to render_to_response.
    Omit the file extension from the template name---it will be added
    automatically.
    For each type to be responded to, reference an attribute of the
    appropriate name (html, js, etc).
    Call the respond function to create a response.
    The response will be created by appending the extension to the filename
    and rendering to response, with the appropriate mimetype.

    To override the default behavior for a given type, treat its
    attribute as a function, and pass a function to it.
    It will be called with the same arguments as the Responder's constructor.
    If the function can modify the passed data, and either return None
    (in which case the template handling proceeds), or return a response.
    Function decorater syntax is a convenient way to do this.

    Example:

        responder = Responder(request, 'mytemplate', { 'foo': 'bar' })

        responder.html

        @responder.json
        def jsonresp(request, templ, data):
            data['foo' : 'baz']

        @responder.js
        def jsresp(request, templ, data):
            return HttpResponse(someJavascript,
                mimetype='application/javascript')

        return responder.response()

    Here an html request is processed as usual.
    A JSON request is processed with changed data.
    A JS request has its own response.

    """
    types = { 'html' : ('text/html',),
              'js' : ('text/javascript',
                      'application/javascript',
                      'application/x-javascript'),
              'json' : ('application/json',),
            }

    def __init__(self, request, *args, **kwargs):
        self.request = request
        self.resp = None
        self.args = [a for a in args]
        self.kwargs = kwargs
        self.priorities = {}
        for t, q in parseAccept(request):
            self.priorities.setdefault(t, q)
        self.defq = self.priorities.get('*/*', 0.0)
        self.bestq = 0.0

    def maybeadd(self, resp):
        try:
            thisq = self.bestq
            for mt in resp.mimetypes:
                q = self.priorities.get(mt, self.defq)
                if q > thisq:
                    resp.mimetype = mt
                    self.resp = resp
                    self.bestq = q
        except:
            pass

    def response(self):
        if self.resp:
            if self.resp.fn:
                result = self.resp.fn(self.request, *self.args, **self.kwargs)
                if result:
                    return result

            # the template name ought to be the first argument
            templ = self.args[0]
            base, ext = os.path.splitext(templ)
            if not ext:
                templ = "%s.%s" % (base, self.resp.ext)
            self.args[0] = templ
            self.kwargs['mimetype'] = self.resp.mimetype
        # if there wasn't a response, default to here
        response = render_to_response(
                context_instance=RequestContext(self.request),
                *self.args, **self.kwargs)
        return response

    def __getattr__(self, attr):
        mtypes = None
        if attr not in self.types:
            mtypes = [mt for mt, enc in [mimetypes.guess_type('.'+attr)]
                        if mt]
        else:
            mtypes = self.types[attr]
        if mtypes:
            resp = _ResponseHelper(attr, mtypes, self)
            self.maybeadd(resp)
            return resp
        else:
            return None

def parseAccept(request):
    """
    Turn a request's HTTP_ACCEPT string into a list
    of mimetype/priority pairs.
    Includes a hack to work around an Opera weirdness.
    """
    strings = request.META['HTTP_ACCEPT'].split(',')
    r = re.compile(r'(.*?)(?:\s*;\s*q\s*\=\s*(.*))?$')
    types = []
    for s in strings:
        m = r.match(s)
        q = float(m.groups()[1]) if m.groups()[1] else 1.0
        t = m.groups()[0].strip()
        if t == 'opera/hack':
            break
        types.append((t, q))
    return types

Some HTML text utilities

May 8, 2008

I’ve just added some utilities to my website:

  • A converter that takes HTML entities to/from the characters they represent. The input can be actual characters (e.g þ), named entity references (&thorn;), or numeric references, decimal or hexadecimal. It also accepts some abbreviations (two back-ticks for “, for example), which I’ll eventually document. Maybe.
  • Lorem Ipsum text, with settable font, font size, and line height.
  • A list of named HTML entities. (Yes, that’s easy to find, but I wanted a place I could get to easily.)

[Yes, I know all these things are easily available many places. I wanted to be able to get to them without having to think about it, and to be able to fiddle with the details.]

I wrote these for a target user base of one—me—so there’s no particular reason to think that they’ll be useful for anyone else. They’re also in a bit of a raw and unfinished state (in which they’ll stay until I get around to doing something about it). But hey, use them if you like.

All the entity names and unicode descriptions come from the python unicodedata and htmlentitydefs libraries. I love the way python includes stuff like that.

Django QuerysetRefactor

April 27, 2008

In major Django news, Malcolm Tredinnick‘s long-awaited QuerysetRefactor branch is in for real; huzzah!  This has little immediate impact on my tiny site.   It did allow (and require, as I expected) me to remove the QLeftOuterJoin workaround from Django Snippets I used in a couple of places.  It also fixes other problems I’ve run into before—with ordering across relations, for example—and looks to be a major nicification in general.    I’m very impressed that so major an internals change could be done with so few backwards incompatibilities.

Authentication and Browser Caching in Django, part II

January 12, 2008

The other day I wrote about turning off browser caching when a user is logged in. Since I’m apparently a clueless n00b, it only occurred to me later that this is the sort of thing belongs in middleware. That way you don’t have to modify individual views, and it works for flatpages as well. Here’s the middleware; it should go in MIDDLEWARE_CLASSES before sessions and flatpages:

import re

def _add_to_header(response, key, value):
    if response.has_header(key):
        values = re.split(r'\s*,\s*', response[key])
        if not value in values:
            response[key] = ', '.join(values + [value])
    else:
        response[key] = value

def _nocache_if_auth(request, response):
    if request.user.is_authenticated():
        _add_to_header(response, 'Cache-Control', 'no-store')
        _add_to_header(response, 'Cache-Control', 'no-cache')
        _add_to_header(response, 'Pragma', 'no-cache')
    return response

class NoCacheIfAuthenticatedMiddleware(object):
    def process_response(self, request, response):
        try:
            return _nocache_if_auth(request, response)
        except:
            return response

Oh, and an annoying note: it’s still possible for firefox to keep an authenticated page cached, I can get that to happen with a sequence of Back and Reloads. Maybe that’s because the Back button is trying to respect history rather than the cache? Oh well, I told you not to mistake this for a security fix.

Authentication and browser caching in Django

January 10, 2008

While adding bits of authentication niceness to my website, I noticed a bit of ugliness. If I logged in, looked at a page that took account of the login, logged out, and hit Back in the browser, I still saw the logged-in page. That’s because the browser cached it, and just redisplayed on Back. I don’t care so much about caching non-authenticated views, but it just seems wrong to cache authenticated ones, so I have (I hope!) disabled it.

What disabling caching requires (here‘s a tutorial) is adding the directives Cache-Control:no-cache and Pragma:no-cache (different browsers may pay attention to one or the other!) to the html header, which is easily done. I have two methods for this; one is a replacement for render_to_response (which I was already using, to simplify using RequestContext), and the other is a wrapper for existing view functions, which can be used as a decorator. Code below.

Two important notes:

  • This should not be considered any sort of security fix.
  • I am not using the cache middleware; my site is way too small to need it. As far as I can tell this shouldn’t interact badly with the cache middleware, but sure don’t promise that.

Here’s the code:

def _add_to_header(response, key, value):
    if response.has_header(key):
        values = re.split(r'\s*,\s*', response[key])
        if not value in values:
            response[key] = ', '.join(values + [value])
    else:
        response[key] = value

def _nocache_if_auth(request, response):
    if request.user.is_authenticated():
        _add_to_header(response, 'Cache-Control', 'no-store')
        _add_to_header(response, 'Cache-Control', 'no-cache')
        _add_to_header(response, 'Pragma', 'no-store')
    return response

def rtr(request, *args, **kwargs):
    """
    If the request includes an authenticated user, disable browser caching.
    """
    response = render_to_response(
            context_instance=RequestContext(request),
            *args, **kwargs)
    return _nocache_if_auth(request, response)

def nocache_if_authenticated(fn):
    """
    Wrap the given view function so that browser caching is disabled
    with authenticated users.
    """
    def wrapped(request, *args, **kwargs):
        response = fn(request, *args, **kwargs)
        return _nocache_if_auth(request, response)
    return wrapped

UPDATE: D’oh! This should be middleware. I’ll implement that this weekend.

Django, AJAX, Scriptaculous

January 7, 2008

I looked a little at Scriptaculous this weekend.  I like it–it seems easy to use, and nice and concise.  It’s more focussed on glitzy effects (and I mean that in a good way) and less on widgets than the much larger DOJO; it also struck me (quite possibly wrongly, I’m just at the poking around stage) as easier to figure out and use.  Both effects and AJAX requests are wrapped pretty nicely; you have to write very little code to use them.

Unlike DOJO it does pollute the global js namespace a bit (as mentioned here); that might be annoying for people planning to use more of their own js than I’m ever likely to.

Here‘s what appears to be a nice scriptaculous-in-django tutorial, not that I’ve gone through it in any detail.

There seems to have been much debate in Django circles about whether to “include AJAX” in Django, which would (I assume) mean bundling a toolkit and providing a set of tags that wrap it, a la Rails and Scriptaculous.  The Django developers are reluctant to do that, not wanting to commit to a toolkit, and pointing out that there’s really not that much to wrap.  Now that I at least know what they’re talking about, I can sorta see their point.  Which doesn’t mean a some rails-like toolkit-wrapping tags wouldn’t be nice.
I’m working (in a slow and desultory fashion) on something where I might actually make some use of this stuff; so maybe I’ll be able to form an informed opinion in a few weeks.

Django and AJAX

January 3, 2008

The other day I fiddled a tiny bit with AJAX in Django (is that a nerdly way to spend New Year’s Day, or what?). I have no use at all for it now, but (i) I might someday, and (ii) it’s cool.

Brief summary, for n00bs like me. The “usual” way a browser communicates with a server is to send HttpRequests, to which the server responds with some response, generally a whole page of HTML which the browser loads. Loading a whole page, though, is inefficient if the browser only wants to change its existing page a little, and doesn’t need much new data to do it. So the point of AJAX is to ask only for the new data, and update the page in place. Also, the browser doesn’t need to wait for the data to come back. Instead, it can supply a callback function to do the updating whenever the data is available.

For a better example than mine of how this works in Django, see this.

It turns out to be a lot simpler than I had expected–I think I’ve spent more time writing about it than actually doing it. By using a javascript toolkit you can avoid the more disgusting details. I used Dojo, although really for my quick test it didn’t matter at all; I just needed the “xhrGET” function, an equivalent of which would be in any toolkit.

[What, by the way, should one look for in a js toolkit? I have very little idea.]

As an exercise, I rewrote the search results page from my music website, so that the search results would be updated in place in the “onchange” method of the search box. That’s a pretty silly use of AJAX, actually, and I won’t be putting it in the real site, but it served as practice.

So what to do? The existing page is is rendered by a search_results view, which finds all the objects (of models Work, Author, and Collection) that match the search criteria (passed in GET data), and renders them to a template. To AJAXify that, I added an onchange handler to the search input, which sends and XML request to retrieve the new search results, supplying a callback to update the page. The URL those requests go to is just the existing search page’s URL, with “xhr” added to the GET data. I could have created a new URL/view for that, but the existing one already did almost exactly what this needs; see below. In a more typical case, I think the same strategy would also work, but more logic in the view would be dependent on the type of the request.

I had to do very little to the view. The AJAX request is expected to return just the “guts” of the page–the lists of matches–which can be done just by using a different template.

It may be more typical, and in some ways is preferable, for AJAX requests to return not html but data in json format. In this case, that would require javascript code to attach the data to the html, which would duplicate the existing template code in a nasty way. Maybe there is some clever way to avoid that code duplication…

Anyway, here’s the code:

def search_results(request):
    works = []
    people = []
    collections = []
    searchwords = ""
    try:
        searchwords = request.GET['searchwords']
        works = utils.searchfields(Work, searchwords)
        people = utils.searchfields(Author, searchwords)
        collections = utils.searchfields(Collection, searchwords)
    except KeyError:
        pass

    if request.has_key('xhr'):
        templatename = 'music/searchresults.html'
    else:
        templatename = 'music/search.html'

    return render_to_response(
            templatename,
            { 'searchwords' : searchwords,
              'works' : works,
              'people' : people,
              'collections' : collections })

The template needed more work. First, of course, it needs a js function to respond to fire off the AJAX request. That I cribbed from the DOJO documentation. Then, to avoid code duplication, I refactored the page a bit. I abstracted the html code that displays the search results out into its own tag, used in the view function above, and put the tag inside a new div. The tag is just a simple inclusion/context tag, using the same template as the view above used for xhr requests. On a normal HTTP request the tag is rendered using the data passed in from the view function; on an AJAX/xhr request the view function generates exactly the same html as the tag does initially (if given the same searchwords), and the callback shoves it into the div. And that’s pretty much it. The code:

{% extends "music/base.html" %}
{% block scripts %}
<script type="text/javascript"
    src="http://o.aolcdn.com/dojo/1.0.0/dojo/dojo.xd.js"></script%gt;
<script type="text/javascript">
    function updateajax() {
      dojo.xhrGet( {
        url: "?searchwords=" + dojo.byId("searchwords").value + ";xhr",
        handleAs: "text",

        timeout: 5000, // Time in milliseconds

        load: function(response, ioArgs) {
          dojo.byId("searchresults").innerHTML = response;
          return response;
        },

        error: function(response, ioArgs) {
          console.error("HTTP status code: ", ioArgs.xhr.status);
          return response;
          }
        });
      }
</script>
{% endblock %}

{% block content_search %}
{% simple_search '/music/search/' searchwords %}
{% endblock %}

{% block content %}
<h2>Search Results</h2>
<div>
    {% searchresults %}
</div>
<script type="text/javascript">
    dojo.byId("searchwords").onchange = updateajax
</script>
{% endblock %}

Some thoughts on What I Learned From This to follow.

Django serialization

December 29, 2007

I figured out the django serialization problem I mentioned in the previous post, more or less. With unicode, json serialization needs a utf8-aware stream. XML serialization, oddly, needs to NOT have one (that part I don’t get). The easy workaround is to wrap the output stream in a StreamWriter. Here’s the relevant part of my dump script:

def dump_data(strm, format='json', indent=None):
    """
    Output the current contents of the database as a fixture of the given format
    """
    from django.db.models import get_apps, get_models
    from django.core import serializers

    app_list = get_apps()

    # Check that the serialization format exists; this is a shortcut to
    # avoid collating all the objects and _then_ failing.
    try:
        serializers.get_serializer(format)
    except KeyError:
        sys.stderr.write("Unknown serialization format: %s\n" % format)
        return

    objects = (o for app in app_list \
                    for model in get_models(app) \
                        for o in model.objects.all())
    try:
        # Bah!
        # json barfs without a codec
        # xml barfs WITH a codec
        if format == 'json':
            strm = codecs.getwriter('utf-8')(strm)
        serializers.serialize(format, objects,
                stream = strm,
                indent=indent, ensure_ascii=False)
    except Exception, e:
        sys.stderr.write("Unable to serialize database: %s\n" % e)