By default, Django traffics in the single settings.py-configured timezone.

Timezone handling appear incomplete. The two options you’ll run across are this snippet for a timezone-localizing filter and django-timezones.

django-timezones has support for a TimeZone model/view field, and a model/view LocalizedDateTimeField. The View field, however, will only do the parse-time conversion of the datetime value in to the settings-defined TZ. This only solves half of the problem, since you still need to convert the settings-defined TZ into the user’s TZ, and there’s no component for that.

The following is a LocalizedDateTimeInput widget which will handle this last step, building on django.forms.DateTimeInput and django-timezones’ utility code:

from django import forms
from timezones.utils import adjust_datetime_to_timezone
 
class LocalizedDateTimeInput (forms.DateTimeInput):
    def __init__(self, tz):
        self._tz = tz
        super(LocalizedDateTimeInput, self).__init__()
 
    def render(self, name, value, attrs=None):
        if isinstance(value, datetime):
            value = adjust_datetime_to_timezone(value, 'UTC', self._tz)
        # @fixme: output the string rep of the timezone, probably after the <input />
        return super(LocalizedDateTimeInput, self).render(name, value, attrs)

Unfortunately, you’ll need to add a level of indirection to your view Form constructors to bind the ‘tz’ argument on the widget. Instead of a normal:

class MumbleForm (forms.Form):
    name = forms.CharField()
    date = forms.DateTimeField()

You’ll want to do something like:

def MumbleForm(user, *args, **kwargs):
    tz = settings.TIME_ZONE
    if user and hasattr(user, 'get_profile'):
        tz = user.get_profile().timezone
 
    class _MumbleForm (forms.Form):
        name = forms.CharField()
        date = LocalizedDateTimeField(tz, widget=LocalizedDateTimeInput(tz))
 
    return _MumbleForm(*args, **kwargs)

You can see more usage examples from the brew-journal commit.

Any tips or improvements welcome, of course.

Update, 2009-11-28: I finally got around to moving to using mysql instead of sqlite for this project, and discovered that the above solution needs one more piece. The default LocalizedDatetimeField seems to return datetimes which are “[tz] aware” in python’s vernacular, but mysql complains that datetimes with a tz portion are not allowed. So, we add a SafeLocalizedDateTimeField that returns “naive” datetimes instead:

class SafeLocalizedDateTimeField (LocalizedDateTimeField):
    def clean(self, value):
        val = super(SafeLocalizedDateTimeField, self).clean(value)
        if val is not None:
            val = val.replace(tzinfo=None)
        return val

There is a property, cargo.protocol, where you can specify https, however, the resulting server.xml that it generates will not be quite correct. A hack-fix is to edit org/codehaus/cargo/container/internal/resources/tomcat5x/server.xml in the jar file to include «SSLEnabled=”true”». As well, you might want to add the appropriate attributes for the relevant keyfile-path, otherwise, it seemed to use $(HOME)/.keystore, which I was happy to provide.

    for (Type var : someIterable) { ... }

As well, there is now an Iterable<t> interface, and Arrays are directly iterable, allowing you to write:

    String[] thingys = {"a","b","c"};
    for (String thingy : thingys) { ... }

At work, we have a collection of utility iterators, most written before these were available. As such, we have an ArrayIter utility, and a ZipIterator, inspired by Python’s itertools.izip.

I’ve been going through these classes and their usages on a lazy basis to update them to the new syntax. I finally got around to a usage of the ZipIterator, which happened to compose an ArrayIter … it zipped together an array of String names with the results of a test.

So, I changed it to:

    String[] names = { "foo", "bar", "baz" };
    List results;
    for (Object[] pair : new ZipIterator(names, results))
    {
    // ...

No dice, says Java:

/home/jsled/stuff/work/[...]TestMumble.java:46: cannot find symbol symbol  : constructor ZipIterator(java.lang.String[],java.util.List)
location: class com.spokesoftware.util.iterator.ZipIterator
for (Object pairObj : new ZipIterator(data, results))

WTF? Okay, let me help you out:

    for (Object[] pair : new ZipIterator((Iterable)names, results))
    // ...

FUCK YOU, says Java:

/home/jsled/stuff/work/[...]/TestMumble.java:46: inconvertible types
found   : java.lang.String[]
required: java.lang.Iterable
for (Object pairObj : new ZipIterator((Iterable)data, results))

It turns out that the string “iter” isn’t even in the text of the section about Arrays in the Java Language Spec. Array instances aren’t Iterable. They’re a special case in the handling of the for-each loop syntax.

This code ended up as:

    for (Object pair : new ZipIterator(Arrays.asList(names), results))

Java is totally shitrude.

edit 2009-10-10: added some formatting/syntax highlighting of the java blocks

The problem stems from use of the GateClassLoader and the dynamic creation of classes in loading JAPE grammars. Generally, classes are loaded into the Permanent Generation section of the heap, and are only eligible for garbage collection when the classloader from which they were sourced is itself unreachable. As the GateClassLoader is never reset or reinitialized, over time the classes will accumulate and fill the heap, and cause OutOfMemory errors in VM. While a first attempt to resolve this would be to cache the loaded JAPE objects, there are still classes created on each use of that grammar, which will slowly cause the same problem.

The caller needs a facility to reset the class loader, as provided by the patches to src/gate/Gate.java (and the compiler classes). It is expected that the caller invokes Gate.resetClassLoader() periodically, either on a fixed frequency or after a set number of uses of the library; our application resets every 15 minutes.

One complication is the difference in time between the generation/loading of the classes for a particular JAPE grammar and the instantiation of objects of those classes. Obviously, the objects needs to be instantiated from a valid class, and in particular from the classloader from which they were created; if the classloader is reset in the middle, there will be instantiation exceptions thrown when using a different classloader. A change was thus made to src/gate/jape/RightHandSide.java to retain a reference to the relevant GateClassLoader. Note, that there’s still a race condition present, but the window of time is significantly less than before.

Our (application-level) reset operation resets both the Gate classloader as well as invalidating our JAPE cache: current threads will finish using their JAPE objects, references to the classloader are eventually lost (from gate.Gate via reset and the jape RightHandSide objects through reference), and the garbage collector is allowed to collect the whole shebang.

This patch is much more of a workaround than a fix for the underlying problems. I don’t care if these patches are applied, so much as they’re useful in demonstrating the problem so it can be more appropriately fixed. Really, though, JAPE shouldn’t need to create classes per use.

• Patches against SVN:
• /2006/06/gate-svn-classloader.patch

• /2006/06/gate-svn-jape-rhs.patch

• Patches against 3.0 (B1846):

• /2006/06/gate-b1846-classloader.patch

• /2006/06/gate-b1846-jape-rhs.patch

• Previous memory issues thread

Also, while I’m at it, here are patches to:

• fix the mangled end of prj.el
• replace the ad-hoc chained exceptions with java’s

Update, 2006-07-10 I was pressed for detail about my claim that JAPE creates classes on use, and found that I couldn’t. Here’s my reply to gate-users:

I went to reproduce this, and found I can’t. Now I believe I mis-spoke.

The classes I remember being created were…

japeactionclasses.postprocessNewLineActionClass[#] japeactionclasses.postprocessVBnegActionClass[#] japeactionclasses.postprocesssimpleJoinActionClass[#]

…which are from the tokenizer’s grammar. Thinking back to the period I was debugging the issue, I don’t believe I was caching the tokenizers, but I had instituted the other JAPE caching. As well, I didn’t remember that the tokenizer was implemented in terms of JAPE. As such, I tricked myself into believing that the simple use of JAPE grammars from a different processing step was causing the loading of the aforementioned classes, when really it was just not caching the tokenizers.

One simple obvious way is logging, but I found that text-based statement logging doesn’t get me quite the level of detail and structure I want and ultimately need to be productive. When I was playing with RDF more heavily earlier last year, I started to format my log messages as N3 pseudo-statements:

** 2006-01-25 12:34:56 INFO Thread-0 Processor: [ object "foo"; count 12; disposition "thumbs-up" ] ** 2006-01-25 12:34:56 INFO Thread-0 Processor: [ object "bar"; count 7; disposition "thumbs-down" ]

Of course, the “real” form of this would be to have the entire logging pipeline emit actual N3 statements, including the logging-provided (meta-)data:

[ # a log:Entry log:ts "2006-01-25 12:34:56" ; log:lvl log:INFO ; log:thread "Thread-0" ; log:class "Processor" ; log:msg [ :object "foo"; :count 12; :disposition "thumbs-up" ] ].

N3’s great because of its simple syntax for representing structured data, but I’ve recently started using JSON instead, which has a more straightforward syntax and semantics for loose, non-RDF data representation. If you’re unfamiliar with JSON, it is a subset of javascript representing dictionaries, lists, and the basic data types (integers, floats, booleans and strings). An example would be:

{”favorite int”: 42, “floating value”: 42.01, “this is an example”: true, “a list of things”: [42, 42.01, true, {foo: bar, baz: quuz}]}

For example, I’ve been developing a content parser, and running it against a fixed set of test cases to assess the impact of changes I make. Each test result consists of 4 parts:

1. the test-case label (the input filename)
2. the expected-and-produced pairs
3. the expected-but-missed cases
4. the unexpected-but-produced cases

Each test-case and result is on the order of 10 “things”, so they’re well suited to manual review. The test run outputs a set of log statements like:

** 2006-01-25 12:34:56 INFO Thread-0 Testing: {”case”:”/path/to/test/case/file.name”,”expectedProduced”:[[«expected object», «produced object»], «…»], “expectedMissed”:[ «...» ], “unexpectedProduced”:[ «...» ]} ** 2006-01-25 12:34:57 INFO Thread-0 Testing: {”case”:”/path/to/test/case/file.name”,”expectedProduced”:[[{"foo":"bar"}, {"foo":"bar"}], «…»], “expectedMissed”:[{"foo":"baz"}], “unexpectedProduced”:[{"foo":"quux"]}

The process of logging general objects is made much easier by using a reflection-based utility class I wrote that will json-stringify random objects thrown at it, but that’s for another post…

A nice feature of JSON – being a subset of javascript – is that it is trivially parsed in javascript with an eval statement. As such, it’s simple to write an HTML page that accepts pasted JSON and renders it as nested tables. As this accepts and evaluates arbitrary javascript, I won’t link to it, but instead present the (mostly) short page here:

json:

decoded: