Planet Jython

Ed Taekema: Firefox 3.1 - Beta 1 Love the Tab Switching View

2008-10-14T17:47:24+00:00

Take a quick peek at whats coming in Firefox 3.1 over at Lifehacker. The tab switching view is pretty awesome, although from the comments, it appears it isn’t quite all there yet. In the meantime, you can install the Showcase Firefox plugin. It isn’t nearly as nice looking but accomplishes the same thing, allowing you to visually select the tab you want to switch to by selecting thumbnail pictures of the contents of each tab.

Ed Taekema: I don’t get Twitter … but I’m trying

2008-10-14T17:00:44+00:00

Ok, so I’m late to the party and am having to play catchup with twitter. I don’t quite think I get it entirely yet, but will give it a go. I’ve setup a twitter account (fishandcross) and twitting(?) about topics similar to this blog and who knows what else. Does anyone have tips for a twitter noob?

Ed Taekema: Recycled Comment Spam

2008-10-14T16:38:54+00:00

Here is recycling I am not in favour of. I’m starting to see weird, almost-not-spam comments. The comment is good english and makes sense. They use some words in common with the post, and if it weren’t for the off-topic site linked to (only 1 per comment) and the age of the post it responded to, I probably wouldn’t have thought twice. It seems someone / something is grabbing legit comments from somewhere and reposting them to other sites. Askimet caught it and held it, but it didn’t catch it as spam. Seems someone is pretty set on destroying blog comments … Has anyone else seen this? Any ideas of how to catch these automatically with WordPress?

Ed Taekema: 4 Great Windows File Copy Tools

2008-10-14T15:29:04+00:00

Readers of this blog suggested some great tools for helping windows file copies (especially network file copies). To make sure these don’t get lost down in the comments, I thought it would make a good blog post … and help me find this again when I need it. So here are the reader suggested tools:

Ycopy - This is a great tool for moving lots of data from one disk to another. Its main intended use is for copying files from an old disk to a new target disk without stopping when an error occurs. The errors are added to a log file. At the end of copy you get a report of was not copied for further investigation.
Terracopy - This is a free for home users tool that promises fast file copies, retries copies upon failure, along with the ability to pause and resume copies.
Robocopy - Stands for Robust File Copy and is known for being able to handle network outages, resume incomplete / failed file transfers and an ability to copy a larger number of files than xcopy without crashing. Oh, and this is a Microsoft utility included in the Windows Resource kit and in Vista. There is also a gui available for it from Microsoft Technet.
cygwin + rsync - This is the old reliable solution from unix … but moved to windows. Rsync lets you synchronize individual files or entire directories. Its claim to fame is that it uses specially delta handling to ensure that file copies use a minimal amount of network. This is handy if you are continually copying a set of files, as only the deltas will go over the net. In order to use this, you will need to install cygwin, so this is the heavy weight of the 4 tools.

Thanks for the great tool suggestions.

Ed Taekema: Open Office 3.0 Officially Released

2008-10-14T13:39:59+00:00

The latest version of Open Office is now officially released. This is what we run on all our home computers, so I’ll e upgrading things later this week … the open office servers are struggling with the demand. Here are the things I am looking forward to in this release:

Office 2007 File format support
View Multiple Pages in Layout View for Writer
Margin Notes for review comments
Its faster than my current open office install

They had planned to add Google Docs like multiple user edit capability. Not sure if that made it in, but it would be pretty cool for those homework review sessions when I am on the road.

Philip Jenvey: Debugging the Jython compiler

2008-10-12T14:36:31+00:00

Debugging the Jython compiler (or most compilers for that matter) isn't the most straightforward process, so here I'll describe the typical steps I go through to figure it out.

The Problem

I've found a bug while trying out the SymPy project:

(jython)pjenvey@golgo13:~/src/python/sympy$ jython -c "import sympy"

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "sympy/__init__.py", line 18, in <module>
 from polys import *
File "sympy/polys/__init__.py", line 4, in <module>
 from polynomial import Poly, PolynomialError, SymbolsError, \
java.lang.VerifyError: (class: sympy/polys/polynomial$py, method: __mul__$52 signature: (Lorg/python/core/PyFrame;)Lorg/python/core/PyObject;) Accessing value from uninitialized register 9

java.lang.VerifyError means the JVM's bytecode verifier found a problem in the bytecode you've attempted to load. In this case the __mul__ method defined in the sympy.polys.polynomial module is the culprit. My blog may cut off some of the VerifyError's error message, it was: "(class: sympy/polys/polynomial$py, method: __mul__$52 signature: (Lorg/python/core/PyFrame;)Lorg/python/core/PyObject;) Accessing value from uninitialized register 9".

Steps

First I open up the polynomial Python module to see what this __mul__ function is doing. Luckily there is only one __mul__ defined. Unluckily it's a large method, almost 100 lines. Usually I'd take a good look at the Python code to flag likely problematic areas in the Jython compiler (like all the brand new 2.4 and 2.5 features we've been working on). Since this is a larger method I do a very quick skim and nothing catches my eye -- no funky with statements or generator expressions.

Then I take a quick look at the bytecode the compiler generated via javap, the JDK's bundled java class disassembler. Just to double check that I'm dealing with the same method the verifier is talking about. This 100 lines of Python code is translated to about 2000 lines of javap output.

So next I narrow down the test case: copy and paste the __mul__ function into its own module and cut it down until I have the smallest piece possible that reproduces the problem. Bisect, bisect, remove that and dedent, bisect, etc. Turns out to be this line of code:

q[sum([ x*y for x, y in zip(monom, degs) ])] = coeff

Interesting -- this is an old list comprehension call inside of an old method call inside of an old item assignment? Old as in the compiler has handled these expressions for a long time now, and they likely haven't been touched lately.

I can probably narrow this statement down even further, for example I doubt the method call to zip has any relevance. Finally I'm down to this:

a[[b for b, c in d]] = e

Removing the lisp comp or even the tuple unpack won't reproduce the bug. I've renamed the variables to make them easily identifiable to me in the bytecode.

This one line of code translates to roughly this javap output:

Compiled from "bad2.py"
public class bad2$py extends org.python.core.PyFunctionTable implements org.python.core.PyRunnable{
static final bad2$py self;

static final org.python.core.PyString _0;

static final org.python.core.PyCode f$0;

public org.python.core.PyObject f$0(org.python.core.PyFrame);
Code:
0:    aload_1
1:    ldc    #4; //String __file__
3:    getstatic    #10; //Field _0:Lorg/python/core/PyString;
6:    invokevirtual    #16; //Method org/python/core/PyFrame.setglobal:(Ljava/lang/String;Lorg/python/core/PyObject;)V
9:    aload_1
10:    iconst_1
11:    invokevirtual    #20; //Method org/python/core/PyFrame.setline:(I)V
14:    aload_1
15:    ldc    #22; //String e
17:    invokevirtual    #26; //Method org/python/core/PyFrame.getname:(Ljava/lang/String;)Lorg/python/core/PyObject;
20:    astore_2
21:    aload_1
22:    ldc    #28; //String a
24:    invokevirtual    #26; //Method org/python/core/PyFrame.getname:(Ljava/lang/String;)Lorg/python/core/PyObject;
27:    new    #30; //class org/python/core/PyList
30:    dup
31:    invokespecial    #34; //Method org/python/core/PyList."":()V
34:    dup
35:    ldc    #36; //String append
37:    invokevirtual    #41; //Method org/python/core/PyObject.__getattr__:(Ljava/lang/String;)Lorg/python/core/PyObject;
40:    astore_3
41:    aload_1
42:    ldc    #43; //String _[1_3]
44:    aload_3
45:    invokevirtual    #46; //Method org/python/core/PyFrame.setlocal:(Ljava/lang/String;Lorg/python/core/PyObject;)V
48:    aconst_null
49:    astore_3
50:    aload_1
51:    iconst_1
52:    invokevirtual    #20; //Method org/python/core/PyFrame.setline:(I)V
55:    aload_1
56:    ldc    #48; //String d
58:    invokevirtual    #26; //Method org/python/core/PyFrame.getname:(Ljava/lang/String;)Lorg/python/core/PyObject;
61:    invokevirtual    #52; //Method org/python/core/PyObject.__iter__:()Lorg/python/core/PyObject;
64:    astore_3
65:    goto    131
68:    aload    4
70:    iconst_2
71:    invokestatic    #58; //Method org/python/core/Py.unpackSequence:(Lorg/python/core/PyObject;I)[Lorg/python/core/PyObject;
74:    astore    5
76:    aload    5
78:    iconst_0
79:    aaload
80:    astore    6
82:    aload_1
83:    ldc    #60; //String b
85:    aload    6
87:    invokevirtual    #46; //Method org/python/core/PyFrame.setlocal:(Ljava/lang/String;Lorg/python/core/PyObject;)V
90:    aconst_null
91:    astore    6
93:    aload    5
95:    iconst_1
96:    aaload
97:    astore    6
99:    aload_1
100:    ldc    #62; //String c
102:    aload    6
104:    invokevirtual    #46; //Method org/python/core/PyFrame.setlocal:(Ljava/lang/String;Lorg/python/core/PyObject;)V
107:    aconst_null
108:    astore    6
110:    aload_1
111:    iconst_1
112:    invokevirtual    #20; //Method org/python/core/PyFrame.setline:(I)V
115:    aload_1
116:    ldc    #43; //String _[1_3]
118:    invokevirtual    #26; //Method org/python/core/PyFrame.getname:(Ljava/lang/String;)Lorg/python/core/PyObject;
121:    aload_1
122:    ldc    #60; //String b
124:    invokevirtual    #26; //Method org/python/core/PyFrame.getname:(Ljava/lang/String;)Lorg/python/core/PyObject;
127:    invokevirtual    #66; //Method org/python/core/PyObject.__call__:(Lorg/python/core/PyObject;)Lorg/python/core/PyObject;
130:    pop
131:    aload_1
132:    iconst_1
133:    invokevirtual    #20; //Method org/python/core/PyFrame.setline:(I)V
136:    aload_3
137:    invokevirtual    #69; //Method org/python/core/PyObject.__iternext__:()Lorg/python/core/PyObject;
140:    astore    4
142:    aload    4
144:    ifnonnull    68
147:    aload_1
148:    iconst_1
149:    invokevirtual    #20; //Method org/python/core/PyFrame.setline:(I)V
152:    aload_1
153:    ldc    #43; //String _[1_3]
155:    invokevirtual    #73; //Method org/python/core/PyFrame.dellocal:(Ljava/lang/String;)V
158:    aload    6
160:    invokevirtual    #77; //Method org/python/core/PyObject.__setitem__:(Lorg/python/core/PyObject;Lorg/python/core/PyObject;)V
163:    aconst_null
164:    astore_2
165:    aload_1
166:    iconst_m1
167:    putfield    #81; //Field org/python/core/PyFrame.f_lasti:I
170:    getstatic    #85; //Field org/python/core/Py.None:Lorg/python/core/PyObject;
173:    areturn

}

Tangent

"Accessing value from uninitialized register 9"

By the way JVM 1.5 -- you're a stack based virtual machine and you're bitching to me about unitialized registers? Sure I have bad bytecode, but registers are your problem buddy. I digress..

Tools of the Trade

I've been using javap to look at the actual bytecode we produce, but now I'm also going to look at what that bytecode looks like as Java code with jad, a Java decompiler. I usually start this process with javap because it's more reliable on bad byte code.

Luckily jad handles this one; here's what it looks like:

public PyObject f$0(PyFrame pyframe)
{
    pyframe.setglobal("__file__", _0);
    pyframe.setline(1);
    PyObject pyobject = pyframe.getname("e");
    PyList pylist = new PyList();
    PyObject pyobject1 = pylist.__getattr__("append");
    pyframe.setlocal("_[1_3]", pyobject1);
    pyobject1 = null;
    pyframe.setline(1);o
    pyobject1 = pyframe.getname("d").__iter__();
    PyObject pyobject3;
    do
    {
        pyframe.setline(1);
        pyobject2 = pyobject1.__iternext__();
        if(pyobject2 != null)
        {
            PyObject apyobject[] = Py.unpackSequence(pyobject2, 2);
            pyobject3 = apyobject[0];
            pyframe.setlocal("b", pyobject3);
            pyobject3 = null;
            pyobject3 = apyobject[1];
            pyframe.setlocal("c", pyobject3);
            pyobject3 = null;
            pyframe.setline(1);
            pyframe.getname("_[1_3]").__call__(pyframe.getname("b"));
        } else
        {
            break;
        }
    } while(true);
    pyframe.setline(1);
    pyframe.dellocal("_[1_3]");
    pyframe.getname("a").__setitem__(pylist, pyobject3);
    pyobject = null;
    pyframe.f_lasti = -1;
    return Py.None;
}

There's an oddity in the jad output: pyobject2 doesn't have any type information. Is that related to the bug I'm tracking down or is that a jad 'bug', or maybe both?

So with jad and javap (or even just soley javap) I myself can become a Java byte code verifier by walking through each instruction line by line and eventually (hopefully) noticing a problem.

Fortunately that isn't necessary with a good software byte code verifier. In this context the JVM's own bytecode verifier is *not* good. The error messages from the java.lang.VerifyErrors it throws are always pretty vague.

The ASM project has a great one called CheckClassAdapter. It can't detect every problem, but when it can, it's incredibly helpful. Here it goes:

(jython)pjenvey@golgo13:~/src/java/jython-trunk-clean3/asm-3.1/lib$ java -cp ~/src/java//jython-trunk-clean3/dist/jython.jar:asm-3.1.jar:asm-tree-3.1.jar:asm-analysis-3.1.jar:asm-util-3.1.jar  org.objectweb.asm.util.CheckClassAdapter /tmp/bad2\$py.class
org.objectweb.asm.tree.analysis.AnalyzerException: Error at instruction 90: Expected an object reference, but found .
at org.objectweb.asm.tree.analysis.Analyzer.analyze(Unknown Source)
at org.objectweb.asm.util.CheckClassAdapter.verify(Unknown Source)
at org.objectweb.asm.util.CheckClassAdapter.main(Unknown Source)
Caused by: org.objectweb.asm.tree.analysis.AnalyzerException: Expected an object reference, but found .
at org.objectweb.asm.tree.analysis.BasicVerifier.copyOperation(Unknown Source)
at org.objectweb.asm.tree.analysis.Frame.execute(Unknown Source)
... 3 more
f$0(Lorg/python/core/PyFrame;)Lorg/python/core/PyObject;
00000 Lbad2$py; PyFrame . . . . .  :  :    FRAME FULL [] []
00001 Lbad2$py; PyFrame . . . . .  :  :     ALOAD 1
00002 Lbad2$py; PyFrame . . . . .  : PyFrame  :     LDC "__file__"
00003 Lbad2$py; PyFrame . . . . .  : PyFrame String  :     GETSTATIC bad2$py._0 : Lorg/python/core/PyString;
00004 Lbad2$py; PyFrame . . . . .  : PyFrame String PyString  :     INVOKEVIRTUAL org/python/core/PyFrame.setglobal (Ljava/lang/String;Lorg/python/core/PyObject;)V
00005 Lbad2$py; PyFrame . . . . .  :  :     ALOAD 1
00006 Lbad2$py; PyFrame . . . . .  : PyFrame  :     ICONST_1
00007 Lbad2$py; PyFrame . . . . .  : PyFrame I  :     INVOKEVIRTUAL org/python/core/PyFrame.setline (I)V
00008 Lbad2$py; PyFrame . . . . .  :  :     ALOAD 1
00009 Lbad2$py; PyFrame . . . . .  : PyFrame  :     LDC "e"
00010 Lbad2$py; PyFrame . . . . .  : PyFrame String  :     INVOKEVIRTUAL org/python/core/PyFrame.getname (Ljava/lang/String;)Lorg/python/core/PyObject;
00011 Lbad2$py; PyFrame . . . . .  : PyObject  :     ASTORE 2
00012 Lbad2$py; PyFrame PyObject . . . .  :  :     ALOAD 1
00013 Lbad2$py; PyFrame PyObject . . . .  : PyFrame  :     LDC "a"
00014 Lbad2$py; PyFrame PyObject . . . .  : PyFrame String  :     INVOKEVIRTUAL org/python/core/PyFrame.getname (Ljava/lang/String;)Lorg/python/core/PyObject;
00015 Lbad2$py; PyFrame PyObject . . . .  : PyObject  :     NEW org/python/core/PyList
00016 Lbad2$py; PyFrame PyObject . . . .  : PyObject PyList  :     DUP
00017 Lbad2$py; PyFrame PyObject . . . .  : PyObject PyList PyList  :     INVOKESPECIAL org/python/core/PyList. ()V
00018 Lbad2$py; PyFrame PyObject . . . .  : PyObject PyList  :     DUP
00019 Lbad2$py; PyFrame PyObject . . . .  : PyObject PyList PyList  :     LDC "append"
00020 Lbad2$py; PyFrame PyObject . . . .  : PyObject PyList PyList String  :     INVOKEVIRTUAL org/python/core/PyObject.__getattr__ (Ljava/lang/String;)Lorg/python/core/PyObject;
00021 Lbad2$py; PyFrame PyObject . . . .  : PyObject PyList PyObject  :     ASTORE 3
00022 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList  :     ALOAD 1
00023 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList PyFrame  :     LDC "_[1_3]"
00024 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList PyFrame String  :     ALOAD 3
00025 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList PyFrame String PyObject  :     INVOKEVIRTUAL org/python/core/PyFrame.setlocal (Ljava/lang/String;Lorg/python/core/PyObject;)V
00026 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList  :     ACONST_NULL
00027 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList Lnull;  :     ASTORE 3
00028 Lbad2$py; PyFrame PyObject Lnull; . . .  : PyObject PyList  :     ALOAD 1
00029 Lbad2$py; PyFrame PyObject Lnull; . . .  : PyObject PyList PyFrame  :     ICONST_1
00030 Lbad2$py; PyFrame PyObject Lnull; . . .  : PyObject PyList PyFrame I  :     INVOKEVIRTUAL org/python/core/PyFrame.setline (I)V
00031 Lbad2$py; PyFrame PyObject Lnull; . . .  : PyObject PyList  :     ALOAD 1
00032 Lbad2$py; PyFrame PyObject Lnull; . . .  : PyObject PyList PyFrame  :     LDC "d"
00033 Lbad2$py; PyFrame PyObject Lnull; . . .  : PyObject PyList PyFrame String  :     INVOKEVIRTUAL org/python/core/PyFrame.getname (Ljava/lang/String;)Lorg/python/core/PyObject;
00034 Lbad2$py; PyFrame PyObject Lnull; . . .  : PyObject PyList PyObject  :     INVOKEVIRTUAL org/python/core/PyObject.__iter__ ()Lorg/python/core/PyObject;
00035 Lbad2$py; PyFrame PyObject Lnull; . . .  : PyObject PyList PyObject  :     ASTORE 3
00036 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList  :     GOTO L0
00037 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList  :    L1
00038 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList  :    FRAME FULL [bad2$py org/python/core/PyFrame org/python/core/PyObject org/python/core/PyObject org/python/core/PyObject] [org/python/core/PyObject org/python/core/PyList]
00039 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList  :     ALOAD 4
00040 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList PyObject  :     ICONST_2
00041 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList PyObject I  :     INVOKESTATIC org/python/core/Py.unpackSequence (Lorg/python/core/PyObject;I)[Lorg/python/core/PyObject;
00042 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList PyObject  :     ASTORE 5
00043 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject .  : PyObject PyList  :     ALOAD 5
00044 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject .  : PyObject PyList PyObject  :     ICONST_0
00045 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject .  : PyObject PyList PyObject I  :     AALOAD
00046 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject .  : PyObject PyList PyObject  :     ASTORE 6
00047 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject PyObject  : PyObject PyList  :     ALOAD 1
00048 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject PyObject  : PyObject PyList PyFrame  :     LDC "b"
00049 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject PyObject  : PyObject PyList PyFrame String  :     ALOAD 6
00050 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject PyObject  : PyObject PyList PyFrame String PyObject  :     INVOKEVIRTUAL org/python/core/PyFrame.setlocal (Ljava/lang/String;Lorg/python/core/PyObject;)V
00051 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject PyObject  : PyObject PyList  :     ACONST_NULL
00052 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject PyObject  : PyObject PyList Lnull;  :     ASTORE 6
00053 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList  :     ALOAD 5
00054 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList PyObject  :     ICONST_1
00055 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList PyObject I  :     AALOAD
00056 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList PyObject  :     ASTORE 6
00057 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject PyObject  : PyObject PyList  :     ALOAD 1
00058 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject PyObject  : PyObject PyList PyFrame  :     LDC "c"
00059 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject PyObject  : PyObject PyList PyFrame String  :     ALOAD 6
00060 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject PyObject  : PyObject PyList PyFrame String PyObject  :     INVOKEVIRTUAL org/python/core/PyFrame.setlocal (Ljava/lang/String;Lorg/python/core/PyObject;)V
00061 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject PyObject  : PyObject PyList  :     ACONST_NULL
00062 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject PyObject  : PyObject PyList Lnull;  :     ASTORE 6
00063 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList  :     ALOAD 1
00064 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList PyFrame  :     ICONST_1
00065 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList PyFrame I  :     INVOKEVIRTUAL org/python/core/PyFrame.setline (I)V
00066 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList  :     ALOAD 1
00067 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList PyFrame  :     LDC "_[1_3]"
00068 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList PyFrame String  :     INVOKEVIRTUAL org/python/core/PyFrame.getname (Ljava/lang/String;)Lorg/python/core/PyObject;
00069 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList PyObject  :     ALOAD 1
00070 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList PyObject PyFrame  :     LDC "b"
00071 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList PyObject PyFrame String  :     INVOKEVIRTUAL org/python/core/PyFrame.getname (Ljava/lang/String;)Lorg/python/core/PyObject;
00072 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList PyObject PyObject  :     INVOKEVIRTUAL org/python/core/PyObject.__call__ (Lorg/python/core/PyObject;)Lorg/python/core/PyObject;
00073 Lbad2$py; PyFrame PyObject PyObject PyObject PyObject Lnull;  : PyObject PyList PyObject  :     POP
00074 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList  :    L0
00075 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList  :    FRAME FULL [bad2$py org/python/core/PyFrame org/python/core/PyObject org/python/core/PyObject] [org/python/core/PyObject org/python/core/PyList]
00076 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList  :     ALOAD 1
00077 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList PyFrame  :     ICONST_1
00078 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList PyFrame I  :     INVOKEVIRTUAL org/python/core/PyFrame.setline (I)V
00079 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList  :     ALOAD 3
00080 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList PyObject  :     INVOKEVIRTUAL org/python/core/PyObject.__iternext__ ()Lorg/python/core/PyObject;
00081 Lbad2$py; PyFrame PyObject PyObject . . .  : PyObject PyList PyObject  :     ASTORE 4
00082 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList  :     ALOAD 4
00083 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList PyObject  :     IFNONNULL L1
00084 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList  :     ALOAD 1
00085 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList PyFrame  :     ICONST_1
00086 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList PyFrame I  :     INVOKEVIRTUAL org/python/core/PyFrame.setline (I)V
00087 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList  :     ALOAD 1
00088 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList PyFrame  :     LDC "_[1_3]"
00089 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList PyFrame String  :     INVOKEVIRTUAL org/python/core/PyFrame.dellocal (Ljava/lang/String;)V
00090 Lbad2$py; PyFrame PyObject PyObject PyObject . .  : PyObject PyList  :     ALOAD 6
00091 ?             :     INVOKEVIRTUAL org/python/core/PyObject.__setitem__ (Lorg/python/core/PyObject;Lorg/python/core/PyObject;)V
00092 ?             :     ACONST_NULL
00093 ?             :     ASTORE 2
00094 ?             :     ALOAD 1
00095 ?             :     ICONST_M1
00096 ?             :     PUTFIELD org/python/core/PyFrame.f_lasti : I
00097 ?             :     GETSTATIC org/python/core/Py.None : Lorg/python/core/PyObject;
00098 ?             :     ARETURN

The first column shown is the instruction number, the second the contents of local variable table, the third the contents of the operand stack, then finally the instruction.

#90 is that last ALOAD 6 instruction. This tells me the problem is in the prepartion for invoking the __setitem__ method (the following instruction), which is Jython's method for actually doing item assignment, the final step in the order of operations of this line of code.

Basically an item assignment of a[c] = e results in a Java method call of a.__setitem__(c, e).

The ALOAD 6 instruction pushes the local variable at index 6 onto the stack. That's preparing the second argument to the __setitem__ method -- which should be the right hand side of the statement, variable e.

However at that point there's nothing in the local variable 6 (represented by a '.' in CheckClassAdapter's output) -- not even a null, thus it is uninitialzed. Which explains the error message, though not totally the 'register 9' part.

My point earlier about the word 'register' is that the JVM is not a register based virtual machine, so any notion of machine registers should be abstracted away from a user of it. Though it's obvious that at least some local variables would be mapped onto the underlying real machine's registers for performance, so I'll cut that message some slack.

Comparing CheckClassAdapters' output vs jad's I see that jad's pyobject3 is (was) at index 6. That's not even the value we want to assign -- pyobject3 was a temporary variable used for collecting the items in the list comprehension.

To the compiler!

So what does this mean the compiler is doing? In particular, what is the item assignment code doing when calling __setitem__? Here's the method responsible for genearting the item assignment bytecode, from org.python.compiler.CodeCompiler. Ignore the yellow sections, they're not called in this case:

 @Override
 public Object visitSubscript(Subscript node) throws Exception {
        if (node.slice instanceof Slice) {
            return Slice(node, (Slice) node.slice);
        }

     expr_contextType ctx = node.ctx;
        if (node.ctx == expr_contextType.AugStore && augmode == expr_contextType.Store) {
            restoreAugTmps(node, 2);
            ctx = expr_contextType.Store;
        } else {
         visit(node.value);
         visit(node.slice);

            if (node.ctx == expr_contextType.AugStore && augmode == expr_contextType.Load) {
                saveAugTmps(node, 2);
                ctx = expr_contextType.Load;
            }
        }

     switch (ctx) {
        case Del:
            code.invokevirtual("org/python/core/PyObject", "__delitem__", "(" + $pyObj + ")V");
            return null;
        case Load:
            code.invokevirtual("org/python/core/PyObject", "__getitem__", "(" + $pyObj + ")" + $pyObj);
            return null;
        Store:
         code.aload(temporary);
         code.invokevirtual("org/python/core/PyObject", "__setitem__", "(" + $pyObj + $pyObj + ")V");
         return null;
     }
     return null;
 }

The sole argument passed to visitSubscript is an AST node. So the AST for this assignment is visited as:

item assignment (visitSubscript):
  value being e (visit(node.value) in visitSubscript),
  key being (visit(node.slice) in visitSubscript) ->
    list comp ->
      tuple unpack

Our astview.py utility can show you the real AST nitty gritty:

(jython)pjenvey@golgo13:~/src/java/jython-trunk-clean3$ jython ast/astview.py ~/src/python/sympy/bad2.py

('Module',
 ('body',
  ('Assign (1,0)',
   ('targets',
    ('Subscript (1,0)',
     ('value', ('Name (1,0)', ('id', 'a'), ('ctx', ('Load',)))),
     ('slice',
      ('Index',
       ('value',
        ('ListComp (1,3)',
         ('elt', ('Name (1,3)', ('id', 'b'), ('ctx', ('Load',)))),
         ('generators',
          ('comprehension',
           ('target',
            ('Tuple (1,5)',
             ('elts',
              ('Name (1,9)', ('id', 'b'), ('ctx', ('Store',))),
              ('Name (1,12)', ('id', 'c'), ('ctx', ('Store',)))),
             ('ctx', ('Store',)))),
           ('iter',
            ('Name (1,17)', ('id', 'd'), ('ctx', ('Load',)))),
           ('ifs',))))))),
     ('ctx', ('Store',)))),
   ('value', ('Name (1,23)', ('id', 'e'), ('ctx', ('Load',)))))))

This means that some interaction with the list comp/tuple unpack has caused the item assignment to lose track of its state for __setitem__.

The bad ALOAD is generated by the line: code.aload(temporary);. Meaning the variable temporary is set to 6. The value we should be assigning, the e object, is one of the first things created in the generated method. We can see from the CheckClassAdapter output that e is stored at local 2:

00009 Lbad2$py; PyFrame . . . . .  : PyFrame  :     LDC "e"
00010 Lbad2$py; PyFrame . . . . .  : PyFrame String  :     INVOKEVIRTUAL org/python/core/PyFrame.getname (Ljava/lang/String;)Lorg/python/core/PyObject;
00011 Lbad2$py; PyFrame . . . . .  : PyObject  :     ASTORE 2
00012 Lbad2$py; PyFrame PyObject . . . .  :  :     ALOAD 1

(Index 2 being the PyObject after PyFrame in the last line)

So 6 should be 2. I print out the value of temporary from both before and after we visit the child list comp/tuple unpack nodes -- the lines visit(node.value); visit(node.slice); and as I guessed the values are 2 and 6, respectively.

The Fix

I'm not that familliar with this 'temporary' instance variable in the compiler. It doesn't seem to be the best design as a few different methods use it to store a local variable index which they'll reference later, or which another method will grab later (in this case temporary is already setup for our visitSubscript likely by visitAssign). With multiple methods using it a clash was bound to happen at some point in a Visitor like our compiler.

I briefly look at the lisp comp/tuple unpack bytecode generation code and wonder why it's overwriting the temporary variable on us, but I don't think I'm going to fix this there. Since temporary is correct before visiting the child nodes, I opt to save the correct value of temporary immediately at the beginning of visitSubscript, so I can use it later.

Cleaning up the temporary variable business would be preferable, but this will do for now. Our compiler could use a lot of cleanup but that's for the future and has already started. I do at least double check the other uses of this variable to ensure this isn't a problem elsewhere.

the fix:

Index: src/org/python/compiler/CodeCompiler.java
===================================================================
--- src/org/python/compiler/CodeCompiler.java (revision 5379)
+++ src/org/python/compiler/CodeCompiler.java (working copy)
@@ -1604,6 +1604,7 @@
             return Slice(node, (Slice) node.slice);
         }
 
+        int value = temporary;
         expr_contextType ctx = node.ctx;
         if (node.ctx == expr_contextType.AugStore && augmode == expr_contextType.Store) {
             restoreAugTmps(node, 2);
@@ -1627,7 +1628,7 @@
             return null;
         case Param:
         case Store:
-            code.aload(temporary);
+            code.aload(value);
             code.invokevirtual("org/python/core/PyObject", "__setitem__", "(" + $pyObj + $pyObj + ")V");
             return null;
         }

Finally

Since this is old code, I double checked if the bug was present in the 2.2 release of Jython. Turns out it is, so this is probably a really old bug, possbily going back 8 years to Jython 2.0 when list comprehensions were first introduced.
.hll { background-color: #ffffcc } .c { color: #408080; font-style: italic } /* Comment */ .err { border: 1px solid #FF0000 } /* Error */ .k { color: #008000; font-weight: bold } /* Keyword */ .o { color: #666666 } /* Operator */ .cm { color: #408080; font-style: italic } /* Comment.Multiline */ .cp { color: #BC7A00 } /* Comment.Preproc */ .c1 { color: #408080; font-style: italic } /* Comment.Single */ .cs { color: #408080; font-style: italic } /* Comment.Special */ .gd { color: #A00000 } /* Generic.Deleted */ .ge { font-style: italic } /* Generic.Emph */ .gr { color: #FF0000 } /* Generic.Error */ .gh { color: #000080; font-weight: bold } /* Generic.Heading */ .gi { color: #00A000 } /* Generic.Inserted */ .go { color: #808080 } /* Generic.Output */ .gp { color: #000080; font-weight: bold } /* Generic.Prompt */ .gs { font-weight: bold } /* Generic.Strong */ .gu { color: #800080; font-weight: bold } /* Generic.Subheading */ .gt { color: #0040D0 } /* Generic.Traceback */ .kc { color: #008000; font-weight: bold } /* Keyword.Constant */ .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */ .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */ .kp { color: #008000 } /* Keyword.Pseudo */ .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */ .kt { color: #B00040 } /* Keyword.Type */ .m { color: #666666 } /* Literal.Number */ .s { color: #BA2121 } /* Literal.String */ .na { color: #7D9029 } /* Name.Attribute */ .nb { color: #008000 } /* Name.Builtin */ .nc { color: #0000FF; font-weight: bold } /* Name.Class */ .no { color: #880000 } /* Name.Constant */ .nd { color: #AA22FF } /* Name.Decorator */ .ni { color: #999999; font-weight: bold } /* Name.Entity */ .ne { color: #D2413A; font-weight: bold } /* Name.Exception */ .nf { color: #0000FF } /* Name.Function */ .nl { color: #A0A000 } /* Name.Label */ .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */ .nt { color: #008000; font-weight: bold } /* Name.Tag */ .nv { color: #19177C } /* Name.Variable */ .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */ .w { color: #bbbbbb } /* Text.Whitespace */ .mf { color: #666666 } /* Literal.Number.Float */ .mh { color: #666666 } /* Literal.Number.Hex */ .mi { color: #666666 } /* Literal.Number.Integer */ .mo { color: #666666 } /* Literal.Number.Oct */ .sb { color: #BA2121 } /* Literal.String.Backtick */ .sc { color: #BA2121 } /* Literal.String.Char */ .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */ .s2 { color: #BA2121 } /* Literal.String.Double */ .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */ .sh { color: #BA2121 } /* Literal.String.Heredoc */ .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */ .sx { color: #008000 } /* Literal.String.Other */ .sr { color: #BB6688 } /* Literal.String.Regex */ .s1 { color: #BA2121 } /* Literal.String.Single */ .ss { color: #19177C } /* Literal.String.Symbol */ .bp { color: #008000 } /* Name.Builtin.Pseudo */ .vc { color: #19177C } /* Name.Variable.Class */ .vg { color: #19177C } /* Name.Variable.Global */ .vi { color: #19177C } /* Name.Variable.Instance */ .il { color: #666666 } /* Literal.Number.Integer.Long *

Philip Jenvey: Jython @ JavaOne 2008

2008-10-12T04:13:05+00:00

I attended a couple days of JavaOne last week and luckily avoided the norovirus outbreak. However I did notice something else spreading through Moscone Center; interest in languages on the JVM. For example:

The number of sessions/BoFs covering Dynamic Language topics: around 8 on Groovy, 5 on JRuby, 2 on Jython and also a couple on Scala and Rhino each, with some very good turnouts. That's not even including CommunityOne. Also featured was a Scripting language bowl: a faceoff between JRuby, Groovy, Scala and Jython.

The JavaOne book store was stocked with just about every JRuby and Groovy book out there. Of the top 10 selling books at JavaOne, 3 were about languages on top of the JVM (Groovy and JavaFX). Wednesday's top 10 sellers also included JRuby committer Ola Bini's Practical JRuby on Rails as well as another Rails book. I'm hoping to see a whole lot of Python books on the shelf next year.

A number of Ruby/JRuby folks (and not just the ones employed by Sun either) told me that they are quite happy with NetBeans' support for Ruby. It has code completion and even some refactoring support. Tor Norbye and the NetBeans crew are now working on JavaScript support (Look Mom, JavaScript type inference) and are slated to add Python support next. Ted Leung has already begun talking to them about the details.

Ruby is still on the rise, and JRuby is a big contributing factor. JRuby definitely had the attention of many attendees and definitely has a growing userbase in Java land.

Groovy and the Groovy on Grails combo is also on the rise. linkedin.com, Sky TV and SAP's Composition on Grails Product are a few notable users of Grails. IBM's Project Zero (aka the WebSphere sMash product) is also utilizing Groovy, as well as their own PHP on the JVM implementation.

As for a language he'd use *now* on top of the JVM, except Java, James Gosling endorses Scala.

Sun's own new scripting lanuguage on top of the JVM, JavaFX, was of course all over the place.

John Rose: "I think we are on the right track here, letting the JVM grow independently of the Java language. (The language, if it has room to grow, will catch up.) James Gosling expressed a similar sentiment at his February Java Users Group talk “The Feel of Java, Revisited”, when he said he sometimes felt more interested in the future of the JVM than that of the Java language. “I don’t really care about the Java language. All the magic is in the JVM specification.” (Yes, I think that is hyperbolic. No, he is not abandoning Java.) I love both Java and the JVM, and I am pushing on the JVM this year."

The potential Java 7 new features. They're still potential because the Java 7 JSR isn't out yet. While some are nice (like John Rose's JSR 292, which will be a big help for JVM languages like Jython), some are arguably not nice. I hear more disdain than ever over where Java the language is going, which makes other languages on top of the JVM even more desirable.

I met many new people (more than I expected) who have used Jython at some point in their career, and some who are using it now (like pushToTest's Frank Cohen). They're all eager about its recent progress and can't wait for the 2.5 release.

Reminder: the Java world is large. Over 10,000 attendees, probably 10 times as many as this year's PyCon. All potential Python converts, right?

Ok, I mostly covered what I observed about different JVM languages at JavaOne, not just Jython. What I'm getting at here is that there's a lot of potential for the Jython 2.5 release (and all that code waiting around on Pypi to run on it) in this large Java ecosystem.

Ed Taekema: I Switched my Asus eee 900 to Ubuntu eee 8.04.1

2008-10-11T12:53:15+00:00

I’ve made the switch to Ubuntu for my Asus eee 900. I played around with the ubuntu-eee distribution a while ago and convinced myself that the install would work. What finally pushed me to do this what the very stale app repository for Xandros … So I backed up all my personal files to a 16 gb HC SD card, and used a USB stick to install Ubuntu eee 8.04.1.

Ubuntu eee 8.04.1 install and recovery media

So far I am enjoying the Netbook Remix interface, but may eventually switch back to full mode. Given that my Asus is not one of the Atom based units, I want to get the most performance out of the system so I’ll compare the two modes and take which ever one is fastest.

So far the only problem I’ve encountered is that automounting inserted SD cards (and maybe USB sticks) doesn’t work. The mount needs to be performed by root, then it all works nicely. The performance is really good, maybe even better than on Xandros, and the updated apps are great. This is where I’ll stay I think. But just in case, I still travel with a USB sticks with ubuntu and the recovery image in the unlikely event that I do need to go back to Xandros. Since these two sticks will be in my bag from now on, I put my label maker to good use…

Ed Taekema: The Low Carb Grocery - Finally a store in Toronto Area

2008-10-11T12:23:43+00:00

I don’t have to stuff my suitcase full of CarbQuick anymore! There is now a great Low carb grocery store in the Toronto area (Markham 404 & Steeles area for locals). They have a great selection of Low Carb products and are also connected to some pretty talented bakers because their line of very low carb cheese cake and 4 net carbs / slice whole wheat bread is amazing. Check them out at The Low Carb Grocery.

Ed Taekema: Vim Mode for Firefox - Perfect for Small Screens

2008-10-10T03:17:57+00:00

For those who are keyboard focused, now you can run Firefox using all the same key bindings as VIM. Take a look at vimperator, a VIM mode plugin for Firefox. When you install it, the menu bar, navigation bar and bookmarks are hidden, and all your surfing is controlled through a VIM like input area by the status bar.

Vimperator Vim Mode for Firefox

It has all the usual page navigation keys like j - k to scroll up and down, h - l for horizontal scrolling, and space bar to scroll down a page. They also have a neat way to highlight links with number that can then be selected via a command + the number to navigate to the link. The search capabilities are there as are using standard Vim commands to go the next match. There are also commands for navigating through the tabs, opening and closing tabs … all in all a very complete VIM experience for Firefox.

Hints let you quickly navigate to a link by entering a number and pressing Enter. Fast link mode is activated by pressing F … each link on the page gets a highlight number in front of it. Type that number and press enter and Firefox navigates to the link.

This is a pretty interesting idea. The only complaint I have is that it breaks the shortcut keys I use in Google’s GMail and Reader and of course if I am typing text into a textarea on an html form, it happily ignores all vimperator behavior. I think that is to be expected. I think I may keep this for a while. Its starting to grow on me. It may make it over to my Asus eee … this gives me about as much screen real estate as possible for actual content.

Ed Taekema: How to Hire A Technology Implementor

2008-10-10T00:54:18+00:00

David over at 37 Signals makes the following assertion:

You should never hire anyone for something you haven’t first struggled to do on your own. It’ll teach you most of what you need to know to actually interview candidates, it’ll allow you to understand the nature of the work better (do I even need to hire or can we outsource?), and you’ll know exactly what a job well done will look like. It’ll also give you a sense of whether the job is big enough for a full-time hire yet or if you can skimp by on your own (the latter is preferable if possible).

I think this is good advice for more than just a hiring situation. So many organizations purchase software tools and along with it a small dose of services from a vendor to get up and running. While I understand that purchasing services together with the product works well for both you and the vendor, there are some huge risks getting the services locked in so early. You can:

Underestimate the complexity of the implementation and run the tool into the ground.
Miss the opportunities to get additional value out of the tool because you are focused the ideas that generated the purchase and that is what the vendor builds for you.
Hire the wrong people for the implementation.
Under/Over estimate the amount of help you will need to implement a solution.

Here are some suggestions for how to do this a bit better.

If the solution involves new technology and skills, invest in limited up front help from outside the organization to get you proficient so you can try things yourself. Note the emphasis.
If you determine you need additional help, after you and your organization have a better feel for what needs to be done, then pursue external help from the vendor and other sources.

If you follow that approach you will know what you need, have an idea of what the finished solution will look like, and know when you are done. Believe it or not, the supplier will probably be happier with this kind of arrangement. We all like working for prople who know what they want and have a good feel for reality!

Ed Taekema: Another Windows Copy Utility - FastCopy

2008-10-09T16:18:29+00:00

With a name like that it has to be good. I ran across Fast Copy while looking for a solution to my failing windows file copies and it looks like a winner. It basically gets better performance for file copies since it doesn’t use the MFC at all. Instead it hits the Win32 api directly and can get near to actual device limit speeds for file copies.

Fastest File Copy on Windows

Try it out and let me know if it measures up to its claim to be the fastest file copy on windows.

Ed Taekema: Security Issues with Mozilla Lab’s Ubiquity

2008-10-09T00:21:34+00:00

Chris Soghoian has some pretty legitimate sounding security concerns with Ubiquity. I’ve blogged about Ubiquity before, and have it installed on my various Firefox installations, but I have to confess, I haven’t added any extra commands to it. Good thing too, since that could be a pretty serious security risk. Here are issues that Chris identifies:

There is no protection in Ubiquity for malicious JavaScript commands. The user is responsible for only installing code the trust, but most users will be unable to catch dangerous JavaScript just by reading it. If someone was really clever they could certainly write a command that looked pretty clean but was still dangerous.
Auto-Update of commands could make it easy for a malicious user to write a useful and popular command and once it was widely used, send out an update that was dangerous.
Updates for commands are also served over a non-encrypted http connection, which is vulnerable to being intercepted and used in a “man in the middle” style attack.

Ubiquity is certainly not finished and I am sure that Mozilla labs will move to address these issues. In the mean time, it’s proably a good idea to use with care.

Ed Taekema: Asus eee - Upgrade Solid State Storage

2008-10-08T16:52:11+00:00

Super Talent has line the availability of a new line of mini PCI-Express SSDs, in 16GB, 32GB and 64GB sizes. These are designed and tested with Asus eee.

The current pricing is $53 for the 16 Gig, $79 for the 32 Gig, and $149 for the 65 Gig models. So if you are running into storage limitations and want to expand your on board flash space, these look like the way to go.

via Liliputing

Ed Taekema: Make Windows File Share Copy Recoverable?

2008-10-08T13:55:25+00:00

I’m digging for a tool that will allow me to resume a failed file copy from a Windows file share. I have to move a bunch of data and am limited to using Windows SP sp2 to do it .. but I seem to get random failures at various points in the 220 meg file copy … Is there a tool out there that will somehow wrap the file copy and let it recover from such errors? Large file copies seem kinda like a gamble with windows …