Drop-Thru Code Considered Harmful

Recently, I was describing my frustrations with some Python code that had been written by a now-departed co-worker.  I describe this code as "Drop-Thru Code".  The main characteristic is that it uses module scope for a non-trivial number of variables.

Module scope is functionally almost-global.  That is, it's so close to global you'll want to use it, but it's far enough away that you'll end up shooting yourself in the foot.

Consider this code snippet:

#!/bin/env python2.6
import os
import sys
varname = 33
vardict = { 'a' : 44 }
# other imports here
print "Starting!"
def something():
print "thing", varname
print "next"
something()
print "end"

This is what I call drop-thru code.  All the variables are global, and the call to something() will fail because varname is out of scope.

Contrast this with what I prefer, well-encapsulated code:

#!/bin/env python2.6
import os
import sys

class SomeThing(object):
    def __init__(self):
        self.varname = 33 
    def something(self):
        print "thing", self.varname
    def main(self):
        self.vardict = { 'a' : 44 }
        # other imports here
        print "Starting!"
        print "next"
        self.something()
        print "end" 

st = SomeThing()
st.main()

Nothing is global, it's obvious what the scope for things is.  Clean, beautiful, easy to understand.

There are lots of examples of scope problems.  I created this one for my team so they understood my frustration:

outside = 1
def function1():
    try:
        print "f1 outside: %s" % (outside)
        outside += 1
    except:
        print "no outside in function1."
print "a outside: %s" % (outside)
function1()                                     
print "b outside: %s" % (outside)
def function2():
    global outside
    try:
        print "f2 outside: %s" % (outside)
        outside += 1
    except:
        print "no outside in function2"
function2()
function2()
class Dum(object):
    def __init__(self):
        print "dum instantiated."
    def main(self):
        print "Dum main outside: %s" % (outside)
    def changeOutside(self, inval):
        outside = inval
        print "Dum changed: outside: %s" % (outside)
    def changeGlobalOutside(self, inval):
        global outside
        outside = inval
        print "Dum changed: outside: %s" % (outside)
d = Dum()
print "c outside: %s" % (outside)
d.main()
print "d outside: %s" % (outside)
d.changeOutside(33)
print "e outside: %s" % (outside)
d.main()
d.changeGlobalOutside(44)
d.main()
print "f outside: %s" % (outside)
______________________________________________________
output:
krice4@zaphod:~/checkouts/userSandboxes/krice4$ python scopeTest.py 
a outside: 1
no outside in function1.
b outside: 1
f2 outside: 1
f2 outside: 2
dum instantiated.
c outside: 3
Dum main outside: 3
d outside: 3
Dum changed: outside: 33
e outside: 3
Dum main outside: 3
Dum changed: outside: 44
Dum main outside: 44
f outside: 44

In short, Drop-Thru Code is considered harmful.  It allows for lots of scope problems that show up as bugs and frustrations later.  So, avoid them.  Put all the vars you can in a class and invoke that class, you'll be glad you did.  IMHO.

Subversion pre-commit hook script - Python files: prevent tabs, verify svn properties

Here's a precommit hook script I've modified from one I've used before.  I hope it comes in handy.  I'm going to try to submit it to the dev group of subversion itself for inclusion in the contrib/hook-scripts directory.

#!/bin/env python

"""
    pre-commit hook script that does several things:
    * prevents committing any python file containing a tab character.
    * checks if there are tabs in the source file and warns if so;
    * aborts if incorrect properties of eol-style and keywords 'id'.
"""
import sys
import os
import traceback
from optparse import OptionParser

#sys.stderr.write("NOTE:  pre-commit hook script enabled - checks for tabs, svn eol-style and id properties...\n")

def command_output(cmd):
    " Capture a command's standard output. "
    import subprocess
    return subprocess.Popen(
        cmd.split(), stdout=subprocess.PIPE).communicate()[0]

def files_changed(look_cmd):
    """ List the files added or updated by this transaction.

        "svnlook changed" gives output like:
          U   trunk/file1.cpp
          A   trunk/file2.py
    """
    def filename(line):
        return line[4:]

    def added_or_updated(line):
        return line and line[0] in ("A", "U")

    retval = []
    for line in command_output(look_cmd % "changed").split("\n"):
        if added_or_updated(line):
            retval.append(filename(line))
    #sys.stderr.write("files changed: %s" % (retval))
    return retval

def file_contents(filename, look_cmd):
    " Return a file's contents for this transaction. "
    return command_output("%s %s" % (look_cmd % "cat", filename))

def file_get_properties(filename, look_cmd):
    propslines = command_output("%s %s" % (look_cmd % "proplist -v", filename))
    res = {}
    for line in propslines.split('\n'):
        line = line.strip()
        if not line:
            continue
        k, v = line.split(' : ')
        res[k] = v
    return res

def contains_tabs(filename, look_cmd):
    " Return True if this version of the file contains tabs. "
    return "\t" in file_contents(filename, look_cmd)

def check_py_files(look_cmd):
    " Check Python files in this transaction are tab-free. "
   
    def is_py_file(fname):
        return os.path.splitext(fname)[1] == ".py"
   
    py_files_with_tabs    = set()
    py_files_bad_eolstyle = set()
    py_files_bad_exec     = set()
    py_files_bad_keywords = set()
    for ff in files_changed(look_cmd):
        if not is_py_file(ff):
            continue
        if contains_tabs(ff, look_cmd):
            py_files_with_tabs.add(ff)
        props = file_get_properties(ff, look_cmd)
       if props.get('svn:special'):
            sys.stderr.write("file %s has svn:special flag, probably a symlink. don't check other props." % ff)
            continue
        eolstyle = props.get('svn:eol-style')
        #sys.stderr.write("props: %s\neolstyle: '%s'\n" % (props, eolstyle))
        if eolstyle in ('native', 'LFFFFF'):
            py_files_bad_eolstyle.add(ff)
        execut   = props.get('svn:executable')
        if execut not in ['ON', '*']:
            py_files_bad_exec.add(ff)
        keywords = props.get('svn:keywords')
        if (not keywords) or ('Id' not in keywords.split()):
            py_files_bad_keywords.add(ff)

    preventCommit = False
    if len(py_files_with_tabs) > 0:
        sys.stderr.write("The following files contain tabs:\n%s\n"                                                                              % "\n".join(py_files_with_tabs))
        preventCommit = True
    if len(py_files_bad_exec) > 0:
        sys.stderr.write("The following py files are missing 'executable' property, committing anyway, but please fix this:\n%s\n"              % "\n".join(py_files_bad_exec))
        # note, do not prevent commit over this, just warn.
    if len(py_files_bad_keywords) > 0:
        sys.stderr.write("The following files don't have keywords property set to 'Id' at least.  Please fix this before committing:\n%s\n"     % "\n".join(py_files_bad_keywords))
        preventCommit = True
    if len(py_files_bad_eolstyle) > 0:
        sys.stderr.write("The following files don't have svn propset svn:eol-style 'LF', please do so before committing:\n%s\n"                 % "\n".join(py_files_bad_eolstyle))
        preventCommit = True

    return preventCommit

def main():
    usage = """usage: %prog REPOS TXN
        Run pre-commit options on a repository transaction."""

    parser = OptionParser(usage=usage)
    parser.add_option("-r", "--revision",
                      help="Test mode. TXN actually refers to a revision.",
                      action="store_true", default=False)
    errors = 0
    try:
        (opts, (repos, txn_or_rvn)) = parser.parse_args()
        look_opt = ("--transaction", "--revision")[opts.revision]
        look_cmd = "svnlook %s %s %s %s" % (
            "%s", repos, look_opt, txn_or_rvn)
        errors += check_py_files(look_cmd)
    except:
        parser.print_help()
        errors += 1
        sys.stderr.write("Pre-commit hook traceback: %s" % (traceback.format_exc()))
    return errors

if __name__ == "__main__":
    sys.exit(main())

Setting up Graphite / Django - MemoryError in Glyph

I'm setting up Graphite (under Django, of course) on a new box, and just saw the following error:

Exception Type: MemoryError
/opt/graphite/webapp/graphite/render/glyph.py in getExtents, line 221

specifically on the line: 

 F = self.ctx.font_extents()

turns out the problem was that I didn't have fonts installed.  New box, right?  Yah.

So, did apt-get install:

apt-get install cairo
apt-get install pycairo
apt-get install bitmap-fonts
 

The last one took a while.  But, now it's up and going!

BTW, the total list of apt-get I ran to get Graphite running on Centos-6.1 with python 2.6 was as follows:

apt-get install subversion
apt-get install httpd
apt-get install Django 
apt-get install mod_wsgi
apt-get install memcached
apt-get install python-devel
apt-get install  python-tools python-setuptools python-memcached python-pycurl
easy_install django-tagging
easy_install -U distribute
apt-get install libmysqlclient-dev
apt-get install mysql-libs
easy_install python-mysqldb
apt-get install MySQL-python
apt-get install cairo
apt-get install pycairo
apt-get install bitmap-fonts
 

Enjoy!

Oh, and if you are not satisfied with the quality of this posting, Complaints should be addressed to: 

Complaints Dept.
British Airways
Ingrams Drive 
Redditch, B79 5UT  
UNITED KINGDOM 

Too Smart for their Britches

I just ran into a bit of bash script that did this:

conflist=`cat www-conf/* |sort -u |sed ':a;N;$!ba;s/\n/ /g'`

On first glance, it's getting cat'ing all the files in a dir, sorting, and returning the result.  BUT WAIT, there's a funky sed script in there.

Sure, I get the 's/\n/ /g' -> get rid of intervening newlines.  But what of the other stuff?

It turns out, some smart-ass decided to code up some fancy schmantzy "I'm Smarter Than You" thumb-to-nose action.   How it works, courtesy of http://www.grymoire.com/Unix/Sed.html :

• The semicolons separate commands.
  
• The :a is a tag, so we can 'goto a' later.
  
• The N command says append lines together including their newline character;
  
  •  The "n" command will print out the current pattern space (unless the "-n" flag is used), empty the current pattern space, and read in the next line of input. The "N" command does not print out the current pattern space and does not empty the pattern space. It reads in the next line, but appends a new line character along with the input line itself to the pattern space.
• the $!ba command says, 'unless you're the last line of the file, branch to a (goto a).
  
  •   An easier way is to use the special character "$," which means the last line in the file.

So, the goal was to collapse all this data into a sorted list without newlines.  Note that if I echo $conflist, I see "a b c d e..." just like if I had left out those fancy branches.

What an utter waste of 10 minutes to figure this out, and another 10 to write this up for future reference and helping anyone else.

Questions:

• Will I ever use this construct?  NO.  I'll find some other way to do it.  If I even need to, given Bash collapses things onto one line with an echo anyway.
  
• Do I actively dislike the person who wrote this?  YES.  I don't know who it was, specifically.  But, it doesn't matter, I know what I need to know:  He wasted my time, and was what I would consider the professional equivalent of a braggart.  Code should be easy to read as a primary goal.
  

Note also, thanks for help:  http://www.catonmat.net/blog/sed-one-liners-explained-part-one/

Solved: MongoDB - How to add Binary data to a document using pymongo

I'm trying to add data to a MongoDB document. My data is tuples of doubles, and I'm adding to an array of them. This is a good example of how to add binary data to a mongodb document using pymongo. My doc looks like this (presume I have created it already):

$ mongo
mongos> use kevintest1
switched to db kevintest1
mongos> show collections
metricValue
system.indexes
mongos> db.metricValue.find()
{ "_id" : ObjectId("5159f82f64524c06f5cb1208"), "mtid" : 2, "seqno" : 0, "vals" : [ ] }
mongos> mtid2 = db.metricValue.findOne({ "mtid" : 2})
{
    "_id" : ObjectId("5159f82f64524c06f5cb1208"),
    "mtid" : 2,
    "seqno" : 0,
    "vals" : [ ]
}
mongos> mtid2
{
    "_id" : ObjectId("5159f82f64524c06f5cb1208"),
    "mtid" : 2,
    "seqno" : 0,
    "vals" : [ ]
}
mongos> mtid2.bsonsize()
Tue Apr  2 17:23:57 TypeError: mtid2.bsonsize is not a function (shell):1
mongos> Object.bsonsize(mtid2)
54
mongos> Object.bsonsize(db.metricValue.findOne({ "mtid" : 2}))
54

NOW, I take a break, and in a different window, I run the following program:

#!/usr/bin/python
import struct
import bson
import pymongo
from pymongo import Connection
from bson import Binary
from pprint import pprint, pformat
conn = pymongo.Connection('myboxname', 7333)
db = conn['kevintest1']
mvc = db.metricValue
print "have mvc.find: %s" % ([x for x in mvc.find()])
mybuffer = struct.pack("dd", 65535.7, 65535.8)
print "binary info to add:  %s" % (pformat(struct.unpack("dd", mybuffer)))
retval = mvc.update({ 'mtid': 2, 'seqno': 0}, { '$push': { 'vals' : Binary(mybuffer) }}, w=1)
print "retval of update: %s" % (retval)
print "After update, have mvc.find: %s" % ([x for x in mvc.find()]) 

So, I run it.  Then, I go back to the mongos window, and each time I run it, I check the bson size:

mongos> db.metricValue.findOne({ "mtid" : 2})
{
    "_id" : ObjectId("5159f82f64524c06f5cb1208"),
    "mtid" : 2,
    "seqno" : 0,
    "vals" : [
        BinData(0,"ZmZmZvb/70CamZmZ+f/vQA==")
    ]
} 

mongos> Object.bsonsize(db.metricValue.findOne({ "mtid" : 2}))
78
mongos> Object.bsonsize(db.metricValue.findOne({ "mtid" : 2}))
102
mongos> Object.bsonsize(db.metricValue.findOne({ "mtid" : 2}))
126

There you have it. 24 bytes per inserted pair of doubles. Without packing, it is 30 bytes, so I would be saving 24/30=12/15=4/5 => 20%.

Solved: Pymongo findAndModify example collection object not callable

I just blew an hour with a silly mistake (or two!)

I have an MongoDB collection with a bunch of documents I'm accessing via pymongo.  The docs have data in an array.  In the latest doc, I want to append an element to the array.  So, I use $push, this is normal.  Finding the right doc to modify, this was harder.

Below is the terminal session where I did this.  I made several mistakes, I hope you find it useful to see them.

The biggest one is that THERE IS NO findAndModify in pymongo.  It's called 'find_and_modify()'.   I kept getting the dreaded error: 

TypeError: 'Collection' object is not callable. If you meant to call the 'findAndUpdate' method on a 'Database' object it is failing because no such method exists.

This was reasonable - there is no such function. 

Other mistakes below might be useful, too, in that I think they're probably common.

[esm@myboxname:scripts]$ python
Python 2.6.6 (r266:84292, Jul 20 2011, 10:22:43) 
[GCC 4.4.5 20110214 (Red Hat 4.4.5-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pymongo import Connetion
>>> c = Connection('myboxname', 40000)
>>> mm = c.mymassivecollection
>>> ret = mm.delme2.insert({'metrictypeid': 20, 'start':1000, 'vals':[3,4]});
>>> ret
ObjectId('5150c9bd64524c04169fd612')
>>> mm.delme2.insert({'metrictypeid': 20, 'start':1010, 'vals':[3,4]});
ObjectId('5150c9cd64524c04169fd613')
>>> mm.delme2.insert({'metrictypeid': 20, 'start':1020, 'vals':[3,4]});
ObjectId('5150c9d264524c04169fd614')
>>> mm.delme2.insert({'metrictypeid': 20, 'start':1030, 'vals':[3,4]});
ObjectId('5150c9d764524c04169fd615')
>>> mm.delme2.insert({'metrictypeid': 10, 'start':1030, 'vals':[3,4]});
ObjectId('5150c9dd64524c04169fd616')
>>> mm.delme2.find()

>>> [x for x in mm.delme2.find()]
[{u'vals': [3, 4], u'start': 1000, u'_id': ObjectId('5150c9bd64524c04169fd612'), u'metrictypeid': 20}, {u'vals': [3, 4], u'start': 1010, u'_id': ObjectId('5150c9cd64524c04169fd613'), u'metrictypeid': 20}, {u'vals': [3, 4], u'start': 1020, u'_id': ObjectId('5150c9d264524c04169fd614'), u'metrictypeid': 20}, {u'vals': [3, 4], u'start': 1030, u'_id': ObjectId('5150c9d764524c04169fd615'), u'metrictypeid': 20}, {u'vals': [3, 4], u'start': 1030, u'_id': ObjectId('5150c9dd64524c04169fd616'), u'metrictypeid': 10}]
>>> import pprint
>>> pprint.pprint([x for x in mm.delme2.find()])
[{u'_id': ObjectId('5150c9bd64524c04169fd612'),
  u'metrictypeid': 20,
  u'start': 1000,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9cd64524c04169fd613'),
  u'metrictypeid': 20,
  u'start': 1010,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9d264524c04169fd614'),
  u'metrictypeid': 20,
  u'start': 1020,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9d764524c04169fd615'),
  u'metrictypeid': 20,
  u'start': 1030,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9dd64524c04169fd616'),
  u'metrictypeid': 10,
  u'start': 1030,
  u'vals': [3, 4]}]
>>> mm.runCommand( { 'findAndModify':'delme2', 'query': {'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1401, in __call__
    self.__name)
TypeError: 'Collection' object is not callable. If you meant to call the 'runCommand' method on a 'Database' object it is failing because no such method exists.
>>> mm.delme2.runCommand( { 'findAndModify':'delme2', 'query': {'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1405, in __call__
    self.__name.split(".")[-1])
TypeError: 'Collection' object is not callable. If you meant to call the 'runCommand' method on a 'Collection' object it is failing because no such method exists.
>>> mm.command( { 'findAndModify':'delme2', 'query': {'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/database.py", line 395, in command
    msg, allowable_errors)
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/helpers.py", line 144, in _check_command_response
    raise OperationFailure(msg % details["errmsg"])
pymongo.errors.OperationFailure: command {'sort': {'start': 1}, 'query': {'metrictypeid': 20}, 'findAndModify': 'delme2', 'update': {'$push': {'vals': 5}}, 'new': True} failed: no such cmd: sort
>>> mm.command( { 'findAndModify':'delme2', 'query': {'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/database.py", line 395, in command
    msg, allowable_errors)
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/helpers.py", line 144, in _check_command_response
    raise OperationFailure(msg % details["errmsg"])
pymongo.errors.OperationFailure: command {'sort': {'start': 1}, 'query': {'metrictypeid': 20}, 'findAndModify': 'delme2', 'update': {'$push': {'vals': 5}}, 'new': True} failed: no such cmd: sort
>>> mm.delme2.findAndModify({'query': 'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
  File "", line 1
    mm.delme2.findAndModify({'query': 'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
                                                    ^
SyntaxError: invalid syntax
>>> mm.delme2.findAndModify({'query': {'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1405, in __call__
    self.__name.split(".")[-1])
TypeError: 'Collection' object is not callable. If you meant to call the 'findAndModify' method on a 'Collection' object it is failing because no such method exists.
>>> mm.delme2.findAndModify({query: {metrictypeid:20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'metrictypeid' is not defined
>>> mm.delme2.findAndModify({'query': {'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1405, in __call__
    self.__name.split(".")[-1])
TypeError: 'Collection' object is not callable. If you meant to call the 'findAndModify' method on a 'Collection' object it is failing because no such method exists.
>>> mm.delme2.findAndModify({'query': {'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1405, in __call__
    self.__name.split(".")[-1])
TypeError: 'Collection' object is not callable. If you meant to call the 'findAndModify' method on a 'Collection' object it is failing because no such method exists.
>>> mm.delme2.find()

>>> pprint([x for x in mm.delme2.find()])
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'module' object is not callable
>>> pprint.pprint([x for x in mm.delme2.find()])
[{u'_id': ObjectId('5150c9bd64524c04169fd612'),
  u'metrictypeid': 20,
  u'start': 1000,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9cd64524c04169fd613'),
  u'metrictypeid': 20,
  u'start': 1010,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9d264524c04169fd614'),
  u'metrictypeid': 20,
  u'start': 1020,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9d764524c04169fd615'),
  u'metrictypeid': 20,
  u'start': 1030,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9dd64524c04169fd616'),
  u'metrictypeid': 10,
  u'start': 1030,
  u'vals': [3, 4]}]
>>> mm.delme2.findAndModify({'metrictypeid':20}, { '$push' : { 'vals' : 5 }});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1405, in __call__
    self.__name.split(".")[-1])
TypeError: 'Collection' object is not callable. If you meant to call the 'findAndModify' method on a 'Collection' object it is failing because no such method exists.
>>> mm.delme2.findAndUpdate()
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1405, in __call__
    self.__name.split(".")[-1])
TypeError: 'Collection' object is not callable. If you meant to call the 'findAndUpdate' method on a 'Collection' object it is failing because no such method exists.
>>> mm.findAndUpdate()
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1401, in __call__
    self.__name)
TypeError: 'Collection' object is not callable. If you meant to call the 'findAndUpdate' method on a 'Database' object it is failing because no such method exists.
>>> db
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'db' is not defined
>>> mm.delme2.find_and_modify({'metrictypeid':20}, { '$push' : { 'vals' : 5 }});
{u'vals': [3, 4], u'start': 1000, u'_id': ObjectId('5150c9bd64524c04169fd612'), u'metrictypeid': 20}
>>> pprint.pprint([x for x in mm.delme2.find()])
[{u'_id': ObjectId('5150c9cd64524c04169fd613'),
  u'metrictypeid': 20,
  u'start': 1010,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9d264524c04169fd614'),
  u'metrictypeid': 20,
  u'start': 1020,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9d764524c04169fd615'),
  u'metrictypeid': 20,
  u'start': 1030,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9dd64524c04169fd616'),
  u'metrictypeid': 10,
  u'start': 1030,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9bd64524c04169fd612'),
  u'metrictypeid': 20,
  u'start': 1000,
  u'vals': [3, 4, 5]}]
>>> mm.delme2.find_and_modify(query={'metrictypeid':20}, update={'$push':{'vals':6}}, sort={'start':1},new=True);
/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py:1364: DeprecationWarning: Passing mapping types for `sort` is deprecated, use a list of (key, direction) pairs instead
  DeprecationWarning)
{u'vals': [3, 4, 5, 6], u'start': 1000, u'_id': ObjectId('5150c9bd64524c04169fd612'), u'metrictypeid': 20}
>>> mm.delme2.find_and_modify(query={'metrictypeid':20}, update={'$push':{'vals':6}}, sort=['start',1],new=True);
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1357, in find_and_modify
    kwargs['sort'] = helpers._index_document(sort)
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/helpers.py", line 68, in _index_document
    for (key, value) in index_list:
ValueError: too many values to unpack
>>> mm.delme2.find_and_modify(query={'metrictypeid':20}, update={'$push':{'vals':6}}, sort=[['start',1]],new=True);
{u'vals': [3, 4, 5, 6, 6], u'start': 1000, u'_id': ObjectId('5150c9bd64524c04169fd612'), u'metrictypeid': 20}
>>> mm.delme2.find_and_modify(query={'metrictypeid':20}, update={'$push':{'vals':7}}, sort=[['start',1]],new=True);
{u'vals': [3, 4, 5, 6, 6, 7], u'start': 1000, u'_id': ObjectId('5150c9bd64524c04169fd612'), u'metrictypeid': 20}
>>> mm.delme2.find_and_modify(query={'metrictypeid':20}, update={'$push':{'vals':7}}, sort=[['start',-1]],new=True);
{u'vals': [3, 4, 7], u'start': 1030, u'_id': ObjectId('5150c9d764524c04169fd615'), u'metrictypeid': 20}

Trying to build Python and a set of Python libraries on an AIX 6.1 box.

Starting with both gcc and IBM xlc compilers handy, this will be a set of notes, as opposed to a strict recipe. 

Have in /opt/freeware/bin:  gcc, make, tar.

Python notes:

    AIX:    A complete overhaul of the shared library support is now in
            place.  See Misc/AIX-NOTES for some notes on how it's done.
            (The optimizer bug reported at this place in previous releases
            has been worked around by a minimal code change.) If you get
            errors about pthread_* functions, during compile or during
            testing, try setting CC to a thread-safe (reentrant) compiler,
            like "cc_r".  For full C++ module support, set CC="xlC_r" (or
            CC="xlC" without thread support).
   
    AIX 5.3: To build a 64-bit version with IBM's compiler, I used the
            following:
   
            export PATH=/usr/bin:/usr/vacpp/bin
            ./configure --with-gcc="xlc_r -q64" --with-cxx="xlC_r -q64" \
                        --disable-ipv6 AR="ar -X64"
            make

So, I know a couple of things from previous adventures here.  First, I need working libraries for a variety of things:
* readline
* sqlite3
* zlib

LIBPATH=/opt/freeware/lib:/opt/freeware/lib64:/usr/lib:/usr/local/lib:/opt/recon/dcm/platforms/AIX/lib:/opt/recon/lib:

I've run into problems when compiling with xlc. The tests don't pass, and I get an exit status 11.  yuck.  More later.