Tuesday, November 13, 2012

Postgres pgdb connect with connect timeout

So, here's a little FYI.

I'm trying to connect to a Postgres database and pass in a connection timeout. Whatever the default is seems to be way too short. This is via Python 2.7 on Linux Ubuntu 12.10, if you're interested, but I don't suppose it matters, I'll be doing the same thing in a moment on Centos 6.

I finally figured out how to do this on the command line:

psql connect_timeout=5 -h myhost.namegoeshere.com -p 5432 -U myname

But, I'm trying to do this programmatically from python using pgdb pgdb.connect(). Here's the command I'm trying:

    connString = "%s:%s:%s:%s:%s" % (self.pgServer, 'xxx', 'xxxzzzzx', 'xyyyy', "connect_timeout=5")
    db_connection = pgdb.connect(dsn=connString, host="myhost.namegoeshere.com :5432")
    db_cursor = db_connection.cursor()
    db_cursor.execute("SELECT 42;")
    if 42 not in db_cursor.fetchone():
        raise Exception("Test query failed")


No joy yet.

Saturday, October 13, 2012

Sysstat make: o: Command not found

I was compiling sysstat for Centos 6 with a --prefix option, but kept getting a set of errors like:
make: o: Command not found make: [nls/sk.gmo] Error 127 (ignored) o nls/sr.gmo nls/sr.po make: o: Command not found make: [nls/sr.gmo] Error 127 (ignored) o nls/sv.gmo nls/sv.po make: o: Command not found make: [nls/sv.gmo] Error 127 (ignored) o nls/uk.gmo nls/uk.po make: o: Command not found make: [nls/uk.gmo] Error 127 (ignored) o nls/vi.gmo nls/vi.po make: o: Command not found make: [nls/vi.gmo] Error 127 (ignored) o nls/zh_CN.gmo nls/zh_CN.po make: o: Command not found make: [nls/zh_CN.gmo] Error 127 (ignored) o nls/zh_TW.gmo nls/zh_TW.po make: o: Command not found make: [nls/zh_TW.gmo] Error 127 (ignored) make distclean
Which really reduces to just "make: o: Command not found" repeated for lots of objects. What gives? I tried playing with CXX=g++ stuff, but it turned out the problem was only with the NLS support, which I disabled in the ./configure step, and the compile worked (note: still won't 'make test'). Successfull line:
./configure --prefix=/opt/recon/dcm/platforms/CENTOS6 --disable-documentation --disable-stripping --disable-sensors --disable-nls
And, voila, it works.

Wednesday, October 03, 2012

Compiling rsyslog for CENTOS 6.1

So, I was in dependency hell yesterday. It's one of the outer circles, nearby the region for people who drop gum on carpet. I was trying to get a recent version of rsyslog to install from source tgz on a centos 6.1 box. Unfortunately, the good people who have brought us rsyslog have put in requirements for libee and libestr, two not-production-ready libraries in the sense that they don't compile cleanly and easily with configure options that point to each other, etc. this is one of the concepts I started with, but it didn't turn out so well. [krice4@boxname rsyslog-7.1.7]$ ./configure LIBESTR_CFLAGS=/usr/local/include LIBESTR_LIBS=/usr/local/lib/libestr.so LIBEE_CFLAGS=/usr/local/include LIBEE_LIBS=/usr/local/lib/libee.so --enable-imfile --enable-mail --enable-extended-tests --enable-omstdout --enable-omprog --enable-zlib --enable-debug JSON_C_LIBS=/usr/lib/libjson.so JSON_C_CFLAGS=/usr/local/include After a while, i had to give up and install an RPM created by someone else, and even then, it broke the box on reboot so we had to re-image the vm. Ug. Advice: don't try to build 7.1 yourself without serious impact protection and a large bottle of 'Scotch' (or whatever your Scotch is) to take the edge off.

Thursday, August 16, 2012

Mongo DB Map-Reduce for Group-By w/ extra fields

So, I was trying to do a query similar to the following:

select field_a, max(field_b) as maxb from tableA group by field_a

But, when I found field_a, I needed another field from that record. This would be straightforward in SQL with a subselect or join, but MongoDB on a sharded DB requires some Map-Reduce foo.

The problem is that reduce only allows you to return one value. But, I needed both the accumulation value AND another field to come back. Solved via this test program:

from pprint import pprint, pformat
from pymongo.code import Code
from pymongo import Connection

conn = Connection()
db  = conn["test"]
db.testcol.remove() 
db.testcol.insert({  "name" : "bob",  "val" : 5, "host":"blah1"})
db.testcol.insert({  "name" : "bob",  "val" : 3, "host":"blah2"})
db.testcol.insert({  "name" : "bob",  "val" : 7, "host":"blah3"})
db.testcol.insert({  "name" : "john", "val" : 2, "host":"blah4"})
db.testcol.insert({  "name" : "john", "val" : 4, "host":"blah5"})
db.testcol.insert({  "name" : "john", "val" : 8, "host":"blah6"})
db.testcol.insert({  "name" : "john", "val" : 6, "host":"blah7"})
#print "Data: "
#print pformat(list(db.testcol.find()))

mapfunc = Code(''' function() { emit( this.name, { name : this.name, val: this.val, host: this.host }); } ''')

redfunc = Code(''' function(key, values) { 
    var latestRec  = { val: 0, host: "", name: ""};
    for (var i=1; i < values.length; i++) {
        thisName = values[i].name
        thisVal  = values[i].val; 
        thisHost = values[i].host;
        if (thisVal > latestRec.val) { 
            latestRec.name = thisName
            latestRec.val  = thisVal;
            latestRec.host = thisHost;
            }
        }
    return latestRec
    } ''')

res = db.testcol.map_reduce(mapfunc, redfunc, "myresults")
for doc in res.find():
    print doc

print "*" * 80
conn.close()

Saturday, July 21, 2012

SOLVED: How to Install Netgear WG311 under Ubuntu 12.04

Put Netgear WG311 in box. turn on box. follow directions at https://help.ubuntu.com/community/WifiDocs/Device/Netgear_WG311_v3 BUT: No matter how many times I rebooted, the sudo modprobe ndiswrapper kept giving me the message FATAL. Solved via:
kevin@odin:~$ sudo modprobe ndiswrapper
[sudo] password for kevin: 
FATAL: Module ndiswrapper not found.


kevin@odin:~$ sudo apt-cache search dkms
... (omitting extra stuff)....
dkms - Dynamic Kernel Module Support Framework
ndiswrapper-dkms - Source for the ndiswrapper Linux kernel module (DKMS)
... (omitting extra stuff)....

kevin@odin:~$ sudo apt-get install dkms
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  fakeroot patch
Suggested packages:
  diffutils-doc
The following NEW packages will be installed:
  dkms fakeroot patch
0 upgraded, 3 newly installed, 0 to remove and 9 not upgraded.
Need to get 247 kB of archives.
After this operation, 890 kB of additional disk space will be used.
Do you want to continue [Y/n]? Y
Get:1 http://mirror.anl.gov/pub/ubuntu/ precise/main patch i386 2.6.1-3 [86.0 kB]
Get:2 http://mirror.anl.gov/pub/ubuntu/ precise/main dkms all 2.2.0.3-1ubuntu3 [73.1 kB]
Get:3 http://mirror.anl.gov/pub/ubuntu/ precise/main fakeroot i386 1.18.2-1 [87.9 kB]
Fetched 247 kB in 0s (436 kB/s)
Selecting previously unselected package patch.
(Reading database ... 143936 files and directories currently installed.)
Unpacking patch (from .../patch_2.6.1-3_i386.deb) ...
Selecting previously unselected package dkms.
Unpacking dkms (from .../dkms_2.2.0.3-1ubuntu3_all.deb) ...
Selecting previously unselected package fakeroot.
Unpacking fakeroot (from .../fakeroot_1.18.2-1_i386.deb) ...
Processing triggers for man-db ...
Setting up patch (2.6.1-3) ...
Setting up dkms (2.2.0.3-1ubuntu3) ...
Setting up fakeroot (1.18.2-1) ...
update-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode.

kevin@odin:~$ sudo apt-get install dkms ndiswrapper-dkms
Reading package lists... Done
Building dependency tree       
Reading state information... Done
dkms is already the newest version.
The following NEW packages will be installed:
  ndiswrapper-dkms
0 upgraded, 1 newly installed, 0 to remove and 9 not upgraded.
Need to get 176 kB of archives.
After this operation, 778 kB of additional disk space will be used.
Do you want to continue [Y/n]? Y
Get:1 http://mirror.anl.gov/pub/ubuntu/ precise/universe ndiswrapper-dkms all 1.57-1ubuntu1 [176 kB]
Fetched 176 kB in 0s (285 kB/s)      
Selecting previously unselected package ndiswrapper-dkms.
(Reading database ... 144030 files and directories currently installed.)
Unpacking ndiswrapper-dkms (from .../ndiswrapper-dkms_1.57-1ubuntu1_all.deb) ...
Setting up ndiswrapper-dkms (1.57-1ubuntu1) ...
Loading new ndiswrapper-1.57 DKMS files...
First Installation: checking all kernels...
Building only for 3.2.0-23-generic-pae
Building initial module for 3.2.0-23-generic-pae
Done.

ndiswrapper:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.2.0-23-generic-pae/updates/dkms/

depmod.......

DKMS: install completed.
kevin@odin:~$ iwconfig
lo        no wireless extensions.

eth0      no wireless extensions.

kevin@odin:~$ sudo modprobe ndiswrapper
kevin@odin:~$ iwconfig
lo        no wireless extensions.

wlan0     IEEE 802.11g  ESSID:off/any  
          Mode:Managed  Channel:0  Access Point: Not-Associated   
          Bit Rate:1 Mb/s   Sensitivity=-200 dBm  
          RTS thr=2346 B   Fragment thr=2346 B   
          Power Management:off
          Link Quality:0  Signal level:0  Noise level:0
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:0   Missed beacon:0

eth0      no wireless extensions.

Wednesday, July 18, 2012

Just had to create a median function in Python. We compute median values by ordering them and choosing the middle one. I found a routine, but it lacked unittest coverage and the edge case of an empty input array. So, here goes:
import unittest
###############################################################################

def getMedian(numericValues):
    
    theValues = sorted(numericValues)
    if not theValues:
        return None
    
    if len(theValues) % 2 == 1:
        return theValues[(len(theValues)+1)/2-1]
    else:
        lower = theValues[len(theValues)/2-1]
        upper = theValues[len(theValues)/2]
  
        return (float(lower + upper)) / 2  

###############################################################################

if __name__ == '__main__':
    unittest.main()

###############################################################################

class TestCommonUtils(unittest.TestCase):

    def test_trivial(self):
        return True

    def isValid(self, functionName, valueShouldBe, valueIs):
        assert (valueShouldBe == valueIs), "ERROR: Function %s returned value %.6fs, but value should be: %.6f." % (functionName, valueIs, valueShouldBe)  

    def test_getMedian(self):
        self.isValid('getMedian', 2.5,      getMedian([0,1,2,3,4,5]))
        self.isValid('getMedian', 2,        getMedian([0,1,2,3,4]))
        self.isValid('getMedian', 2,        getMedian([3,1,2]))
        self.isValid('getMedian', 3,        getMedian([3,2,3]))
        self.isValid('getMedian', 1.234,    getMedian([1.234, 3.678, -2.467]))
        self.isValid('getMedian', 1.345,    getMedian([1.234, 3.678, 1.456, -2.467]))
        self.isValid('getMedian', 3,        getMedian([3]))
        self.isValid('getMedian', None,     getMedian([]))


$ nosetests util.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK

Wednesday, July 11, 2012

How to Build ZeroMQ (and Python binding) on AIX

Okay, so I may be stupid. I'm trying to run a Python script on an AIX box. In order to run it, I need a ZeroMQ (see http://zeromq.org). So, I'm trying to compile the zmq library using the standard compile-link-run thing.

BTW, this is AIX 6.1 using the gcc compiler [ TO START WITH ]

First, I had to get the sysadmins to install gcc and g++. then, I kept getting the error:

...
checking whether we are using clang C compiler... no
checking whether we are using gcc >= 4 C compiler... yes
checking whether the C++ compiler works... no
configure: error: Unable to find a working C++ compiler

Ug. Turns out, I didn't have libm installed. Some sysadmin time playing with puppet and NIM later, I have it. Now, I see:

...
checking alloca.h presence... yes
checking for alloca.h... yes
checking whether SOCK_CLOEXEC is supported... no
configure: creating ./config.status
./config.status: line 765: print: command not found
./config.status: line 765: print: command not found
./config.status: line 765: print: command not found
./config.status: line 765: print: command not found
... (times 200 or so)

Ah, fun with vi. Not vim, which tells you line numbers, just vi. Because IBM doesn't know from modern. So, I look up line number and find it's colon-number for go-to-line-number.

I look in the config.status line 765 (with :765), and I see it's referring to $ECHO shell var, I look for 'print' and find $ECHO is defined as:

...
pic_mode='default'
enable_fast_install='yes'
SHELL='/opt/freeware/bin/bash'
ECHO='print -r --'
host_alias=''
host='powerpc-ibm-aix6.1.0.0'
...

ECHO is 'print -r --' ????? Where did that come from?

So, I tried again, doing export ECHO=/usr/bin/echo first. No go. Next, tried with:

ECHO=/usr/bin/echo ./configure --prefix=/destination/directory

No go again.

-- time passes --

to fix this:

  1. in your ~/.profile or .bashrc, define:
    CONFIG_SHELL=/usr/bin/bash
  2. Then, invoke ./configure as follows:
    ${CONFIG_SHELL} ./configure --prefix=...

Continuing, I found out I had to install:

  • gnu automake
  • gnu libtool
  • gnu M4

Now, I'm stuck with:

[myuser@mybox ~/making/zeromq-2.2.0]: make
Making all in src
make[1]: Entering directory `/home/myuser/making/zeromq-2.2.0/src'
make  all-am
make[2]: Entering directory `/home/myuser/making/zeromq-2.2.0/src'
  CXX      libzmq_la-clock.lo
../libtool: line 880: X--tag=CXX: command not found
../libtool: line 913: libtool: ignoring unknown tag : command not found
../libtool: line 880: X--mode=compile: command not found
../libtool: line 1047: *** Warning: inferring the mode of operation is deprecated.: command not found
../libtool: line 1048: *** Future versions of Libtool will require --mode=MODE be specified.: command not found
../libtool: line 1191: Xg++: command not found
../libtool: line 1191: X-DHAVE_CONFIG_H: command not found
../libtool: line 1191: X-I.: command not found
../libtool: line 1191: X-pedantic: command not found
../libtool: line 1191: X-Werror: command not found
../libtool: line 1191: X-Wall: command not found
../libtool: line 1191: X-D_REENTRANT: command not found
../libtool: line 1191: X-D_THREAD_SAFE: command not found
../libtool: line 1191: X-DZMQ_FORCE_POLL: command not found
../libtool: line 1191: X-g: command not found
../libtool: line 1191: X-O2: command not found
../libtool: line 1191: X-MT: command not found
../libtool: line 1191: Xlibzmq_la-clock.lo: command not found
../libtool: line 1191: X-MD: command not found
../libtool: line 1191: X-MP: command not found
../libtool: line 1191: X-MF: command not found
../libtool: line 1191: X.deps/libzmq_la-clock.Tpo: No such file or directory
../libtool: line 1191: X-c: command not found
../libtool: line 1244: Xlibzmq_la-clock.lo: command not found
../libtool: line 1249: libtool: compile: cannot determine name of library object from `': command not found
make[2]: *** [libzmq_la-clock.lo] Error 1
make[2]: Leaving directory `/home/myuser/making/zeromq-2.2.0/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/myuser/making/zeromq-2.2.0/src'
make: *** [all-recursive] Error 1
[myuser@mybox ~/making/zeromq-2.2.0]: 
So, thinking I might not have a valid C/C++ compiler (silly, it should work), I download gcc source 4.2 and try to compile. Error message is:
checking for .preinit_array/.init_array/.fini_array support... no
checking if mkdir takes one argument... no
*** Configuration rs6000-ibm-aix not supported
make[2]: *** [configure-stage1-gcc] Error 1
make[2]: Leaving directory `/home/myuser/making/gcc-4.2.4/gcc-4.2.4'
make[1]: *** [stage1-bubble] Error 2
make[1]: Leaving directory `/home/myuser/making/gcc-4.2.4/gcc-4.2.4'
make: *** [all] Error 2
so out of luck there, too. Ug.

What to do? Punt.

I went to IBM's site and signed up for a free copy of their XLC compiler. I downloaded it and handed it over to the sysadmins, who did their magic to it. It comes in a package-thing that is only installable by IBM's installer, so this requires sysadmins.

Once I had it, I changed my env vars to be:

export PYTHONPATH=/opt/recon/dcm/platforms/AIX/pylibs:/opt/recon/dcm/platforms/AIX/pylibs/lib/python2.6/site-packages:$PYTHONPATH
export PATH=$PATH:/opt/recon/dcm/bin:/opt/recon/dcm/AIX/bin:/opt/recon/dcm/platforms/AIX/pylibs/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/recon/dcm/lib:/opt/recon/dcm/platforms/AIX/lib
export CONFIG_SHELL=/opt/freeware/bin/bash
export CONFIG_ENV_ARGS=/opt/freeware/bin/bash
export PS1="[\u@\H \w]: "

export CONFIG_SHELL=/opt/freeware/bin/bash
export CONFIG_ENV_ARGS=/opt/freeware/bin/bash
export CC=cc
export CFLAGS="-qmaxmem=16384 -DSYSV -D_AIX -D_AIX32 -D_AIX41 -D_AIX43 -D_AIX51 -D_ALL_SOURCE -DFUNCPROTO=15 -O -I/opt/freeware/include"
export CXX=xlC
export CXXFLAGS=$CFLAGS
export F77=xlf
export FFLAGS="-qmaxmem=16384 -O -I/opt/freeware/include"
export LD=ld
export LDFLAGS="-L/opt/freeware/lib -Wl,-bmaxdata:0x80000000"
export PATH=/opt/recon/dcm/platforms/AIX/bin:/usr/bin:/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/vac/bin:/usr/vacpp/bin:/usr/ccs/bin:/usr/dt/bin:/usr/opt/perl5/bin:/opt/freeware/bin:/opt/freeware/sbin:/usr/local/bin:/usr/lib/instl
then LOGGED OUT and RELOGGED-IN to pick up the env changes, and proceeded.

I then rebuilt libtool (because the aix version was way old) by doing the standard

${CONFIG_SHELL} ./configure --prefix=...
make
make check
make install
(and it worked!),

I re-untarred the zeromq library, and did:

${CONFIG_SHELL} ./configure --prefix=...
make
make check
but this failed. Problem was
make[1]: Entering directory `/opt/recon/making/zeromq-2.2.0/tests'
make  check-TESTS
make[2]: Entering directory `/opt/recon/making/zeromq-2.2.0/tests'
Could not load program /opt/recon/making/zeromq-2.2.0/tests/.libs/lt-test_pair_inproc:
Could not load module /opt/recon/dcm/platforms/AIX/lib/libzmq.a(libzmq.so.1).
        Dependent module /opt/recon/dcm/platforms/AIX/lib/libcrypto.a(libcrypto.so.1.0.0) could not be loaded.
        Member libcrypto.so.1.0.0 is not found in archive 
Could not load module lt-test_pair_inproc.
        Dependent module /opt/recon/dcm/platforms/AIX/lib/libzmq.a(libzmq.so.1) could not be loaded.
Could not load module .
FAIL: test_pair_inproc
Could not load program /opt/recon/making/zeromq-2.2.0/tests/.libs/lt-test_pair_tcp:
Could not load module /opt/recon/dcm/platforms/AIX/lib/libzmq.a(libzmq.so.1).
        Dependent module /opt/recon/dcm/platforms/AIX/lib/libcrypto.a(libcrypto.so.1.0.0) could not be loaded.
        Member libcrypto.so.1.0.0 is not found in archive 
Could not load module lt-test_pair_tcp.
...

Long story short, the fix for this is:

We have to add the /opt/freeware/lib to the LIBPATH **** FIRST ****. Otherwise, it can't know to find the libcrypto.so.1.0.0 there. You'll get this same error if /opt/freeware/lib is there but not first in line.
export LIBPATH="/opt/freeware/lib:/opt/freeware/lib64:/usr/lib:/usr/local/lib"
and
export LDFLAGS="-L/opt/freeware/lib -L/opt/freeware/lib64 -Wl,-bmaxdata:0x80000000"
then, success:
make[1]: Entering directory `/opt/recon/making/zeromq-2.2.0/tests'
make  check-TESTS
make[2]: Entering directory `/opt/recon/making/zeromq-2.2.0/tests'
PASS: test_pair_inproc
PASS: test_pair_tcp
PASS: test_reqrep_inproc
PASS: test_reqrep_tcp
PASS: test_hwm
PASS: test_shutdown_stress
PASS: test_pair_ipc
PASS: test_reqrep_ipc
test_timeo running...
PASS: test_timeo
==================
All 9 tests passed
==================
make[2]: Leaving directory `/opt/recon/making/zeromq-2.2.0/tests'

SUCCESS. Next, it's pyzmq.

Too much good luck now, I know this is going to fail soon.

... And I'm right. Ug. ... Okay, so (some time later) I've gotten it to compile. It kept giving me problems with:

    ******************************************
    building 'zmq.core._poll' extension
    cc -DNDEBUG -O -qmaxmem=16384 -DSYSV -D_AIX -D_AIX32 -D_AIX41 -D_AIX43 -D_AIX51 -D_ALL_SOURCE -DFUNCPROTO=15 -O -I/opt/freeware/include -I/opt/recon/dcm/platforms/AIX/include -I/opt/recon/dcm/platforms/AIX/include -Izmq/utils -Izmq/core -Izmq/devices -I/opt/freeware/include/python2.6 -c zmq/core/_poll.c -o build/temp.aix-6.1-2.6/zmq/core/_poll.o -Wno-unused-function -Wno-strict-aliasing
    cc: 1501-289 (W) Option -Wno-unused-function was incorrectly specified. The option will be ignored.
    cc: 1501-289 (W) Option -Wno-strict-aliasing was incorrectly specified. The option will be ignored.
    "zmq/utils/zmq_compat.h", line 22.11: 1506-189 (S) Floating-point constant 2.2.0 is not valid.
    "zmq/utils/zmq_compat.h", line 30.11: 1506-189 (S) Floating-point constant 3.0.0 is not valid.
    "zmq/utils/zmq_compat.h", line 70.15: 1506-189 (S) Floating-point constant 3.0.0 is not valid.
    "zmq/utils/zmq_compat.h", line 120.11: 1506-189 (S) Floating-point constant 4.0.0 is not valid.
    "zmq/utils/zmq_compat.h", line 132.15: 1506-189 (S) Floating-point constant 4.0.0 is not valid.
    "zmq/utils/zmq_compat.h", line 147.41: 1506-209 (S) Character constants must end before the end of a line.
    "zmq/utils/zmq_compat.h", line 147.31: 1506-076 (W) Character constant 's fd.hpp)' has more than 4 characters. No more than rightmost 4 characters are used.
    "zmq/utils/zmq_compat.h", line 1.1: 1506-046 (S) Syntax error.
    error: command 'cc' failed with exit status 1
there was nothing about this online. Doomed. ug. So, I peeked in the zmq/utils/zmq_compat.h file to see what it was objecting over. The lines were freaking comments. For some unknown inane stupid reason, the IBM cc (c compiler) objected to perfectly valid comment lines "// new in 3.3.0". So, I know how to edit out comment lines!!! I backed up the file and removed them. Success compiling! I see lots of copies to the prefix directory!

Now, to test: invoke python, import zmq...:

: python
Python 2.6.7 (r267:88850, Nov 22 2011, 20:59:18) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import zmq
Traceback (most recent call last):
  File "", line 1, in 
  File "zmq/__init__.py", line 26, in 
    from zmq.utils import initthreads # initialize threads
ImportError: cannot import name initthreads
>>> 
Yuck, but ... memory arises - Ah ha! I've seen this before. This happens BECAUSE I'M IN THE WRONG DIRECTORY. It's picking up something wrong because i"m in the directory I built pyzmq in. So, I go up one dir:
[esm@aixengsb301p ~/making/pyzmq-2.2.0]: cd ..
[esm@aixengsb301p ~/making]: python
Python 2.6.7 (r267:88850, Nov 22 2011, 20:59:18) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import zmq
>>> 
[esm@aixengsb301p ~/making]: 

SUCCESS!!!!

Monday, July 02, 2012

Compiling Python with Bz2

Building python from source on RHEL5, I've had some troubles. Python keeps not finding bz2, giving me the complaint:

Python build finished, but the necessary bits to build these modules were not found:
_tkinter           bsddb185           dl              
gdbm               imageop            sunaudiodev     
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

Failed to build these modules:
_curses            _curses_panel   _bz2

I already BUILT bz2 and installed it with the right prefix. UG.

FINALLY I found the following hint. Turns out bz2 source compile / make doesn't make everything by default. It just makes the executables bzunzip2, etc., but not the all-important (to python compiling) libbz2_so.

There's a file there, though, you just have to invoke it!

$ cd ~/temp
$ wget http://www.bzip.org/1.0.5/bzip2-1.0.5.tar.gz
$ tar zxvf bzip2-1.0.5.tar.gz
$ cd bzip2-1.0.5
next, TWO MAKES. INVOKE BOTH MAKEFILES:
$ make -f Makefile-libbz2_so
$ make
then, it works. NOTE: for the rest of the instructions, I like the page: http://rajaseelan.com/2012/01/28/installing-python-2-dot-7-2-on-centos-5-dot-2/

Wednesday, April 11, 2012

Calling Mail from command line - What works and doesn't

I had some trouble recently calling /bin/mail with -s for subject, quoting the actual subject in single or double quotes, and passing in the destination addresses in a shell / environment variable. So, here's the truth table of what works and what doesn't.
# Explanation of what works and doesn't when calling mail with various params:

#[krice@test1 krice]$ EMDEST="Kevin@justanyone.com"
#[krice@test1 krice]$ ps | mail -s 1132 $EMDEST          OK
#[krice@test1 krice]$ ps | mail -s 1132A "$EMDEST"       OK
#[krice@test1 krice]$ ps | mail -s "1132B" "$EMDEST"     OK
#[krice@test1 krice]$ ps | mail -s "1132C" '$EMDEST'     BAD
#[krice@test1 krice]$ EMDEST="Kevin\@justanyone.COM"
#[krice@test1 krice]$ ps | mail -s "1132D" '$EMDEST'     BAD
#[krice@test1 krice]$ ps | mail -s "1132E" $EMDEST       BAD
#[krice@test1 krice]$ EMDEST="Kevin@justanyone.COM"
#[krice@test1 krice]$ ps | mail -s "1132F" $EMDEST       OK

Thursday, April 05, 2012

How to Pretty Print json.dumps() with indent, compactly

I'm working in Python and trying to pretty print some JSON objects. The trouble is, they print all weird. Indentation really messes up with TOO MANY LINES of data, one number per line. In other words, when I do this:
out = json.dumps(tfarray, sort_keys = True, indent = 2)
the indent=2 line causes each array element (in my case, a number) to be printed on its own line. This sucks and takes lots of space. Here's my program to pretty print the json.dumps result into something nicer:
#!/usr/bin/python
import re
instr = """    
       {
          "abbbb": 52,
          "acccc": 32220,
          "asdfasdfasfd": 32220,
          "asdfasdf": [
            0,
            1,
            2,
            3
            ],
          "somethingElse": 13,
          "asdfasdfasfd": [
            10,
            11,
            12,
            13
            ],
        }
"""
y = re.sub(r'\s\s+(\d+)', lambda match: r' {}'.format(match.group(1), instr), instr)
print "Y=%s" % (y)
the instr is the input string as pretty printed. The same sort of thing happens when I call pprint() or pformat() on a complex data structure - the result is hard to read. With the above re:sub, I get:
[krice@loadtest1 HD]$ python test_resub.py
Y=
       {
          "abbbb": 52,
          "acccc": 32220,
          "asdfasdfasfd": 32220,
          "asdfasdf": [ 0, 1, 2, 3
            ],
          "somethingElse": 13,
          "asdfasdfasfd": [ 10, 11, 12, 13
            ],
        }
Lots nicer, eh? Certainly more compact. Hurray for re.sub regex substitutions and lambda functions.

Wednesday, February 01, 2012

Python Robot Framework "FOR loop contains no keywords"

I've just been having a spot of trouble with Robot Framework and :FOR loops. The darn thing wouldn't recognize that the loop contained anything. Turns out, the problem is due to my using a .TXT file (text file with space delimited format). The :FOR loop didn't know how to parse the leading spaces into two empty columns. It had to presume there was one. With only one, the FOR loop contents were empty. This always leads to an error: "FOR loop contains no keywords". The solution for this is to just use pipe delimiters for that keyword. For instance, consider:
| Then I should see that it is below "${critical_value}" |
|   | Log               | In ThenIShouldSeethatItIsBelow keyword now.  | 
|   | ${table_lines}=   | Get Lines Containing String | ${full_output} | Table ${table} is served by  |
|   | Should Not Be Empty | ${table_lines} |
|   | @{lines}= | Split To Lines | ${table_lines}   |
|   | :FOR      | ${line} | IN | @{lines} |
|   |           | Log | havealine ${line} now. |
|   |           | ${value}=  | Fetch From Right | ${line} | ${SPACE} |
|   |           | Convert To Integer | ${value}                         |
|   |           | Should Be True     | ${value} <= ${critical_value}    |
|   | Log       | ending ThenIShouldSeethatItIsBelow now.  |
This allows the pipe delimiter to do its work. A more succinct example is:
| *Setting*  |     *Value*     |
| Library    | OperatingSystem |

| *Variable* |     *Value*     |
| ${MESSAGE} | Hello, world!   |

| *Test Case*  | *Action*        | *Argument*   |
| My Test      | [Documentation] | Example test |
|              | Log             | ${MESSAGE}   |
|              | My Keyword      | /tmp         |
| Another Test | Should Be Equal | ${MESSAGE}   | Hello, world! |
| TestTwo      |
|              | Log             | \nStarting   | WARN |
|              | MyForLoop       |
|              | Log             | \nEnding     | WARN |

| *Keyword*     |
| My Keyword    | [Arguments] | ${path}
|               | Directory Should Exist | ${path}
| MyForLoop     |
|               | Log | In myforloop now. | WARN |
|               | @{gvar}=  | Create List | a | b | c |
|               | Log | gvar is @{gvar} now. | WARN |
|               | :FOR  | ${x} | in | @{gvar} |
|               |       | Log | havealine ${x} now. | WARN |
|               | Log | ending myforloop now.  | WARN |
Enjoy your newfound power and use it responsibly. Or not.