Compiling Python with Bz2

Building python from source on RHEL5, I've had some troubles. Python keeps not finding bz2, giving me the complaint:

Python build finished, but the necessary bits to build these modules were not found:
_tkinter           bsddb185           dl              
gdbm               imageop            sunaudiodev     
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

Failed to build these modules:
_curses            _curses_panel   _bz2

I already BUILT bz2 and installed it with the right prefix. UG.

FINALLY I found the following hint. Turns out bz2 source compile / make doesn't make everything by default. It just makes the executables bzunzip2, etc., but not the all-important (to python compiling) libbz2_so.

There's a file there, though, you just have to invoke it!

$ cd ~/temp
$ wget http://www.bzip.org/1.0.5/bzip2-1.0.5.tar.gz
$ tar zxvf bzip2-1.0.5.tar.gz
$ cd bzip2-1.0.5

next, TWO MAKES. INVOKE BOTH MAKEFILES:

$ make -f Makefile-libbz2_so
$ make

then, it works. NOTE: for the rest of the instructions, I like the page: http://rajaseelan.com/2012/01/28/installing-python-2-dot-7-2-on-centos-5-dot-2/

Calling Mail from command line - What works and doesn't

I had some trouble recently calling /bin/mail with -s for subject, quoting the actual subject in single or double quotes, and passing in the destination addresses in a shell / environment variable. So, here's the truth table of what works and what doesn't.

# Explanation of what works and doesn't when calling mail with various params:

#[krice@test1 krice]$ EMDEST="Kevin@justanyone.com"
#[krice@test1 krice]$ ps | mail -s 1132 $EMDEST          OK
#[krice@test1 krice]$ ps | mail -s 1132A "$EMDEST"       OK
#[krice@test1 krice]$ ps | mail -s "1132B" "$EMDEST"     OK
#[krice@test1 krice]$ ps | mail -s "1132C" '$EMDEST'     BAD
#[krice@test1 krice]$ EMDEST="Kevin\@justanyone.COM"
#[krice@test1 krice]$ ps | mail -s "1132D" '$EMDEST'     BAD
#[krice@test1 krice]$ ps | mail -s "1132E" $EMDEST       BAD
#[krice@test1 krice]$ EMDEST="Kevin@justanyone.COM"
#[krice@test1 krice]$ ps | mail -s "1132F" $EMDEST       OK

How to Pretty Print json.dumps() with indent, compactly

I'm working in Python and trying to pretty print some JSON objects. The trouble is, they print all weird. Indentation really messes up with TOO MANY LINES of data, one number per line. In other words, when I do this:

out = json.dumps(tfarray, sort_keys = True, indent = 2)

the indent=2 line causes each array element (in my case, a number) to be printed on its own line. This sucks and takes lots of space. Here's my program to pretty print the json.dumps result into something nicer:

#!/usr/bin/python
import re
instr = """    
       {
          "abbbb": 52,
          "acccc": 32220,
          "asdfasdfasfd": 32220,
          "asdfasdf": [
            0,
            1,
            2,
            3
            ],
          "somethingElse": 13,
          "asdfasdfasfd": [
            10,
            11,
            12,
            13
            ],
        }
"""
y = re.sub(r'\s\s+(\d+)', lambda match: r' {}'.format(match.group(1), instr), instr)
print "Y=%s" % (y)

the instr is the input string as pretty printed. The same sort of thing happens when I call pprint() or pformat() on a complex data structure - the result is hard to read. With the above re:sub, I get:

[krice@loadtest1 HD]$ python test_resub.py
Y=
       {
          "abbbb": 52,
          "acccc": 32220,
          "asdfasdfasfd": 32220,
          "asdfasdf": [ 0, 1, 2, 3
            ],
          "somethingElse": 13,
          "asdfasdfasfd": [ 10, 11, 12, 13
            ],
        }

Lots nicer, eh? Certainly more compact. Hurray for re.sub regex substitutions and lambda functions.

Python Robot Framework "FOR loop contains no keywords"

I've just been having a spot of trouble with Robot Framework and :FOR loops. The darn thing wouldn't recognize that the loop contained anything. Turns out, the problem is due to my using a .TXT file (text file with space delimited format). The :FOR loop didn't know how to parse the leading spaces into two empty columns. It had to presume there was one. With only one, the FOR loop contents were empty. This always leads to an error: "FOR loop contains no keywords". The solution for this is to just use pipe delimiters for that keyword. For instance, consider:

| Then I should see that it is below "${critical_value}" |
|   | Log               | In ThenIShouldSeethatItIsBelow keyword now.  | 
|   | ${table_lines}=   | Get Lines Containing String | ${full_output} | Table ${table} is served by  |
|   | Should Not Be Empty | ${table_lines} |
|   | @{lines}= | Split To Lines | ${table_lines}   |
|   | :FOR      | ${line} | IN | @{lines} |
|   |           | Log | havealine ${line} now. |
|   |           | ${value}=  | Fetch From Right | ${line} | ${SPACE} |
|   |           | Convert To Integer | ${value}                         |
|   |           | Should Be True     | ${value} <= ${critical_value}    |
|   | Log       | ending ThenIShouldSeethatItIsBelow now.  |

This allows the pipe delimiter to do its work. A more succinct example is:

| *Setting*  |     *Value*     |
| Library    | OperatingSystem |

| *Variable* |     *Value*     |
| ${MESSAGE} | Hello, world!   |

| *Test Case*  | *Action*        | *Argument*   |
| My Test      | [Documentation] | Example test |
|              | Log             | ${MESSAGE}   |
|              | My Keyword      | /tmp         |
| Another Test | Should Be Equal | ${MESSAGE}   | Hello, world! |
| TestTwo      |
|              | Log             | \nStarting   | WARN |
|              | MyForLoop       |
|              | Log             | \nEnding     | WARN |

| *Keyword*     |
| My Keyword    | [Arguments] | ${path}
|               | Directory Should Exist | ${path}
| MyForLoop     |
|               | Log | In myforloop now. | WARN |
|               | @{gvar}=  | Create List | a | b | c |
|               | Log | gvar is @{gvar} now. | WARN |
|               | :FOR  | ${x} | in | @{gvar} |
|               |       | Log | havealine ${x} now. | WARN |
|               | Log | ending myforloop now.  | WARN |

Enjoy your newfound power and use it responsibly. Or not.

I've recently run into the need to create a simple StatefulTelnetProtocol Telnet server. But, most of the examples have lots of extra cruft in the way. So, here's what I came up with by removing some extras from a post I found online: --> Note: I don't know the difference (yet) between doing a self.sendLine() and a self.transport.write() call. Feel free to comment if you know the distinction.

from twisted.internet.protocol import ServerFactory, Protocol
from twisted.conch.telnet import StatefulTelnetProtocol
from twisted.internet import reactor
from time import sleep

class MyProtocol(StatefulTelnetProtocol):
    
    def connectionMade(self):
        print "DEBUG: connectionMade called"
        self.sendLine("***************************************\r\n")
        self.sendLine("Welcome to the Simplified Telnet Server\r\n")
        self.sendLine("***************************************\r\n")
        sleep(2)
        self.transport.write("Can also send a line this way.\r\n")
        self.clearLineBuffer()
     
    def lineReceived(self, line):
        print "DEBUG: lineReceived called with %s" % line.strip()
        if line.strip() != "exit":
            self.sendLine("***************************************\r\n")
            #self.transport.write("TRANSPORT_WRITE: GOT LINE='%s'\n" % (str(line.strip())))
        else:
            print "telnet killed in line: %s" % str(line.strip())
            self.sendLine("TelnetShell killed. Bye Bye ...")
            self.transport.loseConnection()
            self.clearLineBuffer()
     
    def connectionLost(self, reason):
        print "DEBUG: connectionLost called with: %s" % str(reason)
     
def CreateMyFactory():
    factory = ServerFactory()
    factory.protocol = MyProtocol
    return factory
 
if __name__ == "__main__":
    MaFactory = CreateMyFactory()
    reactor.listenTCP(8023, MaFactory)
    reactor.run()

Python sdist package directory not found: Solved

Just was trying to create a new Python module and submit it to PyPi. Trouble was, when I was doing the standard:

me@mybox ~/myusername/blah
$ python setup.py sdist --formats=gztar,zip,bztar,ztar,tar
running sdist
error: package directory 'filenameX' does not exist

turns out the problem was, my setup.py had the following line:

py_modules = ['filenameX.py'],

This should be:

py_modules = ['filenameX'],

FIXED IT! Yay! Now it proceeds nicely.

Using ssh id_rsa without password - another username

Just had to login to another box via ssh, but the other account has a different username than my current one. Further, had to use a different id_dsa / id_dsa.pub set of files. I couldn't figure this out for a while. Turns out, the right way to do this is:

• generate the keys
• copy the pub key into authorized_keys2 on the remote box
• ssh -i ~/.ssh/privateKeyFileName otheruser@remoteboxname

Pretty straightforward, found out. Kinda cool - didn't know about the -i parameter.