Trying to build Python and a set of Python libraries on an AIX 6.1 box.

Starting with both gcc and IBM xlc compilers handy, this will be a set of notes, as opposed to a strict recipe. 

Have in /opt/freeware/bin:  gcc, make, tar.

Python notes:

    AIX:    A complete overhaul of the shared library support is now in
            place.  See Misc/AIX-NOTES for some notes on how it's done.
            (The optimizer bug reported at this place in previous releases
            has been worked around by a minimal code change.) If you get
            errors about pthread_* functions, during compile or during
            testing, try setting CC to a thread-safe (reentrant) compiler,
            like "cc_r".  For full C++ module support, set CC="xlC_r" (or
            CC="xlC" without thread support).
   
    AIX 5.3: To build a 64-bit version with IBM's compiler, I used the
            following:
   
            export PATH=/usr/bin:/usr/vacpp/bin
            ./configure --with-gcc="xlc_r -q64" --with-cxx="xlC_r -q64" \
                        --disable-ipv6 AR="ar -X64"
            make

So, I know a couple of things from previous adventures here.  First, I need working libraries for a variety of things:
* readline
* sqlite3
* zlib

LIBPATH=/opt/freeware/lib:/opt/freeware/lib64:/usr/lib:/usr/local/lib:/opt/recon/dcm/platforms/AIX/lib:/opt/recon/lib:

I've run into problems when compiling with xlc. The tests don't pass, and I get an exit status 11.  yuck.  More later.

MongoDB - How to Easily Pre-Split Chunks

Okay, so I'm doing some performance testing on MongoDB, spinning up a new configuration, and then throwing lots of data at it to see what the characteristics are.

Early on, I discovered that database locking was severely limiting performance.  MongoDB locks the database/collection (as of MongoDB 2.2.3) during every write operation.  However, it only locks it for the specific shard being written to.  So, massive numbers of writes dictates large numbers of shards, to minimize the lock/unlock bottleneck.  That's the theory.

In practice, this indeed proved correct.  We changed from a running (on 4 boxes - 2 primaries and 2 replicas) with 2 shards to running with 48 shards.  YES, they all magically cooperate nicely and share memory and CPU equally.  Actually, the boxes were all 196 GB memory w/ 32 cores each, so we weren't limited on machine capacity.  YMMV.

BTW, at peak operations, we're only seeing CPU of 50% per daemon, so we're not CPU limited. 

But, during my tests, I had to spin up first 4 shards, then 8, then 24, then 48.  Waiting for each test to settle into a steady state took time.  Each shard had to be active and equally used.  Likewise, there had to be enough data for it to be reading and writing at the same time, where the data was read out to send to the replicas.

The startup was handicapped by there being all activity on one shard, and not the others.  I would start things off, and all the activity would just be on 1 shard.  Since I was maxing out the capacity, there wasn't any spare IO available to do the splits and balancing.

To fix this, I found that I could add chunk split-points ahead of time, a process called 'pre-splitting chunks'.  To do this:

• Turn off the balancer.
• Run the command 'split' with a middle value, multiple times according to your number of shards.  
• Turn on the balancer again.

I had 48 shards on a key with range 000-999, so I pretended I had 50 shards and gave each shard 20.

No, you don't have to stop the database from doing anything, like failover to secondaries or anything like that.

So, my commands were:

$ mongo --port 99999999 --host myhost
> use config
> sh.stopBalancer();
> use admin
> db.runCommand({split: "myDb.myCollection" middle:{"key1":000,'key2':1}});

> db.runCommand({split: "myDb.myCollection" middle:{"key1":020,'key2':1}});
> ... (46+ more of these)
> use config
> sh.startBalancer();



NOTE:  Always leave the balancer off more than 1 minute or it doesn't count.  That is, if you're 'bouncing' the balancer, leave it off for a full 60+ seconds or the bounce doesn't do anything.

MongoDB Shard Stats One-Liner

I've been testing, and I want to get the shard stats in one line.  That is I want this:

    krice@zaphod:~/$ mongo --host boxname303.example.com --port 40000 --eval "sh.status()"
    MongoDB shell version: 2.0.6
    connecting to: boxname303.example.com:40000/test
    --- Sharding Status ---
      sharding version: { "_id" : 1, "version" : 3 }
      shards:
            {  "_id" : "shard-00",  "host" : "shard-00/boxname301.example.com:30001,boxname303.example.com:30000" }
            {  "_id" : "shard-01",  "host" : "shard-01/boxname302.example.com:30101,boxname304.example.com:30100" }
            {  "_id" : "shard-02",  "host" : "shard-02/boxname301.example.com:30201,boxname303.example.com:30200" }
            {  "_id" : "shard-03",  "host" : "shard-03/boxname302.example.com:30301,boxname304.example.com:30300" }
            {  "_id" : "shard-04",  "host" : "shard-04/boxname301.example.com:30401,boxname303.example.com:30400" }
            {  "_id" : "shard-05",  "host" : "shard-05/boxname302.example.com:30501,boxname304.example.com:30500" }
            {  "_id" : "shard-06",  "host" : "shard-06/boxname301.example.com:30601,boxname303.example.com:30600" }
            {  "_id" : "shard-07",  "host" : "shard-07/boxname302.example.com:30701,boxname304.example.com:30700" }
            {  "_id" : "shard-08",  "host" : "shard-08/boxname301.example.com:30801,boxname303.example.com:30800" }
            {  "_id" : "shard-09",  "host" : "shard-09/boxname302.example.com:30901,boxname304.example.com:30900" }
            {  "_id" : "shard-10",  "host" : "shard-10/boxname301.example.com:31011,boxname303.example.com:31010" }
            {  "_id" : "shard-11",  "host" : "shard-11/boxname302.example.com:31111,boxname304.example.com:31110" }
            {  "_id" : "shard-12",  "host" : "shard-12/boxname301.example.com:31211,boxname303.example.com:31210" }
            {  "_id" : "shard-13",  "host" : "shard-13/boxname302.example.com:31311,boxname304.example.com:31310" }
            {  "_id" : "shard-14",  "host" : "shard-14/boxname301.example.com:31411,boxname303.example.com:31410" }
            {  "_id" : "shard-15",  "host" : "shard-15/boxname302.example.com:31511,boxname304.example.com:31510" }
      databases:
            {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
            {  "_id" : "megamaid",  "partitioned" : true,  "primary" : "shard-00" }
                    megamaid.metricValue chunks:
                                    shard-05        2
                                    shard-13        2
                                    shard-10        2
                                    shard-15        2
                                    shard-01        2
                                    shard-08        2
                                    shard-04        2
                                    shard-07        2
                                    shard-12        2
                                    shard-03        2
                                    shard-09        2
                                    shard-06        2
                                    shard-11        2
                                    shard-14        2
                                    shard-00        2
                                    shard-02        5
                            too many chunks to print, use verbose if you want to force print
            {  "_id" : "test",  "partitioned" : false,  "primary" : "shard-14" }
    krice4@zaphod:~/$


But, I want just the numbers so I can see if I'm balanced.  Here's the one-liner:

alias shardbal="mongo --host boxname.example.com --port 40000 --eval \"sh.status()\" | egrep 'shard\-[0-9]{2}[^\"/]' | awk '{print \$2}' | tr '\n' ',' | xargs echo \"Shard Balance is: \" "
krice4@zaphod:~/$ shardbal
Shard Balance is:  4,4,4,3,4,4,4,4,4,4,4,4,4,4,4,4,

Postgres pgdb connect with connect timeout

So, here's a little FYI.

I'm trying to connect to a Postgres database and pass in a connection timeout. Whatever the default is seems to be way too short. This is via Python 2.7 on Linux Ubuntu 12.10, if you're interested, but I don't suppose it matters, I'll be doing the same thing in a moment on Centos 6.

I finally figured out how to do this on the command line:

psql connect_timeout=5 -h myhost.namegoeshere.com -p 5432 -U myname

But, I'm trying to do this programmatically from python using pgdb pgdb.connect(). Here's the command I'm trying:

    connString = "%s:%s:%s:%s:%s" % (self.pgServer, 'xxx', 'xxxzzzzx', 'xyyyy', "connect_timeout=5")
    db_connection = pgdb.connect(dsn=connString, host="myhost.namegoeshere.com :5432")
    db_cursor = db_connection.cursor()
    db_cursor.execute("SELECT 42;")
    if 42 not in db_cursor.fetchone():
        raise Exception("Test query failed")

No joy yet.

Sysstat make: o: Command not found

I was compiling sysstat for Centos 6 with a --prefix option, but kept getting a set of errors like:

make: o: Command not found make: [nls/sk.gmo] Error 127 (ignored) o nls/sr.gmo nls/sr.po make: o: Command not found make: [nls/sr.gmo] Error 127 (ignored) o nls/sv.gmo nls/sv.po make: o: Command not found make: [nls/sv.gmo] Error 127 (ignored) o nls/uk.gmo nls/uk.po make: o: Command not found make: [nls/uk.gmo] Error 127 (ignored) o nls/vi.gmo nls/vi.po make: o: Command not found make: [nls/vi.gmo] Error 127 (ignored) o nls/zh_CN.gmo nls/zh_CN.po make: o: Command not found make: [nls/zh_CN.gmo] Error 127 (ignored) o nls/zh_TW.gmo nls/zh_TW.po make: o: Command not found make: [nls/zh_TW.gmo] Error 127 (ignored) make distclean

Which really reduces to just "make: o: Command not found" repeated for lots of objects. What gives? I tried playing with CXX=g++ stuff, but it turned out the problem was only with the NLS support, which I disabled in the ./configure step, and the compile worked (note: still won't 'make test'). Successfull line:

./configure --prefix=/opt/recon/dcm/platforms/CENTOS6 --disable-documentation --disable-stripping --disable-sensors --disable-nls

And, voila, it works.

Compiling rsyslog for CENTOS 6.1

So, I was in dependency hell yesterday. It's one of the outer circles, nearby the region for people who drop gum on carpet. I was trying to get a recent version of rsyslog to install from source tgz on a centos 6.1 box. Unfortunately, the good people who have brought us rsyslog have put in requirements for libee and libestr, two not-production-ready libraries in the sense that they don't compile cleanly and easily with configure options that point to each other, etc. this is one of the concepts I started with, but it didn't turn out so well. [krice4@boxname rsyslog-7.1.7]$ ./configure LIBESTR_CFLAGS=/usr/local/include LIBESTR_LIBS=/usr/local/lib/libestr.so LIBEE_CFLAGS=/usr/local/include LIBEE_LIBS=/usr/local/lib/libee.so --enable-imfile --enable-mail --enable-extended-tests --enable-omstdout --enable-omprog --enable-zlib --enable-debug JSON_C_LIBS=/usr/lib/libjson.so JSON_C_CFLAGS=/usr/local/include After a while, i had to give up and install an RPM created by someone else, and even then, it broke the box on reboot so we had to re-image the vm. Ug. Advice: don't try to build 7.1 yourself without serious impact protection and a large bottle of 'Scotch' (or whatever your Scotch is) to take the edge off.

Mongo DB Map-Reduce for Group-By w/ extra fields

So, I was trying to do a query similar to the following:

select field_a, max(field_b) as maxb from tableA group by field_a

But, when I found field_a, I needed another field from that record. This would be straightforward in SQL with a subselect or join, but MongoDB on a sharded DB requires some Map-Reduce foo.

The problem is that reduce only allows you to return one value. But, I needed both the accumulation value AND another field to come back. Solved via this test program:

from pprint import pprint, pformat
from pymongo.code import Code
from pymongo import Connection

conn = Connection()
db  = conn["test"]
db.testcol.remove() 
db.testcol.insert({  "name" : "bob",  "val" : 5, "host":"blah1"})
db.testcol.insert({  "name" : "bob",  "val" : 3, "host":"blah2"})
db.testcol.insert({  "name" : "bob",  "val" : 7, "host":"blah3"})
db.testcol.insert({  "name" : "john", "val" : 2, "host":"blah4"})
db.testcol.insert({  "name" : "john", "val" : 4, "host":"blah5"})
db.testcol.insert({  "name" : "john", "val" : 8, "host":"blah6"})
db.testcol.insert({  "name" : "john", "val" : 6, "host":"blah7"})
#print "Data: "
#print pformat(list(db.testcol.find()))

mapfunc = Code(''' function() { emit( this.name, { name : this.name, val: this.val, host: this.host }); } ''')

redfunc = Code(''' function(key, values) { 
    var latestRec  = { val: 0, host: "", name: ""};
    for (var i=1; i < values.length; i++) {
        thisName = values[i].name
        thisVal  = values[i].val; 
        thisHost = values[i].host;
        if (thisVal > latestRec.val) { 
            latestRec.name = thisName
            latestRec.val  = thisVal;
            latestRec.host = thisHost;
            }
        }
    return latestRec
    } ''')

res = db.testcol.map_reduce(mapfunc, redfunc, "myresults")
for doc in res.find():
    print doc

print "*" * 80
conn.close()