Tuesday, March 26, 2013

Solved: Pymongo findAndModify example collection object not callable


I just blew an hour with a silly mistake (or two!)

I have an MongoDB collection with a bunch of documents I'm accessing via pymongo.  The docs have data in an array.  In the latest doc, I want to append an element to the array.  So, I use $push, this is normal.  Finding the right doc to modify, this was harder.

Below is the terminal session where I did this.  I made several mistakes, I hope you find it useful to see them.

The biggest one is that THERE IS NO findAndModify in pymongo.  It's called 'find_and_modify()'.   I kept getting the dreaded error: 

TypeError: 'Collection' object is not callable. If you meant to call the 'findAndUpdate' method on a 'Database' object it is failing because no such method exists.

This was reasonable - there is no such function. 

Other mistakes below might be useful, too, in that I think they're probably common.

[esm@myboxname:scripts]$ python
Python 2.6.6 (r266:84292, Jul 20 2011, 10:22:43) 
[GCC 4.4.5 20110214 (Red Hat 4.4.5-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pymongo import Connetion
>>> c = Connection('myboxname', 40000)
>>> mm = c.mymassivecollection
>>> ret = mm.delme2.insert({'metrictypeid': 20, 'start':1000, 'vals':[3,4]});
>>> ret
ObjectId('5150c9bd64524c04169fd612')
>>> mm.delme2.insert({'metrictypeid': 20, 'start':1010, 'vals':[3,4]});
ObjectId('5150c9cd64524c04169fd613')
>>> mm.delme2.insert({'metrictypeid': 20, 'start':1020, 'vals':[3,4]});
ObjectId('5150c9d264524c04169fd614')
>>> mm.delme2.insert({'metrictypeid': 20, 'start':1030, 'vals':[3,4]});
ObjectId('5150c9d764524c04169fd615')
>>> mm.delme2.insert({'metrictypeid': 10, 'start':1030, 'vals':[3,4]});
ObjectId('5150c9dd64524c04169fd616')
>>> mm.delme2.find()

>>> [x for x in mm.delme2.find()]
[{u'vals': [3, 4], u'start': 1000, u'_id': ObjectId('5150c9bd64524c04169fd612'), u'metrictypeid': 20}, {u'vals': [3, 4], u'start': 1010, u'_id': ObjectId('5150c9cd64524c04169fd613'), u'metrictypeid': 20}, {u'vals': [3, 4], u'start': 1020, u'_id': ObjectId('5150c9d264524c04169fd614'), u'metrictypeid': 20}, {u'vals': [3, 4], u'start': 1030, u'_id': ObjectId('5150c9d764524c04169fd615'), u'metrictypeid': 20}, {u'vals': [3, 4], u'start': 1030, u'_id': ObjectId('5150c9dd64524c04169fd616'), u'metrictypeid': 10}]
>>> import pprint
>>> pprint.pprint([x for x in mm.delme2.find()])
[{u'_id': ObjectId('5150c9bd64524c04169fd612'),
  u'metrictypeid': 20,
  u'start': 1000,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9cd64524c04169fd613'),
  u'metrictypeid': 20,
  u'start': 1010,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9d264524c04169fd614'),
  u'metrictypeid': 20,
  u'start': 1020,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9d764524c04169fd615'),
  u'metrictypeid': 20,
  u'start': 1030,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9dd64524c04169fd616'),
  u'metrictypeid': 10,
  u'start': 1030,
  u'vals': [3, 4]}]
>>> mm.runCommand( { 'findAndModify':'delme2', 'query': {'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1401, in __call__
    self.__name)
TypeError: 'Collection' object is not callable. If you meant to call the 'runCommand' method on a 'Database' object it is failing because no such method exists.
>>> mm.delme2.runCommand( { 'findAndModify':'delme2', 'query': {'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1405, in __call__
    self.__name.split(".")[-1])
TypeError: 'Collection' object is not callable. If you meant to call the 'runCommand' method on a 'Collection' object it is failing because no such method exists.
>>> mm.command( { 'findAndModify':'delme2', 'query': {'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/database.py", line 395, in command
    msg, allowable_errors)
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/helpers.py", line 144, in _check_command_response
    raise OperationFailure(msg % details["errmsg"])
pymongo.errors.OperationFailure: command {'sort': {'start': 1}, 'query': {'metrictypeid': 20}, 'findAndModify': 'delme2', 'update': {'$push': {'vals': 5}}, 'new': True} failed: no such cmd: sort
>>> mm.command( { 'findAndModify':'delme2', 'query': {'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/database.py", line 395, in command
    msg, allowable_errors)
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/helpers.py", line 144, in _check_command_response
    raise OperationFailure(msg % details["errmsg"])
pymongo.errors.OperationFailure: command {'sort': {'start': 1}, 'query': {'metrictypeid': 20}, 'findAndModify': 'delme2', 'update': {'$push': {'vals': 5}}, 'new': True} failed: no such cmd: sort
>>> mm.delme2.findAndModify({'query': 'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
  File "", line 1
    mm.delme2.findAndModify({'query': 'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
                                                    ^
SyntaxError: invalid syntax
>>> mm.delme2.findAndModify({'query': {'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1405, in __call__
    self.__name.split(".")[-1])
TypeError: 'Collection' object is not callable. If you meant to call the 'findAndModify' method on a 'Collection' object it is failing because no such method exists.
>>> mm.delme2.findAndModify({query: {metrictypeid:20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'metrictypeid' is not defined
>>> mm.delme2.findAndModify({'query': {'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1405, in __call__
    self.__name.split(".")[-1])
TypeError: 'Collection' object is not callable. If you meant to call the 'findAndModify' method on a 'Collection' object it is failing because no such method exists.
>>> mm.delme2.findAndModify({'query': {'metrictypeid':20}, 'sort': { 'start' : 1}, 'update' : { '$push' : { 'vals' : 5 } }, 'new': True});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1405, in __call__
    self.__name.split(".")[-1])
TypeError: 'Collection' object is not callable. If you meant to call the 'findAndModify' method on a 'Collection' object it is failing because no such method exists.
>>> mm.delme2.find()

>>> pprint([x for x in mm.delme2.find()])
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'module' object is not callable
>>> pprint.pprint([x for x in mm.delme2.find()])
[{u'_id': ObjectId('5150c9bd64524c04169fd612'),
  u'metrictypeid': 20,
  u'start': 1000,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9cd64524c04169fd613'),
  u'metrictypeid': 20,
  u'start': 1010,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9d264524c04169fd614'),
  u'metrictypeid': 20,
  u'start': 1020,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9d764524c04169fd615'),
  u'metrictypeid': 20,
  u'start': 1030,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9dd64524c04169fd616'),
  u'metrictypeid': 10,
  u'start': 1030,
  u'vals': [3, 4]}]
>>> mm.delme2.findAndModify({'metrictypeid':20}, { '$push' : { 'vals' : 5 }});
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1405, in __call__
    self.__name.split(".")[-1])
TypeError: 'Collection' object is not callable. If you meant to call the 'findAndModify' method on a 'Collection' object it is failing because no such method exists.
>>> mm.delme2.findAndUpdate()
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1405, in __call__
    self.__name.split(".")[-1])
TypeError: 'Collection' object is not callable. If you meant to call the 'findAndUpdate' method on a 'Collection' object it is failing because no such method exists.
>>> mm.findAndUpdate()
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1401, in __call__
    self.__name)
TypeError: 'Collection' object is not callable. If you meant to call the 'findAndUpdate' method on a 'Database' object it is failing because no such method exists.
>>> db
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'db' is not defined
>>> mm.delme2.find_and_modify({'metrictypeid':20}, { '$push' : { 'vals' : 5 }});
{u'vals': [3, 4], u'start': 1000, u'_id': ObjectId('5150c9bd64524c04169fd612'), u'metrictypeid': 20}
>>> pprint.pprint([x for x in mm.delme2.find()])
[{u'_id': ObjectId('5150c9cd64524c04169fd613'),
  u'metrictypeid': 20,
  u'start': 1010,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9d264524c04169fd614'),
  u'metrictypeid': 20,
  u'start': 1020,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9d764524c04169fd615'),
  u'metrictypeid': 20,
  u'start': 1030,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9dd64524c04169fd616'),
  u'metrictypeid': 10,
  u'start': 1030,
  u'vals': [3, 4]},
 {u'_id': ObjectId('5150c9bd64524c04169fd612'),
  u'metrictypeid': 20,
  u'start': 1000,
  u'vals': [3, 4, 5]}]
>>> mm.delme2.find_and_modify(query={'metrictypeid':20}, update={'$push':{'vals':6}}, sort={'start':1},new=True);
/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py:1364: DeprecationWarning: Passing mapping types for `sort` is deprecated, use a list of (key, direction) pairs instead
  DeprecationWarning)
{u'vals': [3, 4, 5, 6], u'start': 1000, u'_id': ObjectId('5150c9bd64524c04169fd612'), u'metrictypeid': 20}
>>> mm.delme2.find_and_modify(query={'metrictypeid':20}, update={'$push':{'vals':6}}, sort=['start',1],new=True);
Traceback (most recent call last):
  File "", line 1, in 
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/collection.py", line 1357, in find_and_modify
    kwargs['sort'] = helpers._index_document(sort)
  File "/path/not/important/here/pylibs/lib/python2.6/site-packages/pymongo-2.4.1-py2.6-linux-x86_64.egg/pymongo/helpers.py", line 68, in _index_document
    for (key, value) in index_list:
ValueError: too many values to unpack
>>> mm.delme2.find_and_modify(query={'metrictypeid':20}, update={'$push':{'vals':6}}, sort=[['start',1]],new=True);
{u'vals': [3, 4, 5, 6, 6], u'start': 1000, u'_id': ObjectId('5150c9bd64524c04169fd612'), u'metrictypeid': 20}
>>> mm.delme2.find_and_modify(query={'metrictypeid':20}, update={'$push':{'vals':7}}, sort=[['start',1]],new=True);
{u'vals': [3, 4, 5, 6, 6, 7], u'start': 1000, u'_id': ObjectId('5150c9bd64524c04169fd612'), u'metrictypeid': 20}
>>> mm.delme2.find_and_modify(query={'metrictypeid':20}, update={'$push':{'vals':7}}, sort=[['start',-1]],new=True);
{u'vals': [3, 4, 7], u'start': 1030, u'_id': ObjectId('5150c9d764524c04169fd615'), u'metrictypeid': 20}


Friday, March 08, 2013

Trying to build Python and a set of Python libraries on an AIX 6.1 box.

Starting with both gcc and IBM xlc compilers handy, this will be a set of notes, as opposed to a strict recipe. 

Have in /opt/freeware/bin:  gcc, make, tar.

Python notes:

    AIX:    A complete overhaul of the shared library support is now in
            place.  See Misc/AIX-NOTES for some notes on how it's done.
            (The optimizer bug reported at this place in previous releases
            has been worked around by a minimal code change.) If you get
            errors about pthread_* functions, during compile or during
            testing, try setting CC to a thread-safe (reentrant) compiler,
            like "cc_r".  For full C++ module support, set CC="xlC_r" (or
            CC="xlC" without thread support).
   
    AIX 5.3: To build a 64-bit version with IBM's compiler, I used the
            following:
   
            export PATH=/usr/bin:/usr/vacpp/bin
            ./configure --with-gcc="xlc_r -q64" --with-cxx="xlC_r -q64" \
                        --disable-ipv6 AR="ar -X64"
            make
So, I know a couple of things from previous adventures here.  First, I need working libraries for a variety of things:
* readline
* sqlite3
* zlib

LIBPATH=/opt/freeware/lib:/opt/freeware/lib64:/usr/lib:/usr/local/lib:/opt/recon/dcm/platforms/AIX/lib:/opt/recon/lib:

I've run into problems when compiling with xlc. The tests don't pass, and I get an exit status 11.  yuck.  More later.

MongoDB - How to Easily Pre-Split Chunks

Okay, so I'm doing some performance testing on MongoDB, spinning up a new configuration, and then throwing lots of data at it to see what the characteristics are.

Early on, I discovered that database locking was severely limiting performance.  MongoDB locks the database/collection (as of MongoDB 2.2.3) during every write operation.  However, it only locks it for the specific shard being written to.  So, massive numbers of writes dictates large numbers of shards, to minimize the lock/unlock bottleneck.  That's the theory.

In practice, this indeed proved correct.  We changed from a running (on 4 boxes - 2 primaries and 2 replicas) with 2 shards to running with 48 shards.  YES, they all magically cooperate nicely and share memory and CPU equally.  Actually, the boxes were all 196 GB memory w/ 32 cores each, so we weren't limited on machine capacity.  YMMV.

BTW, at peak operations, we're only seeing CPU of 50% per daemon, so we're not CPU limited. 

But, during my tests, I had to spin up first 4 shards, then 8, then 24, then 48.  Waiting for each test to settle into a steady state took time.  Each shard had to be active and equally used.  Likewise, there had to be enough data for it to be reading and writing at the same time, where the data was read out to send to the replicas.

The startup was handicapped by there being all activity on one shard, and not the others.  I would start things off, and all the activity would just be on 1 shard.  Since I was maxing out the capacity, there wasn't any spare IO available to do the splits and balancing.

To fix this, I found that I could add chunk split-points ahead of time, a process called 'pre-splitting chunks'.  To do this:
  • Turn off the balancer.
  • Run the command 'split' with a middle value, multiple times according to your number of shards.  
  • Turn on the balancer again.

I had 48 shards on a key with range 000-999, so I pretended I had 50 shards and gave each shard 20.

No, you don't have to stop the database from doing anything, like failover to secondaries or anything like that.

So, my commands were:

$ mongo --port 99999999 --host myhost
> use config
> sh.stopBalancer();
> use admin
> db.runCommand({split: "myDb.myCollection" middle:{"key1":000,'key2':1}});

> db.runCommand({split: "myDb.myCollection" middle:{"key1":020,'key2':1}});
> ... (46+ more of these)
> use config
> sh.startBalancer();



NOTE:  Always leave the balancer off more than 1 minute or it doesn't count.  That is, if you're 'bouncing' the balancer, leave it off for a full 60+ seconds or the bounce doesn't do anything.

Friday, January 04, 2013

MongoDB Shard Stats One-Liner

I've been testing, and I want to get the shard stats in one line.  That is I want this:

    krice@zaphod:~/$ mongo --host boxname303.example.com --port 40000 --eval "sh.status()"
    MongoDB shell version: 2.0.6
    connecting to: boxname303.example.com:40000/test
    --- Sharding Status ---
      sharding version: { "_id" : 1, "version" : 3 }
      shards:
            {  "_id" : "shard-00",  "host" : "shard-00/boxname301.example.com:30001,boxname303.example.com:30000" }
            {  "_id" : "shard-01",  "host" : "shard-01/boxname302.example.com:30101,boxname304.example.com:30100" }
            {  "_id" : "shard-02",  "host" : "shard-02/boxname301.example.com:30201,boxname303.example.com:30200" }
            {  "_id" : "shard-03",  "host" : "shard-03/boxname302.example.com:30301,boxname304.example.com:30300" }
            {  "_id" : "shard-04",  "host" : "shard-04/boxname301.example.com:30401,boxname303.example.com:30400" }
            {  "_id" : "shard-05",  "host" : "shard-05/boxname302.example.com:30501,boxname304.example.com:30500" }
            {  "_id" : "shard-06",  "host" : "shard-06/boxname301.example.com:30601,boxname303.example.com:30600" }
            {  "_id" : "shard-07",  "host" : "shard-07/boxname302.example.com:30701,boxname304.example.com:30700" }
            {  "_id" : "shard-08",  "host" : "shard-08/boxname301.example.com:30801,boxname303.example.com:30800" }
            {  "_id" : "shard-09",  "host" : "shard-09/boxname302.example.com:30901,boxname304.example.com:30900" }
            {  "_id" : "shard-10",  "host" : "shard-10/boxname301.example.com:31011,boxname303.example.com:31010" }
            {  "_id" : "shard-11",  "host" : "shard-11/boxname302.example.com:31111,boxname304.example.com:31110" }
            {  "_id" : "shard-12",  "host" : "shard-12/boxname301.example.com:31211,boxname303.example.com:31210" }
            {  "_id" : "shard-13",  "host" : "shard-13/boxname302.example.com:31311,boxname304.example.com:31310" }
            {  "_id" : "shard-14",  "host" : "shard-14/boxname301.example.com:31411,boxname303.example.com:31410" }
            {  "_id" : "shard-15",  "host" : "shard-15/boxname302.example.com:31511,boxname304.example.com:31510" }
      databases:
            {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
            {  "_id" : "megamaid",  "partitioned" : true,  "primary" : "shard-00" }
                    megamaid.metricValue chunks:
                                    shard-05        2
                                    shard-13        2
                                    shard-10        2
                                    shard-15        2
                                    shard-01        2
                                    shard-08        2
                                    shard-04        2
                                    shard-07        2
                                    shard-12        2
                                    shard-03        2
                                    shard-09        2
                                    shard-06        2
                                    shard-11        2
                                    shard-14        2
                                    shard-00        2
                                    shard-02        5
                            too many chunks to print, use verbose if you want to force print
            {  "_id" : "test",  "partitioned" : false,  "primary" : "shard-14" }
    krice4@zaphod:~/$



But, I want just the numbers so I can see if I'm balanced.  Here's the one-liner:

alias shardbal="mongo --host boxname.example.com --port 40000 --eval \"sh.status()\" | egrep 'shard\-[0-9]{2}[^\"/]' | awk '{print \$2}' | tr '\n' ',' | xargs echo \"Shard Balance is: \" "
krice4@zaphod:~/$ shardbal
Shard Balance is:  4,4,4,3,4,4,4,4,4,4,4,4,4,4,4,4,



Tuesday, November 13, 2012

Postgres pgdb connect with connect timeout

So, here's a little FYI.

I'm trying to connect to a Postgres database and pass in a connection timeout. Whatever the default is seems to be way too short. This is via Python 2.7 on Linux Ubuntu 12.10, if you're interested, but I don't suppose it matters, I'll be doing the same thing in a moment on Centos 6.

I finally figured out how to do this on the command line:

psql connect_timeout=5 -h myhost.namegoeshere.com -p 5432 -U myname

But, I'm trying to do this programmatically from python using pgdb pgdb.connect(). Here's the command I'm trying:

    connString = "%s:%s:%s:%s:%s" % (self.pgServer, 'xxx', 'xxxzzzzx', 'xyyyy', "connect_timeout=5")
    db_connection = pgdb.connect(dsn=connString, host="myhost.namegoeshere.com :5432")
    db_cursor = db_connection.cursor()
    db_cursor.execute("SELECT 42;")
    if 42 not in db_cursor.fetchone():
        raise Exception("Test query failed")


No joy yet.

Saturday, October 13, 2012

Sysstat make: o: Command not found

I was compiling sysstat for Centos 6 with a --prefix option, but kept getting a set of errors like:
make: o: Command not found make: [nls/sk.gmo] Error 127 (ignored) o nls/sr.gmo nls/sr.po make: o: Command not found make: [nls/sr.gmo] Error 127 (ignored) o nls/sv.gmo nls/sv.po make: o: Command not found make: [nls/sv.gmo] Error 127 (ignored) o nls/uk.gmo nls/uk.po make: o: Command not found make: [nls/uk.gmo] Error 127 (ignored) o nls/vi.gmo nls/vi.po make: o: Command not found make: [nls/vi.gmo] Error 127 (ignored) o nls/zh_CN.gmo nls/zh_CN.po make: o: Command not found make: [nls/zh_CN.gmo] Error 127 (ignored) o nls/zh_TW.gmo nls/zh_TW.po make: o: Command not found make: [nls/zh_TW.gmo] Error 127 (ignored) make distclean
Which really reduces to just "make: o: Command not found" repeated for lots of objects. What gives? I tried playing with CXX=g++ stuff, but it turned out the problem was only with the NLS support, which I disabled in the ./configure step, and the compile worked (note: still won't 'make test'). Successfull line:
./configure --prefix=/opt/recon/dcm/platforms/CENTOS6 --disable-documentation --disable-stripping --disable-sensors --disable-nls
And, voila, it works.

Wednesday, October 03, 2012

Compiling rsyslog for CENTOS 6.1

So, I was in dependency hell yesterday. It's one of the outer circles, nearby the region for people who drop gum on carpet. I was trying to get a recent version of rsyslog to install from source tgz on a centos 6.1 box. Unfortunately, the good people who have brought us rsyslog have put in requirements for libee and libestr, two not-production-ready libraries in the sense that they don't compile cleanly and easily with configure options that point to each other, etc. this is one of the concepts I started with, but it didn't turn out so well. [krice4@boxname rsyslog-7.1.7]$ ./configure LIBESTR_CFLAGS=/usr/local/include LIBESTR_LIBS=/usr/local/lib/libestr.so LIBEE_CFLAGS=/usr/local/include LIBEE_LIBS=/usr/local/lib/libee.so --enable-imfile --enable-mail --enable-extended-tests --enable-omstdout --enable-omprog --enable-zlib --enable-debug JSON_C_LIBS=/usr/lib/libjson.so JSON_C_CFLAGS=/usr/local/include After a while, i had to give up and install an RPM created by someone else, and even then, it broke the box on reboot so we had to re-image the vm. Ug. Advice: don't try to build 7.1 yourself without serious impact protection and a large bottle of 'Scotch' (or whatever your Scotch is) to take the edge off.