I've been testing, and I want to get the shard stats in one line. That is I want this:
krice@zaphod:~/$ mongo --host boxname303.example.com --port 40000 --eval "sh.status()"
MongoDB shell version: 2.0.6
connecting to: boxname303.example.com:40000/test
--- Sharding Status ---
sharding version: { "_id" : 1, "version" : 3 }
shards:
{ "_id" : "shard-00", "host" : "shard-00/boxname301.example.com:30001,boxname303.example.com:30000" }
{ "_id" : "shard-01", "host" : "shard-01/boxname302.example.com:30101,boxname304.example.com:30100" }
{ "_id" : "shard-02", "host" : "shard-02/boxname301.example.com:30201,boxname303.example.com:30200" }
{ "_id" : "shard-03", "host" : "shard-03/boxname302.example.com:30301,boxname304.example.com:30300" }
{ "_id" : "shard-04", "host" : "shard-04/boxname301.example.com:30401,boxname303.example.com:30400" }
{ "_id" : "shard-05", "host" : "shard-05/boxname302.example.com:30501,boxname304.example.com:30500" }
{ "_id" : "shard-06", "host" : "shard-06/boxname301.example.com:30601,boxname303.example.com:30600" }
{ "_id" : "shard-07", "host" : "shard-07/boxname302.example.com:30701,boxname304.example.com:30700" }
{ "_id" : "shard-08", "host" : "shard-08/boxname301.example.com:30801,boxname303.example.com:30800" }
{ "_id" : "shard-09", "host" : "shard-09/boxname302.example.com:30901,boxname304.example.com:30900" }
{ "_id" : "shard-10", "host" : "shard-10/boxname301.example.com:31011,boxname303.example.com:31010" }
{ "_id" : "shard-11", "host" : "shard-11/boxname302.example.com:31111,boxname304.example.com:31110" }
{ "_id" : "shard-12", "host" : "shard-12/boxname301.example.com:31211,boxname303.example.com:31210" }
{ "_id" : "shard-13", "host" : "shard-13/boxname302.example.com:31311,boxname304.example.com:31310" }
{ "_id" : "shard-14", "host" : "shard-14/boxname301.example.com:31411,boxname303.example.com:31410" }
{ "_id" : "shard-15", "host" : "shard-15/boxname302.example.com:31511,boxname304.example.com:31510" }
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "megamaid", "partitioned" : true, "primary" : "shard-00" }
megamaid.metricValue chunks:
shard-05 2
shard-13 2
shard-10 2
shard-15 2
shard-01 2
shard-08 2
shard-04 2
shard-07 2
shard-12 2
shard-03 2
shard-09 2
shard-06 2
shard-11 2
shard-14 2
shard-00 2
shard-02 5
too many chunks to print, use verbose if you want to force print
{ "_id" : "test", "partitioned" : false, "primary" : "shard-14" }
krice4@zaphod:~/$
But, I want just the numbers so I can see if I'm balanced. Here's the one-liner:
alias shardbal="mongo --host boxname.example.com --port 40000 --eval \"sh.status()\" | egrep 'shard\-[0-9]{2}[^\"/]' | awk '{print \$2}' | tr '\n' ',' | xargs echo \"Shard Balance is: \" "
krice4@zaphod:~/$ shardbal
Shard Balance is: 4,4,4,3,4,4,4,4,4,4,4,4,4,4,4,4,
Friday, January 04, 2013
Tuesday, November 13, 2012
Postgres pgdb connect with connect timeout
So, here's a little FYI.
I'm trying to connect to a Postgres database and pass in a connection timeout. Whatever the default is seems to be way too short. This is via Python 2.7 on Linux Ubuntu 12.10, if you're interested, but I don't suppose it matters, I'll be doing the same thing in a moment on Centos 6.
I finally figured out how to do this on the command line:
But, I'm trying to do this programmatically from python using pgdb pgdb.connect(). Here's the command I'm trying:
No joy yet.
I'm trying to connect to a Postgres database and pass in a connection timeout. Whatever the default is seems to be way too short. This is via Python 2.7 on Linux Ubuntu 12.10, if you're interested, but I don't suppose it matters, I'll be doing the same thing in a moment on Centos 6.
I finally figured out how to do this on the command line:
psql connect_timeout=5 -h myhost.namegoeshere.com -p 5432 -U myname
But, I'm trying to do this programmatically from python using pgdb pgdb.connect(). Here's the command I'm trying:
connString = "%s:%s:%s:%s:%s" % (self.pgServer, 'xxx', 'xxxzzzzx', 'xyyyy', "connect_timeout=5")
db_connection = pgdb.connect(dsn=connString, host="myhost.namegoeshere.com :5432")
db_cursor = db_connection.cursor()
db_cursor.execute("SELECT 42;")
if 42 not in db_cursor.fetchone():
raise Exception("Test query failed")
No joy yet.
Saturday, October 13, 2012
Sysstat make: o: Command not found
I was compiling sysstat for Centos 6 with a --prefix option, but kept getting a set of errors like:
make: o: Command not found make: [nls/sk.gmo] Error 127 (ignored) o nls/sr.gmo nls/sr.po make: o: Command not found make: [nls/sr.gmo] Error 127 (ignored) o nls/sv.gmo nls/sv.po make: o: Command not found make: [nls/sv.gmo] Error 127 (ignored) o nls/uk.gmo nls/uk.po make: o: Command not found make: [nls/uk.gmo] Error 127 (ignored) o nls/vi.gmo nls/vi.po make: o: Command not found make: [nls/vi.gmo] Error 127 (ignored) o nls/zh_CN.gmo nls/zh_CN.po make: o: Command not found make: [nls/zh_CN.gmo] Error 127 (ignored) o nls/zh_TW.gmo nls/zh_TW.po make: o: Command not found make: [nls/zh_TW.gmo] Error 127 (ignored) make distcleanWhich really reduces to just "make: o: Command not found" repeated for lots of objects. What gives? I tried playing with CXX=g++ stuff, but it turned out the problem was only with the NLS support, which I disabled in the ./configure step, and the compile worked (note: still won't 'make test'). Successfull line:
./configure --prefix=/opt/recon/dcm/platforms/CENTOS6 --disable-documentation --disable-stripping --disable-sensors --disable-nlsAnd, voila, it works.
Wednesday, October 03, 2012
Compiling rsyslog for CENTOS 6.1
So, I was in dependency hell yesterday. It's one of the outer circles, nearby the region for people who drop gum on carpet.
I was trying to get a recent version of rsyslog to install from source tgz on a centos 6.1 box. Unfortunately, the good people who have brought us rsyslog have put in requirements for libee and libestr, two not-production-ready libraries in the sense that they don't compile cleanly and easily with configure options that point to each other, etc.
this is one of the concepts I started with, but it didn't turn out so well.
[krice4@boxname rsyslog-7.1.7]$ ./configure LIBESTR_CFLAGS=/usr/local/include LIBESTR_LIBS=/usr/local/lib/libestr.so LIBEE_CFLAGS=/usr/local/include LIBEE_LIBS=/usr/local/lib/libee.so --enable-imfile --enable-mail --enable-extended-tests --enable-omstdout --enable-omprog --enable-zlib --enable-debug JSON_C_LIBS=/usr/lib/libjson.so JSON_C_CFLAGS=/usr/local/include
After a while, i had to give up and install an RPM created by someone else, and even then, it broke the box on reboot so we had to re-image the vm. Ug. Advice: don't try to build 7.1 yourself without serious impact protection and a large bottle of 'Scotch' (or whatever your Scotch is) to take the edge off.
Thursday, August 16, 2012
Mongo DB Map-Reduce for Group-By w/ extra fields
So, I was trying to do a query similar to the following:
select field_a, max(field_b) as maxb from tableA group by field_a
But, when I found field_a, I needed another field from that record. This would be straightforward in SQL with a subselect or join, but MongoDB on a sharded DB requires some Map-Reduce foo.
The problem is that reduce only allows you to return one value. But, I needed both the accumulation value AND another field to come back. Solved via this test program:
from pprint import pprint, pformat
from pymongo.code import Code
from pymongo import Connection
conn = Connection()
db = conn["test"]
db.testcol.remove()
db.testcol.insert({ "name" : "bob", "val" : 5, "host":"blah1"})
db.testcol.insert({ "name" : "bob", "val" : 3, "host":"blah2"})
db.testcol.insert({ "name" : "bob", "val" : 7, "host":"blah3"})
db.testcol.insert({ "name" : "john", "val" : 2, "host":"blah4"})
db.testcol.insert({ "name" : "john", "val" : 4, "host":"blah5"})
db.testcol.insert({ "name" : "john", "val" : 8, "host":"blah6"})
db.testcol.insert({ "name" : "john", "val" : 6, "host":"blah7"})
#print "Data: "
#print pformat(list(db.testcol.find()))
mapfunc = Code(''' function() { emit( this.name, { name : this.name, val: this.val, host: this.host }); } ''')
redfunc = Code(''' function(key, values) {
var latestRec = { val: 0, host: "", name: ""};
for (var i=1; i < values.length; i++) {
thisName = values[i].name
thisVal = values[i].val;
thisHost = values[i].host;
if (thisVal > latestRec.val) {
latestRec.name = thisName
latestRec.val = thisVal;
latestRec.host = thisHost;
}
}
return latestRec
} ''')
res = db.testcol.map_reduce(mapfunc, redfunc, "myresults")
for doc in res.find():
print doc
print "*" * 80
conn.close()
Saturday, July 21, 2012
SOLVED: How to Install Netgear WG311 under Ubuntu 12.04
Put Netgear WG311 in box.
turn on box.
follow directions at
https://help.ubuntu.com/community/WifiDocs/Device/Netgear_WG311_v3
BUT: No matter how many times I rebooted, the sudo modprobe ndiswrapper kept giving me the message FATAL.
Solved via:
kevin@odin:~$ sudo modprobe ndiswrapper
[sudo] password for kevin:
FATAL: Module ndiswrapper not found.
kevin@odin:~$ sudo apt-cache search dkms
... (omitting extra stuff)....
dkms - Dynamic Kernel Module Support Framework
ndiswrapper-dkms - Source for the ndiswrapper Linux kernel module (DKMS)
... (omitting extra stuff)....
kevin@odin:~$ sudo apt-get install dkms
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
fakeroot patch
Suggested packages:
diffutils-doc
The following NEW packages will be installed:
dkms fakeroot patch
0 upgraded, 3 newly installed, 0 to remove and 9 not upgraded.
Need to get 247 kB of archives.
After this operation, 890 kB of additional disk space will be used.
Do you want to continue [Y/n]? Y
Get:1 http://mirror.anl.gov/pub/ubuntu/ precise/main patch i386 2.6.1-3 [86.0 kB]
Get:2 http://mirror.anl.gov/pub/ubuntu/ precise/main dkms all 2.2.0.3-1ubuntu3 [73.1 kB]
Get:3 http://mirror.anl.gov/pub/ubuntu/ precise/main fakeroot i386 1.18.2-1 [87.9 kB]
Fetched 247 kB in 0s (436 kB/s)
Selecting previously unselected package patch.
(Reading database ... 143936 files and directories currently installed.)
Unpacking patch (from .../patch_2.6.1-3_i386.deb) ...
Selecting previously unselected package dkms.
Unpacking dkms (from .../dkms_2.2.0.3-1ubuntu3_all.deb) ...
Selecting previously unselected package fakeroot.
Unpacking fakeroot (from .../fakeroot_1.18.2-1_i386.deb) ...
Processing triggers for man-db ...
Setting up patch (2.6.1-3) ...
Setting up dkms (2.2.0.3-1ubuntu3) ...
Setting up fakeroot (1.18.2-1) ...
update-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode.
kevin@odin:~$ sudo apt-get install dkms ndiswrapper-dkms
Reading package lists... Done
Building dependency tree
Reading state information... Done
dkms is already the newest version.
The following NEW packages will be installed:
ndiswrapper-dkms
0 upgraded, 1 newly installed, 0 to remove and 9 not upgraded.
Need to get 176 kB of archives.
After this operation, 778 kB of additional disk space will be used.
Do you want to continue [Y/n]? Y
Get:1 http://mirror.anl.gov/pub/ubuntu/ precise/universe ndiswrapper-dkms all 1.57-1ubuntu1 [176 kB]
Fetched 176 kB in 0s (285 kB/s)
Selecting previously unselected package ndiswrapper-dkms.
(Reading database ... 144030 files and directories currently installed.)
Unpacking ndiswrapper-dkms (from .../ndiswrapper-dkms_1.57-1ubuntu1_all.deb) ...
Setting up ndiswrapper-dkms (1.57-1ubuntu1) ...
Loading new ndiswrapper-1.57 DKMS files...
First Installation: checking all kernels...
Building only for 3.2.0-23-generic-pae
Building initial module for 3.2.0-23-generic-pae
Done.
ndiswrapper:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/3.2.0-23-generic-pae/updates/dkms/
depmod.......
DKMS: install completed.
kevin@odin:~$ iwconfig
lo no wireless extensions.
eth0 no wireless extensions.
kevin@odin:~$ sudo modprobe ndiswrapper
kevin@odin:~$ iwconfig
lo no wireless extensions.
wlan0 IEEE 802.11g ESSID:off/any
Mode:Managed Channel:0 Access Point: Not-Associated
Bit Rate:1 Mb/s Sensitivity=-200 dBm
RTS thr=2346 B Fragment thr=2346 B
Power Management:off
Link Quality:0 Signal level:0 Noise level:0
Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
Tx excessive retries:0 Invalid misc:0 Missed beacon:0
eth0 no wireless extensions.
Wednesday, July 18, 2012
Just had to create a median function in Python. We compute median values by ordering them and choosing the middle one.
I found a routine, but it lacked unittest coverage and the edge case of an empty input array. So, here goes:
import unittest
###############################################################################
def getMedian(numericValues):
theValues = sorted(numericValues)
if not theValues:
return None
if len(theValues) % 2 == 1:
return theValues[(len(theValues)+1)/2-1]
else:
lower = theValues[len(theValues)/2-1]
upper = theValues[len(theValues)/2]
return (float(lower + upper)) / 2
###############################################################################
if __name__ == '__main__':
unittest.main()
###############################################################################
class TestCommonUtils(unittest.TestCase):
def test_trivial(self):
return True
def isValid(self, functionName, valueShouldBe, valueIs):
assert (valueShouldBe == valueIs), "ERROR: Function %s returned value %.6fs, but value should be: %.6f." % (functionName, valueIs, valueShouldBe)
def test_getMedian(self):
self.isValid('getMedian', 2.5, getMedian([0,1,2,3,4,5]))
self.isValid('getMedian', 2, getMedian([0,1,2,3,4]))
self.isValid('getMedian', 2, getMedian([3,1,2]))
self.isValid('getMedian', 3, getMedian([3,2,3]))
self.isValid('getMedian', 1.234, getMedian([1.234, 3.678, -2.467]))
self.isValid('getMedian', 1.345, getMedian([1.234, 3.678, 1.456, -2.467]))
self.isValid('getMedian', 3, getMedian([3]))
self.isValid('getMedian', None, getMedian([]))
$ nosetests util.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.001s
OK
Subscribe to:
Posts (Atom)