Example of how to set JournalCommitInterval in MongoDB

I have found no way yet to find the current value of journalCommitInterval.

But, I have found how to set the journalCommitInterval.

We've got our mongodb set up to use a local disk for our journal, and data resides on a SAN-mounted mount point.

We'd like to set our journalCommitInterval up to a larger value to cut down on the IOPS to our local disk. This might have the added benefit of helping us run faster.

I've run the following command against a mongos, and it complains:

mongos> db.adminCommand({ "setParameter" : 1, "journalCommitInterval": 499});
{ "ok" : 0, "errmsg" : "journaling is off" }

When I run this against a specific daemon, it works up to a value of 499 (setting it to 500 generates a complaint/error):

shard-000:PRIMARY> db.adminCommand({ "setParameter" : 1, "journalCommitInterval": 499});
{ "ok" : 1 }

I have posed the follwoing questions to 10Gen:

• Question 1: do I need to run this against each mongod individually?
• Answer 1: Yes.
• Question 2: does this persist across restarts of the daemon?
• Answer 2: No. That daemon's journalCommitInterval returns to its default when a daemon are restarted. Thus, it's better to set it in the config file you have defined for the shard. Or, if you define it via command line, do something like:

  /path/to/mongod --journalCommitInterval=499 ...
  

I have just created a minor request in core-server to expose the current value so we can verify it has been set correctly on all shards. If you agree this would be a handy feature to have, please watch and VOTE on the case, https://jira.mongodb.org/browse/SERVER-10508

Solved: Installing Linux on Acer Aspire V5-122P-0643 Quadcore AMD laptop

So, my wife wanted an Acer Aspire laptop because of its size, primarily. The form factor was just right for her. But, it came with Windows 8, which she really hates (she loves Ubuntu Linux and Libre Office Writer). So, I downloaded Ubuntu Linux 13.04 Raring Ringtail and installed it on this Aspire V5 122P 0643 box. Complications?

1. windows 8 doesn't like to give up control of the BIOS. Going to system settings and boot from USB key (which I'd already put Ubuntu's install image on via another box).

2. Once installed, it wanted to reboot. I did so.

3. VAST multi-day hassles due to black screen after boot, then prompting for user login: instead of going to the graphical user login page familiar to all Ubuntu users. I messed around for a long time trying various things. Running startx failed with a message about no screens. Trying to reboot with some setting for recovery mode got me success once, but I couldn't reproduce it.

The command to show all hw installed revealed the video chipset is AMD Radeon 8280. This appears to be the only laptop, or in fact any device whatsoever, that uses this chipset, though Ubuntu seems to think it's on some desktops somewhere (in their compatibility pages).

4. FINALLY, solved the problem. On my other laptop, navigated to find the download for AMD's proprietary driver, and found the download destination.

Logged into Ubuntu using the user created during installation. Did 'sudo ls' and reentered my password so I had sudo privs without prompting. Then, downloaded the AMD driver from the link on this page: http://support.amd.com/us/gpudownload/linux/Pages/radeon_linux.aspx

the download link was http://www2.ati.com/drivers/linux/amd-driver-installer-catalyst-13-4-linux-x86.x86_64.zip, so I retyped it at the command prompt as:

$ wget http://www2.ati.com/drivers/linux/amd-driver-installer-catalyst-13-4-linux-x86.x86_64.zip

and it downloaded. I

then did: unzip amd* and then did: chmod 755 amd* then agreed to everything and installed it, rebooted per instructions, and voila! IT WORKED!

So, the answer is that the fglrx drivers (automatically installed as part of the amd drivers download, methinks) are the correct solution here.

The touch screen does work , but is of limited usefulness since the icons are so small in a normal linux desktop screen, as they should be. After all , windows 8 sucks at trying to unite the phone with the desktop environment, because they're disparate platforms and everyone except Microsoft seems to know this.

Enjoy!

How to Disable password prompt during ssh login with key failure

I wanted to login to a bunch of servers and test whether I could get in, or if I got error messages.

After messing around a while, I worked it into:

for n in `cat allhosts.txt`; do echo "host: ${n}"; ssh -oConnectTimeout=2 -oKbdInteractiveAuthentication=no -oPasswordAuthentication=no -oStrictHostKeyChecking=no -oChallengeResponseAuthentication=no myuser@${n} echo '----------'; done

Question on StackOverflow: Best Unittest for Python programs using OptionParser() ?

Just posted this on stackoverflow. http://stackoverflow.com/questions/17223530/python-how-do-i-usefully-unittest-a-method-invoking-optionparser

MagicMock Returns different values each call

Recent discoveries about mock objects!

I'm trying to create a Python unittest that exercises a class method with a time() call in it.

That is I have this:

class DumObj(object):

  def methodOne(self, intime):

    while time.time() < intime:

      ... do stuff ...

so, I refactored this to:

class DumObj(object):

  def nowTime(self):

    return time.time() 

  def methodOne(self, intime):

    while self.nowTime() < intime:

      ... do stuff ...

and have to test methodOne. How to get it to go through the loop once (or twice)?

I tried setting the return value to the output of a function:

import unittest

from mock import Mock, MagicMock

class TestDumObj(unittest.TestCase) 

 def test_methodOne(self):

 d = DumObj()

    retvals=[1,2,3,4,5] 

    def mf():

      ret = retvals.pop()

      return ret

 d.nowTime = MagicMock(return_value=mf()) d.methodOne()

Doesn't work. Here's the interactive version:

>>> from mock import MagicMock

>>> retvals = [ 1, 2, 3, 4, 5 ] 

>>> def mse(*args, **kwargs):

... ret = retvals.pop()

... return ret

... 

>>> mm = MagicMock(return_value=mse())

>>> mm()

5

>>> mm()

5

Damn! I want MagicMock to return different values each time it's called. I want to have an array of return values for MagicMock, and each call it returns the next one. How to do this? Turns out it's using the side_effect param. Continuing from above:

>>> mm = MagicMock(side_effect=mse)

>>> mm()

4

>>> mm()

3

Yay! Solved: MagicMock returns different values each call. Now, tie it into the overall test:

import unittest

from mock import Mock, MagicMock

class TestDumObj(unittest.TestCase) 

 def test_methodOne(self):

 d = DumObj()

    retvals=[1,2,3,4,5] 

    def mse():

      ret = retvals.pop()

      return ret

 d.nowTime = MagicMock(side_effect=mse) d.methodOne()

Works!

Drop-Thru Code Considered Harmful

Recently, I was describing my frustrations with some Python code that had been written by a now-departed co-worker.  I describe this code as "Drop-Thru Code".  The main characteristic is that it uses module scope for a non-trivial number of variables.

Module scope is functionally almost-global.  That is, it's so close to global you'll want to use it, but it's far enough away that you'll end up shooting yourself in the foot.

Consider this code snippet:

#!/bin/env python2.6
import os
import sys
varname = 33
vardict = { 'a' : 44 }
# other imports here
print "Starting!"
def something():
print "thing", varname
print "next"
something()
print "end"

This is what I call drop-thru code.  All the variables are global, and the call to something() will fail because varname is out of scope.

Contrast this with what I prefer, well-encapsulated code:

#!/bin/env python2.6
import os
import sys

class SomeThing(object):
    def __init__(self):
        self.varname = 33 
    def something(self):
        print "thing", self.varname
    def main(self):
        self.vardict = { 'a' : 44 }
        # other imports here
        print "Starting!"
        print "next"
        self.something()
        print "end" 

st = SomeThing()
st.main()

Nothing is global, it's obvious what the scope for things is.  Clean, beautiful, easy to understand.

There are lots of examples of scope problems.  I created this one for my team so they understood my frustration:

outside = 1
def function1():
    try:
        print "f1 outside: %s" % (outside)
        outside += 1
    except:
        print "no outside in function1."
print "a outside: %s" % (outside)
function1()                                     
print "b outside: %s" % (outside)
def function2():
    global outside
    try:
        print "f2 outside: %s" % (outside)
        outside += 1
    except:
        print "no outside in function2"
function2()
function2()
class Dum(object):
    def __init__(self):
        print "dum instantiated."
    def main(self):
        print "Dum main outside: %s" % (outside)
    def changeOutside(self, inval):
        outside = inval
        print "Dum changed: outside: %s" % (outside)
    def changeGlobalOutside(self, inval):
        global outside
        outside = inval
        print "Dum changed: outside: %s" % (outside)
d = Dum()
print "c outside: %s" % (outside)
d.main()
print "d outside: %s" % (outside)
d.changeOutside(33)
print "e outside: %s" % (outside)
d.main()
d.changeGlobalOutside(44)
d.main()
print "f outside: %s" % (outside)
______________________________________________________
output:
krice4@zaphod:~/checkouts/userSandboxes/krice4$ python scopeTest.py 
a outside: 1
no outside in function1.
b outside: 1
f2 outside: 1
f2 outside: 2
dum instantiated.
c outside: 3
Dum main outside: 3
d outside: 3
Dum changed: outside: 33
e outside: 3
Dum main outside: 3
Dum changed: outside: 44
Dum main outside: 44
f outside: 44

In short, Drop-Thru Code is considered harmful.  It allows for lots of scope problems that show up as bugs and frustrations later.  So, avoid them.  Put all the vars you can in a class and invoke that class, you'll be glad you did.  IMHO.

Subversion pre-commit hook script - Python files: prevent tabs, verify svn properties

Here's a precommit hook script I've modified from one I've used before.  I hope it comes in handy.  I'm going to try to submit it to the dev group of subversion itself for inclusion in the contrib/hook-scripts directory.

#!/bin/env python

"""
    pre-commit hook script that does several things:
    * prevents committing any python file containing a tab character.
    * checks if there are tabs in the source file and warns if so;
    * aborts if incorrect properties of eol-style and keywords 'id'.
"""
import sys
import os
import traceback
from optparse import OptionParser

#sys.stderr.write("NOTE:  pre-commit hook script enabled - checks for tabs, svn eol-style and id properties...\n")

def command_output(cmd):
    " Capture a command's standard output. "
    import subprocess
    return subprocess.Popen(
        cmd.split(), stdout=subprocess.PIPE).communicate()[0]

def files_changed(look_cmd):
    """ List the files added or updated by this transaction.

        "svnlook changed" gives output like:
          U   trunk/file1.cpp
          A   trunk/file2.py
    """
    def filename(line):
        return line[4:]

    def added_or_updated(line):
        return line and line[0] in ("A", "U")

    retval = []
    for line in command_output(look_cmd % "changed").split("\n"):
        if added_or_updated(line):
            retval.append(filename(line))
    #sys.stderr.write("files changed: %s" % (retval))
    return retval

def file_contents(filename, look_cmd):
    " Return a file's contents for this transaction. "
    return command_output("%s %s" % (look_cmd % "cat", filename))

def file_get_properties(filename, look_cmd):
    propslines = command_output("%s %s" % (look_cmd % "proplist -v", filename))
    res = {}
    for line in propslines.split('\n'):
        line = line.strip()
        if not line:
            continue
        k, v = line.split(' : ')
        res[k] = v
    return res

def contains_tabs(filename, look_cmd):
    " Return True if this version of the file contains tabs. "
    return "\t" in file_contents(filename, look_cmd)

def check_py_files(look_cmd):
    " Check Python files in this transaction are tab-free. "
   
    def is_py_file(fname):
        return os.path.splitext(fname)[1] == ".py"
   
    py_files_with_tabs    = set()
    py_files_bad_eolstyle = set()
    py_files_bad_exec     = set()
    py_files_bad_keywords = set()
    for ff in files_changed(look_cmd):
        if not is_py_file(ff):
            continue
        if contains_tabs(ff, look_cmd):
            py_files_with_tabs.add(ff)
        props = file_get_properties(ff, look_cmd)
       if props.get('svn:special'):
            sys.stderr.write("file %s has svn:special flag, probably a symlink. don't check other props." % ff)
            continue
        eolstyle = props.get('svn:eol-style')
        #sys.stderr.write("props: %s\neolstyle: '%s'\n" % (props, eolstyle))
        if eolstyle in ('native', 'LFFFFF'):
            py_files_bad_eolstyle.add(ff)
        execut   = props.get('svn:executable')
        if execut not in ['ON', '*']:
            py_files_bad_exec.add(ff)
        keywords = props.get('svn:keywords')
        if (not keywords) or ('Id' not in keywords.split()):
            py_files_bad_keywords.add(ff)

    preventCommit = False
    if len(py_files_with_tabs) > 0:
        sys.stderr.write("The following files contain tabs:\n%s\n"                                                                              % "\n".join(py_files_with_tabs))
        preventCommit = True
    if len(py_files_bad_exec) > 0:
        sys.stderr.write("The following py files are missing 'executable' property, committing anyway, but please fix this:\n%s\n"              % "\n".join(py_files_bad_exec))
        # note, do not prevent commit over this, just warn.
    if len(py_files_bad_keywords) > 0:
        sys.stderr.write("The following files don't have keywords property set to 'Id' at least.  Please fix this before committing:\n%s\n"     % "\n".join(py_files_bad_keywords))
        preventCommit = True
    if len(py_files_bad_eolstyle) > 0:
        sys.stderr.write("The following files don't have svn propset svn:eol-style 'LF', please do so before committing:\n%s\n"                 % "\n".join(py_files_bad_eolstyle))
        preventCommit = True

    return preventCommit

def main():
    usage = """usage: %prog REPOS TXN
        Run pre-commit options on a repository transaction."""

    parser = OptionParser(usage=usage)
    parser.add_option("-r", "--revision",
                      help="Test mode. TXN actually refers to a revision.",
                      action="store_true", default=False)
    errors = 0
    try:
        (opts, (repos, txn_or_rvn)) = parser.parse_args()
        look_opt = ("--transaction", "--revision")[opts.revision]
        look_cmd = "svnlook %s %s %s %s" % (
            "%s", repos, look_opt, txn_or_rvn)
        errors += check_py_files(look_cmd)
    except:
        parser.print_help()
        errors += 1
        sys.stderr.write("Pre-commit hook traceback: %s" % (traceback.format_exc()))
    return errors

if __name__ == "__main__":
    sys.exit(main())