Author Archives: colinnewell

OpenERP debugging tip – turn off cron

While we’ve been doing a lot of OpenERP deployment we have been discovering various ways to configure it and one turns out to be very handy for debugging. If you’ve ever debugged OpenERP using the --debug flag and dropped into the debugger you have probably noticed the system carries on doing things while you’re sat at the debugger prompt. Often obliterating what you were looking at. This generally happens because OpenERP generally runs with multiple threads out of the box, and some of those threads do the ‘cron’ jobs, the background tasks, so even if you haven’t tried to do anything on the website, there will be activity. If you want to prevent the cron activity from making your debugging session more confusing than it needs to be add the --max-cron-threads=0 flag when you run OpenERP.

Tagged

DBIx::Class and Postgres tweaks at startup

After connection you can do simple commands thanks to the on_connect_do connection setting. One thing I sometimes do is turn down the whining. Postgres can be quite noisy when you’re creating tables,

NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "blah_pkey" for table "blah"
NOTICE:  CREATE TABLE will create implicit sequence "blah_id_seq" for serial column "blah.id"
...

So I sometimes SET client_min_messages=WARNING at connection. That makes deploying a schema a lot less verbal.

If you want to deploy to a different schema within a database you can also do a ‘SET search_path TO’ statement at that point too. That can be quite handy if you want to deploy the same tables again to an alternative schema within a database.

perl -I lib -MMyModule::Schema::DB -e "MyModule::Schema::DB->connect('dbi:Pg:dbname=database;host=postgres', 'username', 'password', { on_connect_do => 'SET search_path TO temp' })->deploy"

If you’re setting up a Catalyst config for DBIC you can set the connection options like this,

<Model::MyDB>
    connect_info dbi:Pg:dbname=database
    connect_info username
    connect_info password
    <connect_info>
        on_connect_do  [ SET client_min_messages=WARNING ]
        quote_names 1
...
Tagged ,

Adhoc parameters to joins in DBIx::Class

Update: there are a few updates to the caveats on this post based on the comments by Peter Rabbitson.

I’ve been using DBIx::Class for a couple of years now and I’ve always been impressed with it’s flexibility.  I’ve rarely felt the need to write my own SQL queries because it’s simply so good at it, and it’s generally easy to get it to do what I want.

The one exception to that was custom adhoc joins.  In general DBIx::Class wants to know about how you’re going to join up front.  Anything else tends to require a custom view.  

The other night I realised I could come up with a way to deal with slightly more complex joins while still making the most of all my result definitions.  The extended relationships explained by frew on his blog demonstrate how to add decent join conditions.  The one thing missing was adhoc parameters.  They can be added by providing a bind parameter.  Since relationships don’t traditionally require any extra search parameters, I’d recommend indicating that the relationship isn’t for public consumption, and providing a wrapper method around it.

For example, here is a the relationship in the Result class,

__PACKAGE__->has_many(
  _date_range => 'DBICTest::Schema::CD',
  sub {
    my $args = shift;
    return (
      { "$args->{foreign_alias}.artist" => { -ident => "$args->{self_alias}.artistid" },
        -and => [
            "$args->{foreign_alias}.year"   => { '>' => \"?" },
            "$args->{foreign_alias}.year"   => { '<' => \"?" },
        ],
      }
    );
  }
);

And then a method to exploit it in the ResultSet class.

sub join_date_range
{
    my $self = shift;
    my $start = shift;
    my $end = shift;
    $self->search(undef, {
        join => ['_by_name'],
        bind => [ $start, $end ],
    });
}

Then you can use it like this,

$schema->resultset("Artist")->join_date_range(1990, 1992)->search(...)->all;

You can of course do more complex joins too, even joining up multiple times.  All you need to do is specify all the necessary bind parameters.

The most impressive thing is that the joins work fine even with the chained searches that DBIC is so good at.  You can of course also do search_related and most of the usual links, you just need to specify the bind parameters.  

There are a few of caveats however.  This isn’t strictly speaking an intentional feature, or at least it wasn’t consciously designed to work this way (as far as I know).

  1. Attempting to do a prefetch across the join fails with the error “Can't prefetch has_many _date_range (join cond too complex)”. Update: there is a new version of DBIC in the pipeline that fixes this. Right now that version is 0.08241-TRIAL. The feature you’re probably looking for in the Changes file is “Multiple has_many prefetch” when checking if a new live release has it.
  2. You might have looked at that -and => [] construction and thought that’s a bit pointless, a standard hash would be simpler and achieve the same effect. Unfortunately the bind parameters are positional, and a hash doesn’t have a guaranteed order. That means you need to be extra careful with your query when you have multiple parameters you need to specify, to ensure the binds happen to the correct place holders. Update: as Peter Rabbitson pointed out, it’s not actually that simple. DBIC does try to make sure you have a guaranteed order by sorting the keys of the hashes so that it always produces the same SQL. This means that you probably just need to try it and see which order in which it requires the parameters most of the time.
  3. Update: I was incorrect about not being able to use extra binding information with bind, the syntax Peter Rabbitson suggested works perfectly. i.e. bind => [ [ $opt1 => $val1 ], [ $opt => $val 2 ]… ]The final caveat is that the bind parameters don’t currently take any extra type information. Normally most of the places you are exposed directly to bindings you can specify types in order to help DBIC create the correct query. It doesn’t appear to be possible to provide that information via the bind parameter on a search.

This isn’t strictly a documented feature, but hopefully it’s helpful to a few people. If you’re wondering why you’d need to do this at all, consider the difference between these two queries.

SELECT *
FROM a LEFT JOIN b ON a.id = b.owner_id AND a.name = b.name

and

SELECT *
FROM a LEFT JOIN b ON a.id = b.owner_id
WHERE a.name = b.name

In the course of figuring this out, I also discovered the -ident key which indicates that you’re dealing with something like a column name, and should therefore be quoted if you have column name quoting turned on.  A handy feature to go along with using references to pass in literal bits of SQL.

Tagged

GDB

I only occasionally use gdb so I end up spending my time relearning it each time.  Hopefully these notes will make that process easier in the future.  This document assumes you already know the s and n commands and how to list the source code (l).  These are the commands I don’t know off by heart, and have found useful. It’s just a subset, but a useful one so far.  It’s also well worth downloading the manual pdf.

To compile a program with the debug symbols use the -ggdb command line switch.

Here is a basic summary of some of the useful commands.

set args [command line args]
r
bt # stack trace
up/down # move up the stack
p # print
p/x # display hex value
p/x (int[5]) *0xffffd320 # display values at address as integers
x address # display contents of memory location
x/16b address # display 16 bytes
x/32cb address # display 16 bytes as characters
x/5i address # display as instructions (i.e. assembly)
x/5i $pc - 6 # displays current code
info registers # display register contents
$pc, $sp # program counter and stack pointer, can also use $eip
set $sp += 4 # add 4 to stack pointer
set var variablename = 4 # set a program variable
set {int}0x800321 = 4 # sets memory location 0x800321 to 4
j 0x32211 # jump to 0x32211
find # search memory
b *0xaddress # set breakpoint at an address (b nnn sets at a source line)

I’ve also been using lxc a lot recently and being able to create an x86 based machine is useful.  This creates a machine named x86 for testing x86 code.  The apt-get line is to be run on the machine once it’s loaded to install gdb and the compilers.

lxc-create -n x86 -t ubuntu -- -a i386
apt-get install build-essential gdb
Tagged , ,

Catalyst Config Hack

With a lot of modules for our Catalyst systems we have separate models. We then use a subset of them in a single application, and it makes sense to actually store all those database models in a single physical database. This means we end up with a lot of duplicate model config keys in our catalyst config.

<Model::Processor>
    connect_info dbi:Pg:dbname=app_db;host=db-server
</Model::Processor>
<Model::SysParams>
    connect_info dbi:Pg:dbname=app_db;host=db-server
</Model::SysParams>
<Model::AuditTrail>
    connect_info dbi:Pg:dbname=app_db;host=db-server
</Model::AuditTrail>

A lot of database configurations aren’t just a single line, and you end up spending forever copy/pasting and then modifying the config. I wanted to come up with a way to avoid all that repetition.

The Catalyst::Plugin::ConfigLoader provides two potential hooks for things to do after the configuration has been loaded. One is a finalize_config, the other is config_substitutions, via the substitutions settings. Because we are using CatalystX::AppBuilder the finalize_config doesn’t appear to be hookable, or at least I didn’t figure out how to. The substitutions is however perfectly usable because that just requires config setup in code.

   $config->{'Plugin::ConfigLoader'}->{substitutions} = {
        duplicate => sub {
            my $c = shift;
            my $from = shift;
            my $to = shift;
            $c->config->{$to} = $c->config->{$from};
        }
    };

Then this lets me do this in the config file.

<Model::Processor>
    connect_info dbi:Pg:dbname=app_db;host=db-server
    connect_info dbusername
    connect_info dbpassword
    <connect_info>
      quote_char "\""
      name_sep .
    </connect_info>
</Model::Processor>

__duplicate(Model::TokenProcessor,Model::SysParams)__
__duplicate(Model::TokenProcessor,Model::AuditTrail)__
__duplicate(Model::TokenProcessor,Model::AuthDB)__

This copies the configuration specified for the Processor to the SysParams, AuditTrail and AuthDB model config settings. This happens right after the configuration has been loaded, and before the models are loaded so all the settings are there just in time. That saves me lots of copy/paste, and even more editing. I don’t even need to copy those directives into my _local.conf because the _local.conf settings for the Processor model will be what get copied.

Tagged ,

Lenovo S205 wifi woes

I finally decided to stop mincing about my netbooks wifi compatibility woes with Linux and fix them.  My friends and colleagues should be relieved that my constant moaning about the worst linux compatibility problem I’ve had with a computer in a long time will be over.  I should definitely thank them though, as it was their suggestions and research that has helped me solve this issue in the end.

The core problem I’ve had, along with a lot of other S205 users, is that the wifi doesn’t work out of the box with Linux.  This won’t affect every S205, just those that report their wifi hardware switch in a particular way.  The irony is that the wifi driver is loaded and works fine, it’s just that the kernel believes that the hardware disable switch for the wifi has been permanently set to disabled.  

As with lots of manufacturers, Lenovo produce their laptops with lots of variants, so you can choose the specification that suits you.  There may also be variants based on what parts they decided to put under the hood, even though two computers may functionally have the same spec.  They quite handily label these differences as models.  Commonly things like 10382JG and 1038DPG (Lenovo tend to use a consistent length string for the model info which also turns out to be handy).  You can find this model name here, /sys/class/dmi/id/product_name.

This bug appears to reference my problem, but unfortunately, it didn’t actually cure *my* problem.  

Those 3 links are various relevant bugs and patches, but a quick comparison of the models revealed that this didn’t fix my problem, simply because my model was different.  Looking at that patch shows that there is a quirks mechanism to detect  the S205 based on the product name.  If that’s the case then it looks at EC register 0x78 for the hardware switch.

The first thing to do was to confirm whether my computer used that same register, in which case I’d just need to enable the quirk.  Otherwise, I’d have to delve deeper, and add my own.

I downloaded acer_ec.pl from this site – http://code.google.com/p/aceracpi/wiki/EmbeddedController.  And gave it a run.

When I flipped the hardware switch it did appear to signal using the 0x78 EC register as suggested in the patch.  Since it did, it looked like all I’d need to do is enable the quirk the patch is designed to enable.  

Here be dragons – don’t do this unless you’ve read it all, and made backups.

I then started to prepare my computer to build kernel modules before laziness kicked in.  I found my acer-wmi.ko module in /lib/modules/$(uname -r)/kernel/drivers/platform/x86/ and  took a backup of it.  I then took a second copy of the file and modified it.  I found the existing reference to 10382JG and changed the 2 letters ‘2J’ at the end to DP to make it specify my module.  This was possible because the new value I wanted was an identical length to the existing one.  I then did an ls -l to check the files were the same length as a sanity check.  Once I copied the modified module back to it’s original location and rebooted I had wifi!  Theory confirmed.

The tool I used to modify the file was bless, since obviously using a text editor for that is generally not a good idea.  Just remember to make sure you switch to overwrite mode and that you don’t append any characters to the file.

Now all I need to do is ask the nice kernel people to add a reference to the quirk for this model.

Tagged , ,

Skippng Python unit tests if a dependency is missing (fixed)

I got some feedback on my previous post about skipping tests in python unittests pointing out my solution was flawed.  As Mu Mind pointed out, the denizens of stackoverflow pointed out the solution has a problem when run directly from python.  At first I didn’t realise how flawed; technically I had run my tests via python and nosetest regularly.  I just hadn’t realised that I’d never run the tests via python when I was missing the dependency.  If you do that you get this ugly error,

ERROR: test_openihm_gui_interface_db_mixin (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_openihm_gui_interface_db_mixin
Traceback (most recent call last):
  File "python2.7/unittest/loader.py", line 252, in _find_tests
    module = self._get_module_from_name(name)
  File "python2.7/unittest/loader.py", line 230, in _get_module_from_name
    __import__(name)
  File "tests/test_openihm_gui_interface_db_mixin.py", line 6, in 
    raise unittest.SkipTest("Need PyQt4 installed to do gui tests")
SkipTest: Need PyQt4 installed to do gui tests

It does tell you what the problem was clearly, but it really wasn’t the intention.  The idea was to silently skip the test.

From the answers and comments on the stackoverflow post I stitched together this ugly but hopefully working solution for whatever your unit test runner of choice is.

import unittest
try:
    import PyQt4
    # the rest of the imports


    class TestDataEntryMixin(unittest.TestCase):
        def test_somefeature(self):
            # actual tests go here.

except ImportError, e:
    if e.message.find('PyQt4') >= 0:
        class TestMissingDependency(unittest.TestCase):

            @unittest.skip('Missing dependency - ' + e.message)
            def test_fail():
                pass
    else:
        raise

if __name__ == '__main__':
    unittest.main()

I dislike the fact that I can’t hide away the logic at the top, but surrounding your whole test with the code works.  Then if the import fails we create a dummy test case that does a skip to indicate the problem. I’ve also tried to ensure that we only catch the exception we’re expecting, and pass through any we aren’t.

Now if you run the tests in verbose mode you’ll see this when there is a missing dependency.

test_fail (test_openihm_gui_interface_mixins.TestMissingDependency) ... skipped 'Missing dependency - No module named PyQt4'
Tagged ,

Hooking into the OpenERP ORM

I’ve been hooking into the OpenERP ORM layer of a few of the models to add full text search via an external engine. Thanks to the way OpenERP is structured that appears to be quite a reliable approach. As I was doing it I found that I wanted a common set of hooks on several models. Doing that suggested I should refactor the hooks to a mixin or a base class. After playing about with my OpenERP module I’ve come to the conclusion that creating a base class seems to be the most reliable way to hook the methods. The one trick you need to be aware of is the _register flag that you want to set to False for your base class.

class search_base(osv.osv):

    _register = False

    def write(self, cr, user, ids, vals, context=None):
       success = super(search_base, self).write(cr, user, ids,
                                                    vals, context)
       … do stuff
       return success


class product_template_search(search_base):
    _name = "product.template"
    _inherit = "product.template"
    _register = True


class product_search(search_base):
    _name = "product.product"
    _inherit = "product.product"
    _register = True

product_search()
product_template_search()

Without that you’ll end up with the orm whining that the search_base class has no _name attribute as it tries to register it as a model.

openerp.osv.orm: The class search_base has to have a _name attribute
openerp.netsvc: ValueError
The class search_base has to have a _name attribute
> /usr/lib/pymodules/python2.7/openerp/osv/orm.py(967)__init__()
-> raise except_orm('ValueError', msg)
Tagged ,

Skippng Python unit tests if a dependency is missing

Update: this is a flawed method of skipping the tests, I’ve written up an improved version based on the feedback I received.

In all the examples of using python unittest to skip tests I’ve not seen anyone explaining how to skip tests if a library isn’t installed.

The simplest way to do it appears to be to simply try to import the relevant libraries and catch the exception thrown if the library isn’t there.

import unittest
try:
    import PyQt4.QtCore
    import PyQt4.QtGui
except:
    raise unittest.SkipTest("Need PyQt4 installed to do gui tests")

This example at the top of a test file simply skips all the tests if PyQt4 isn’t available.

Tagged

Perlbrew with options like threading

This blog post on the *n*x blog gives a great description of how to install perl with threading, something that you need to do if you want to run padre.

The only thing I’d add that you can use the –as to install perl with an alias. This is useful if you want to build a threaded version of a perl you already have installed. You can simply do,

perlbrew install perl-5.14.2 -Dusethreads -Duselargefiles -Dcccdlflags=-fPIC -Doptimize=-O2 -Duseshrplib -Duse64bitall -Darchname=x86_64-linux-gnu -Dccflags=-DDEBIAN --as threaded-perl-5.14.2

(Note that I’ve customised this for my Ubuntu 64 bit os).

Tagged
Follow

Get every new post delivered to your Inbox.

Join 64 other followers