The Dumping Ground

Stripping lines from a file, a quick perl one liner

colinnewell — Tue, 11 Aug 2009 17:39:00 +0000

I’m getting bored of figuring this one out from first principles.

perl –n –e "if(!/^(SOMESTART|SOMEOTHERSTART)/) { print; }" filename.csv > smaller.csv

This is using the loop –n does, similar to the trick I’ve used with –p, it’s just that the –n doesn’t print automatically. Instead I call print manually if a pattern is matched, or in this case, doesn’t match.

Stack overflow rocks

colinnewell — Tue, 16 Jun 2009 18:11:00 +0000

I’ve still yet to gain any reputation on Stack overflow but it’s already proving to be useful. It’s already answered some of my questions on MVC development and now it’s just saved me some pain and frustration while wondering why my code to use a FoxPro database isn’t working on my laptop.

As a side note on someone’s question Brian Behm has made a note that you need to be in 32 bit mode to use the Fox Pro Ole DB driver. That explains why I was getting the ‘The 'vfpoledb' provider is not registered on the local machine.’ exception despite having installed the driver. All I need to do is add an x86 platform to my project and when I build with msbuild specify the platform using /p:Platform=x86. I go through this pain so the users don’t have to!

The only down side is that it’s not exactly searchable. The only reason I found it was because I’d reached desperation search mode. I guess I should just ask the question properly on Stack overflow and then he can answer it again there and we’ll both gain some reputation points and it will be searchable for others.

I suppose it would be neat if you could break existing threads up on Stack overflow so that you can mark a comment as being the answer to a different question.

VO –> Vulcan regex

colinnewell — Wed, 27 May 2009 23:07:56 +0000

We’re about to do a bunch of ‘Transporting’ from VO to Vulcan in the next few days so I’m probably going to figure out a bunch of useful tricks for helping to automate the process.

So far things are looking good, our unit tests our now transporting cleanly and there’s no incompatibilities exposed by them.

One thing I did need to do was remove all the PSZ <-> STRING conversions. The easiest way to do that was to use Visual Studio’s ‘Replace in files’ option and some regex’s.

String2Psz({.*})
AsPsz({.*})
Psz({.*})
AsString({.*})

They are the things you want to find, replace them with \1

Note that Visual Studio doesn’t appear to support .*? so those matches may be too eager. In my case I just ended up with extra brackets left around my variables. Syntactically okay but ugly looking.

Thinking about it, I have the feeling last time I gave up on Visual Studio and just used perl the last time I needed to do this. I guess if I have any more complex things to do I will.

More Logging

colinnewell — Mon, 04 May 2009 21:38:18 +0000

Now that I’ve figured out what I want a logging framework to do I’ve taken a look at some interesting existing frameworks. log4net is a .net logging framework based on a java framework – log4j. It’s hosted on the apache website which gives it extra credibility in my view. Looking at the feature set it looks like it covers pretty much what I need.

As far as sending emails go there’s a convenient way to get emails out when there’s a problem with a sample config in the examples.

What that does is send an email when a warning comes along and also sends upto the previous 512 messages (of any kind) as context. That wasn’t exactly what I was thinking of but it’s certainly good enough for the emails.

What I want from a logging framework

colinnewell — Mon, 04 May 2009 00:43:29 +0000

I keep being unsatisfied with our applications logging and I’ve decided to take a look at what I really want.

Obviously we want different levels of severity,

Exception
Error
Warning
Note
Debug

Those severities imply that we can filter by them and that we can possibly do different things based on encountering them. We also need to be able to specify different targets – file, debug view, event log and email.

There’s also another aspect that we don’t often think about but which I think is quite important, tying together the log messages into some sort of unit. This might be something like the import of an order on a system or perhaps a web page request.

There’s another thing related to that, timing. Not everything wants to be logged immediately. When we log straight to a file or a debug log we normally just keep writing the stream as we encounter it. If we want to log to an email we might want to batch together some of those message though. That also ties in quite neatly with the idea of units of work.

This would mean that if I have an error occur within a task I could see all the messages within that task, not just the errors. This would be an ‘intelligent’ mode of operation, rather than just one tied to a severity level.

This gives us a set of features something like this,

Severity filtering
Tying messages together into a unit of work or task
Delayed sending
Possibly optional logging based on severity, i.e. only display the notes if an error occurs.

Honestly we probably also want to have several separate configurations in action at once too so there’s the another feature to consider,

A flexible configuration mechanism – this should be able to deal with multiple setups running at the same time

This makes me think a simple interface like this would be best,

public interface Logger
{
	void StartLevel(string name);
	void EndLevel(string name);
	void Error(string message);
	void Note(string message);
	void Warning(string message);
	void Debug(string message);
	void Exception(string message);
}

All that we would do is to create new classes that implement that to do the functionality we want. We presumably want them to be able to be composed of different ones and have different configurations but essentially when we log we log through an interface like that and it gets sent to a bunch of objects exposing that API.

In practice we would probably use it something like this,

Log.StartLevel("Interface run");
Log.Note("Interface running");
Log.StartLevel("Order Import");
Log.Note("Importing orders...");
Log.Note("Importing order XXXXX1");
Log.Note(" for destination WHS1");
Log.Error("Product D898223 does not exist, order XXXXX1 not imported");
Log.Note("Importing order XXXXX2");
Log.Note("Importing order XXXXX3");
Log.Note("Importing order XXXXX4");
Log.EndLevel("Order Import");
Log.EndLevel("Interface run"); // an email gets sent if an error occurred

One thing that irks is that if we have several paths storing the messages for some condition to become true, like sending an email when the interface has run or logging if an error occurs. We could even be storing all the debug messages when we are highly unlikely to send them. In practice though I think it’s worth keeping it simple, if there are really enough messages building up between flushes I suspect that we must have the applications architecture wrong. If not it can always be optimized then.

The configuration is the tricky bit.


c:\logs\error_log.txt
Warning




colin+newell@rugby-software.co.uk
smtp.someserver.com
interface@rugby-software.co.uk
Interface problems

Error
Interface run
Note

I think that makes sense as a basic API and configuration mechanism. I think in practice it will need tidying and some more complexity adding but that should be the basics. All that I now need to do is the magic in between to make it work.

There’s also the problem of what to do in the case of exceptions to resolve. Ideally in those cases you would shortcut the trigger levels; especially since if we’ve just caught an exception it’s probably just busted through all those carefully constructed levels. Alternatively perhaps you simply add an alternative configuration block for them. Then it’s explicit and there’s no hard wired behaviour or complex contortions to try to express something complex in xml. Perhaps we’d just add a ‘flush log’ to implicitly dump all the previous log messages stored along with the actual exception report.

.Net deployment

colinnewell — Fri, 24 Apr 2009 07:46:41 +0000

Now that I’ve given the talk on .Net Deployment I figured it would be worth putting together some links for people wanting to find out more about what I said.

As a refresher here are the titles of my slides,

Introduction
.Net avoids DLL hell
Get the right DLL’s
Strong Names
32 Bit & 64 Bit
Vista
Virtualization
Manifests
Code access privileges
Settings
Vulcan
Installer technology
UAC
Installers and x64
Automating things
Debug information
Other dependencies
Testing the installer

Here are some links to some of the things I talked about,

Figuring out dependencies

Strong names

Code Access Security

Virtualization/UAC

64 bit

Installer technology

Settings

http://bytes.com/groups/net-c/522191-how-maintain-user-config-settings-when-upgrading

VO 2.7 Build system

colinnewell — Sun, 19 Apr 2009 19:58:08 +0000

After far too long without a proper way to build our applications from the command line in VO 2.7 we’ve finally created a program to do that.

There was one hitch we had when building our program; getting the Assembler to compile the modules that needed compiling. We use the VO27NASM.DLL plug in which is essentially just NASM embedded into a dll to compile some assembler code.

There were 2 problems with the assembler code. The first was that after importing the AEF the assembly code wouldn’t compile, even from within VO itself. This turns out to be because they weren’t imported as source code entities. We fixed that by replacing those entities with an identical entity that we created as a source entity. Once we’d done that the application could be built from within the IDE.

The second problem was that compiling from Adam within our app didn’t work. To fix that we just needed to call SubSysInit in the CAVOSSYS . This loads in the plug-ins like the assembler.

This is the import you need for the SubSysInit function. You can call it with all 0’s.

_DLL FUNC SubSysInit (hMenu AS PTR, uiFirstMenuID AS DWORD, wMenuCount AS DWORD) AS LOGIC STRICT:CAVOSSYS._SubSysInit@12

We haven’t tried VO 2.5, 2.6 or2.8 yet but I’m guessing that it should all be essentially the same.

Note: VO refers to CA Visual Objects – http://www.cavo.com/. If you don’t know it don’t worry about it!

Birmingham Perl QA Hackathon

colinnewell — Tue, 31 Mar 2009 20:34:11 +0000

I’m back from the 2009 Perl QA Hackathon (http://qa-hackathon.org/) where I acted as a spare programmer.

Barbie and JJ and the Birmingham Perl Mongers (http://birmingham.pm.org/) organized an excellent event. I’m really going to have to do some exercise after eating so well too.

For my own reference if no-one else’s, here are some of the places I’ve just eaten in Birmingham so that I can remember them.

Shimla Pinks – An upmarket Indian restaurant
The Handmade burger company – decent burgers
The Thai Orchid on Bennetts Hill – Excellent food and service.
Bar Room Bar at the Mail box – excellent pizzas

Perl QA Hackathon – Test::Builder2

colinnewell — Tue, 31 Mar 2009 20:20:58 +0000

At the QA Hackathon (http://qa-hackathon.org/) I helped Michael G Schwern (http://use.perl.org/~schwern/journal/) get over his metaphorical hump with the design of the new Test::Builder2. I was one of the programmers without an agenda at the hackathon there just to try things. A disinterested+ party, I pitched in (at least a bit) when a bit of feedback and programming was called for.

To explain what we tried to do and why we were doing it I’ll explain what the Test::Builder is first. The Test::Builder is the library used under the hood by various testing libraries including Test::Simple and Test::More to produce the TAP output and manage the test numbers so that they don’t end up out of line. It also does a few other convenient things but those were the core bits we were focusing on. To do that it’s a singleton and has a bunch of helper methods for those libraries that use it. While this Builder library may change, the existing test libraries aren’t likely to change, except perhaps to add new facilities.

Schwern had some goals with the new version,

Make the output configurable and replaceable (then you wouldn’t be tied to TAP)
To make it so that you can tell whether you are currently passing the tests (so that you can tag on extra debug output at the end if you’re not)
To allow extra information like diagnostics to be tied to tests more easily

The second item, the telling whether you were currently passing your tests was something he’d already done with a history object.

The configurable output was a relatively simple thing to do. For now we’ve created a class that implements a few basic output messages for start, end and test result and set the test builder to be created with a default implementation that outputs TAP. Schwern is now looking at other testing frameworks to see if other messages are likely to be needed. His look at the POSIX testing framework for example suggested there may need to be some additional start/end messages at different levels.

Allowing extra information to be associated with the tests is a little trickier. Schwern had already figured out that the way to do it was to use a result object so that you had something to tack the extra information to. This way rather than simply returning a pass or fail the test methods will return a ‘Result’ object. You can then tack on extra diagnostics or mark the fact that it’s a todo.

In the simplest case to mark an object with some diagnostics,

ok($func('a'), 'some test')->diag('some cheap diagnostics');

Or if the diagnostics are expensive to run and you only want to run them when the test fails,


{
	my $result = ok $func('a'), 'A test';
	if(!$result) {
		$result->diag("expensive diagnostics we'd rather not run");
	}
}

There are a couple of things to note in this instance. The first is that you can do a true/false comparison on the object to see if it’s passed or failed. One delegate described this as the ‘truthiness’ of the object. This way we can check easily whether it’s a pass or fail without checking the ‘is_fail’ property (as it was named at the time I was writing this).

The other thing to note are the scope blocks. They are probably good practice around tests in general but in this case they are essential. The way that the builder knows when to display the output is by only doing so when you are finished with the object, i.e. by waiting for the object to be destroyed. Without those scope blocks the destructor wouldn’t be called in the appropriate place. This design decision seems like the best way to give you a simple interface while allowing you to deal with the more complex scenario’s without too much difficulty. You can of course make a mess of it, but that’s programming for you.

Another bit of ugliness is that the result isn’t actually the real result but a wrapper that implements the destructor functionality and pretends to be the result. That’s because the result is a pure data object and because it’s also stored in the history and so wouldn’t naturally be destructed at the correct point just because your code doesn’t reference it at the test any more. The wrapper just acts as a facade on the result and is given the output object by the builder so that it can output the test result when the wrapper is destroyed.

The syntax also allows you to set a single test as a TODO very easily.

ok($func('a'), 'some test')->todo('still need to implement this');
You can chain multiple pieces of data together and each setter returns the result back so that you can keep doing it. It’s also possible to set a todo without a message. The net effect of that is that it’s not possible to use the todo call to tell whether or not the result is a todo or what the reason is. Sounds obvious when I say it like that but it was something we had to think about! Instead there is a ‘is_todo’ property and a ‘reason’ property for those pieces of information.

Note that setting the information like that doesn’t mean that you cannot use the existing libraries features like Test::More’s TODO blocks. They can still be used for multiple tests where they are appropriate; it’s just that now you can also do quick todo’s with minimal fuss too.

The weekend was spent figuring out how to meet these goals with an implementation that was reasonably clean. Schwern had already figured out the basic mechanics of it, we just thrashed out what worked practically and what didn’t. After a few rewrites of parts of the structure it’s starting to look like something useful. What I’ve mentioned above was roughly where we were by the end of the weekend. Whether it still looks like that now is another matter. Of course how much you’ll actually see of these changes I don’t know, this library is used within most of the other test libraries so most changes will be invisible to the users.

+ disinterested in the sense that I’ve not got a vested interest in the development of the QA or testing code. As far as I’m concerned it already works great and I’m really grateful for it.

Tweaking the AntiForgeryToken on ASP.Net MVC RC2

colinnewell — Thu, 05 Mar 2009 09:34:42 +0000

After a bit of thought I realised that my old helper extension method to set the path of the anti-forgery cookie was broken by RC2. Steve Sanderson did point out that they now allow you to specify the path up front when you create the token though so it actually makes my helper even simpler.

public static string MyAntiForgeryToken(this HtmlHelper helper, string salt)
{
    string fragment = helper.AntiForgeryToken(salt, null, helper.ViewContext.HttpContext.Request.ApplicationPath);
    return fragment;
}