Whip IT Good (hopefully)

backbone.js with require.js: a couple quick notes including _ is undefined

2012-01-07T09:04:00.000-08:00

Require.js is a JavaScript library which allows for use of the AMD spec which can allow JavaScript code to finally become organized and allow for powerful optimization. Backbone.js is one of the several MVC solutions which essentially allows for a more declarative style of programming rich Internet apps without getting lost in a tangle of primarily imperative code. Or more succinctly it takes care of most of the typical plumbing and SoC so you can focus more on app specific code.

I just spent a little bit of time trying to get the 2 libraries to cooperate. There are several recipes that address this out on the Internet, but I encountered some issues and so spent a little more time figuring out what was going on. This may have been primarily due to an issue with my inadvertently using a leading slash in the require.js script tags data-main attribute and thereby canceling out the module loading behavior. This was a result of the way the application's (server side) view layer was being implemented, and I'm not feeling ambitious enough to do further testing on it to isolate the behavior down enough to have a firm answer.

The primary quirk in this combination of libraries is that there is presently partial support for require/AMD within Backbone and Backbone's "suspenders" underscore.js (I would guess this support is evolving). As it stands right now, underscore will export itself as an AMD module if require is present. Backbone will actually use this definition to load underscore if needed...however Backbone does not export itself as a module.

As many of the other recipes for this set-up advocate I've adopted the practice of using proxy modules to load libraries which do not have module support: the proxies basically serve to rip whatever is needed out of the global/window namespace and return it explicitly. In my present set-up therefore there are loaders for jQuery and for Backbone, but underscore can be served directly.

There was actually an additional complication due to this issue. Because underscore supports require.js, a call to require("underscore"), assuming the path is properly set-up, will return the expected object that can be assigned to "_". Unfortunately, the Backbone code that I was working with was attempting to assign along the lines of 'var _ = require("underscore")._'. That trailing selector was therefore trying to get the "_" member of the typical "_" object and was returning undefined. This reflects a possible incompatibility between the 2 immediate versions of the 2 libraries. My solution was to simply remove the trailing '._'. A more robust workaround though less conceptually sound solution would be to create the "_" member within the underscore module (or more appropriately a loader proxy) which returned a reference to itself.

My Adoption of PHP

2011-12-24T08:17:00.001-08:00

Sooner or later I'll get this whole blogging rhythm down.

The majority of my independent work is going to be switched to PHP. This is a decision dictated by pragmatism, which is I suppose the principal reason one would adopt PHP. PHP from what I've seen appears to be the ubiquitous language for independent Web development, including some projects with which I am involved and cannot port. More directly I'd like the option to hand off any of my work and therefore the numbers of PHP developers running around will make that hand off more available.

As outlined in a previous post, I want to try to stay true to what I view as the strengths of PHP. This can more or less be reduced to the idea that the moment I start to think "I wish I was using language X" (most likely Java) it's time to re-evaluate what I'm doing and look at shorter paths to all the out of the box functionality that PHP provides.

This post is an introduction to a new thread of development. I am in the process of designing a PHP framework which will pursue this goal while following best practices. It will start life as heavily object oriented to ensure backwards compatibility. To further fly in the face of my goals it will also use the Java style declared Interfaces that PHP offers so that objects can be doubled for TDD. Once the framework is in place, however, other paradigms and the simplest most fitting approaches will be integrated (such as using first class functions from PHP 5.3 and providing richer support for procedural code where sensible). I will also attempt to borrow from other frameworks, and possibly abandon the whole project if I find one that doesn't carry too much baggage with it.

Non-PHP tools will also be used for some parts of the development process. Likely due to the common treatment of PHP as part of larger LAMPy stacks PHP lags behind languages like Ruby when it comes to automated building, deployment, and functional testing. I unfortunately have found PHing to be too Java-esque: adding layers of abstraction to a language which is capable of simple scripting. I will be using Gradle, which is (ironically) a popular Groovy/Java build tool. Ruby has a killer tool selection and is also a very good choice (and what I am actually moving away from). But Gradle wins out for now since: it's what I've started using at my day job; could be used to run PHP with Quercus or something similar; can tie in with Geb (www.gebish.org) which at a glance seems slightly cooler than the Ruby WebDriver tools; runs on the JVM and can be self contained rather than expecting a less common Ruby environment.

So...there's a quick preview of what will be upcoming on this blog.

How I Use Security through Obscurity

2011-10-30T09:10:00.000-07:00

Security through obscurity (http://en.wikipedia.org/wiki/Security_through_obscurity) is the often deried practice of what amounts to hiding. It is potentially a dangerous technique in that it can introduce a false sense of security where none necessarily exists. On the Internet it is the equivalent of putting a lamp shade over your head and hoping not to be seen.

It is something, however, that I do make heavy use of in solutions that have light security needs. In particular, I often run services on non-standard ports. I normally do this to hide from script kiddies: those that use automated scripts to look for exploitable services on the Internet. Normally these scripts will work in bulk, and scan massive amounts of machines and therefore cannot spend any significant amount of time looking at any one machine. Using the lamp shade metaphor, it is the equivalent of looking through a window will driving past in a car. Of course if anyone where to walk past or even drive slowly enough, the service would be discovered.

The end result is therefore that I don't rely on it to keep those service protected. What I do use it for is to keep my logs clean of endless failed, unimportant authentication attempts and to keep the services and systems from being hammered by blind brute force attacks. In an installation requiring robust securit these issues would be better addressed by using more intelligent firewalling and monitoring/IDS solutions, but in a quick implementation with minimal need for security, using obscurity can offer the most cost effective initial solution to minimizing, and highlighting, unwanted traffic.

Why formencode/htmlfill style form handling doesn't belong in PHP

2011-08-07T14:04:00.000-07:00

A couple posts ago I wrote about starting a project in PHP that more or less recreates what the htmlfill component of Python's formencode does...allowing a stream of HTML to have the proper error messages added for form validation without having to actually change the HTML. As I went to use this library again it suddenly struck me that this was a potentially very stupid idea. Here's the deal:

There is the immediate thought that this type of filter-based solution would introduce a performance hit when compared to integrating error messages during initial rendering. Since I'm one of those lazy programmers that doesn't care about these types of issues until I have to (or have reason to believe I may have to), that wasn't a particular deal breaker. More significantly, intelligently dealing with a stream of data can be fairly complicated, particularly when adding flexibility. I think I had worked out a system in my mind that handled this fairly decently (I hadn't yet gotten around to just stealing algorithms from formencode), but it still had potential to induce head scratching.

Still...neither of these problems are necessarily a deal breaker. What did stop my ambitions dead however was the realization that what makes this an attractive solution in Python does not apply to PHP. Formencode in Python allows for minimally intrusive form validation which can operate fairly independently of the other libraries involved: most relevantly the view/templating technologies that provide the stream in the first place. When writing with PHP on the other hand there is a technology that is a constant in that layer...PHP. By using well structured PHP in the form code you get better performance, more flexibility, and most importantly of all...the simplest solution for rendering form errors. I would recommend keeping the code that is intermingled with the HTML more on the declarative side, but it can still be a short path to grok all the logic.

Now the fate of the complementary validation library I had started is substantially more in doubt...

Spring and Tiles on Google App Engine 1 - Ivy

2011-07-24T06:22:00.000-07:00

So my plans on posting more regularly have still not solidified. Also my idea about doing a series on PHP is going to wait a bit until I venture back into spending some time in that language. For now, I'll put together some information for Java, a language I've more or less avoided for several years but am now finally breaking down and spending some quality time with.

Introduction & Technologies Used

This is going to be some quick tutorials on getting a project set up to run Spring MVC on GAE using Apache Tiles to handle the view. Google offers a very nice plugin for Eclipse (which I'm also breaking down and using again) to help with project management. My personal preference is to do things more explicitly and with more portability, so although I'm using Eclipse I'll be sticking to tools that work outside of Eclipse (and are more transparent within Eclipse). For the moment at least that leads me to Ant since it is far more fine-grained than the main alternative (Maven) and it is also the tool used by Google.

Ivy is the dependency management complement to Ant. This will be used to...well...manage dependencies. I prefer an automatic system primarily so that I can check all the info that is relevant for a project into a source management system without including files that don't need to be tracked.

Files

Ant is generally driven by an XML file, traditionally "build.xml" in the root directory of a project. Ivy typically uses an "ivy.xml" in the same location. The tutorials on the Ivy site cover the basics of loading the Ivy namespace into the Ant file, creating the resolve task, and creating the Ivy file.

Retrieving to /war/WEB-INF/lib

Ivy typically retrieves files into a "lib" directory in the project directory. In order to place the downloaded files into the typical location for web projects the pattern needs to be modified. There is an additional issue also in that Ivy by default will retrieve multiple types of resources (jar, source, etc.) to satisfy dependencies. Therefore the pattern either needs to separate the types to avoid collisions, or the types need to be filtered. In this case I'm only concerned with JAR files, so I'll be specifying that I only want JAR files. Some types will also have additional dependencies, so specifying jars keeps everything focused.

So in the build.xml file I want to specify where I want the libraries placed:

<property location="war/WEB-INF/lib" name="lib.dir"></property>

And then within the resolve task I want to modify the pattern and specify that I only want JAR files. The end result is something along the lines of:
<target description="Resolve Ivy deps" name="resolve">
<ivy:retrieve pattern="${lib.dir}/[artifact].[ext]" type="jar"/>
</target>

Now Ivy should be all set to download dependencies and shove them into the proper place.

Dependencies

Now it's simply a matter of specifying dependencies in the Ivy file and running the resolve task. As mentioned I want to use Spring, Tiles, and GAE. Dependencies can initially be located at http://mvnrepository.com, and there's a tab to display Ivy format so that the elements can be cut and pasted. I'm going to use Spring 3 since Spring 2.5 has issues with newer versions of Tiles. So the dependencies element is along the lines of:

<dependencies>
<dependency org="com.google.appengine" name="appengine-api-1.0-sdk" rev="1.5.1"/>
<dependency org="org.springframework" name="spring-webmvc-portlet" rev="3.0.5.RELEASE" />
<dependency org="org.apache.tiles" name="tiles-servlet" rev="2.2.2"></dependency>
<dependency org="org.apache.tiles" name="tiles-jsp" rev="2.2.2"></dependency>
</dependencies>

Now after running the resolve task the required JARS will be downloaded into the appropriate place, and code can be written using the specified technologies.

What PHP means to me

2011-07-01T17:53:00.001-07:00

I've been meaning to write on this blog more regularly since at some point it actually started acquiring traffic, and now my schedule is starting to stabilize enough to actually do so. I'm going to start with some PHP posts and this one will cover my take on PHP.

Introduction

To paraphrase Donald Knuth, programming is the art of telling another person what you want a computer to do. In other words, you're speaking through the language designer. I personally don't just want to tell anyone what I'd want a computer to do, so I therefore want to know why and when I should bother working with a particular language. I also enjoy being understood, so I don't want to be jamming grammar in that doesn't quite fit and be left thinking "this would be so much easier if I were speaking...".

A brief history of my working with PHP

I began using PHP about 10 years ago. At that point I started to write a decently complex application with it, but coming from a desktop application programming perspective I found PHP a little rough around the edges and ended up trading it in for Perl, which seemed more equipped to deal with my needs at that time. Since then I've done assorted one-off jobs with the language, and have recently begun dealing with it more seriously and have now finally found how to fit it in without thinking of it as the worst of the P's (the others being Perl and Python).

A brief history of PHP

PHP was created as a toolkit of functions written in C, and the language itself was created to work with those functions. The language has since evolved following the same pragmatic perspective of creating utility functionality and associated support. This is in contrast to other languages which are designed rather than evolved. Most languages are focused around the concepts of primitives and means of combination so that libraries can be created in the language itself and concepts like self-hosting can be attained; the language makes the libraries. PHP is somewhat the inverse in that the language fits around the libraries.

The Community

PHP probably has the most diverse user base of any server side Web language. I tend to feel that most gravitate towards two extremes: amateurs that hack and slash to make things work, and people coming from other languages who carry with them all of their favorite patterns and ideas about how things are supposed to work. The post-Rails era and all of the OO goodies of PHP5 have probably furthered this fragmentation.

The Diamond

PHP's power is the reason behind its popularity: utility functionality. PHP is built around nice black boxed doWhatIWant($WithThis) functions rather than libraries composed of smaller pieces. My take on the language then is to embrace those functions as directly as possible. This is, of course, potentially horrifying to methodology evangelists since it amounts to advocating *gasp* unadorned structured programming. It took me a bit of regressing and looking at extending fopen() for this benefit to coalesce out of the fog of a language that otherwise might seem like a poor man's Java. Since then it's been reinforced through dealing with even those object related functions in the core language which do not use a method invocation (likely not so much for consistency as to allow for a cleaner interface to the lower level code).

Conclusion

Multi-paradigm languages like PHP too often result in "use your favorite paradigm" rather than "mix and match for the task at hand". PHP provides a wide array of accessible functionality out of the box. This is the ugly, practical heart of PHP and what separates it from general purpose programming languages. The language itself provides for diverse approaches in structuring an application (an OO-approach is probably the best bet for the time being), but the standard, tested power and reliability of the accessible core (or extension) functionality should be utilized as much as possible and as transparently as possible. The more PHP is dressed up, the more it looks like an ugly programming language; the more it is stripped down, the more it feels like a very powerful set of tools.

First Impressions of the Go Language and Gratitude for Implcit Interface Implementation

2011-06-06T07:29:00.000-07:00

Google has released the Go Runtime for Google App Engine which uses their newish Go language (http://golang.org/). Being a fan of GAE and also trying new things I took Go out for a test drive.

I didn't use it enough to appreciate any of it's advanced usage, such as it appearing to have strong concurrency support, but wrote enough to get a feel for the day to day coding.

Raw

One of the big complaints about the language is that it is "ugly". The language definitely reflects the designers's backgrounds in 1970's programming. It it is a very terse syntax reminiscent of C and other older Von Neumann languages which can seem a bit out of place among modern languages. The syntax feels far closer to the metal than even Java, which is furthered by the fact that the language seems to utilize practices (such as object orientation) as tools rather than enveloping methodologies.

I personally enjoy this approach as a more accurate correspondence of what and why is taking place, but it will likely limit the adoption of the language. It can easily appeal to those coming from a C background or some functional languages, but it could be a big turn off for even Java programmers, let alone higher level languages like Ruby.

Implicit Interface Implementation

Really the purpose of this review is to rave about this particular feature. First some complaining: Java uses the concept of declared "Interface"s to separate interface from implementation and to avoid the complexities of multiple inheritance (it may be from an older language, but Java's is the dominant implementation now). This idea has been integrated into other languages, even sometimes jammed into where it doesn't really fit. The basic idea in Java is as follows...an Interface defines behavior that is to be implemented, called code says that it requires an object implementing an Interface, and then any class that is used in client code must implement the interface behavior and is declared to do so. So that's a decent way to ensure that the requirements of the called code are met without imposing too much on the client code.

The concept is sound, it provides a nice declarative way to check at compile time that interfaces are being satisfied. There's a questionable aspect though: the called code requiring an Interface ensures the behavior will be provided...but should the class have to declare that it implements a particular interface? That has the subtle implication that the class's concerns cross the fine line between how it's going to be used and simply what it's doing. In a static language that also implies that both the class and and the called code need to have access to the Interface definition. That may be a trivial concern but could also introduce breakdown in modular boundaries that aren't necessary. As an example of where related issues may cause a headache, consider a situation where called code only relies on a subset of a previously defined Interface, therefore accepting any classes that implement that Interface in addition to any new classes which implement the subset Interface. With explicit implementation there are several ways to work through this scenario, but all of them involve introducing extra, polluting code in one place or another to work around a feature of the language which is supposed to make things easier.

Go uses implicit implementations, so only the called code need be concerned with the Interface. This consolidates validation neatly at the point where it is needed. The created classes need only be concerned with what they want to be concerned with, and as this is a compiled language the appropriate errors will still be thrown during compilation and the performance won't be impacted.

htmlfill style filter based form validation in PHP: Zend Framework - Iteration 1

2011-06-04T12:25:00.000-07:00

Background

Motivation

The handling of forms is always a fun topic in Web application development, with plenty of people and plenty of opinions. Out of the assorted solutions that I've used, the one that I've essentially established as a baseline is Python's formencode with htmlfill. Unlike many rivals formencode avoids form generation, focusing entirely on validation and conversion to a native (Python) structure. htmlfill is a nice complement that allows standard "static" HTML forms to be filled in with the appropriate values and error messages. This approach can easily be unobtrusive, lightweight, and customizable. The lack of code-generation of forms is also more designer friendly, and can allow the code that it present to focus on more relevant tasks.

On Using Zend and assorted thoughts

I'd ultimately like to move this over to a general purpose PHP library, but the project that I'm presently working is using Zend and therefore some of the functionality afforded by the framework will be used in this iteration (and later more general interfaces may be added). In particular, Zend_Form will be used to represent the required Schema of the data (obviously without the rendering aspect). Depending on how Zend_Form is written (I haven't looked yet), this may be overly wasteful. One of my principal issues with a lot of these libraries (including formencode), is that they move validation to what is essentially the View layer. This may be applicable in some cases, but if the validation is to enforce invariants that apply to the objects themselves, I would personally like to see this handled in some way within...well...the objects themselves. If I continue to develop this that is likely a direction I will take.

On Filtering

As this particular case is dealing with XHTML, I'm going to use functions that focus on XML. In particular I've adopted the SAX based xml_parser PHP object. This can be made to scale well particularly as it is assumed that most of the content will pass through unchanged. Further, there is a natural mapping between the selective modifications and the use of callbacks.

Objectives

My particular objectives in this case are to

tag of fields with errors (through use of the class attribute

add a span element with an error message

repopulate the form with previously entered data

Disclaimer

I had some issues with capturing returned values that are seemingly destined for output. Someone more familiar with PHP and/or ZF will likely be able to significantly improve the code.
Also this is written for 5.2 so some fun stuff 5.3 offers was out of bounds (closures would help). Also very little has been cleaned up, and there are plenty of points for customization and extension, but right now it's KISSing rigid.

Name

I'm calling the package FormFillter at the moment. The 2 "l"'s are likely to cause consistent typos but serve to make the name a recursive portmanteau of "fill" (referencing htmlfill) and "filter".

Zend Setup

As already mentioned I created a Zend_Form to represent the data constraints. In order to run the filter I've used the postDispatch hook. Presently this is done in the action controller, though it could likely be moved to a front controller plugin. Ideally I'd have each action be more self-contained and process the output itself (perhaps using an annotation), but I didn't have any immediate luck with allowing for integrated captured rendering. As a result, some information will be stashed in the controller:


class MyTestController extends Zend_Controller_Action
{
  protected $_fillterErrors;

The action consists of a typical validation fork which process or re-displays depending on validity:


public function startAction()
{
//The script for this action will have the HTML form as first displayed
}

public function testAction()
{
 $form = new Application_Form_Test();
 if $(form->isValid($this->getRequest()->getParams())) {
    //Do some stuff
    $this->_redirect('/stuff_done');
 }
 else {
   //stash aforementioned info
   $this->_fillterErrors = $form->getMessages();
   $this->render('start');
 }
}

And now the postDispatch hook will call the filter if there's stashed data


public function postDispatch()
{
 //If no errors, get out of here
 if (! $errorStruct = $this->_fillterErrors) { return; }

 //Get the body, filter it, and give it back
 $pre = $this->getResponse()->getBody();
 $filter = new FormFillter();
 $post = $filter->parse($pre, $errorStruct, $this->getRequest()->getParams());
 $this->getResponse()->setBody($post);
}
}

And now the SAXy code

Some prep objects

XContext

I'm not really using this much, but this is a simple data object that roughly represents the dynamic context of the XML structure as it is being parsed. The present structure is stored a simple attribute of the filter object, and the ancestral contexts are stored in stack. Anything that is associated with a particular node can be jammed in here.


class XContext
{
public $elementName; //the name given by PHP...the tag type
public $id; //the id attribute
public $errorWriter; //what's gonna be shown for an error for this object

public function __construct($elementName) {
  $this->elementName = $elementName;
}
}

The errorWriter is an object which uses the Strategy pattern on a hook. At the moment I'm using a singleton default that writes nothing.


class FormFillter_EmptyErrorWriter
{
private static $singleton;
public static function getInstance()
{
  if (!isset(self::$singleton)) {
    self::$singleton = new $c;
  }
  return self::$singleton;
}
public function output() {}
}

And a simple error writer:


class FormFillter_FormatErrorWriter
{
public $messages;
public static function getInstance($messages)
{
  $c = __CLASS__;
  $new = new $c;
  $new->messages = $messages;
  return $new;
}
public function output() {
  foreach($this->messages as $error_ {
    printf("%s", $error);
  }
}
}

And finally the Fillter class itself:


class FormFillter
{
public $errStruct; //Error information
public $valueStruct;  //Values to refill
public $context;
public $ancestry = array(); //Context objects

A little ugly ATM, but the code that goes through the attributes. This is particularly significant since this handles the actual element id.


protected function attrFilter($attrs)
{
  $id = array_key_exists('id', $attrs) ? $attrs['id'] : "";
  $this->context->id = $id;

  'If this field is marked as having errors
  if (array_key_exists($id, $this->errStruct)) {
    //Add error class
    $value='error';
    if (array_key_exists('class', $attrs)) { $value .= " ".$attrs['class']; }
    attrs['class'] = $value;

    //Add writer for the errors
    $this->context->errorWriter
       = FormFillter_FormatErrorWriter::getInstance($this->errStruct[$id]);
  }

  //Refill values
  if (array_key_exists($id, $this->valueStruct)) {
    $attrs['value'] = $this->valueStruct[$id];
  }
  return $attrs;
}

And now the SAX callbacks


 protected function startElement($parser, $name, $attrs)
 {
   //This is part of a kludge to fake a single root element
   if ($name == "tmp_root") { return; }
 
   //Store previous context if exists (it should except for document node) and load present
   if ($this->context) { array_push($this->ancestry, $this->context); }
   $this->context = new XContext($name);
   $this->context->errorWriter = FormFillter_EmptyOutput::getInstance();

   $attrs = $this->attrFilter($attr, $name);

   //Output tag and processed attrs
   echo "<.$name;      foreach ($attrs as $key => $value) {
     $echo " $key=\"$value\"";
   }
   echo ">";
}

protected function endElement($parser, $name)
{
  if ($name == "temp_root") {return;}  //klduge again
  $this->context->errorWriter->output();  //show any error messages
  echo "";
  $this->context = array_pop($this->ancestry);
}

//Callback that just echoes
protected function passThrough($parser, $data) {
  echo $data;
}

And FINALLY the code that is actually called and puts everything together


 public function parse($string, $errStruct, $valueStruct)
 {
   ob_start();
   $this->errStruct = $errStruct;
   $this->valueString = $valueStruct;
   $string = "".$string.";  //Start of kludge, the particular way this is done smells
   $parser = xml_parse_create();
   xml_parser_set_option($parser, XML_OPTION_CASE_FOLDING, false); //This should really be the default for XML

   //Attach handlers
   xml_set_default_handler($parser, array($this, "passThrough"));
   xml_set_element_handler($parser, array($this, "startElement"), array($this, "endElement"));
   xml_set_character_data_handler($parser, array($this, "passThrough"));
   if (!xml_parse($parser, $string)) {
     throw new Exception(xml_error_string(xml_get_error_code($parser)));
   }
   return ob_get_clean();
 }
}

I'll throw a version up on some code repository in a little while after it's a bit more organized and has some tests written.

Issues with PHP's Zend Framework

2011-06-03T04:56:00.001-07:00

I've used the Zend Framework off and on for about two years now, while bouncing between other frameworks and other languages. Now that I'm returning to touch up a previous project, I'm left facing some strong opinions on ZF. I don't profess to be any type of expert, but since I have the personality that drew me to ZF, these concerns may apply to others that follow the same path.

After a quick Internet search it may appear that there are those that use ZF as a powerful solution, and on the other side of the fence are those that view it as too difficult. I'm hoping to touch upon some of my concerns which are a bit more beneath the surface and which don't seem to be mentioned elsewhere. ZF is certainly not targeted for beginners. If you need help configuring Apache for example, you're probably best off steering clear.

Support

"You know ZF isn't going anywhere and will be well supported because it's backed by Zend" (paraphrased often-touted benefit). In the time that I've used Zend I've seen very little notable progress and very little active support on the documentation. This includes bugs and lack of features on the main ZF site, both of which are ridiculously ironic for a site that is dedicated to a product that is supposed to help provide some of the same features without the bugs. This is theoretically an open source project, but there is very little in the way of promoting an open community around it. There is little in the way of comprehensive narrative documentation that is both insightful and up to date. Overall it seems as though the project has somewhat stalled since 2009.

Use at Will

One of the big selling points for Zend Framework is that it is a "use at will" loosely coupled framework. Unfortunately I presently don't find this to be the case. ZF offers a large collection of independent components, and that is an enormous strength. Zend's library is great, but as soon as you take that step into using the MVC framework, the concept of a loosely coupled architecture flies out the window. Describing the framework as "use at will" may lead one to think that the functionality of request dispatching, for example, may be able to be easily separated out. It is soon discovered that these integral function have scattered dependencies and a relatively complex architecture. The will to use something is inversely proportional to how complex or idiosyncratic it is.

Too Enterprise-y

The reasons for the coupling within the MVC components are that the framework is so flexible. I would argue that the component hierarchy is rich to a fault. This is PHP, not Java. PHP provides quite a bit of out of the box functionality and is intended to be able to allow you to quickly write what you need. There is therefore a point where a complex class hierarchy suddenly becomes as or more difficult to work with than scratch PHP (but with the added benefit of complexity and loss of code re-usability). This would be the flaw that those people that denounce frameworks would tear apart. This is also a result of the coupling covered above. If the core components were actually created so that they used clearly defined simple interfaces, then the creation of needed Plain Old PHP Objects that integrated with those components would be an attractive option.

Flexibility?

ZF advocates use of OO techniques for extending and flexing the framework (along with advocating OO for pretty much anything and everything (filters? using references in said filters? really?)...but that's a whole different rant). In my case that has led me in the past to write a decent amount of code to get ZF to behave the way that I want. Ultimately it left me feeling like I was fighting ZF which in turn I felt like was fighting PHP. For a framework that is positioned to be the professional solution, and considering it's beefy class structure, it could stand to offer some more declarative metaprogramming to allow developers to leverage the power of the framework rather than practically fork the base.

All the Mystery without the Magic

ZF has positioned itself as a glue framework, but due to the reasons outlined above that glue can seem to be stuck to everything. This leaves either accepting what is being dragged along, or investing in unraveling it. In either case there's a high potential for high investment, but there's less of the fun wizardly behavior of the "full stack" frameworks.

Conclusion

Ultimately what all of this amounts to is that I feel as though ZF has a bit of an identity crisis. It is a very useful and powerful component library, but it is also an entangled framework that can seem to ensnare even those well encapsulated components. I look forward to ZF 2, as it should be a fresh design with the experience of the growing pains, but I seriously question how long it will take and whether "use at will" will be carried through the entire library, or whether at some point it still switches to "bend to the will of the framework".

Functional vs Conceptual Demarcation in MVC (and family)

2011-05-19T08:50:00.000-07:00

First a disclaimer: how to use the MVC pattern properly is somewhat of a religious issue. The best single answer is to use it the way that makes the most sense for the parties involved. I'd also re-enforce that patterns are made to be adapted, and strict adherence to one definition or another is just as likely to mean that your solution isn't optimized for the problem as it is a sign of a robust architecture...in other words it doesn't mean much by itself.

With the wave of adoption of MVC based Web frameworks there follows a horde of devotees. Most of them seem to divide applications into one of the three areas, categorized by functionality. In some form or another the Controller interacts with the Model and the View renders the data. Depending on what a particular piece of code does, you can fit it into the proper area. This encourages all of the logic and persistence to be jammed into the Model (which ideally should be nicely structured but that's a whole other ball of wax...many frameworks equate the Model with an ORM or something similar), the Controller to be more or less a courier and dispatcher for Model and View, and the View to spit out what's left. Many systems advocate a lack of logic in the View. That all sounds great, if you keep it straight (and you know it's true because it rhymes).

Sticking to the above division, in my experience, leads to a lot of extra work. More relevantly, it leads to a lot of extra coupling, since by the time you get to the View everything else is supposed to be prepped and ready for action. This also leads to endless discussions about where certain responsibilities lie. If, on the other hand, the MVC pattern is viewed with conceptual divisions, there's far less of chance of having to think about where something should go (though there would still be room for discretion).

With conceptual demarcation, the Model represents the raw, independent part of your application. This is where all of the fun stuff goes. This should have its own structure which allows for a nice pure interface and representation of what your application is actually doing, which is then supported by ideally loosely coupled implementation code which provides things like persistence, security, and whatever else falls more into "how" than "what".

The Controller is the glue, and the code that is most bound to the implementation.

The View represents a particular representation of that data (or perhaps more properly a UI, since it also provides the means for input). This would include all of the supporting junk for that representation. The MVC pattern exists partly so that different Views can be created from the same application. This seems to often be overlooked in Web frameworks, where the View is left focused on Web formats (probably also due to the shakier association with input). So anything that is desired for display but that doesn't fit in to that core application belongs in the View area. Think about writing a bare minimal command line version of your application: anything that doesn't fall into what you would expect that version to display probably belongs in a View. The View may contain a composite of a couple different actions, but it will also often consist of code that exists for that View. In a Web application this may consist of supporting navigation structure or shared layouts. In a View that allowed the information to be represented on a MIDI instrument, it would include all of the mapping necessary to translate between the I/O of the program, and the MIDI voices and events. The fact that those two possible types of output are so different are part of the power of the MVC pattern. Thinking of how globally a function could be applied to disparate representations can serve as an easy guideline as to where it should be placed.

It should be mentioned that I am by no means advocating bloated templates (I do however strongly advocate fairly self-contained template systems which some frameworks seem to discourage). One of the other issues with many MVC frameworks is that the flow consists of a "Controller" which passes to a template "View". This leaves the initial impression that in the case of a complex view there's going to be quite a bit of code added to one or the other. Of course if you are one of the better "functional" MVC practitioners you may shove that complexity into the Model: even though it may have no use outside of the View, it may look to be acting like a Model. The fundamental issue however is that the perspective of the MVC pattern within these frameworks has become too limited, and even those frameworks that have adopted semantic variations that they consider to be more fitting still prescribe a basic flow and fitting things into that flow. The ultimate solution is the need for a developer to be able to use a framework for what it provides, but realizing when it may need to be supplemented by additional code; this also includes the admission that frameworks (like any tool) help with particular tasks but shouldn't be stretched into places where they don't reach (it will be there when you get back).

In closing, I'll reiterate that people should do what makes sense to them (and will still make sense to them in a year). I'll also say that I'm not a Patternologist, and I'm also not necessarily even advocating the use of MVC for Web applications. I do enjoy well structured systems, however, and think that MVC has lessons if nothing else but those lessons are best appreciated if MVC is viewed from a perspective larger than the ubiquitous Web frameworks waving its banner.

Pyramid on Google App Engine, Take 1 (for me)

2011-04-11T07:32:00.000-07:00

I'm in the process of streamlining project creation for Pyramid Apps running on Google App Engine. For me, the combination of the 2 technologies is a solution that I actually enjoy rather than suffer through. The same cannot be said about getting them to work together however.

The official appengine_monkey approach outlined on the Pyramid site worked for me after a little bit of caressing, and is the configuration presently used for most of the sites I have running at the moment. Unfortunately this configuration has the annoyance that the development server must be manually restarted after code changes, and more importantly the config.scan() method throws an error which stop use of the Venusian decorators. So this set-up is monkeying around with stuff that it shouldn't be.

The other standard approach appears to be using buildout recipes as outlined here: http://code.google.com/p/bfg-pages/wiki/PyramidTutorial. This initially didn't work for me at all, but I ended up spending some time bending it in to submission after I realized that the issues with the monkey set-up were, in fact, caused by that set-up and so trying to make the buildout option work would likely be easier than fixing the problems with the monkey set-up.

Here's my present process which I'll be iteratively automating, though I'm still undecided as to which tools I'll use (buildout/paste/something else).

- Follow the instructions at http://code.google.com/p/bfg-pages/wiki/PyramidTutorial...in my case that means the instructions in the comment after the article with the virtualenv in a separate directory (though I'm using "venv" rather than "python").

- Patch main.py with the 'os.mkdir = None' hack (or the slicker lambda one)

- Patch main.py with the block:


try:
    import pkg_resources
except ImportError:
    pass
else:
    if hasattr(os, '__loader__'):
       # This only seems to apply to the SDK
       pkg_resources.register_loader_type(type(os.__loader__), pkg_resources.DefaultProvider)

- Run the dev_appserver.py provided from Google not the one in the bin directory.

In addition to automating this set-up I'll also be refining it. I'm not crazy about the way the above set-up is structured and have normally modified it to make things feel more package-centric rather than dictated by environment. These concerns will also be addressed in future revisions.

The relational data model.

2011-02-14T11:44:00.000-08:00

When data is stored in computers it is broken down into a sequence of flat, serial bits. The efficient storage and retrieval of complex data structures accordingly requires assorted techniques and algorithms to convert all those 1's and 0's to something a little more meaningful. From the top-down perspective this is accompanied by techniques to dissect data so that it can be represented in ways that can then be consumed. There have been various data models created to approach this problem.

Through the 1980's the relational data model rose to be the standard for representing large amounts of data. Storing data as relations (tables) keeps the data structure fairly fixed which allows the computer to quickly calculate positions within the table, and combined with indexing allows for very quick operations on the data. The relational model maintained it's dominance through the rise in popularity of dynamic web-sites, and became second nature to many programmers to the point that it became THE way to represent data.

The relational model is very limited in that it also consists of flat structures (tables). Relationships between tables are the standard way to create references to other data and some semblance of structure, but this can quickly become complicated. The benefit of relations in their fixed structure is also a liability when the data obtained is significantly less than uniform. Continuing with the blessing being curses, the strict definition of data also means that the structure must be thoroughly specified which can easily increase maintenance at the least, and can complicate development when the data to be used (or any edge cases there in) is less than certain. This model has, again, become ingrained into developers though, so workarounds for all of these problems become instinctual.

Recently, however, alternate data models have once again become viable. These can provide an enormous benefit in that they may be significantly more analogous to your data than a relational data model. Generally it's a very nice thing when the model matches the reality without having to be jammed into shapes it doesn't like. The freedom of some of these models allows for more attention to be paid to properly handling application and domain logic, rather than worrying about how each change has to be handled in the underlying database. In addition many of them are more scalable and Internet friendly. The relational data model generally focuses on providing a singular database (or cluster) which has all of your data and the schema (and integrity constraints) for that data and therefore isn't conducive to distribution. Many, many cases therefore fit better into alternate data models and I don't think much beyond Maslow's Hammer would explain adoption of a relational (which is evidenced by most proponents of relational data models and their seeming blind faith).

There are plenty of cases where relational databases are a better fit. Some data fits quite nicely into relations. Relational Databases are also still the best choice for aggregation of data and similar bulk operations.

In general for an application the best solution normally draws on both, and therefore would best be addressed through the use of a hybrid data model. Persistence of objects is normally best handled by a more hierarchical data model such as (surprise) an object database, while things like statistics are best still done with a relational. Combining models allows for the best of both worlds.

This should be considered in the data persistence of an application. Deploying a hybrid data model would be overkill for most projects particularly during initial development. The important concern therefore becomes (as it should be anyway due to countless design mantras) not coupling your application model to a data model. From the top layer perspective your code should only worry about what it is supposed to do (interface) and not how it is done (implementation). In addition to standard abstraction, this means that all of those remnants of relational tricks should be discarded in favor of writing code that focuses on the problems at hand. Your application can then be ready to move on to multiple, alternate, possibly more fitting persistence mechanisms.

Fun with Microsoft Access and Rollovers

2010-12-17T19:42:00.000-08:00

Oh wait, not fun...the other one...abject misery.

Visual Basic and Access will always have a place in my heart. They were both early stepping stones on my way to other technologies. After having recently stumbled into a consulting job which involves Access and therefore by association VB(A), I quickly went from amusement to frustration. After reading up a bit on Access I started to grow fond of it...but the fondness quickly disappeared upon actually using it. Access fits in to a particular space where there's a decently high investment in learning the technology for an average user, and yet ridiculous constraints on a more powerful user. VBA similarly remains a very limited high level language which seems to still suffer from some of the quirks of BASIC. In other words, neither of them seem worth the trouble. After about 10 years of absence, they've quickly worn their welcome thin.

So...rollovers. Not something I care much about but this database already had them and they should be something that is simple enough to do in this day and age. The existing approach was a bewildering maze of events scattered around elements combined with what amounted to a continuous loop which constantly inspected their meta-data. In hindsight I should have left well enough alone after investing too much time, but at some point I crossed the threshold of taking it as a challenge.

Access doesn't allow for any type of reusable event handler, the type where you can create a function that has access to the caller. You're left with either individual procedures, or module code which is oblivious. This can be easily solved by passing an instance of the control manually to the function (Though annoyingly).

As far as rollovers go there's a mouse over (or Mouse Move), but no mouse out, or any other comparable callback mechanism. The advocated solution is to therefore have an event on whatever is outside the control (i.e. everything else). Already off to a good start. I originally planned on using slightly larger elements to catch the event as soon as the mouse left, but mouse move seems to not be handled reliably well. So attaching to the form it is.

Image elements are normally only rendered on initial form display, so modifying the image source would be tricky...multiple images it is.

I had a decent solution going but then got a little crazy and decided the best solution all things considered would be to create a stack of flags for each group used for a rollover. Anytime a rollover was left in a changed state, a flag would be pushed onto the form's stack. The form mouse move would then check and pop off of its stack if necessary. The form then does nearly nothing when everything is in the proper state, and the form doesn't need to worry about anything but the stack.

To make this even more enjoyable, I'm uncertain of the production environment, so I wanted to use vanilla Access rather than leveraging any external references. Most notably, using a dictionary/hash would have been fantastic. This is another feature which is missing from Access 2010 for reasons I can't fathom. After all of this there was a nice payoff in the fact that not only was the design less of a mess, but the CPU usage caused by moving the mouse around (which I admittedly didn't test beforehand) dropped from being actually significant (up to 80% or so on the aging development PC), down to well under 20% for even the most spastic motions. I can't say how proper the code is, since I'm not particularly interested in relearning more VBA than necessary to do things decently.

Here's the function which creates the stack hash. This should be a class. Collections also appear to not want to support assigning to existing keys. Can't seem to cut and paste through VNC, will have to post code later.

The Joys of Comcast

2010-11-29T07:51:00.000-08:00

Over the past year I've had nothing but bad experiences with Comcast broadband, shifting my preference on Internet providers from indifference to whatever option isn't Comcast. They have the usual mild aggravations which I begrudgingly expect from any large company. The rundown of issues that I've dealt with are:
* a botched (and less importantly messy) installation
* an impartial installation followed by an internal service call of late, clueless techs who wasted 5 hours of my time before disappearing without doing anything
* lack of IMAP
* a gateway device which needs to be constantly rebooted, can't be configured, but they seemingly won't replace

These have already amounted to many wasted hours in many different locations, with no instances of any positive moments. Last night I was finally personally affected when their DNS service went down in this area. It took me a little bit to jury rig a fix since I didn't have another DNS server address readily available so I had to retrieve an IP for remote access to a machine that could still resolve Internet addresses.

This was a DNS issue. DNS is a fairly simple service with no dependencies to note for most deployments. That translates to something that should be able to be fixed within no more than 30 minutes. A company like Comcast which provides the service to all of it's customers should have at least an interim fix available which they could deploy within 10 minutes (if it doesn't automatically happen nearly instantly). In attempting to solve the issue I logged in to a device 30 miles away in what should be treated as a different administrative area when considering population...and the devices were using the same DNS servers. This leads me to believe that all of the systems affected (ranging throughout New England) were all hitting the same 2 servers. Performance-wise that's not a big deal (again DNS is simple and normally cached all over the place), but redundancy wise that's just asking for...well...an outage to affect a large area and to potentially make recovery more difficult. BIND (the reigning heavyweight DNS champ) provides for tons of redundancy to minimize downtime, which was apparently not used.

My dealings with Comcast leave me with the impression that they are one of the lopsided companies with too much attention on making money off of their services and not enough on actually providing the services. Ultimately and cynically I suppose the difference in providers is determined by accountability. Verizon is involved with the Internet (having the telecom association with the physical connections and configuration), while Comcast is just a reseller to consumers. To Verizon it's a part of their livelihood, to Comcast it's a product and an expense. Thanks to lack of competition in the cable industry, they only have to care enough to not be glaringly deficient. Chalk one up for advocates of big business in the name of free enterprise.

I'm Gone Daddy (to GoDaddy)

2010-10-21T06:49:00.000-07:00

I woke up this morning to no new email. Highly unusual, especially since there are scheduled emails that I should receive every day. Oh...that's right, my network is all switched around at the moment because I'm configuring some routers to be deployed elsewhere, maybe that was the problem. Assuming my inability to contact my hosts is related to a lookup or routing issue I spend several minutes checking everything before realizing that it's only my hosts (the hosts on my domain) that I'm having issues with.

The settings at my registrar (Webhero/Catalog.com) were apparently broken...everything appeared to be working normally but disabled. The support was alright, after the tech spent some time first making sure that I was blaming the right person, and then figuring out what exactly was wrong. It turns out that my zone files were purged. Awesome. Since I run most of my stuff off the cloud or a VPS I don't have anything hosted with my registrar. Apparently my hosting with Catalog.com just expired and although the domain registration is still active, they decided to clean up my account to the point where it was totally useless, ruins my morning, and causes some some potentially costly loss while the domain was inactive (here's hoping my clients slept in today).

I've kept several domains with Catalog since I've been using them for years now. I was toying with the idea of moving the domains before this renewal, but ultimately decided to not rock the boat. In a business where a lot is riding on a domain, and in a time where a domain is expected to be working, casually purging zone files is a fairly egregious offense. To make matters worse, Catalog.com is slow to update their files from the Web interface (which then still has to propagate), so my recovery is still in process.

I started using GoDaddy for a couple domains out of convenience earlier this year. I'd never particularly been drawn to them since I normally attach the evils of bloat to larger more popular companies in certain industries (worse support, less offered power). I've had nothing but good experiences with them though (with surprisingly good support), including one time when I was letting a domain expire and received a phone call before the domain was deactivated.

Part of this likely comes down to my loyalty to the relative underdog, but in this case the underdog is definitely showing that the odds are justified in certain respects. My particular wants in this case revolve entirely around the very narrow issue of domain registration, and should not be extended outside of that aspect. It is a very crucial service that is fundamental to doing business on the web and is again very narrow, so in order to provide the service competitively after this long, there shouldn't be visibly unmanaged spaces.

Pretty URLs vs Query strings

2010-10-05T09:18:00.000-07:00

For the past several years there's been multiple schools of thought on whether and when a web application should favor pretty URL's (such as /item/245) or query strings (such as /item?id=245). The latter is the older style and has somewhat fallen out of favor and left people running to pretty URL's. I'm all for keeping URL's clean, but part of that is keeping them free from all the extraneous query arguments that you may want to add to your application. Query strings are also more self documenting, so the argument is similar to fixed place vs named function parameters in programming. So, I'm personally of the opinion that they both have their place and while reading up a little bit on Django, a clear separation occurred to me.

URL paths should identify resources (that's what the R is), query strings should identify queries on resources. That's stating the obvious and not the recent revelation. An easy way to differentiate what qualifies as a resource is implied by RESTful interfaces. A resource should be something that in some way maps not only to GETs, but to PUTs (or POSTs). In the django book (djangobook.com) the example is given of adding hours to a datetime by using a URL pattern of |time/plus/\d+/|. This is a modified version of a resource which doesn't have it's own identity, most likely not something that you would PUT, and therefore a query which should be handled by a query string. With, the example in the introduction using book ids, each id is a resource and therefore should be part of the URL path.

There may be some cases which are somewhat unclear, but the general rule would be of URL paths handling responses which contain unmodified, unfiltered resources that would be sanely subject to modification, while query strings offer the filtering and modification. Queries are therefore reports generated from the underlying base data (represented by the URL paths).

Dividing resources into components is subject to the same rules...if the displayed component is something which can be logically represented by itself and subject to updating, then it can be given a URL path. As an example a collection may be divided into categories where each category effectively becomes a collection and therefore is it's own resource. You may want to add items to that category. On the other hand filtering collections by arbitrary properties such as dates is a query. You would not add an item to that filtered collection, it would be added to the main collection and then filtered according to its properties.

Individual resources and their components would generally map to URL paths. In this case the qualification would again come down to how logical it would be to use the representation for mutation, and how it differs from the canonical data representation. As a simple example I'd suggest that a book's author could suitably be mapped to '/book/${book_id}/author', but if you wanted to show book reviews with a spoiler omitted it should map to '/book/${book_id}/review/{$review_id}?spoiler_visible=false'

This may be an issue subject to a great deal of opinion and taste. I personally try to view URLs as powerful tools which can be used in the identification and CRUD of distributed data, and therefore look for ways to ensure their clear definition. For these purposes, I think the above guidelines are useful.

Design Patterns in different languages

2010-10-01T06:01:00.000-07:00

I'm a big fan of design patterns. Particularly since I've normally programmed sporadically, my discovery of design patterns gave me the insight to keep my code organized in ways that I wasn't confused and/or disgusted upon rereading the code a bit later. One of the consistent cautions when reading about design patterns is to not over-apply them...to not introduce needless complexity. There also appears to be an overlooked problem with design patterns in that most references say something along the lines of "although this example is implemented in language0, this pattern can easily be adapted to other languages", but there is no indication about whether this pattern should be used in other languages.

The Gang of Four book has become the seminal work on the topic and it is the pattern format there that is most often copied. Considering most subsequent works seem more language-centric, the GoF also seems regarded to be more universal. The book is dated, however, and predates the practical, popular use of dynamic scripting languages for complex applications (i.e. the modern Web). All of the patterns are therefore written with the focus on compiled languages and the related necessary trappings such as static typing. The rise of dynamic languages seems to have followed in the wake of the explosion of Java, and therefore patterns mostly retained their compiled tendencies with the perspective of Java.

In more flexible languages the patterns can be implemented, and often are implemented, but they don't need to be. They introduce needless complexity rather than using some of the strengths of the language...the strengths that the speed of compilation is being traded for. I propose then that design pattern catalogs should contain additional information describing what limitations of a language a pattern is made to overcome, and at the opposite end, what language features would obviate the need for the pattern.

It's not because I'm an early riser

2010-09-23T12:14:00.000-07:00

I just pulled my first all-nighter working at a computer for the first time in a long time. They used to be a part of my work schedule, back in the glorious days of NT4 when it seemed as though you had to do a fresh install to adjust the system clock, and so all night marathons to minimize workday downtime became a ritual. Lately I've rarely made it to 3am though.

This time the culprit was moving data after data recovery. Generally I give myself ample time to do data recovery, but I've apparently gotten comfortable enough with it that I have a firm plan ahead of time...but do it infrequently enough to forget how long some things take. In particular, I seem to have forgotten how long it takes to move large amounts of data around. The recovery (nothing too exotic yet, blind usage of ddrescue (not to be confused with dd_rescue) took about as long as I expected and had allocated (left it running for about 36 hours before the error size began to hold steady).

The error in my ways was that I normally only move large amounts of data once or twice during a tech project like this, and normally that's something I set and walk away from/sleep on. In this project I ended up shifting a lot of data around and it quickly caught up with me. I had the recovery image target drive, and the replacement drive intended for the customer. This was a favor fix, so these drives were some that I had sitting around, so the first bit was cleaning off the drives...and then it turns out I needed a larger replacement drive so everything was shuffled around again...after this the replacement Windows install was broken (which was initially done concurrently with data shuffling)....reinstall....copy the files over from the target recovery drive (only room for mounting one drive in the system).

This wouldn't have been such a disaster if I hadn't kept underestimating how long this would take, leading to the system's owner continually expecting it to be done when it wasn't and ultimately pushing to make a deadline. All in all some lessons learned though:
* I need to organize my dormant data better (already knew this one but this drove it home a bit)
* Don't use compression when copying and time is an issue (probably obvious, but after my normal background long-term copying I seem to have lost touch with how much of an impact this has)
* Image copying is markedly the fastest (this isn't surprising either, but the difference actually was. Copying an image of a half full drive (with all that empty space) took far less time than actually accessing the file system and copying the files themselves. This would certainly be dependent on the file system and the file structure itself, but image copying will be the new first stop for me when it's an option)
* Don't screw around with Windows installations any more than necessary (they're fickle and not worth dealing with. It should be a simple matter of files but bottom line is it isn't, and even though I've gotten good at fixing the little boot issues that may appear, it's not worth the risk of time investment if that doesn't work out).

Overall the project was a slight disaster, but forces me to reassess the best practices for fast turn around coupled with long term solutions (i.e. what to do now and what to schedule for later).

As a side note on data recovery...on another drive the freezer trick seems to have worked for me. My slight variation was using a vacuum sealed freezer bag.

database not found error when installing ADempiere with PostgreSQL

2010-09-20T08:31:00.001-07:00

ADempiere is one of the premier open source ERP solutions, and out of the contenders appears to be the one that best seizes the spirit of open source software. This carries with it many of the benefits of the bazaar, but also at least one of the serious drawbacks: the product is for the most part not packaged into a polished deliverable suitable for end users. As such certain tasks require a little bit more work on behalf of whoever the person is in charge of the software.

One of these tasks is installation itself, which rather than an integrated wizard consists of manually installing the assorted support aspects and piecing them together (aside from the integrated JBoss application server). This is now the second time I've installed it and the second time I've run into an error when constructing the database.

Error messages scroll across the screen, the initial one of which is along the lines of "database could not be found" even though the database certainly seems to be anxiously awaiting contact. The first time I ran into this I went through standard troubleshooting and took care of the issue fairly quickly: quickly enough that when I ran into the problem again I didn't remember what it was. There is an ADempiere book (mildly recommended) which mentions the error and a link to a script to work around it, but doesn't explain what the cause is and is therefore of limited use if the script isn't available or is out of date.

So now that I have this blog going and have fixed the issue again I'll post for posterity. The issue is caused by template database encoding within PostgreSQL. For whatever reason, postgresql is still often installed with ASCII rather than UTF-8 encoding as a default, and when creating a new database using UTF-8 an option must be passed to use the blank template0 rather than copying the ASCII template1. This causes an issue when ADempiere tries to create the DB, because it recreates without that option even if the DB has already been created with the appropriate settings. A more long term solution may be to edit the shell script that creates the database, but in my case I'd rather have PostgreSQL using UTF-8 so I just created a new template1 which uses that encoding.

Dynamic DNS on OpenWRT with web based IP retrieval

2010-09-07T14:33:00.000-07:00

Not long ago I was implementing Dynamic DNS updating for an OpenWRT kamikaze installation as outlined at http://wiki.openwrt.org/oldwiki/ddnshowto. This is the new and improved version which uses shell scripts rather than the abandoned ez-ipupdate. It didn't seem too improved to me as it didn't work, but being shell scripts it was easy enough to hack it into submission.

The problem is in the /usr/lib/ddns/dynamic_dns_updater.sh script which is the file that actually performs the update (the shell function file listed in the how-to is a simple wrapper which calls this script with the config section(s) as an argument). At the end is a loop which calls to sed to substitute patterns in an update URI with the variables supplied in the configuration. Unfortunately when using an update_url, there isn't any escaping so sed complains and everything falls to pieces. The first time I encountered this my quick hack was just to specify the particular values I needed substituted rather than looping through all available with "for option_var in $ALL_OPTION_VARIABLES". This worked (I didn't need the URL or anything with any values beyond alphanumeric) and saved some me from having to make sure data passed cleanly through the busybox sed and bash. It is not the best solution, so I wanted to return to it later, if for no other reason than to brush up on my neglected bash and sed skills (or lack thereof).

Not being overly familiar with sed and not having enough interest beyond getting it to work a proper way, I'm supposing that the primary issue is the "/"'s in the URL's conflicting with sed's use of / in the regex. sed, like most regex processing utilities, can use most anything as a delimiter so swapping that out is the first stop (ideally with something which won't be used in any of the values). To ensure this passes through bash and sed decently strong string quoting should be used, the sed recommended approach appears to be single quotes for literals which can then be directly concatenated with interspersed double quoted shell expansions. When working with the script, I also had to deal with the fact that the endless loop wants to run forever, no matter what you tell it, so a signal trap was added to make it more obedient. Here's the final loop with the couple minor changes indicated. The OpenWRT wiki doesn't seem editable quite yet otherwise I would post this there. I suppose I'll post a note in the forums so that this may work it's way into the package (I won't have time to play with opkg or anything for a bit longer).

while [ true ]
do
trap exit INT TERM

current_ip=$(get_current_ip)

current_time=$(date +%s)
time_since_update=$(($current_time - $last_update))

verbose_echo "Running IP check..."
verbose_echo "current system ip = $current_ip"
verbose_echo "registered domain ip = $registered_ip"

if [ "$current_ip" != "$registered_ip" ] || [ $force_interval_seconds -lt $time_since_update ]
then
verbose_echo "update necessary, performing update ..."

#do replacement
final_url=$update_url
for option_var in $ALL_OPTION_VARIABLES
do
replace_name=$(echo "\[$option_var\]" | tr 'a-z' 'A-Z')
replace_value=$(eval echo "\$option_var")
final_url=$(echo $final_url | sed 's|'"$replace_name"'|'"$replace_value"'|g' )
done
final_url=$(echo $final_url | sed s/"\[IP\]"/"$current_ip"/g )

verbose_echo "updating with url=\"$final_url\""

#here we actually connect, and perform the update
update_output=$( $retrieve_prog "$final_url" )

verbose_echo "Update Output:"
verbose_echo "$update_output"
verbose_echo ""

#save the time of the update
current_time=$(date +%s)
last_update=$current_time
time_since_update='0'
registered_ip=$current_ip

human_time=$(date)
verbose_echo "update complete, time is: $human_time"

echo "$last_update" > "/var/run/dynamic_dns/$service_id.update"
else
human_time=$(date)
human_time_since_update=$(( $time_since_update / ( 60 * 60 ) ))
verbose_echo "update unnecessary"
verbose_echo "time since last update = $human_time_since_update hours"
verbose_echo "the time is now $human_time"
fi

#sleep for 10 minutes, then re-check ip && time since last update
sleep $check_interval_seconds
done

Mailman, Postifx, & Virtual Mailboxes

2010-08-16T07:15:00.000-07:00

I recently adopted GNU Mailman to manage a mailing list, and to serve as a starting point for a more complete service solution that I'll be developing over time. Mailman is a respected piece of software which should handle all of the things I initially need it for, but it has the common problem of poor documentation. In this particular case it has plenty of what are basically How-To's which explain what to do, but not why it's being done. How everything is fitting together begins to resolve itself as it's being implemented, but a couple quick explanations would save estimations during set-up. I'm not trying to be one of the horde of open-source users whose complaints are their biggest contribution to their communities, and am planning on putting at least an architectural overview together when I start to dissect the software (right now I'm just quickly trying to get it to work).

I found myself a little lost in particular when it came to the integration of Mailman with the MTA (in my case Postfix). The uncertainty of why this integration was needed and how much it applied to my set-up led to wasted time during configuration. Ultimately (as it becomes obvious in hindsight), this is all done for the sake of handling the various addresses which mailman uses for managing the lists: mailman creates aliases for the users, postfix reads the files from mailman so that mail is delivered.

Virtual domains are covered in the documentation, but I was using virtual mailboxes. Since I wasn't sure how the integration was being done, I wasn't sure if that made any difference in mailman's generation of its aliases (particularly considering the emphasis on postfix configuration of the domain in the mailman docs). After breaking my configuration by implementing virtual domains in addition to virtual mailboxes (hoping Postfix would sort it all out), it seems that the interface between postfix and mailman is as simple as it should be and that postfix simply needs to include the new aliases and mailman generates everything more or less in isolation.

Transparent proxied Google Web Address Mapping

2010-07-17T10:20:00.000-07:00

At the moment I've switched my Web site (www.mattwhipple.com) over to a Google site. Unfortunately I also have the older version of the site running a secure version. I personally haven't spent much time with it, but as far as I know the limitations of the present TLS/SSL protocol don't play well with Google Sites. So as a quick little hack I'm going to attempt to use the existing Web server to proxy through to the Google site and try to keep that transparent. As a disclaimer, the better solution all things considered would likely be to use a separate host name for the secured version of the site which could then be proxied or otherwise handled, but I'm sure there are cases where this would be helpful. One of the other things that I'm presently using this for is to continue to serve some resources from the old server (for instance the /images directory is still a directory on that server). This allows me to keep Cool URI's until I get around to moving my site to a Web App (once I put a design together).

First some background:

There sensibly doesn't appear to be much ability to play with this stuff on Google side so that has to be handled elsewhere (in this case my VPS)
Google Sites generates absolute URL's for internal navigation, so the site itself needs to think it's using the right domain name (most likely through the host in the HTTP requests).
Because of all the abstraction in clouds, CNAME is the way to go with DNS.

I was worried that Google actually verified the DNS settings after creating the mapping, but so far this doesn't appear to be the case. So my game plan is to configure a DNS daemon on my VPS (bind) to allow me to override how that particular host resolves on that server, and then configure the Web server (nginx) to reverse proxy to that host which will send it off to Google and hopefully be accepted and create the appropriate URL's on the site.

The goal for the Web traffic is:

Look up the host name in DNS which is resolved to the front-end server (VPS)
Be accepted by the configured VPS Web server which is then told to proxy to the same host name
That host name is resolved on the VPS to be the Google site and then passed to that
The request is made to the Google site with all the appropriate headers so all those detours are nicely hidden

I also am going to try 2 slight variations. One in which the CNAME corresponds to the normal Google DNS resolver which would normally be placed into the authoritative DNS, and a second using a second hostname which is additionally mapped to the site (i.e. web.mattwhipple.com)

So...the first variation...WORKS!. I'm actually blogging as I'm doing it so I'm relieved I'm documenting something that does what it's supposed to rather than saving people wasted time at best.

To reiterate (top-level view, more specific instructions will vary depending on the software used):

DNS left alone (i.e. pointing to the front-end Web server)
DNS resolver on front-end server overrides canonical DNS (in this case bind installed locally, 127.0.0.1 added to /etc/resolv.conf and a zone for that domain created with the www host changed to CNAME pointing to ghs.google.com. .
Front end web-server proxied to the same address it received the request on (in nginx amounts to server_name www.mattwhipple.com;...location / { proxy_pass http://www.mattwhipple.com; }

One thing to keep strongly in mind is that any request that the server resolves for that host will end up being sent off to Google. Combined with the way that TLS works with host names this means that basically any absolute request to a secure site as outlined above needs to be initiated form outside of the server (which most would be anyway).

Ok...so I'm going to scrap the second variation. As I was thinking of the changes to make it occurred to me that mapping to another host in the same domain in this case would require some form of DNS inheritance so that only particular records are changed. I was happy enough to get bind working simply without having to spend a couple minutes refreshing, so I'm not inspired at the moment to spend time on a solution that I don't need and can't think of any reasons it would be necessary in place of the one above.

Welcome!

2010-07-15T15:06:00.000-07:00

I'm finally breaking down and doing a blog on an actual blog service rather than one squirreled away somewhere (often away from prying eyes). This is all part of my big migration to using cloud computing, something which I've been toying with for a while and am now fully embracing. That will likely often be a topic of blog posts. Presently I'm using Google's offerings which will likely be complemented by some Erlang floating around EC2 or something similar. Right now I'm breaking down and using Django so I already have wisdom to share/vent. Rants and chaos will be here and then cleaned up stuff will appear on my Web site

As an aside...I've enabled the advertising so I can rake in the pennies and happened to notice that the top initial ad was for Sunbelt Software's VIPRE. This is software that I have deployed in enterprise through home environments and one that I strongly recommend (to the point where some people near me get sick of hearing it). I personally don't use it since I'm rarely on Windows, but it's the first anti-virus program that I consistently consider worth paying for in a long time, and it's highly affordable to boot. It is a newer AV program and seems to have benefited immensely from a fresh start with modern ideas, as opposed to most of the bigger names who seem to be working with built up versions of the same engines that they used on Windows 95. It seemingly doesn't slow the system down (seriously) and does a good job of keeping a wide variety of threats out. It also is presently good at removing viruses that other AV programs can't, though that may just be because it's the new kid on the block so a lot of baddies don't know to hide from it yet. My one time dealing with their support was also very smooth.