Since chatting with Peter Bell at 360 Flex, I’ve been doing a lot of thinking about the base model for my Flex-and-ColdFusion Intranet application, “Stream”.

This application was pseudo reverse engineered from a Visual Basic application and has nearly 200 tables of inter-related data - mostly without primary or foreign keys, but that’s another story entirely. For a long while there has been no clear and overall model of the application. The tables and views were mostly responsible for driving the application but there is still a lot of business logic tied up in various peoples’ heads. This is obviously not an ideal situation, as it dramatically raises the bar of entry on this project for any other developers.

As a direct result of the planned migration from Flex 1.5, to new-and-improved Flex 2.0.1 (including the usage of Flex Data Services), I now have to rework the ColdFusion tier. This seems like a good time to rethink how the application works, apply some consistency to the application as well as build up a global model to develop from. Hopefully there will also be a page or two of documentation.

When working on “Version 1″ of Stream I spearheaded the generation of a lot of the code on for the ColdFusion and Flex tiers, which included various parts of code for usage with Cairngorm in Flex as well as unit tests for ColdFusion. However this code was fairly brittle and an increasing number of complicated edge cases meant that eventually the generation was one-time only. The result of this was that this was barely better than no generation at all.

So for the last few weeks, I’ve been playing around with ideas for the new model and how to generate code for the three effective tiers - ActionScript, ColdFusion and database (accessed via an ORM). This is resulting in an bespoke generator application that I’ve named Dynamo. But code generation is only a part of the purpose of Dynamo. The source for my model is the database, so obviously I’ll need to validate my model against this - checking usage of fields and automatically using the database as the basis for any typing information; and as the end result of the application is to display these fields in a Flex UI, I need to validate that I am using all the fields required for each part of the model. This combination of techniques should allow rapid development based on our existing data while dramatically minimise the number of errors caused by missing fields, typos or incorrect data types.

For the generation part of Dynamo, after much thought and reviewing of options I’ve fallen back on an old favourite technique - using XML for the model and XSLT for the generated files. Based on ideas from Peter, I was originally intended to put all of this model meta data into the database itself. However after some initial attempts at putting a hierarchical data model into a database, I realised that I was overly complicating things and that the “simplest thing that could possibly work” was to work with an XML file for the time being - if required later I can always rework my model code to use a database. One drawback with using XSLT is that many people find it complicated and difficult to work with; at some point I’ll replace this with a simpler templating method.

However, one big advantage of using XML and XSLT is that I can incorporate this into my ant build script so that all of the generated code is created when “Stream 2″ is compiled and deployed, and the generated files are not checked into source control. This keeps the focus of change entirely on the model and the templates, not on the generated files themselves.

Ideally I’d have a dynamic and active generator - change my model, next request all the changes are reflected. The problem is that I need my VOs for ActionScript to be generated to be compiled into my application, and ColdFusion requires physical files to automatically translate classes to ActionScript. As an aside, I was talking with one of my friends who works in the finance industry and when I told him I was using XML and XSLT he embraced my like a brother exalting the virtues of XLST programming. But when I told him I was generating my code, he became disgusted and seemed to want to pray for my eternal programming soul

These are still early days, but I’ve switched back from focusing on Dynamo to Stream -which of course was the whole point of this whole exercise. I’ve now started wiring up a number of screens, and apart from the occasional typo or missing field (which is why I need to validate), the development is proceeding rather quickly.

[Addendum: a few weeks later]

The base idea behind this modelling has worked really well - I was able to wire up a lot of screens with data, in rapid succession, with only a small amount of code required - the rest was generated. I’d like to componentise more on the Flex side, but so far the ColdFusion side is working well. However, I’m yet to need to customise the ColdFusion code - and this is where generated code tends to fall over. Dynamo at the moment is sliding away from being a separate application, and may end up being tied into the main code-base of Stream. One unfortunate and unforeseen side effect of this generation system is that I now need to generate my code for the application to work - otherwise there is no base ColdFusion components to handle data accesses. This means that my build times in CFEclipse are in the realm of 20 seconds - hardly ideal.

Well, its been a very hard month or so of work, but last week I finally managed to get the first alpha release of my Intranet project released. It’s still quite a bit rough around the edges, but it works as more than just a proof of concept.

And as far as all the cool buzzwords go, I’ve got Flex Data Services, Remoting, XML and ColdFusion on the server side; a SWC for the UI customisation; a specialised data component to automatically connect to FDS objects, and on edit validate and then auto submit the form; and of course a one click build from source to deployment. Not too bad, even if I do say so myself.

There are still some hurdles to work out,  and there’s a lot of data integrity problems to be fixing, but its finally all coming together properly.

Tomorrow I might even hit “publish” on some of my outstanding blog posts

I’ve been back from San Jose for a few days now, but I’ve been too caught up with resuming my London-life to post a review.

360Flex was awesome. Without a doubt the best conference I’ve been to. Good choice of venue in eBay, the conference itself was well organised, even in the more basic sessions I learnt something, there was a lot of tasty breakfasts and lunches and of course the requisite range of intelligent speakers. The area around eBay’s campus was devoid of pretty much anything, but it was made up for with the suitably warm and sunny weather of California.

• Most animated speaker: Jesse Warden demonstrating Flash and Flex Together
• Coolest demonstration: Mapping Components by Mansour Raad, demonstrating ESRI’s about-to-be-released mapping components for Flex 2.
• Most annoying clash: Building Apollo Apps vs Mapping Components vs Model Driven Integration
• Most illuminating speaker: Ely Greenfield presenting Making Components
• Best meal: Tuesday nights BBQ Chicken
• Most hungover: Ryan Stewart

And, of course, I put lots of faces to names and met a lot of new friends.

Very much looking forward to the next 360Flex, Boston or otherwise.

Introduction

This is a reworking of my session at BarCampLondon2. I started about 5 minutes late, which means I ended up zooming through the following links at a rate of about one per minute.

Historical

Charles Minard - Napolean’s Russian Campaign of 1812
This graphic, first published in 1861, contrasts the size of the French army on the attack and retreat parts of the campaign, and also compares the temperatures faced by the troops on the retreat.

Florence Nightingale’s Causes of Mortality in the Crimean War
This chart was used by Florence Nightingale to show the government the true causes of death during the Crimean war. The graph is not the clearest, but the blue area represents deaths through preventable disease, red is death through wounds and black is all other causes. Its fairly obvious what the main cause of deaths were.

Common

Weighted List
The weighted list is a fairly common technique. used on del.icio.us, flickr, and the example used here is from 43Things. The more popular an item is the larger it’s size is - this provides a quick and easy way to determine popularity.

O’Reilly Treemaps
Tim O’Reilly uses Treemaps to analyse the ups and downs of the computer book industry. Treemaps display hierachical data using areas of comparable size, thus highlighting disproportionate areas and in this example enables easy comparison of shifts in the market place.

Newsmap
This interactive Flash application compares news articles sourced via Google News. Using brightness to indicate freshness, and colour to indicate grouping, the actual relative popularity of news stories can be seen.

Sparklines
Not all data visualisations need to be complicated. Sparklines are a streamlined view of one set of changing data. The example here compares the performance of the stock market, other examples include things such as comparing winning and losing streaks for a sporting team.

Elections

US Election Results - Red vs Blue States
After the recent US election, maps appeared of ‘Blue’ vs ‘Red’ states - this site looks into greater detail the ways of representing this data displaying the influence of population, and the middle ground of purple, as opposed to pure Red and Blue.

Californian Vote Distribution
This site applies some statistical normalisation to results of the Californian Gubernatorial elections and then reworks the map of California to draw insights from the original data.

UK Election Shift
This interactive Java applet shows the shifting of seats that occurred during the recent UK elections. Seats that have become more marginal shift away from the edges and gains and losses are represented by seats migrating into a different section entirely.

Travel

Travel Time Maps in the UK
In these sets of UK maps, coloured contour are are applied to represent the time taken to travel around the UK. Both train and car are contrasted, showing a slightly asymmetric map of travel times. Journeys within Cambridge and London are also displayed, highlighting comparatively unconnected areas of both cities.

Travel Time Maps in Europe
These maps use map distortion to compare rail travel times in around Europe in 1993 and 2020.

World Data

Worldmapper
Worldmapper applies changes to the size and shape of countries based on various data sets to produce distorted maps. Population, economic wealth, air passenger numbers are all vary easily compared across the world. They have produced nearly 300 maps so far, all of which are available in PDF poster format.

World Processor
World processor doesn’t constraint itself to size and shape - it applies images on top of the existing globe of the Earth, changes the topography or removes countries entirely. Not all the examples work well, but those that do highlight the usefulness of this technique.

Others

Data Fountain
The data fountain shows that not all visualisations need to be software. This installation contrasts the relative worth of the Yen, the Euro and US dollar with the height of their associated fountains.

Suicides on Golden Gate Bridge
This infographic shows the reported jumping point for people committing suicides based upon witness and surveillance footage. People are more likely to jump facing San Francisco, than the Pacific Ocean, and light pole #69 is by far the most popular.

Quantum Level Particles
This page demonstrates some conceptual items for representing quantum particles, and their combinations and interactions. While it isn’t ideal, it is an interesting and different way at looking these things.

Circular Calendar
This poster size calendar is designed to be drawn upon. Events and journeys can be non-contiguously marked, and collisions in time can easily become highlighted.

Family Wheel
An interesting look at the family tree, this technique transforms branches into slices of a circle. It is, however, unlikely to be clearer when common ancestors are found.

Lightweight Data Visualisation in Excel
Not all data visualisations need to be complex. This technique for Excel creates easily comparable data using only conditional formatting and a simple command to repeat characters.

Multi-Touch Research
While not quite a visualisation method, this technology - coming soon in the new iPhone - looks to provide a real world interactive with data on screen. The video provided by this research group at NYU shows some quite amazing ways of working with information on the screen.

Resources

Edward Tufte

Tufte is a legend in the field of data visualisation and his books are well worth a read for anyone interested in this topic. He also runs courses on this subject, but unfortunately they seem to be US only.

Periodic Table of Visualisation Methods

While this list is inherently flawed, due to its shoehorning of the Periodic table into comparing visualisation methods, it has some use however, in detailing a large number of techniques, both simple and advanced.

Blogs

information aesthetics and visualcomplexity

I didn’t get a chance to mention these in my talk, but I felt it worthwhile to include them here. Information Aesthetics is a site focused entirely on data visualisation and is a fantastic resource on the subject while VisualComplexity is focused more narrowed onto the domain of complex networks with over 400 examples of different representations.

Conclusion
I think this is a really interesting field of work, getting into the heart of what computers should be used for - making ideas and information clearer to the end user and providing insights which were not previously perceivable.

Feel free to comment below with any additional examples you have found, and without a doubt the visualisation category within my del.icio.us links will continue to grow.

This weekend I shall mostly be doing this: