Stumbled across this post on “micro-etl” at Tom Gleeson’s blog – It misses my points about the value of a framework and structured approach to Tactical Data Integration (Edge Data Integration) but is interesting.
Describes the type of Excel or hand-coded integrations often seen in reconciliations and the like.
“Edge” data integration is needed by organizations of all sizes. In many cases an enterprise class tool (typically a centralized monolithic deployment) is the best alternative if available but for many solutions it is not. Valid reasons for choosing “smaller tools”:
None of the reasons justify skipping , version control, quality assurance and change management which are as critical to tactical “Edge” data integration efforts as they are to behemoth IT projects.
Coming soon … a methodology for tactical/edge data integrations!
Please do not interpret this as a call to load a datamart from an unsupported PC under your desk or prepare payroll advice on your laptop – There are cases were traditional efforts with an IT supported tool are best!
Ever been in a business that did not have some critical analysis that they thought could only come from some long-time employee secretively using Excel, some stored procs, some custom scripts and/or a ton of manual editing (ctl-C,ctl-V)?
I have not either. Chances are only one person (or a few folks) know how this is done each week and the CxO holds his breath when they’re on vacation and the process is not understood or documented it may even be done on an old desktop that has some magic configuration.
This is traditional strategic data integration – This is the worst of Edge Data Integration … often the sources are not an enterprise systems such as the ERP, the ODS or the big _______ (insert billing, manufacturing execution, transaction processing …) system.
In companies where robust integration environments and deep skill sets exist, the IT shop might see sees the “perfect” application for (or justification or) a new EAI framework, monolithic ETL tool or ESB and repeatedly proposes a six figure and multi-month project.
Without enrolling half the IT folks or worse, an army of junior consultants, there are strategies, design patterns, methodologies and even tools that are very appropriate for this type of Edge Data Integration. They can be built with the visibility/maintainability of logic and exception handling that the enterprise tools promise.
It’ll be interesting to watch Wikipedia … someone suggested that the topic “Edge Data Integration” be merged with Core Data Integration – The only pressure I can think of to merge these would be from tool vendors with a commercial interest as they are VERY different things.
Stay tuned and check out www.bus-informatics.com…
