Rhino ETL: First Code Drop
First, let me make it clear, it is not ready yet.
What we have:
- 99% complete on the syntax
- Overall architecture should be stable
- The engine works - but I think of it as a spike, it is likely to change significantly.
What remains to be done:
- Parallelising the work inside a pipeline
- Better error messages
- More logging
- More tests
- Transforms over sets of rows
Here are a few works about how it works. The DSL is compromised of connection, source, destination and transform, which has one to one mapping with the respective Connection, DataSource, DataDestination and Transform class. In some cases, we just fill the data in (Connection), in some cases we pass a generator (think of it as a delegate) to the instance that we create (DataSource, DataDestination), and sometimes we subclass the class to add the new behavior (transform).
A pipeline is a central concept, and is compromised of a set of pipeline associations, which connect the input/output of components.
Places to start looking at:
- EtlContextBuilder - Compile the DSL and spits out an instance of:
- EtlConfigurationContext - the result of the DSL, which can be run using:
- ExecutionPackage - the result of building the EtlConfigurationContext, this one manages the running of all the pipelines.
There is an extensive set of tests (mostly for the syntax), and a couple of integration tests. As I said, anything that happens as a result of a call to ExecutionPackage.Execute() is suspect and will likely change. I may have been somewhat delegate happy in the execution, it is anonymous delegate that calls anonymous delegate, etc, which is probably too complex for what we need here.
I am putting the source out for review, while it can probably handle most simple things, it very bare bone and subject to change.
You can get it here: https://rhino-tools.svn.sourceforge.net/svnroot/rhino-tools/trunk/Rhino-ETL
But it needs references from the root, so it would be easiest to just do:
svn checkout https://rhino-tools.svn.sourceforge.net/svnroot/rhino-tools/trunk/Rhino.ETL