@TransformerBean("Groovy transformer (simple)")
public class GroovySimpleTransformer implements Transformer {
@Configured
InputColumn[] inputs;
@Configured
String code;
private GroovyObject _groovyObject;
public OutputColumns getOutputColumns() {
return new OutputColumns("Groovy output");
}
public String[] transform(InputRow inputRow) {
if (_groovyObject == null) {
_groovyObject = compileCode();
}
final Map map = new LinkedHashMap();
for (InputColumn input : inputs) {
map.put(input.getName(), inputRow.getValue(input));
}
final Object[] args = new Object[] { map };
String result = (String) _groovyObject.invokeMethod("transform", args);
logger.debug("Transformation result: {}", result);
return new String[] { result };
}
private GroovyObject compileCode() {
// omitted
}
public class GroovySimpleTransformerTest extends TestCase {
public void testScenario() throws Exception {
GroovySimpleTransformer transformer = new GroovySimpleTransformer();
InputColumn col1 = new MockInputColumn("foo");
InputColumn col2 = new MockInputColumn("bar");
transformer.inputs = new InputColumn[] { col1, col2 };
transformer.code =
"class Transformer {\n" +
" String transform(map){println(map); return \"hello \" + map.get(\"foo\")}\n" +
"}";
String[] result = transformer.transform(new MockInputRow().put(col1, "Kasper").put(col2, "S"));
assertEquals(1, result.length);
assertEquals("hello Kasper", result[0]);
}
}
| What? | How? |
|---|---|
| My transformer had only a default icon | Icons can be defined by providing PNG icon (32x32 pixels) with the same name as the transformer class, in the JAR file. In my case the transformer class was GroovySimpleTransformer.java, so I made an icon available at GroovySimpleTransformer.png. |
| The 'Code' text field was a single line field and did not look like a code editing field. | Since the API is designed for Convention-over-Configuration, putting a plain String property as the groovy code was maybe a bit naive. There are two strategies to pursue if you have properties which need special rendering on the UI: Provide more metadata about the property (Quite easy), or build your own renderer for it (most flexible, but also more complex). In this case I was able to simply provide more metadata, using the @StringProperty annotation:
@Configured @StringProperty(multiline = true, mimeType = "text/groovy") String codeThe default DataCleaner string property widget will then provide a multi-line text field with syntax coloring for the specific mime-type:  |
| The compilation of the Groovy class was done when the first record hits the transformer, but ideally we would want to do it before the batch even begins. | This point is actually quite important, also to avoid race-conditions in concurrent code and other nasty scenarios. Additionally it will help DataCleaner validation the configuration before actually kicking off a the batch job. The trick is to add a method with the @Initialize annotation. If you have multiple items you need to initialize, you can even add more. In our case, it was quite simple: @Initialize
public void init() {
_groovyObject = compileCode();
}
|
| The transformer was placed in the root of the Transformation menu. | This was fixed by applying the following annotation on the class, moving it into the Scripting category:
@Categorized(ScriptingCategory.class) |
| The transformer had no description text while hovering over it. | The description was added in a similar fashion, with a class-level annotation:
@Description("Perform a data transformation with the use of the Groovy language.")
|
| After execution it would be good to clean up resources used by Groovy. | Similarly to the @Initialize annotation, I can also create one or more descruction methods, annotated with @Close. In the case of the Groovy transformer, there are some classloader-related items that can be cleared after execution this way. |
| In a more advanced edition of the same transformer, I wanted to support multiple output records. | DataCleaner does support transformers that yield multiple (or even zero) output records. To archieve this, you can inject an OutputRowCollector instance into the Transformer:
public class MyTransformer implements Transformer<...> {
@Inject
@Provided
OutputRowCollector collector;
public void transform(InputRow row) {
// output two records, each with two new values
collector.putValues("foo", "bar");
collector.putValues("hello", "world");
}
}
Side-note - Users of Hadoop might recognize the OutputRowCollector as similar to mappers in Map-Reduce. Transformers, like mappers, are in deed quite capable of executing in parallel. |
{
"orderNumber": 123,
"priceSum": 500,
"orderLines": [
{ "productId": "abc", "productName": "Foo", "price": 200 },
{ "productId": "def", "productName": "Bar", "price": 300 },
],
"customer": { "id": "xyz", "name": "John Doe", "country": "USA" }
}
Everytime we publish some work that I've been working on I feel a big sigh of relief. This time it's not software, but a video - and I think the end result is quite well although we are not movie making professionals at Human Inference :)
Here's the stuff: Data Quality monitoring with DataCleaner 3
Enjoy! And do let us know what you think on our discussion forum or our linkedin group.
Monitor the evolution of your data:
Share your data quality analysis with everyone:
Continuously monitor and improve your data's quality:
Connect DataCleaner to your infrastructure using web services:
DataContext dc = DataContextFactory.createExcelDataContext(new File("people.xlsx"));
// getting column by path
Column customerNameColumn = dc.getColumnByQualifiedLabel("customers.name");
// traversing all schemas, tables, columns Schema schema = dc.getDefaultSchema(); Table[] tables = schema.getTables(); Column[] columns = tables[0].getColumns();
// step-wise getting specific elements based on names
Table customersTable = schema.getTableByName("customers");
Column customerBalanceColumn = customerTable.getColumnByName("balance");
// Simple query: Get all customer fields for customers with a credit balance greater than 10000. DataSet ds = dc.query().from(customersTable).select(customersTable.getColumns()) .where(customerBalanceColumn).greaterThan(10000).execute();
// Slightly more advanced query: Join customers with their associated sales representatives
// and group the result to count which sales reps have the most customers
Column salesRepId = customersTable.getColumnByName("sales_rep_id");
Column employeeId = dc.getColumnByQualifiedLabel("employees.id");
DataSet ds = dc.query()
.from(customersTable).innerJoin("employees").on(salesRepId, employeeId)
.selectCount().and("employees.name")
.groupBy("employees.name").execute();
// Batch #1: Create a table and insert a few records
dc.executeUpdate(new UpdateScript() {
public void run(UpdateCallback cb) {
Table muppets = cb.createTable(schema, "muppets")
.withColumn("name").ofType(VARCHAR)
.withColumn("profession").ofType(VARCHAR).execute();
cb.insertInto(muppets).value("name","Kermit the frog")
.value("profession","TV host").execute();
cb.insertInto(muppets).value("name","Miss Piggy")
.value("profession","Diva").execute();
}
});
// Batch #2: Update and delete a record
dc.executeUpdate(new UpdateScript() {
public void run(UpdateCallback cb) {
cb.update("muppets").value("profession","Theatre host")
.where("name").equals("Kermit the frog").execute();
cb.deleteFrom("muppets")
.where("name").like("%Piggy").execute();
}
});
// Batch #3: Drop the table
dc.executeUpdate(new UpdateScript() {
public void run(UpdateCallback cb) {
cb.dropTable("muppets").execute();
}
});