Today I've announced the release of DataCleaner 1.5.2, yay! I'm pretty excited about this release as I think this is probably the biggest of the minor releases to date. And especially I hope to see that our new "single jar file" distribution-option will attract new users. Go read the announcement for more details now :)
20090628
Introducing AnalyzerBeans
It's been some time now since I first designed the core API's of the DataCleaner project and as time goes on, some of my initial assumptions about the design of profilers, validation rules and so on have shown to be less-than-optimal in regards to flexibility and scalability for the application. This is why yesterday I decided to do a major change in the roadmap for the project:
- The idea about the "webmonitor" application (DataCleaner 2.0) have been cancelled for now. If anyone wants to realize this idea it's still something that I am very much interested in, but as you will see I have found that other priorities are more important.
- A new project have been founded - for now as a "sandbox" project: AnalyzerBeans. AnalyzerBeans is a rethought architecture for datastore profiling, validation etc. - in one word: "Analysis". When this project is stable and mature we will probably be ready for something I like to think of as a new DataCleaner 2.0.
The solution is a new annotation based component-model. Each profiler, validation rule etc. will not have to implement certain interfaces because we can now mix and match annotations to the specific type of analysis-component - each "AnalyzerBean". There are a lot more interesting features available when we introduce an annotation-based model, but let me first give you a simple example of how a regular row-processing DataCleaner-style profile would look like:
@AnalyzerBean(name="Row counter", execution=ExecutionType.ROW_PROCESSING)Now this is not so impressive. I've just replaced the IProfile interface of DataCleaner's API's with some annotations. But notice how I've gotten rid of the ProfileDescriptor class which was used to hold metadata about the profiler. Instead the annotations represent the class metadata. This is actually excactly what annotations are for :-) Also notice that I've gotten a type-safe configuration-property using the @Configured annotation. This means that I don't have to parse a string, ask for a Table of the corresponding name etc. And the UI will become a LOT more easy to develop because of type-safe facilities like this.
public class MySerialCounter {
@Configured("Table to count")
private Table table;
private long count = 0l;
@Run
public void run(Row row, long count) {
this.count += count;
}
}
But an even more exciting way to use the new API is when creating a whole new type of profiler, an exploring AnalyzerBean:
@AnalyzerBean(name="Row counter", execution=ExecutionType.EXPLORING)Now this is something totally new: A component that can gain total control of the DataContext and create it's own query based on some @Configured parameters. I imagine that this programming model will give us complete flexibility to do exiting new things that was impossible in the DataCleaner-framework: Join testing, non-serial Value Distribution etc.
public class MySerialCounter {
@Configured("Table to count")
private Table table;
private Number count;
@Run
public void run(DataContext dc) {
DataSet ds = dc.executeQuery(new Query().selectCount().from(table));
ds.next();
this.count = (Number) row.getValue(0);
ds.close();
}
}
There are a few other annotations available to the AnalyzerBean-developers but I will take a look at them in a more in-depth blog-entry later. For now - let me know if you like the ideas and if you have any comments. Anyone who would like to help out in the development of the AnalyzerBeans project should visit our wiki page on the subject.
Update (2010-09-12)
A lot has happened to AnalyzerBeans since this blog entry. Here's a list of blog entries (in chronological order) that will help interested readers dive deeper into the development of AnalyzerBeans:
20090613
Performance benchmark: DataCleaner thrives on lower column counts
Today I've conducted an experiment. After fixing a bug related to CSV-file-reading in DataCleaner, I was wondering how performance was impacted by different kinds of CSV file compositions. The reason that I suspected that this could impact performance is that CSV files with many columns will require a somewhat larger chunk of memory in order to keep a single row in memory compared to CSV files with fewer columns. In the older versions of DataCleaner we discovered that using 200 or more columns would actually make the application run out of memory! Fortunately, this bug is fixed, but there is still a significant performance penalty, as this blog post will hopefully show.
I auto-generated three files for the benchmark: "huge.csv" with 2.000 columns and 16.000 rows, "long.csv" with 250 columns and 128.000 rows and "slim.csv" with only 10 columns and a roaring 3.200.000 rows. Together, each file has 32.000.000 cells to be profiled. I set up a profiler job with the profiles Standard measures and String analysis on all columns.
Here are the (surprising?) results:
| filename | rows | columns | start time | end time | total time |
| huge.csv | 16000 | 2000 | 18:54:48 | 19:40:28 | 45:40 |
| long.csv | 128000 | 250 | 19:44:53 | 19:52:31 | 7:38 |
| slim.csv | 3200000 | 10 | 19:53:46 | 19:55:03 | 1:17 |
So the bottom line is: Lowering the number of columns has a very significant, positive impact on performance. Having a lot of columns means that you will need to hold a lot more data in memory and needless to say you will have to replace this large chunk of memory a lot of times during the execution of a large profiler job. Going all the way from 45 minutes to 1½ is quite an improvement - so don't pre-join tables or anything like that before you run them through your profiler.
20090605
eobjects.org @ JavaOne
I am currently hanging out at the lovely JavaOne conference in San Francisco, checking out cool new Java technology and meeting interesting people. Of course I'm here as a representative from my employer I also do some blogging (in Danish).
Yesterday I saw an interesting session about JFreeChart and surviving as an Open Source professional. Dave Gilbert told us about how he has managed to live from his hobby as a JFreeChart developer, about cool new features of the excellent charting API and about the struggles of making money on Open Source. Very fascinating stuff and I hope that everybody in the chart-consuming business will give it a try. I was quite happily surprised to see the new interactive chart functionality that has been put into the API - I'm wondering how that hadn't found my attention before now! It gave rise to a couple of ideas (or rather: Sparked my motivation) for me to try and implement charting in DataCleaner.
20090429
Seam, EJB's and EAR-packaging in Maven
Lately I've been designing a new course on the splendid JBoss Seam (2.1) Web and Java EE framework. One thing that strikes me, being an enterprise Java developer, is that almost no good examples of setting up Seam using Maven exist. I realize that Seam's appeal comes from a tradition of wanting things to be nice and easy, so the seam-gen tool has absolute merits for creating your project but to me the downside of this approach is that you're bound to use Ant as a build tool and for a lot of reasons I won't go into here, I strongly prefer using Maven.
The examples of using Maven to build Seam applications that I have been able to track down have been limited in a variety of ways:
- Most of them required to use a specific Maven archetype which altered the default project layout.
- Almost all of them where out of date and was bound to a very "custom" maven repository which I don't think is suitable for enterprise application infrastructure.
- Those that didn't fall within the former two points where restricted to web-applications (WAR-files) only and thus didn't support using EJB's in the Seam applications.
Project structure
OK - to meet the demand of packaging an EAR file I've created a Maven project consisting of three modules. It's possible that this part can be optimized a bit since I use a seperate module for doing all the packaging (which to my understanding is the way to do it). The modules are:
- ejbs - containing the EJB's and possibly other "backend"-stuff like JPA-classes and so on.
- web - containing the web application
- packaging - for packaging the EAR file containg the two other modules (and Seam)
One of the big issues that I faced was figuring out how to configure Maven's dependency management framework to generate a correct set of EAR, JAR and WAR files. If this is done wrong it won't work and you'll waste hours (or at least I did) trying to figure out what is wrong.
All provided dependencies go into the parent project. These are all the dependencies provided by your container (plus Seam itself which will be packaged within the EAR file). You will need to add JBoss's Maven repository to resolve these dependencies. Here are the important parts of my parent project pom:
<repositories>In the ejbs module you will not need any dependencies!
<repository>
<id>repository.jboss.org</id>
<name>JBoss Repository</name>
<url>http://repository.jboss.org/maven2</url>
</repository>
</repositories>
<modules>
<module>ejbs</module>
<module>web</module>
<module>packaging</module>
</modules>
<dependencies>
<dependency>
<groupId>org.jboss.seam</groupid>
<artifactId>jboss-seam</artifactid>
<version>2.1.1.GA</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.faces</groupid>
<artifactId>jsf-api</artifactid>
<version>1.2_02</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.hibernate</groupid>
<artifactId>hibernate-entitymanager</artifactid>
<version>3.4.0.GA</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.hibernate</groupid>
<artifactId>hibernate-validator</artifactid>
<version>3.1.0.GA</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.servlet</groupid>
<artifactId>servlet-api</artifactid>
<version>2.5</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.servlet.jsp</groupid>
<artifactId>jsp-api</artifactid>
<version>2.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.ejb</groupid>
<artifactId>ejb-api</artifactid>
<version>3.0</version>
</dependency>
</dependencies>
In the web module you will need to add seam-ui (excluding the "core" seam, because it is provided within the EAR file), facelets and other web-dependencies such as RichFaces, seam-pdf or whatever. Here are the dependencies of my web module pom:
<dependencies>If you want to write Seam components that pertain only to the web module, then you will probably also need to include the ejbs module as a dependency in the web module pom. But remember to set the scope as provided, because it's all within the same EAR.
<dependency>
<groupId>com.sun.facelets</groupid>
<artifactId>jsf-facelets</artifactid>
<version>1.1.11</version>
</dependency>
<dependency>
<groupId>org.jboss.seam</groupid>
<artifactId>jboss-seam-ui</artifactid>
<version>2.1.1.GA</version>
<exclusions>
<exclusion>
<groupId>org.jboss.seam</groupid>
<artifactId>jboss-seam</artifactid>
</exclusion>
</exclusions>
</dependency>
</dependencies>
Last but not least you will need to write the packaging module pom. The Seam Documentation have been used to guide how this pom is structured: The ejbs module needs to be a registered EJB module. The web module needs to be a registered WAR module. And Seam itself needs to be a registered EJB module as well! JBoss EL needs to be placed in the /lib directory of the EAR and you need to make sure to exclude the EL API dependency from several artifacts - otherwise you'll get weird classpath issues when deploying. The important parts of the packaging module pom looks like this (pay attention to the parts in bold):
<build>So that's all the Maven configuration. Now to some "gotchas" in regards to Seam configuration. A lot of these things are being "disguised" by seam-gen so consider this a list of things you might have forgotten if you're building the project from the bottom up:
<plugins>
<plugin>
<artifactId>maven-ear-plugin</artifactid>
<configuration>
<modules>
<jarmodule>
<groupId>org.jboss.el</groupid>
<artifactId>jboss-el</artifactid>
<includeinapplicationxml>false</includeinapplicationxml>
<bundledir>lib</bundledir>
</jarmodule>
<webmodule>
<!-- add web module groupId and artifactId here -->
<contextroot>/seam-ejb-ex</contextroot>
</webmodule>
</modules>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<!-- add ejbs module groupId, artifactId and version here -->
<type>ejb</type>
</dependency>
<dependency>
<!-- add web module groupId, artifactId and version here -->
<type>war</type>
</dependency>
<dependency>
<groupId>org.jboss.seam</groupid>
<artifactId>jboss-seam</artifactid>
<version>2.1.1.GA</version>
<type>ejb</type>
<exclusions>
<exclusion>
<groupId>javax.el</groupid>
<artifactId>el-api</artifactid>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.jboss.el</groupid>
<artifactId>jboss-el</artifactid>
<version>1.0_02.CR2</version>
<type>jar</type>
<exclusions>
<exclusion>
<groupId>javax.el</groupid>
<artifactId>el-api</artifactid>
</exclusion>
</exclusions>
</dependency>
</dependencies>
- Remember to put empty seam.properties files in to the resource folders of both the ejbs and web modules.
- Remember to register the Seam interceptors (as described in the Seam Documentation) in the ejb-jar.xml file, located in the resources/META-INF folder ejbs module.
- The components.xml file should be located in the web module's WEB-INF folder. But you can't just copy the file from a seam-gen project because seam-gen dynamically replaces some "magic strings" in it to see to container compliancy. Instead you will have to add this static entry to the components.xml file yourself:
<core:init pattern="seam-ejb-ex/#{ejbName}/local"></core:init>
Note the seam-ejb-ex part of that static jndi pattern string. You will have to replace this part of the string with the context root entry that was highlighted in the packaging module pom!
I hope this blog entry have cleared up some of the confusion that I met when I tried building Maven-based Seam/EJB projects. Please let me know if it works out and remember that you can download the full example and use it as a reference as you like.
20090420
DataCleaner 1.5.1 released
I'm happy to announce the release of DataCleaner version 1.5.1. This release is a minor release, nevertheless containing a few nice features - especially for the users who are enjoying the exporting features that was introduced in 1.5:
- An additional HTML export format have been added to the built-in export formats (usable when exporting Profiler results in the desktop app and when executing the runjob command-line tool).
- The export format is now choosable directly in the desktop app.
- Four new measures where added to the String Analysis profile: avg. chars and max/min/avg white spaces.
The new version of DataCleaner is (as always) downloadable for free on the downloads page and feedback from users is also greatly appreciated, ie:
- Fill out our online user survey, or
- Post your comments and questions at our discussion forum.
We hope that you all enjoy DataCleaner 1.5.1.
20090211
Data quality pro interview
Dylan Jones over at data quality pro is working on a feature about DataCleaner and I'm very thankful for his work already (and pretty excited to see the final result). The feature have just been started with an interview with a very important person ... Me! :-) For all of those who take an interest in DataCleaner, the visions of the product and it's story I hope that you will head over there and read the article.
To be continued with more posts on the data quality pro articles.