Comments on kasper's source: Pre-processing in DataCleaner 2: Why?

For every kind of business it needs to be updated ...

2011-08-19T13:59:33.664+02:00

For every kind of business it needs to be updated by using new software solutions. Some applications could be developed by outsource company that provides custom software for business companies.

But honestly - any tool can do a SQL query. You ca...

2011-03-17T14:30:04.347+01:00

But honestly - any tool can do a SQL query. You can invoke it using any command line tool that follows with your DB.

And it's not entirely true that SQL is always faster than Java. If you're only looking for a single measure, then yes, it typically is. But if you want to retrieve a lot of measures from the same set of data, then it is typically faster to query once and share the result between multiple measure generators.

Furthermore, SQL doesn't work on anything but relational databases... I find that doing a DQA most often requires incorporating a lot of data from different stores.

Monitoring is something completely different and whatever measures found there should be handled as temporal data stored for trends analysis etc. I don't think a profiling tool will be the obvious choice for that.

rhaa, I lost my previous comment. So here is a sho...

2011-03-17T11:58:25.034+01:00

rhaa, I lost my previous comment. So here is a shorter one:

I just wanted to say that SQL queries are faster than a java program in general. And that they are useful when you need to monitor your data quality on a regular basis (Do I have some nulls data today?).

On the other hand, complex analyses will require the data to be extracted from the datastore. In this case, SQL queries cannot be used.

@Garima, give a try at Talend Open Profiler which provides both types of analysis.

Currently there is not a SQL console or something ...

2011-03-02T09:09:01.249+01:00

Currently there is not a SQL console or something similar in DataCleaner. This was a quite conscious choice, because we want to provide something that will work the same for any database, or rather - datastore! A datastore might simply be a CSV file, or an Excel spreadsheet, so making a SQL console would be something only available for some of the datastore types.

The other thing about doing SQL is that I don't think it's very good for exploration. In DataCleaner there are a lot of filters and transformation steps that you can add, and you can choose to preview and even subset your data ("Write to datastore") after you've performed these steps. In my oppinion that's a better way to do exploration.

I'm not against making something for user-written queries, but it's just not on the top of my radar screen. If someone wants to work on such a feature I think it's great and I will do my best to help them out.

Regarding your question about associated columns, the answer is that you need to include these columns in some kind of analysis. It could be a simple "String analyzer" or something like that. The reason is that the rows that are saved for inspection by the user are based on the rows that are available while DataCleaner is executing. At the same time, DataCleaner will only query for the data it needs. So this means if you need more data (columns) to be available, then you should make sure these data are included in the analysis.

Does DalaCleaner support SQL queries? How can I is...

2011-03-02T08:21:51.344+01:00

Does DalaCleaner support SQL queries? How can I issue SQL queries to explore data? Should it not be a part of a data analysis/profiling tool to allow to explore data?

I can generate various profiles to explore data, but in the output, I can see only the column on which profiling was done. What should I do to see other associated column values too for that row?