I am not suggesting to get away from ETL tool. The point I am driving at, is to bring staging area nearer to DW area. That would give us an option to move data using stored procedures as well if the need arises. So this in my opinion is driven by needs and the infrastructure available. Let’s assume, we have to laod huge amount of data. Now I have these options to laod data:

a. Using ETL tool (which may execute a job from another server) and do the transformation and loading.
b. Along with using ETL tool, in some cases I may prefer to move data from Staging database to DW database using say stored procedure.
c. Use mix of both

So what needs to be done is being done. Only the way it is implemented is different. If there are any updates (loading/updating) required in DW database, they would be done by ETL tool or some SP or some SQL.

Also, I agree, we have really powerful ETL tools available these days. And in my opinion, partitioning, bulk loading etc features of ETL tools are available in databases already. So we have to try various options and test the results. Based on that we can make a decision how to implement the solution.

Thanks,
Jagdish

This was a good article. You have pointed most of the common problems that we face in Datawarehouse load.

However I am a little confused on the use of Stored Procedures. Most of the time we have to implement some difficult logic which is little easier to do using an ETL tool compared to writing procedures (maybe that is the primary purpose of ETL…to make a developer’s life a little easy …. ). Also if we write a stored procedure we may, most of the time, end up using cursors which again does a row by row processing rather than bulk processing. This when we compare to a datawarehouse load (where the number of records is too large), using a cursor can be a bottleneck as it needs a large memory. Don’t you think this problem is better handled by the ETL tool which has its own dedicated memory and some other advance concepts like partitioning and parallelism?

One of the examples we were given was, sorting a large number of records would be better done through a powerful ETL tool than doing it on the database. This is significant as sorting plays an important role in our development.

Another problem we faced was, as the datawarehouse and data marts grows up, even querying a database takes up a long time. In that case if we use ELTL method as you described, won’t it be a little difficult to update (transform) the records in the database after loading them when selecting those records itself takes ages? This problem won’t be that significant if we use ETL approach as we are just loading the database not updating it (and as you said bulk loading feature of ETL is very good at this).

Today we have some very fast databases which are specifically designed for high volume data (like Teradata, Kognitio). This may prove your point too.

As you pointed out, this is a good topic to debate on.

Regards,
Praveen

Thanks for the detailed comments (although little bit blunt).

Let me explain what I meant to say in this post:

<<<<<<<<< where you are endorsing an approach of stored procedure over an use of ETL tool/components, in my experience would eventually lead to ridiculous performance problems, extremely high maintenance cost over the years and the solution would essentially have the same limitations as are applicable to a stored procedure, which are enormous.
>>>>>>>>>>>

I am in no way endorsing the SP approach here. The point I wanted to bring up here is
1. bring in the raw data as near as possible to the data warehouse database, so that we have a choice of using SP if required.
2. we shall be able to get away from row by row transformations if required. For instance, slowly changing dimension type 2. If that creates performance problems, we shall have an option to use Merge SQL instead. Even Ralph Kimbal suggests using it if there are lot of performance problems. You can find his article here:

http://www.kimballgroup.com/html/08dt/KU107_UsingSQL_MERGESlowlyChangingDimension.pdf

At the same time, I agree with your view that SPs have limitations. This the the reason, I called this approach as pattern. It is not a guideline that we shall follow.

<<<<<<<<<<<<>>>>>>>>>>>>>

You are tlking about ETL solutions design vs SP Programming. This is not the essence of this post. ETL tools allows for parallel execution which can not be done with SPs. From a maintenance and cost perspective, I would prefer to use Merge SQL statement for SCD type 2 (if I can use it) over ETL tool approach. I would not require ETL specialists working closely with DBAs and other third party vendors. Also consider the scenario, where a compnay started with say SSIS and later they decide to move the ETL implementation to Business Objects.

Moreover, the reasons (pretty blunt) that you have mentioned above are in the real word very real. So we need to be practical and we shall also not be monolithic in our thinking.

Finally there were my thoughts which I posted. I leave it to the people to think about it when they are implementing their ETL solutions. I am in no way proposing rule of thumb as you have thought about it.

Please email me if you have any further comments at jagdishm@adit.com

Thanks,
Jagdish

Sorry for replying late. Was running busy with work.

Coming to your point, The purpose of staging area is to keep the incoming data at some designated place so that while transforming/validating/etc, we do not need persistent connection to source. The point to remember is that this data is still comsidered raw and not fully massaged. As you said, we do not store more than a days (or some specific period) unless required for some reasons.Thus, it does not matter where is your staging area, it is not usable from OLAP perspective. So you need transformation anyway.

For your second point, I mean staging area should be as close as possible to DW. In that case, we have flexibility to implement transformations in SQL. Let’s consider a scenario: we are using say SSIS, where we need to do SCD type 2 for say 1million incoming records. Even if your job server and DB server are on the same box, it will be slower. You would resort to lookup transformations and some so called advance stuff in SSIS to tune it.
With SQL, you would have more control and best performance.
Hope this helps. Thanks

Jagdish Malani

This sounds like a good idea however what will be the impact on the design of the data marts as they need to accommodate all the leaf level and or staging type of tables. Data marts which are usually supplied with only clean data will contain the raw data as well since the data is directly loaded from source. Shall we still need a DW or DMs alone will suffice? Have you come across any other ETL desgin pattern than this?

Once again, appreciate your idea and would like to hear more from you.

Regards,
Ashish