Wednesday, March 10, 2010

Single Insertion for all values in ETL script

0 comments
It is common practice to get single value from the transactional database and then add to it immediately in the database and then updating rest of the values but it is not efficient because in huge data it is very difficult to update i.e. it is time consuming therefore it is suggested to first select all the values i.e. KPIs from the transactional database using transformation in the Pentaho and store these KPIs values in variable using SET VARIABLE as mentioned below

select your kpis query values using input

SET VARIABLE

Field Name Variable name Scope default value
from input My_Variable valid in root job 0(in case of int)

in the set variable you can store each value you get from the query using INPUT instead of updating get all these values store in the variable using GET VARIABLE in below mentioned way

GET VARIABLE

Name of Variable Name of same variable set as in SET VARIABLE
name to variable as in database ${variable name as defined in SET VARIABLE}

pick a INSERT from OUTPUT get all the variables in the last grid defined in the GET variable and get only those values which want to look up in the database and on the basis of that you want to insert values in the destination............................. in this is way it not required for your ETL to every time get value and update it in the warehouse on the basis of some criteria instead it is better to have all the values ready to insert in one go using single insertion weather it is 1000 columns or a single column think how complected it is to fill single row through 999 updating including a single insertion.......





0 comments: