SSIS Remove redundancy Publisher's description
from SQL Lion
SSIS package to detect and remove only duplicate data (rows) in your SQL Server database. All redundant entries are eliminated except the real one.
Here comes one more challenging yet interesting topic to tide over. The requirement goes something like this: You have some sources, let it be some sales data or some Call center data coming from different sources. Data can be of different media. But you are able to load it in your staging tables i.e. one staging database you are maintaining like StageDB for storing these incremental data.
So, here we have both master i.e. Dimensional data and Detail i.e. Fact data in our stage database. But before processing further and loading it into our Mart or Data warehouse, we need to check if there is any redundancy at the row level for each of these tables in the staging database. Here comes the main problem, what if you don't know how many tables are there in the staging database and how many columns are there for each table in the staging database but still you have to keep only unique records for each table.
This means there must be some mechanism to deal with this problem dynamically, i.e.
It has to detect the number of tables for the StageDB database.
It has to detect all the columns for each table in the StageDB database.
It has to detect only the redundant records for each table in the StageDB database.
What's New in This Release:-- Detect the number of tables for the given database.
-- Detect all the columns for each table in the given database.
-- Detect only the redundant records for each table in the given database.
System Requirements:SQL Server 2005 or SQL Server 2008
Program Release Status: New Release
Program Install Support: No Install support