SSIS Error - Invalid Number ORA-01722

After 5 years,  a select statement on a Oracle 10 database starts to fail...with the error message ORA-01722: Invalid Number.

Hum, strange error message and not really usefull.

I read lots of post and sometimes it was due to an issue with the connector (http://www.attunity.com/forums/microsoft-ssis-oracle-connector/known-limitation-number-fields-1305.html), due  to datatype mismatch (http://social.msdn.microsoft.com/Forums/en/sqlintegrationservices/thread/9acb1c36-bfaf-40eb-bb6e-3a91eeea96ec) or due to other reasons.

In my case, the issue cames from a TO_NUMBER casting that we did to convert a string datatype to a number...I know it'sstupid to create keys as string but I'm not responsible of that ;-)

So let's continue the explanation...first I started to execute the query by removing all the selected fields.
The query ran fine.
Than I added, one by one the selected fields till the error occured again. Like this I was able to identify the column that caused the issue.

Second I remarked that I did a conversion using TO_NUMBER...May it be possible that some values in the source database are not numbers?

I executed again the query without converting the values and was able then to see that one row had the value 'PHONECALL' as key number! Very funny. So I have identified the bug.

Now go to the solution...

1) The workaround: adding a selection criteria in the where clause that avoid selecting rows where the values length is smaller than the expected length.
2) The solution: Asking to the DBA to create a function that detect if a value is a number or not like here follow:

CREATE OR REPLACE FUNCTION is_number
( p_str IN VARCHAR2 )  
RETURN VARCHAR2 IS l_num NUMBER;
BEGIN  
l_num := to_number( p_str );  
RETURN 'Y';
EXCEPTION      
RETURN 'N';
END is_number;

SSIS vs. SQL Stored Procedure

Recently I was asking by the customer to redesign a sequence of stored procedures that were used to load and transform data. This because the performance was not as expected due to the increasing of the mount of data.

Here the observations I will share with you:

• Use SSIS to extract and load data from one or mutliple sources to one destination or multiple destinations.
• Use a stored procedure to update the data instead of using e.g. a SCD component.
• Use SQL Statement for grouping, filtering or sorting a query result instead of e.g. a Sort component
• If the data to join comes from different sources then extract and load first the data in tables of the same e.g Staging area and then use a SQL Select statement where the join conditions will be indexed.

A right indexation and statistics will be the key of success!

SSIS - Kimball SCD Component vs. SCD Component

Some of you certainly know the performance issue of using the Slowly Changing Dimension (SCD) SSIS component.
With a small or medium amount of records, this component works fine but with a bigger amount of data, I’m speaking here about more than 10000 records (which is not huge J), this component “sucks”.

Here at Mobistar I encountered this kind of issue because I had to import more or less 80000 records per day.

By using the SCD component, after 40 minutes I decided to stop the loading and starting to find another solution.
By using the Kimball SCD component, the loading takes a bit more than 2 minutes.

The Kimball SCD component receives input from 2 sorted sources, compares the data and updates, inserts or deletes the records into the destination table. Furthermore, the Kimball SCD applies an insert or update against all rows at the same time. The normal SCD applies the update or the insert against a single row at the time. This is why the normal SCD is slower than the Kimball SCD.

I guess this component could interesting you a lot in your future developments.

Feel free to send me your experience with this component.

http://kimballscd.codeplex.com/

Kind regards,
Jérémie
PS: you can also have a look to http://www.sqlbi.eu/Projects/TableDifference/tabid/74/language/en-US/Default.aspx for information about another improved SCD component

SQL Server 2005 :Insert a flat file in a OLTP table

This article will show you 4 different ways to import a flat file in a SQL Server table.

The first question that you must have is:" Have I to manipulate some of my data?"

If no, you can use the following methods:

BULK INSERT

BULK INSERT is a T-SQL command that allows to import data from a flat file into a table.

Here an example:

BULK INSERT .dbo.[TheDestinationTableName] FROM 'FullPathToTheFlatFile'
WITH
(
FIELDTERMINATOR = '',
ROWTERMINATOR = '\n'
)

Here the end character between each column is '' and the final character to pass to the next record is '\n'

bcp

bcp can copy Microsoft® SQL Server™ data to or from a data file. bcp is a command prompt utility and not a T-SQL command. bcp is used in a batch file or in your .NET application, for instance. The bcp utility is written using the ODBC bulk copy application programming interface (API).

Import/Export Tasks

As the Copy tool of SQL Server 2005, it exists also an import tool in it.

From SSMS, connect you on your destination database (which one that contains the destination table). Right click on this database and select Tasks --> Import Data... The Sql Server Import and Export Wizard window opens itself.

Select Flat File Source as Datasource. Specify the File path and the formatting. You can preview the result in the Columns tab for the columns name and the data in the Preview tab.

Click Next and follow the steps.

if you have a manipulation:

In this case, you will need to use a tool that will make the changes. This tool is SSIS. SSIS is provided with the Developer and Enterprise version of SQL Server 2005.With SSIS you can create a connection to your flat file, make some changes on the values in your DataFlow and importing them in a table thanks to an OLE DB connection.

I hope that this post will help you ...

SSIS 2005 : How to write annotations in a package?

In the SSIS Designer, right-click on the design surface of your Control Flow, Data Flow or Event Handler tab and then click Add annotation.

After you have created your annotation, you can either change the font formatting or group it to an object of your design surface. How to do it? --> right-click on it :-)