dsintbuf_getrow() - row has 71 columns when 6 expected

yinyin61 · Post by **yinyin61** » Fri Jan 06, 2006 2:17 am

Hi all why is this error appearing randomly? I am looking for info...

Thanks in advanced.[/img]

ArndW · Post by **ArndW** » Fri Jan 06, 2006 3:09 am

Aileen,

the message is very clear - you have declared a column separator in your source file stage to separate the columns. DataStage expects a line in the source to have 5 of these separator characters to split the data into 6 columns. Your source data has 71 of these. You will need to check your source data file for corruption, using either UNIX tools or you could also use the debugger in the Designer canvas to do so.

chulett · Post by **chulett** » Fri Jan 06, 2006 7:49 am

Also, if this is coming from a Link Collector which it looks like it might, then double-check that all links into the stage provide the exact same metadata.

yinyin61 · Post by **yinyin61** » Mon Jan 09, 2006 1:19 am

chulett wrote:Also, if this is coming from a Link Collector which it looks like it might, then double-check that all links into the stage provide the exact same metadata.

Hi all,

Thanks for the prompt replies. However, I have yet to solve this problem.

JOB_STG_Staff_Contact_C..ShrContSTGcurrentM.SRT_01.Lnk_SRT_01: dsintbuf_getrow() - row has 66 columns when 6 expected

I have taken Arnd's advice and checked the delimiter of the ODBC source I am using. I have allowed the ODBC source to auto detect the delimiter yet the error still persists.

What's strange is I have 20 different jobs and the failure occurence is randomly appearing in the different jobs. I am out of ideas as of yet...

ray.wurlod · Post by **ray.wurlod** » Mon Jan 09, 2006 1:35 am

This error means that DataStage found 71 columns in a row, whereas your job design has only six columns defined for that link.

Can you please let us know what stage types you are using in your job design, and how they are linked together?

One place this can occur is where reading from a sequential file; because of the nature of a sequential file (you must read every byte to get to the next), you can not skip/select columns as you can when selecting rows from a database table.

Something similar can happen when using row buffering. If you push 71 columns into the row buffer, you must retrieve 71 columns. Row buffering may be implicit (row buffering on) or explicit (use of IPC, Link Partitioner or Link Collector stages).

yinyin61 · Post by **yinyin61** » Mon Jan 09, 2006 2:10 am

ray.wurlod wrote:This error means that DataStage found 71 columns in a row, whereas your job design has only six columns defined for that link.

Can you please let us know what stage types you are using in your job design, and how they are linked together?

One place this can occur is where reading from a sequential file; because of the nature of a sequential file (you must read every byte to get to the next), you can not skip/select columns as you can when selecting rows from a database table.

Something similar can happen when using row buffering. If you push 71 columns into the row buffer, you must retrieve 71 columns. Row buffering may be implicit (row buffering on) or explicit (use of IPC, Link Partitioner or Link Collector stages).

Hi Ray... Thanks for the help. below are some screens to enlighten on the design...

1) An example of the 30+ jobs is shown in the screen below:

As can been seen, the 30+ jobs share the same design. Whereby it extracts data from the ODBC stage... and pushes the data in the container.

2) Below is the sample design of the container:

The jobs will pass data into the container. And the container does the specifc transactions through the various stages.

As noted by the green tick, originally the design involved data going into the hash file on the first ticked hash file stage... and later that same hash file is used as a reference outout as noted on the second ticked hash file. This was an error (UVopen unable to open file) and now this is the modification done to solve the problem.

However this error I am facing now.

JOB_STG_Staff_Contact_C..ShrContSTGcurrentM.AGG_01.Lnk_AGG_01: dsintbuf_getrow() - row has 89 columns when 7 expected

It states that the error occur at Lnk_AGG_01 Aggreator... but I don't see why...

I have auto detect the delimiter in the job's ODBC.

I have also turn off the row buffer option in the Datastage Administrator properties tab.

I have no idea what's wrong. Error still persists.

Thanks. =)

ray.wurlod · Post by **ray.wurlod** » Mon Jan 09, 2006 4:04 pm

How many columns are defined on the input link to Agg01?

How many columns are defined on the output link from Agg01?

How many columns are defined on the other output link from Agg01?

( Optional: What are you trying to achieve by aggregating aggregated data?)

yinyin61 · Post by **yinyin61** » Tue Jan 10, 2006 7:36 pm

ray.wurlod wrote:How many columns are defined on the input link to Agg01?

How many columns are defined on the output link from Agg01?

How many columns are defined on the other output link from Agg01?

( Optional: What are you trying to achieve by aggregating aggregated data?)

6 columns on the input link to Agg01

7 rows on the first output link heading to Agg01

3 rows on the other output link heading to Hash file...

Thanks. =)

ray.wurlod · Post by **ray.wurlod** » Tue Jan 10, 2006 7:54 pm

This is weird. In your first post it was

JOB_STG_Staff_Contact_C..ShrContSTGcurrentM.Trans_01.Lnk_Trans_01: dsintbuf_getrow() - row has 71 columns when 6 expected

Later, it was

JOB_STG_Staff_Contact_C..ShrContSTGcurrentM.SRT_01.Lnk_SRT_01: dsintbuf_getrow() - row has 66 columns when 6 expected

This is a different link from a different stage (the Sort stage?).

Can you post a picture that leaves in the stage and link names (or edit the original post)?

The "int" in dsintbuf_getrow() refers to an internal buffer, such as occurs on active-to-active links. Do you have row buffering enabled or disabled? Despite what you have set in the Administrator, you need to check in job properties (Performance tab) what the setting is for this particular job - the Administrator only sets project-wide defaults. Indeed, uncheck the "use project defaults" box here and leave "enable row buffer" unchecked to guarantee that row buffering is not enabled for this job. Then try again and let us know if it's still being a problem.

JRodriguez · Post by **JRodriguez** » Tue Jan 10, 2006 8:09 pm

YinYin,

We got same error that you are getting, and we found that the situation came out more frequently in jobs that use two active stage together. And has the row buffer feature turned on - any in process or inter process.

In our case we are using Store Procedure Stage conected to a Transformer Stage; in your case you have two agregator stages.

We file a case with ascential and waiting for their answer.

Thanks,

Julio Rodriguez

jzparad · Post by **jzparad** » Tue Jan 10, 2006 9:03 pm

I vaguely remember a similar problem in a previous project and, from memory, the problem disappeared when we put an IPC stage between the active stages.

yinyin61 · Post by **yinyin61** » Tue Jan 10, 2006 9:25 pm

jzparad wrote:I vaguely remember a similar problem in a previous project and, from memory, the problem disappeared when we put an IPC stage between the active stages.

Hi all,

Thanks for the replies!

Yeah I have found out that row buffer is the culprit... Just noticed that although I turned it off in Administrator. Changes are not taking effect because each job that I have error with has not been set to check as project default in the Performance tab.

I have turned off row buffer in each 30 ++ jobs... and it's working so far.

My question is... why is row buffering effecting this and in this production server while not in other servers?

And... Can someone pls give a short elaboration on Active stages (Are they aggregators, sorters etc...) and IPC stage (Issit transformer) and why having 2 active stages linked together is not advisable when row buffer turned on? What is the significance?

Thanks guys so much!

jzparad · Post by **jzparad** » Tue Jan 10, 2006 10:13 pm

Active stages are simply stages that spawn a process. Two active stages should only spawn a single process according to the DataStage manual. By turning inter-process buffering on, you force DataStage to create two processes. On an SMP machine, this means that you can have processes running in parallel and you get the associated benefits. You can achieve a similar result by placing an IPC stage between two active stages.

As to why you sometimes get errors such as the one you experienced - I have no idea. It seems like some data is lost and, of course, this can cause errors such as missing columns or too many columns.

Read the server documentation (servjdev.pdf). It has a very good explanation of how these stages work.

jzparad · Post by **jzparad** » Tue Jan 10, 2006 10:18 pm

Active stages are simply stages that spawn a process. Two active stages should only spawn a single process according to the DataStage manual.

Sorry that should have read

Active stages are simply stages that spawn a process. Two active stages in sequence should only spawn a single process according to the DataStage manual.

ray.wurlod · Post by **ray.wurlod** » Tue Jan 10, 2006 11:44 pm

The discrepancy is so large (71 against 6) that you've got to believe there's a bug in the buffering mechanism. Aileen, please report this through your support provider - it simply shouldn't work like this. That said, it was an interesting bit of detective work to eliminate all other possibilities.