External source stage skipping blank lines
Moderators: chulett, rschirm, roy
External source stage skipping blank lines
Hi ,
Im using the below command to in the source program column in External source stage.
tar -xOvf /home/files/Reques_folder.tar
The output from the external source stage is skipping the Blank lines.
However when i enter the same command in unix prompt it is giving the correct data from the files including Blank lines.
Further setting in External source stage:
Final delimiter:End
Record Delimiter:UNIX Newline
Delimiter:None
Quote:None
I'm trying to read the data into one single column.
Kindly let me what could be the issue.
Thanks,
Chetan
Im using the below command to in the source program column in External source stage.
tar -xOvf /home/files/Reques_folder.tar
The output from the external source stage is skipping the Blank lines.
However when i enter the same command in unix prompt it is giving the correct data from the files including Blank lines.
Further setting in External source stage:
Final delimiter:End
Record Delimiter:UNIX Newline
Delimiter:None
Quote:None
I'm trying to read the data into one single column.
Kindly let me what could be the issue.
Thanks,
Chetan
Re: External source stage skipping blank lines
Hi ,
Is it a problem in the configuration?
Please guide me for a solution.
Thanks,
Chetan.C
Is it a problem in the configuration?
Please guide me for a solution.
Thanks,
Chetan.C
-
- Participant
- Posts: 597
- Joined: Fri Apr 29, 2005 6:19 am
- Location: Singapore
-
- Participant
- Posts: 597
- Joined: Fri Apr 29, 2005 6:19 am
- Location: Singapore
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Hi Ray,ray.wurlod wrote:Specify your record delimiter string property in the Sequential File stage as "DOS style". ...
I tried using DOS style but still the same problem.
Also when i load a tar file which has around 7000 records,the job aborts saying consumed more than 100000 bytes .
I read the posts here and understood about the APT_MAX_DELIMITED_READ_SIZE environment varible.
But my question is if the stage cannot find the record delimiter then why does it show data from file in different rows when i view data?.
Thanks,
Chetan.C
hi Kryt0n,
I just passed the file throught od -xc and i got the below output.
The First two lines of the files looks like this.
[iamie]
TLTSID=E53F96B42B72102BB3D22D89C82DEAEC.
The second portion in the code is with a blank line and next line which has data.
So it is DOS style right ?As there is \r\n.
And in the blank line it has \r\n\r\n.
But one thing that still bothers me is why the exteranl source stage cant find the record delimiter? i read about the stage documention and even tried \r\n in the external source stage.
Can you let me know where i'm going wrong?.
Thanks.
I just passed the file throught od -xc and i got the below output.
The First two lines of the files looks like this.
[iamie]
TLTSID=E53F96B42B72102BB3D22D89C82DEAEC.
The second portion in the code is with a blank line and next line which has data.
So it is DOS style right ?As there is \r\n.
And in the blank line it has \r\n\r\n.
But one thing that still bothers me is why the exteranl source stage cant find the record delimiter? i read about the stage documention and even tried \r\n in the external source stage.
Can you let me know where i'm going wrong?.
Thanks.
Code: Select all
0000000 695b 6d61 6569 0d5d 540a 544c 4953 3d44
[ i a m i e ] \r \n T L T S I D =
0000260 0a0d 0a0d 655b 766e 0d5d 520a 5145 425f
\r \n \r \n [ e n v ] \r \n R E Q _ B
Yes windows newline erros with no record delimiter.
I did read a post here about the error of Consuming 100000 bytes for which the user increased the APT_MAX_DELIMITED_READ_SIZE and the job worked.
But i dont want to do that atleast now without finding out why this is happening.
Any thoughts?
thanks,
Chetan.C
I did read a post here about the error of Consuming 100000 bytes for which the user increased the APT_MAX_DELIMITED_READ_SIZE and the job worked.
But i dont want to do that atleast now without finding out why this is happening.
Any thoughts?
thanks,
Chetan.C
The issue with the external source stage about consuming more bytes than defined i resolved that error by increasing APT_MAX_DELIMITED_READ_SIZE and APT_DEFAULT_TRANSPORT_BLOCK_SIZE.
That seems to have solved the Consuming more than 100000 bytes error.
But still did not get why it says record delimiter not found.
Anways since now i'm facing a new error about job running very slow will be opening a new thread.
Thanks,
Chetan
That seems to have solved the Consuming more than 100000 bytes error.
But still did not get why it says record delimiter not found.
Anways since now i'm facing a new error about job running very slow will be opening a new thread.
Thanks,
Chetan