Need Help in UNIX script

balaiah · Post by **balaiah** » Mon Jul 18, 2011 11:30 pm

Hi All, Greetings!!!!!!

We are trying to split .xml file based file size. below code is not working.

Please help me in it.

Code: Select all

x=du -m $full_file

if ($x -lt 200)
then 
	Num_of_Split=1
elif (($x -gt 200) && ($x -le 400))
	Num_of_Split=2
elif (($x -gt 400) && ($x -le 600))
	Num_of_Split=3
elif (($x -gt 600) && ($x -le 800))
	Num_of_Split=4
elif (($x -gt 800) && ($x -le 1000))
	Num_of_Split=5
elif (($x -gt 1000) && ($x -le 1200))
	Num_of_Split=6
elif (($x -gt 1200) && ($x -le 1400))
	Num_of_Split=7
else 
	Num_of_Split=8
fi

Thanks & Regards.
Balaiah

ray.wurlod · Post by **ray.wurlod** » Mon Jul 18, 2011 11:44 pm

1. You need backquotes to capture the output of a command.

2. You are doing way too much work. Your entire solution is possible in a single command, such as

Code: Select all

split -b 200m $full-file

chulett · Post by **chulett** » Mon Jul 18, 2011 11:46 pm

Define "not working".

May not really matter as, unless you have a great deal of intelligence built into your "split" logic, all you'll have after the split are smaller useless XML file fragments. What are you planning on doing with these pieces?

balaiah · Post by **balaiah** » Tue Jul 19, 2011 12:09 am

Please find full script:

Code: Select all

#!/usr/bin/ksh


#split_customer()

############ Main Program ############

#### DECLARE VARIABLES ####

ENV=$1
full_file=/acis/$ENV/feed/dds1/R_ACIS_DDS_$date.DAT
full_file_copy=/acis/$ENV/feed/dds1/R_ACIS_DDS_$date.DAT.bkup
curr_dir=`pwd`
temp_full_file=$curr_dir/temp_full_file.xml
temp_file2=$curr_dir/temp_file2.xml
Header_File=$curr_dir/Header.xml
Footer_File=$curr_dir/Footer.xml
Files_To_Merge=$curr_dir/Files_To_Merge.txt

cp $full_file $full_file_copy
gzip $full_file_copy

x=du -m $full_file

if ($x -lt 200)
then 
	Num_of_Split=1
elif (($x -gt 200) && ($x -le 400))
	Num_of_Split=2
elif (($x -gt 400) && ($x -le 600))
	Num_of_Split=3
elif (($x -gt 600) && ($x -le 800))
	Num_of_Split=4
elif (($x -gt 800) && ($x -le 1000))
	Num_of_Split=5
elif (($x -gt 1000) && ($x -le 1200))
	Num_of_Split=6
elif (($x -gt 1200) && ($x -le 1400))
	Num_of_Split=7
else 
	Num_of_Split=8
fi


#### PUT <Message> and <Customer> SECTION INTO NEW LINE  ####
echo "PUT <Message> and <Customer> SECTION INTO NEW LINE"

echo "" >> $full_file

sed 's/Total_Customers></Total_Customers>\
</g' < $full_file > $temp_full_file

sed 's/><\/Message/>\
<\/Message/g' < $temp_full_file > $temp_file2
mv $temp_file2 $temp_full_file

sed 's/><\/Customer>/>\
<\/Customer>/g' < $temp_full_file > $temp_file2
mv $temp_file2 $temp_full_file

sed 's/><Customer>/>\
<Customer>/g' < $temp_full_file > $temp_file2
mv $temp_file2 $temp_full_file


#### GET HEADER/FOOTER #### 
echo "GET HEADER/FOOTER"

head_num=`grep -n '</Total_Customers>' $temp_full_file | tail -1 | cut -f 1 -d':'`
head -$head_num $temp_full_file > $Header_File 

echo '</Message>' > $Footer_File

#### GET <Customer> TAGS #### 
echo "GET <Customer> TAGS"

tail +$(($head_num+1))  $temp_full_file | sed 's/<\/Message>//g' > $temp_file2 

mv $temp_file2 $temp_full_file

#### CREATE INDIVIDUAL CUSTOMER FILES ####
echo "CREATE INDIVIDUAL CUSTOMER FILES"



typeset -i Num_of_Cust=`grep '</Customer>' $temp_full_file | wc -l`
typeset -i File_Line=1
typeset -i Cust_File_Len=`wc -l $temp_full_file  | awk '{print $1}'`
Line_Poss=$curr_dir/Line_Poss.txt
prev_sec=0
 
grep -n '</Customer>'  $temp_full_file | cut -f 1 -d':' > $Line_Poss
echo "" > $Files_To_Merge

 
i=0
while [ $i  -lt $Num_of_Split ]
do
        i=$(($i+1))
	typeset -i head_i=$(($Num_of_Cust*$i/$Num_of_Split))
        if [ $i -lt $Num_of_Split ]
        then
                typeset -i sec=`head -$head_i $Line_Poss | tail -1`
        else
                typeset -i sec=`cat $Line_Poss | tail -1`
        fi
        head -$sec  $temp_full_file | tail +$(($prev_sec+1)) > $curr_dir/dds_$((i))_.xml

	echo "i: [$i] head_i: [$head_i] sec:[$sec] prev_sec:[$prev_sec]"

	#### REMOVE INVALID CHARACTERS ####
        tr -cd '\11\12\13\14\40-\176' < $curr_dir/dds_$((i))_.xml  > $temp_file2
	mv $temp_file2 $curr_dir/dds_$((i))_.xml

	prev_sec=$sec
	echo "$curr_dir/dds_$((i))_.xml" >> $Files_To_Merge
done

echo "Files_To_Merge: `cat $Files_To_Merge`"

i=0
for file in `cat $Files_To_Merge`
do
	i=$(($i+1))
	splitted_file=$curr_dir/R_ACIS_DDS_`/usr/bin/date +%Y%m%d`_PART$i.DAT	
	typeset -i total_customers=`grep '</Customer>' $file | wc -l`
	cat $Header_File | sed "s/<Total_Customers>[0-9]*<\/Total_Customers>/<Total_Customers>$total_customers<\/Total_Customers>/g" > $splitted_file 
	cat $file >> $splitted_file
	cat $Footer_File >> $splitted_file
	ssh ftpppdac@dbsa0001.uhc.com
	scp $splitted_file ftpppdac@dbsa0001.uhc.com:/ftpdata/pdr1/ftpppdac/inbox/
done

touch $curr_dir/R_ACIS_DDS.SUCCESS

#### REMOVE TEMP FILES ####
rm -f `cat $Files_To_Merge`
rm -f  $temp_file2 
rm -f $Header_File $Footer_File
rm -f $Line_Poss $Files_To_Merge 
find /acis/$ENV/feed/dds1/R_ACIS_DDS_$date.DAT.bkup -mtime +95 -exec rm {} \;



echo "SECONDS: $SECONDS"

chulett · Post by **chulett** » Tue Jul 19, 2011 12:15 am

As Ray noted -

Code: Select all

x=`du -m $full_file`

ray.wurlod · Post by **ray.wurlod** » Tue Jul 19, 2011 12:47 am

... assuming, of course, that du even works with a file. It does not work on files (only on directories) on some UNIX variants.

balaiahmba · Post by **balaiahmba** » Tue Jul 19, 2011 2:44 am

Hi All,

I have executed the above script after modifying, Please find the error.

[bbirudu@dbsp8073] /etldata/dds1/acis/estage> ksh DDSAcisFilesSplit.ksh /etldata/dds1/acis/inbox
# -------------------------------------------------------------
#split_customer()

############ Main Program ############

#### DECLARE VARIABLES ####

DirName=$1
+ DirName=/etldata/dds1/acis/inbox
full_file=$DirName/R_ACIS_DDS_*.DAT
+ full_file=/etldata/dds1/acis/inbox/R_ACIS_DDS_*.DAT
temp_full_file=$DirName/temp_full_file.xml
+ temp_full_file=/etldata/dds1/acis/inbox/temp_full_file.xml
temp_file2=$DirName/temp_file2.xml
+ temp_file2=/etldata/dds1/acis/inbox/temp_file2.xml
Header_File=$DirName/Header.xml
+ Header_File=/etldata/dds1/acis/inbox/Header.xml
Footer_File=$DirName/Footer.xml
+ Footer_File=/etldata/dds1/acis/inbox/Footer.xml
Files_To_Merge=$DirName/Files_To_Merge.txt
+ Files_To_Merge=/etldata/dds1/acis/inbox/Files_To_Merge.txt

x=`du -m $full_file`
+ + du -m /etldata/dds1/acis/inbox/R_ACIS_DDS_20110701.DAT
x=774.96 /etldata/dds1/acis/inbox/R_ACIS_DDS_20110701.DAT

if ($x -lt 200)
then
Num_of_Split=1
elif (($x -gt 200) && ($x -le 400))
Num_of_Split=2
elif DDSAcisFilesSplit.ksh[34]: syntax error at line 39 : `elif' unexpected

ray.wurlod · Post by **ray.wurlod** » Tue Jul 19, 2011 4:23 am

elif was unexpected.

Verify the syntax of the if..elif..else..end construct in your particular shell on your particular UNIX.