Hello Everyone,
We have a parallel Job reading DB2 AS400 table and writing out to AWS S3 bucket. Now, while this Job is working for all other relatively smaller tables, but for one large table with 200 million records and of size around 50 GB it is failing after few hours with the following error. We do not get this error if we limit the number of rows to lets say couple of thousands.
An exception occurred: com.amazonaws.services.s3.model.AmazonS3Exception: Part number must be an integer between 1 and 10000, inclusive
Now, based upon internet search, it appears that this error was resolved in other AWS S3 upload methods (not Datastage; could not find anything on DataStage) can be resolved by increasing the default chunk size (multipart-chunk-size-mb) from 15MB. to a larger value like 50 MB.
I am not sure what is the corresponding S3 connector property or environment variable to set in DataStage for the chunk size.
I tried with $APT_DEFAULT_TRANSPORT_BLOCK_SIZE and set to its maximum 1048576 but it did not help.
Please help me if you have faced this error before or any ideas on someone who has?
S3 connector error with transferring large files
Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 3
- Joined: Sat Aug 26, 2017 7:55 pm
Return to “IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)”
Jump to
- Moderators' Choice
- ↳ Editor's BLOG Corner
- ↳ Ask the Experts! - Dads and Grads
- ↳ DSXchange Testimonials
- ↳ Cognos (IBM BI)
- FAQs
- ↳ FAQs
- ↳ FAQ Discussion
- DataStage
- ↳ General
- ↳ IBM<sup>®</sup> Infosphere DataStage Server Edition
- ↳ IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
- ↳ Archive of DataStage Users@Oliver.com
- IBM<sup>®</sup>Infosphere Products<sup></sup>
- ↳ Business Glossary
- Suggestions
- ↳ Site/Forum
- ↳ Enhancement Wish List
- Consulting
- ↳ Talent
- ↳ Looking for Talent
- Support
- ↳ Parameter Manager
- ↳ Compile All Plus
- Usergroup Forums
- ↳ Usergroup Central Forum
- ↳ Heartland Usergroup Forum
- The Written Word
- ↳ Articles, White Papers and Tips and Tricks
- ↳ Product Documentation
- Third Party Applications
- ↳ Third Party Applications
- Product Derivatives
- ↳ Functions
- ↳ Routines
- ↳ Jobs
- ↳ Logs
- Tools
- ↳ Tools Forum
- Category
- ↳ Infosphere Master Data Management
- ↳ Data Quality Best Practices
- ↳ IBM QualityStage
- ↳ Information Analyzer (formerly ProfileStage)
- ↳ IBM<sup>®</sup> SOA Editions (Formerly RTI Services)
- ↳ IBM<sup>®</sup> DataStage TX
- ↳ BI
- ↳ Data Integration