Utilizing Write Sharding to Optimize Data Ingestion

Our coworker has written a script to load metadata about images gathered by robots in the Amazon warehouses of bins containing the items for orders. They have encountered some difficulty with the script where writes to their table AmazonBins are being throttled heavily and data is not making it to the table. They have modified the script to only try write operations once, which has sped up the script, but it is still losing data. Our coworker has stood up an environment for us with the AmazonBins table already created, and an EC2 instance has been pre-loaded with files containing the data as well as the script they have written to load the data into the AmazonBins table. We will need to modify the transform function in this script to implement partition sharding, which should eliminate throttling, and ensure all of the data is loaded to the table. Once this modification is made, we will run the script to test that no data is lost, Write Capacity Unit utilization has improved, and throttling has been eliminated.

Code used in the solution video for this lab can be found here.