Hi I have run the crawler(s) many times, but no table is created at the end of the scan and no errors reported. Cannot see where it is going wrong. the S3 bucket path is s3://openaq-fetches/realtime/2018-10-09/. When I look at the DB after the crawler has finished there is just empty space. Any Ideas?
2 Answers
Hi Arthur,
Almost every time this issue comes up, it has to do with insufficent permissions in the IAM role given to the crawler on the S3 buckets. Be sure you explicitly grant access to the OpenAQ Fetches bucket.
–Scott
Did you manually download the data and load it into your S3?..If yes, did you download all the files under that folder and upload it into your S3?
Hi Scott,
I had to leave this for a while and come back to it. But yes I have the IAM as
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::openaq-fetches/realtime/2018-10-9/*"
]
}
]
}
And it still does not work. (forget the lack indentations just a cut and paste). The Pole was created by the glue console.
Hi Arthur, do check out this post as it seems like it might be similar: https://acloud.guru/forums/aws-csa-pro-2019/discussion/-MHEOGPpt1psUkeGyHwE/aws_glue_throwing_com.amazonaw
When you created the IAM role for the crawler did you give it permissions to s3?