in this lab we copied files from one bucket to another in GCS. We used /** to copy all files regardless of folder structure. Why does it not create the folder structure with the copy in destination bucket? i.e. we now have all the objects in the root directory in destination.
4 Answers
my understanding is that "**" lists all objects irrespective of directory location…..-r adds the directory structure
There seems to be a nuance in the usage. When using the wildcard, individual files are taken, and the tree structure is not replicated. From the cp command documentation:
"recursive copies of buckets and bucket subdirectories produce a mirrored filename structure, while copying individually or wildcard-named objects produce flatly-named files."
This version of copying gsutil cp gs://my-bucket** gs://my-bucket seems to be inaccurate now.
Only way for me that I’ve managed to copy between buckets is using the -r flag and without wildcards.
I’ve updated to the latest gsutils and so the way it’s shown in this video, it doesn’t work for me.
UPDATE:
Seems it only works for me using the integrated shell, not when using gsutils with zsh on my Mac.
To copy data from bucket to bucket you should proceed like this:
$ gsutil cp -r gs://my-bucket-1/* gs://my-bucket-2
Note that we are using one start, not two.
Thanks for this. I’m sure it will save me some time.