Google Certified Professional Data Engineer

Sign Up Free or Log In to participate!

Demo Lab: Streaming Pipelines with Cloud Dataflow

The following command line is not working for me:

(tweeper) 2 ~> $python pipeline.py –streaming –runner DataflowRunner –project xxx –region us-central1 ….

and in order to execute localy scripts, I am obliged to use  the first example in this link https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login.

When running the command I have got the following errors. Help please. Thanks in advance.

Traceback (most recent call last):

File "pipeline.py", line 38, in

p = beam.Pipeline(options=PipelineOptions())

File "/home/cloud_user_p_95c41be3/tweeper/lib/python3.7/site-packages/apache_beam/pipeline.py", line 197, in init

errors = PipelineOptionsValidator(self._options, runner).validate()

File "/home/cloud_user_p_95c41be3/tweeper/lib/python3.7/site-packages/apache_beam/options/pipeline_options_validator.py", line 146, in validate

errors.extend(self.options.view_as(cls).validate(self))

File "/home/cloud_user_p_95c41be3/tweeper/lib/python3.7/site-packages/apache_beam/options/pipeline_options.py", line 715, in validate

default_bucket = self._create_default_gcs_bucket()

……………………………………

  File "/home/cloud_user_p_95c41be3/tweeper/lib/python3.7/site-packages/apitools/base/py/base_api.py", line 729, in _RunMethod

http, http_request, **opts)

File "/home/cloud_user_p_95c41be3/tweeper/lib/python3.7/site-packages/apitools/base/py/http_wrapper.py", line 360, in MakeRequest

max_retry_wait, total_wait_sec))

File "/home/cloud_user_p_95c41be3/tweeper/lib/python3.7/site-packages/apache_beam/io/gcp/gcsio_overrides.py", line 47, in retry_func

return http_wrapper.HandleExceptionsAndRebuildHttpConnections(retry_args)

File "/home/cloud_user_p_95c41be3/tweeper/lib/python3.7/site-packages/apitools/base/py/http_wrapper.py", line 304, in HandleExceptionsAndRebuildHttpConnections

raise retry_args.exc

File "/home/cloud_user_p_95c41be3/tweeper/lib/python3.7/site-packages/apitools/base/py/http_wrapper.py", line 350, in MakeRequest

check_response_func=check_response_func)

File "/home/cloud_user_p_95c41be3/tweeper/lib/python3.7/site-packages/apitools/base/py/http_wrapper.py", line 400, in _MakeRequestNoRetry

redirections=redirections, connection_type=connection_type)

File "/home/cloud_user_p_95c41be3/tweeper/lib/python3.7/site-packages/oauth2client/transport.py", line 159, in new_request

credentials._refresh(orig_request_method)

File "/home/cloud_user_p_95c41be3/tweeper/lib/python3.7/site-packages/oauth2client/contrib/gce.py", line 128, in _refresh

self._retrieve_info(http)

File "/home/cloud_user_p_95c41be3/tweeper/lib/python3.7/site-packages/oauth2client/contrib/gce.py", line 113, in _retrieve_info

self.service_account_email = info[’ema

0 Answers

Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!

Get Started
Who’s going to be learning?