Lab
A Cloud Guru

Running a Pyspark Job on Cloud Dataproc Using Google Cloud Storage

This hands-on lab introduces how to use Google Cloud Storage as the primary input and output location for Dataproc cluster jobs. Leveraging GCS over the Hadoop Distributed File System (HDFS) allows us to treat clusters as ephemeral entities, so we can delete clusters that are no longer in use, while still preserving our data.

Try for free Contact sales

Path Info

Level

Intermediate

Duration

30m

Published

Aug 15, 2019

Challenge

Prepare Our Environment

First, we need to enable the Dataproc API:

gcloud services enable dataproc.googleapis.com

Then create a Cloud Storage bucket:

gsutil mb -l us-central1 gs://$DEVSHELL_PROJECT_ID-data

Now create the dataproc cluster:

gcloud dataproc clusters create wordcount --region=us-central1 --zone=us-central1-f --single-node --master-machine-type=n1-standard-2

And finally, download the wordcount.py file that will be used for the pyspark job:

gsutil cp -r gs://acg-gcp-labs-resources/data-engineer/dataproc/* .

Challenge

Submit the Pyspark Job to the Dataproc Cluster

In Cloud Shell, type:

gcloud dataproc jobs submit pyspark wordcount.py --cluster=wordcount --region=us-central1 -- 
gs://acg-gcp-labs-resources/data-engineer/dataproc/romeoandjuliet.txt 
gs://$DEVSHELL_PROJECT_ID-data/output/

Challenge

Review the Pyspark Output
1. In Cloud Shell, download output files from the GCS output location:
```
gsutil cp -r gs://$DEVSHELL_PROJECT_ID-data/output/* .
```
Note: Alternatively, we could download them to our local machine via the web console.
Challenge

Delete the Dataproc Cluster
1. We don't need our cluster any longer, so let's delete it. In the web console, go to the top-left menu and into BIGDATA > Dataproc.
2. Select the wordcount cluster, then click DELETE, and OK to confirm.
Our job output still remains in Cloud Storage, allowing us to delete Dataproc clusters when no longer in use to save costs, while preserving input and output resources.

Author

A Cloud Guru

The Cloud Content team comprises subject matter experts hyper focused on services offered by the leading cloud vendors (AWS, GCP, and Azure), as well as cloud-related technologies such as Linux and DevOps. The team is thrilled to share their knowledge to help you build modern tech solutions from the ground up, secure and optimize your environments, and so much more!

What's a lab?

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.