Welcome to this hands-on lab, Viewing IoT Core Data Using BigQuery.
Google Cloud IoT Core is a fully managed service that manages and ingests data from millions of globally dispersed devices.
You will gain experience by using IoT Core to collect data from a single device. However, you can use the same configuration steps to collect data from millions of devices.
In this lab, we will:
– Get hands-on with IoT Core to create a registry, add a device, and send simulated data from a compute engine VM.
– Configure a Cloud Dataflow Template job to collect the data from a Pub/Sub topic, transform the data from JSON to table format, and store the data in BigQuery.
– Use BigQuery to query the data, sort it by the time collected to prove you have configured the IoT Data Pipeline to move data from Pub/Sub using DataFlow, and store it in BigQuery in the correct format.
– Optional: Export the data to Data Studio and display the heart rate data over time collected.
Learning Objectives
Successfully complete this lab by achieving the following learning objectives:
- Ingest the IoT Core Data
To complete this lab objective, you will need to perform the following tasks:
Create a Google Cloud IoT Core registry using following settings:
- Registry_ID:
us-iot-hr-trial - Pub/Sub topic name:
us-iot-hr-queue - Pub/Sub subscription name:
us-iot-hr-data
- Registry_ID:
Register a device in the IoT Core (VM instance). Use the following settings:
- Device_ID:
hrsensor007 - Create the public/private key pair
- Device_ID:
Send simulated data to the IoT Core device using the Python heart rate simulator. Use the following settings:
- Project_ID
<your project ID> - Device_ID
hrsensor007 - registry_Id
us-iot-hr-trial - private_key_file: =
../.ssh/ec_private.pem
- Project_ID
- Build a Cloud Dataflow Pipeline
Create a BigQuery dataset and table:
- Dataset name:
heartratedata - Table name:
heartratedatatable
- Dataset name:
Collect the endpoints for the Pub/Sub subscription and Cloud Storage Bucket:
- BigQuery Table link
- Storage URL
- Pub/Sub subscription URL
Send the Dataflow template and Pub/Sub subscription to Bigquery using the following endpoints:
- Dataflow job name
- BigQuery table information
- Pub/Sub subscription link
- Storage Bucket location
- View Our Data in BigQuery
Run a query to view the IoT data that was received in BigQuery.
Query the data to return the results sorted by
timecollected, nottimereceived.Optional: Export the results to Data Studio to create a dashboard graph.
