Viewing Cloud IoT Core Data Using BigQuery

1.5 hours
  • 3 Learning Objectives

About this Hands-on Lab

Welcome to this hands-on lab, Viewing IoT Core Data Using BigQuery.

Google Cloud IoT Core is a fully managed service that manages and ingests data from millions of globally dispersed devices.

You will gain experience by using IoT Core to collect data from a single device. However, you can use the same configuration steps to collect data from millions of devices.

In this lab, we will:

– Get hands-on with IoT Core to create a registry, add a device, and send simulated data from a compute engine VM.
– Configure a Cloud Dataflow Template job to collect the data from a Pub/Sub topic, transform the data from JSON to table format, and store the data in BigQuery.
– Use BigQuery to query the data, sort it by the time collected to prove you have configured the IoT Data Pipeline to move data from Pub/Sub using DataFlow, and store it in BigQuery in the correct format.
– Optional: Export the data to Data Studio and display the heart rate data over time collected.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Ingest the IoT Core Data

To complete this lab objective, you will need to perform the following tasks:

  1. Create a Google Cloud IoT Core registry using following settings:

    • Registry_ID: us-iot-hr-trial
    • Pub/Sub topic name: us-iot-hr-queue
    • Pub/Sub subscription name: us-iot-hr-data
  2. Register a device in the IoT Core (VM instance). Use the following settings:

    • Device_ID: hrsensor007
    • Create the public/private key pair
  3. Send simulated data to the IoT Core device using the Python heart rate simulator. Use the following settings:

    • Project_ID <your project ID>
    • Device_ID hrsensor007
    • registry_Id us-iot-hr-trial
    • private_key_file: = ../.ssh/ec_private.pem
Build a Cloud Dataflow Pipeline
  1. Create a BigQuery dataset and table:

    • Dataset name: heartratedata
    • Table name: heartratedatatable
  2. Collect the endpoints for the Pub/Sub subscription and Cloud Storage Bucket:

    • BigQuery Table link
    • Storage URL
    • Pub/Sub subscription URL
  3. Send the Dataflow template and Pub/Sub subscription to Bigquery using the following endpoints:

    • Dataflow job name
    • BigQuery table information
    • Pub/Sub subscription link
    • Storage Bucket location
View Our Data in BigQuery
  1. Run a query to view the IoT data that was received in BigQuery.

  2. Query the data to return the results sorted by timecollected, not timereceived.

  3. Optional: Export the results to Data Studio to create a dashboard graph.

Additional Resources

The Arkan group needs to test a new IoT device that's been giving them problems in the field.

The actual device in the field is a Raspberry Pi 3B+ with a heart rate sensor connected to the device by GPIO pins.

They have sent us a copy of the scripts they use to generate the public/private keys for each device. We also received a Python code file that simulates a sample of the heart rate data collected.

A VM instance named hrsensor007 is provided for us in the lab. The VM already contains the software in the root directory named iotcore-heartrate. Use this VM instance as the IoT device to send a message to IoT Core.

Once the data arrived in Pub/Sub via IoT, we will verify that Dataflow will successfully move the data to BigQuery for later machine learning analytics on the data.

Now, have fun learning about IoT Core, Dataflow, and BigQuery.

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!

Get Started
Who’s going to be learning?