Serverless Data Processing with Dataflow

Price
$1,995.00 USD

Duration
3 Days

 

Delivery Methods
Virtual Instructor Led
Private Group

Course Overview

This training is intended for big data practitioners who want to further their understanding of Dataflow in order to advance their data processing applications. Beginning with foundations, this training explains how Apache Beam and Dataflow work together to meet your data processing needs without the risk of vendor lock-in.The section on developing pipelines covers how you convert your business logic into data processing applications that can run on Dataflow. This training culminates with a focus on operations, which reviews the most important lessons for operating a data application on Dataflow, including monitoring, troubleshooting, testing, and reliability.

Course Objectives

  • Demonstrate how Apache Beam and Dataflow work together to fulfill your organization’s data processing needs.
  • Summarize the benefits of the Beam Portability Framework and enable it for your Dataflow pipelines.
  • Enable Shuffle and Streaming Engine, for batch and streaming pipelines respectively, for maximum performance.
  • Enable Flexible Resource Scheduling for more cost-efficient performance.
  • Select the right combination of IAM permissions for your Dataflow job.
  • Implement best practices for a secure data processing environment.
  • Select and tune the I/O of your choice for your Dataflow pipeline.
  • Use schemas to simplify your Beam code and improve the performance of your pipeline.
  • Develop a Beam pipeline using SQL and DataFrames.
  • Perform monitoring, troubleshooting, testing and CI/CD on Dataflow pipelines.
  • Who Should Attend?

  • Data engineer.
  • Data analysts and data scientists aspiring to develop data engineering skills
    • Top-rated instructors: Our crew of subject matter experts have an average instructor rating of 4.8 out of 5 across thousands of reviews.
    • Authorized content: We maintain more than 35 Authorized Training Partnerships with the top players in tech, ensuring your course materials contain the most relevant and up-to date information.
    • Interactive classroom participation: Our virtual training includes live lectures, demonstrations and virtual labs that allow you to participate in discussions with your instructor and fellow classmates to get real-time feedback.
    • Post Class Resources: Review your class content, catch up on any material you may have missed or perfect your new skills with access to resources after your course is complete.
    • Private Group Training: Let our world-class instructors deliver exclusive training courses just for your employees. Our private group training is designed to promote your team’s shared growth and skill development.
    • Tailored Training Solutions: Our subject matter experts can customize the class to specifically address the unique goals of your team.

    Course Prerequisites

  • Building Batch Data Pipelines
  • Building Resilient Streaming Analytics Systems
  • Agenda

    1 - Introduction

    • Introduce the course objectives.
    • Demonstrate how Apache Beam and Dataflow work together to fulfill your organization’s data processing needs.

    2 - Beam Portability

    • Summarize the benefits of the Beam Portability Framework.
    • Customize the data processing environment of your pipeline using custom containers.
    • Review use cases for cross-language transformations.
    • Enable the Portability framework for your Dataflow pipelines.

    3 - Separating Compute and Storage with Dataflow

    • Enable Shuffle and Streaming Engine, for batch and streaming pipelines respectively, for maximum performance.
    • Enable Flexible Resource Scheduling for more cost-efficient performance.

    4 - IAM, Quotas, and Permissions

    • Select the right combination of IAM permissions for your Dataflow job.
    • Determine your capacity needs by inspecting the relevant quotas for your Dataflow jobs.

    5 - Security

    • Select your zonal data processing strategy using Dataflow, depending on your data locality needs.
    • Implement best practices for a secure data processing environment.

    6 - Beam Concepts Review

    • Review main Apache Beam concepts (Pipeline, PCollections, PTransforms, Runner, reading/writing, Utility PTransforms, side inputs), bundles and DoFn Lifecycle.

    7 - Windows, Watermarks, Triggers

    • Implement logic to handle your late data.
    • Review different types of triggers.
    • Review core streaming concepts (unbounded PCollections, windows).

    8 - Sources and Sinks

    • Write the I/O of your choice for your Dataflow pipeline.
    • Tune your source/sink transformation for maximum performance.
    • Create custom sources and sinks using SDF.

    9 - Schemas

    • Introduce schemas, which give developers a way to express structured data in their Beam pipelines.
    • Use schemas to simplify your Beam code and improve the performance of your pipeline.

    10 - State and Timers

    • Identify use cases for state and timer API implementations.
    • Select the right type of state and timers for your pipeline.

    11 - Best Practices

    • Implement best practices for Dataflow pipelines.

    12 - Dataflow SQL and DataFrames

    • Develop a Beam pipeline using SQL and DataFrames.

    13 - Beam Notebooks

    • Prototype your pipeline in Python using Beam notebooks.
    • Use Beam magics to control the behavior of source recording in your notebook.
    • Launch a job to Dataflow from a notebook.

    14 - Monitoring

    • Navigate the Dataflow Job Details UI.
    • Interpret Job Metrics charts to diagnose pipeline regressions.
    • Set alerts on Dataflow jobs using Cloud Monitoring.

    15 - Logging and Error Reporting

    • Use the Dataflow logs and diagnostics widgets to troubleshoot pipeline issues.

    16 - Troubleshooting and Debug

    • Use a structured approach to debug your Dataflow pipelines.
    • Examine common causes for pipeline failures.

    17 - Performance

    • Understand performance considerations for pipelines.
    • Consider how the shape of your data can affect pipeline performance.

    18 - Testing and CI/CD

    • Testing approaches for your Dataflow pipeline.
    • Review frameworks and features available to streamline your CI/CD workflow for Dataflow pipelines.

    19 - Reliability

    • Implement reliability best practices for your Dataflow pipelines.

    20 - Flex Templates

    • Using flex templates to standardize and reuse Dataflow pipeline code.

    21 - Summary

    • Summary
     

    Get in touch to schedule training for your team
    We can enroll multiple students in an upcoming class or schedule a dedicated private training event designed to meet your organization’s needs.

     



    Do You Have Additional Questions? Please Contact Us Below.

    contact us contact us 
     
    Contact Us about Starting Your Business Training Strategy with New Horizons