Designing data processing systems

Selecting the appropriate storage technologies. Considerations include:

●  Mapping storage systems to business requirements

●  Data modeling

●  Trade-offs involving latency, throughput, transactions

●  Distributed systems

●  Schema design

Designing data pipelines. Considerations include:

●  Data publishing and visualization (e.g., BigQuery)

●  Batch and streaming data (e.g., Dataflow, Dataproc, Apache Beam, Apache Spark and Hadoop ecosystem, Pub/Sub, Apache Kafka)

●  Online (interactive) vs. batch predictions

●  Job automation and orchestration (e.g., Cloud Composer)

Designing a data processing solution. Considerations include:

●  Choice of infrastructure

●  System availability and fault tolerance

●  Use of distributed systems

●  Capacity planning

●  Hybrid cloud and edge computing

●  Architecture options (e.g., message brokers, message queues, middleware, service-oriented architecture, serverless functions)

●  At least once, in-order, and exactly once, etc., event processing

Migrating data warehousing and data processing. Considerations include:

●  Awareness of current state and how to migrate a design to a future state

●  Migrating from on-premises to cloud (Data Transfer Service, Transfer Appliance, Cloud Networking)

●  Validating a migration

Building and operationalizing data processing systems

Building and operationalizing storage systems. Considerations include:

●  Effective use of managed services (Cloud Bigtable, Cloud Spanner, Cloud SQL, BigQuery, Cloud Storage, Datastore, Memorystore)

●  Storage costs and performance

●  Life cycle management of data

Building and operationalizing pipelines. Considerations include:

●  Data cleansing

●  Batch and streaming

●  Transformation

●  Data acquisition and import

●  Integrating with new data sources

Building and operationalizing processing infrastructure. Considerations include:

●  Provisioning resources

●  Monitoring pipelines

●  Adjusting pipelines

●  Testing and quality control

Operationalizing machine learning models

Leveraging pre-built ML models as a service. Considerations include:

●  ML APIs (e.g., Vision API, Speech API)

●  Customizing ML APIs (e.g., AutoML Vision, Auto ML text)

●  Conversational experiences (e.g., Dialogflow)

Deploying an ML pipeline. Considerations include:

●  Ingesting appropriate data

●  Retraining of machine learning models (AI Platform Prediction and Training, BigQuery ML, Kubeflow, Spark ML)

●  Continuous evaluation

Choosing the appropriate training and serving infrastructure. Considerations include:

●  Distributed vs. single machine

●  Use of edge compute

●  Hardware accelerators (e.g., GPU, TPU)

Measuring, monitoring, and troubleshooting machine learning models. Considerations include:

●  Machine learning terminology (e.g., features, labels, models, regression, classification, recommendation, supervised and unsupervised learning, evaluation metrics)

●  Impact of dependencies of machine learning models

●  Common sources of error (e.g., assumptions about data)

Ensuring solution quality

Designing for security and compliance. Considerations include:

●  Identity and access management (e.g., Cloud IAM)

●  Data security (encryption, key management)

●  Ensuring privacy (e.g., Data Loss Prevention API)

●  Legal compliance (e.g., Health Insurance Portability and Accountability Act (HIPAA), Children's Online Privacy Protection Act (COPPA), FedRAMP, General Data Protection Regulation (GDPR))

Ensuring scalability and efficiency. Considerations include:

●  Building and running test suites

●  Pipeline monitoring (e.g., Cloud Monitoring)

●  Assessing, troubleshooting, and improving data representations and data processing infrastructure

●  Resizing and autoscaling resources

Ensuring reliability and fidelity. Considerations include:

●  Performing data preparation and quality control (e.g., Dataprep)

●  Verification and monitoring

●  Planning, executing, and stress testing data recovery (fault tolerance, rerunning failed jobs, performing retrospective re-analysis)

●  Choosing between ACID, idempotent, eventually consistent requirements

Ensuring flexibility and portability. Considerations include:

●  Mapping to current and future business requirements

●  Designing for data and application portability (e.g., multicloud, data residency requirements)

●  Data staging, cataloging, and discovery