8 of the 51 questions in this set, with the correct answer marked and every option explained.
1. A company is setting up a system to manage all of the datasets it stores in Amazon S3. The company would like to automate running transformation jobs on the data and maintaining a catalog of the metadata concerning the datasets. The solution should require the least amount of setup and maintenance. Which solution will allow the company to achieve its goals?
✗Create an AWS Data Pipeline that transforms the data. Then, create an Apache Hive metastore and a script to run transformation jobs on a schedule.
Data Pipeline is a legacy orchestration service, and pairing it with a self-managed Hive metastore means running two components that Glue replaces with one managed service.
✗Create an Amazon EMR cluster with Apache Hive installed. Then, create a Hive metastore and a script to run transformation jobs on a schedule.
Functionally capable, but you now own a cluster to size, patch and pay for. The requirement is the least setup and maintenance, and Glue is serverless.
✓Create an AWS Glue crawler to populate the AWS Glue Data Catalog. Then, author an AWS Glue ETL job, and set up a schedule for data transformation jobs.
AWS Glue is the correct answer because this option requires the least amount of setup and maintenance since it is serverless, and it does not require management of the infrastructure. Refer to this link for supporting information. A, C, and D are all solutions that can solve the problem, but require more steps for configuration, and require higher operational overhead to run and maintain.
✗Create an Amazon EMR cluster with Apache Spark installed. Then, create an Apache Hive metastore and a script to run transformation jobs on a schedule.
Same objection. Spark on EMR is powerful and appropriate for heavy custom processing, but it carries cluster management overhead that a managed crawler and ETL job avoid entirely.
2. A local university wants to track cars in a parking lot to determine which students are parking in the lot. The university is wanting to ingest videos of the cars parking in near-real time, use machine learning to identify license plates, and store that data in an AWS data store. Which solution meets these requirements with the LEAST amount of development effort?
✓Use Amazon Kinesis Video Streams to ingest the videos in near-real time, use the Kinesis Video Streams integration with Amazon Rekognition Video to identify the license plate information, and then store the results in DynamoDB.
Kinesis Video Streams is used to stream videos in near-real time. Amazon Rekognition Video uses Amazon Kinesis Video Streams to receive and process a video stream. After the videos have been processed by Rekognition we can output the results in DynamoDB.
✗Use Amazon Kinesis Data Streams to ingest the video in near-real time, use the Kinesis Data Streams consumer integrated with Amazon Rekognition Video to process the license plate information, and then store results in DynamoDB.
Data Streams carries generic records, not video. Rekognition Video's native stream integration is with Kinesis Video Streams specifically.
✗Use Amazon Kinesis Data Streams to ingest videos in near-real time, call Amazon Rekognition to identify license plate information, and then store results in DynamoDB.
Wrong ingestion service again, and calling Rekognition manually means writing and maintaining the plumbing that the Video Streams integration already provides.
✗Use Amazon Kinesis Firehose to ingest the video in near-real time and outputs results onto S3. Set up a Lambda function that triggers when a new video is PUT onto S3 to send results to Amazon Rekognition to identify license plate information, and then store results in DynamoDB.
Firehose buffers and delivers in batches, so this is not near-real time. It also adds an S3 round trip and custom Lambda code, which is more development effort rather than less.
3. Which service in the Kinesis family allows you to easily load streaming data into data stores and analytics tools?
✗Kinesis Video Streams
Purpose-built for ingesting and storing video and other time-encoded media for playback and analysis. Wrong data type for loading into a warehouse.
✓Kinesis Firehose
Kinesis Firehose is perfect for streaming data into AWS and sending it directly to its final destination - places like S3, Redshift, Elastisearch, and Splunk Instances.
✗Kinesis Data Analytics
Runs SQL or Apache Flink over a stream to compute results in flight. It analyses data rather than delivering it to a destination.
✗Kinesis Streams
A durable buffer that holds records for consumers to read. Getting those records into S3 or Redshift still requires something to do the loading — which is Firehose.
4. Your organization has a standalone Javascript (Node.js) application that streams data into AWS using Kinesis Data Streams. You notice that they are using the Kinesis API (AWS SDK) over the Kinesis Producer Library (KPL). What might be the reasoning behind this?
✗The Kinesis API (AWS SDK) runs faster in Javascript applications over the Kinesis Producer Library.
Raw speed is not the deciding factor. Where the KPL can be used at all, its record aggregation usually delivers higher effective throughput than individual API calls.
✗The Kinesis API (AWS SDK) provides greater functionality over the Kinesis Producer Library.
The reverse. The KPL layers batching, aggregation, automatic retries and CloudWatch metrics on top of the plain API.
✗The Kinesis Producer Library cannot be integrated with a Javascript application because of its asynchronous architecture.
Asynchrony is no obstacle — Node.js is built around it. The real barrier is that the KPL ships as a Java process, which a standalone Node application would have to host separately.
✓The Kinesis Producer Library must be installed as a Java application to use with Kinesis Data Streams.
The KPL must be installed as a Java application before it can be used with your Kinesis Data Streams. There are ways to process KPL serialized data within AWS Lambda, in Java, Node.js, and Python, but not if these answers mentions Lambda.
5. You have been tasked with capturing two different types of streaming events. The first event type includes mission-critical data that needs to immediately be processed before operations can continue. The second event type includes data of less importance, but operations can continue without immediately processing. What is the most appropriate solution to record these different types of events?
✗Capture both event types using the Kinesis Producer Library (KPL).
The KPL deliberately buffers records before sending, up to RecordMaxBufferedTime, so it can aggregate them. That delay is precisely what mission-critical events cannot tolerate.
✓Capture the mission critical events with the PutRecords API call and the second event type with the Kinesis Producer Library (KPL).
The question is about sending data to Kinesis synchronously vs. asynchronously. PutRecords is a synchronous send function, so it must be used for the first event type (critical events). The Kinesis Producer Library (KPL) implements an asynchronous send function, so it can be used for the second event type. In this scenario, the reason to use the KPL over the PutRecords API call is because: KPL can incur an additional processing delay of up to RecordMaxBufferedTime within the library (user-configurable). Larger values of RecordMaxBufferedTime results in higher packing efficiencies and better performance. Applications that cannot tolerate this additional delay may need to use the AWS SDK directly. For more information about using the AWS SDK with Kinesis Data Streams, see Developing Producers Using the Amazon Kinesis Data Streams API with the AWS SDK for Java. For more information about RecordMaxBufferedTime and other user-configurable properties of the KPL, see Configuring the Kinesis Producer Library.
✗Capture both events with the PutRecords API call.
Correct for the urgent stream, but applying it to the bulk stream too gives up the aggregation that makes high-volume ingestion efficient.
✗Capture the mission critical events with the Kinesis Producer Library (KPL) and the second event type with the Putrecords API call.
Exactly inverted. The buffered, higher-latency path is assigned to the data that must move immediately, and the synchronous path to the data that could wait.
6. You are collecting clickstream data from an e-commerce website to make near-real time product suggestions for users actively using the site. Which combination of tools can be used to achieve the quickest recommendations and meets all of the requirements?
✓Use Kinesis Data Streams to ingest clickstream data, then use Kinesis Data Analytics to run real time SQL queries to gain actionable insights and trigger real-time recommendations with AWS Lambda functions based on conditions.
Kinesis Data Analytics gets its input streaming data from Kinesis Data Streams or Kinesis Data Firehose. You can use Kinesis Data Analytics to run real-time SQL queries on your data. Once certain conditions are met you can trigger Lambda functions to make real time product suggestions to users. It is not important that we store or persist the clickstream data.
✗Use Kinesis Data Streams to ingest clickstream data, then use Lambda to process that data and write it to S3. Once the data is on S3, use Athena to query based on conditions that data and make real time recommendations to users.
Writing to S3 and querying with Athena introduces seconds to minutes of latency. Far too slow to influence a shopper who is browsing right now.
✗Use Kinesis Data Firehose to ingest click stream data, then use Kinesis Data Analytics to run real time SQL queries to gain actionable insights and trigger real-time recommendations with AWS Lambda functions based on conditions, then use Lambda to load these results into S3.
Firehose buffers before delivering, which adds latency at the very front of the pipeline. Data Streams provides the sub-second path that real-time recommendations need.
✗Use the Kinesis Data Analytics to ingest the clickstream data directly and run real time SQL queries to gain actionable insights and trigger real-time recommendations with AWS Lambda functions based on conditions.
Data Analytics is not an ingestion endpoint. It reads from a source such as a Data Stream or Firehose delivery stream; it cannot receive events from a website directly.
7. Your organization needs to find a way to capture streaming data from certain events customers are performing. These events are a crucial part of the organization's business development and cannot afford to be lost. You've already set up a Kinesis Data Stream and a consumer EC2 instance to process and deliver the data into S3. You've noticed that the last few days of events are not showing up in S3 and your EC2 instance has been shutdown. What combination of steps can you take to ensure this does not happen again?
✗Set up CloudWatch monitoring for your EC2 instance as well as AutoScaling if your consumer EC2 instance is shutdown. Next, set up multiple Kinesis Data Streams to process the data on the EC2 instance.
Additional streams multiply throughput, not durability. If the consumer is down, records still expire unread — now across several streams instead of one.
✗Set up CloudWatch monitoring for your EC2 instance as well as AutoScaling if your consumer EC2 instance is shutdown. Next, send the data to Kinesis Data Firehose before writing the data into S3. Since Kinesis Data Firehose has retry mechanism built-in, the changes of data being lost is extremely unlikely.
Firehose would genuinely simplify delivery to S3 and is a reasonable redesign. As an answer to this question it sidesteps the requirement rather than addressing it — the default 24-hour retention window is what caused the loss, and Firehose does not extend it.
✗Set up CloudWatch monitoring for your EC2 instance as well as AutoScaling if your consumer EC2 instance is shutdown. Next, set up a Lambda function to poll the Kinesis Data Stream for failed delivered records and then send those requests back into the consumer EC2 instance.
Kinesis has no concept of a failed-delivery queue to poll. Records simply sit in the shard until read or until retention expires, so there is nothing for this Lambda to find.
✓Set up CloudWatch monitoring for your EC2 instance as well as AutoScaling if your consumer EC2 instance is shutdown. Next, ensure that the maximum amount of hours are selected (168 hours) for data retention when creating your Kinesis Data Stream. Finally, write logic on the consumer EC2 instance that handles unprocessed data in the Kinesis Data Stream and failed writes to S3.
In this setup, the data is being ingested by Kinesis Data Streams and processes and delivered using an EC2 instance. It's best practice to always setup CloudWatch monitoring for your EC2 instance as well as AutoScaling if your consumer EC2 instance is shutdown. Since this data is critical data that we cannot afford to lose, we should set the retention period for the maximum number of hours (168 hours or 7 days). Finally, we need to have reprocessed the failed records that are still in the data stream and that fail to write to S3.
8. Which service built by AWS makes it easy to set up a retry mechanism, aggregate records to improve throughput, and automatically submits CloudWatch metrics?
✓Kinesis Producer Library (KPL)
Although the Kinesis API built into the AWS SDK can be used for all of this, the Kinesis Producer Library (KPL) makes it easy to integrate all of this into your applications.
✗Kinesis Consumer Library
Not an AWS product name. The consumer-side library is the Kinesis Client Library.
✗Kinesis API (AWS SDK)
The raw interface the KPL is built on. It sends records exactly as instructed, with no aggregation, no automatic retry policy and no metrics of its own.
✗Kinesis Client Library (KCL)
The consumer-side counterpart. It handles checkpointing, shard leases and load balancing across workers reading from a stream — the mirror image of the producer concerns described here.
43 more questions in the app
Practise the full 51-question set with a timer, scoring and progress tracking.
