8 of the 45 questions in this set, with the correct answer marked and every option explained.
1. A company is setting up a system to manage all of the datasets it stores in Amazon S3. The company would like to automate running transformation jobs on the data and maintaining a catalog of the metadata concerning the datasets. The solution should require the least amount of setup and maintenance. Which solution will allow the company to achieve its goals?
✓Create an AWS Glue crawler to populate the AWS Glue Data Catalog. Then, author an AWS Glue ETL job, and set up a schedule for data transformation jobs.
AWS Glue is the correct answer because this option requires the least amount of setup and maintenance since it is serverless, and it does not require management of the infrastructure. Refer to this link for supporting information. A, C, and D are all solutions that can solve the problem, but require more steps for configuration, and require higher operational overhead to run and maintain.
✗Create an Amazon EMR cluster with Apache Hive installed. Then, create a Hive metastore and a script to run transformation jobs on a schedule.
Delivers the outcome but hands you a cluster to size, patch and pay for around the clock. The requirement explicitly favours minimal setup and maintenance.
✗Create an Amazon EMR cluster with Apache Spark installed. Then, create an Apache Hive metastore and a script to run transformation jobs on a schedule.
Spark is the stronger engine for heavy custom processing, but the operational burden is the same — a managed cluster plus a metastore you run yourself.
✗Create an AWS Data Pipeline that transforms the data. Then, create an Apache Hive metastore and a script to run transformation jobs on a schedule.
Combines a legacy orchestration service with a self-managed metastore. Two components to maintain where Glue provides both as one serverless service.
2. A local university wants to track cars in a parking lot to determine which students are parking in the lot. The university is wanting to ingest videos of the cars parking in near-real time, use machine learning to identify license plates, and store that data in an AWS data store. Which solution meets these requirements with the LEAST amount of development effort?
✗Use Amazon Kinesis Data Streams to ingest the video in near-real time, use the Kinesis Data Streams consumer integrated with Amazon Rekognition Video to process the license plate information, and then store results in DynamoDB.
Rekognition Video's stream processor connects to Kinesis Video Streams, not Data Streams. Data Streams carries opaque records with no notion of frames or codecs.
✗Use Amazon Kinesis Firehose to ingest the video in near-real time and outputs results onto S3. Set up a Lambda function that triggers when a new video is PUT onto S3 to send results to Amazon Rekognition to identify license plate information, and then store results in DynamoDB.
Firehose delivers in buffered batches, so the pipeline is not near-real time. It also adds an S3 hop and bespoke Lambda code, increasing development effort rather than reducing it.
✓Use Amazon Kinesis Video Streams to ingest the videos in near-real time, use the Kinesis Video Streams integration with Amazon Rekognition Video to identify the license plate information, and then store the results in DynamoDB.
Kinesis Video Streams is used to stream videos in near-real time. Amazon Rekognition Video uses Amazon Kinesis Video Streams to receive and process a video stream. After the videos have been processed by Rekognition we can output the results in DynamoDB.
✗Use Amazon Kinesis Data Streams to ingest videos in near-real time, call Amazon Rekognition to identify license plate information, and then store results in DynamoDB.
Wrong ingestion service, and invoking Rekognition by hand means building the frame extraction and call orchestration yourself.
3. Which service in the Kinesis family allows you to easily load streaming data into data stores and analytics tools?
✗Kinesis Video Streams
Handles video and other time-encoded media, with APIs for playback and frame access. A specialised path, not a general loader.
✗Kinesis Streams
Retains records so consumers can read them at their own pace. It holds data; something else has to move it onward.
✓Kinesis Firehose
Kinesis Firehose is perfect for streaming data into AWS and sending it directly to its final destination - places like S3, Redshift, Elastisearch, and Splunk Instances.
✗Kinesis Data Analytics
Computes over a stream with SQL or Flink. It derives results in flight rather than depositing raw data into a store.
4. Your organization has a standalone Javascript (Node.js) application that streams data into AWS using Kinesis Data Streams. You notice that they are using the Kinesis API (AWS SDK) over the Kinesis Producer Library (KPL). What might be the reasoning behind this?
✗The Kinesis API (AWS SDK) provides greater functionality over the Kinesis Producer Library.
Inverted. The KPL sits above the SDK and adds aggregation, batching, retry handling and metrics.
✗The Kinesis API (AWS SDK) runs faster in Javascript applications over the Kinesis Producer Library.
Performance is not what decides this. Where the KPL can run, aggregating many small records into fewer requests typically raises effective throughput.
✓The Kinesis Producer Library must be installed as a Java application to use with Kinesis Data Streams.
The KPL must be installed as a Java application before it can be used with your Kinesis Data Streams. There are ways to process KPL serialized data within AWS Lambda, in Java, Node.js, and Python, but not if these answers mentions Lambda.
✗The Kinesis Producer Library cannot be integrated with a Javascript application because of its asynchronous architecture.
Node.js handles asynchrony natively, so that is no barrier. The genuine obstacle is that the KPL runs as a Java process, which a Node application would have to launch and manage alongside itself.
5. You have been tasked with capturing two different types of streaming events. The first event type includes mission-critical data that needs to immediately be processed before operations can continue. The second event type includes data of less importance, but operations can continue without immediately processing. What is the most appropriate solution to record these different types of events?
✓Capture the mission critical events with the PutRecords API call and the second event type with the Kinesis Producer Library (KPL).
The question is about sending data to Kinesis synchronously vs. asynchronously. PutRecords is a synchronous send function, so it must be used for the first event type (critical events). The Kinesis Producer Library (KPL) implements an asynchronous send function, so it can be used for the second event type. In this scenario, the reason to use the KPL over the PutRecords API call is because: KPL can incur an additional processing delay of up to RecordMaxBufferedTime within the library (user-configurable). Larger values of RecordMaxBufferedTime results in higher packing efficiencies and better performance. Applications that cannot tolerate this additional delay may need to use the AWS SDK directly. For more information about using the AWS SDK with Kinesis Data Streams, see Developing Producers Using the Amazon Kinesis Data Streams API with the AWS SDK for Java. For more information about RecordMaxBufferedTime and other user-configurable properties of the KPL, see Configuring the Kinesis Producer Library.
✗Capture the mission critical events with the Kinesis Producer Library (KPL) and the second event type with the Putrecords API call.
Assigns each path to the wrong stream. The KPL introduces deliberate buffering delay, which is unacceptable for events that must be processed immediately.
✗Capture both event types using the Kinesis Producer Library (KPL).
Applies that same buffering to the urgent stream. RecordMaxBufferedTime is a feature for throughput and a liability for latency.
✗Capture both events with the PutRecords API call.
Right for the urgent events, wasteful for the bulk ones. Sending high-volume low-priority data record by record forgoes the aggregation that keeps costs and request counts down.
6. You are collecting clickstream data from an e-commerce website to make near-real time product suggestions for users actively using the site. Which combination of tools can be used to achieve the quickest recommendations and meets all of the requirements?
✓Use Kinesis Data Streams to ingest clickstream data, then use Kinesis Data Analytics to run real time SQL queries to gain actionable insights and trigger real-time recommendations with AWS Lambda functions based on conditions.
Kinesis Data Analytics gets its input streaming data from Kinesis Data Streams or Kinesis Data Firehose. You can use Kinesis Data Analytics to run real-time SQL queries on your data. Once certain conditions are met you can trigger Lambda functions to make real time product suggestions to users. It is not important that we store or persist the clickstream data.
✗Use Kinesis Data Firehose to ingest click stream data, then use Kinesis Data Analytics to run real time SQL queries to gain actionable insights and trigger real-time recommendations with AWS Lambda functions based on conditions, then use Lambda to load these results into S3.
Firehose buffers before it delivers, so latency is introduced at the very first step. Data Streams provides the sub-second path this scenario needs.
✗Use the Kinesis Data Analytics to ingest the clickstream data directly and run real time SQL queries to gain actionable insights and trigger real-time recommendations with AWS Lambda functions based on conditions.
Data Analytics reads from a stream; it is not itself an ingestion endpoint. A website cannot post events to it directly.
✗Use Kinesis Data Streams to ingest clickstream data, then use Lambda to process that data and write it to S3. Once the data is on S3, use Athena to query based on conditions that data and make real time recommendations to users.
Persisting to S3 and querying with Athena puts seconds to minutes between the click and the recommendation. The shopper will have moved on.
7. Your organization needs to find a way to capture streaming data from certain events customers are performing. These events are a crucial part of the organization's business development and cannot afford to be lost. You've already set up a Kinesis Data Stream and a consumer EC2 instance to process and deliver the data into S3. You've noticed that the last few days of events are not showing up in S3 and your EC2 instance has been shutdown. What combination of steps can you take to ensure this does not happen again?
✓Set up CloudWatch monitoring for your EC2 instance as well as AutoScaling if your consumer EC2 instance is shutdown. Next, ensure that the maximum amount of hours are selected (168 hours) for data retention when creating your Kinesis Data Stream. Finally, write logic on the consumer EC2 instance that handles unprocessed data in the Kinesis Data Stream and failed writes to S3.
In this setup, the data is being ingested by Kinesis Data Streams and processes and delivered using an EC2 instance. It's best practice to always setup CloudWatch monitoring for your EC2 instance as well as AutoScaling if your consumer EC2 instance is shutdown. Since this data is critical data that we cannot afford to lose, we should set the retention period for the maximum number of hours (168 hours or 7 days). Finally, we need to have reprocessed the failed records that are still in the data stream and that fail to write to S3.
✗Set up CloudWatch monitoring for your EC2 instance as well as AutoScaling if your consumer EC2 instance is shutdown. Next, set up a Lambda function to poll the Kinesis Data Stream for failed delivered records and then send those requests back into the consumer EC2 instance.
There is no failed-delivery queue in Kinesis to poll. Records sit in the shard until a consumer reads them or retention expires, so this Lambda would have nothing to collect.
✗Set up CloudWatch monitoring for your EC2 instance as well as AutoScaling if your consumer EC2 instance is shutdown. Next, send the data to Kinesis Data Firehose before writing the data into S3. Since Kinesis Data Firehose has retry mechanism built-in, the changes of data being lost is extremely unlikely.
A reasonable simplification of the delivery path in general. It does not address the actual cause here, which is that records aged out of the stream during the outage — Firehose does not extend retention.
✗Set up CloudWatch monitoring for your EC2 instance as well as AutoScaling if your consumer EC2 instance is shutdown. Next, set up multiple Kinesis Data Streams to process the data on the EC2 instance.
More streams add throughput, not durability. With the consumer offline the same expiry problem occurs across every one of them.
8. Which service built by AWS makes it easy to set up a retry mechanism, aggregate records to improve throughput, and automatically submits CloudWatch metrics?
✗Kinesis Client Library (KCL)
The consumer-side equivalent. It manages shard leases, checkpointing and worker coordination — the reading half of the picture rather than the writing half described here.
✗Kinesis Consumer Library
Not a product name. AWS calls the consumer library the Kinesis Client Library.
✗Kinesis API (AWS SDK)
The primitive the KPL wraps. It transmits exactly what you give it, with no aggregation, no retry policy and no metrics emitted on your behalf.
✓Kinesis Producer Library (KPL)
Although the Kinesis API built into the AWS SDK can be used for all of this, the Kinesis Producer Library (KPL) makes it easy to integrate all of this into your applications.
37 more questions in the app
Practise the full 45-question set with a timer, scoring and progress tracking.
