New 2022 Associate-Developer-Apache-Spark exam questions Welcome to download the newest PassLeaderVCE Associate-Developer-Apache-Spark PDF dumps (179 Q&As)
P.S. Free 2022 Databricks Certification Associate-Developer-Apache-Spark dumps are available on Google Drive shared by PassLeaderVCE
NEW QUESTION 68
Which of the following code blocks returns a 2-column DataFrame that shows the distinct values in column productId and the number of rows with that productId in DataFrame transactionsDf?
- A. transactionsDf.count("productId")
- B. transactionsDf.groupBy("productId").agg(col("value").count())
- C. transactionsDf.groupBy("productId").count()
- D. transactionsDf.count("productId").distinct()
- E. transactionsDf.groupBy("productId").select(count("value"))
Answer: C
Explanation:
Explanation
transactionsDf.groupBy("productId").count()
Correct. This code block first groups DataFrame transactionsDf by column productId and then counts the rows in each group.
transactionsDf.groupBy("productId").select(count("value"))
Incorrect. You cannot call select on a GroupedData object (the output of a groupBy) statement.
transactionsDf.count("productId")
No. DataFrame.count() does not take any arguments.
transactionsDf.count("productId").distinct()
Wrong. Since DataFrame.count() does not take any arguments, this option cannot be right.
transactionsDf.groupBy("productId").agg(col("value").count())
False. A Column object, as returned by col("value"), does not have a count() method. You can see all available methods for Column object linked in the Spark documentation below.
More info: pyspark.sql.DataFrame.count - PySpark 3.1.2 documentation, pyspark.sql.Column - PySpark
3.1.2 documentation
Static notebook | Dynamic notebook: See test 3
NEW QUESTION 69
The code block shown below should return an exact copy of DataFrame transactionsDf that does not include rows in which values in column storeId have the value 25. Choose the answer that correctly fills the blanks in the code block to accomplish this.
- A. transactionsDf.drop(transactionsDf.storeId==25)
- B. transactionsDf.remove(transactionsDf.storeId==25)
- C. transactionsDf.filter(transactionsDf.storeId==25)
- D. transactionsDf.where(transactionsDf.storeId!=25)
- E. transactionsDf.select(transactionsDf.storeId!=25)
Answer: D
Explanation:
Explanation
transactionsDf.where(transactionsDf.storeId!=25)
Correct. DataFrame.where() is an alias for the DataFrame.filter() method. Using this method, it is straightforward to filter out rows that do not have value 25 in column storeId.
transactionsDf.select(transactionsDf.storeId!=25)
Wrong. The select operator allows you to build DataFrames column-wise, but when using it as shown, it does not filter out rows.
transactionsDf.filter(transactionsDf.storeId==25)
Incorrect. Although the filter expression works for filtering rows, the == in the filtering condition is inappropriate. It should be != instead.
transactionsDf.drop(transactionsDf.storeId==25)
No. DataFrame.drop() is used to remove specific columns, but not rows, from the DataFrame.
transactionsDf.remove(transactionsDf.storeId==25)
False. There is no DataFrame.remove() operator in PySpark.
More info: pyspark.sql.DataFrame.where - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3
NEW QUESTION 70
Which of the following is a characteristic of the cluster manager?
- A. Each cluster manager works on a single partition of data.
- B. The cluster manager transforms jobs into DAGs.
- C. The cluster manager does not exist in standalone mode.
- D. The cluster manager receives input from the driver through the SparkContext.
- E. In client mode, the cluster manager runs on the edge node.
Answer: D
Explanation:
Explanation
The cluster manager receives input from the driver through the SparkContext.
Correct. In order for the driver to contact the cluster manager, the driver launches a SparkContext. The driver then asks the cluster manager for resources to launch executors.
In client mode, the cluster manager runs on the edge node.
No. In client mode, the cluster manager is independent of the edge node and runs in the cluster.
The cluster manager does not exist in standalone mode.
Wrong, the cluster manager exists even in standalone mode. Remember, standalone mode is an easy means to deploy Spark across a whole cluster, with some limitations. For example, in standalone mode, no other frameworks can run in parallel with Spark. The cluster manager is part of Spark in standalone deployments however and helps launch and maintain resources across the cluster.
The cluster manager transforms jobs into DAGs.
No, transforming jobs into DAGs is the task of the Spark driver.
Each cluster manager works on a single partition of data.
No. Cluster managers do not work on partitions directly. Their job is to coordinate cluster resources so that they can be requested by and allocated to Spark drivers.
More info: Introduction to Core Spark Concepts * BigData
NEW QUESTION 71
Which of the following code blocks returns a single row from DataFrame transactionsDf?
Full DataFrame transactionsDf:
1.+-------------+---------+-----+-------+---------+----+
2.|transactionId|predError|value|storeId|productId| f|
3.+-------------+---------+-----+-------+---------+----+
4.| 1| 3| 4| 25| 1|null|
5.| 2| 6| 7| 2| 2|null|
6.| 3| 3| null| 25| 3|null|
7.| 4| null| null| 3| 2|null|
8.| 5| null| null| null| 2|null|
9.| 6| 3| 2| 25| 2|null|
10.+-------------+---------+-----+-------+---------+----+
- A. transactionsDf.filter(col("storeId")==25).select("predError","storeId").distinct()
- B. transactionsDf.select("productId", "storeId").where("storeId == 2 OR storeId != 25")
- C. transactionsDf.where(col("storeId").between(3,25))
- D. transactionsDf.filter((col("storeId")!=25) | (col("productId")==2))
- E. transactionsDf.where(col("value").isNull()).select("productId", "storeId").distinct()
Answer: A
Explanation:
Explanation
Output of correct code block:
+---------+-------+
|predError|storeId|
+---------+-------+
| 3| 25|
+---------+-------+
This question is difficult because it requires you to understand different kinds of commands and operators. All answers are valid Spark syntax, but just one expression returns a single-row DataFrame.
For reference, here is what the incorrect answers return:
transactionsDf.filter((col("storeId")!=25) | (col("productId")==2)) returns
+-------------+---------+-----+-------+---------+----+
|transactionId|predError|value|storeId|productId| f|
+-------------+---------+-----+-------+---------+----+
| 2| 6| 7| 2| 2|null|
| 4| null| null| 3| 2|null|
| 5| null| null| null| 2|null|
| 6| 3| 2| 25| 2|null|
+-------------+---------+-----+-------+---------+----+
transactionsDf.where(col("storeId").between(3,25)) returns
+-------------+---------+-----+-------+---------+----+
|transactionId|predError|value|storeId|productId| f|
+-------------+---------+-----+-------+---------+----+
| 1| 3| 4| 25| 1|null|
| 3| 3| null| 25| 3|null|
| 4| null| null| 3| 2|null|
| 6| 3| 2| 25| 2|null|
+-------------+---------+-----+-------+---------+----+
transactionsDf.where(col("value").isNull()).select("productId", "storeId").distinct() returns
+---------+-------+
|productId|storeId|
+---------+-------+
| 3| 25|
| 2| 3|
| 2| null|
+---------+-------+
transactionsDf.select("productId", "storeId").where("storeId == 2 OR storeId != 25") returns
+---------+-------+
|productId|storeId|
+---------+-------+
| 2| 2|
| 2| 3|
+---------+-------+
Static notebook | Dynamic notebook: See test 2
NEW QUESTION 72
Which of the following describes Spark's way of managing memory?
- A. Storage memory is used for caching partitions derived from DataFrames.
- B. Spark uses a subset of the reserved system memory.
- C. Disabling serialization potentially greatly reduces the memory footprint of a Spark application.
- D. As a general rule for garbage collection, Spark performs better on many small objects than few big objects.
- E. Spark's memory usage can be divided into three categories: Execution, transaction, and storage.
Answer: A
Explanation:
Explanation
Spark's memory usage can be divided into three categories: Execution, transaction, and storage.
No, it is either execution or storage.
As a general rule for garbage collection, Spark performs better on many small objects than few big objects.
No, Spark's garbage collection runs faster on fewer big objects than many small objects.
Disabling serialization potentially greatly reduces the memory footprint of a Spark application.
The opposite is true - serialization reduces the memory footprint, but may impact performance in a negative way.
Spark uses a subset of the reserved system memory.
No, the reserved system memory is separate from Spark memory. Reserved memory stores Spark's internal objects.
More info: Tuning - Spark 3.1.2 Documentation, Spark Memory Management | Distributed Systems Architecture, Learning Spark, 2nd Edition, Chapter 7
NEW QUESTION 73
Which of the following describes a narrow transformation?
- A. A narrow transformation is an operation in which no data is exchanged across the cluster.
- B. A narrow transformation is an operation in which data is exchanged across the cluster.
- C. A narrow transformation is a process in which 32-bit float variables are cast to smaller float variables, like 16-bit or 8-bit float variables.
- D. A narrow transformation is a process in which data from multiple RDDs is used.
- E. narrow transformation is an operation in which data is exchanged across partitions.
Answer: A
Explanation:
Explanation
A narrow transformation is an operation in which no data is exchanged across the cluster.
Correct! In narrow transformations, no data is exchanged across the cluster, since these transformations do not require any data from outside of the partition they are applied on. Typical narrow transformations include filter, drop, and coalesce.
A narrow transformation is an operation in which data is exchanged across partitions.
No, that would be one definition of a wide transformation, but not of a narrow transformation. Wide transformations typically cause a shuffle, in which data is exchanged across partitions, executors, and the cluster.
A narrow transformation is an operation in which data is exchanged across the cluster.
No, see explanation just above this one.
A narrow transformation is a process in which 32-bit float variables are cast to smaller float variables, like
16-bit or 8-bit float variables.
No, type conversion has nothing to do with narrow transformations in Spark.
A narrow transformation is a process in which data from multiple RDDs is used.
No. A resilient distributed dataset (RDD) can be described as a collection of partitions. In a narrow transformation, no data is exchanged between partitions. Thus, no data is exchanged between RDDs.
One could say though that a narrow transformation and, in fact, any transformation results in a new RDD being created. This is because a transformation results in a change to an existing RDD (RDDs are the foundation of other Spark data structures, like DataFrames). But, since RDDs are immutable, a new RDD needs to be created to reflect the change caused by the transformation.
More info: Spark Transformation and Action: A Deep Dive | by Misbah Uddin | CodeX | Medium
NEW QUESTION 74
Which of the following is not a feature of Adaptive Query Execution?
- A. Reroute a query in case of an executor failure.
- B. Coalesce partitions to accelerate data processing.
- C. Replace a sort merge join with a broadcast join, where appropriate.
- D. Collect runtime statistics during query execution.
- E. Split skewed partitions into smaller partitions to avoid differences in partition processing time.
Answer: A
Explanation:
Explanation
Reroute a query in case of an executor failure.
Correct. Although this feature exists in Spark, it is not a feature of Adaptive Query Execution. The cluster manager keeps track of executors and will work together with the driver to launch an executor and assign the workload of the failed executor to it (see also link below).
Replace a sort merge join with a broadcast join, where appropriate.
No, this is a feature of Adaptive Query Execution.
Coalesce partitions to accelerate data processing.
Wrong, Adaptive Query Execution does this.
Collect runtime statistics during query execution.
Incorrect, Adaptive Query Execution (AQE) collects these statistics to adjust query plans. This feedback loop is an essential part of accelerating queries via AQE.
Split skewed partitions into smaller partitions to avoid differences in partition processing time.
No, this is indeed a feature of Adaptive Query Execution. Find more information in the Databricks blog post linked below.
More info: Learning Spark, 2nd Edition, Chapter 12, On which way does RDD of spark finish fault-tolerance?
- Stack Overflow, How to Speed up SQL Queries with Adaptive Query Execution
NEW QUESTION 75
The code block displayed below contains an error. The code block should count the number of rows that have a predError of either 3 or 6. Find the error.
Code block:
transactionsDf.filter(col('predError').in([3, 6])).count()
- A. Numbers 3 and 6 need to be passed as string variables.
- B. The number of rows cannot be determined with the count() operator.
- C. Instead of filter, the select method should be used.
- D. The method used on column predError is incorrect.
- E. Instead of a list, the values need to be passed as single arguments to the in operator.
Answer: D
Explanation:
Explanation
Correct code block:
transactionsDf.filter(col('predError').isin([3, 6])).count()
The isin method is the correct one to use here - the in method does not exist for the Column object.
More info: pyspark.sql.Column.isin - PySpark 3.1.2 documentation
NEW QUESTION 76
Which of the following code blocks generally causes a great amount of network traffic?
- A. DataFrame.coalesce()
- B. DataFrame.rdd.map()
- C. DataFrame.collect()
- D. DataFrame.count()
- E. DataFrame.select()
Answer: C
Explanation:
Explanation
DataFrame.collect() sends all data in a DataFrame from executors to the driver, so this generally causes a great amount of network traffic in comparison to the other options listed.
DataFrame.coalesce() just reduces the number of partitions and generally aims to reduce network traffic in comparison to a full shuffle.
DataFrame.select() is evaluated lazily and, unless followed by an action, does not cause significant network traffic.
DataFrame.rdd.map() is evaluated lazily, it does therefore not cause great amounts of network traffic.
DataFrame.count() is an action. While it does cause some network traffic, for the same DataFrame, collecting all data in the driver would generally be considered to cause a greater amount of network traffic.
NEW QUESTION 77
Which of the following code blocks performs a join in which the small DataFrame transactionsDf is sent to all executors where it is joined with DataFrame itemsDf on columns storeId and itemId, respectively?
- A. itemsDf.join(transactionsDf, broadcast(itemsDf.itemId == transactionsDf.storeId))
- B. itemsDf.join(transactionsDf, itemsDf.itemId == transactionsDf.storeId, "broadcast")
- C. itemsDf.join(broadcast(transactionsDf), itemsDf.itemId == transactionsDf.storeId)
- D. itemsDf.join(transactionsDf, itemsDf.itemId == transactionsDf.storeId, "right_outer")
- E. itemsDf.merge(transactionsDf, "itemsDf.itemId == transactionsDf.storeId", "broadcast")
Answer: C
Explanation:
Explanation
The issue with all answers that have "broadcast" as very last argument is that "broadcast" is not a valid join type. While the entry with "right_outer" is a valid statement, it is not a broadcast join. The item where broadcast() is wrapped around the equality condition is not valid code in Spark. broadcast() needs to be wrapped around the name of the small DataFrame that should be broadcast.
More info: Learning Spark, 2nd Edition, Chapter 7
Static notebook | Dynamic notebook: See test 1
tion and explanation?
NEW QUESTION 78
The code block displayed below contains an error. The code block should write DataFrame transactionsDf as a parquet file to location filePath after partitioning it on column storeId. Find the error.
Code block:
transactionsDf.write.partitionOn("storeId").parquet(filePath)
- A. The partitioning column as well as the file path should be passed to the write() method of DataFrame transactionsDf directly and not as appended commands as in the code block.
- B. No method partitionOn() exists for the DataFrame class, partitionBy() should be used instead.
- C. The partitionOn method should be called before the write method.
- D. The operator should use the mode() option to configure the DataFrameWriter so that it replaces any existing files at location filePath.
- E. Column storeId should be wrapped in a col() operator.
Answer: B
Explanation:
Explanation
No method partitionOn() exists for the DataFrame class, partitionBy() should be used instead.
Correct! Find out more about partitionBy() in the documentation (linked below).
The operator should use the mode() option to configure the DataFrameWriter so that it replaces any existing files at location filePath.
No. There is no information about whether files should be overwritten in the question.
The partitioning column as well as the file path should be passed to the write() method of DataFrame transactionsDf directly and not as appended commands as in the code block.
Incorrect. To write a DataFrame to disk, you need to work with a DataFrameWriter object which you get access to through the DataFrame.writer property - no parentheses involved.
Column storeId should be wrapped in a col() operator.
No, this is not necessary - the problem is in the partitionOn command (see above).
The partitionOn method should be called before the write method.
Wrong. First of all partitionOn is not a valid method of DataFrame. However, even assuming partitionOn would be replaced by partitionBy (which is a valid method), this method is a method of DataFrameWriter and not of DataFrame. So, you would always have to first call DataFrame.write to get access to the DataFrameWriter object and afterwards call partitionBy.
More info: pyspark.sql.DataFrameWriter.partitionBy - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3
NEW QUESTION 79
Which of the following describes a difference between Spark's cluster and client execution modes?
- A. In cluster mode, executor processes run on worker nodes, while they run on gateway nodes in client mode.
- B. In cluster mode, the Spark driver is not co-located with the cluster manager, while it is co-located in client mode.
- C. In cluster mode, a gateway machine hosts the driver, while it is co-located with the executor in client mode.
- D. In cluster mode, the driver resides on a worker node, while it resides on an edge node in client mode.
- E. In cluster mode, the cluster manager resides on a worker node, while it resides on an edge node in client mode.
Answer: D
Explanation:
Explanation
In cluster mode, the driver resides on a worker node, while it resides on an edge node in client mode.
Correct. The idea of Spark's client mode is that workloads can be executed from an edge node, also known as gateway machine, from outside the cluster. The most common way to execute Spark however is in cluster mode, where the driver resides on a worker node.
In practice, in client mode, there are tight constraints about the data transfer speed relative to the data transfer speed between worker nodes in the cluster. Also, any job in that is executed in client mode will fail if the edge node fails. For these reasons, client mode is usually not used in a production environment.
In cluster mode, the cluster manager resides on a worker node, while it resides on an edge node in client execution mode.
No. In both execution modes, the cluster manager may reside on a worker node, but it does not reside on an edge node in client mode.
In cluster mode, executor processes run on worker nodes, while they run on gateway nodes in client mode.
This is incorrect. Only the driver runs on gateway nodes (also known as "edge nodes") in client mode, but not the executor processes.
In cluster mode, the Spark driver is not co-located with the cluster manager, while it is co-located in client mode.
No, in client mode, the Spark driver is not co-located with the driver. The whole point of client mode is that the driver is outside the cluster and not associated with the resource that manages the cluster (the machine that runs the cluster manager).
In cluster mode, a gateway machine hosts the driver, while it is co-located with the executor in client mode.
No, it is exactly the opposite: There are no gateway machines in cluster mode, but in client mode, they host the driver.
NEW QUESTION 80
Which of the following options describes the responsibility of the executors in Spark?
- A. The executors accept tasks from the driver, execute those tasks, and return results to the driver.
- B. The executors accept jobs from the driver, plan those jobs, and return results to the cluster manager.
- C. The executors accept tasks from the cluster manager, execute those tasks, and return results to the driver.
- D. The executors accept jobs from the driver, analyze those jobs, and return results to the driver.
- E. The executors accept tasks from the driver, execute those tasks, and return results to the cluster manager.
Answer: A
Explanation:
Explanation
More info: Running Spark: an overview of Spark's runtime architecture - Manning (https://bit.ly/2RPmJn9)
NEW QUESTION 81
Which of the following code blocks reads JSON file imports.json into a DataFrame?
- A. spark.read.json("/FileStore/imports.json")
- B. spark.read().mode("json").path("/FileStore/imports.json")
- C. spark.read.format("json").path("/FileStore/imports.json")
- D. spark.read("json", "/FileStore/imports.json")
- E. spark.read().json("/FileStore/imports.json")
Answer: A
Explanation:
Explanation
Static notebook | Dynamic notebook: See test 1
(https://flrs.github.io/spark_practice_tests_code/#1/25.html ,
https://bit.ly/sparkpracticeexams_import_instructions)
NEW QUESTION 82
Which of the following statements about Spark's execution hierarchy is correct?
- A. In Spark's execution hierarchy, a job may reach over multiple stage boundaries.
- B. In Spark's execution hierarchy, a stage comprises multiple jobs.
- C. In Spark's execution hierarchy, tasks are one layer above slots.
- D. In Spark's execution hierarchy, executors are the smallest unit.
- E. In Spark's execution hierarchy, manifests are one layer above jobs.
Answer: A
Explanation:
Explanation
In Spark's execution hierarchy, a job may reach over multiple stage boundaries.
Correct. A job is a sequence of stages, and thus may reach over multiple stage boundaries.
In Spark's execution hierarchy, tasks are one layer above slots.
Incorrect. Slots are not a part of the execution hierarchy. Tasks are the lowest layer.
In Spark's execution hierarchy, a stage comprises multiple jobs.
No. It is the other way around - a job consists of one or multiple stages.
In Spark's execution hierarchy, executors are the smallest unit.
False. Executors are not a part of the execution hierarchy. Tasks are the smallest unit!
In Spark's execution hierarchy, manifests are one layer above jobs.
Wrong. Manifests are not a part of the Spark ecosystem.
NEW QUESTION 83
Which of the following code blocks returns a DataFrame that has all columns of DataFrame transactionsDf and an additional column predErrorSquared which is the squared value of column predError in DataFrame transactionsDf?
- A. transactionsDf.withColumnRenamed("predErrorSquared", pow(predError, 2))
- B. transactionsDf.withColumn("predErrorSquared", pow(col("predError"), lit(2)))
- C. transactionsDf.withColumn("predError", pow(col("predErrorSquared"), 2))
- D. transactionsDf.withColumn("predErrorSquared", pow(predError, lit(2)))
- E. transactionsDf.withColumn("predErrorSquared", "predError"**2)
Answer: B
Explanation:
Explanation
While only one of these code blocks works, the DataFrame API is pretty flexible when it comes to accepting columns into the pow() method. The following code blocks would also work:
transactionsDf.withColumn("predErrorSquared", pow("predError", 2))
transactionsDf.withColumn("predErrorSquared", pow("predError", lit(2))) Static notebook | Dynamic notebook: See test 1 (https://flrs.github.io/spark_practice_tests_code/#1/26.html ,
https://bit.ly/sparkpracticeexams_import_instructions)
NEW QUESTION 84
Which of the following describes characteristics of the Spark driver?
- A. If set in the Spark configuration, Spark scales the Spark driver horizontally to improve parallel processing performance.
- B. In a non-interactive Spark application, the Spark driver automatically creates the SparkSession object.
- C. The Spark driver requests the transformation of operations into DAG computations from the worker nodes.
- D. The Spark driver processes partitions in an optimized, distributed fashion.
- E. The Spark driver's responsibility includes scheduling queries for execution on worker nodes.
Answer: B
Explanation:
Explanation
The Spark driver requests the transformation of operations into DAG computations from the worker nodes.
No, the Spark driver transforms operations into DAG computations itself.
If set in the Spark configuration, Spark scales the Spark driver horizontally to improve parallel processing performance.
No. There is always a single driver per application, but one or more executors.
The Spark driver processes partitions in an optimized, distributed fashion.
No, this is what executors do.
In a non-interactive Spark application, the Spark driver automatically creates the SparkSession object.
Wrong. In a non-interactive Spark application, you need to create the SparkSession object. In an interactive Spark shell, the Spark driver instantiates the object for you.
NEW QUESTION 85
......
Associate-Developer-Apache-Spark exam questions from PassLeaderVCE dumps: https://www.passleadervce.com/Databricks-Certification/reliable-Associate-Developer-Apache-Spark-exam-learning-guide.html (179 Q&As)
Free 2022 Databricks Certification Associate-Developer-Apache-Spark dumps are available on Google Drive shared by PassLeaderVCE: https://drive.google.com/open?id=1xa7LTOQK-I8kJn-Q8QsiZZgbSxS64Kvm