PySpark Coding Interview Questions & Answers

Q: Find employees whose salary is greater than 55,000

Expected output Priya earns exactly 55,000, so > 55000 excludes her; use >= to include it. Practise this: Filter rows with a condition (runs in the browser, with test cases).

pyspark·

2d agoInterview Question

Q.1

Scenario: for each customer, show the month-over-month change and a running total.

Medium

Given monthly spend per customer, add the previous month's amount, the percentage change and the cumulative spend so far.

from pyspark.sql import SparkSession, functions as F
from pyspark.sql.window import Window
 
spark = SparkSession.builder.appName("MonthOverMonth").getOrCreate()
 
sales = spark.createDataFrame([
    ("C1", "2026-01", 1200.0), ("C1", "2026-02", 1500.0), ("C1", "2026-03", 900.0),
    ("C2", "2026-01", 400.0), ("C2", "2026-02", 400.0), ("C2", "2026-03", 650.0),
], ["customer_id", "month", "amount"])
 
w = Window.partitionBy("customer_id").orderBy("month")
running = w.rowsBetween(Window.unboundedPreceding, Window.currentRow)
 
result = (sales
    .withColumn("prev_amount", F.lag("amount").over(w))
    .withColumn("change_pct", F.round((F.col("amount") - F.col("prev_amount")) / F.col("prev_amount") * 100, 1))
    .withColumn("running_total", F.sum("amount").over(running)))
 
result.orderBy("customer_id", "month").show()
▶ Run to see output

Expected output

+-----------+-------+------+-----------+----------+-------------+
|customer_id|  month|amount|prev_amount|change_pct|running_total|
+-----------+-------+------+-----------+----------+-------------+
|         C1|2026-01|1200.0|       NULL|      NULL|       1200.0|
|         C1|2026-02|1500.0|     1200.0|      25.0|       2700.0|
|         C1|2026-03| 900.0|     1500.0|     -40.0|       3600.0|
|         C2|2026-01| 400.0|       NULL|      NULL|        400.0|
|         C2|2026-02| 400.0|      400.0|       0.0|        800.0|
|         C2|2026-03| 650.0|      400.0|      62.5|       1450.0|
+-----------+-------+------+-----------+----------+-------------+
▶ Run to see output

Points interviewers look for

The first month has no previous month, so lag returns NULL and so does the change: do not turn it into 0%.
An explicit rowsBetween(unboundedPreceding, currentRow) frame. With only orderBy, the default frame is RANGE-based, so rows with the same month would all get the same running total.
A missing month (no row for February) makes lag compare March with January; fill the calendar first if that matters.

Practise this: Month-over-month growth percentage (runs in the browser, with test cases).

pyspark·

2d agoInterview Question

Q.2

Scenario: find the top 2 salaries in each department, keeping employees who tie.

Medium

Return every employee whose salary is among the two highest salary levels of their department. Two people with the same salary must both appear.

from pyspark.sql import SparkSession, functions as F
from pyspark.sql.window import Window
 
spark = SparkSession.builder.appName("TopNPerGroup").getOrCreate()
 
emp = spark.createDataFrame([
    ("Anil", "IT", 90000), ("Bina", "IT", 85000), ("Chetan", "IT", 85000), ("Divya", "IT", 70000),
    ("Esha", "HR", 60000), ("Farid", "HR", 55000), ("Gita", "HR", 50000),
    ("Hari", "Sales", 65000),
], ["name", "department", "salary"])
 
w = Window.partitionBy("department").orderBy(F.col("salary").desc())
 
top2 = emp.withColumn("rnk", F.dense_rank().over(w)).filter("rnk &lt;= 2")
top2.orderBy("department", "rnk", "name").show()
▶ Run to see output

Expected output

+------+----------+------+---+
|  name|department|salary|rnk|
+------+----------+------+---+
|  Esha|        HR| 60000|  1|
| Farid|        HR| 55000|  2|
|  Anil|        IT| 90000|  1|
|  Bina|        IT| 85000|  2|
|Chetan|        IT| 85000|  2|
|  Hari|     Sales| 65000|  1|
+------+----------+------+---+
▶ Run to see output

Choosing the ranking function

Function	IT ranks for 90k, 85k, 85k, 70k	Use when
`row_number`	1, 2, 3, 4	Exactly N rows per group; ties broken arbitrarily
`rank`	1, 2, 2, 4	Ties share a rank and the next rank is skipped
`dense_rank`	1, 2, 2, 3	Top N distinct values; ties kept, no gaps

Sales has one employee, so it returns one row: “top 2” never invents rows.

Practise this: Top 3 highest-paid employees in each department (runs in the browser, with test cases).

pyspark·

2d agoInterview Question

Q.3

Scenario: a table receives several updates per customer. How do you keep only the latest record per key?

Medium

A CDC feed or a daily extract often carries more than one version of the same customer. Keep the newest version of each key, deterministically, even when two updates share a timestamp.

from pyspark.sql import SparkSession, functions as F
from pyspark.sql.window import Window
 
spark = SparkSession.builder.appName("LatestPerKey").getOrCreate()
 
updates = spark.createDataFrame([
    (101, "Asha", "Pune", "2026-01-05 10:00:00", 1),
    (101, "Asha", "Mumbai", "2026-03-12 09:30:00", 4),
    (102, "Ravi", "Delhi", "2026-02-01 08:00:00", 2),
    (103, "Meera", "Chennai", "2026-01-20 18:45:00", 3),
    (103, "Meera", "Bengaluru", "2026-02-28 11:15:00", 5),
    (103, "Meera", "Hyderabad", "2026-02-28 11:15:00", 6),
], ["customer_id", "name", "city", "updated_at", "ingest_seq"])
 
updates = updates.withColumn("updated_at", F.to_timestamp("updated_at"))
 
# Newest first; ingest_seq breaks a tie between two updates at the same timestamp
w = Window.partitionBy("customer_id").orderBy(F.col("updated_at").desc(), F.col("ingest_seq").desc())
 
latest = (updates
    .withColumn("rn", F.row_number().over(w))
    .filter("rn = 1")
    .drop("rn"))
 
latest.orderBy("customer_id").show(truncate=False)
▶ Run to see output

Expected output

+-----------+-----+---------+-------------------+----------+
|customer_id|name |city     |updated_at         |ingest_seq|
+-----------+-----+---------+-------------------+----------+
|101        |Asha |Mumbai   |2026-03-12 09:30:00|4         |
|102        |Ravi |Delhi    |2026-02-01 08:00:00|2         |
|103        |Meera|Hyderabad|2026-02-28 11:15:00|6         |
+-----------+-----+---------+-------------------+----------+
▶ Run to see output

What the interviewer is checking

row_number(), not rank(): rank would keep both tied rows for customer 103.
A deterministic tie-breaker (an ingestion sequence or offset). Without it, which tied row survives can change between runs.
Why not dropDuplicates(["customer_id"]): it keeps an arbitrary row, not the latest.
On Delta Lake, the same window result feeds a MERGE so the target table holds one current row per key.

Practise this: Deduplicate: keep the latest record per customer (runs in the browser, with test cases).

Vishal Taneja·

Sep 13, 2026Interview Question

Q.4

You receive employee data in Spark. How would you calculate a 15% bonus column?

Easy

from pyspark.sql import SparkSession
from pyspark.sql.functions import col
 
# Create SparkSession
spark = SparkSession.builder \
    .appName("EmployeeBonus") \
    .master("local[*]") \
    .getOrCreate()
 
# Sample data
data = [
    (1, "Amit", 50000),
    (2, "Rahul", 60000)
]
 
# Column names
columns = ["id", "name", "salary"]
 
# Create DataFrame
df = spark.createDataFrame(data, columns)
 
# Display original data
print("Original Data:")
df.show()
 
# Calculate 15% bonus
result = df.withColumn(
    "bonus",
    col("salary") * 0.15
)
 
# Display result
print("Employee Bonus:")
result.show()
 
# Stop Spark
spark.stop()
▶ Run to see output

Expected output

+---+-----+------+------+
| id| name|salary| bonus|
+---+-----+------+------+
|  1| Amit| 50000|7500.0|
|  2|Rahul| 60000|9000.0|
+---+-----+------+------+
▶ Run to see output

Practise this: Add a bonus column with withColumn (runs in the browser, with test cases).

Vishal Taneja·

Sep 13, 2026Interview Question

Q.5

Find employees whose salary is greater than 55,000

Easy

from pyspark.sql import SparkSession
from pyspark.sql.functions import col
# 1. Create SparkSession
spark = SparkSession.builder \
    .appName("EmployeeFiltering") \
    .master("local[*]") \
    .getOrCreate()
# 2. Create sample employee data
data = [
    (1, "Amit", 50000),
    (2, "Rahul", 60000),
    (3, "John", 75000),
    (4, "Priya", 55000),
    (5, "Neha", 90000)
]
 
# 3. Define columns
columns = ["employee_id", "name", "salary"]
# 4. Create DataFrame
df = spark.createDataFrame(data, columns)
# 5. Display original DataFrame
print("Original Data:")
df.show()
# 6. Filter employees whose salary is greater than 55,000
result = df.filter(col("salary") &gt; 55000)
# 7. Display filtered result
print("Employees with salary greater than 55,000:")
result.show()
# 8. Stop Spark
spark.stop()
▶ Run to see output

Expected output

+-----------+-----+------+
|employee_id| name|salary|
+-----------+-----+------+
|          2|Rahul| 60000|
|          3| John| 75000|
|          5| Neha| 90000|
+-----------+-----+------+
▶ Run to see output

Priya earns exactly 55,000, so > 55000 excludes her; use >= to include it.

Practise this: Filter rows with a condition (runs in the browser, with test cases).

Vishal Taneja·

Sep 4, 2026Interview Question

Q.6

A bank wants to identify customers whose salary has significantly declined over the last 3 months.

Hard

Business Requirement

A bank wants to identify customers whose salary has significantly declined over the last 3 months.

Risk Criteria

Customer should have monthly salary credits.
Compare the latest month's salary with the average salary of the previous 3 months.
If the salary has dropped by more than 40%, flag the customer.
Ignore customers having fewer than 4 salary credits.

Input DataFrame

`account_transactions`

account_id	txn_date	txn_type	amount
1001	2024-01-05	CREDIT	100000
1001	2024-02-05	CREDIT	105000
1001	2024-03-05	CREDIT	98000
1001	2024-04-05	CREDIT	50000
1002	2024-01-05	CREDIT	70000
1002	2024-02-05	CREDIT	72000
1002	2024-03-05	CREDIT	71000
1002	2024-04-05	CREDIT	70000

Expected Output

account_id	avg_previous_3_months	current_salary	drop_percentage	risk_flag
1001	101000.00	50000	50.50%	FLAG
1002	71000.00	70000	1.41%	NOT_FLAG

from pyspark.sql import SparkSession
from pyspark.sql.functions import (
    col,
    avg,
    count,
    round,
    when,
    row_number
)
from pyspark.sql.window import Window
# ============================================================
# 1. Create Spark Session
# ============================================================
spark = SparkSession.builder \
    .appName("SalaryDropAnalysis") \
    .master("local[*]") \
    .getOrCreate()
# ============================================================
# 2. Create Source Data
# ============================================================
data = [
    (1001, "2024-01-05", "CREDIT", 100000),
    (1001, "2024-02-05", "CREDIT", 105000),
    (1001, "2024-03-05", "CREDIT", 98000),
    (1001, "2024-04-05", "CREDIT", 50000),
    (1002, "2024-01-05", "CREDIT", 70000),
    (1002, "2024-02-05", "CREDIT", 72000),
    (1002, "2024-03-05", "CREDIT", 71000),
    (1002, "2024-04-05", "CREDIT", 70000),
 
    # Only 3 credits - should be ignored
    (1003, "2024-02-05", "CREDIT", 80000),
    (1003, "2024-03-05", "CREDIT", 82000),
    (1003, "2024-04-05", "CREDIT", 40000)
]
 
columns = [
    "account_id",
    "txn_date",
    "txn_type",
    "amount"
]
 
# ============================================================
# 3. Create DataFrame
# ============================================================
df = spark.createDataFrame(data, columns)
print("SOURCE DATA")
df.show()
# ============================================================
# 4. Convert transaction date to date
# ============================================================
df = df.withColumn(
    "txn_date",
    col("txn_date").cast("date")
)
 
# ============================================================
# 5. Keep only salary CREDIT transactions
# ============================================================
salary_df = df.filter(
    col("txn_type") == "CREDIT"
)
print("SALARY CREDIT DATA")
salary_df.show()
 
# ============================================================
# 6. Check customers having at least 4 salary credits
# ============================================================
valid_customers = salary_df \
    .groupBy("account_id") \
    .agg(
        count("*").alias("credit_count")
    ) \
    .filter(
        col("credit_count") &gt;= 4
    )
 
print("CUSTOMERS WITH AT LEAST 4 CREDITS")
valid_customers.show()
# ============================================================
# 7. Keep only valid customers
# ============================================================
salary_df = salary_df.join(
    valid_customers.select("account_id"),
    on="account_id",
    how="inner"
)
 
# ============================================================
# 8. Create Window
#
# Latest transaction gets rn = 1
# Previous transactions get rn = 2, 3, 4
# ============================================================
window_spec = Window \
    .partitionBy("account_id") \
    .orderBy(col("txn_date").desc())
# ============================================================
# 9. Assign row number
# ============================================================
salary_df = salary_df.withColumn(
    "rn",
    row_number().over(window_spec)
)
print("DATA WITH ROW NUMBER")
salary_df.show()
# ============================================================
# 10. Get latest salary
# ============================================================
current_salary_df = salary_df \
    .filter(col("rn") == 1) \
    .select(
        "account_id",
        col("amount").alias("current_salary")
    )
print("CURRENT SALARY")
current_salary_df.show()
# ============================================================
# 11. Get previous 3 months average salary
# ============================================================
previous_salary_df = salary_df \
    .filter(col("rn").between(2, 4)) \
    .groupBy("account_id") \
    .agg(
        avg("amount").alias("avg_previous_3_months")
    )
print("AVERAGE OF PREVIOUS 3 MONTHS")
previous_salary_df.show()
# ============================================================
# 12. Join current salary and previous salary
# ============================================================
 
result_df = current_salary_df.join(
    previous_salary_df,
    on="account_id",
    how="inner"
)
 
# ============================================================
# 13. Calculate salary drop percentage
#
# Drop % =
# ((Previous Average - Current Salary)
#  / Previous Average) * 100
# ============================================================
 
result_df = result_df.withColumn(
    "drop_percentage",
    round(
        (
            (
                col("avg_previous_3_months")
                - col("current_salary")
            )
            / col("avg_previous_3_months")
        ) * 100,
        2
    )
)
 
# ============================================================
# 14. Apply Risk Rule
#
# Drop &gt; 40% --&gt; FLAG
# Otherwise --&gt; NOT_FLAG
# ============================================================
 
result_df = result_df.withColumn(
    "risk_flag",
    when(
        col("drop_percentage") &gt; 40,
        "FLAG"
    ).otherwise("NOT_FLAG")
)
 
# ============================================================
# 15. Final Result
# ============================================================
 
final_df = result_df.select(
    "account_id",
    "avg_previous_3_months",
    "current_salary",
    "drop_percentage",
    "risk_flag"
).orderBy("account_id")
 
print("FINAL RESULT")
 
final_df.show(truncate=False)
 
# ============================================================
# 16. Stop Spark
# ============================================================
 
spark.stop()
▶ Run to see output

Practise this: Detect significant salary declines (runs in the browser, with test cases).

pyspark·

Sep 4, 2026Interview Question

Q.7

Given a DataFrame containing flight/train routes and fares, find the cheapest fare between a source and destination, including the possibility of travelling through an intermediate city?

Medium

Input DataFrame

Suppose we have this route data:

source	destination	fare
Delhi	Mumbai	5000
Delhi	Pune	3000
Pune	Mumbai	1500
Delhi	Bangalore	4000
Bangalore	Mumbai	2500
Delhi	Chennai	4500
Chennai	Mumbai	3000

We want to find the cheapest way to travel:

Delhi → Mumbai

Possible routes

Route	Calculation	Total Fare
Delhi → Mumbai	5000	5000
Delhi → Pune → Mumbai	3000 + 1500	4500
Delhi → Bangalore → Mumbai	4000 + 2500	6500
Delhi → Chennai → Mumbai	4500 + 3000	7500

Therefore:

Cheapest fare = ₹4,500

Route = Delhi → Pune → Mumbai

from pyspark.sql import SparkSession, functions as F
 
spark = SparkSession.builder.appName("CheapestFare").getOrCreate()
 
routes = spark.createDataFrame([
    ("Delhi", "Mumbai", 5000), ("Delhi", "Pune", 3000), ("Pune", "Mumbai", 1500),
    ("Delhi", "Bangalore", 4000), ("Bangalore", "Mumbai", 2500),
    ("Delhi", "Chennai", 4500), ("Chennai", "Mumbai", 3000),
], ["source", "destination", "fare"])
 
src, dst = "Delhi", "Mumbai"
 
# Direct routes
direct = (routes
    .filter((F.col("source") == src) &amp; (F.col("destination") == dst))
    .select(F.concat_ws(" -&gt; ", "source", "destination").alias("route"),
            F.col("fare").alias("total_fare")))
 
# One-stop routes: self join, first leg's destination = second leg's source
a, b = routes.alias("a"), routes.alias("b")
one_stop = (a.join(b, F.col("a.destination") == F.col("b.source"))
    .filter((F.col("a.source") == src) &amp; (F.col("b.destination") == dst))
    .select(F.concat_ws(" -&gt; ", "a.source", "a.destination", "b.destination").alias("route"),
            (F.col("a.fare") + F.col("b.fare")).alias("total_fare")))
 
direct.unionByName(one_stop).orderBy("total_fare").limit(1).show(truncate=False)
▶ Run to see output

Expected output

+-----------------------+----------+
|route                  |total_fare|
+-----------------------+----------+
|Delhi -&gt; Pune -&gt; Mumbai|4500      |
+-----------------------+----------+
▶ Run to see output

Why include the direct route?

The question asks for the cheapest fare including one stop, so a direct flight must compete too. With only the self join, a cheaper direct fare is never considered: change Delhi → Mumbai to 4000 and the one-stop-only version still answers 4500, while this one answers 4000 (Delhi -> Mumbai).

How to explain it: “I build the candidates two ways: direct routes by filtering, and one-stop routes by self-joining the table on destination = source. I union them, sort by total fare and take the first row. For more than one stop this becomes a graph problem (shortest path), not a fixed number of joins.”

Remember the logic

Direct routes ∪ (self join → add fares) → sort ascending → limit(1)

pyspark·

Sep 4, 2026Interview Question

Q.8

Given a sales DataFrame containing product_id, unit_price, and quantity, write PySpark code to calculate the total amount for each product and the overall total sales amount.

Easy

1. Input DataFrame

Think of the DataFrame like this table:

product_id	unit_price	quantity
101	100.0	2
102	200.0	3
103	50.0	4
104	150.0	2

For each row:

Total Amount = unit_price × quantity

🐍 script.py 32 lines

from pyspark.sql import SparkSession

from pyspark.sql.functions import col, sum

# Create Spark Session

spark = SparkSession.builder \

.appName("SalesCalculation") \

.getOrCreate()

# Sample sales data

data = [

(101, 100.0, 2),

(102, 200.0, 3),

(103, 50.0, 4),

(104, 150.0, 2)

]

columns = ["product_id", "unit_price", "quantity"]

# Create DataFrame

sales_df = spark.createDataFrame(data, columns)

print("Input Data:")

sales_df.show()

# Calculate total amount for each row

sales_with_amount = sales_df.withColumn(

"total_amount",

col("unit_price") * col("quantity")

)

print("Sales with Total Amount:")

sales_with_amount.show()

# Calculate overall total sales

total_sales = sales_with_amount.agg(

sum("total_amount").alias("total_sales_amount")

)

print("Overall Total Sales:")

total_sales.show()

spark.stop()

▶ Run to see output

✦ AI Explanation

Expected output

+----------+----------+--------+------------+
|product_id|unit_price|quantity|total_amount|
+----------+----------+--------+------------+
|       101|     100.0|       2|       200.0|
|       102|     200.0|       3|       600.0|
|       103|      50.0|       4|       200.0|
|       104|     150.0|       2|       300.0|
+----------+----------+--------+------------+
 
+------------------+
|total_sales_amount|
+------------------+
|            1300.0|
+------------------+
▶ Run to see output

Vishal Taneja·

Apr 20, 2026Interview Question

Q.9

Find the number of transactions that occurred for each product. Output the product name along with the transaction count and sort by product_id

Hard

You are given two datasets:

1. Inventory Table

product_id

product_name

2. Transactions Table

transaction_id

time

product_id

Output the product name along with the transaction count and sort by product_id.

Handle duplicate transactions (keep latest)

Count the number of transactions per product

Exclude products with no transactions

Sort the result by product_id

inventory_data = [
    (1, 'strawberry'),
    (2, 'apple_fuji'),
    (3, 'orange'),
    (4, 'clementines'),
    (5, 'blood_orange'),
    (6, 'banana')  # no transactions (to test filtering)
]
 
inv_cols = ["product_id", "product_name"]
 
transaction_data = [
    (153, '2016-01-06 08:57:52', 1),
    (91,  '2016-01-07 12:17:27', 1),
    (31,  '2016-01-05 13:19:25', 1),
    (24,  '2016-01-03 10:47:44', 3),
    (4,   '2016-01-06 17:57:42', 3),
    (163, '2016-01-03 10:11:22', 3),
    (92,  '2016-01-08 12:03:20', 2),
    (32,  '2016-01-04 19:37:14', 4),
    (253, '2016-01-06 14:15:20', 5),
    (118, '2016-01-06 14:27:33', 5),
    (118, '2016-01-07 14:27:33', 5),  # duplicate (keep latest)
    (300, '2016-01-02 09:00:00', 7)   # product not in inventory (to test join)
]
 
trans_cols = ["transaction_id", "time", "product_id"]
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, row_number, count
from pyspark.sql.window import Window
 
spark = SparkSession.builder.appName("InterviewSolution").getOrCreate()
 
# DataFrames
inv_df = spark.createDataFrame(inventory_data, ["product_id", "product_name"])
trans_df = spark.createDataFrame(transaction_data, ["transaction_id", "time", "product_id"])
 
# Step 1: Cast time to timestamp
trans_df = trans_df.withColumn("time", col("time").cast("timestamp"))
 
# Step 2: Deduplicate using window
window_spec = Window.partitionBy("transaction_id").orderBy(col("time").desc())
 
dedup_df = trans_df.withColumn("rn", row_number().over(window_spec)) \
                   .filter(col("rn") == 1) \
                   .drop("rn")
 
# Step 3: Inner join
joined_df = dedup_df.join(inv_df, on="product_id", how="inner")
 
# Step 4: Aggregate
result_df = joined_df.groupBy("product_id", "product_name") \
                    .agg(count("*").alias("transaction_count"))
 
# Step 5: Sort
final_df = result_df.orderBy("product_id")
 
final_df.show()
▶ Run to see output

Expected output

+----------+------------+-----------------+
|product_id|product_name|transaction_count|
+----------+------------+-----------------+
|         1|  strawberry|                3|
|         2|  apple_fuji|                1|
|         3|      orange|                3|
|         4| clementines|                1|
|         5|blood_orange|                2|
+----------+------------+-----------------+
▶ Run to see output

Transaction 118 appears twice for product 5; keeping only its latest row leaves 2 transactions, not 3. Banana (no transactions) and product 7 (not in the inventory) drop out of the inner join.

Vishal Taneja·

Apr 20, 2026Interview Question

Q.10

Find customers who have placed at least one order (PySpark join)

Easy

Write a PySpark program to perform an JOIN and return only customers who have placed at least one order.

customers = [
    ("C001", "Arjun", "Chennai"),
    ("C002", "Priya", None),
    ("C003", "Ravi", None)
]
 
orders = [
    ("C001", "Idli", 50),
    ("C002", "Coffee", 120),
    ("C004", "Dosa", 60)
]
 
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
 
# Create Spark session
spark = SparkSession.builder.appName("CustomerOrders").getOrCreate()
 
# Create DataFrames
customers_df = spark.createDataFrame(customers, ["customer_id", "name", "city"])
orders_df = spark.createDataFrame(orders, ["customer_id", "item", "amount"])
 
# Perform INNER JOIN
result_df = customers_df.join(orders_df, on="customer_id", how="inner")
 
# Select only customer columns
final_df = result_df.select("customer_id", "name", "city")
 
# Show result
 
final_df.show()
▶ Run to see output

Better: a left semi join

The inner join returns a customer once per order. Give C001 a second order and C001 appears twice. A left_semi join keeps each matching customer once and returns only the customer columns:

result_df = customers_df.join(orders_df, on="customer_id", how="left_semi")
result_df.show()
▶ Run to see output

Expected output

+-----------+-----+-------+
|customer_id| name|   city|
+-----------+-----+-------+
|       C001|Arjun|Chennai|
|       C002|Priya|   NULL|
+-----------+-----+-------+
▶ Run to see output

The opposite question, customers with no orders, is how="left_anti".

Practise this: Customers with at least one order (runs in the browser, with test cases).

PySpark Coding Interview Questions

Scenario: for each customer, show the month-over-month change and a running total.

Expected output

Points interviewers look for

Scenario: find the top 2 salaries in each department, keeping employees who tie.

Expected output

Choosing the ranking function

Scenario: a table receives several updates per customer. How do you keep only the latest record per key?

Expected output

What the interviewer is checking

You receive employee data in Spark. How would you calculate a 15% bonus column?

Expected output

Find employees whose salary is greater than 55,000

Expected output

A bank wants to identify customers whose salary has significantly declined over the last 3 months.

Business Requirement

Risk Criteria

Input DataFrame

account_transactions

Expected Output

Given a DataFrame containing flight/train routes and fares, find the cheapest fare between a source and destination, including the possibility of travelling through an intermediate city?

​Input DataFrame

Possible routes

Expected output

Why include the direct route?

Remember the logic

Given a sales DataFrame containing product_id, unit_price, and quantity, write PySpark code to calculate the total amount for each product and the overall total sales amount.

1. Input DataFrame

Expected output

Find the number of transactions that occurred for each product. Output the product name along with the transaction count and sort by product_id

Expected output

Find customers who have placed at least one order (PySpark join)

Better: a left semi join

Expected output

PySpark Coding
Interview Questions

`account_transactions`

Input DataFrame