Learn PySpark, SQL & Data Engineering

• PySpark groupBy(): group and aggregate a DataFrame — PySpark `groupBy()` groups the rows of a DataFrame by one or more columns so you can compute aggregates (count, sum, avg
• PySpark join(): combine two DataFrames — PySpark `join()` combines two DataFrames on one or more key columns. You control the join type with the `how` argument —
• PySpark window functions explained — PySpark window functions compute a value across a set of rows related to the current row without collapsing them (unlike
• SQL window functions with examples — SQL window functions perform a calculation across a set of rows related to the current row using the `OVER()` clause, wi
• ETL vs ELT: what's the difference? — ETL (Extract, Transform, Load) transforms data **before** loading it into the destination; ELT (Extract, Load, Transform
• PySpark withColumn(): add or transform a column — PySpark `withColumn(name, expr)` returns a new DataFrame with a column added or replaced. The second argument is a Colum
• PySpark filter() and where(): keep rows that match a condition — PySpark `filter()` (and its alias `where()`) returns a new DataFrame keeping only the rows that satisfy a Boolean condit
• PySpark select(): pick and compute columns — PySpark `select()` returns a new DataFrame containing only the columns (or column expressions) you name. You can pass co
• PySpark: read a CSV into a DataFrame — In PySpark you load a CSV with `spark.read.csv(path)` or `spark.read.option(...).csv(path)`. Set `header=True` to use th
• PySpark orderBy() and sort(): order rows — PySpark `orderBy()` (alias `sort()`) returns a new DataFrame with rows sorted by one or more columns. By default sorting
• PySpark dropDuplicates() and distinct(): deduplicate rows — PySpark `distinct()` removes fully duplicate rows; `dropDuplicates(subset)` removes rows that are duplicates on a chosen
• PySpark UDFs: custom functions on columns — A PySpark UDF wraps a Python function so it can run on DataFrame columns. Define it with `udf(fn, returnType)` or the `@
• SQL CTEs: the WITH clause — A SQL CTE (Common Table Expression) is a named temporary result set defined with the `WITH` clause and referenced later
• repartition() vs coalesce() in PySpark — Both `repartition()` and `coalesce()` change the number of partitions of a PySpark DataFrame. `repartition(n)` does a fu
• PySpark fillna(): replace null values — PySpark `fillna(value)` (alias `df.na.fill(value)`) replaces null values in a DataFrame. Pass a single value to fill all
• What is Data Engineering? — Data Engineering is the practice of designing, building and maintaining the systems that collect, store, transform and s