Как получить доступ к столбцу в Dataframe, где DataFrame создается строкой

Question

Я новичок в pyspark
Требуется получить доступ к столбцу из DataFrame, который создан строкой.
Пожалуйста, см. Код ниже из моего .py файла
Это выдает ошибку AttributeError: у объекта 'DataFrame' нет атрибут «товар»

import findspark

findspark.init("/opt/spark")

from pyspark.sql import SparkSession
from pyspark.sql import Row
from pyspark.sql import SQLContext



productRevenue = Row("product", "category", "revenue")
spark = SparkSession \
    .builder \
    .appName("DataFrame Learning") \
    .getOrCreate()

sqlContext = SQLContext(spark)

productRevenue1 = productRevenue("product", "Cell phone", 6000)
productRevenue2 = productRevenue("Normal", "Tablet", 1500)
productRevenue3 = productRevenue("Mini", "Tablet", 5500)
productRevenue4 = productRevenue("Ultra thin", "Cell phone", 5000)
productRevenue5 = productRevenue("Very thin", "Cell phone", 6000)
productRevenue6 = productRevenue("Big", "Tablet", 2500)
productRevenue7 = productRevenue("Bendable", "Cell phone", 3000)
productRevenue8 = productRevenue("Foldable", "Cell phone", 3000)
productRevenue9 = productRevenue("Pro", "Tablet", 5500)
productRevenue10 = productRevenue("Pro2", "Tablet", 5500)

productRevenueAll = Row(
    productRevenue=[productRevenue1, productRevenue2, productRevenue3, productRevenue4, productRevenue5,
                    productRevenue6, productRevenue7, productRevenue8, productRevenue9, productRevenue10])

dataFrame = spark.createDataFrame(productRevenueAll)



filter_df = dataFrame.filter((dataFrame.product=="product") )

Answer 1

Вы вкладываете объект Row, который приводит к получению struct полей.

• Вот способы, которыми вы можете createDataFrame из Row объекта.

---

Example:

#using .toDF to create dataframe
sc.parallelize([productRevenue1, productRevenue2, productRevenue3, productRevenue4, productRevenue5,productRevenue6, productRevenue7, productRevenue8, productRevenue9, productRevenue10]).toDF().show()

#using spark.createDataFrame to create dataframe
spark.createDataFrame([productRevenue1, productRevenue2, productRevenue3, productRevenue4, productRevenue5,productRevenue6, productRevenue7, productRevenue8, productRevenue9, productRevenue10]).show()

#creating dataframe from rdd
productRevenue=sc.parallelize([productRevenue1, productRevenue2, productRevenue3, productRevenue4, productRevenue5,productRevenue6, productRevenue7, productRevenue8, productRevenue9, productRevenue10])

#creating dataframe from list
productRevenue=[productRevenue1, productRevenue2, productRevenue3, productRevenue4, productRevenue5,productRevenue6, productRevenue7, productRevenue8, productRevenue9, productRevenue10]

spark.createDataFrame(productRevenue).show()
#+----------+----------+-------+
#|   product|  category|revenue|
#+----------+----------+-------+
#|   product|Cell phone|   6000|
#|    Normal|    Tablet|   1500|
#|      Mini|    Tablet|   5500|
#|Ultra thin|Cell phone|   5000|
#| Very thin|Cell phone|   6000|
#|       Big|    Tablet|   2500|
#|  Bendable|Cell phone|   3000|
#|  Foldable|Cell phone|   3000|
#|       Pro|    Tablet|   5500|
#|      Pro2|    Tablet|   5500|
#+----------+----------+-------+


dataFrame=spark.createDataFrame(productRevenue)

dataFrame.filter((dataFrame.product=="product") ).show()
#+-------+----------+-------+
#|product|  category|revenue|
#+-------+----------+-------+
#|product|Cell phone|   6000|
#+-------+----------+-------+

Answer 2

Чтобы создать DataFrame из строк, один из подходов заключается в вызове SparkSession.createDataFrame() в списке строк.

Если вы хотите создать DataFrame наподобие

# +----------+----------+-------+
# |   product|  category|revenue|
# +----------+----------+-------+
# |   product|Cell phone|   6000|
# |    Normal|    Tablet|   1500|
# |      Mini|    Tablet|   5500|
# |             ...             |
# +----------+----------+-------+

# with Schema:

# root
#  |-- product: string (nullable = true)
#  |-- category: string (nullable = true)
#  |-- revenue: long (nullable = true)

Тогда вместо того, чтобы иметь productRevenueAll в качестве ряда строк, измените его на список строк, например:

productRevenueAll = [
    productRevenue1, productRevenue2, productRevenue3, 
    productRevenue4, productRevenue5, productRevenue6, 
    productRevenue7, productRevenue8, productRevenue9,
    productRevenue10,
]

dataFrame = spark.createDataFrame(productRevenueAll)

# then use it like:

dataFrame.product
# Column<b'product'>

dataFrame.select(dataFrame.product).show()
# +----------+
# |   product|
# +----------+
# |   product|
# |    Normal|
# |      Mini|
# |   ...    |
# +----------+

---

Однако, если вы действительно намеревались создать вложенную структуру, как:

# +-----------------------------+
# |      productRevenue         |
# +----------+----------+-------+
# |   product|  category|revenue|
# +----------+----------+-------+
# |   product|Cell phone|   6000|
# |    Normal|    Tablet|   1500|
# |             ...             |
# +----------+----------+-------+

# with Schema:

# root
#  |-- productRevenue: array (nullable = true)
#  |    |-- element: struct (containsNull = true)
#  |    |    |-- product: string (nullable = true)
#  |    |    |-- category: string (nullable = true)
#  |    |    |-- revenue: long (nullable = true)

feed createDataFrame() со списком из одного элемента, например:

productRevenueAllNested = Row(
    productRevenue=[
        productRevenue1, productRevenue2, productRevenue3, 
        productRevenue4, productRevenue5, productRevenue6, 
        productRevenue7, productRevenue8, productRevenue9, 
        productRevenue10,
    ])

dataFrameNested = spark.createDataFrame([productRevenueAllNested]) 

# then access it like
dataFrameNested.printSchema()

dataFrameNested.select(dataFrameNested.productRevenue).show()
# +----------------------+
# |productRevenue.product|
# +----------------------+
# |  [product, Normal,...|
# +----------------------+