Pyspark Oracle JDB c Схема клиента - PullRequest
Новая
       27

Pyspark Oracle JDB c Схема клиента

0 голосов
/ 24 апреля 2020

Я пытаюсь прочитать таблицу из oracle, используя jdb c pyspark. Таблица oracle содержит множество столбцов типа FLOAT (126). 

from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql import functions as f
import json
jdbcUrl = "jdbc:oracle:thin:@{0}:{1}/{2}".format("hostname","1521","SCHEMA")

c_schema = "KEYCOL DECIMAL(10,0), DMDUNIT VARCHAR(50), LOC VARCHAR(50), STARTDATE DATE,  TYPE DECIMAL(10,0),  FCSTID VARCHAR(50),  MODEL VARCHAR(50),PERIOD1 DOUBLE ,PERIOD2 DOUBLE ,PERIOD3 DOUBLE ,PERIOD4 DOUBLE, PERIOD5 DOUBLE ,PERIOD6 DOUBLE ,PERIOD7 DOUBLE ,PERIOD8 DOUBLE ,PERIOD9 DOUBLE ,PERIOD10 DOUBLE ,PERIOD11 DOUBLE ,PERIOD12 DOUBLE ,PERIOD13 DOUBLE, PERIOD14 DOUBLE ,PERIOD15 DOUBLE ,PERIOD16 DOUBLE ,PERIOD17 DOUBLE ,PERIOD18 DOUBLE ,PERIOD19 DOUBLE ,PERIOD20 DOUBLE ,PERIOD21 DOUBLE, PERIOD22 DOUBLE ,PERIOD23 DOUBLE ,PERIOD24 DOUBLE ,PERIOD25 DOUBLE ,PERIOD26 DOUBLE ,PERIOD27 DOUBLE ,PERIOD28 DOUBLE ,PERIOD29 DOUBLE, PERIOD30 DOUBLE ,PERIOD31 DOUBLE ,PERIOD32 DOUBLE ,PERIOD33 DOUBLE ,PERIOD34 DOUBLE ,PERIOD35 DOUBLE ,PERIOD36 DOUBLE ,PERIOD37 DOUBLE, PERIOD38 DOUBLE ,PERIOD39 DOUBLE ,PERIOD40 DOUBLE ,PERIOD41 DOUBLE ,PERIOD42 DOUBLE ,PERIOD43 DOUBLE ,PERIOD44 DOUBLE ,PERIOD45 DOUBLE, PERIOD46 DOUBLE ,PERIOD47 DOUBLE ,PERIOD48 DOUBLE ,PERIOD49 DOUBLE ,PERIOD50 DOUBLE ,PERIOD51 DOUBLE ,PERIOD52 DOUBLE ,PERIOD53 DOUBLE"

connectionProperties = {
  "user" : "username",
  "password" : "paswowrd",
  "driver" : "oracle.jdbc.driver.OracleDriver",
  "fetchsize" : "100000"
  "customSchema" : c_schema
}

pushdown_query = "(SELECT  KEYCOL,  COL1,  COL2,  COl3,  COL4,  COL5,  COL6,  PERIOD1,  PERIOD2,  PERIOD3,  PERIOD4, PERIOD5,  PERIOD6,  PERIOD7,  PERIOD8,  PERIOD9,  PERIOD10,  PERIOD11,  PERIOD12,  PERIOD13, PERIOD14,  PERIOD15,  PERIOD16,  PERIOD17,  PERIOD18,  PERIOD19,  PERIOD20,  PERIOD21, PERIOD22,  PERIOD23,  PERIOD24,  PERIOD25,  PERIOD26,  PERIOD27,  PERIOD28,  PERIOD29, PERIOD30,  PERIOD31,  PERIOD32,  PERIOD33,  PERIOD34,  PERIOD35,  PERIOD36,  PERIOD37, PERIOD38,  PERIOD39,  PERIOD40,  PERIOD41,  PERIOD42,  PERIOD43,  PERIOD44,  PERIOD45, PERIOD46,  PERIOD47,  PERIOD48,  PERIOD49,  PERIOD50,  PERIOD51,  PERIOD52,  PERIOD53 FROM DB.TABLE ) TAB_ALS"

df = spark.read.jdbc(url=jdbcUrl, table=pushdown_query, column="KEYCOL", lowerBound=0, upperBound=10000, numPartitions=5, properties=connectionProperties)

>>> df.printSchema()
root
 |-- KEYCOL: decimal(10,0) (nullable = true)
 |-- COL1: string (nullable = true)
 |-- COl2: string (nullable = true)
 |-- COL3: date (nullable = true)
 |-- COL4: decimal(10,0) (nullable = true)
 |-- COL5: string (nullable = true)
 |-- COL6: string (nullable = true)
 |-- PERIOD1: double (nullable = true)
 |-- PERIOD2: double (nullable = true)
 |-- PERIOD3: double (nullable = true)
 |-- PERIOD4: double (nullable = true)
 |-- PERIOD5: double (nullable = true)
 |-- PERIOD6: decimal(38,10) (nullable = true)
 |-- PERIOD7: decimal(38,10) (nullable = true)
 |-- PERIOD8: decimal(38,10) (nullable = true)
 |-- PERIOD9: decimal(38,10) (nullable = true)
 |-- PERIOD10: decimal(38,10) (nullable = true)
...

Значение из oracle обрезается для формы столбцов PERIOD6 

+------+-------+----+----------+----+-----------------+-----+-------------------+---------------------+---------------------+--------------------+---------------------+-------------+-------------+-------------+-------------+-------------+-------------+------------+-------------+-------------+-------------+------------+------------+-------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|KEYCOL|COL1|COL2 |COL3 |COL4|COL5           |COL6|PERIOD1            |PERIOD2              |PERIOD3              |PERIOD4             |PERIOD5              |PERIOD6      |PERIOD7      |PERIOD8      |PERIOD9      |PERIOD10     |PERIOD11     |PERIOD12    |PERIOD13     |PERIOD14     |PERIOD15     |PERIOD16    |PERIOD17    |PERIOD18     |PERIOD19    |PERIOD20    |PERIOD21    |PERIOD22    |PERIOD23     |PERIOD24     |PERIOD25     |PERIOD26     |PERIOD27     |PERIOD28     |PERIOD29     |PERIOD30     |PERIOD31     |PERIOD32     |PERIOD33     |PERIOD34     |PERIOD35     |PERIOD36     |PERIOD37     |PERIOD38     |PERIOD39     |PERIOD40     |PERIOD41     |PERIOD42     |PERIOD43     |PERIOD44     |PERIOD45     |PERIOD46     |PERIOD47     |PERIOD48     |PERIOD49     |PERIOD50     |PERIOD51     |PERIOD52     |PERIOD53     |
+------+-------+----+----------+----+-----------------+-----+-------------------+---------------------+---------------------+--------------------+---------------------+-------------+-------------+-------------+-------------+-------------+-------------+------------+-------------+-------------+-------------+------------+------------+-------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
|1871  |B1234 |A23423|2020-03-07|1   |                 |ABCD  |0.36033156148977036|0.36033156148977036  |0.36033156148977036  |0.36033156148977036 |0.36033156148977036  |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615|0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615|0.3603315615|0.3603315615 |0.3603315615|0.3603315615|0.3603315615|0.3603315615|0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |0.3603315615 |

Почему это не работает, если я не предоставляю пользовательскую схему, все значения обрезаются

...