Я создал список оператора alter table. Когда я выполняю инструкцию alter в l oop в искре, это занимает больше времени.
Я хотел увеличить производительность запроса. Поэтому я следовал нижеследующему подходу, где я создал RDD в списке и выполнил rdd. foreachPartition. Мое намерение состоит в том, чтобы выполнить каждый оператор alter на рабочих узлах параллельно. Я получаю ошибку ниже.
Причина: java .io.IOException: org. apache .spark.SparkException: Не удалось получить broadcast_16_piece0 of broadcast_16
Правильно ли подходит мой подход? или я что-то упускаю?
Ниже приведен код.
scala> list1.foreach(println)
ALTER TABLE stk000h.csd_item_visit_dly_spark ADD IF NOT EXISTS PARTITION (rpt_dt='2018-01-01',op_cmpny_cd='WMT.COM',tenant_ste_cd='US_ANDROID') LOCATION '/user/hive/userdbs/stk000h.db/csd_item_visit_dly_spark/rpt_dt=2018-01-01/op_cmpny_cd=WMT.COM/tenant_ste_cd=US_ANDROID'
ALTER TABLE stk000h.csd_item_visit_dly_spark ADD IF NOT EXISTS PARTITION (rpt_dt='2018-01-01',op_cmpny_cd='WMT.COM',tenant_ste_cd='US_CORESITE') LOCATION '/user/hive/userdbs/stk000h.db/csd_item_visit_dly_spark/rpt_dt=2018-01-01/op_cmpny_cd=WMT.COM/tenant_ste_cd=US_CORESITE'
.
.
.
.
sccala>val rdd=spark.sparkContext.parallelize(list1).repartition(200)
scala> rdd.foreachPartition({ x=>
| x.foreach {e=>
| println(e)
| val conf = new SparkConf().setMaster("local[2]").set("spark.cleaner.ttl","10000").set("spark.scheduler.mode", "FAIR")
| val ctx = new SparkContext(conf)
| val sqlContext = new SQLContext(ctx)
| sqlContext.sql(e)
| }
| })
warning: there was one deprecation warning; re-run with -deprecation for details
20/03/31 06:38:20 INFO SparkContext: Starting job: foreachPartition at <console>:68
20/03/31 06:38:20 INFO DAGScheduler: Registering RDD 31 (repartition at <console>:63)
20/03/31 06:38:20 INFO DAGScheduler: Got job 15 (foreachPartition at <console>:68) with 200 output partitions
20/03/31 06:38:20 INFO DAGScheduler: Final stage: ResultStage 27 (foreachPartition at <console>:68)
20/03/31 06:38:20 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 26)
20/03/31 06:38:20 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 26)
20/03/31 06:38:20 INFO DAGScheduler: Submitting ShuffleMapStage 26 (MapPartitionsRDD[31] at repartition at <console>:63), which has no missing parents
20/03/31 06:38:20 INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 3.0 KB, free 912.3 MB)
20/03/31 06:38:20 WARN DFSClient: Slow ReadProcessor read fields took 46978ms (threshold=30000ms); ack: seqno: 104 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 611418 flag: 0 flag: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.72.153.11:1019,DS-3c71a15e-03e6-4642-ac2b-77fd71f3d5cb,DISK], DatanodeInfoWithStorage[10.72.150.27:1019,DS-b6c0678b-e097-4faa-bb24-ddc6cda17b65,DISK], DatanodeInfoWithStorage[10.72.150.24:1019,DS-bfd85c92-2bb6-4a65-bf59-b661f01605bd,DISK]]
20/03/31 06:38:20 INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 1928.0 B, free 912.3 MB)
20/03/31 06:38:20 INFO BlockManagerInfo: Added broadcast_17_piece0 in memory on 10.24.84.156:40819 (size: 1928.0 B, free: 912.3 MB)
20/03/31 06:38:20 INFO SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1006
20/03/31 06:38:20 INFO DAGScheduler: Submitting 140 missing tasks from ShuffleMapStage 26 (MapPartitionsRDD[31] at repartition at <console>:63) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
20/03/31 06:38:20 INFO YarnScheduler: Adding task set 26.0 with 140 tasks
20/03/31 06:38:20 INFO TaskSetManager: Starting task 0.0 in stage 26.0 (TID 1969, osec400684.wal-mart.com, executor 21, partition 0, PROCESS_LOCAL, 4949 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 1.0 in stage 26.0 (TID 1970, osec400925.wal-mart.com, executor 9, partition 1, PROCESS_LOCAL, 4950 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 2.0 in stage 26.0 (TID 1971, osec401251.wal-mart.com, executor 27, partition 2, PROCESS_LOCAL, 4948 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 3.0 in stage 26.0 (TID 1972, osec400288.wal-mart.com, executor 23, partition 3, PROCESS_LOCAL, 4947 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 4.0 in stage 26.0 (TID 1973, osec401186.wal-mart.com, executor 3, partition 4, PROCESS_LOCAL, 5070 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 5.0 in stage 26.0 (TID 1974, osec400442.wal-mart.com, executor 5, partition 5, PROCESS_LOCAL, 4948 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 6.0 in stage 26.0 (TID 1975, osec401331.wal-mart.com, executor 7, partition 6, PROCESS_LOCAL, 4947 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 7.0 in stage 26.0 (TID 1976, osec400899.wal-mart.com, executor 12, partition 7, PROCESS_LOCAL, 4949 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 8.0 in stage 26.0 (TID 1977, oser405773.wal-mart.com, executor 14, partition 8, PROCESS_LOCAL, 4950 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 9.0 in stage 26.0 (TID 1978, osec400603.wal-mart.com, executor 11, partition 9, PROCESS_LOCAL, 5066 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 10.0 in stage 26.0 (TID 1979, osec400223.wal-mart.com, executor 32, partition 10, PROCESS_LOCAL, 4949 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 11.0 in stage 26.0 (TID 1980, oser405320.wal-mart.com, executor 10, partition 11, PROCESS_LOCAL, 4950 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 12.0 in stage 26.0 (TID 1981, osec401186.wal-mart.com, executor 3, partition 12, PROCESS_LOCAL, 4950 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 13.0 in stage 26.0 (TID 1982, osec400442.wal-mart.com, executor 5, partition 13, PROCESS_LOCAL, 5065 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 14.0 in stage 26.0 (TID 1983, osec400442.wal-mart.com, executor 5, partition 14, PROCESS_LOCAL, 4952 bytes)
20/03/31 06:38:20 INFO TaskSetManager: Starting task 15.0 in stage 26.0 (TID 1984, osec400442.wal-mart.com, executor 5, partition 15, PROCESS_LOCAL, 4951 bytes)
20/03/31 06:38:20 WARN TaskSetManager: Lost task 13.0 in stage 26.0 (TID 1982, osec400442.wal-mart.com, executor 5): java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_17_piece0 of broadcast_17
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1310)
at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:206)
at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:66)
at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:66)
at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:96)
at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:86)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Failed to get broadcast_17_piece0 of broadcast_17
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply$mcVI$sp(TorrentBroadcast.scala:178)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:150)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:150)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$readBlocks(TorrentBroadcast.scala:150)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:222)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1303)
... 12 more