Java-Spark: невозможно прочитать файл свойств в режиме кластера - PullRequest
Новая
       3

Java-Spark: невозможно прочитать файл свойств в режиме кластера

0 голосов
/ 03 июля 2019

У меня есть очень простой тестовый код с целью чтения файла java properties , переданного --files, и отображения значения одного из его ключей.
У меня есть файл свойств testprop.prop: 

name:aiman
location:india

Я использую команду spark-submit , например: 

spark-submit --class org.main.ReadLocalFile --master yarn --deploy-mode cluster --queue orion --files /path/to/testprop.prop#testprop.prop spark_cluster_file_read-0.0.1-SNAPSHOT-jar-with-dependencies.jar testprop.prop

Я передал имя файла, используя --files /path/to/file/testprop.prop#testprop.prop, и передал testprop.prop в качестве аргумента в код, чтобы оно могло быть прочитано FileInputStream.
Мой код как: 

package org.main;

import java.io.FileInputStream;
import java.io.InputStream;
import java.util.Properties;

import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.spark.sql.SparkSession;


public class ReadLocalFile {
    public static void main(String args[]) throws Exception
    {
        SparkSession spark = SparkSession.builder().getOrCreate();
        String filename = args[0];
        Properties prop = new Properties();
        InputStream in = null;
        try{
            in = new FileInputStream(filename);
            prop.load(in);
        }
        catch(Exception e){
            e.printStackTrace();
            System.out.println("=========Exception Thrown============");
            System.exit(1);
        }  

        System.out.println("====================Value: "+prop.getProperty("name"));

    }

}

Код выполняется до конца, но не показывает никакого вывода. Ожидаемый результат должен был быть либо значением клавиши name, либо, возможно, FileNotFoundException из-за обработки try-catch.
Сгенерированные журналы: 

19/07/03 11:32:52 INFO O: Set a new configuration for the first time.
19/07/03 11:32:52 INFO d: Method not implemented in this version of Hadoop: org.apache.hadoop.fs.FileSystem$Statistics.getBytesReadLocalHost
19/07/03 11:32:52 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
19/07/03 11:32:52 INFO u: Scheduling statistics report every 2000 millisecs
19/07/03 11:32:52 INFO RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
19/07/03 11:32:53 INFO RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]
19/07/03 11:32:53 INFO Client: Requesting a new application from cluster with 24 NodeManagers
19/07/03 11:32:53 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (102400 MB per container)
19/07/03 11:32:53 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
19/07/03 11:32:53 INFO Client: Setting up container launch context for our AM
19/07/03 11:32:53 INFO Client: Setting up the launch environment for our AM container
19/07/03 11:32:53 INFO Client: Preparing resources for our AM container
19/07/03 11:32:53 INFO HadoopFSCredentialProvider: getting token for: hdfs://clustername/user/serviceuser
19/07/03 11:32:53 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 6977007 for serviceuser on ha-hdfs:clustername
19/07/03 11:32:55 INFO metastore: Trying to connect to metastore with URI thrift://XX.XX.XX.XX:9083
19/07/03 11:32:55 INFO metastore: Connected to metastore.
19/07/03 11:32:56 INFO HiveCredentialProvider: Get Token from hive metastore: Kind: HIVE_DELEGATION_TOKEN, Service: , Ident: 00 1a 65 62 64 70 62 75 73 73 40 43 41 42 4c 45 2e 43 4f 4d 43 41 53 54 2e 43 4f 4d 04 68 69 76 65 00 8a 01 6b b7 9b e6 e3 8a 01 6b db a8 6a e3 8e 9e e1 8e 02 f2
19/07/03 11:32:56 INFO Client: Use hdfs cache file as spark.yarn.archive for HDP, hdfsCacheFile:hdfs://clustername/hdp/apps/2.6.3.20-2/spark2/spark2-hdp-yarn-archive.tar.gz
19/07/03 11:32:56 INFO Client: Source and destination file systems are the same. Not copying hdfs://clustername/hdp/apps/2.6.3.20-2/spark2/spark2-hdp-yarn-archive.tar.gz
19/07/03 11:32:56 INFO Client: Uploading resource file:/home/serviceuser/aiman/spark_cluster_file_read-0.0.1-SNAPSHOT-jar-with-dependencies.jar -> hdfs://clustername/user/serviceuser/.sparkStaging/application_1561094073414_101648/spark_cluster_file_read-0.0.1-SNAPSHOT-jar-with-dependencies.jar
19/07/03 11:32:57 INFO Client: Uploading resource file:/home/serviceuser/aiman/testprop.prop#testprop.prop -> hdfs://clustername/user/serviceuser/.sparkStaging/application_1561094073414_101648/testprop.prop
19/07/03 11:32:57 INFO Client: Uploading resource file:/tmp/spark-02d69650-9fb9-4f5e-9947-d8fa629323f4/__spark_conf__3111084457019278305.zip -> hdfs://clustername/user/serviceuser/.sparkStaging/application_1561094073414_101648/__spark_conf__.zip
19/07/03 11:32:57 INFO SecurityManager: Changing view acls to: serviceuser
19/07/03 11:32:57 INFO SecurityManager: Changing modify acls to: serviceuser
19/07/03 11:32:57 INFO SecurityManager: Changing view acls groups to:
19/07/03 11:32:57 INFO SecurityManager: Changing modify acls groups to:
19/07/03 11:32:57 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(serviceuser); groups with view permissions: Set(); users  with modify permissions: Set(serviceuser); groups with modify permissions: Set()
19/07/03 11:32:57 INFO Client: Submitting application application_1561094073414_101648 to ResourceManager
19/07/03 11:32:57 INFO YarnClientImpl: Submitted application application_1561094073414_101648
19/07/03 11:32:58 INFO Client: Application report for application_1561094073414_101648 (state: ACCEPTED)
19/07/03 11:32:58 INFO Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: AM container is launched, waiting for AM container to Register with RM
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: orion
         start time: 1562153577621
         final status: UNDEFINED
         tracking URL: http://XX.XX.XX.XX:8088/proxy/application_1561094073414_101648/
         user: serviceuser
19/07/03 11:32:59 INFO Client: Application report for application_1561094073414_101648 (state: ACCEPTED)
19/07/03 11:33:00 INFO Client: Application report for application_1561094073414_101648 (state: ACCEPTED)
19/07/03 11:33:01 INFO Client: Application report for application_1561094073414_101648 (state: ACCEPTED)
19/07/03 11:33:02 INFO Client: Application report for application_1561094073414_101648 (state: ACCEPTED)
19/07/03 11:33:03 INFO Client: Application report for application_1561094073414_101648 (state: ACCEPTED)
19/07/03 11:33:04 INFO Client: Application report for application_1561094073414_101648 (state: ACCEPTED)
19/07/03 11:33:05 INFO Client: Application report for application_1561094073414_101648 (state: RUNNING)
19/07/03 11:33:05 INFO Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: N/A
         ApplicationMaster host: XX.XX.XX.XX
         ApplicationMaster RPC port: 0
         queue: orion
         start time: 1562153577621
         final status: UNDEFINED
         tracking URL: http://XX.XX.XX.XX:8088/proxy/application_1561094073414_101648/
         user: serviceuser
19/07/03 11:33:06 INFO Client: Application report for application_1561094073414_101648 (state: RUNNING)
19/07/03 11:33:07 INFO Client: Application report for application_1561094073414_101648 (state: RUNNING)
19/07/03 11:33:08 INFO Client: Application report for application_1561094073414_101648 (state: RUNNING)
19/07/03 11:33:09 INFO Client: Application report for application_1561094073414_101648 (state: RUNNING)
19/07/03 11:33:10 INFO Client: Application report for application_1561094073414_101648 (state: RUNNING)
19/07/03 11:33:11 INFO Client: Application report for application_1561094073414_101648 (state: RUNNING)
19/07/03 11:33:12 INFO Client: Application report for application_1561094073414_101648 (state: FINISHED)
19/07/03 11:33:12 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: XX.XX.XX.XX
         ApplicationMaster RPC port: 0
         queue: orion
         start time: 1562153577621
         final status: SUCCEEDED
         tracking URL: http://XX.XX.XX.XX:8088/proxy/application_1561094073414_101648/
         user: serviceuser
19/07/03 11:33:12 INFO ShutdownHookManager: Shutdown hook called
19/07/03 11:33:12 INFO ShutdownHookManager: Deleting directory /tmp/spark-02d69650-9fb9-4f5e-9947-d8fa629323f4

Скажите, пожалуйста, куда я иду или что-то упустил?
Это что в режиме кластера вывод не просматривается? Что я могу сделать, чтобы получить вывод на консоль?

1 Ответ

1 голос
/ 05 июля 2019

опция files копирует ваши файлы поддержки в узлы вашего исполнителя. это как вы можете загрузить файл. 

  def readProperties(propertiesPath: String) = {

val url = getClass.getResource("/" + propertiesPath)
assert(url != null, s"Could not create URL to read $propertiesPath properties file")
val source = Source.fromURL(url)
val properties = new Properties
properties.load(source.bufferedReader)

properties
}

и ваш звонок должен быть примерно таким прочитайте это в var 

val myProp= readProperties(args(0))

и для доступа к свойствам вы можете использовать что-то вроде этого 

val getnamefromProp= myProp.getProperty("name")

если это не работает, попробуйте использовать Soruce.fromFile (Path) в качестве альтернативы.

...