Flink 1.7.1 не может аутентифицировать s3a с core-site.xml - PullRequest
Новая
       15

Flink 1.7.1 не может аутентифицировать s3a с core-site.xml

0 голосов
/ 25 января 2019

Использование Flink 1.7.1 при построении его для кластера с одним заданием на kubernetes flink не может загрузить XML-файл core-site, несмотря на то, что он находится на пути к классам, что приводит к игнорированию конфигурации, однако, если я добавлю ENV-переменные AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID, он будет работать найти его, но если я зависим от core-site.xml, он никогда не будет работать без переменных env.

В настоящее время я копирую файл core-site.xml, как показано в Dockerfile, и в документации сказано, что HADOOP_CONF_DIR указывается в качестве переменной env. Тем не менее он не загружает его, в результате чего NoCredentialsProvider.

Исключение составляет: 

Caused by: org.apache.flink.fs.s3base.shaded.com.amazonaws.AmazonClientException: No AWS Credentials provided by 
BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : org.apache.flink.fs.s3base.shaded.com.amazonaws.SdkClientException: 
Unable to load credentials from service endpoint

Classpath загружен менеджером заданий / диспетчером задач 

-  Classpath: 


/opt/flink-1.7.1/lib/aws-java-sdk-core-1.11.489.jar
:/opt/flink-1.7.1/lib/aws-java-sdk-kms-1.11.489.jar
:/opt/flink-1.7.1/lib/aws-java-sdk-s3-1.10.6.jar
:/opt/flink-1.7.1/lib/flink-python_2.12-1.7.1.jar
:/opt/flink-1.7.1/lib/flink-s3-fs-hadoop-1.7.1.jar
:/opt/flink-1.7.1/lib/flink-shaded-hadoop2-uber-1.7.1.jar
:/opt/flink-1.7.1/lib/hadoop-aws-2.8.0.jar:/opt/flink-1.7.1/lib/httpclient-4.5.6.jar
:/opt/flink-1.7.1/lib/httpcore-4.4.11.jar
:/opt/flink-1.7.1/lib/jackson-annotations-2.9.8.jar
:/opt/flink-1.7.1/lib/jackson-core-2.9.8.jar
:/opt/flink-1.7.1/lib/jackson-databind-2.9.8.jar
:/opt/flink-1.7.1/lib/job.jar
:/opt/flink-1.7.1/lib/joda-time-2.10.1.jar
:/opt/flink-1.7.1/lib/log4j-1.2.17.jar
:/opt/flink-1.7.1/lib/slf4j-log4j12-1.7.15.jar
:/opt/flink-1.7.1/lib/flink-dist_2.12-1.7.1.jar
:
:/hadoop/conf:

Dockerfile для создания образа Docker: 

################################################################################
#  Licensed to the Apache Software Foundation (ASF) under one
#  or more contributor license agreements.  See the NOTICE file
#  distributed with this work for additional information
#  regarding copyright ownership.  The ASF licenses this file
#  to you under the Apache License, Version 2.0 (the
#  "License"); you may not use this file except in compliance
#  with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
# limitations under the License.
################################################################################

FROM openjdk:8-jre-alpine

# Install requirements
# Modification to original Dockerfile to support rocksdb
# RUN apk add --no-cache bash snappy
# This is a fix for RocksDB compatibility


# Flink environment variables
ENV FLINK_INSTALL_PATH=/opt
ENV FLINK_HOME $FLINK_INSTALL_PATH/flink
ENV FLINK_LIB_DIR $FLINK_HOME/lib
ENV PATH $PATH:$FLINK_HOME/bin
ENV FLINK_CONF $FLINK_HOME/conf
ENV FLINK_OPT $FLINK_HOME/opt
ENV FLINK_HADOOP_CONF /hadoop/conf

# flink-dist can point to a directory or a tarball on the local system
ARG flink_dist=NOT_SET
ARG job_jar=NOT_SET

# Install build dependencies and flink
ADD $flink_dist $FLINK_INSTALL_PATH
ADD $job_jar $FLINK_INSTALL_PATH/job.jar

RUN set -x && \
  ln -s $FLINK_INSTALL_PATH/flink-* $FLINK_HOME && \
  ln -s $FLINK_INSTALL_PATH/job.jar $FLINK_LIB_DIR && \
  addgroup -S flink && adduser -D -S -H -G flink -h $FLINK_HOME flink && \
  chown -R flink:flink $FLINK_INSTALL_PATH/flink-* && \
  chown -h flink:flink $FLINK_HOME

# Modification to original Dockerfile
RUN apk add --no-cache bash libc6-compat snappy 'su-exec>=0.2'

COPY core-site.xml $FLINK_HADOOP_CONF/core-site.xml
ENV HADOOP_CONF_DIR=$FLINK_HADOOP_CONF

RUN echo "fs.hdfs.hadoopconf: $FLINK_HADOOP_CONF" >> $FLINK_CONF/flink-conf.yaml

RUN echo "akka.ask.timeout: 30 min" >> $FLINK_CONF/flink-conf.yaml

RUN echo "akka.client.timeout: 30 min" >> $FLINK_CONF/flink-conf.yaml
RUN echo "web.timeout: 180000" >> $FLINK_CONF/flink-conf.yaml

RUN mv $FLINK_OPT/flink-s3-fs-hadoop-1.7.1.jar $FLINK_LIB_DIR

COPY docker-entrypoint.sh /

RUN chmod +x docker-entrypoint.sh

RUN wget -O $FLINK_LIB_DIR/hadoop-aws-2.8.0.jar https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/2.8.0/hadoop-aws-2.8.0.jar
RUN wget -O $FLINK_LIB_DIR/aws-java-sdk-s3-1.10.6.jar http://central.maven.org/maven2/com/amazonaws/aws-java-sdk-s3/1.10.6/aws-java-sdk-s3-1.10.6.jar
#Transitive Dependency of aws-java-sdk-s3
RUN wget -O $FLINK_LIB_DIR/aws-java-sdk-core-1.11.489.jar http://central.maven.org/maven2/com/amazonaws/aws-java-sdk-core/1.11.489/aws-java-sdk-core-1.11.489.jar
RUN wget -O $FLINK_LIB_DIR/aws-java-sdk-kms-1.11.489.jar http://central.maven.org/maven2/com/amazonaws/aws-java-sdk-kms/1.11.489/aws-java-sdk-kms-1.11.489.jar
RUN wget -O $FLINK_LIB_DIR/jackson-annotations-2.9.8.jar http://central.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.9.8/jackson-annotations-2.9.8.jar
RUN wget -O $FLINK_LIB_DIR/jackson-core-2.9.8.jar http://central.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.9.8/jackson-core-2.9.8.jar
RUN wget -O $FLINK_LIB_DIR/jackson-databind-2.9.8.jar http://central.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.9.8/jackson-databind-2.9.8.jar
RUN wget -O $FLINK_LIB_DIR/joda-time-2.10.1.jar http://central.maven.org/maven2/joda-time/joda-time/2.10.1/joda-time-2.10.1.jar
RUN wget -O $FLINK_LIB_DIR/httpcore-4.4.11.jar http://central.maven.org/maven2/org/apache/httpcomponents/httpcore/4.4.11/httpcore-4.4.11.jar
RUN wget -O $FLINK_LIB_DIR/httpclient-4.5.6.jar http://central.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.6/httpclient-4.5.6.jar
#Modification to original Dockerfile

USER flink
EXPOSE 8081 6123
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["--help"]

ядро-site.xml 

<configuration>

    <property>
        <name>fs.s3.impl</name>
        <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
    </property>

    <!-- Comma separated list of local directories used to buffer
         large results prior to transmitting them to S3. -->
    <property>
        <name>fs.s3a.buffer.dir</name>
        <value>/tmp</value>
    </property>

    <property>
        <name>fs.s3a.access.key</name>
        <description>AWS access key ID.
            Omit for IAM role-based or provider-based authentication.</description>
        <value>*</value>
    </property>

    <property>
        <name>fs.s3a.secret.key</name>
        <description>AWS secret key.
            Omit for IAM role-based or provider-based authentication.</description>
        <value>*</value>
    </property>


    <property>
        <name>fs.s3a.aws.credentials.provider</name>
        <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
    </property>


</configuration>

Ответы [ 2 ]

0 голосов
/ 27 января 2019

Решено,

Помогло добавление hadoop-common с соответствующей версией в Dockerfile.

0 голосов
/ 26 января 2019

Хорошо, это было решено, если у вас есть затененный хэдуп на пути к классам (перемещая его из / opt в / lib), вам нужно указать свои ключи в flink-conf, однако теперь я получаю следующее исключение 

Caused by: java.io.IOException: org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider 
constructor exception.  A class specified in fs.s3a.aws.credentials.provider must provide a public constructor accepting URI and Configuration, or a public factory method named getInstance that accepts no arguments, or a public default constructor.

Есть идеи?

...