Я пытаюсь настроить Spark (точнее, pySpark) на новую виртуальную машину Ubuntu 18.10. Я настроил Hadoop, SSH без пароля (localhost), другого пользователя с именем hduser, который мне нужен для Hadoop. Теперь я поместил папку Spark рядом с ней, добавил пути к bashrc:
export HADOOP_HOME=/usr/local/hadoop-3.1.2
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
# Native path
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
# Java path
export JAVA_HOME="/usr"
# OS path
export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/sbin
export SPARK_HOME=/usr/local/spark-2.4.1-bin-hadoop2.7
export PATH=$PATH:/usr/local/spark-2.4.1-bin-hadoop2.7/bin
Так что теперь из hduser я захожу в $ SPARK_HOME и пытаюсь запустить ./sbin/start-all.sh, и я получаю это сообщение:
chown: changing ownership of '/usr/local/spark-2.4.1-bin-hadoop2.7/logs': Operation not permitted
starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out
/usr/local/spark-2.4.1-bin-hadoop2.7/sbin/spark-daemon.sh: line 128: /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out: Permission denied
failed to launch: nice -n 0 /usr/local/spark-2.4.1-bin-hadoop2.7/bin/spark-class org.apache.spark.deploy.master.Master --host cristi-VirtualBox --port 7077 --webui-port 8080
tail: cannot open '/usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out' for reading: No such file or directory
full log in /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out
localhost: chown: changing ownership of '/usr/local/spark-2.4.1-bin-hadoop2.7/logs': Operation not permitted
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out
localhost: /usr/local/spark-2.4.1-bin-hadoop2.7/sbin/spark-daemon.sh: line 128: /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out: Permission denied
localhost: failed to launch: nice -n 0 /usr/local/spark-2.4.1-bin-hadoop2.7/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://cristi-VirtualBox:7077
localhost: tail: cannot open '/usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out' for reading: No such file or directory
localhost: full log in /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out
В чем конкретно проблема? Я не понимаю, что происходит.