Я новичок в Bigdata и Hadoop.Вот моя программа MapReduce
package com.maxtemp;
import java.io.IOException;
import java.util.Iterator;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class MaxTemperature {
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "maxtemperature");
job.setJarByClass(MaxTemperature.class);
job.setMapperClass(MapMaxTemperature.class);
job.setReducerClass(ReduceMaxTemperature.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
}
public class MapMaxTemperature extends Mapper<Text, IntWritable, Text, IntWritable> {
IntWritable intWritable = new IntWritable(1);
Text word = new Text();
public void map(IntWritable key, Text value, Context context) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString(), " ");
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, intWritable);
}
}
}
public class ReduceMaxTemperature extends Reducer<Text, IntWritable, Text, IntWritable> {
private int max_temp = Integer.MIN_VALUE;
private int temp = 0;
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
Iterator<IntWritable> itr = values.iterator();
while (itr.hasNext()) {
temp = itr.next().get();
if (temp > max_temp) {
max_temp = temp;
}
}
context.write(key, new IntWritable(max_temp));
}
}
}
Это команда, которую я использую для запуска-
hadoop jar /home/hduser/HadoopPOC/eclipse/workspace/Assignment4/Assignment4.jar com.maxtemp.MaxTemperature /home/hduser/maxtempinput/Temperature.txt /home/hduser/maxtempoutput5/
И это вывод, который я получаю, вместе со stacktrace-
19/06/01 13:18:03 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/06/01 13:18:04 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
19/06/01 13:18:04 INFO input.FileInputFormat: Total input paths to process : 1
19/06/01 13:18:04 INFO mapreduce.JobSubmitter: number of splits:1
19/06/01 13:18:05 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1559320460461_0010
19/06/01 13:18:05 INFO impl.YarnClientImpl: Submitted application application_1559320460461_0010
19/06/01 13:18:05 INFO mapreduce.Job: The url to track the job: http://hduser-HP-Pavilion-g7-Notebook-PC:8088/proxy/application_1559320460461_0010/
19/06/01 13:18:05 INFO mapreduce.Job: Running job: job_1559320460461_0010
19/06/01 13:18:13 INFO mapreduce.Job: Job job_1559320460461_0010 running in uber mode : false
19/06/01 13:18:13 INFO mapreduce.Job: map 0% reduce 0%
19/06/01 13:18:17 INFO mapreduce.Job: Task Id : attempt_1559320460461_0010_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.NoSuchMethodException: com.maxtemp.MaxTemperature$MapMaxTemperature.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NoSuchMethodException: com.maxtemp.MaxTemperature$MapMaxTemperature.<init>()
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:128)
... 7 more
19/06/01 13:18:22 INFO mapreduce.Job: Task Id : attempt_1559320460461_0010_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.NoSuchMethodException: com.maxtemp.MaxTemperature$MapMaxTemperature.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NoSuchMethodException: com.maxtemp.MaxTemperature$MapMaxTemperature.<init>()
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:128)
... 7 more
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
19/06/01 13:18:26 INFO mapreduce.Job: Task Id : attempt_1559320460461_0010_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.NoSuchMethodException: com.maxtemp.MaxTemperature$MapMaxTemperature.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NoSuchMethodException: com.maxtemp.MaxTemperature$MapMaxTemperature.<init>()
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:128)
... 7 more
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
19/06/01 13:18:32 INFO mapreduce.Job: map 100% reduce 100%
19/06/01 13:18:32 INFO mapreduce.Job: Job job_1559320460461_0010 failed with state FAILED due to: Task failed task_1559320460461_0010_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
19/06/01 13:18:33 INFO mapreduce.Job: Counters: 13
Job Counters
Failed map tasks=4
Killed reduce tasks=1
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=11494
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=11494
Total time spent by all reduce tasks (ms)=0
Total vcore-milliseconds taken by all map tasks=11494
Total vcore-milliseconds taken by all reduce tasks=0
Total megabyte-milliseconds taken by all map tasks=11769856
Total megabyte-milliseconds taken by all reduce tasks=0
Я не уверен, что мне здесь не хватает, так как я написал другие программы подобным образом, и все они работали.