Клавиша разделения текста в редукторе (Hadoop MapReduce) - PullRequest
Новая
       21

Клавиша разделения текста в редукторе (Hadoop MapReduce)

0 голосов
/ 11 мая 2018

У меня есть такая проблема: я делаю приложение уменьшения карты для проекта больших данных. 

So, i must split a string in my reduce method. My string's type is: 0006641040-2003.

So, if i split with the method String.split(), i can't take the secondo part of my string after split with index 1. I recived  java.lang.ArrayIndexOutOfBoundsException: 1

Но я могу взять часть строки после разделения с индексом 0.

Почему?

Спасибо. 

    Hi, I have this problem: i'm doing a map-reduce application for a big data project.

So, i must split a string in my reduce method. My string's type is: 0006641040-2003.

So, if i split with the method String.split(), i can't take the secondo part of my string after split with index 1. I recived  java.lang.ArrayIndexOutOfBoundsException: 1

Но я могу взять часть строки после разделения с индексом 0.

Почему?

Спасибо. 

public class ProductReducerClass extends Reducer<Text, MapWritable, Text, MapWritable> {

MapWritable mw1 = new MapWritable();
String[] out;
public void reduce(Text key, Iterable<MapWritable> values, Context context) 
        throws IOException, InterruptedException {
    out = key.toString().split("-");
    for (MapWritable m : values) {
        context.write(new Text(out[1]),m);

    }



}

} 

public class ProductMapperClass extends Mapper<LongWritable, Text, Text, MapWritable> {
private static final IntWritable one = new IntWritable(40);
private static final MapWritable missing = new MapWritable();
MapWritable mw = new MapWritable();
ArrayWritable aw = new ArrayWritable(IntWritable.class);
int [] score;
public void map(LongWritable key, Text value, Context context)
        throws IOException, InterruptedException {
    missing.put(new LongWritable(0), new IntWritable(0));
    String line = value.toString();
    String[] fields = line.split(",");
    try {

        Calendar date=Calendar.getInstance();
        date.setTimeInMillis(Long.parseLong((fields[7]))*1000); 
        long anno=date.get(Calendar.YEAR);
        if(anno>=2003 && anno<=2012) {
            mw.put(new Text("score"), new IntWritable(Integer.parseInt(fields[6])));
            context.write(new Text(fields[1].concat("-").concat(Long.toString(anno))), mw);
        }
    }
    catch (NumberFormatException e) {
        context.write(new Text("perso"),missing);
    }


}

} 

public class MainClass {

public static void main (String args[]) throws Exception {

    Job job = new Job(new Configuration(), "MainClass");

    job.setJarByClass(MainClass.class);
    job.setMapperClass(ProductMapperClass.class);
    job.setReducerClass(ProductReducerClass.class);


    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));


    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(MapWritable.class);

    job.waitForCompletion(true);


}

}

...