Я работаю над созданием функции экспорта в Excel для моего приложения. Из-за большого количества строк, большого количества столбцов и значения времени ожидания, равного 5 минутам, я разделил полезную нагрузку задания на несколько одинаковых заданий с разными параметрами, а затем просто добавил новые строки в существующий файл Excel.
Я получаю следующую ошибку:
[2019-07-08 14:13:04] local.ERROR: Test\Jobs\CalculateExport has been attempted too many times or run too long. The job may have previously timed out. {"exception":"[object] (Illuminate\\Queue\\MaxAttemptsExceededException(code: 0): Test\\Jobs\\CalculateExport has been attempted too many times or run too long. The job may have previously timed out. at /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:394)
[stacktrace]
#0 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(314): Illuminate\\Queue\\Worker->markJobAsFailedIfAlreadyExceedsMaxAttempts('beanstalkd', Object(Illuminate\\Queue\\Jobs\\BeanstalkdJob), 1)
#1 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(270): Illuminate\\Queue\\Worker->process('beanstalkd', Object(Illuminate\\Queue\\Jobs\\BeanstalkdJob), Object(Illuminate\\Queue\\WorkerOptions))
#2 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(227): Illuminate\\Queue\\Worker->runJob(Object(Illuminate\\Queue\\Jobs\\BeanstalkdJob), 'beanstalkd', Object(Illuminate\\Queue\\WorkerOptions))
#3 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(101): Illuminate\\Queue\\Worker->runNextJob('beanstalkd', 'default', Object(Illuminate\\Queue\\WorkerOptions))
#4 /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(85): Illuminate\\Queue\\Console\\WorkCommand->runWorker('beanstalkd', 'default')
#5 [internal function]: Illuminate\\Queue\\Console\\WorkCommand->handle()
#6 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(29): call_user_func_array(Array, Array)
#7 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#8 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#9 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(549): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#10 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(183): Illuminate\\Container\\Container->call(Array)
#11 /var/www/html/vendor/symfony/console/Command/Command.php(255): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#12 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(170): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#13 /var/www/html/vendor/symfony/console/Application.php(960): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#14 /var/www/html/vendor/symfony/console/Application.php(255): Symfony\\Component\\Console\\Application->doRunCommand(Object(Illuminate\\Queue\\Console\\WorkCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Ob
ject(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#15 /var/www/html/vendor/symfony/console/Application.php(148): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#16 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Application.php(88): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#17 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(121): Illuminate\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#18 /var/www/html/artisan(35): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#19 {main}
"}
Beanstalkd используется в качестве обработчика очереди. Похоже, никаких исключений не происходит, кроме приведенного выше.
Класс работы выглядит следующим образом:
<?php
namespace Test\Jobs;
use Log;
use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use PanelApi\v2\Export;
class CalculateExport implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
private $export;
private $params;
public $tries = 1;
public $timeout = 0;
/**
* CalculateExport constructor.
* @param Export $export
* @param null $params
*/
public function __construct(Export $export, $params = null)
{
$this->export = $export;
$this->params = $params;
Log::info($this->params);
}
/**
* Execute the job.
*
* @return void
*/
public function handle()
{
if (empty($this->params)) {
$this->export->calculateExport();
$this->delete();
} else {
$skip = isset($this->params['skip']) ? $this->params['skip'] : 0;
$excelFilePath = isset($this->params['excelFilePath']) ? $this->params['excelFilePath'] : null;
$totalCount = isset($this->params['totalCount']) ? $this->params['totalCount'] : null;
$this->export->calculateExport($skip, $excelFilePath, $totalCount);
$this->delete();
}
}
}
Метод расчета экспорта, подобный этому:
public function calculateExport($skip = 0, $excelFilePath = null, $totalCount = null)
{
try {
// FIXME: investigate this more
ini_set('memory_limit', '1024M');
set_time_limit(0);
$this->load('hard_filters_condition', 'test', 'exportAttributes');
$test_id = $this->test->id;
$p = \Test\v2\Test::where('test_id', $test_id);
$testColumns = collect($this->exportAttributes)
->filter(function ($item) {
return $item['attribute_type'] == 'metadata';
})
->map(function ($item) {
return $item['attribute_code'];
})
->toArray();
array_unshift($testColumns, 'email');
array_unshift($testColumns, 'id');
$questionColumnIds = collect($this->exportAttributes)
->filter(function ($item) {
return $item['attribute_type'] == 'question';
})
->map(function ($item) {
return explode('_', $item['attribute_code'])[1];
})
->toArray();
$hard_filter = $this->hard_filters_condition;
if ($hard_filter) {
$p = $p->where(function ($query) use ($hard_filter) {
$query = $hard_filter->getInvitationQuery($query);
});
}
// get the actual questions by ids
$questions = Question::whereIn('id', $questionColumnIds)->get();
$textQuestionIds = implode(',', $questions
->filter(function ($question) {
return $question->type == 'text';
})
->map(function ($question) {
return $question->id;
})
->toArray());
$nonTextQuestionIds = implode(',', $questions
->filter(function ($question) {
return $question->type != 'text';
})
->map(function ($question) {
return $question->id;
})
->toArray());
// test select columns
$testColumnsToSelect = collect($testColumns)
->map(function ($item) {
return "tests.$item";
})
->toArray();
$testCount = empty($totalCount) ? $p->count() : $totalCount;
$LIMIT = 50;
$p = $p->select($testColumnsToSelect)
->skip($skip)
->take($LIMIT);
$testCursor = $p->cursor();
if (empty($totalCount)) {
$this->hard_filtered_count = $testCount;
$this->hard_filtered_at = Carbon::now()->toDateTimeString();
$this->save();
Log::info('HARD FILTERED COUNT:' . $testCount);
}
$PROGRESS_FACTOR = 0.01;
$progressIncrement = round($testCount * $PROGRESS_FACTOR);
$fileName = "";
if (empty($excelFilePath)) {
$bytes = random_bytes(32);
$randomId = bin2hex($bytes);
// 1. Init Excel file
$date = Carbon::now()->toDateString();
$fileName = "Test_export_{$this->id}_" . $randomId . "_$date";
} else {
$fileName = $excelFilePath;
}
$excelHeaders = array_merge($testColumns, collect($questions)
->map(function ($item) {
return $item->code;
})
->toArray());
if (empty($excelFilePath)) {
$counter = $skip;
Log::info('Starting first job for excel file: ' . $excelFilePath . ' with test count: ' . $testCount . ' and questions count ' . $questions->count());
Excel::create($fileName, function ($excel) use ($testCursor, $questions, $progressIncrement, $testCount, $excelHeaders, $skip, &$counter, $nonTextQuestionIds, $textQuestionIds) {
$excel->sheet('Sheet 1', function ($sheet) use ($testCursor, $questions, $progressIncrement, $testCount, $excelHeaders, $skip, &$counter, $nonTextQuestionIds, $textQuestionIds) {
$sheet->prependRow($excelHeaders);
$sheet->row(1, function ($row) {
$row->setFontWeight('bold');
});
$sheet->freezeFirstRow();
foreach ($testCursor as $test) {
$row = $test->toArray();
// get answers to all questions for that user
Log::info('First job - iteration ' . ($counter + 1));
$nonTextAnswers = collect(DB::select(DB::raw("
SELECT pa.question_id, qo.name
FROM test_answers pa, question_options qo
WHERE pa.test_id = :test_id
AND pa.question_id IN ($nonTextQuestionIds)
AND qo.id = pa.choice_option_id;
"), ['test_id' => $test->id]));
$textAnswers = collect(DB::select(DB::raw("
SELECT pa.question_id, pa.text
FROM test_answers pa
WHERE pa.test_id = :test_id
AND pa.question_id IN ($textQuestionIds);
"), ['test_id' => $test->id]));
Log::info('First job - iteration ' . ($counter + 1) . ' gotten test answers');
foreach ($questions as $question) {
if ($question->type != 'text') {
$answers = $nonTextAnswers
->filter(function ($item) use ($question) {
return $item->question_id == $question->id;
})
->map(function ($item) {
return $item->name;
})
->toArray();
$answer = implode(', ', $answers);
$row[$question->code] = $answer;
} else {
$answer = array_values($textAnswers
->filter(function ($item) use ($question) {
return $item->question_id == $question->id;
})
->toArray());
if (count($answer) > 0) {
$row[$question->code] = $answer[0]->text;
}
}
}
$excelRow = [];
foreach ($excelHeaders as $header) {
array_push($excelRow, isset($row[$header]) ? $row[$header] : "");
}
Log::info('First job - iteration ' . ($counter + 1) . ' appending excel row');
$sheet->appendRow($excelRow);
$counter++;
}
$this->calculated_percent = round($counter / $testCount * 100, 2);
Log::info('First job - iteration ' . ($counter + 1) . ' updating export row in db');
$this->save();
});
})->store('xlsx');
if ($counter < $testCount) {
// dispatch next job
Log::info("Processed $counter tests, starting next job");
dispatch(new CalculateExport($this, [
'skip' => $counter,
'totalCount' => $testCount,
'excelFilePath' => $fileName
]));
return;
}
} else {
$ds = DIRECTORY_SEPARATOR;
$counter = $skip;
Excel::load(storage_path('exports') . $ds . $fileName . '.xlsx', function ($reader) use ($testCursor, $questions, $excelHeaders, $progressIncrement, $testCount, &$counter, $nonTextQuestionIds, $textQuestionIds) {
$reader->sheet('Sheet 1', function ($sheet) use ($testCursor, $questions, $excelHeaders, $progressIncrement, $testCount, &$counter, $nonTextQuestionIds, $textQuestionIds) {
foreach ($testCursor as $test) {
$row = $test->toArray();
// get answers to all questions for that user
Log::info('Update job - iteration ' . ($counter + 1));
$nonTextAnswers = collect(DB::select(DB::raw("
SELECT pa.question_id, qo.name
FROM test_answers pa, question_options qo
WHERE pa.test_id = :test_id
AND pa.question_id IN ($nonTextQuestionIds)
AND qo.id = pa.choice_option_id;
"), ['test_id' => $test->id]));
$textAnswers = collect(DB::select(DB::raw("
SELECT pa.question_id, pa.text
FROM test_answers pa
WHERE pa.ptest_id = :test_id
AND pa.question_id IN ($textQuestionIds);
"), ['test_id' => $test->id]));
Log::info('Update job - iteration ' . ($counter + 1) . ' gotten answers');
foreach ($questions as $question) {
if ($question->type != 'text') {
$answers = $nonTextAnswers
->filter(function ($item) use ($question) {
return $item->question_id == $question->id;
})
->map(function ($item) {
return $item->name;
})
->toArray();
$answer = implode(', ', $answers);
$row[$question->code] = $answer;
} else {
$answer = array_values($textAnswers
->filter(function ($item) use ($question) {
return $item->question_id == $question->id;
})
->toArray());
if (count($answer) > 0) {
$row[$question->code] = $answer[0]->text;
}
}
}
$excelRow = [];
foreach ($excelHeaders as $header) {
array_push($excelRow, isset($row[$header]) ? $row[$header] : "");
}
Log::info('Update job - iteration ' . ($counter + 1) . ' appending excel row');
$sheet->appendRow($excelRow);
$counter++;
}
$this->calculated_percent = round($counter / $testCount * 100, 2);
Log::info('Update job - iteration ' . ($counter + 1) . ' update export row in db');
$this->save();
});
})->store('xlsx');
if ($counter < $testCount) {
// dispatch next job
Log::info("Processed $counter tests, starting next job");
CalculateExport::dispatch($this, [
'skip' => $counter,
'totalCount' => $testCount,
'excelFilePath' => $fileName
]);
dispatch(new CalculateExport($this, [
'skip' => $counter,
'totalCount' => $testCount,
'excelFilePath' => $fileName
]));
return;
}
}
Log::info("Processing Excel finished, uploading to s3");
// upload to S3
$ds = DIRECTORY_SEPARATOR;
$diskPath = storage_path('exports') . $ds . $fileName . '.xlsx';
$contents = file_get_contents($diskPath);
Storage::disk('s3_circle_public')->put($fileName . '.xlsx', $contents);
$path = Storage::disk('s3_circle_public')->url($fileName . '.xlsx');
$this->export_url = $path;
$this->save();
// update percent
$this->calculated_percent = 100;
$this->calculated_at = Carbon::now()->toDateTimeString();
$this->save();
} catch (\Exception $e) {
Log::error($e->getMessage());
Log::error($e->getTraceAsString());
}
}
Я попытался изменить тайм-аут задания для самого задания, изменив CalculateExport :: dispatch с глобальной отправкой, установив попытку на единицу. В консоли beastalkd я вижу, что задания отправляются одна за другой. Но как только 5 минут достигнуто, оно заканчивается Я подозреваю, что он так или иначе никогда не завершал предыдущее задание, поэтому я добавил задание на удаление.
Спасибо!