Разница между JavaCV 1.5 и 1.4.4 в индексаторе, использующем в обнаружении объектов с Yolo - PullRequest
Новая
       18

Разница между JavaCV 1.5 и 1.4.4 в индексаторе, использующем в обнаружении объектов с Yolo

0 голосов
/ 09 июня 2019

Моя среда:

Ubuntu 18.04 LTS 64bit

Java: OpenJDK8 + Groovy 2.5.6

Версия JavaCV: 1.4.4 против 1.5

Yolo версия: Yolo v3 крошечный

Я использую open_dnn в JavaCV для обнаружения объектов с помощью Yolo. Мой код хорошо работает с JavaCV 1.4.4. и может обнаружить объект без каких-либо проблем. Но с тем же кодом (настроив пакеты импорта для соответствия JavaCV 1.5), той же конфигурацией yolo, теми же весами, тем же тестовым изображением, он ничего не обнаруживает. Я пробовал несколько раз, и результаты совпадают.

Я просто перечисляю соответствующие коды и регистрируюсь как ниже: Для JavaCV 1.4.4 код выглядит следующим образом: 

import groovy.util.logging.Slf4j
import org.bytedeco.javacpp.FloatPointer
import org.bytedeco.javacpp.IntPointer
import org.bytedeco.javacpp.indexer.FloatIndexer
import org.bytedeco.opencv.global.opencv_core
import org.bytedeco.opencv.global.opencv_dnn
import org.bytedeco.opencv.global.opencv_imgproc
import org.bytedeco.javacv.Java2DFrameUtils

import  org.bytedeco.opencv.opencv_dnn.Net
import  org.bytedeco.opencv.opencv_core.Rect
import  org.bytedeco.opencv.opencv_core.RectVector
import  org.bytedeco.opencv.opencv_core.Scalar
import  org.bytedeco.opencv.opencv_core.Size
import  org.bytedeco.opencv.opencv_core.IplImage
import  org.bytedeco.opencv.opencv_core.Mat
import  org.bytedeco.opencv.opencv_core.MatVector

import org.bytedeco.javacv.Frame
import vehicle.detectedobject.DetectedVehicle

@Slf4j
class YoloV3TinyVehicleDetector  {
    private Net net
    private List<String> classNameList
    private float confThreshold = 0.5; // Confidence threshold
    private float nmsThreshold = 0.4;
    private final static Scalar scalar = new Scalar(0)
    private final static Size sz = new Size(416, 416)
    private final static Size sz1 = new Size(1280, 720)
    private final static float scale = 1F/255F
    private final static boolean swapRB = true
    private Rect[] lastBoxArray
    private boolean foundNothing = false

    public YoloV3TinyVehicleDetector(
                String moduleConfigFilename, String moduleWeightFilename,
                int backendId, int targetId,
                List<String> classNameList
    ) {
        net = opencv_dnn.readNetFromDarknet(moduleConfigFilename, moduleWeightFilename)
        net.setPreferableBackend(opencv_dnn.DNN_BACKEND_DEFAULT)
        net.setPreferableTarget(opencv_dnn.DNN_TARGET_CPU)
        this.classNameList = classNameList
    }    

    List<DetectedVehicle> detectVehicles(Mat frame) {

        Mat inputBlob = opencv_dnn.blobFromImage(frame, scale, sz,scalar, swapRB, false, opencv_core.CV_32F)

        net.setInput(inputBlob)
        MatVector outputBlobs = new MatVector()
        net.forward(outputBlobs, net.getLayerNames())

        List<Integer> classIds =[]
        List<Float> confidences = []
        List<Rect> boxes = []
        List<DetectedVehicle> detectedVehicleList = []
        FloatIndexer indexer
        for(int ii = 0; ii < outputBlobs.size(); ii++) {
            Mat detectionMat = outputBlobs.get(ii)
            if (detectionMat.empty()) {
                foundNothing = true
                log.debug("detectVehicles(): No object detected!")
                return detectedVehicleList
            }
            int rowCount = detectionMat.rows()
            log.debug("detectVehicles(): ii[${ii}]. detectionMat.cols(), rows()=${detectionMat.cols()},${rowCount}")
            if (rowCount <=0) continue
            if (rowCount > 0) {
                indexer = detectionMat.createIndexer()

            }
            for(int i = 0; i < rowCount; i++) {
                int probability_index = 5;
                int size = (int) (detectionMat.cols() * detectionMat.channels());
                float[] data = new float[size]
                indexer.get(i, 0, data)
                log.debug("detectVehicles(): i[${i}/${detectionMat.rows()}]:size:${size}, data:"+data) // <-- here is the last line of output
            ...
            ...
            ...
    }
    ...
    ...
}

сообщения журнала для JavaCV 1.4.4 приведены ниже. Различная часть журнала между двумя кодами находится в последней строке вывода 

> Task :TestPlayWithYoloV3.main()
detectVehicles(): image wxh=800x480
detectVehicles(): mat wxh=800x480
19/06/09 06:06:41.793 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[0]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.795 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[1]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.795 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[2]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.795 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[3]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.795 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[4]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.795 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[5]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.795 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[6]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.795 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[7]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.795 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[8]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.795 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[9]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.795 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[10]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.795 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[11]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.795 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[12]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.796 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[13]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.796 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[14]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.796 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[15]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.796 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[16]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[17]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[18]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[19]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[20]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[21]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[22]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[23]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[24]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[25]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[26]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[27]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[28]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[29]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[30]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[31]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[32]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.797 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[33]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.798 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[34]. detectionMat.cols(), rows()=-1,-1
19/06/09 06:06:41.798 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[35]. detectionMat.cols(), rows()=12,507
19/06/09 06:06:41.817 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): i[0/507]:size:12, data:[0.03504561, 0.035271123, 0.03801521, 0.20626219, 1.4282074E-27, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
19/06/09 06:06:41.818 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): i[1/507]:size:12, data:[0.043720312, 0.027087418, 0.17653388, 0.10258253, 5.213337E-34, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

Для JavaCV 1.5 код приведен ниже (как я уже говорил выше, без изменений, кроме настройки импорта 

import groovy.util.logging.Slf4j
import org.bytedeco.javacpp.FloatPointer
import org.bytedeco.javacpp.IntPointer
import org.bytedeco.javacpp.indexer.FloatIndexer
import org.bytedeco.opencv.global.opencv_core
import org.bytedeco.opencv.global.opencv_dnn
import org.bytedeco.opencv.global.opencv_imgproc
import org.bytedeco.javacv.Java2DFrameUtils

import  org.bytedeco.opencv.opencv_dnn.Net
import  org.bytedeco.opencv.opencv_core.Rect
import  org.bytedeco.opencv.opencv_core.RectVector
import  org.bytedeco.opencv.opencv_core.Scalar
import  org.bytedeco.opencv.opencv_core.Size
import  org.bytedeco.opencv.opencv_core.IplImage
import  org.bytedeco.opencv.opencv_core.Mat
import  org.bytedeco.opencv.opencv_core.MatVector

import org.bytedeco.javacv.Frame
import vehicle.detectedobject.DetectedVehicle

import java.awt.image.BufferedImage

@Slf4j
class YoloV3TinyVehicleDetector  {
    private Net net
    private List<String> classNameList
    private float confThreshold = 0.5; // Confidence threshold
    private float nmsThreshold = 0.4;
    private final static Scalar scalar = new Scalar(0)
    private final static Size sz = new Size(416, 416)
    private final static Size sz1 = new Size(1280, 720)
    private final static float scale = 1F/255F
    private final static boolean swapRB = true
    private Rect[] lastBoxArray
    private boolean foundNothing = false

    public YoloV3TinyVehicleDetector(
            String moduleConfigFilename, String moduleWeightFilename,
            int backendId, int targetId,
            List<String> classNameList
    ) {
        net = opencv_dnn.readNetFromDarknet(moduleConfigFilename, moduleWeightFilename)
        net.setPreferableBackend(opencv_dnn.DNN_BACKEND_DEFAULT)
        net.setPreferableTarget(opencv_dnn.DNN_TARGET_CPU)
        this.classNameList = classNameList
    }


    List<DetectedVehicle> detectVehicles(Mat frame) {

        log.debug("detectVehicles(): im wxh=${frame.cols()}x${frame.rows()}")
        Mat inputBlob = opencv_dnn.blobFromImage(frame, scale, sz,scalar, swapRB, false, opencv_core.CV_32F)
        net.setInput(inputBlob)
        MatVector outputBlobs = new MatVector()
        net.forward(outputBlobs, net.getLayerNames())
        List<Integer> classIds =[]
        List<Float> confidences = []
        List<Rect> boxes = []
        List<DetectedVehicle> detectedVehicleList = []
        FloatIndexer indexer
        log.info("outputBlobs.size()=" + outputBlobs.size())
        for(int ii = 0; ii < outputBlobs.size(); ii++) {
            Mat detectionMat = outputBlobs.get(ii)
            if (detectionMat.empty()) {
                foundNothing = true
                log.debug("detectVehicles(): detectionMat is empty, so no object detected!")
                return detectedVehicleList
            }
            int rowCount = detectionMat.rows()
            log.debug("detectVehicles(): ii[${ii}]. detectionMat.cols(),rows()=${detectionMat.cols()}, ${rowCount}")
            if (rowCount <=0) continue
            if (rowCount > 0) {
                indexer = detectionMat.createIndexer()
            }
            for(int i = 0; i < rowCount; i++) {
                int probability_index = 5;
                int size = (int) (detectionMat.cols() * detectionMat.channels());
                float[] data = new float[size]
                indexer.get(i, 0, data)
                log.debug("detectVehicles(): i[${i}/${detectionMat.rows()}]:size:${size}, data:"+data) // <-- here is the last line of output
                ...
                ...
                ...
    }
}

Вывод журнала для моего детектора JavaCV1.5: 

detectVehicles(): image wxh=800x480
detectVehicles(): mat wxh=800x480
19/06/08 18:07:23.553 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): im wxh=800x480
19/06/08 18:07:23.976 INFO YoloV3TinyVehicleDetector(): outputBlobs.size()=48
19/06/08 18:07:23.979 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[0]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.979 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[1]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.979 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[2]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.979 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[3]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.979 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[4]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.979 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[5]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.980 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[6]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.980 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[7]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.980 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[8]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.980 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[9]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.980 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[10]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.980 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[11]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.980 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[12]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.980 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[13]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.981 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[14]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.981 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[15]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.981 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[16]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.981 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[17]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.981 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[18]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.981 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[19]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.982 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[20]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.982 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[21]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.982 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[22]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.982 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[23]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.982 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[24]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.982 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[25]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.982 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[26]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.982 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[27]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.982 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[28]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.982 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[29]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.982 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[30]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.982 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[31]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.982 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[32]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.983 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[33]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.983 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[34]. detectionMat.cols(),rows()=-1, -1
19/06/08 18:07:23.983 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): ii[35]. detectionMat.cols(),rows()=12, 507
19/06/08 18:07:23.997 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): i[0/507]:size:12, data:[NaN, NaN, NaN, NaN, NaN, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
19/06/08 18:07:23.997 DEBUG YoloV3TinyVehicleDetector(): detectVehicles(): i[1/507]:size:12, data:[NaN, NaN, NaN, NaN, NaN, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

Однако по тому же алгоритму я просто использую opencv 4.1.0 (не JavaCV), он все еще работает и может обнаруживать объекты. Поэтому я сомневаюсь, что в JavaCV 1.5 есть что-то другое.

Или я пропустил что-то более новое в JavaCV 1.5?

Спасибо за вашу помощь.

...