为什么我不能用 java opencv 正确 select 矩形区域?

Why i can't correctly select the rectangular areas with java opencv?

通过 opencv 我想 select 像这样的图像的收据区域:

https://i.imgur.com/BnXzWPe.jpg

然后裁剪它们并将它们保存在单独的文件中。

图像是在 1200 Dpi 下扫描的,方法是在收据上叠加黑色纸板以更好地识别边缘。

到目前为止我得到了以下结果:

https://i.imgur.com/tRaPocd.jpg

您可以在其中看到包含收据的红色边界框矩形和具有最小面积的绿色矩形。

尽管使用 Canny 正确识别了边缘:

https://i.imgur.com/O5o2DWr.jpg

中心图像绿色区域不正确,我不明白为什么。

这是源代码的摘录,您可以在此处找到完整的源代码:https://pastebin.com/NNc18pzA

Imgproc.resize (srcMat, srcMat, new Size (0,0), 0.5, 0.5, Imgproc.INTER_AREA);
Imgproc.cvtColor (srcMat, grayMat, Imgproc.COLOR_BGR2GRAY);
Imgproc.threshold (grayMat, grayMat, 177, 200, Imgproc.THRESH_BINARY);
Imgproc.GaussianBlur (grayMat, blurredMat, new Size (21,21), 0, 0, Core.BORDER_DEFAULT); // 3,3, 9,9 15,15, ....

Mat rectKernel = Imgproc.getStructuringElement (Imgproc.MORPH_RECT, new Size (21,21));

Imgproc.dilate (blurredMat, dilatedMat, rectKernel, new Point (0,0), 1);
Imgproc.Canny (dilatedMat, cannyMat, 100,200.3);

List <MatOfPoint> contours = new ArrayList <MatOfPoint> ();
final Mat hierarchy = new Mat ();

Imgproc.findContours (cannyMat, contours, hierarchy, Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_SIMPLE);

这些是在中间阶段处理的图像:

变灰: https://i.imgur.com/ufeIMQT.jpg

模糊: https://i.imgur.com/lQQER6T.jpg

扩张: https://i.imgur.com/sHmfZyI.jpg

这是使用的opencv版本:

<dependency>
    <groupId>org.openpnp</groupId>
    <artifactId>opencv</artifactId>
    <version>4.3.0-2</version>
</dependency> 

请帮帮我。

谢谢大家

主要问题是在 Canny 边缘检测之后应用 findContours

Canny 算子在外部轮廓中创建小间隙:

结合 approxPolyDP,我们得到了奇怪的结果。


简单的解决方案是跳过 Canny 运算符并在 dilatedMat 上应用 findContours

Imgproc.findContours(cannyMat, contours, hierarchy, Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_SIMPLE);替换为:

Imgproc.findContours(dilatedMat, contours, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);

注意:建议使用 RETR_EXTERNAL,但在这种情况下不是必须的。


一个小问题是噪声导致的小轮廓。
我们可以使用 opening 形态学操作来去除小(噪声)轮廓(最好在扩张之前应用)。

另一个简单的解决方案是跳过小面积的轮廓。
我发现低于 10000 像素的区域可能被视为“小”区域。

在for循环中添加一个if语句:

double area = Imgproc.contourArea(cnt);
if (area > 10000) { // Exclude small contours (noise)...

输出:


我把代码转成了Python(JAVA代码保留在注释里)

完整代码示例:

import cv2
import numpy as np

srcMat = cv2.imread('receipts.jpg')  # Read input image
cv2.resize(srcMat, (0, 0), srcMat, 0.1, 0.1, cv2.INTER_AREA)  # Imgproc.resize (srcMat, srcMat, new Size (0,0), 0.5, 0.5, Imgproc.INTER_AREA);
grayMat = cv2.cvtColor(srcMat, cv2.COLOR_BGR2GRAY)  # grayMat = Imgproc.cvtColor (srcMat, grayMat, Imgproc.COLOR_BGR2GRAY);
grayMat = cv2.threshold(grayMat, 177, 200, cv2.THRESH_BINARY)[1]  # Imgproc.threshold (grayMat, grayMat, 177, 200, Imgproc.THRESH_BINARY);
blurredMat = cv2.GaussianBlur(grayMat, ksize=(21, 21), sigmaX=0, sigmaY=0)  # Imgproc.GaussianBlur (grayMat, blurredMat, new Size (21,21), 0, 0, Core.BORDER_DEFAULT); // 3,3, 9,9 15,15, ....

rectKernel = cv2.getStructuringElement(cv2.MORPH_RECT, (21, 21))  # Mat rectKernel = Imgproc.getStructuringElement (Imgproc.MORPH_RECT, new Size (21,21));

dilatedMat = cv2.dilate(blurredMat, rectKernel)  # Imgproc.dilate (blurredMat, dilatedMat, rectKernel, new Point (0,0), 1);
cannyMat = cv2.Canny(dilatedMat, 100, 200.3)  # Imgproc.Canny (dilatedMat, cannyMat, 100,200.3);

# List <MatOfPoint> contours = new ArrayList <MatOfPoint> ();
# final Mat hierarchy = new Mat ();
#contours, hierarchy = cv2.findContours(cannyMat, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)  # Imgproc.findContours(cannyMat, contours, hierarchy, Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_SIMPLE);

# Find contours over dilatedMat (the Canny operator creates gaps in the external contour).
contours, hierarchy = cv2.findContours(dilatedMat, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)  # Imgproc.findContours(dilatedMat, contours, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);


# Use cv2.RETR_EXTERNAL instead of cv2.RETR_TREE
#contours, hierarchy = cv2.findContours(cannyMat, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)  # Imgproc.findContours(cannyMat, contours, hierarchy, Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_SIMPLE);

cntImg = srcMat.copy()
# Mark best_cnt with green line - used for testing
cv2.drawContours(cntImg, contours, -1, (0, 255, 0), 20)

# for(MatOfPoint cnt : contours) {
for cnt  in contours:
    area = cv2.contourArea(cnt)

    # Exclude small contours (noise)
    if area > 10000:
        mop2f = cv2.approxPolyDP(cnt, 0.02 * cv2.arcLength(cnt, True), True)  # Imgproc.approxPolyDP(mop2f, mop2f, 0.02*Imgproc.arcLength(mop2f, true), true);
        rr = cv2.minAreaRect(mop2f) #  RotatedRect rr = Imgproc.minAreaRect(mop2f);
        m = cv2.boxPoints(rr) # Imgproc.boxPoints(rr, m);  

        # Point[] rectPoints = new Point[4];
        # rr.points(rectPoints);
        rectPoints = np.int0(m)  # Convert all coordinates floating point values to int

        # for (int j = 0; j < 4; ++j) {
        #     Imgproc.line(srcMat, rectPoints[j], rectPoints[(j + 1) % 4], new Scalar(0,255,0), 20); }    
        #for j in range(4):
        #    cv2.line(srcMat, tuple(rectPoints[j]), tuple(rectPoints[(j + 1) % 4]), (0, 255, 0), 20)  # Imgproc.line(srcMat, rectPoints[j], rectPoints[(j + 1) % 4], new Scalar(0,255,0), 20);

        # Deaw the rectangles using drawContours instead of drawing lines
        # 
        cv2.drawContours(srcMat, [rectPoints], 0, (0, 255, 0), 20)

        boundingRect = cv2.boundingRect(cnt)  # Rect boundingRect = Imgproc.boundingRect(cnt);
        cv2.rectangle(srcMat, boundingRect, (0, 0, 255), 20)  # Imgproc.rectangle(srcMat, boundingRect, new Scalar(0,0,255),20); //scalar not is RGB but BGR !

简化实施(建议):

  • 使用 Imgproc.THRESH_OTSU 进行自动阈值选择。
  • 不需要申请GaussianBlur
  • 使用闭合而不是扩张。
  • 不需要申请Canny

建议的JAVA代码:

package myproject; //package it.neo7bf;

import java.util.ArrayList;
import java.util.List;

import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.core.MatOfPoint;
import org.opencv.core.MatOfPoint2f;
import org.opencv.core.Point;
import org.opencv.core.Rect;
import org.opencv.core.RotatedRect;
import org.opencv.core.Scalar;
import org.opencv.core.Size;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.imgproc.Imgproc;

//import nu.pattern.OpenCV;

public class SeparationTest3 {
    
    static { System.loadLibrary(Core.NATIVE_LIBRARY_NAME); }
    
    static class I {
        public String name;
        public int v;
        I(String name, int v) {
            this.name = name;
            this.v = v;
        }
    }
    
    public static void cannyTest() {

        List<I> images = List.of(
            new I("2022-04-16_085329",3)
        );

        for(I image : images) {

            Mat srcMat = Imgcodecs.imread("C:\ProgettoScontrino\scontrini\campioni-test\test-separazione\"+image.name+".jpg");
            
            Mat grayMat = new Mat();
            //Mat blurredMat = new Mat();
            Mat dilatedMat = new Mat();
            //Mat cannyMat = new Mat();
            

            Imgproc.resize(srcMat, srcMat, new Size(0,0), 0.5, 0.5, Imgproc.INTER_AREA);
            Imgproc.cvtColor(srcMat, grayMat, Imgproc.COLOR_BGR2GRAY);
            //Imgproc.threshold(grayMat, grayMat, 177, 200, Imgproc.THRESH_BINARY);
            Imgproc.threshold(grayMat, grayMat, 0, 255, Imgproc.THRESH_OTSU);  //Use automatic threshold
            
            //There is no need to blur the image after threshold
            //Imgproc.GaussianBlur(grayMat, blurredMat, new Size(21,21),0, 0,Core.BORDER_DEFAULT); //3,3, 9,9 15,15,....
                
            Mat rectKernel = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(21,21));
            
            //Imgproc.dilate(blurredMat, dilatedMat, rectKernel, new Point(0,0),1);
            Imgproc.morphologyEx(grayMat, dilatedMat, Imgproc.MORPH_CLOSE, rectKernel);  // Use closing instead of dilate
            //Imgproc.Canny(dilatedMat,cannyMat,100,200,3); //No need for Canny
            
            List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
            final Mat hierarchy = new Mat();
    
            //Imgproc.findContours(cannyMat, contours, hierarchy, Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_SIMPLE);
            
            Imgproc.findContours(dilatedMat, contours, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);
            
            //contours = getMaxContours(contours,image.v);
            
            for(MatOfPoint cnt : contours) {
                double area = Imgproc.contourArea(cnt);
                if (area > 10000) //Ignore small contours
                {
                    MatOfPoint2f mop2f = new MatOfPoint2f(cnt.toArray());
                    Imgproc.approxPolyDP(mop2f, mop2f, 0.02*Imgproc.arcLength(mop2f, true), true);
                    RotatedRect rr = Imgproc.minAreaRect(mop2f);
                    MatOfPoint m = new MatOfPoint();                    
                    Imgproc.boxPoints(rr, m);
                    Point[] rectPoints = new Point[4];          
                    rr.points(rectPoints);
                    for (int j = 0; j < 4; ++j) {
                      Imgproc.line(srcMat, rectPoints[j], rectPoints[(j + 1) % 4], new Scalar(0,255,0), 20); 
                    }

                    //BoundingBox
                    Rect boundingRect = Imgproc.boundingRect(cnt);
                    Imgproc.rectangle(srcMat, boundingRect, new Scalar(0,0,255),20); //scalar not is RGB but BGR !
                }
            }
                    
            //C:\ProgettoScontrino\scontrini\campioni-test\test-separazione\output\
            Imgcodecs.imwrite("C:\ProgettoScontrino\scontrini\campioni-test\test-separazione\output\"+image.name+"gray.jpg", grayMat);
            //Imgcodecs.imwrite("C:\ProgettoScontrino\scontrini\campioni-test\test-separazione\output\"+image.name+"blurred.jpg", blurredMat);
            Imgcodecs.imwrite("C:\ProgettoScontrino\scontrini\campioni-test\test-separazione\output\"+image.name+"dilated.jpg", dilatedMat);
            //Imgcodecs.imwrite("C:\ProgettoScontrino\scontrini\campioni-test\test-separazione\output\"+image.name+"canny.jpg", cannyMat);
            Imgcodecs.imwrite("C:\ProgettoScontrino\scontrini\campioni-test\test-separazione\output\"+image.name+"contours.jpg", srcMat);
        }
    }
    
    public static void main(String[] args) {
        cannyTest();
    }
}