如何为 Metal Performance Shaders 设置 MTLTexture 和 MTLBuffers 找到关键点
How to setup MTLTexture and MTLBuffers for Metal Performance Shaders Find Keypoints
问题
我是第一次尝试性能着色器，遇到了运行时问题。 MTLTexture MTKTextureLoader returns 似乎与 Metal Performance Shaders 的 MPSImageFindKeypoints 编码器不兼容。
到目前为止，我发现的唯一提示来自@warrenm 在 MPS 上的示例代码，它像我一样指定了 MTKTextureLoaderOptions。我没有在文档中找到任何其他提及。
非常感谢任何帮助。
错误
/BuildRoot/Library/Caches/com.apple.xbs/Sources/MetalImage/MetalImage-121.0.2/MPSImage/Filters/MPSKeypoint.mm:166: failed assertion `Source 0x282ce8fc0 texture type (80) is unsupported
其中 0x282ce8fc0 是来自纹理加载器的 MTLTexture。
据我所知，没有 MTLTexture 类型 80，枚举范围最多为 8（不是十六进制）。
代码
CGFloat w = CGImageGetWidth(_image);
CGFloat h = CGImageGetHeight(_image);
id<MTLDevice> device = MTLCreateSystemDefaultDevice();
id<MTLCommandQueue> commandQueue = [device newCommandQueue];
NSDictionary* textureOptions = @{ MTKTextureLoaderOptionSRGB: [[NSNumber alloc] initWithBool:NO] };
id<MTLTexture> texture = [[[MTKTextureLoader alloc] initWithDevice:device] newTextureWithCGImage:_image
options:textureOptions
error:nil];
id<MTLBuffer> keypointDataBuffer;
id<MTLBuffer> keypointCountBuffer;
MTLRegion region = MTLRegionMake2D(0, 0, w, h);
id<MTLCommandBuffer> commandBuffer = [commandQueue commandBuffer];
MPSImageKeypointRangeInfo rangeInfo = {100,0.5};
MPSImageFindKeypoints* imageFindKeypoints = [[MPSImageFindKeypoints alloc] initWithDevice:device
info:&rangeInfo];
[imageFindKeypoints encodeToCommandBuffer:commandBuffer
sourceTexture:texture
regions:®ion
numberOfRegions:1
keypointCountBuffer:keypointCountBuffer
keypointCountBufferOffset:0
keypointDataBuffer:keypointDataBuffer
keypointDataBufferOffset:0];
[commandBuffer commit];
NSLog(keypointCountBuffer);
NSLog(keypointDataBuffer);
编辑
将我的图像转换为正确的像素格式后，我现在正在初始化缓冲区，如下所示：
id<MTLBuffer> keypointDataBuffer = [device newBufferWithLength:maxKeypoints*(sizeof(MPSImageKeypointData)) options:MTLResourceOptionCPUCacheModeDefault];
id<MTLBuffer> keypointCountBuffer = [device newBufferWithLength:sizeof(int) options:MTLResourceOptionCPUCacheModeDefault];
没有错误了。但是我现在怎么看内容呢？
((MPSImageKeypointData*)[keypointDataBuffer contents])[0].keypointCoordinate returns (0,0) 用于所有索引。我也不知道如何阅读 keypointsCountBuffer。转换为 int 值的缓冲区内容显示的值高于定义的 maxKeypoints。我看不到文档在哪里说明计数缓冲区的格式。
最后代码是 运行，为了完整起见，我认为我应该 post 整个代码作为答案
代码
id<MTLDevice> device = MTLCreateSystemDefaultDevice();
id<MTLCommandQueue> commandQueue = [device newCommandQueue];
// init textures
NSDictionary* textureOptions = @{ MTKTextureLoaderOptionSRGB: [[NSNumber alloc] initWithBool:NO] };
id<MTLTexture> texture = [[[MTKTextureLoader alloc] initWithDevice:device] newTextureWithCGImage:_lopoImage
options:textureOptions
error:nil];
MTLTextureDescriptor *descriptor = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:(MTLPixelFormatR8Unorm) width:w height:h mipmapped:NO];
descriptor.usage = (MTLTextureUsageShaderRead | MTLTextureUsageShaderWrite);
id<MTLTexture> unormTexture = [device newTextureWithDescriptor:descriptor];
// init arrays and buffers for keypoint finder
int maxKeypoints = w*h;
id<MTLBuffer> keypointDataBuffer = [device newBufferWithLength:sizeof(MPSImageKeypointData)*maxKeypoints options:MTLResourceOptionCPUCacheModeWriteCombined];
id<MTLBuffer> keypointCountBuffer = [device newBufferWithLength:sizeof(int) options:MTLResourceOptionCPUCacheModeWriteCombined];
MTLRegion region = MTLRegionMake2D(0, 0, w, h);
// init colorspace converter
CGColorSpaceRef srcColorSpace = CGColorSpaceCreateWithName(kCGColorSpaceSRGB);
CGColorSpaceRef dstColorSpace = CGColorSpaceCreateWithName(kCGColorSpaceLinearGray);
CGColorConversionInfoRef conversionInfo = CGColorConversionInfoCreate(srcColorSpace, dstColorSpace);
MPSImageConversion *conversion = [[MPSImageConversion alloc] initWithDevice:device
srcAlpha:(MPSAlphaTypeAlphaIsOne)
destAlpha:(MPSAlphaTypeNonPremultiplied)
backgroundColor:nil
conversionInfo:conversionInfo];
// init keypoint finder
MPSImageKeypointRangeInfo rangeInfo = {maxKeypoints,0.75};
MPSImageFindKeypoints* imageFindKeypoints = [[MPSImageFindKeypoints alloc] initWithDevice:device
info:&rangeInfo];
// encode command buffer
id<MTLCommandBuffer> commandBuffer = [commandQueue commandBuffer];
[conversion encodeToCommandBuffer:commandBuffer sourceTexture:texture destinationTexture:unormTexture];
[imageFindKeypoints encodeToCommandBuffer:commandBuffer
sourceTexture:unormTexture
regions:®ion
numberOfRegions:1
keypointCountBuffer:keypointCountBuffer
keypointCountBufferOffset:0
keypointDataBuffer:keypointDataBuffer
keypointDataBufferOffset:0];
// run command buffer
[commandBuffer commit];
[commandBuffer waitUntilCompleted];
// read keypoints
int count = ((int*)[keypointCountBuffer contents])[0];
MPSImageKeypointData* keypointDataArray = ((MPSImageKeypointData*)[keypointDataBuffer contents]);
for (int i = 0 ; i<count;i++) {
simd_ushort2 coordinate = keypointDataArray[i].keypointCoordinate;
NSLog(@"color:%f | at:(%u,%u)", keypointDataArray[i].keypointColorValue, coordinate[0], coordinate[1] );
}
我想应该有一种更聪明的方法来使用 [device newBufferWithBytesNoCopy] 分配关键点缓冲区，这样您就不需要将内容复制回分配的数组中。它只是没有弄清楚正确对齐缓冲区。
另外我要提一下，我想通常在任何类型的特征检测之后你都会有一个灰度纹理，这样就不需要图像转换部分了。
问题
我是第一次尝试性能着色器，遇到了运行时问题。 MTLTexture MTKTextureLoader returns 似乎与 Metal Performance Shaders 的 MPSImageFindKeypoints 编码器不兼容。
到目前为止，我发现的唯一提示来自@warrenm 在 MPS 上的示例代码，它像我一样指定了 MTKTextureLoaderOptions。我没有在文档中找到任何其他提及。
非常感谢任何帮助。
错误
/BuildRoot/Library/Caches/com.apple.xbs/Sources/MetalImage/MetalImage-121.0.2/MPSImage/Filters/MPSKeypoint.mm:166: failed assertion `Source 0x282ce8fc0 texture type (80) is unsupported
其中 0x282ce8fc0 是来自纹理加载器的 MTLTexture。
据我所知，没有 MTLTexture 类型 80，枚举范围最多为 8（不是十六进制）。
代码
CGFloat w = CGImageGetWidth(_image);
CGFloat h = CGImageGetHeight(_image);
id<MTLDevice> device = MTLCreateSystemDefaultDevice();
id<MTLCommandQueue> commandQueue = [device newCommandQueue];
NSDictionary* textureOptions = @{ MTKTextureLoaderOptionSRGB: [[NSNumber alloc] initWithBool:NO] };
id<MTLTexture> texture = [[[MTKTextureLoader alloc] initWithDevice:device] newTextureWithCGImage:_image
options:textureOptions
error:nil];
id<MTLBuffer> keypointDataBuffer;
id<MTLBuffer> keypointCountBuffer;
MTLRegion region = MTLRegionMake2D(0, 0, w, h);
id<MTLCommandBuffer> commandBuffer = [commandQueue commandBuffer];
MPSImageKeypointRangeInfo rangeInfo = {100,0.5};
MPSImageFindKeypoints* imageFindKeypoints = [[MPSImageFindKeypoints alloc] initWithDevice:device
info:&rangeInfo];
[imageFindKeypoints encodeToCommandBuffer:commandBuffer
sourceTexture:texture
regions:®ion
numberOfRegions:1
keypointCountBuffer:keypointCountBuffer
keypointCountBufferOffset:0
keypointDataBuffer:keypointDataBuffer
keypointDataBufferOffset:0];
[commandBuffer commit];
NSLog(keypointCountBuffer);
NSLog(keypointDataBuffer);
编辑
将我的图像转换为正确的像素格式后，我现在正在初始化缓冲区，如下所示：
id<MTLBuffer> keypointDataBuffer = [device newBufferWithLength:maxKeypoints*(sizeof(MPSImageKeypointData)) options:MTLResourceOptionCPUCacheModeDefault];
id<MTLBuffer> keypointCountBuffer = [device newBufferWithLength:sizeof(int) options:MTLResourceOptionCPUCacheModeDefault];
没有错误了。但是我现在怎么看内容呢？
((MPSImageKeypointData*)[keypointDataBuffer contents])[0].keypointCoordinate returns (0,0) 用于所有索引。我也不知道如何阅读 keypointsCountBuffer。转换为 int 值的缓冲区内容显示的值高于定义的 maxKeypoints。我看不到文档在哪里说明计数缓冲区的格式。
最后代码是 运行，为了完整起见，我认为我应该 post 整个代码作为答案
代码
id<MTLDevice> device = MTLCreateSystemDefaultDevice();
id<MTLCommandQueue> commandQueue = [device newCommandQueue];
// init textures
NSDictionary* textureOptions = @{ MTKTextureLoaderOptionSRGB: [[NSNumber alloc] initWithBool:NO] };
id<MTLTexture> texture = [[[MTKTextureLoader alloc] initWithDevice:device] newTextureWithCGImage:_lopoImage
options:textureOptions
error:nil];
MTLTextureDescriptor *descriptor = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:(MTLPixelFormatR8Unorm) width:w height:h mipmapped:NO];
descriptor.usage = (MTLTextureUsageShaderRead | MTLTextureUsageShaderWrite);
id<MTLTexture> unormTexture = [device newTextureWithDescriptor:descriptor];
// init arrays and buffers for keypoint finder
int maxKeypoints = w*h;
id<MTLBuffer> keypointDataBuffer = [device newBufferWithLength:sizeof(MPSImageKeypointData)*maxKeypoints options:MTLResourceOptionCPUCacheModeWriteCombined];
id<MTLBuffer> keypointCountBuffer = [device newBufferWithLength:sizeof(int) options:MTLResourceOptionCPUCacheModeWriteCombined];
MTLRegion region = MTLRegionMake2D(0, 0, w, h);
// init colorspace converter
CGColorSpaceRef srcColorSpace = CGColorSpaceCreateWithName(kCGColorSpaceSRGB);
CGColorSpaceRef dstColorSpace = CGColorSpaceCreateWithName(kCGColorSpaceLinearGray);
CGColorConversionInfoRef conversionInfo = CGColorConversionInfoCreate(srcColorSpace, dstColorSpace);
MPSImageConversion *conversion = [[MPSImageConversion alloc] initWithDevice:device
srcAlpha:(MPSAlphaTypeAlphaIsOne)
destAlpha:(MPSAlphaTypeNonPremultiplied)
backgroundColor:nil
conversionInfo:conversionInfo];
// init keypoint finder
MPSImageKeypointRangeInfo rangeInfo = {maxKeypoints,0.75};
MPSImageFindKeypoints* imageFindKeypoints = [[MPSImageFindKeypoints alloc] initWithDevice:device
info:&rangeInfo];
// encode command buffer
id<MTLCommandBuffer> commandBuffer = [commandQueue commandBuffer];
[conversion encodeToCommandBuffer:commandBuffer sourceTexture:texture destinationTexture:unormTexture];
[imageFindKeypoints encodeToCommandBuffer:commandBuffer
sourceTexture:unormTexture
regions:®ion
numberOfRegions:1
keypointCountBuffer:keypointCountBuffer
keypointCountBufferOffset:0
keypointDataBuffer:keypointDataBuffer
keypointDataBufferOffset:0];
// run command buffer
[commandBuffer commit];
[commandBuffer waitUntilCompleted];
// read keypoints
int count = ((int*)[keypointCountBuffer contents])[0];
MPSImageKeypointData* keypointDataArray = ((MPSImageKeypointData*)[keypointDataBuffer contents]);
for (int i = 0 ; i<count;i++) {
simd_ushort2 coordinate = keypointDataArray[i].keypointCoordinate;
NSLog(@"color:%f | at:(%u,%u)", keypointDataArray[i].keypointColorValue, coordinate[0], coordinate[1] );
}
我想应该有一种更聪明的方法来使用 [device newBufferWithBytesNoCopy] 分配关键点缓冲区，这样您就不需要将内容复制回分配的数组中。它只是没有弄清楚正确对齐缓冲区。
另外我要提一下，我想通常在任何类型的特征检测之后你都会有一个灰度纹理，这样就不需要图像转换部分了。