使用 MTKView 显示 JPEG 图片
Display JPEG image using MTKView
有没有办法通过 MTKView 和 MTLBuffer（在 iPhone 6+ 内）显示 JPEG 图像。我已经通过以下方式尝试过（这只是测试）：
- (id<MTLBuffer>)testBuffer
{
if (!_testBuffer)
{
//
NSString *path = [[NSBundle mainBundle] pathForResource:@"testImage" ofType:@"jpg"];
NSData *imageData = [NSData dataWithContentsOfFile:path];
_testBuffer = [self.device newBufferWithBytes:imageData.bytes length:imageData.length options:MTLResourceCPUCacheModeWriteCombined];
}
return _testBuffer;
}
- (void)drawInMTKView:(MTKView *)view
{
MTLRenderPassDescriptor* renderPassDescriptor = view.currentRenderPassDescriptor;
id <MTLCommandBuffer> commandBuffer = [_commandQueue commandBuffer];
id <MTLRenderCommandEncoder> renderEncoder = [commandBuffer renderCommandEncoderWithDescriptor:renderPassDescriptor];
[renderEncoder drawPrimitives:MTLPrimitiveTypePoint indirectBuffer:self.testBuffer indirectBufferOffset:0];
[renderEncoder endEncoding];
[commandBuffer presentDrawable:view.currentDrawable];
[commandBuffer commit];
}
我在此方法调用中遇到错误：
[renderEncoder drawPrimitives:MTLPrimitiveTypePoint indirectBuffer:self.testBuffer indirectBufferOffset:0];
/BuildRoot/Library/Caches/com.apple.xbs/Sources/Metal/Metal-54.31/ToolsLayers/Debug/MTLDebugRenderCommandEncoder.mm:2123:
failed assertion `-[MTLDebugRenderCommandEncoder
drawPrimitives:indirectBuffer:indirectBufferOffset:] is only supported
on MTLFeatureSet_iOS_GPUFamily3_v1 and later'
这是因为 API 只有 iPhone 6s(+) 支持。
而且我认为我做的事情完全错误。也许我需要换个方向思考。有人可以帮助我或指出正确的方向吗？
谢谢
没有简单的方法可以使用MTLBuffer 通过MTKView 呈现图像数据。在这种情况下，您必须为 MTKView 在 drawble 层中用于显示的正确输出纹理准备正确的像素表示。最简单的方法是使用 MTKTextureLoader，或者从 UIImage/CGImage 实例加载数据到 MTLTexture 对象。
我做了一个类似的实验，使用 MTL 层和 MTLTexture（而不是 MTLBuffer）在 iOS 设备上呈现图像，我已经决定了显示图像的最佳方式（如果这个只需要：到显示图像）它使用直通内核函数而不渲染。
在 Swift2 中，如下所示：
...
let textureLoader = MTKTextureLoader(device: self.device!)
if let image = UIImage(named: file){
imageTexture = try! textureLoader.newTextureWithCGImage(image.CGImage!, options: nil)
threadGroups = MTLSizeMake(
(imageTexture.width+threadGroupCount.width)/threadGroupCount.width,
(imageTexture.height+threadGroupCount.height)/threadGroupCount.height, 1)
}
}
...
let function:MTLFunction! = library.newFunctionWithName("kernel_passthrough")
...
pipeline = try! self.device.newComputePipelineStateWithFunction(function)
...
let commandBuffer = commandQueue.commandBuffer()
let encoder = commandBuffer.computeCommandEncoder()
encoder.setComputePipelineState(pipeline)
encoder.setTexture(actualImageTexture, atIndex: 0)
encoder.setTexture(metalView.currentDrawable!.texture, atIndex: 1)
encoder.setBuffer(self.saturationUniform, offset: 0, atIndex: 0)
encoder.dispatchThreadgroups(threadGroups!, threadsPerThreadgroup: threadGroupCount)
encoder.endEncoding()
commandBuffer.presentDrawable(metalView.currentDrawable!)
commandBuffer.commit()
...
在金属着色文件中：
kernel void kernel_passthrough(texture2d<float, access::read> inTexture [[texture(0)]],
texture2d<float, access::write> outTexture [[texture(1)]],
uint2 gid [[thread_position_in_grid]])
{
float4 inColor = inTexture.read(gid);
//
// flip texture vertically if it needs to display with right orientation
//
outTexture.write(inColor, gid);
}
完整示例来源：ImageMetalling-00
最近比较忙忘记写解决方案了。 @Denn 的回答看起来是正确的，所以我会接受它。
我的解决方案如下（在 objective-c 中）。我简化了苹果示例（以及它们的着色器文件）并制作了 test project。
我创建了名为 MetalView 的 MTKView 的子class。它有“configure”方法，为绘图准备 MetalView：设置 MTKView 的属性并创建 pipelineState 对象和顶点缓冲区。
- (void)configure
{
self.device = MTLCreateSystemDefaultDevice();
self.colorPixelFormat = MTLPixelFormatBGRA8Unorm;
self.framebufferOnly = YES;
self.sampleCount = 1;
// create command queue for usage during drawing
self.commandQueue = [self.device newCommandQueue];
// load shaders functions from texturedQuad.metal file. These functions needed for configuration of MTLRenderPipelineDescriptor
id <MTLLibrary> shaderLibrary = [self.device newDefaultLibrary];
id <MTLFunction> fragmentProgram = [shaderLibrary newFunctionWithName:@"texturedQuadFragment"];
id <MTLFunction> vertexProgram = [shaderLibrary newFunctionWithName:@"texturedQuadVertex"];
// create a pipeline state
MTLRenderPipelineDescriptor *pQuadPipelineStateDescriptor = [MTLRenderPipelineDescriptor new];
pQuadPipelineStateDescriptor.colorAttachments[0].pixelFormat = MTLPixelFormatBGRA8Unorm;
pQuadPipelineStateDescriptor.sampleCount = self.sampleCount;
pQuadPipelineStateDescriptor.vertexFunction = vertexProgram;
pQuadPipelineStateDescriptor.fragmentFunction = fragmentProgram;
NSError *pipErr = nil;
self.pipelineState = [self.device newRenderPipelineStateWithDescriptor:pQuadPipelineStateDescriptor
error:&pipErr];
if (pipErr) NSLog(@"newRenderPipelineStateWithDescriptor err: %@", pipErr);
// buffers for MTLRenderCommandEncoder in drawRect: method
self.vertexBuffer = [self.device newBufferWithBytes:kQuadVertices
length:kSzQuadVertices
options:MTLResourceOptionCPUCacheModeDefault];
self.texCoordBuffer = [self.device newBufferWithBytes:kQuadTexCoords
length:kSzQuadTexCoords
options:MTLResourceOptionCPUCacheModeDefault];
self.paused = YES;
self.enableSetNeedsDisplay = NO;
}
drawRect: 方法中的 renderEncoder 将使用这些东西进行实际绘制：
- (void)drawRect:(CGRect)rect
{
id <MTLCommandBuffer> commandBuffer = [self.commandQueue commandBuffer];
MTLRenderPassDescriptor *renderPassDescriptor = self.currentRenderPassDescriptor;
id <MTLRenderCommandEncoder> renderEncoder = [commandBuffer renderCommandEncoderWithDescriptor:renderPassDescriptor];
// Encode into a renderer
[renderEncoder setRenderPipelineState:self.pipelineState];
[renderEncoder setVertexBuffer:self.vertexBuffer
offset:0
atIndex:0];
[renderEncoder setVertexBuffer:self.texCoordBuffer
offset:0
atIndex:1];
[renderEncoder setFragmentTexture:self.texture
atIndex:0];
// tell the render context we want to draw our primitives. We will draw triangles that's
// why we need kQuadVertices and kQuadTexCoords (arrays of points)
[renderEncoder drawPrimitives:MTLPrimitiveTypeTriangle
vertexStart:0
vertexCount:6
instanceCount:1];
[renderEncoder endEncoding];
[commandBuffer presentDrawable:self.currentDrawable];
[commandBuffer commit];
}
着色器文件代码(texturedQuad.metal)：
struct VertexInOut
{
float4 m_Position [[position]];
float2 m_TexCoord [[user(texturecoord)]];
};
vertex VertexInOut texturedQuadVertex(constant float4 *pPosition [[ buffer(0) ]],
constant packed_float2 *pTexCoords [[ buffer(1) ]],
constant float4x4 *pMVP [[ buffer(2) ]],
uint vid [[ vertex_id ]])
{
VertexInOut outVertices;
outVertices.m_Position = pPosition[vid];
outVertices.m_TexCoord = pTexCoords[vid];
return outVertices;
}
fragment half4 texturedQuadFragment(VertexInOut inFrag [[ stage_in ]],
texture2d<half> tex2D [[ texture(0) ]])
{
constexpr sampler quad_sampler;
half4 color = tex2D.sample(quad_sampler, inFrag.m_TexCoord);
return color;
}
MetalView class 在 ViewController 的故事板中用作视图出口的 class。 ViewController 的 viewDidLoad 方法检索指定图像的对象并将该对象设置为 MetalView。
- (void)viewDidLoad
{
[super viewDidLoad];
self.metalView = (MetalView *)self.view;
NSString *path = [[NSBundle mainBundle] pathForResource:@"testImage"
ofType:@"jpg"];
MTKTextureLoader *loader = [[MTKTextureLoader alloc] initWithDevice:self.metalView.device];
NSError *err = nil;
self.metalView.texture = [loader newTextureWithContentsOfURL:[NSURL fileURLWithPath:path] options:nil error:&err];
if (err) NSLog(@"newTextureWithContentsOfURL err: %@", [err localizedDescription]);
}
最后，我们需要为 MetalView 调用 draw 方法
- (void)viewDidLayoutSubviews
{
[self.metalView draw];
}
示例代码是 here.
有没有办法通过 MTKView 和 MTLBuffer（在 iPhone 6+ 内）显示 JPEG 图像。我已经通过以下方式尝试过（这只是测试）：
- (id<MTLBuffer>)testBuffer
{
if (!_testBuffer)
{
//
NSString *path = [[NSBundle mainBundle] pathForResource:@"testImage" ofType:@"jpg"];
NSData *imageData = [NSData dataWithContentsOfFile:path];
_testBuffer = [self.device newBufferWithBytes:imageData.bytes length:imageData.length options:MTLResourceCPUCacheModeWriteCombined];
}
return _testBuffer;
}
- (void)drawInMTKView:(MTKView *)view
{
MTLRenderPassDescriptor* renderPassDescriptor = view.currentRenderPassDescriptor;
id <MTLCommandBuffer> commandBuffer = [_commandQueue commandBuffer];
id <MTLRenderCommandEncoder> renderEncoder = [commandBuffer renderCommandEncoderWithDescriptor:renderPassDescriptor];
[renderEncoder drawPrimitives:MTLPrimitiveTypePoint indirectBuffer:self.testBuffer indirectBufferOffset:0];
[renderEncoder endEncoding];
[commandBuffer presentDrawable:view.currentDrawable];
[commandBuffer commit];
}
我在此方法调用中遇到错误：
[renderEncoder drawPrimitives:MTLPrimitiveTypePoint indirectBuffer:self.testBuffer indirectBufferOffset:0];
/BuildRoot/Library/Caches/com.apple.xbs/Sources/Metal/Metal-54.31/ToolsLayers/Debug/MTLDebugRenderCommandEncoder.mm:2123: failed assertion `-[MTLDebugRenderCommandEncoder drawPrimitives:indirectBuffer:indirectBufferOffset:] is only supported on MTLFeatureSet_iOS_GPUFamily3_v1 and later'
这是因为 API 只有 iPhone 6s(+) 支持。
而且我认为我做的事情完全错误。也许我需要换个方向思考。有人可以帮助我或指出正确的方向吗？ 谢谢
没有简单的方法可以使用MTLBuffer 通过MTKView 呈现图像数据。在这种情况下，您必须为 MTKView 在 drawble 层中用于显示的正确输出纹理准备正确的像素表示。最简单的方法是使用 MTKTextureLoader，或者从 UIImage/CGImage 实例加载数据到 MTLTexture 对象。
我做了一个类似的实验，使用 MTL 层和 MTLTexture（而不是 MTLBuffer）在 iOS 设备上呈现图像，我已经决定了显示图像的最佳方式（如果这个只需要：到显示图像）它使用直通内核函数而不渲染。
在 Swift2 中，如下所示：
...
let textureLoader = MTKTextureLoader(device: self.device!)
if let image = UIImage(named: file){
imageTexture = try! textureLoader.newTextureWithCGImage(image.CGImage!, options: nil)
threadGroups = MTLSizeMake(
(imageTexture.width+threadGroupCount.width)/threadGroupCount.width,
(imageTexture.height+threadGroupCount.height)/threadGroupCount.height, 1)
}
}
...
let function:MTLFunction! = library.newFunctionWithName("kernel_passthrough")
...
pipeline = try! self.device.newComputePipelineStateWithFunction(function)
...
let commandBuffer = commandQueue.commandBuffer()
let encoder = commandBuffer.computeCommandEncoder()
encoder.setComputePipelineState(pipeline)
encoder.setTexture(actualImageTexture, atIndex: 0)
encoder.setTexture(metalView.currentDrawable!.texture, atIndex: 1)
encoder.setBuffer(self.saturationUniform, offset: 0, atIndex: 0)
encoder.dispatchThreadgroups(threadGroups!, threadsPerThreadgroup: threadGroupCount)
encoder.endEncoding()
commandBuffer.presentDrawable(metalView.currentDrawable!)
commandBuffer.commit()
...
在金属着色文件中：
kernel void kernel_passthrough(texture2d<float, access::read> inTexture [[texture(0)]],
texture2d<float, access::write> outTexture [[texture(1)]],
uint2 gid [[thread_position_in_grid]])
{
float4 inColor = inTexture.read(gid);
//
// flip texture vertically if it needs to display with right orientation
//
outTexture.write(inColor, gid);
}
完整示例来源：ImageMetalling-00
最近比较忙忘记写解决方案了。 @Denn 的回答看起来是正确的，所以我会接受它。 我的解决方案如下（在 objective-c 中）。我简化了苹果示例（以及它们的着色器文件）并制作了 test project。 我创建了名为 MetalView 的 MTKView 的子class。它有“configure”方法，为绘图准备 MetalView：设置 MTKView 的属性并创建 pipelineState 对象和顶点缓冲区。
- (void)configure
{
self.device = MTLCreateSystemDefaultDevice();
self.colorPixelFormat = MTLPixelFormatBGRA8Unorm;
self.framebufferOnly = YES;
self.sampleCount = 1;
// create command queue for usage during drawing
self.commandQueue = [self.device newCommandQueue];
// load shaders functions from texturedQuad.metal file. These functions needed for configuration of MTLRenderPipelineDescriptor
id <MTLLibrary> shaderLibrary = [self.device newDefaultLibrary];
id <MTLFunction> fragmentProgram = [shaderLibrary newFunctionWithName:@"texturedQuadFragment"];
id <MTLFunction> vertexProgram = [shaderLibrary newFunctionWithName:@"texturedQuadVertex"];
// create a pipeline state
MTLRenderPipelineDescriptor *pQuadPipelineStateDescriptor = [MTLRenderPipelineDescriptor new];
pQuadPipelineStateDescriptor.colorAttachments[0].pixelFormat = MTLPixelFormatBGRA8Unorm;
pQuadPipelineStateDescriptor.sampleCount = self.sampleCount;
pQuadPipelineStateDescriptor.vertexFunction = vertexProgram;
pQuadPipelineStateDescriptor.fragmentFunction = fragmentProgram;
NSError *pipErr = nil;
self.pipelineState = [self.device newRenderPipelineStateWithDescriptor:pQuadPipelineStateDescriptor
error:&pipErr];
if (pipErr) NSLog(@"newRenderPipelineStateWithDescriptor err: %@", pipErr);
// buffers for MTLRenderCommandEncoder in drawRect: method
self.vertexBuffer = [self.device newBufferWithBytes:kQuadVertices
length:kSzQuadVertices
options:MTLResourceOptionCPUCacheModeDefault];
self.texCoordBuffer = [self.device newBufferWithBytes:kQuadTexCoords
length:kSzQuadTexCoords
options:MTLResourceOptionCPUCacheModeDefault];
self.paused = YES;
self.enableSetNeedsDisplay = NO;
}
drawRect: 方法中的 renderEncoder 将使用这些东西进行实际绘制：
- (void)drawRect:(CGRect)rect
{
id <MTLCommandBuffer> commandBuffer = [self.commandQueue commandBuffer];
MTLRenderPassDescriptor *renderPassDescriptor = self.currentRenderPassDescriptor;
id <MTLRenderCommandEncoder> renderEncoder = [commandBuffer renderCommandEncoderWithDescriptor:renderPassDescriptor];
// Encode into a renderer
[renderEncoder setRenderPipelineState:self.pipelineState];
[renderEncoder setVertexBuffer:self.vertexBuffer
offset:0
atIndex:0];
[renderEncoder setVertexBuffer:self.texCoordBuffer
offset:0
atIndex:1];
[renderEncoder setFragmentTexture:self.texture
atIndex:0];
// tell the render context we want to draw our primitives. We will draw triangles that's
// why we need kQuadVertices and kQuadTexCoords (arrays of points)
[renderEncoder drawPrimitives:MTLPrimitiveTypeTriangle
vertexStart:0
vertexCount:6
instanceCount:1];
[renderEncoder endEncoding];
[commandBuffer presentDrawable:self.currentDrawable];
[commandBuffer commit];
}
着色器文件代码(texturedQuad.metal)：
struct VertexInOut
{
float4 m_Position [[position]];
float2 m_TexCoord [[user(texturecoord)]];
};
vertex VertexInOut texturedQuadVertex(constant float4 *pPosition [[ buffer(0) ]],
constant packed_float2 *pTexCoords [[ buffer(1) ]],
constant float4x4 *pMVP [[ buffer(2) ]],
uint vid [[ vertex_id ]])
{
VertexInOut outVertices;
outVertices.m_Position = pPosition[vid];
outVertices.m_TexCoord = pTexCoords[vid];
return outVertices;
}
fragment half4 texturedQuadFragment(VertexInOut inFrag [[ stage_in ]],
texture2d<half> tex2D [[ texture(0) ]])
{
constexpr sampler quad_sampler;
half4 color = tex2D.sample(quad_sampler, inFrag.m_TexCoord);
return color;
}
MetalView class 在 ViewController 的故事板中用作视图出口的 class。 ViewController 的 viewDidLoad 方法检索指定图像的对象并将该对象设置为 MetalView。
- (void)viewDidLoad
{
[super viewDidLoad];
self.metalView = (MetalView *)self.view;
NSString *path = [[NSBundle mainBundle] pathForResource:@"testImage"
ofType:@"jpg"];
MTKTextureLoader *loader = [[MTKTextureLoader alloc] initWithDevice:self.metalView.device];
NSError *err = nil;
self.metalView.texture = [loader newTextureWithContentsOfURL:[NSURL fileURLWithPath:path] options:nil error:&err];
if (err) NSLog(@"newTextureWithContentsOfURL err: %@", [err localizedDescription]);
}
最后，我们需要为 MetalView 调用 draw 方法
- (void)viewDidLayoutSubviews
{
[self.metalView draw];
}
示例代码是 here.