如何修改（添加过滤器）WebRTC 发送给其他人的相机流 peers/server

Question

范围

我正在使用 RTCCameraPreviewView 显示本地相机流

    let videoSource = self.pcFactory.avFoundationVideoSource(with: nil)
    let videoTrack = self.pcFactory.videoTrack(with: sVideoSource, trackId: "video0")

    //setting the capture session to my RTCCameraPreviewView:
    (self.previewView as! RTCCameraPreviewView).captureSession = (videoTrack.source as! RTCAVFoundationVideoSource).captureSession

    stream = self.pcFactory.mediaStream(withStreamId: "unique_label")
    audioTrack = self.pcFactory.audioTrack(withTrackId: "audio0")
    stream.addAudioTrack(audioTrack)

    var device: AVCaptureDevice?
    for captureDevice in AVCaptureDevice.devices(withMediaType: AVMediaTypeVideo) {
        if (captureDevice as AnyObject).position == AVCaptureDevicePosition.front {
            device = captureDevice as? AVCaptureDevice
            break
        }
    }
    if device != nil && videoTrack != nil {
        stream.addVideoTrack(videoTrack)
    }

    configuration = RTCConfiguration()

    configuration.iceServers = iceServers
    peerConnection = self.pcFactory.peerConnection(with: configuration, constraints: RTCMediaConstraints(mandatoryConstraints: nil, optionalConstraints: ["DtlsSrtpKeyAgreement": "true"]), delegate: self)
    peerConnection.add(stream)

一切正常，正如预期的那样。

问题

现在我想从相机中获取帧并对它们进行预处理以添加一些过滤器（棕褐色，b/w 等），然后将帧中继到 WebRTC。在浏览了 webrtc 文档之后，我仍然无法找到从哪里开始和做什么。

我们将不胜感激任何形式的提醒！

Answer 1

我找到了出路。所以基本上您需要构建自己的 WebRTC pod，然后您可以添加一个挂钩以在 videoOutput 对象上使用自定义 AVCaptureVideoDataOutputSampleBufferDelegate。然后处理sampleBuffer，修改buffer再传给webrtc。

实施

打开文件webrtc/sdk/objc/Frameworks/Classes/RTCAVFoundationVideoCapturerInternal.mm

并在线：

[videoDataOutput setSampleBufferDelegate:self queue:self.frameQueue];

使用自定义委托代替 self。

在那位代表中

  class YourDelegate : AVCaptureVideoDataOutputSampleBufferDelegate {
 func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!) {
        let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)

        //modify the pixelBuffer
        //get the modifiedSampleBuffer from modified pixelBuffer
        DispatchQueue.main.async {
            //show the modified buffer to the user
        }

        //To pass the modified buffer to webrtc (warning: [this is objc code]):
        //(_capturer object is found in RTCAVFoundationVideoCapturerInternal.mm)
        _capturer->CaptureSampleBuffer(modifiedSampleBuffer, _rotation);

    }
}

如何修改（添加过滤器）WebRTC 发送给其他人的相机流 peers/server

How to modify (add filters to) the camera stream that WebRTC is sending to other peers/server

iphone

ios

webrtc

apprtcdemo

swift