尝试使用 TFjs 和 React Native 进行实时对象检测，总是给出相同的预测并在打开相机时停止

Question

当相机打开时，空白相机会出现几秒钟，并且始终给出相同的以下输出并停止。

prediction: [{"className":"nematode, nematode worm, roundworm","probability":0.050750732421875},{"className":"matchstick","probability":0.043731689453125},{"className":"lighter, light, igniter, ignitor","probability":0.021453857421875}]

知道如何进行实时预测吗？没有像上面那样得到一次错误的预测

下面是相机屏幕代码，当用户扫描某个周围环境时，预测应该在实时相机馈送中发生

export function CameraScreen() {
  const [word, setWord] = useState("");
  const [predictionFound, setPredictionFound] = useState(false);
  const [hasPermission, setHasPermission] = useState();
  const [model, setModel] = useState();
  const TensorCamera = cameraWithTensors(Camera);
  let requestAnimationFrameId = 0;
  const textureDims =
    Platform.OS === "ios"
      ? { width: 1080, height: 1920 }
      : { width: 1600, height: 1200 };
  const tensorDims = { width: 152, height: 200 };

  async function loadModel() {
    try {
      const model = await mobilenet.load();
      setModel(model);
      console.log("set loaded Model");
    } catch (err) {
      console.log(err);
      console.log("failed load model");
    }
  }

  useEffect(() => {
    (async () => {
      const { status } = await Camera.requestPermissionsAsync();
      setHasPermission(status === "granted");

      await tf.ready();
      await loadModel();
      console.log("after model load");
    })();
  }, []);

  const getPrediction = async (tensor) => {
    if (!tensor) {
      return;
    }

    const prediction = await model.classify(tensor);
    console.log(`prediction: ${JSON.stringify(prediction)}`);

    if (!prediction || prediction.length === 0) {
      cancelAnimationFrame(requestAnimationFrameId);
      console.log("no predictions found");
      setPredictionFound(false);
      return;
    } else {
      setPredictionFound(true);
    }
  };

  const handleCameraStream = (imageAsTensors) => {
    const loop = async () => {
      const nextImageTensor = await imageAsTensors.next().value;
      await getPrediction(nextImageTensor);
      requestAnimationFrameId = requestAnimationFrame(loop);
    };
    if (!predictionFound) {
      loop();
    }
  };

  const renderCameraView = () => {
    return (
      <View style={styles.cameraView}>
        <TensorCamera
          style={styles.camera}
          type={Camera.Constants.Type.back}
          zoom={0}
          cameraTextureHeight={textureDims.height}
          cameraTextureWidth={textureDims.width}
          resizeHeight={tensorDims.height}
          resizeWidth={tensorDims.width}
          resizeDepth={3}
          onReady={handleCameraStream}
        />
      </View>
    );
  };

  useEffect(() => {
    return () => {
      cancelAnimationFrame(requestAnimationFrameId);
    };
  }, [requestAnimationFrameId]);

  return (
    <View style={styles.container}>
      <View style={styles.header}>
        <Text style={styles.title}>My Pictionary</Text>
      </View>
      {model ? (
        <View style={styles.body}>{renderCameraView()}</View>
      ) : (
        <Text>Still loading</Text>
      )}
    </View>
  );
}

Answer 1

在函数 handleCameraStream 中，一旦找到预测就停止循环函数。在你的情况下，你会想要不断运行循环，因为你想对所有帧而不是单个帧进行预测。

const handleCameraStream = (imageAsTensors) => {
    const loop = async () => {
      const nextImageTensor = await imageAsTensors.next().value;
      await getPrediction(nextImageTensor);
      requestAnimationFrameId = requestAnimationFrame(loop);
    };
    loop();
  };

尝试使用 TFjs 和 React Native 进行实时对象检测，总是给出相同的预测并在打开相机时停止

Trying to make a real time object detection with TFjs and React Native, always gives the same prediction and stops when camera is opened

machine-learning

object-detection

react-native

tensorflow

tensorflow.js