Posenet 的视频输入（而不是网络摄像头）P5.js

Question

我正在尝试创建一个网络程序，该程序使用 body 的特定部分使用 P5.js 和 ML5.js 中的 Posenet 进行绘制。我已经成功创建了网络模型，该模型在设置中使用 createCapture(VIDEO) 使用实时摄像头提要，如下所示

function setup() {
  socket = io();
  cnv = createCanvas(windowWidth,windowHeight);
  capture = createCapture(VIDEO);
  background(255);
  
  PoseZero.init();
  
  local = new Agent();

    socket.emit('game-start', local.data)
    socket.on('heartbeat', function(data){
        world = data;
    })
  
}

var localPreviousPose = {};
var remotePreviousPose = {};

function draw() {
  
  image(capture, 0, 0, 220, 140);

（完整代码见https://glitch.com/edit/#!/makeyourmark-final）

不过为了方便测试，我想在本地使用本地视频，我尝试以这种方式实现它：

function setup() {
  socket = io();

  cnv = createCanvas(windowWidth, windowHeight);
  capture = createVideo(
    [
      "https://cdn.glitch.com/e36b62ca-cf57-47f9-a16d-caa525a0d130%2FScreen%20Recording%202021-05-25%20at%2013.06.25.MOV?v=1621952189772"
    ],
    onVideoLoad
  );
  background(255);
  PoseZero.init();

  local = new Agent();

  socket.emit("game-start", local.data);
  socket.on("heartbeat", function(data) {
    world = data;
  });
}

var localPreviousPose = {};
var remotePreviousPose = {};

function onVideoLoad() {
  // The media will play as soon as it is loaded.
  capture.autoplay();
  capture.loop();
}

function draw() {
  image(capture, 0, 0, 220, 140);

（完整代码见https://glitch.com/edit/#!/makeyourmark-video）

错误似乎是无关的（例如“未捕获的类型错误：无法读取未定义的属性 'autoplay'”当它已经在自动播放时）

有什么办法可以解决这个问题吗？任何 insight/solution 表示赞赏。

谢谢！

Answer 1

您的代码存在一些问题，none 您在 post 中包含的内容很明显：

pose.js

.bind(this)

在 pose.js 中声明的 class 上的 init 函数中，您将 onVideoLoad 函数作为回调传递给 createVideo。但是 onVideoLoad 函数引用 this。任何时候你要从作为回调传递的函数中使用 this，你需要调用 .bind(this):

  this.init = function() {
    this.video = createVideo(
      ['https://cdn.glitch.com/e36b62ca-cf57-47f9-a16d-caa525a0d130%2FScreen%20Recording%202021-05-25%20at%2013.06.25.MOV?v=1621952189772'],
      // !!! Here, you need to use onVideoLoad.bind(this)
      onVideoLoad
    );

    // ...
  }
  
  function onVideoLoad() {
    // The media will play as soon as it is loaded.
    this.video.autoplay();
    this.video.loop();
  }

由于 i 未声明而损坏的代码

在 client.js 的 draw() 函数中，您对 world[i].data.pose 进行了多次引用，但尚未声明 [=17=].

调用 pop() 而不匹配调用 push()

在 pose.js 中，您必须使用以下代码：


    push();
    blendMode(MULTIPLY);
    // line(pose.leftWrist.x, pose.leftWrist.y, previousPose.data.leftWrist.x, previousPose.data.leftWrist.y);

    colorMode(HSB, 255);
    fill.apply(this, args.color);
    noStroke();
    beginShape();
    vertex(previousPose.data.leftWrist.x, previousPose.data.leftWrist.y);
    vertex(pose.leftWrist.x, pose.leftWrist.y);
    vertex(pose.leftElbow.x, pose.leftElbow.y);
    vertex(previousPose.data.leftElbow.x, previousPose.data.leftElbow.y);
    endShape(CLOSE);
    pop();
    
    // Presumably there should be another call to push() here
    colorMode(HSB, 255);
    fill.apply(this, args.color);
    noStroke();
    beginShape();
    vertex(previousPose.data.rightWrist.x, previousPose.data.rightWrist.y);
    vertex(pose.rightWrist.x, pose.rightWrist.y);
    vertex(pose.rightElbow.x, pose.rightElbow.y);
    vertex(previousPose.data.rightElbow.x, previousPose.data.rightElbow.y);
    endShape(CLOSE);
    pop();

需要在用户交互时启动视频播放

并非在所有情况下都可以在您的视频加载后立即调用 loop()。相反，你应该有一些提示让用户点击开始播放。

工作混音

尽管我对您的预期行为不是 100% 满意，但我认为这是“有效”的混音：https://glitch.com/edit/#!/Whosebug-67694586。我做了一些可能不是您想要的更改，例如让它创建一个视频元素并将其传递给 PoseZero.init（而不是 PoseZero 拥有自己的视频），并隐藏视频元素，因为您无论如何都要重新绘制到 canvas。

结论

我认为你应该注意以下建议：

向使用 p5.js 的 javascript 文件（下面的完整示例）添加 /* globals ... */ 注释，这样小故障就不会在您使用 p5.js 全局变量的地方显示错误没有宣布他们。这样你确实看到错误的地方它们将是有意义的。
单击常规上的格式化此文件按钮。这将使您的代码更易于阅读。
注意 javascript 控制台。当它开始发出警告时，不要忽略它，修复它们。

这是 p5.js 的示例全局评论（可能并不详尽）：

/* globals ADD, ALT, ARROW, AUDIO, AUTO, AXES, BACKSPACE, BASELINE, BEVEL, BEZIER, BLEND, BLUR, BOLD, BOLDITALIC, BOTTOM, BURN, CENTER, CHORD, CLAMP, CLOSE, CONTROL, CORNER, CORNERS, CROSS, CURVE, DARKEST, DEGREES, DEG_TO_RAD, DELETE, DIFFERENCE, DILATE, DODGE, DOWN_ARROW, ENTER, ERODE, ESCAPE, EXCLUSION, FALLBACK, FILL, GRAY, GRID, HALF_PI, HAND, HARD_LIGHT, HSB, HSL, IMAGE, IMMEDIATE, INVERT, ITALIC, LABEL, LANDSCAPE, LEFT, LEFT_ARROW, LIGHTEST, LINEAR, LINES, LINE_LOOP, LINE_STRIP, MIRROR, MITER, MOVE, MULTIPLY, NEAREST, NORMAL, OPAQUE, OPEN, OPTION, OVERLAY, P2D, PI, PIE, POINTS, PORTRAIT, POSTERIZE, PROJECT, QUADRATIC, QUADS, QUAD_STRIP, QUARTER_PI, RADIANS, RADIUS, RAD_TO_DEG, REMOVE, REPEAT, REPLACE, RETURN, RGB, RIGHT, RIGHT_ARROW, ROUND, SCREEN, SHIFT, SOFT_LIGHT, SQUARE, STROKE, SUBTRACT, TAB, TAU, TESS, TEXT, TEXTURE, THRESHOLD, TOP, TRIANGLES, TRIANGLE_FAN, TRIANGLE_STRIP, TWO_PI, UP_ARROW, VIDEO, WAIT, WEBGL, accelerationX, accelerationY, accelerationZ, deltaTime, deviceOrientation, displayHeight, displayWidth, focused, frameCount, height, isKeyPressed, key, keyCode, keyIsPressed, mouseButton, mouseIsPressed, mouseX, mouseY, movedX, movedY, pAccelerationX, pAccelerationY, pAccelerationZ, pRotateDirectionX, pRotateDirectionY, pRotateDirectionZ, pRotationX, pRotationY, pRotationZ, pixels, pmouseX, pmouseY, pwinMouseX, pwinMouseY, rotationX, rotationY, rotationZ, touches, turnAxis, width, winMouseX, winMouseY, windowHeight, windowWidth, abs, acos, alpha, ambientLight, ambientMaterial, angleMode, append, applyMatrix, arc, arrayCopy, asin, atan, atan2, background, beginContour, beginShape, bezier, bezierDetail, bezierPoint, bezierTangent, bezierVertex, blend, blendMode, blue, boolean, box, brightness, byte, camera, ceil, char, circle, clear, clearStorage, color, colorMode, concat, cone, constrain, copy, cos, createA, createAudio, createButton, createCamera, createCanvas, createCapture, createCheckbox, createColorPicker, createDiv, createElement, createFileInput, createGraphics, createImage, createImg, createInput, createNumberDict, createP, createRadio, createSelect, createShader, createSlider, createSpan, createStringDict, createVector, createVideo, createWriter, cursor, curve, curveDetail, curvePoint, curveTangent, curveTightness, curveVertex, cylinder, day, debugMode, degrees, describe, describeElement, directionalLight, displayDensity, dist, downloadFile, ellipse, ellipseMode, ellipsoid, emissiveMaterial, endContour, endShape, erase, exitPointerLock, exp, fill, filter, float, floor, fract, frameRate, frustum, fullscreen, get, getFrameRate, getItem, getURL, getURLParams, getURLPath, green, gridOutput, hex, hour, httpDo, httpGet, httpPost, hue, image, imageMode, int, isLooping, join, keyIsDown, lerp, lerpColor, lightFalloff, lightness, lights, line, loadBytes, loadFont, loadImage, loadJSON, loadModel, loadPixels, loadShader, loadStrings, loadTable, loadXML, log, loop, mag, map, match, matchAll, max, millis, min, minute, model, month, nf, nfc, nfp, nfs, noCanvas, noCursor, noDebugMode, noErase, noFill, noLights, noLoop, noSmooth, noStroke, noTint, noise, noiseDetail, noiseSeed, norm, normalMaterial, orbitControl, ortho, perspective, pixelDensity, plane, point, pointLight, pop, popMatrix, popStyle, pow, print, push, pushMatrix, pushStyle, quad, quadraticVertex, radians, random, randomGaussian, randomSeed, rect, rectMode, red, redraw, registerPromisePreload, removeElements, removeItem, requestPointerLock, resetMatrix, resetShader, resizeCanvas, reverse, rotate, rotateX, rotateY, rotateZ, round, saturation, save, saveCanvas, saveFrames, saveGif, saveJSON, saveJSONArray, saveJSONObject, saveStrings, saveTable, scale, second, select, selectAll, set, setAttributes, setCamera, setFrameRate, setMoveThreshold, setShakeThreshold, shader, shearX, shearY, shininess, shorten, shuffle, sin, smooth, sort, specularColor, specularMaterial, sphere, splice, split, splitTokens, spotLight, sq, sqrt, square, storeItem, str, stroke, strokeCap, strokeJoin, strokeWeight, subset, tan, text, textAlign, textAscent, textDescent, textFont, textLeading, textOutput, textSize, textStyle, textWidth, texture, textureMode, textureWrap, tint, torus, translate, triangle, trim, unchar, unhex, updatePixels, vertex, writeFile, year */

Posenet 的视频输入（而不是网络摄像头）P5.js

Video Input (Instead of Webcam) for Posenet P5.js

javascript

video

webcam

p5.js

pose

工作混音

结论