如何使用 AccessibilityService 在 Android 上执行拖动（基于 X、Y 鼠标坐标）？

Question

我想知道如何根据 X、Y 鼠标坐标对 android 执行拖动？考虑两个简单的例子，Team Viewer/QuickSupport 分别在远程 smartphone 和 Windows Paint 的 Pen 上绘制 "password pattern"。

我能做的就是（dispatchGesture() 和 AccessibilityNodeInfo.ACTION_CLICK）。

我找到了这些相关链接，但不知道它们是否有用：

下面是我的工作代码，用于将鼠标坐标（在 PictureBox 控件内部）发送到远程 phone 并模拟触摸。

Windows 申请表格：

private void pictureBox1_MouseDown(object sender, MouseEventArgs e)
{
    foreach (ListViewItem item in lvConnections.SelectedItems)
    {
        // Remote screen resolution
        string[] tokens = item.SubItems[5].Text.Split('x'); // Ex: 1080x1920

        int xClick = (e.X * int.Parse(tokens[0].ToString())) / (pictureBox1.Size.Width);
        int yClick = (e.Y * int.Parse(tokens[1].ToString())) / (pictureBox1.Size.Height);

        Client client = (Client)item.Tag;

        if (e.Button == MouseButtons.Left)
            client.sock.Send(Encoding.UTF8.GetBytes("TOUCH" + xClick + "<|>" + yClick + Environment.NewLine));
    }
}

编辑：

我最后一次尝试是 "swipe screen" 使用鼠标坐标（C# Windows Forms Application）和自定义 android 例程（参考上面链接的 "swipe screen" 代码), 分别为:

private Point mdownPoint = new Point();

private void pictureBox1_MouseDown(object sender, MouseEventArgs e)
{
    foreach (ListViewItem item in lvConnections.SelectedItems)
    {
        // Remote screen resolution
        string[] tokens = item.SubItems[5].Text.Split('x'); // Ex: 1080x1920

        Client client = (Client)item.Tag;

        if (e.Button == MouseButtons.Left)
        {
            xClick = (e.X * int.Parse(tokens[0].ToString())) / (pictureBox1.Size.Width); 
            yClick = (e.Y * int.Parse(tokens[1].ToString())) / (pictureBox1.Size.Height);

            // Saving start position:

            mdownPoint.X = xClick; 
            mdownPoint.Y = yClick; 

            client.sock.Send(Encoding.UTF8.GetBytes("TOUCH" + xClick + "<|>" + yClick + Environment.NewLine));
        }
    }
}

private void PictureBox1_MouseMove(object sender, MouseEventArgs e)
{
    foreach (ListViewItem item in lvConnections.SelectedItems)
    {
        // Remote screen resolution
        string[] tokens = item.SubItems[5].Text.Split('x'); // Ex: 1080x1920

        Client client = (Client)item.Tag;

        if (e.Button == MouseButtons.Left)
        {
            xClick = (e.X * int.Parse(tokens[0].ToString())) / (pictureBox1.Size.Width);
            yClick = (e.Y * int.Parse(tokens[1].ToString())) / (pictureBox1.Size.Height);

            client.sock.Send(Encoding.UTF8.GetBytes("MOUSESWIPESCREEN" + mdownPoint.X + "<|>" + mdownPoint.Y + "<|>" + xClick + "<|>" + yClick + Environment.NewLine));
        }
    }
}

android AccessibilityService:

public void Swipe(int x1, int y1, int x2, int y2, int time) {

if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.N) {
    System.out.println(" ======= Swipe =======");

    GestureDescription.Builder gestureBuilder = new GestureDescription.Builder();
    Path path = new Path();
    path.moveTo(x1, y1);
    path.lineTo(x2, y2);

    gestureBuilder.addStroke(new GestureDescription.StrokeDescription(path, 100, time));
    dispatchGesture(gestureBuilder.build(), new GestureResultCallback() {
        @Override
        public void onCompleted(GestureDescription gestureDescription) {
            System.out.println("SWIPE Gesture Completed :D");
            super.onCompleted(gestureDescription);
        }
    }, null);
}

}

产生以下结果（但仍然无法像 TeamViewer 那样绘制 "pattern password"）。但是就像下面的评论所说的那样，我认为使用类似的方法可以使用 Continued gestures 来实现。欢迎提出这方面的任何建议。

编辑 2：

当然，解决方案是 continued gestures 就像之前 Edit.

Simulating joystick movement using AccessibilityService

下面是我发现的假定固定代码 here =>

android 辅助功能服务：

// Simulates an L-shaped drag path: 200 pixels right, then 200 pixels down.
Path path = new Path();
path.moveTo(200,200);
path.lineTo(400,200);

final GestureDescription.StrokeDescription sd = new GestureDescription.StrokeDescription(path, 0, 500, true);

// The starting point of the second path must match
// the ending point of the first path.
Path path2 = new Path();
path2.moveTo(400,200);
path2.lineTo(400,400);

final GestureDescription.StrokeDescription sd2 = sd.continueStroke(path2, 0, 500, false); // 0.5 second

HongBaoService.mService.dispatchGesture(new GestureDescription.Builder().addStroke(sd).build(), new AccessibilityService.GestureResultCallback(){

@Override
public void onCompleted(GestureDescription gestureDescription){
super.onCompleted(gestureDescription);
HongBaoService.mService.dispatchGesture(new GestureDescription.Builder().addStroke(sd2).build(),null,null);
}

@Override
public void onCancelled(GestureDescription gestureDescription){
super.onCancelled(gestureDescription);
}
},null);

那么，我的疑惑是：如何为上面的代码正确发送鼠标坐标，可以执行向任意方向拖动的方式？ 一些想法？

编辑 3：

我找到了两个用于执行拖动的例程，但它们正在使用 UiAutomation + injectInputEvent(). AFAIK, injection of event works only in a system app like said and here 而我不需要它。

这是找到的例程：

然后为了实现我的目标，我认为第二个例程更适合使用（遵循逻辑，不包括事件注入）和 Edit 2 上显示的代码并发送所有点pictureBox1_MouseDown 和 pictureBox1_MouseMove（C# Windows Forms Application）分别动态填充 Point[] 并在 pictureBox1_MouseUp 上发送 cmd 以执行例程并使用此数组填充。如果您对第一个例程有想法，请告诉我:D.

如果在阅读此编辑后您有可能的解决方案，请在答案中告诉我，同时我会尝试测试这个想法。

Answer 1

您是否尝试过使用 AutoIt 脚本？

您可以在特定的范围内保存坐标 windows/screens。您可以在绘制图案时按住鼠标点击。

如果您需要，我还有一些示例代码/脚本供您使用！

编辑：

根据 this tutorial，您可以在 C# 上使用 Auto-IT。

按照以下步骤操作：

安装 Auto-IT
在参考管理器中添加 Auto-IT 作为参考 (AutoItX3.dll)
然后导入您添加的库：Using AutoItX3Lib;
创建名为 'auto' 的新 AutoItX3 对象：AutoItX3 auto = new AutoItX3();
您现在可以执行 Auto It 命令。

这是执行鼠标点击的完整示例：

Using AutoItX3Lib;
AutoItX3 auto = new AutoItX3();
auto.MouseClick("left", 78, 1133, 1, 35)

使用 AutoIt Window Info Tool 您可以检查您要使用的坐标。

请注意鼠标坐标模式之间存在差异：

例如：auto.AutoItSetOption("MouseCoordMode", 1) 将使用绝对屏幕坐标。请参阅来源 here。

按住鼠标点击，可以勾选MouseDown Function

Answer 2

这是一个基于编辑问题 3 的解决方案示例。

C# Windows 来自应用程序“formMain.cs”:

using System.Net.Sockets;

private List<Point> lstPoints;

private void pictureBox1_MouseDown(object sender, MouseEventArgs e) 
{
    if (e.Button == MouseButtons.Left)
    {
        lstPoints = new List<Point>();
        lstPoints.Add(new Point(e.X, e.Y));
    }
}

private void PictureBox1_MouseMove(object sender, MouseEventArgs e)
{
    if (e.Button == MouseButtons.Left)
    {
        lstPoints.Add(new Point(e.X, e.Y));
    }
}

private void PictureBox1_MouseUp(object sender, MouseEventArgs e)
{
    lstPoints.Add(new Point(e.X, e.Y));

    StringBuilder sb = new StringBuilder();

    foreach (Point obj in lstPoints)
    {
        sb.Append(Convert.ToString(obj) + ":");
    }

    serverSocket.Send("MDRAWEVENT" + sb.ToString() + Environment.NewLine);
}

android 服务“SocketBackground.java”:

import java.net.Socket;

String xline;

while (clientSocket.isConnected()) {

    BufferedReader xreader = new BufferedReader(new InputStreamReader(clientSocket.getInputStream(), StandardCharsets.UTF_8));

    if (xreader.ready()) {

        while ((xline = xreader.readLine()) != null) {
                xline = xline.trim();

            if (xline != null && !xline.trim().isEmpty()) {

                if (xline.contains("MDRAWEVENT")) {

                    String coordinates = xline.replace("MDRAWEVENT", "");
                    String[] tokens = coordinates.split(Pattern.quote(":"));
                    Point[] moviments = new Point[tokens.length];

                    for (int i = 0; i < tokens.length; i++) {
                       String[] coordinates = tokens[i].replace("{", "").replace("}", "").split(",");

                       int x = Integer.parseInt(coordinates[0].split("=")[1]);
                       int y = Integer.parseInt(coordinates[1].split("=")[1]);

                       moviments[i] = new Point(x, y);
                    }

                    MyAccessibilityService.instance.mouseDraw(moviments, 2000);
                }
            }
        }
    }
}

android AccessibilityService "MyAccessibilityService.java":

public void mouseDraw(Point[] segments, int time) {
    if (android.os.Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {

        Path path = new Path();
        path.moveTo(segments[0].x, segments[0].y);

        for (int i = 1; i < segments.length; i++) {

            path.lineTo(segments[i].x, segments[i].y);

            GestureDescription.StrokeDescription sd = new GestureDescription.StrokeDescription(path, 0, time);

            dispatchGesture(new GestureDescription.Builder().addStroke(sd).build(), new AccessibilityService.GestureResultCallback() {

                @Override
                public void onCompleted(GestureDescription gestureDescription) {
                    super.onCompleted(gestureDescription);
                }

                @Override
                public void onCancelled(GestureDescription gestureDescription) {
                    super.onCancelled(gestureDescription);
                }
            }, null);
        }
    }
}

Answer 3

对不起，兄弟，但下面的代码和我的英语一样糟糕：

public void mouseDraw(Point[] segments, int time) {
    if (android.os.Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {

        Path path = new Path();
        path.moveTo(segments[0].x, segments[0].y);

        for (int i = 1; i < segments.length; i++) {

            path.lineTo(segments[i].x, segments[i].y);

            GestureDescription.StrokeDescription sd = new GestureDescription.StrokeDescription(path, 0, time);

            dispatchGesture(new GestureDescription.Builder().addStroke(sd).build(), new AccessibilityService.GestureResultCallback() {

                @Override
                public void onCompleted(GestureDescription gestureDescription) {
                    super.onCompleted(gestureDescription);
                }

                @Override
                public void onCancelled(GestureDescription gestureDescription) {
                    super.onCancelled(gestureDescription);
                }
            }, null);
        }
    }
}

这里程序调用 dispatchGesture segments.length-1 次，但只有最后一个手势会完成，因为每个手势都会被下一个手势取消。来自官方documentation：当前正在进行的任何手势，无论是来自用户、此服务还是其他服务，都将被取消。

右：

public void mouseDraw(Point[] segments, int time) {
    if (android.os.Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {
        Path path = new Path();
        path.moveTo(segments[0].x, segments[0].y);
        for (int i = 1; i < segments.length; i++) {
            path.lineTo(segments[i].x, segments[i].y);
        }
        GestureDescription.StrokeDescription sd = new GestureDescription.StrokeDescription(path, 0, time);
        dispatchGesture(new GestureDescription.Builder().addStroke(sd).build(), new AccessibilityService.GestureResultCallback() {
            @Override
            public void onCompleted(GestureDescription gestureDescription) {
                super.onCompleted(gestureDescription);
            }
            @Override
            public void onCancelled(GestureDescription gestureDescription) {
                super.onCancelled(gestureDescription);
            }
        }, null);
    }
}

这里我们先构建Path，然后使用dispatchGesture一次。

如果你需要等待每个派遣的手势然后使用 Semaphore，例如，像这样 (Kotlin):

    val sem = Semaphore(0, true)
    for (i in 1 until segments.size) {
        path.lineTo(segments.get(i).x.toFloat(), segments.get(i).y.toFloat())
        val sd = GestureDescription.StrokeDescription(path, 0, 1500)
        dispatchGesture(
                GestureDescription.Builder().addStroke(sd).build(),
                object : GestureResultCallback() {
                    override fun onCancelled(gestureDescription: GestureDescription?) {
                        super.onCancelled(gestureDescription)
                        sem.release()
                    }
                    override fun onCompleted(gestureDescription: GestureDescription?) {
                        super.onCompleted(gestureDescription)
                        sem.release()
                    }
                },
                null
        )
        sem.acquire()
    }

如何使用 AccessibilityService 在 Android 上执行拖动（基于 X、Y 鼠标坐标）？

How perform a drag (based in X,Y mouse coordinates) on Android using AccessibilityService?

java

c#

android

accessibilityservice

mouse-coordinates