[Appium] How to find element only by image?

[Abush]

Hi,

I am using appium latest (1.11.1)
Working on Win 10
IDE tool: Intellij
Writing in JAVA
java client - 7.0.0
Maven project

I am trying to find element by image only by using: driver.findElementByImage("path to png image)
Can someone simplified the how to make it work ?
The app is built with opengl, so no element exsist on the screen at all,
This is Android app, any help will be appriciate, tutorials are very general in this subject,

Thanks

[Abush]

Anybody?
Nobosy knows any tool that works with appium to catch element by recognize image?

[Aleksei]

Write what you have tried and problem met. Hardly we can help without knowing your problem

[Abush]

I tried the appium find by image but The appium never recognize the opencv in the npm, I have tried every method to Install opencv, it installed auccess but appium doctor keep. Saying that I need to install opencv.
I have tried sikuli but it always fails.
So I wanted to know if there is working method to fond element by image?

What I need:
Sample code that use exist screenshot to find it on the screen(as element).

Thanks

[Aleksei]

1. there is guide https://appium.readthedocs.io/en/latest/en/writing-running-appium/image-comparison/#image-comparison-features
2. step ONE is have OpenCV 3+ native libraries. You need install it on your MAC/PC.

it is not quite obvious. but it is. follow guide from official site: e.g. with Linux: https://docs.opencv.org/master/d7/d9f/tutorial_linux_install.html

[Abush]

Hi Aleksei,

I have already installed the opencv through npm,
When I run any driver.findByImage(“path.png”)
I get this error from appium server:

mime must be a string
[W3C] Encountered internal error running command: Error: opencv4nodejs module is required to use OpenCV features. Please install it first (npm i -g opencv4nodejs) and restart Appium. Read https://github.com/justadudewhohacks/opencv4nodejs#how-to-install for more details on this topic.
[W3C] at initOpenCV (C:\node_modules\appium\node_modules\appium-support\lib\image-util.js:100:11)
[W3C] at Object.getImageOccurrence$ (C:\node_modules\appium\node_modules\appium-support\lib\image-util.js:420:3)
[W3C] at tryCatch (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:67:40)
[W3C] at GeneratorFunctionPrototype.invoke [as _invoke] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:315:22)
[W3C] at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:100:21)
[W3C] at invoke (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:136:37)
[W3C] at enqueueResult (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:185:17)
[W3C] at new Promise ()
[W3C] at new F (C:\node_modules\appium\node_modules\core-js\library\modules$.export.js:30:36)
[W3C] at AsyncIterator.enqueue (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:184:12)
[W3C] at AsyncIterator.prototype.(anonymous function) [as next] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:100:21)
[W3C] at Object.runtime.async (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:209:12)
[W3C] at Object.getImageOccurrence (C:\node_modules\appium\node_modules\appium-support\build\lib\image-util.js:632:30)
[W3C] at AndroidDriver.callee$0$0$ (C:\node_modules\appium\node_modules\appium-base-driver\lib\basedriver\commands\images.js:47:32)
[W3C] at tryCatch (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:67:40)
[W3C] at GeneratorFunctionPrototype.invoke [as _invoke] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:315:22)
[W3C] at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:100:21)
[W3C] at invoke (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:136:37)
[W3C] at enqueueResult (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:185:17)
[W3C] at new Promise ()
[W3C] at new F (C:\node_modules\appium\node_modules\core-js\library\modules$.export.js:30:36)
[W3C] at AsyncIterator.enqueue (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:184:12)
[W3C] at AsyncIterator.prototype.(anonymous function) [as next] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:100:21)
[W3C] at Object.runtime.async (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:209:12)
[W3C] at AndroidDriver.callee$0$0 [as compareImages] (C:\node_modules\appium\node_modules\appium-base-driver\build\lib\basedriver\commands\images.js:55:30)
[W3C] at AndroidDriver.condition$ (C:\node_modules\appium\node_modules\appium-base-driver\lib\basedriver\commands\find.js:111:26)
[W3C] at tryCatch (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:67:40)
[W3C] at GeneratorFunctionPrototype.invoke [as _invoke] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:315:22)
[W3C] at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:100:21)
[W3C] at GeneratorFunctionPrototype.invoke (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:136:37)

I am mantion that I already use install opencv in various ways:
npm install --save opencv4nodejs
npm i -g opencv4nodejs

And also tried mannually, the appium doctor keep saying:
WARN AppiumDoctor opencv4nodejs cannot be found

But I got indication that the opencv is installed success when install from npm,
maybe Windows 10 path is wrong ? I add mannually all the pathes that I could,
I don’t know what else to do in order that the npm/Windows recognize the opencv and activate him.

Any help will be appriciate!

[Aleksei]

@Abush once again. With npm you installing libraries which used to translate your java commands into commands that uses openCV. To install openCV follow guide. In general ot has 2 steps: download sources and compile then on your mac/pc.

PS use link in error to install openCV

[Abush]

@Aleksei
I did it, I follow the guide…
I don’t understand why it fails as I attached the info,
I did exactly as the guide but it still fails,
Can you direct me more spesific?

[Aleksei]

So you did “cmake” command and compiled openCV. Correct me if i am wrong.

[Abush]

Cmake is installed and windows path works with cmake,
How to compile? Did I miss something?

[Aleksei]

i do not use image any more. last time i was installing it - i used guide from https://docs.opencv.org/master/d7/d9f/tutorial_linux_install.html with macOS machine

try also: https://github.com/appium/appium/issues/11865

[Abush]

So if not image then what are you using?

[Aleksei]

just do not need search by image any more. another project

[Abush]

Any other suggestions?
How can I found images?

[Aleksei]

i suggest fix your case. other case harder.

1. check you installed appium also with “npm install -g appium”
2. check appium version up to date
3. i checked “npm i -g opencv4nodejs” and looks like it DOES ALL needed. it compiles also openCV libraries.

[Abush]

I did all of this… Exactly as you write…

[Aleksei]

do we have exact appium command that failing?

[Abush]

Every usage of opencv fails

[mykola-mokhnach]

Sometimes providing a correct value to NODE_ROOT environment variable solves module resolution issues.

[Abush]

How can I do it?
Appium is installed through npm global and works great, I use the appium server from java code, but every time I tried to use opencv it fails as I mention above.