改造思寒的AppCrawler，使其支持Appium最新版本

思绪

最近完成了自定义Appium的需求，让Appium内置了自动识别权限框并点击的能力，参考我的知乎专栏：自定义Appium之路

但遇到另外一个问题，就是testerhome思寒开发的AppiumCrawler并不支持Appium最新版，也就是当前的1.12版本，只支持到1.8版本，让人很是捉急。

本来是想基于1.8重新自定义一个appium，但是发现这个appium实在太老了，下载下来编译都有各种问题，况且后续还要自定义appium-android-driver，appium-uiautomator2-driver和appium-uiaumator2-server，工作量至少得3天，太费时间。

索性，我来替思寒把AppCrawler来升级一下，让它支持最新appium。

刚开始觉得挺难的，毕竟我对scala只略知一二，编译打包方面还要学，但事后发现，这个工程做的确实不错，升级改造过程比预计的要简单很多，这里要给思寒大佬一个赞👍！

改造

先看看出了什么问题？

我开启最新的appium：

appium

执行appcrawler测试：

java -jar appcrawler-2.1.3.jar -a ApiDemos-debug.apk

执行过程中在appium和appcrawler两端都报错：

[HTTP] <-- GET /wd/hub/session/efdf97d8-cf46-4ffb-b2d4-7d8feb931cee/window/rect 200 7 ms - 50
[HTTP] 
[HTTP] --> POST /wd/hub/session/efdf97d8-cf46-4ffb-b2d4-7d8feb931cee/execute/sync
[HTTP] {"script":"var source = document.documentElement.outerHTML; \nif (!source) { source = new XMLSerializer().serializeToString(document); }\nreturn source;","args":[]}
[debug] [W3C (efdf97d8)] Calling AppiumDriver.execute() with args: ["var source = document.documentElement.outerHTML; \nif (!source) { source = new XMLSerializer().serializeToString(document); }\nreturn source;",[],"efdf97d8-cf46-4ffb-b2d4-7d8feb931cee"]
[debug] [W3C (efdf97d8)] Encountered internal error running command: NotImplementedError: Method is not implemented
[debug] [W3C (efdf97d8)]     at AndroidDriver.extensions.execute (/usr/local/lib/node_modules/appium/node_modules/appium-android-driver/lib/commands/execute.js:12:9)
[debug] [W3C (efdf97d8)]     at curCommandCancellable._bluebird.default.resolve.then (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/driver.js:291:18)
[debug] [W3C (efdf97d8)]     at tryCatcher (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/util.js:26:23)
[debug] [W3C (efdf97d8)]     at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/promise.js:510:31)
[debug] [W3C (efdf97d8)]     at Promise._settlePromiseAt (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/promise.js:584:18)
[debug] [W3C (efdf97d8)]     at Promise._settlePromiseAtPostResolution (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/promise.js:248:10)
[debug] [W3C (efdf97d8)]     at Async._drainQueue (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/async.js:128:12)
[debug] [W3C (efdf97d8)]     at Async._drainQueues (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/async.js:133:10)
[debug] [W3C (efdf97d8)]     at Immediate.Async.drainQueues (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/async.js:15:14)
[debug] [W3C (efdf97d8)]     at runCallback (timers.js:705:18)
[debug] [W3C (efdf97d8)]     at tryOnImmediate (timers.js:676:5)
[debug] [W3C (efdf97d8)]     at processImmediate (timers.js:658:5)
[HTTP] <-- POST /wd/hub/session/efdf97d8-cf46-4ffb-b2d4-7d8feb931cee/execute/sync 405 11 ms - 1600
[HTTP] 
[HTTP] --> POST /wd/hub/session/efdf97d8-cf46-4ffb-b2d4-7d8feb931cee/execute/sync
[HTTP] {"script":"var source = document.documentElement.outerHTML; \nif (!source) { source = new XMLSerializer().serializeToString(document); }\nreturn source;","args":[]}
[debug] [W3C (efdf97d8)] Calling AppiumDriver.execute() with args: ["var source = document.documentElement.outerHTML; \nif (!source) { source = new XMLSerializer().serializeToString(document); }\nreturn source;",[],"efdf97d8-cf46-4ffb-b2d4-7d8feb931cee"]
[debug] [W3C (efdf97d8)] Encountered internal error running command: NotImplementedError: Method is not implemented
[debug] [W3C (efdf97d8)]     at AndroidDriver.extensions.execute (/usr/local/lib/node_modules/appium/node_modules/appium-android-driver/lib/commands/execute.js:12:9)
[debug] [W3C (efdf97d8)]     at curCommandCancellable._bluebird.default.resolve.then (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/driver.js:291:18)
[debug] [W3C (efdf97d8)]     at tryCatcher (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/util.js:26:23)
[debug] [W3C (efdf97d8)]     at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/promise.js:510:31)
[debug] [W3C (efdf97d8)]     at Promise._settlePromiseAt (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/promise.js:584:18)
[debug] [W3C (efdf97d8)]     at Promise._settlePromiseAtPostResolution (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/promise.js:248:10)
[debug] [W3C (efdf97d8)]     at Async._drainQueue (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/async.js:128:12)
[debug] [W3C (efdf97d8)]     at Async._drainQueues (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/async.js:133:10)
[debug] [W3C (efdf97d8)]     at Immediate.Async.drainQueues (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/async.js:15:14)
[debug] [W3C (efdf97d8)]     at runCallback (timers.js:705:18)
[debug] [W3C (efdf97d8)]     at tryOnImmediate (timers.js:676:5)
[debug] [W3C (efdf97d8)]     at processImmediate (timers.js:658:5)
[HTTP] <-- POST /wd/hub/session/efdf97d8-cf46-4ffb-b2d4-7d8feb931cee/execute/sync 405 4 ms - 1600
[HTTP] 
[HTTP] --> POST /wd/hub/session/efdf97d8-cf46-4ffb-b2d4-7d8feb931cee/execute/sync
[HTTP] {"script":"var source = document.documentElement.outerHTML; \nif (!source) { source = new XMLSerializer().serializeToString(document); }\nreturn source;","args":[]}
[debug] [W3C (efdf97d8)] Calling AppiumDriver.execute() with args: ["var source = document.documentElement.outerHTML; \nif (!source) { source = new XMLSerializer().serializeToString(document); }\nreturn source;",[],"efdf97d8-cf46-4ffb-b2d4-7d8feb931cee"]
[debug] [W3C (efdf97d8)] Encountered internal error running command: NotImplementedError: Method is not implemented
[debug] [W3C (efdf97d8)]     at AndroidDriver.extensions.execute (/usr/local/lib/node_modules/appium/node_modules/appium-android-driver/lib/commands/execute.js:12:9)
[debug] [W3C (efdf97d8)]     at curCommandCancellable._bluebird.default.resolve.then (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/driver.js:291:18)
[debug] [W3C (efdf97d8)]     at tryCatcher (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/util.js:26:23)
[debug] [W3C (efdf97d8)]     at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/promise.js:510:31)
[debug] [W3C (efdf97d8)]     at Promise._settlePromiseAt (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/promise.js:584:18)
[debug] [W3C (efdf97d8)]     at Promise._settlePromiseAtPostResolution (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/promise.js:248:10)
[debug] [W3C (efdf97d8)]     at Async._drainQueue (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/async.js:128:12)
[debug] [W3C (efdf97d8)]     at Async._drainQueues (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/async.js:133:10)
[debug] [W3C (efdf97d8)]     at Immediate.Async.drainQueues (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/node_modules/bluebird/js/main/async.js:15:14)
[debug] [W3C (efdf97d8)]     at runCallback (timers.js:705:18)
[debug] [W3C (efdf97d8)]     at tryOnImmediate (timers.js:676:5)
[debug] [W3C (efdf97d8)]     at processImmediate (timers.js:658:5)
[HTTP] <-- POST /wd/hub/session/efdf97d8-cf46-4ffb-b2d4-7d8feb931cee/execute/sync 405 9 ms - 1600
[HTTP]

2019-05-08 11:34:01 WARN [AppiumClient.$anonfun$getPageSource$1.340] get page source error
2019-05-08 11:34:01 WARN [Crawler.refreshPage.562] page source get fail, go back
2019-05-08 11:34:01 INFO [Crawler.setElementAction.660] set action to back
2019-05-08 11:34:01 INFO [Crawler.runStartupScript.236] first refresh
2019-05-08 11:34:01 INFO [Crawler.doElementAction.976] current element = _startupActions-Start-0
2019-05-08 11:34:01 INFO [Crawler.doElementAction.977] current index = 0
2019-05-08 11:34:01 INFO [Crawler.doElementAction.978] current action = 
2019-05-08 11:34:01 INFO [Crawler.doElementAction.979] current url = 
2019-05-08 11:34:01 INFO [Crawler.doElementAction.980] current xpath = startupActions-Start-0
2019-05-08 11:34:01 INFO [Crawler.doElementAction.981] current tag path = _startupActions-Start-0
2019-05-08 11:34:01 INFO [Crawler.doElementAction.982] current file name = _
2019-05-08 11:34:01 INFO [Crawler.doElementAction.983] current uri = 	startupActions-Start-0			startupActions
Exception in thread "main" java.util.NoSuchElementException: last of empty ListBuffer
	at scala.collection.mutable.ListBuffer.last(ListBuffer.scala:401)
	at com.testerhome.appcrawler.DataRecord.last(DataRecord.scala:40)
	at com.testerhome.appcrawler.Crawler.doElementAction(Crawler.scala:985)
	at com.testerhome.appcrawler.Crawler.runStartupScript(Crawler.scala:238)
	at com.testerhome.appcrawler.Crawler.start(Crawler.scala:152)
	at com.testerhome.appcrawler.AppCrawler$.startCrawl(AppCrawler.scala:344)
	at com.testerhome.appcrawler.AppCrawler$.parseParams(AppCrawler.scala:312)
	at com.testerhome.appcrawler.AppCrawler$.main(AppCrawler.scala:92)
	at com.testerhome.appcrawler.AppCrawler.main(AppCrawler.scala)

分析原因

我们看到Appcrawler中报了个get page source error，我们追查appcrawler的代码发现是在这里报错的：

override def getPageSource(): String = {
    currentPageSource=null
    currentPageDom=null
    log.info("start to get page source from appium")
    //获取页面结构, 最多重试3次
    1 to 3 foreach (i => {
      asyncTask(20)(driver.getPageSource) match {
        case Some(v) => {
          log.trace("get page source success")
          //todo: wda返回的不是标准的xml
          val xmlStr=v match {
            case json if json.trim.charAt(0)=='{' => {
              log.info("json format maybe from wda")
              DataObject.fromJson[Map[String, String]](v).getOrElse("value", "")
            }
            case xml if xml.trim.charAt(0)=='<' =>{
              log.info("xml format ")
              xml
            }
          }
          Try(XPathUtil.toDocument(xmlStr)) match {
            case Success(v) => {
              currentPageDom = v
            }
            case Failure(e) => {
              log.warn("convert to xml fail")
              log.warn(xmlStr)
              currentPageDom=null
            }
          }

          currentPageSource = XPathUtil.toPrettyXML(xmlStr)
          return currentPageSource
        }
        case None => {
          log.warn("get page source error")
        }
      }
    })
    currentPageSource
  }

我们看appium的源码，发现在appcrawler给我们的appium传递了一段js代码来获取控件树

{"script":"var source = document.documentElement.outerHTML; \nif (!source) { source = new XMLSerializer().serializeToString(document); }\nreturn source;","args":[]}

然而，我们的appium代码对get page source这个功能接口做了限制，源码在appium-android-driver中的lib/execute.js中：

extensions.execute = async function execute (script, args) {
  if (script.match(/^mobile:/)) {
    script = script.replace(/^mobile:/, '').trim();
    return await this.executeMobile(script, _.isArray(args) ? args[0] : args);
  }

  throw new errors.NotImplementedError();
};

我们可以看到，这里抛出异常了，说明可能是接口变动了，那么我这里有个大胆猜想，appcrawler所使用的java-client过老。

解决问题

ok，立马开始行动，替换上最新的java-client，也就是7.0，同时我们使用最新的appcrawler2.4.0

<dependency>
  <groupId>com.github.appium</groupId>
  <artifactId>java-client</artifactId>
  <version>v7.0.0</version>
</dependency>

同时添加对应的仓库：

<repository>
  <id>jitpack.io</id>
  <url>https://jitpack.io</url>
</repository>

打包

mvn assembly:assembly

打包完成，会再target目录下生成一个完整依赖的jar包：appcrawler-2.4.0-jar-with-dependencies.jar

重新执行，你会发现美妙的事情发生，最新appium完美支持！

提出质疑

上面的完美支持，是不是因为我更新了最新版本2.4.0，而不是使用的最开始的2.1.3版本呢？

有可能！！！

撤销修改，直接打包2.4.0，执行测试看是否正常。

结果就是：最开始的get page source问题没了，但出现另外一个问题：

2019-05-08 11:55:46 INFO [AppiumClient.30.initLog] already exist
Exception in thread "main" scala.MatchError: [app, appium, deviceName, dontStopAppOnReset, fullReset, noReset] (of class java.util.Collections$UnmodifiableSet)
        at com.testerhome.appcrawler.driver.AppiumClient.appium(AppiumClient.scala:94)
        at com.testerhome.appcrawler.driver.AppiumClient.<init>(AppiumClient.scala:40)
        at com.testerhome.appcrawler.Crawler.setupAppium(Crawler.scala:277)
        at com.testerhome.appcrawler.Crawler.restart(Crawler.scala:221)
        at com.testerhome.appcrawler.Crawler.crawl(Crawler.scala:201)
        at com.testerhome.appcrawler.Crawler.start(Crawler.scala:170)
        at com.testerhome.appcrawler.AppCrawler$.startCrawl(AppCrawler.scala:323)
        at com.testerhome.appcrawler.AppCrawler$.parseParams(AppCrawler.scala:291)
        at com.testerhome.appcrawler.AppCrawler$.main(AppCrawler.scala:91)
        at com.testerhome.appcrawler.AppCrawler.main(AppCrawler.scala)
chengmingdeMacBook-Pro:AppCrawler cmlanche$

[debug] [W3C (dd159942)] Encountered internal error running command: NoSuchDriverError: A session is either terminated or not started
[debug] [W3C (dd159942)]     at asyncHandler (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/protocol/protocol.js:298:15)
[debug] [W3C (dd159942)]     at asyncHandler (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/protocol/protocol.js:489:15)
[debug] [W3C (dd159942)]     at Layer.handle [as handle_request] (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/layer.js:95:5)
[debug] [W3C (dd159942)]     at next (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/route.js:137:13)
[debug] [W3C (dd159942)]     at Route.dispatch (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/route.js:112:3)
[debug] [W3C (dd159942)]     at Layer.handle [as handle_request] (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/layer.js:95:5)
[debug] [W3C (dd159942)]     at /usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:281:22
[debug] [W3C (dd159942)]     at param (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:354:14)
[debug] [W3C (dd159942)]     at param (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:365:14)
[debug] [W3C (dd159942)]     at Function.process_params (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:410:3)
[debug] [W3C (dd159942)]     at next (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:275:10)
[debug] [W3C (dd159942)]     at logger (/usr/local/lib/node_modules/appium/node_modules/morgan/index.js:144:5)
[debug] [W3C (dd159942)]     at Layer.handle [as handle_request] (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/layer.js:95:5)
[debug] [W3C (dd159942)]     at trim_prefix (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:317:13)
[debug] [W3C (dd159942)]     at /usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:284:7
[debug] [W3C (dd159942)]     at Function.process_params (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:335:12)
[debug] [W3C (dd159942)]     at next (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:275:10)
[debug] [W3C (dd159942)]     at jsonParser (/usr/local/lib/node_modules/appium/node_modules/body-parser/lib/types/json.js:110:7)
[debug] [W3C (dd159942)]     at Layer.handle [as handle_request] (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/layer.js:95:5)
[debug] [W3C (dd159942)]     at trim_prefix (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:317:13)
[debug] [W3C (dd159942)]     at /usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:284:7
[debug] [W3C (dd159942)]     at Function.process_params (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:335:12)
[debug] [W3C (dd159942)]     at next (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:275:10)
[debug] [W3C (dd159942)]     at Layer.handle [as handle_request] (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/layer.js:91:12)
[debug] [W3C (dd159942)]     at trim_prefix (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:317:13)
[debug] [W3C (dd159942)]     at /usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:284:7
[debug] [W3C (dd159942)]     at Function.process_params (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:335:12)
[debug] [W3C (dd159942)]     at next (/usr/local/lib/node_modules/appium/node_modules/express/lib/router/index.js:275:10)
[HTTP] <-- GET /wd/hub/session/dd159942-ed6d-411c-8dcc-b43d7fc26284/source 404 5 ms - 3173

还是错的！

那就用我的最新java-client 7.0吧，重新执行一次完整的测试，发现没有任何问题！