Git操作偶尔会在Windows上的Jenkins中挂起

We are running continuous Jenkins builds of a Git project hosted at Assembla. Jenkins is running on Tomcat 6 under its own user and generally works fine.

然而,每隔一段时间(例如每10次构建一次),构建作业开始时的结帐操作就会挂起。在其他时候,构建结束时的Git标记操作也会挂起。我相信这在命令行操作中不会发生(在具有相同用户的同一台主机上)。

挂起时,Windows进程树显示taskhost.exe? tomcat6.exe? git.exe? ssh.exe

当从外部查杀挂起标记命令的Git和ssh进程时,在作业控制台输出中会看到以下堆栈跟踪 - 该错误很奇怪,因为提到的目录已经存在并且已经安装了私钥。

hudson.plugins.git.GitException: Command "C:/Program Files (x86)/Git/bin/git.exe push  " returned status code 1:
stdout:
stderr: Could not create directory 'c/Users//.ssh'.
    at hudson.plugins.git.GitAPI.launchCommandIn(GitAPI.java:779)
    at hudson.plugins.git.GitAPI.launchCommand(GitAPI.java:741)

这里是进程和句柄的完整列表(除了Tomcat线程的长列表)。构建任务目前挂在拉。另一个奇怪的现象是,每5分钟就有一次计划的SCM轮询,但它没有运行几天 - 它可能也以某种方式被卡住了。

Process                PID      CPU      Private   Working     Description                                      Company Name
                                         Bytes     Set
-----------------------------------------------------------------------------------------------------------------------------------------
System Idle          Process 0  84.09        0 K       24 K
 Interrupts            n/a       0.28        0 K        0 K    Hardware Interrupts
 DPCs                  n/a       0.85        0 K        0 K    Deferred Procedure Calls
 System                   4                112 K      300 K
  smss.exe              240                620 K    1,196 K    Windows Session Manager                                 Microsoft Corporation
sppsvc.exe             2664              3,312 K    9,100 K    Microsoft Software Protection Platform Service          Microsoft Corporation
csrss.exe               344              3,516 K    5,120 K    Client Server Runtime Process                           Microsoft Corporation
 conhost.exe           1316              1,184 K    2,804 K    Console Window Host                                     Microsoft Corporation
 conhost.exe           3148              1,140 K    2,696 K    Console Window Host                                     Microsoft Corporation
wininit.exe             396              1,944 K    4,624 K    Windows Start-Up Application                            Microsoft Corporation
 services.exe           496              5,096 K   10,444 K    Services and Controller app                             Microsoft Corporation
  svchost.exe           616              4,776 K    9,940 K    Host Process for Windows Services                       Microsoft Corporation
   WmiPrvSE.exe        2468              2,692 K    6,052 K    WMI Provider Host                                       Microsoft Corporation
   dllhost.exe         2180      2.27    2,160 K    5,392 K    COM Surrogate                                           Microsoft Corporation
  svchost.exe           692      0.28    4,512 K    8,568 K    Host Process for Windows Services                       Microsoft Corporation
  svchost.exe           776              9,804 K   12,528 K    Host Process for Windows Services                       Microsoft Corporation
  svchost.exe           832             22,052 K   34,980 K    Host Process for Windows Services                       Microsoft Corporation
  svchost.exe           888              7,988 K   14,528 K    Host Process for Windows Services                       Microsoft Corporation
  svchost.exe           944              8,844 K   15,740 K    Host Process for Windows Services                       Microsoft Corporation
   dwm.exe              800              1,692 K    4,636 K    Desktop Window Manager                                  Microsoft Corporation
   dwm.exe             3908              1,800 K    4,748 K    Desktop Window Manager                                  Microsoft Corporation
  svchost.exe           984             13,036 K   17,004 K    Host Process for Windows Services                       Microsoft Corporation
  svchost.exe           284              8,536 K   11,152 K    Host Process for Windows Services                       Microsoft Corporation
  spoolsv.exe          1064              9,808 K   16,696 K    Spooler SubSystem App                                   Microsoft Corporation
  svchost.exe          1168              1,116 K    2,740 K    Host Process for Windows Services                       Microsoft Corporation
  Tomcat6.exe          1308      0.28  331,512 K  302,568 K    Commons Daemon Service Runner                           Apache Software Foundation
   git.exe             1812              3,308 K    4,880 K
    ssh.exe            2996              2,940 K    5,792 K
  vmtoolsd.exe         1352              8,260 K   12,892 K    VMware Tools Core Service                               VMware, Inc.
  VMUpgradeHelper.exe  1416              2,452 K    6,588 K    VMware virtual hardware upgrade helper application      VMware, Inc.
  svchost.exe          1880              3,796 K    9,224 K    Host Process for Windows Services                       Microsoft Corporation
   rdpclip.exe         3020              2,584 K    7,076 K    RDP Clip Monitor                                        Microsoft Corporation
   rdpclip.exe         4072              1,948 K    6,236 K    RDP Clip Monitor                                        Microsoft Corporation
  svchost.exe          1932              1,964 K    5,532 K    Host Process for Windows Services                       Microsoft Corporation
  dllhost.exe          1996              4,500 K   11,340 K    COM Surrogate                                           Microsoft Corporation
  msdtc.exe            1284              3,604 K    7,880 K    Microsoft Distributed Transaction Coordinator Service   Microsoft Corporation
  taskhost.exe         2492              3,076 K    6,252 K    Host Process for Windows Tasks                          Microsoft Corporation
  taskhost.exe         3548              2,896 K    6,088 K    Host Process for Windows Tasks                          Microsoft Corporation
 lsass.exe              504              8,516 K   16,548 K    Local Security Authority Process                        Microsoft Corporation
 lsm.exe                512              3,468 K    6,480 K    Local Session Manager Service                           Microsoft Corporation
csrss.exe               408              1,836 K    3,796 K    Client Server Runtime Process                           Microsoft Corporation
winlogon.exe            436              1,780 K    4,392 K    Windows Logon Application                               Microsoft Corporation
 LogonUI.exe            784              7,344 K   14,460 K    Windows Logon User Interface Host                       Microsoft Corporation
csrss.exe              2184              2,756 K    7,532 K    Client Server Runtime Process                           Microsoft Corporation
winlogon.exe           2952              1,960 K    5,192 K    Windows Logon Application                               Microsoft Corporation
explorer.exe           1836             23,536 K   45,060 K    Windows Explorer                                        Microsoft Corporation
 VMwareTray.exe        2168              2,824 K    6,400 K    VMware Tools tray application                           VMware, Inc.
 regedit.exe           2772              6,212 K    9,584 K    Registry Editor                                         Microsoft Corporation
 procexp64.exe         3648      11.93  21,904 K   37,056 K    Sysinternals Process Explorer                           Sysinternals - www.sysinternals.com
csrss.exe              3140              2,732 K    5,612 K    Client Server Runtime Process                           Microsoft Corporation
 conhost.exe           2500              1,312 K    3,452 K    Console Window Host                                     Microsoft Corporation
winlogon.exe           3172              1,900 K    4,980 K    Windows Logon Application                               Microsoft Corporation
explorer.exe            868             28,840 K   45,200 K    Windows Explorer                                        Microsoft Corporation
 VMwareTray.exe        3300              2,672 K    6,252 K    VMware Tools tray application                           VMware, Inc.
 rundll32.exe          3328              1,828 K    5,584 K    Windows host process (Rundll32)                         Microsoft Corporation
 cmd.exe               2832              2,240 K    2,588 K    Windows Command Processor                               Microsoft Corporation

Process: Tomcat6.exe Pid: 1308

Type        Name
Desktop      \Default
Directory    \KnownDlls
Directory    \BaseNamedObjects
Event        \BaseNamedObjects\TOMCAT6SIGNAL
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\commons-daemon.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\tomcat6-stdout.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\tomcat6-stdout.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\tomcat6-stderr.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\tomcat6-stderr.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0
File    C:\Users\\AppData\Local\Temp\hsperfdata_\1308
File    C:\Program Files\Java\jre7\lib\rt.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin\bootstrap.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin\tomcat-juli.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\catalina.2012-05-30.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\localhost.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\manager.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\host-manager.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\tomcat6-stderr.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\tomcat6-stdout.2012-05-24.log
File    C:\Program Files\Java\jre7\lib\ext\dnsns.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\annotations-api.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\catalina-ant.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\catalina-ha.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\catalina-tribes.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\catalina.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\ecj-3.7.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\el-api.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\jasper-el.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\jasper.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\jsp-api.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\servlet-api.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\tomcat-coyote.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\tomcat-dbcp.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\tomcat-i18n-es.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\tomcat-i18n-fr.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\tomcat-i18n-ja.jar
File    C:\Program Files\Java\jre7\lib\resources.jar
File    C:\Program Files\Java\jre7\lib\ext\localedata.jar
File    C:\Program Files\Java\jre7\lib\jsse.jar
File    C:\Program Files\Java\jre7\lib\jce.jar
File    C:\Program Files\Java\jre7\lib\ext\sunec.jar
File    C:\Program Files\Java\jre7\lib\ext\sunmscapi.jar
File    C:\Program Files\Java\jre7\lib\ext\sunjce_provider.jar
File    \
0
额外 编辑
意见: 1
是的,它是同一个用户。
额外 作者 user1414274,
无论如何,@VonC感谢您的帮助
额外 作者 user1414274,
@Vonc将ssh find的结果添加到问题主体中。控制台没有显示与以前不同的东西(单行 - 由用户启动,就是这样)。再次,构建卡住了,并且更改日志轮询已过时,并以“获取上游更改”行结束。顺便说一句,同样的詹金斯服务器上的另一项工作运行正常,而这项工作卡住了。
额外 作者 user1414274,
将添加跟踪。您是否期望在控制台输出中看到任何结果,或者在哪里可以看到跟踪输出?至于手柄 - 不知道你的意思要检查。上面的列表包含了机器上的所有内容(procexp以管理员身份运行)。
额外 作者 user1414274,
顺便说一句,我认为这是因为SCM投票悬而未决 - 我杀死了GIT和SSH进程,并且突然之间所有新的进程都被创建出来,并且工作开始真正从SCM中提取并运行。
额外 作者 user1414274,
见上面的问题主体
额外 作者 user1414274,
这里是一个完整的流程树,并处理jenkins当前挂在GIT结账处的转储。
额外 作者 user1414274,
将在下次发生时尝试
额外 作者 user1414274,
是的,所有相同的用户。我怀疑,如果问题只与用户有关,那么这根本就没有用。但它确实有效 - 并且只是偶尔挂起。
额外 作者 user1414274,
你可以在你的作业环境变量中尝试一个 GIT_TRACE = 1 以便在git ssh调用挂起时查看是否有任何有用的跟踪?你有没有检查引用这些可执行文件的句柄? (检查是否与其他进程有冲突)
额外 作者 VonC,
某种资源冲突然后......在杀死所有进程树之前,在 ssh.exe 上查找句柄(通过进程资源管理器: technet.microsoft.com/fr-fr/sysinternals/bb896653.aspx
额外 作者 VonC,
控制台应该足够了。关于句柄,其思想是检查是否有其他进程在 ssh.exe 上打开了句柄:在Process Explorer中使用 CTRL + F 并键入 ssh (见 geekyninja.com/archives/…
额外 作者 VonC,
很奇怪。我很抱歉不能提供任何确凿的答案,但我想确保涵盖通常的调试选项。
额外 作者 VonC,
并且 tomcat6.exe 进程也由同一用户拥有?
额外 作者 VonC,
user-name 与Jenkins运行的帐户相同吗?
额外 作者 VonC,
这个问题是否解决?
额外 作者 Maximin,

4 答案

这在以下讨论:

https://issues.jenkins-ci.org/browse/JENKINS-5413

您可以在脚本控制台中运行以下脚本:

Jenkins.instance.getTrigger("SCMTrigger").getRunners().each()
{
  item ->
    println(item.getTarget().name)
    println(item.getDuration())
    println(item.getStartTime())
    long millis = Calendar.instance.time.time - item.getStartTime()

    if(millis > (1000 * 60 * 3))//1000 millis in a second * 60 seconds in a minute * 3 minutes
    {
      Thread.getAllStackTraces().keySet().each()
      { 
        tItem ->
          if (tItem.getName().contains("SCM polling") && tItem.getName().contains(item.getTarget().name))
          { 
            println "Interrupting thread " + tItem.getName(); 
            tItem.interrupt()
          }
       }
    }
}

至少要澄清问题...

0
额外
这会做什么?我刚刚得到:结果:[[email protected]]
额外 作者 Josh Santangelo,

在我的情况下,从 PuTTY网站下载plink并设置使用它的GIT_SSH环境变量解决了问题。

0
额外

我解决了它:

  • Git可执行文件路径设置为 C:\ Program Files文件(x86)\ Git \ cmd \ git.exe (即 cmd ,而不是bin!)和
  • 通过将环境变量 %HOME% 设置为 $ USERPROFILE
    (否则 HOME 默认为 $ HOMEDRIVE $ HOMEPATH ,在我的情况下是 H:\ ,但通常是 $ HOME >设置为 $ USERPROFILE )。
0
额外
我们有这两个设置,并且仍然遇到这个问题...
额外 作者 gliptak,
优秀。感谢您花时间更新这个旧问题。 +1
额外 作者 VonC,
很酷,谢谢。这对我帮助很大!
额外 作者 SGrebenkin,
我认为这已经解决了我的问题,但经过一周的健康,问题今天返回=(
额外 作者 brianfeucht,

我可能解决了这个问题。我最初通过在Linux机器上使用ssh-keygen生成SSH密钥来设置我的SSH密钥。这通常有效,但Git偶尔会从Jenkins挂起。

现在我已经尝试使用PuTTY工具为我的Git服务器设置基于密钥的身份验证,使用这些说明。然后,我将GIT_SSH环境变量设置为plink.exe的路径。它似乎现在正在工作,但它始终是一个间歇性问题。希望它不会返回...

0
额外