Hanged TeamCity builds when cloning git repository

Last time we started watching nodejs hanging on several TeamCity build agents when they builds website static content with webpack. Hanging was detected when nodejs was loading npm-package from our locally installed GitLab Community repository (here and after gitlab.com):

  • nodejs cloned git-repository by ssh
  • git executes ssh
  • ssh thought git-host untrusted and issue user prompt for adding this host to known_hosts file like:

    The authenticity of host 'gitlab.com (<ip-address>)' can't be established.
    RSA key fingerprint is SHA256:<public-key>.
    Are you sure you want to continue connecting (yes/no)?
    
  • executing git-cloning hanging until user enters yes in console

TL;DR

This kind of hanging take place on different build agents. Of course we had already added our gitlab server to known_hosts on each of them on first git clone command:

gitlab.com ssh-rsa <public-key>
gitlab.com ecdsa-sha2-nistp256 <public-key>

And sometimes by unknown reasons known_hosts was empty.

Initial thoughts

The first try to fix things down was to add gitlab.com automatically to known_hosts right before build. We wrote simple batch-file:

set GIT_KEY1=<public-key>
set GIT_KEY2=<public-key>
(echo gitlab.com ssh-rsa %GIT_KEY1% && echo gitlab.com ecdsa-sha2-nistp256 %GIT_KEY2%) > %USERPROFILE%\.ssh\known_hosts

This solution did help on some hosts, but on others this didn't help. In some hosts there was warning message:

Warning: Permanently added the RSA host key for IP address '<ip>' to the list of known hosts.

In some hanged hosts, right after entering yes in console, ip address with public key has beed added to known_hosts by ssh, in form:

<ip> ecdsa-sha2-nistp256 <public-key>

We guessed we need to write dns-name and ip address directly to known_hosts. But ip address is subject to change and we need to resolve dns-name to ip address.

In windows hosts resolving dns-name can be done with nslookup utility: nslookup gitlab.com. Unforunately, it is very difficult to parse output of this utility to batch variables.

Powershell to the resque

Then we implement a powershell script for this task.

# ps-patch-knownhost.ps1 file
$GIT_KEY1 = '<public-key>'
$GIT_KEY2 = '<public-key>'
$ipAddresses = [System.Net.Dns]::GetHostAddresses("gitlab.com") | where { $_.AddressFamily -eq [System.Net.Sockets.AddressFamily]::InterNetwork }
foreach ($ip in $ipAddresses) {
    Write-Host "gitlab.com,$ip ssh-rsa $GIT_KEY1" -NoNewline
    Write-Host ""
    Write-Host "gitlab.com,$ip ecdsa-sha2-nistp256 $GIT_KEY2" -NoNewline
    Write-Host ""
}

In this script we resolve ip addresses with .NET Framework class System.Net.Dns, and filter them to contain only IPv4 adddresses. To stream two strings into stdout we use Write-Host with -NoNewline option because in other cases powershell may break lines by console width (known_host will be corrupted then). And Write-Host with empty string is used just to add new lines.

It is important to note that known_hosts file always should be ends with new line.

Running powershell script from batch file

So, powershell script is written and we need to run it from command line. Powershell has several ways to do this.

The standard way is to use -File command line argument to specify concrete script:

powershell -File ps-patch-knownhost.ps1

Unfortunately, it is turned off in powershell by security reasons. To turn it on, run this command from command line with administrator permissions:

powershell -Command "Set-ExecutionPolicy -ExecutionPolicy RemoteSigned"

This way is not secure and not recommended on production servers (but it is OK to use on development).

There was hope that we can start powershell script from stdin. Fast found way to do this, but get error:

type "ps-patch-knownhost.ps1" | powershell -noprofile -noninteractive -command { $input | iex }
"iex" is not recognized as an internal or external command, operable program or batch file.

type "ps-patch-knownhost.ps1" | powershell -noprofile -noninteractive -command '$input | iex'
"iex'" is not recognized as an internal or external command, operable program or batch file.

type "ps-patch-knownhost.ps1" | powershell -noprofile -noninteractive -command '$input | Invoke-Expression'
"Invoke-Expression'" is not recognized as an internal or external command, operable program or batch file

Finally, I got this command works and patch knownhost file:

(powershell -NoLogo -ExecutionPolicy Unrestricted -Command "& '.\ps-patch-knownhost.ps1'") > %USERPROFILE%\.ssh\known_hosts

Summary

To patch known_hosts file you can use powershell script with ip address resolving for dns name:

# ps-patch-knownhost.ps1 file
$GIT_KEY1 = '<public-key>'
$GIT_KEY2 = '<public-key>'
$ipAddresses = [System.Net.Dns]::GetHostAddresses("gitlab.com") | where { $_.AddressFamily -eq [System.Net.Sockets.AddressFamily]::InterNetwork }
foreach ($ip in $ipAddresses) {
    Write-Host "gitlab.com,$ip ssh-rsa $GIT_KEY1" -NoNewline
    Write-Host ""
    Write-Host "gitlab.com,$ip ecdsa-sha2-nistp256 $GIT_KEY2" -NoNewline
    Write-Host ""
}

and run it from batch file:

(powershell -NoLogo -ExecutionPolicy Unrestricted -Command "& '.\ps-patch-knownhost.ps1'") > %USERPROFILE%\.ssh\known_hosts

Happy troubleshooting!

Comments