UNKNOWN - "http response error: 401 - invalid content type #8

aheinhold · 2020-08-19T10:57:28Z

Hello,

i tested the check_by_powershell with the following commandline calls:

./check_by_powershell --host "hostname.fqdn" --auth NTLM --user "DOMAIN\domainuser" --password 'password' --icingacmd "Invoke-IcingaCheckCPU"
./check_by_powershell --host "hostname.fqdn" --auth NTLM --user "hostname\localuser" --password 'password' --icingacmd "Invoke-IcingaCheckCPU"
./check_by_powershell --host "hostname.fqdn" --auth NTML --user "DOMAIN\domainuser" --password 'password' --icingacmd "Invoke-IcingaCheckCPU"
./check_by_powershell --host "hostname.fqdn" --user "hostname\localuser" --password 'password' --icingacmd "Invoke-IcingaCheckCPU"
./check_by_powershell --host hostname.fqdn --auth NTML --user "DOMAIN\domainuser" --password 'password' --icingacmd "Invoke-IcingaCheckCPU"
./check_by_powershell --host hostname.fqdn --auth NTML --user "domainuser@DOMAIN" --password "password" --icingacmd "Invoke-IcingaCheckCPU"
./check_by_powershell --host hostname.fqdn --auth NTML --user "domainuser@DOMAIN" --password 'password' --icingacmd "Invoke-IcingaCheckCPU"
./check_by_powershell --host xxx.xxx.xxx.xxx --auth "NTML" --user "domainuser" --password "password" --icingacmd "Invoke-IcingaCheckCPU"
./check_by_powershell --host xxx.xxx.xxx.xxx --auth "NTML" --user "localuser" --password "password" --icingacmd Invoke-IcingaCheckCPU
./check_by_powershell --host xxx.xxx.xxx.xxx --auth NTML --user "DOMAIN\domainuser" --password "password" --icingacmd "Invoke-IcingaCheckCPU"
./check_by_powershell --host xxx.xxx.xxx.xxx --auth NTML --user "domainuser" --password "password" --icingacmd "Invoke-IcingaCheckCPU"
./check_by_powershell --host xxx.xxx.xxx.xxx --auth NTML --user "domainuser@DOMAIN" --password "password" --icingacmd "Invoke-IcingaCheckCPU"

The result was the followng message:
UNKNOWN - "http response error: 401 - invalid content type"

ref/NC/649281

The text was updated successfully, but these errors were encountered:

pdorschner · 2020-08-19T14:31:59Z

At the moment there is no option for AD-Users. This should be implemented.

We evaluate this problem. I discussed that with @LordHepipud

lazyfrosch · 2020-08-28T14:08:07Z

Usually NTLM should work with domains, but be aware of the shall handling for \.

You need to use one of these:

--user 'domain\user'
--user "domain\\user"

Icinga will take care of escaping a single \ correctly from Director, Icinga 2 config will require vars.xxx = "domain\\user"

I still need to test --user "fqdn.corp.de\\user"

Also see #9

pdorschner · 2020-09-03T08:09:02Z

We have a new pre-release version. I would appreciate if you could test the new build v0.2.0 pre-release.

The error message: UNKNOWN - "http response error: 401 - invalid content type" could be a problem with the configuration of the windows system, e.g. Firewall, WinRM. On my tests I can execute a command like that with no errors:

./check_by_powershell-darwin-amd64 --host 'example.ip' --auth NTLM --user "DOMAIN\Username" --password 'examplePass' --icingacmd "Invoke-IcingaCheckCPU"

[OK] Check package "CPU Load" | 'core_2'=47.41397%;;;0;100 'core_total'=49.99663%;;;0;100 'core_0'=70.58625%;;;0;100 'core_1'=29.40701%;;;0;100 'core_3'=45.4517%;;;0;100

Unfortunately NTLM won't work like @lazyfrosch mentioned above

I still need to test --user "fqdn.corp.de\user"

If you have feedback, please write us in #11

lazyfrosch · 2020-09-07T07:11:33Z

Okay, let's discuss the problems here, #11 is for a TLS client certificate, and not TLS connections by default.

lazyfrosch · 2020-09-07T07:11:56Z

@K0nne wrote in #11

--snip--

Hi!

I tested the latest release against our own test server:

./check_by_powershell-linux-amd64 --version
check_by_powershell version 0.2.0 - 72488a89b2e9fcf36e244b3394687e23838af55f

Test1:

./check_by_powershell-linux-amd64 --host 'IP' --auth NTLM --user 'DOMAIN\\USR' --password 'PWD'  --icingacmd 'Invoke-IcingaCheckCPU'
./check_by_powershell-linux-amd64 --host 'FQDN' --auth NTLM --user 'DOMAIN\\USR' --password 'PWD'  --icingacmd 'Invoke-IcingaCheckCPU'

Here we get the following error:

UNKNOWN - execution failed: execution of remote cmd failed: http response error: 401 - invalid content type

Test2:

./check_by_powershell-linux-amd64 --host 'IP' --auth NTLM --user 'DOMAIN\USR' --password 'PWD'  --icingacmd 'Invoke-IcingaCheckCPU'
./check_by_powershell-linux-amd64 --host 'IP' --auth NTLM --user 'USR' --password 'PWD'  --icingacmd 'Invoke-IcingaCheckCPU'
./check_by_powershell-linux-amd64 --host 'FQDN' --auth NTLM --user 'DOMAIN\USR' --password 'PWD'  --icingacmd 'Invoke-IcingaCheckCPU'
./check_by_powershell-linux-amd64 --host 'FQDN' --auth NTLM --user 'USR' --password 'PWD'  --icingacmd 'Invoke-IcingaCheckCPU'

With the other spelling of the user we get this message:
UNKNOWN - execution failed: execution of remote cmd failed: Command has already been closed

We will investigate this with our colleagues.

lazyfrosch · 2020-09-07T07:14:05Z

@K0nne: You are not using a HTTPS/TLS connection, therefor you would need to add --tls as option, or reconfigure winrm:

winrm set winrm/config/service '@{AllowUnencrypted="true"}'

I guess it might be a good idea to change the --tls switch, so TLS is used by default, and HTTP can be used when needed...

K0nne · 2020-09-07T07:32:49Z

@aheinhold can you please take over?
@lazyfrosch I'm currently on vacation.

lazyfrosch · 2020-09-07T07:34:42Z

@K0nne enjoy! 🍹

lazyfrosch · 2020-09-07T13:09:09Z

@aheinhold I've updated the README, please tell me if I can help 😉

aheinhold · 2020-09-08T06:54:57Z

Hi @lazyfrosch
i downloaded and compiled the check_by_winrm package.

After this i tested the check as follows:

./check_by_winrm -H FQDN --user 'DOMAIN\USR' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU'
UNKNOWN - Timeout reached
./check_by_winrm -H FQDN --user 'USR' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU'
UNKNOWN - Timeout reached
./check_by_winrm -H FQDN --user 'USR' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU' --timeout 60
UNKNOWN - execution failed: execution of remote cmd failed: unknown error Post "https://FQDN:5986/wsman": dial tcp IP:5986: i/o timeout
./check_by_winrm -H FQDN --user 'DOMAIN\USR' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU' --timeout 60
UNKNOWN - execution failed: execution of remote cmd failed: unknown error Post "https://FQDN:5986/wsman": dial tcp IP:5986: i/o timeout
./check_by_winrm -H IP --user 'DOMAIN\USR' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU' --timeout 60
UNKNOWN - execution failed: execution of remote cmd failed: unknown error Post "https://IP:5986/wsman": dial tcp IP:5986: i/o timeout
./check_by_winrm -H IP --user 'USR' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU' --timeout 60
UNKNOWN - execution failed: execution of remote cmd failed: unknown error Post "https://IP:5986/wsman": dial tcp IP:5986: i/o timeout
./check_by_winrm -H IP --user 'USR@DOMAIN' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU' --timeout 60
UNKNOWN - execution failed: execution of remote cmd failed: unknown error Post "https://IP:5986/wsman": dial tcp IP:5986: i/o timeout

The i/o timeout occurs after ~30 seconds
There is no additional output when using the argument --verbose

lazyfrosch · 2020-09-08T07:06:33Z

Have you configured WinRM on the host? When yes, how?
Is the firewall opened or disabled?
Can you access the HTTPs URL manually from your desktop?

aheinhold · 2020-09-09T06:21:08Z

Hi @lazyfrosch

sorry for my late response.
NOW i have configured the host for tls communication... I couldn't find out how the host was configured before.

I configured the host as follows:

requested a certificate for the host.
imported the certificate in the local machine cert store
Get-ChildItem -Path cert:\LocalMachine\My -Recurse; -> copied the thumbprint
$CertThumbprint = 'my-certificates-thumbprint';
Enable-PSRemoting -SkipNetworkProfileCheck -Force;
New-Item -Path WSMan:\LocalHost\Listener -Transport HTTPS -Address * -CertificateThumbPrint $CertThumbprint -Force;
Set-Item WSMan:\localhost\Service\EnableCompatibilityHttpsListener -Value true;
tested the connection from the satellite via telnet over port 5986
tested the check_by_winrm-command

I got the following output:

./check_by_winrm -H FQDN --user 'DOMAIN\USR' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU
UNKNOWN - execution failed: execution of remote cmd failed: http response error: 401 - invalid content type

lazyfrosch · 2020-09-09T06:25:19Z

How about enabling Basic Auth?

Or using --auth ntlm

aheinhold · 2020-09-09T07:42:46Z

--auth ntlm

UNKNOWN - execution failed: execution of remote cmd failed: Command has already been closed

when I try to configure basic auth:

winrm set winrm/config/service/Auth '@{Basic="true"}'
WSManFault
    Message
        ProviderFault
            WSManFault
                Message = The config setting Basic cannot be changed because is controlled by policies. The policy would need to be set to "Not Configured" in order to change the config setting.

Error number:  -2144108406 0x8033808A
Cannot change GPO controlled setting.

Changing this settings is prohibited by GPO. Would it help for the tests if i let this be changed?

lazyfrosch · 2020-09-11T14:07:23Z

Well Basic Auth might not be the best choice for default, I opened #15 for that.

I've seen the error "Command has already been closed" once in the CI, but not sure where that comes from, I thought of a timing issue during setup of the CI VM.

Have you restarted winrm Restart-Service winrm or tried again later?

aheinhold · 2020-09-15T07:34:54Z

I've seen the error "Command has already been closed" once in the CI, but not sure where that comes from, I thought of a timing issue during setup of the CI VM.

I don't know. Maybe @K0nne can say something about it next week.

Have you restarted winrm Restart-Service winrm or tried again later?

Yes, i restarted WinRM after the configuration.

Result/Plugin output:

UNKNOWN - execution failed: execution of remote cmd failed: http response error: 401 - invalid content type

lazyfrosch · 2020-09-28T12:08:33Z

Invalid content type basically means that authentication didn't work. You should find some info in the security log of the server.

But when you use NTLM with TLS - which is default in master now - it should be fine.

I'm not sure what can cause the error UNKNOWN - execution failed: execution of remote cmd failed: Command has already been closed. Maybe we should check the logs on server side here as well.

It could also be related to additional security policies.

/cc @mwaldmueller

lazyfrosch · 2020-10-30T08:54:44Z

Current status: We can't really determine why NTLM authentication fails.

There is a limitation with the library, that you can only auth with users of the same domain as the computer, and not even local users. Azure/go-ntlmssp#16

For this issue that shouldn't be the cause.

Events recorded:

Audit Success	27.10.2020 07:23:49	Microsoft-Windows-Security-Auditing	4627	Group Membership	"Group membership information.

<non interesting details>

Audit Success	27.10.2020 07:23:49	Microsoft-Windows-Security-Auditing	4624	Logon	"An account was successfully logged on.

<non interesting details>

Audit Failure	27.10.2020 07:23:49	Microsoft-Windows-Security-Auditing	4776	Credential Validation	"The computer attempted to validate the credentials for an account.

Authentication Package:	MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon Account:	user
Source Workstation:	REMOTECOMPUTER
Error Code:	0xC0000064"

Audit Success	27.10.2020 07:23:49	Microsoft-Windows-Security-Auditing	4634	Logoff	"An account was logged off.

<non interesting details>

What stands out here, is that the logon is successful, and afterwards some verification fails.

K0nne · 2020-10-30T19:05:55Z

On Monday I will provide you some audit details from our pwsh-version and winrm.

K0nne · 2020-11-10T14:24:51Z

I executed both checks from the same satellite against the same target. To my surprise check_by_winrm produced no security eventlog entries. Instead it thows the http 401 message in the same second it is executed. Is it posssible that the check dies right after execution before it can send anything?

I've sent the logs from our incarnation to @LordHepipud.

pdorschner added the enhancement New feature or request label Aug 19, 2020

pdorschner self-assigned this Aug 19, 2020

lazyfrosch mentioned this issue Sep 7, 2020

Use a TLS connection by default #12

Merged

lazyfrosch added this to the 1.0.0 milestone Sep 7, 2020

aheinhold closed this as completed Jan 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UNKNOWN - "http response error: 401 - invalid content type #8

UNKNOWN - "http response error: 401 - invalid content type #8

aheinhold commented Aug 19, 2020 •

edited by lazyfrosch

Loading

pdorschner commented Aug 19, 2020

lazyfrosch commented Aug 28, 2020

pdorschner commented Sep 3, 2020 •

edited

Loading

lazyfrosch commented Sep 7, 2020

lazyfrosch commented Sep 7, 2020

lazyfrosch commented Sep 7, 2020

K0nne commented Sep 7, 2020

lazyfrosch commented Sep 7, 2020

lazyfrosch commented Sep 7, 2020

aheinhold commented Sep 8, 2020

lazyfrosch commented Sep 8, 2020

aheinhold commented Sep 9, 2020

lazyfrosch commented Sep 9, 2020

aheinhold commented Sep 9, 2020

lazyfrosch commented Sep 11, 2020

aheinhold commented Sep 15, 2020

lazyfrosch commented Sep 28, 2020

lazyfrosch commented Oct 30, 2020

K0nne commented Oct 30, 2020

K0nne commented Nov 10, 2020

UNKNOWN - "http response error: 401 - invalid content type #8

UNKNOWN - "http response error: 401 - invalid content type #8

Comments

aheinhold commented Aug 19, 2020 • edited by lazyfrosch Loading

pdorschner commented Aug 19, 2020

lazyfrosch commented Aug 28, 2020

pdorschner commented Sep 3, 2020 • edited Loading

lazyfrosch commented Sep 7, 2020

lazyfrosch commented Sep 7, 2020

lazyfrosch commented Sep 7, 2020

K0nne commented Sep 7, 2020

lazyfrosch commented Sep 7, 2020

lazyfrosch commented Sep 7, 2020

aheinhold commented Sep 8, 2020

lazyfrosch commented Sep 8, 2020

aheinhold commented Sep 9, 2020

lazyfrosch commented Sep 9, 2020

aheinhold commented Sep 9, 2020

lazyfrosch commented Sep 11, 2020

aheinhold commented Sep 15, 2020

lazyfrosch commented Sep 28, 2020

lazyfrosch commented Oct 30, 2020

K0nne commented Oct 30, 2020

K0nne commented Nov 10, 2020

aheinhold commented Aug 19, 2020 •

edited by lazyfrosch

Loading

pdorschner commented Sep 3, 2020 •

edited

Loading