New crc release start failed on RHEL and MacOS where previous release deployed #815
Comments
morningspace
changed the title
morningspace
changed the title
morningspace
changed the title
|
You are probably in an inconsistent state by removing the ~/.crc folder
manually. Please use the hypervisor manager, or the CLI tool on Linux to
remove the artifacts. like `virsh undefine crc`, after which you perform
`crc delete -f`. (note the force option)
The same issue happened even I remove ~/.crc.
This is the reason why you get this error.
…On Mon, Nov 18, 2019 at 1:11 PM MorningSpace ***@***.***> wrote:
General information
- OS: Linux / macOS
- Hypervisor: KVM / hyperkit
- Did you run crc setup before starting it (Yes)?
CRC version
$ crc version
crc version: 1.1.0+95966a9
OpenShift version: 4.2.2 (embedded in binary)
CRC status
# For RHEL
$ crc status
Machine 'crc' does not exist. Use 'crc start' to create it.
# For Mac
$ crc status
ERRO error: stat /Users/morningspace/.crc/machines/crc/kubeconfig: no such file or directory
- exit status 1
CRC config
# Put the output of `crc config view`
Nothing returned
Host Operating System
# Put the output of `cat /etc/os-release` in case of Linux
NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.6 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.6"
# Put the output of `sw_vers` in case of Mac
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.15.1
BuildVersion: 19B88
Steps to reproduce
1. Both the MacOS and RHEL machines have been used to successfully
deploy a previous release of crc (1.0.0).
2. After I download the new release (1.1.0), and run crc setup, then crc
start:
- On RHEL, it failed at domain 'crc' already exists with uuid xxxxx,
e.g.:
INFO Extracting bundle: crc_libvirt_4.2.2.crcbundle ...
INFO Creating CodeReady Containers VM for OpenShift 4.2.2...
ERRO Error creating host: Error creating the VM: Error creating machine: Error in driver during machine creation: virError(Code=9, Domain=20, Message='operation failed: domain 'crc' already exists with uuid 84b5685a-03bd-4a27-9d44-266d8f2a9272')
- On MacOS, it keeps hanging at INFO Creating CodeReady Containers VM
for OpenShift 4.2.2... for ever. I've been waiting for a few hours,
before press Ctrl+C. e.g.:
$ crc start -p ~/.crc/pull-secret.txt
INFO Checking if running as non-root
INFO Checking if oc binary is cached
INFO Checking if HyperKit is installed
INFO Checking if crc-driver-hyperkit is installed
INFO Checking file permissions for /etc/resolver/testing
INFO Checking file permissions for /etc/hosts
INFO Extracting bundle: crc_hyperkit_4.2.2.crcbundle ...
INFO Creating CodeReady Containers VM for OpenShift 4.2.2...
1. Run crc stop, crc delete, crc status after start failed.
On RHEL:
$ crc stop
Machine 'crc' does not exist. Use 'crc start' to create it.
$ crc delete
Machine 'crc' does not exist. Use 'crc start' to create it.
$ crc status
Machine 'crc' does not exist. Use 'crc start' to create it.
On MacOS:
$ crc stop
Stopping the OpenShift cluster, this may take a few minutes...
ERRO Machine "crc" is already stopped.
$ crc delete
Do you want to delete the OpenShift cluster? [y/N]: y
Deleted the OpenShift cluster
$ crc status
Machine 'crc' does not exist. Use 'crc start' to create it.
1. The same issue happened even I remove ~/.crc.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#815>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAOZRVGQJMBBL76H6QBN3QUIPYNANCNFSM4JOOF4GA>
.
--
Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]
|
|
Hmmm... so, it's for RHEL, how about MacOS... I tried But, after that, when I run INFO Creating CodeReady Containers VM for OpenShift 4.2.2...
ERRO Error creating host: Error creating the VM: Error creating machine: Error in driver during machine creation: virError(Code=9, Domain=20, Message='operation failed: domain 'crc' already exists with uuid 84b5685a-03bd-4a27-9d44-266d8f2a9272')It makes me confused. I guess I can revert my machine to a clean state (since it's a VM), so as to avoid the error. But I'd like to know whether there's way to fix it before I revert it. |
|
you likely used an administrative account:
`sudo virsh undefine crc`
For macOS? Remove the ~/.crc/machines folder, use `crc delete -f` and try
to start again.
…On Mon, Nov 18, 2019 at 3:38 PM MorningSpace ***@***.***> wrote:
Hmmm... so, it's for RHEL, how about MacOS...
I tried virsh undefine crc, but got the following error:
error: failed to get domain 'crc'
error: Domain not found: no domain with matching name 'crc'
But, after that, when I run crc start again, it says:
INFO Creating CodeReady Containers VM for OpenShift 4.2.2...
ERRO Error creating host: Error creating the VM: Error creating machine: Error in driver during machine creation: virError(Code=9, Domain=20, Message='operation failed: domain 'crc' already exists with uuid 84b5685a-03bd-4a27-9d44-266d8f2a9272')
It makes me confused. I guess I can revert my machine to a clean state
(since it's a VM), so as to avoid the error. But I'd like to know whether
there's way to fix it before I revert it.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#815>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAOZSN6T7H7ZLLQGOR26DQUJA7XANCNFSM4JOOF4GA>
.
--
Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]
|
|
Thanks @gbraad, I have successfully launched crc on RHEL based on your comment above. But for MacOS, it still failed, any ideas? $ crc start -p ~/pull-secret.txt
INFO Checking if running as non-root
INFO Checking if oc binary is cached
INFO Checking if HyperKit is installed
INFO Checking if crc-driver-hyperkit is installed
INFO Checking file permissions for /etc/resolver/testing
INFO Checking file permissions for /etc/hosts
INFO Starting CodeReady Containers VM for OpenShift 4.2.2...
ERRO Failed to connect to the CRC VM with SSH
$ crc status
ERRO error: stat /Users/moyingbj/.crc/machines/crc/kubeconfig: no such file or directory
- exit status 1
$ ls ~/.crc/machines/crc/
config.json console-ring crc.disk hyperkit.json hyperkit.pid tty |
|
/Users/moyingbj/.crc/machines/crc/kubeconfig:
have seen this before, but I would suggest to search the issue tracker or
create a new issue around this.
This kind of error should in fact never completely prevent startup to
happen, so it is surely an issue.
…On Tue, Nov 19, 2019 at 12:25 PM MorningSpace ***@***.***> wrote:
Thanks @gbraad <https://github.com/gbraad>, I have successfully launched
crc on RHEL based on your comment above
<#815 (comment)>.
But for MacOS, it still failed, any ideas?
$ crc start -p ~/pull-secret.txt
INFO Checking if running as non-root
INFO Checking if oc binary is cached
INFO Checking if HyperKit is installed
INFO Checking if crc-driver-hyperkit is installed
INFO Checking file permissions for /etc/resolver/testing
INFO Checking file permissions for /etc/hosts
INFO Starting CodeReady Containers VM for OpenShift 4.2.2...
ERRO Failed to connect to the CRC VM with SSH
$ crc status
ERRO error: stat /Users/moyingbj/.crc/machines/crc/kubeconfig: no such file or directory
- exit status 1
$ ls ~/.crc/machines/crc/
config.json console-ring crc.disk hyperkit.json hyperkit.pid tty
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#815>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAOZWQE5474VQKG3NN5O3QUNTFJANCNFSM4JOOF4GA>
.
--
Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]
|
|
@morningspace Can you share you mac details like |
|
This looks like a partially created crc instance, so I'd start with |
|
@cfergeau I did run delete before start, just it did not work... |
|
Yup, if there are no more logs, it looks like ssh connection to 192.168.64.56 (VM IP) fails. |
|
does a manual
ssh -i /Users/moyingbj/.crc/cache/crc_hyperkit_4.2.2/id_rsa_crc core@192.168.64.56
work?
|
|
@gbraad I tried, but no luck: $ ssh -i /Users/moyingbj/.crc/cache/crc_hyperkit_4.2.2/id_rsa_crc core@192.168.64.56
ssh: connect to host 192.168.64.56 port 22: Operation timed outA couple of interesting things I noticed, and hope that helps to debug the issue. Before the ssh timeout error, from the log, it seems crc was trying to lookup the IP for the VM via dhcp. It got the result after a few retries, e.g. (crc) DBG | dhcp entry: {Name:crc-shdl4-master-0 IPAddress:192.168.64.57 HWAddress:8a:5d:af:b4:a9:23 ID:1,8a:5d:af:b4:a9:23 Lease:0x5dd717f3}
(crc) DBG | Found match: 8a:5d:af:b4:a9:23
(crc) DBG | IP: 192.168.64.57Suppose that's the VM IP, 192.168.64.52 api.crc.testing oauth-openshift.apps-crc.testingAnd, the port 53
nameserver 192.168.64.52
search_order 1Here, Then, the interesting thing is, if I try to modify /etc/hosts manually and save, I can see an Error like this: $ ls -l /etc/hosts
-rw-r--r-- 1 moyingbj wheel 2094 Nov 21 08:33 /etc/hostsThis keeps happening until I change the owner of the parent folder /private/etc to be $ ls -l /private/
total 0
drwxr-xr-x 128 moyingbj wheel 4096 Nov 21 08:40 etc
...But even with that, it looks still I cannot get the crc up. Also, by using the new VM IP, neither I can ping or ssh to it: $ ping 192.168.64.57
PING 192.168.64.57 (192.168.64.57): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
^C
--- 192.168.64.57 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet lossMy previous log pasted above was truncated for its first half part due to too many ssh retries which exceeded the terminal buffer, here's a complete one with less retries: crc-start.log Also to note, I just upgrade my MacOS to Catalina v10.15.1. And, before that, I use High Sierra v10.13.4, which never see such an issue. |
|
If you crc delete/crc start, then yes, the VM ip will change. At the beginning of the previous comment, you try to ssh to 192.168.64.56, but then you say the VM IP is .57, so a failure seems normal. Your permissions issues are not when trying to modify /etc/hosts, but when your editor tries to create a temporary file in the same directory. This is expected to fail. |
The log you uploaded seems truncated too? It ends with ^C and is not showing the messages which would be shown after a successful crc start. |
Yep, it's for the run prior to the IP changed to 57, the log that I pasted here is the next run which the IP was increased to 57. But no matter its 56 or 57, all the same.
Exactly, that's what I'm confused, and probably the cause to the issue.
No, I just stop it by manually pressing Ctrl + C, after a few retries. Actually, crc keeps retry until failed at last. |
|
Hi @morningspace Can you try following on that mac and let us know if still the same issue? |
|
@morningspace I upgraded to Catalina and didn't see any issue to run the CRC, Are you able to check with your colleagues if they are facing the same? |
|
@morningspace Same here as @praveenkumar. I am on Catalina but once I work around #836 , everything works fine and cluster comes up. |
|
@praveenkumar It's good to know that's not reproducible on your side. I am asking my colleague today to have a try on his Catalina Mac, and will cycle back when done. @zeenix I checked #836, and tried the workaround, but didn't see the warning dialog. It looks my Mac is in a very special situation, which is good to others but bad to me :-) |
@morningspace Thanks, we are waiting for the feedback on this one, do let us know. |
|
@praveenkumar , sorry for my late response! My colleague did try on his Mac a few days ago and cannot reproduce the issue as well... So, I assume that's an issue particular to my Mac if there's no other people reporting similar issue on Catalina, neither new installation nor upgrade... With that, I'm happy to close it for now, and keep eyes on it. Thanks a lot for taking time to look into it! |
|
I am getting the same issue here. Exactly as per @morningspace |
|
It turned out to be because of a VPN "Cisco Anyconnect" - when this was running, I got the same behaviour as you. When this was not running and closed, crc behaved as expected. |
|
Aha, that sounds a reasonable cause... I've taken a new MacBook and deployed a new CRC, so that's never happened... But, IIRC, I was seeing the same issue happened when I was in office where I don't need Cisco AnyConnect. |
|
I solved it in RHEL.
|
General information
crc setupbefore starting it (Yes)?CRC version
$ crc version crc version: 1.1.0+95966a9 OpenShift version: 4.2.2 (embedded in binary)CRC status
CRC config
# Put the output of `crc config view` Nothing returnedHost Operating System
Steps to reproduce
crc setup, thencrc start:domain 'crc' already exists with uuid xxxxx, e.g.:INFO Creating CodeReady Containers VM for OpenShift 4.2.2...for ever. I've been waiting for a few hours, before press Ctrl+C. e.g.:crc stop,crc delete,crc statusafter start failed.~/.crc.The text was updated successfully, but these errors were encountered: