Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cant update to 2023.11.3 #4709

Closed
jensjakobandersen opened this issue Nov 14, 2023 · 51 comments
Closed

Cant update to 2023.11.3 #4709

jensjakobandersen opened this issue Nov 14, 2023 · 51 comments
Labels

Comments

@jensjakobandersen
Copy link

Describe the issue you are experiencing

Update to 2023.11.3 fails

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Steps to reproduce the issue

  1. Try to update supervisor - either from GUI or CLI

Anything in the Supervisor logs that might be useful for us?

23-11-14 21:11:00 INFO (MainThread) [supervisor.os.manager] Rauc: B - marked slot kernel.1 as good
23-11-14 21:11:00 INFO (MainThread) [supervisor.supervisor] Fetching AppArmor profile https://version.home-assistant.io/apparmor_stable.txt
23-11-14 21:11:00 INFO (MainThread) [supervisor.host.apparmor] Adding/updating AppArmor profile: hassio-supervisor
23-11-14 21:11:01 INFO (MainThread) [supervisor.supervisor] Update Supervisor to version 2023.11.3
23-11-14 21:11:01 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/aarch64-hassio-supervisor with tag 2023.11.3.
23-11-14 21:11:05 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
23-11-14 21:11:05 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state startup
23-11-14 21:11:05 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
23-11-14 21:11:05 WARNING (MainThread) [supervisor.jobs] 'StoreManager.reload' blocked from execution, supervisor needs to be updated first
23-11-14 21:11:22 INFO (MainThread) [supervisor.homeassistant.core] Detect a running Home Assistant instance
23-11-14 21:13:57 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3")
23-11-14 21:13:57 INFO (MainThread) [supervisor.resolution.module] Create new issue update_failed - supervisor / None
23-11-14 21:13:57 ERROR (MainThread) [supervisor.supervisor] Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3")
23-11-14 21:13:57 CRITICAL (MainThread) [supervisor.core] Can't update Supervisor! This will break some Add-ons or affect future versions of Home Assistant!

System Health information

System Information

version core-2023.10.3
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.11.5
os_name Linux
os_version 6.1.56
arch aarch64
timezone Europe/Copenhagen
config_dir /config
Home Assistant Community Store
GitHub API ok
GitHub Content ok
GitHub Web ok
GitHub API Calls Remaining 4954
Installed Version 1.33.0
Stage running
Available Repositories 1398
Downloaded Repositories 6
Home Assistant Cloud
logged_in true
subscription_expiration December 12, 2023 at 01:00
relayer_connected true
relayer_region eu-central-1
remote_enabled true
remote_connected true
alexa_enabled false
google_enabled false
remote_server eu-central-1-6.ui.nabu.casa
certificate_status ready
can_reach_cert_server ok
can_reach_cloud_auth ok
can_reach_cloud ok
Home Assistant Supervisor
host_os Home Assistant OS 11.0
update_channel stable
supervisor_version supervisor-2023.10.1
agent_version 1.6.0
docker_version 24.0.6
disk_total 28.0 GB
disk_used 6.1 GB
healthy failed to load: Unhealthy
supported true
board odroid-n2
supervisor_api ok
version_api ok
installed_addons deCONZ (6.20.0), Terminal & SSH (9.7.1)
Dashboards
dashboards 16
resources 2
views 16
mode storage
Recorder
oldest_recorder_run November 5, 2023 at 22:35
current_recorder_run November 14, 2023 at 21:11
estimated_db_size 155.77 MiB
database_engine sqlite
database_version 3.41.2
Sonoff
version 3.5.3 (a8c6d45)
cloud_online 7 / 9
local_online 0 / 0
debug failed to load:

Supervisor diagnostics

config_entry-hassio-6854af4adda5d85d74a459955abf37b0.json.txt

Additional information

Has also filed bug for 2023.11.1 here:
#4703

@frenck
Copy link
Member

frenck commented Nov 14, 2023

@jensjakobandersen It most definitely exists:

CleanShot 2023-11-14 at 21 22 12@2x

It has quite the downloads already as well.

You can see/test for yourself by taking the image name from the log you shared and put https in front of it. https://ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3

If this all fails, you might be dealing with corruption in your filesystem, or maybe you are simply running out of diskspace? You could try running ha supervisor repair in an terminal window to let the Supervisor trying to actively repair the situation.

../Frenck

@jensjakobandersen
Copy link
Author

Thank you. I ran the ha supervisor repair - it ran without any problems.

Can you explain me this part of the error message?

I dont run docker (I guess), I run the native HAOS on Odroid.
404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3/json

@frenck
Copy link
Member

frenck commented Nov 14, 2023

Can you explain me this part of the error message?

It means the local image cannot be found (the 404 means, not found on your system).

@jensjakobandersen
Copy link
Author

jensjakobandersen commented Nov 15, 2023 via email

@bazb
Copy link

bazb commented Nov 16, 2023

I have the same issue. Diskspace at 13% and the (very scary) ha supervisor repair didn't fix it.

@xgolubev
Copy link

Same here. I got 6Gb free space

@GJSchroeder
Copy link

Same issue for me. I tried running the "ha supervisor repair", which completed successfully but I still cannot do the update. I am only using 6% of my disk space, so that's not the issue. I can access https://ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3 from the browser.

Any other ideas for troubleshooting where the problem is?

@agners
Copy link
Member

agners commented Nov 20, 2023

@jensjakobandersen can you still reproduce that problem?

@bazb @xgolubev @GJSchroeder do you see the very same symptoms (Downloading docker image ghcr.io/home-assistant/aarch64-hassio-supervisor with tag 2023.11.3. followed by 23-11-14 21:13:57 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3")? Can you share maybe your Supervisor log as well to get a bit wider perspective on this issue?

@bazb
Copy link

bazb commented Nov 20, 2023

@agners Yes, same logs.

23-11-20 14:52:29 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3")
23-11-20 14:52:29 ERROR (MainThread) [supervisor.supervisor] Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3")

@GJSchroeder
Copy link

GJSchroeder commented Nov 20, 2023

@agners Yes, same error log for me as well. The only thing I can see that might be related is I installed HA on a new HA Yellow, and restored my backup from my HA running on Pi. That was the 2023.10.5 version. Then I tried to upgrade to 2023.11 and have never been able to because of this error. Everything else works fine though after the migration to Yellow.

23-11-20 20:58:53 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3")
23-11-20 20:58:53 ERROR (MainThread) [supervisor.supervisor] Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3")

@jensjakobandersen
Copy link
Author

jensjakobandersen commented Nov 20, 2023 via email

@jensjakobandersen
Copy link
Author

jensjakobandersen commented Nov 20, 2023

I got this in the Home Assistant Core log as well:
Log details (ERROR)
Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/commands.py:226
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 15:18:47 (1 occurrences)
Last logged: 15:18:47

[281473038033344] Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3")
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 258, in async_install
await async_update_supervisor(self.hass)
File "/usr/src/homeassistant/homeassistant/components/hassio/handler.py", line 53, in _wrapper
raise HassioAPIError(data["message"])
homeassistant.components.hassio.handler.HassioAPIError: Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 226, in handle_call_service
await hass.services.async_call(
File "/usr/src/homeassistant/homeassistant/core.py", line 2012, in async_call
response_data = await coro
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/core.py", line 2049, in _execute_service
return await target(service_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
return await service.entity_service_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 876, in entity_service_call
response_data = await _handle_entity_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 948, in _handle_entity_call
result = await task
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/update/init.py", line 157, in async_install
await entity.async_install_with_progress(version, backup)
File "/usr/src/homeassistant/homeassistant/components/update/init.py", line 432, in async_install_with_progress
await self.async_install(version, backup)
File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 260, in async_install
raise HomeAssistantError(
homeassistant.exceptions.HomeAssistantError: Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.3")

@unhandyman
Copy link

Similar error here. I can't update Core from 2023.10.5 to 2023.11.3 - or update any other Addon - on my HAOS on a RasPi3.

Core Logs

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/commands.py:226
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 21. November 2023 um 21:30:27 (7 occurrences)
Last logged: 17:29:34

[548311224000] Error updating Home Assistant Core 'HomeAssistantCore.update' blocked from execution, was unable to update plugin(s) cli and all plugins must be up to date first
[547808230720] Error updating Home Assistant Core 'HomeAssistantCore.update' blocked from execution, was unable to update plugin(s) cli and all plugins must be up to date first
[547806811200] Error updating Home Assistant Core 'HomeAssistantCore.update' blocked from execution, was unable to update plugin(s) cli and all plugins must be up to date first
[548145128896] Error updating Home Assistant Core 'HomeAssistantCore.update' blocked from execution, was unable to update plugin(s) cli and all plugins must be up to date first
[547763761344] Error updating Home Assistant Core 'HomeAssistantCore.update' blocked from execution, was unable to update plugin(s) cli and all plugins must be up to date first
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 303, in async_install
    await async_update_core(self.hass, version=version, backup=backup)
  File "/usr/src/homeassistant/homeassistant/components/hassio/handler.py", line 53, in _wrapper
    raise HassioAPIError(data["message"])
homeassistant.components.hassio.handler.HassioAPIError: 'HomeAssistantCore.update' blocked from execution, was unable to update plugin(s) cli and all plugins must be up to date first

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 226, in handle_call_service
    await hass.services.async_call(
  File "/usr/src/homeassistant/homeassistant/core.py", line 2012, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2049, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 876, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 948, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/update/__init__.py", line 157, in async_install
    await entity.async_install_with_progress(version, backup)
  File "/usr/src/homeassistant/homeassistant/components/update/__init__.py", line 432, in async_install_with_progress
    await self.async_install(version, backup)
  File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 305, in async_install
    raise HomeAssistantError(
homeassistant.exceptions.HomeAssistantError: Error updating Home Assistant Core 'HomeAssistantCore.update' blocked from execution, was unable to update plugin(s) cli and all plugins must be up to date first

Supervisor Logs

23-11-22 17:29:34 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/aarch64-hassio-cli:2023.06.0 to ghcr.io/home-assistant/aarch64-hassio-cli:2023.10.0
23-11-22 17:29:34 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/aarch64-hassio-cli with tag 2023.10.0.
23-11-22 17:29:34 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/aarch64-hassio-cli:2023.10.0: 500 Server Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-cli:2023.10.0/json: Internal Server Error ("readlink /mnt/data/docker/overlay2: invalid argument")
23-11-22 17:29:34 ERROR (MainThread) [supervisor.plugins.cli] CLI update failed

I can access https://github.com/home-assistant/plugin-cli/pkgs/container/aarch64-hassio-cli via Browser. Tried several restarts, HA and host itself.

I see in my logs it says Server Error whereas the other logs say Client Error.

Is this related? Or can anyone point me in the right direction?

@nabob67
Copy link

nabob67 commented Nov 23, 2023

Hello guys i'm a Home Assistant new user, have the same problem
and resolve it by using dns 8.8.8.8 instead of my classic 192.168.1.1

@jensjakobandersen
Copy link
Author

jensjakobandersen commented Nov 23, 2023 via email

@agners
Copy link
Member

agners commented Nov 23, 2023

@jensjakobandersen what type of installation are you using? Are there maybe errors related to NetworkManager in the host logs?

@agners
Copy link
Member

agners commented Nov 23, 2023

What would be interesting here is the host logs of Docker after a failed attempt. You can use the following command to get them ha host logs -t dockerd. Maybe the host logs in general also have a clue (just ha host logs, or from the user interface log provider host).

@jensjakobandersen
Copy link
Author

jensjakobandersen commented Nov 23, 2023 via email

@jensjakobandersen
Copy link
Author

jensjakobandersen commented Nov 23, 2023 via email

@Frankymuc
Copy link

Frankymuc commented Nov 23, 2023

hi, the new version 2023.11.3 after 2 hours i have break because its hang and not update finish. i start HA and the version is 2023.11.2 but supervisor is 2023.11.3
the update window is also visible to update to 2023.11.3 if i will start after 1 or 2 seconds its finish but no 2023.11.3.. i can this do now again and again no error no update runing. i habe test to upgrade to 2023.11.0 and than open the update window to 11.3 but its no update to .3 have update to 2023.11.2 again and now its stay on .2
the update windows is always visible but not update.

what is wrong with this build? what can i do?

ps.: i used linux mx debian kde plasma and home asistant in virtual machine..oracle

@agners
Copy link
Member

agners commented Nov 29, 2023

Supervisor 2023.11.6 is now on the stable channel. If you encounter the problem again, please check the following things:

ha network info

Is host_internet and supervisor_internet set to true?

ha resolution info

Are there any issues listed?

@GJSchroeder
Copy link

@agners It still failed for me again trying to update to 2023.11.6.

I ran ha network info and it showed both host_internet and supervisor_internet set to true. I would include the entire output but got too frustrated trying to figure out how to copy/paste the output from a terminal window....what a mess that is.

I ran ha resolution info and got the following:

[core-ssh ~]$ ha resolution info
checks:

  • enabled: true
    slug: free_space
  • enabled: true
    slug: multiple_data_disks
  • enabled: true
    slug: addon_pwned
  • enabled: true
    slug: docker_config
  • enabled: true
    slug: network_interface_ipv4
  • enabled: true
    slug: core_security
  • enabled: true
    slug: dns_server_ipv6
  • enabled: true
    slug: supervisor_trust
  • enabled: true
    slug: backups
  • enabled: true
    slug: dns_server
    issues:
  • context: supervisor
    reference: null
    type: update_failed
    uuid: 75b9b0551abe49858be284d00d36c5a7
    suggestions: []
    unhealthy:
  • supervisor
    unsupported: []

@jensjakobandersen
Copy link
Author

jensjakobandersen commented Nov 29, 2023 via email

@xgolubev
Copy link

still the same problem

host_internet and supervisor_internet both are true

image

@jensjakobandersen
Copy link
Author

Sorry,. still cant update to 23.11.06

[281472299704896] Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6")
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 258, in async_install
await async_update_supervisor(self.hass)
File "/usr/src/homeassistant/homeassistant/components/hassio/handler.py", line 53, in _wrapper
raise HassioAPIError(data["message"])
homeassistant.components.hassio.handler.HassioAPIError: Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 226, in handle_call_service
await hass.services.async_call(
File "/usr/src/homeassistant/homeassistant/core.py", line 2012, in async_call
response_data = await coro
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/core.py", line 2049, in _execute_service
return await target(service_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
return await service.entity_service_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 876, in entity_service_call
response_data = await _handle_entity_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 948, in _handle_entity_call
result = await task

@jensjakobandersen
Copy link
Author

Ands from the Supervisor log:
23-11-29 21:41:59 INFO (MainThread) [supervisor.supervisor] Fetching AppArmor profile https://version.home-assistant.io/apparmor_stable.txt
23-11-29 21:41:59 INFO (MainThread) [supervisor.host.apparmor] Adding/updating AppArmor profile: hassio-supervisor
23-11-29 21:41:59 INFO (MainThread) [supervisor.supervisor] Update Supervisor to version 2023.11.6
23-11-29 21:41:59 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/aarch64-hassio-supervisor with tag 2023.11.6.

23-11-29 21:43:25 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6")
23-11-29 21:43:25 ERROR (MainThread) [supervisor.supervisor] Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6")

@jensjakobandersen
Copy link
Author

jensjakobandersen commented Nov 29, 2023

More Supervisor logs (with debug level log enabled and diagnostics enabled)

23-11-29 23:37:01 INFO (MainThread) [supervisor.supervisor] Repairing Supervisor 2023.10.1
23-11-29 23:37:01 DEBUG (MainThread) [supervisor.homeassistant.websocket] Sending: {'type': <WSType.SUPERVISOR_EVENT: 'supervisor/event'>, 'data': {'event': <WSEvent.JOB: 'job'>, 'data': {'name': 'docker_supervisor_retag', 'reference': 'hassio_supervisor', 'uuid': '9d6dde288a19417c91fae37894060e2e', 'progress': 0, 'stage': None, 'done': False, 'parent_id': None}}, 'id': 101}
23-11-29 23:37:01 DEBUG (MainThread) [supervisor.homeassistant.websocket] Received: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"id":101,"type":"result","success":true,"result":null}', extra='')
23-11-29 23:37:01 INFO (MainThread) [supervisor.core] Finished repair of Supervisor Environment
23-11-29 23:37:01 DEBUG (MainThread) [supervisor.homeassistant.websocket] Sending: {'type': <WSType.SUPERVISOR_EVENT: 'supervisor/event'>, 'data': {'event': <WSEvent.JOB: 'job'>, 'data': {'name': 'docker_supervisor_retag', 'reference': 'hassio_supervisor', 'uuid': '9d6dde288a19417c91fae37894060e2e', 'progress': 0, 'stage': None, 'done': True, 'parent_id': None}}, 'id': 102}
23-11-29 23:37:01 DEBUG (MainThread) [supervisor.homeassistant.websocket] Received: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"id":102,"type":"result","success":true,"result":null}', extra='')
23-11-29 23:37:03 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:37:03 DEBUG (MainThread) [supervisor.api.middleware.security] /info access from Home Assistant
23-11-29 23:37:03 DEBUG (MainThread) [supervisor.api.middleware.security] /core/info access from Home Assistant
23-11-29 23:37:03 DEBUG (MainThread) [supervisor.api.middleware.security] /core/stats access from Home Assistant
23-11-29 23:37:03 DEBUG (MainThread) [supervisor.api.middleware.security] /supervisor/info access from Home Assistant
23-11-29 23:37:03 DEBUG (MainThread) [supervisor.api.middleware.security] /supervisor/stats access from Home Assistant
23-11-29 23:37:03 DEBUG (MainThread) [supervisor.api.middleware.security] /os/info access from Home Assistant
23-11-29 23:37:05 DEBUG (MainThread) [supervisor.api.middleware.security] /addons/core_deconz/stats access from Home Assistant
23-11-29 23:37:05 DEBUG (MainThread) [supervisor.api.middleware.security] /addons/core_ssh/stats access from Home Assistant
23-11-29 23:37:07 DEBUG (MainThread) [supervisor.api.middleware.security] /addons/core_deconz/changelog access from Home Assistant
23-11-29 23:37:07 DEBUG (MainThread) [supervisor.api.middleware.security] /addons/core_ssh/changelog access from Home Assistant
23-11-29 23:37:07 DEBUG (MainThread) [supervisor.api.middleware.security] /addons/core_deconz/info access from Home Assistant
23-11-29 23:37:07 DEBUG (MainThread) [supervisor.api.middleware.security] /addons/core_ssh/info access from Home Assistant
23-11-29 23:37:09 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:37:14 DEBUG (MainThread) [supervisor.api.middleware.security] /backups access from Home Assistant
23-11-29 23:37:14 DEBUG (MainThread) [supervisor.api.middleware.security] /host/info access from Home Assistant
23-11-29 23:37:16 DEBUG (MainThread) [supervisor.api.middleware.security] /addons access from Home Assistant
23-11-29 23:37:18 DEBUG (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from Home Assistant
23-11-29 23:37:19 DEBUG (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from Home Assistant
23-11-29 23:37:21 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:37:27 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:37:36 DEBUG (MainThread) [supervisor.api.middleware.security] /supervisor/update access from Home Assistant
23-11-29 23:37:36 INFO (MainThread) [supervisor.supervisor] Fetching AppArmor profile https://version.home-assistant.io/apparmor_stable.txt
23-11-29 23:37:37 INFO (MainThread) [supervisor.host.apparmor] Adding/updating AppArmor profile: hassio-supervisor
23-11-29 23:37:37 DEBUG (MainThread) [supervisor.utils.dbus] D-Bus call - io.hass.os.AppArmor.call_load_profile on /io/hass/os/AppArmor
23-11-29 23:37:37 INFO (MainThread) [supervisor.supervisor] Update Supervisor to version 2023.11.6
23-11-29 23:37:37 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/aarch64-hassio-supervisor with tag 2023.11.6.
23-11-29 23:37:37 DEBUG (MainThread) [supervisor.homeassistant.websocket] Sending: {'type': <WSType.SUPERVISOR_EVENT: 'supervisor/event'>, 'data': {'event': <WSEvent.JOB: 'job'>, 'data': {'name': 'docker_interface_install', 'reference': 'hassio_supervisor', 'uuid': '1371218e22c948e4a7e77a71ba8fa6d3', 'progress': 0, 'stage': None, 'done': False, 'parent_id': None}}, 'id': 103}
23-11-29 23:37:37 DEBUG (MainThread) [supervisor.homeassistant.websocket] Received: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"id":103,"type":"result","success":true,"result":null}', extra='')
23-11-29 23:37:58 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['AccessPoints'] changed & [] invalidated
23-11-29 23:37:58 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['AccessPoints'] changed & [] invalidated
23-11-29 23:37:58 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['AccessPoints'] changed & [] invalidated
23-11-29 23:37:58 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['AccessPoints'] changed & [] invalidated
23-11-29 23:37:58 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['AccessPoints'] changed & [] invalidated
23-11-29 23:37:58 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['AccessPoints'] changed & [] invalidated
23-11-29 23:37:58 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['AccessPoints'] changed & [] invalidated
23-11-29 23:37:58 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['AccessPoints'] changed & [] invalidated
23-11-29 23:37:58 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['LastScan'] changed & [] invalidated
23-11-29 23:38:03 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:38:05 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6")
23-11-29 23:38:05 ERROR (MainThread) [supervisor.supervisor] Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6")
23-11-29 23:38:05 DEBUG (MainThread) [supervisor.homeassistant.websocket] Sending: {'type': <WSType.SUPERVISOR_EVENT: 'supervisor/event'>, 'data': {'event': <WSEvent.JOB: 'job'>, 'data': {'name': 'docker_interface_install', 'reference': 'hassio_supervisor', 'uuid': '1371218e22c948e4a7e77a71ba8fa6d3', 'progress': 0, 'stage': None, 'done': True, 'parent_id': None}}, 'id': 104}
23-11-29 23:38:05 DEBUG (MainThread) [supervisor.homeassistant.websocket] Received: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"id":104,"type":"result","success":true,"result":null}', extra='')
23-11-29 23:38:09 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:38:21 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:38:33 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:38:39 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:38:51 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:38:57 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:39:03 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:39:09 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:39:15 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:39:21 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:39:26 DEBUG (MainThread) [supervisor.api.middleware.security] /backups access from Home Assistant
23-11-29 23:39:26 DEBUG (MainThread) [supervisor.api.middleware.security] /host/info access from Home Assistant
23-11-29 23:39:27 DEBUG (MainThread) [supervisor.utils.dbus] Property change for org.freedesktop.NetworkManager-/org/freedesktop/NetworkManager/Devices/3: ['Bitrate'] changed & [] invalidated
23-11-29 23:39:27 DEBUG (MainThread) [supervisor.api.middleware.security] /addons access from Home Assistant
23-11-29 23:39:29 DEBUG (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from Home Assistant
23-11-29 23:39:29 DEBUG (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from Home Assistant

@agners
Copy link
Member

agners commented Nov 30, 2023

@jensjakobandersen thanks for the logs. It really seems that the fetching operation runs for quite some time, but in the end things just go south.

Can you setup host SSH access on port 22222 and run the following command from the HAOS shell?

docker pull ghcr.io/home-assistant/aarch64-hassio-supervisor:2023.11.6

@agners
Copy link
Member

agners commented Nov 30, 2023

So I was able to reproduce a problem with the same symptoms today after a power outage. I think these are the same problems underneath.

Unfortunately there is no easy solution as this seems to be caused by data corruption. Please see #4738 (comment) for more details.

@hubtub2
Copy link

hubtub2 commented Nov 30, 2023

I seem to have the same problem here getting the not found / no such image message.
Updating the supervisor to 2023.11.6, currently 2023.11.3 is installed.

I tried to ignore the update for now, but each time I have to reboot, e.g. after installing a hacs repository, the raspberry does not come up again. I have to physically remove power to make it boot again.

ha network info looks ok, host_internet and supervisor_internet is true

ha resolution info shows

  • supervisor issue: update failed
  • system issue: no current backup
  • unhealthy:supervisor

Running on raspberry 3 with sd card. Will try to reinstall from SSD ...

@xgolubev
Copy link

xgolubev commented Dec 1, 2023

Can you setup host SSH access on port 22222 and run the following command from the HAOS shell?

I've tried it in Terminal couple of times addon. Had to disable Protection mode
here's the result:
image

@GJSchroeder
Copy link

@agners I tried re-installing HA on my Yellow and now I can't even get it installed. It seems I'm getting similar/same errors as before when I tried updating. Below is the errors I see. So unfortunately now I have no HA that I can even get running at all. :( Are you sure nothing else changed with the 2023.11.x release?? It seems a bit too unusual that this many of us would mysteriously have corruption.

image

@jasperzz
Copy link

jasperzz commented Dec 1, 2023

Here the same issue as above with a fresh install.

Did a fresh install because of some issues with the supervisor.
Install is on Odroid N2.

image

@jensjakobandersen
Copy link
Author

@agners I tried re-installing HA on my Yellow and now I can't even get it installed. It seems I'm getting similar/same errors as before when I tried updating. Below is the errors I see. So unfortunately now I have no HA that I can even get running at all. :( Are you sure nothing else changed with the 2023.11.x release?? It seems a bit too unusual that this many of us would mysteriously have corruption.

image

Pretty strange. Can you install the 2023.10.x version ? Just for verifikation?

@hubtub2
Copy link

hubtub2 commented Dec 2, 2023

Current state: each time i reboot, the raspberry does not come up again, I guess because supervisor update failed. Then I have to switch power off/on and it boots again.
What is weired: some files I deleted previously from custom_components are suddenly restored.
Is the last backup restored automatically when supervisor update fails or something?
I could reproduce this twice ...

@Daz42
Copy link

Daz42 commented Dec 3, 2023

This has happened to me also, prior to this the Supervisor was reporting issues. Before this perfectly healthy and I haven't made any changes for quite some time now.

@agners
Copy link
Member

agners commented Dec 5, 2023

@xgolubev in your cases, it definitely is that Docker storage issue described in #4738 (comment).

Are you sure nothing else changed with the 2023.11.x release??

@GJSchroeder you mean in the Supervisor 2023.11.x release? Nothing really which stands out/would explain this issue. But I could miss something, I can't rule out that...

We do have sooo many installations these days, it is quite often the case that several people independently run into the same problem, even when it involves power cuts.

That you can observe it in a new installation is a bit strange. Did you remove power at any point when the LEDs were on?

(same question to @jasperzz)

@agners
Copy link
Member

agners commented Dec 5, 2023

@hubtub2 such type of behavior I've read from folks having a dying SD card. It seem that they become a bit flaky before they die entirely. Is that still on the SD card?

@jensjakobandersen
Copy link
Author

Interesting .- there might be a solution:
#4738 (comment)

@jasperzz
Copy link

jasperzz commented Dec 5, 2023

Solved my issue yesterday. Indeed a dying sd card.
Had eventually different error codes during installation

@jensjakobandersen
Copy link
Author

I solved it by using the script referenced in
#4738 (comment)

With one addition - I had to run the commandline "ha banner" to get home assistant started.

Here is the result of my ha info command (after waiting quite a while for supervisor to update itself at startup):
Harekærgårdsvej
571313174111707809

ha info

arch: aarch64
channel: stable
docker: 24.0.6
features:

  • reboot
  • shutdown
  • services
  • network
  • hostname
  • timedate
  • os_agent
  • haos
  • resolved
  • journal
  • disk
  • mount
    hassos: "11.0"
    homeassistant: 2023.10.3
    hostname: Plantagen
    logging: error
    machine: odroid-n2
    operating_system: Home Assistant OS 11.0
    state: running
    supervisor: 2023.11.6
    supported: true
    supported_arch:
  • aarch64
  • armv7
  • armhf
    timezone: Europe/Copenhagen

@hubtub2
Copy link

hubtub2 commented Dec 5, 2023

@hubtub2 such type of behavior I've read from folks having a dying SD card. It seem that they become a bit flaky before they die entirely. Is that still on the SD card?

Yes, still SD card. Parts are here to update to SSD. I hope you are right, that this is just a statistical effect because of the large installed user based. If this is the case we should have such a thread with every future release ;-)

@agners
Copy link
Member

agners commented Dec 6, 2023

If this is the case we should have such a thread with every future release ;-)

As mentioned in #4738 (comment), HAOS 11.2 comes with a new Docker version which includes a fix for that particular type of data corruption. So I am expecting that the amount of people seeing this issues slowly declines as people update to HAOS 11.2. Let's see.

@GJSchroeder
Copy link

@GJSchroeder you mean in the Supervisor 2023.11.x release? Nothing really which stands out/would explain this issue. But I could miss something, I can't rule out that...

We do have sooo many installations these days, it is quite often the case that several people independently run into the same problem, even when it involves power cuts.

That you can observe it in a new installation is a bit strange. Did you remove power at any point when the LEDs were on?

@agners When I first installed HA on the Yellow, I had a lot of difficulty and following the normal procedure never worked. I had to do the RPIBoot method and eventually got it to work and all my automations etc worked fine after restoring the backup from my previous Raspberry Pi. I had the same issues when trying to re-install 2023.11.5 and got those errors but even with RPIBoot method it still failed. I then tried again to install with the 2023.10.5, and it fails part way thru the install. It seems to get mostly done, and I can see it on the network but when I try to connect to it I get a message saying "Preparing Home Assistant. This may up to 20 minutes." I've left it for hours and it never completes. I'm honestly disappointed with the installation process for Yellow -- trying to figure out problems just by watching blinking LEDs is painful at best.

But the short answer is yes, it's very likely that during one of those attempts I removed power when the LEDs were on and maybe there's some corrupted files there that are causing the issue. Between each attempt, I restart the Yellow box with both the red and blue buttons pushed, which I thought was supposed to basically do a factory reset. But maybe it's not actually doing a clean wipe of everything (I'm using CM4 with I think 32GB storage, not the CM4 Lite). I'm away now but will try some more things when I return next week. If there's a different/better way to completely wipe clean the Yellow CM4 storage, I'm happy to try it. Worst case, I'll go back to my previous Raspberry Pi and see if I can get that updated to a newer build and if so, maybe will have to just ditch the Yellow (unfortunately).

@agners
Copy link
Member

agners commented Dec 6, 2023

@GJSchroeder the rpi-boot method is definitely not the regular expierence people should go through. It is hard to tell why the regular procedure did not work out for you. I am sorry for the hassle 😢

Not sure when did those installations, but in October 25th there was an update to the installer (see https://github.com/NabuCasa/buildroot-installer/releases/tag/yellow-installer-20231025). At least on my end, that version of the installer worked flawlessly every time.

If you have been able to update to HAOS 11.2, then the device wipe mechanism should be good enough (as it wipes the complete Docker storage).

@hubtub2
Copy link

hubtub2 commented Dec 10, 2023

@hubtub2 such type of behavior I've read from folks having a dying SD card. It seem that they become a bit flaky before they die entirely. Is that still on the SD card?

Update:
I tried to reinstall on my Raspberry PI 3B .

  1. Using the SD Card + SSD Method. I took a different 4GB SD card and installed Homeassistant for PI 3 from the "Raspberry PI Imager". Interestingly, this failed with a similar error:
Screenshot 2023-12-09 at 23 31 58
  1. Using the SSD/USB only method I finally succeeded (first I set the boot usb flag using a normal linux image as needed for 3B). I first thought it did not work, since there was no output on HDMI console and the red PI LED was blinking - but with some patients it was visible in my network. Then I could restore the full HA backup, that I downloaded from the old system during setup of HA. Every came up as expected.

So the case is closed to me, but since it happened on a fresh install on a different SD card, it does not look like a coincidence to me. Hope my information helps. Maybe it is the old PI 3B? Or some network IPv4/v6 problem?

@GJSchroeder
Copy link

@agners I spent several hours yesterday trying many, many times to get a clean install and it is still failing. I tried installing the latest HA OS 11.2 and still get the same error during install as noted earlier (and included below). I also tried using an older version 10.5 from August, and it fails in a different way in that it gets part way installed, I can see the homeassistant on the network, but when I try to connect it shows a message saying it's installing and it will take up to 20 minutes. I've left it that way overnight and it never progressed. So at this point, it seems I basically have a bricked HA Yellow that I can't even do a fresh install on.

What I've tried doing typically is the following:

  1. Run RPIBoot. It seems to complete ok, although there is a message in the output that says "Cannot open file fixup4.dat". I did a little looking into that and seems that it can be ignored?
  2. Use Raspberry Pi Imager to image HA Yellow onto the device. I've tried just selecting the HA Yellow from the choices and assume it picks the latest, as well as downloading the latest image from github and pointing the imager to use that image, as well as downloading older versions and trying those.
  3. Restart Yellow

I'm using a CM4 with eMMC. I do also have an NVMe SSD but have not been using it in order to eliminate any other extra issues.

Here is the log from the failed attempt at clean install of 11.2:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/udev.sh
[15:32:47] INFO: Using udev information from host
cont-init: info: /etc/cont-init.d/udev.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun supervisor (no readiness notification)
services-up: info: copying legacy longrun watchdog (no readiness notification)
[15:32:48] INFO: Starting local supervisor watchdog...
s6-rc: info: service legacy-services successfully started
23-12-12 15:32:51 INFO (MainThread) [__main__] Initializing Supervisor setup
23-12-12 15:32:51 INFO (MainThread) [supervisor.docker.network] Can't find Supervisor network, creating a new network
23-12-12 15:32:52 INFO (MainThread) [supervisor.bootstrap] Seting up coresys for machine: yellow
23-12-12 15:32:52 INFO (MainThread) [supervisor.docker.supervisor] Attaching to Supervisor ghcr.io/home-assistant/aarch64-hassio-supervisor with version 2023.11.6
23-12-12 15:32:52 INFO (MainThread) [supervisor.docker.supervisor] Connecting Supervisor to hassio-network
23-12-12 15:32:52 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state initialize
23-12-12 15:32:52 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
23-12-12 15:32:52 INFO (MainThread) [__main__] Setting up Supervisor
23-12-12 15:32:52 INFO (MainThread) [supervisor.api] Starting API on 172.30.32.2
23-12-12 15:32:52 INFO (MainThread) [supervisor.hardware.monitor] Started Supervisor hardware monitor
23-12-12 15:32:52 INFO (MainThread) [supervisor.dbus.manager] Connected to system D-Bus.
23-12-12 15:32:52 INFO (MainThread) [supervisor.dbus.agent] Load dbus interface io.hass.os
23-12-12 15:32:52 INFO (MainThread) [supervisor.dbus.hostname] Load dbus interface org.freedesktop.hostname1
23-12-12 15:32:52 INFO (MainThread) [supervisor.dbus.logind] Load dbus interface org.freedesktop.login1
23-12-12 15:32:52 INFO (MainThread) [supervisor.dbus.network] Load dbus interface org.freedesktop.NetworkManager
23-12-12 15:32:52 INFO (MainThread) [supervisor.dbus.rauc] Load dbus interface de.pengutronix.rauc
23-12-12 15:32:52 INFO (MainThread) [supervisor.dbus.resolved] Load dbus interface org.freedesktop.resolve1
23-12-12 15:32:52 INFO (MainThread) [supervisor.dbus.systemd] Load dbus interface org.freedesktop.systemd1
23-12-12 15:32:52 INFO (MainThread) [supervisor.dbus.timedate] Load dbus interface org.freedesktop.timedate1
23-12-12 15:32:53 INFO (MainThread) [supervisor.host.services] Updating service information
23-12-12 15:32:53 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information
23-12-12 15:32:53 WARNING (SyncWorker_1) [supervisor.host.sound] Can't update PulseAudio data: Failed to connect to pulseaudio server
23-12-12 15:32:53 INFO (MainThread) [supervisor.host.network] Updating local network information
23-12-12 15:32:53 INFO (MainThread) [supervisor.host.apparmor] Loading AppArmor Profiles: {'hassio-supervisor'}
23-12-12 22:32:53 INFO (MainThread) [supervisor.docker.monitor] Started docker events monitor
23-12-12 22:32:53 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
23-12-12 22:32:53 INFO (MainThread) [supervisor.docker.interface] Found ghcr.io/home-assistant/aarch64-hassio-cli versions: [<AwesomeVersion CalVer '2023.10.0'>]
23-12-12 22:32:53 INFO (MainThread) [supervisor.docker.interface] Attaching to ghcr.io/home-assistant/aarch64-hassio-cli with version 2023.10.0
23-12-12 22:32:53 INFO (MainThread) [supervisor.plugins.cli] Starting CLI plugin
23-12-12 22:32:54 INFO (MainThread) [supervisor.docker.cli] Starting CLI ghcr.io/home-assistant/aarch64-hassio-cli with version 2023.10.0 - 172.30.32.5
23-12-12 22:32:54 INFO (MainThread) [supervisor.docker.interface] Found ghcr.io/home-assistant/aarch64-hassio-dns versions: [<AwesomeVersion CalVer '2023.06.2'>]
23-12-12 22:32:54 INFO (MainThread) [supervisor.docker.interface] Attaching to ghcr.io/home-assistant/aarch64-hassio-dns with version 2023.06.2
23-12-12 22:32:54 INFO (MainThread) [supervisor.plugins.dns] Starting CoreDNS plugin
23-12-12 22:32:55 INFO (MainThread) [supervisor.docker.dns] Starting DNS ghcr.io/home-assistant/aarch64-hassio-dns with version 2023.06.2 - 172.30.32.3
23-12-12 22:32:55 INFO (MainThread) [supervisor.plugins.dns] Updated /etc/resolv.conf
23-12-12 22:32:55 INFO (MainThread) [supervisor.docker.interface] Found ghcr.io/home-assistant/aarch64-hassio-audio versions: [<AwesomeVersion CalVer '2023.10.0'>]
23-12-12 22:32:55 INFO (MainThread) [supervisor.docker.interface] Attaching to ghcr.io/home-assistant/aarch64-hassio-audio with version 2023.10.0
23-12-12 22:32:55 INFO (MainThread) [supervisor.plugins.audio] Starting Audio plugin
23-12-12 22:32:56 INFO (MainThread) [supervisor.docker.audio] Starting Audio ghcr.io/home-assistant/aarch64-hassio-audio with version 2023.10.0 - 172.30.32.4
23-12-12 22:32:56 INFO (MainThread) [supervisor.docker.interface] Found ghcr.io/home-assistant/aarch64-hassio-observer versions: [<AwesomeVersion CalVer '2023.06.0'>]
23-12-12 22:32:56 INFO (MainThread) [supervisor.docker.interface] Attaching to ghcr.io/home-assistant/aarch64-hassio-observer with version 2023.06.0
23-12-12 22:32:56 INFO (MainThread) [supervisor.plugins.observer] Starting observer plugin
23-12-12 22:32:57 INFO (MainThread) [supervisor.docker.observer] Starting Observer ghcr.io/home-assistant/aarch64-hassio-observer with version 2023.06.0 - 172.30.32.6
23-12-12 22:32:57 INFO (MainThread) [supervisor.docker.interface] Found ghcr.io/home-assistant/aarch64-hassio-multicast versions: [<AwesomeVersion CalVer '2023.06.2'>]
23-12-12 22:32:57 INFO (MainThread) [supervisor.docker.interface] Attaching to ghcr.io/home-assistant/aarch64-hassio-multicast with version 2023.06.2
23-12-12 22:32:57 INFO (MainThread) [supervisor.plugins.multicast] Starting Multicast plugin
23-12-12 22:32:58 INFO (MainThread) [supervisor.docker.multicast] Starting Multicast ghcr.io/home-assistant/aarch64-hassio-multicast with version 2023.06.2 - Host
23-12-12 22:32:58 INFO (MainThread) [supervisor.plugins.manager] cli does not have the latest version 2023.11.0, updating
23-12-12 22:32:58 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/aarch64-hassio-cli:2023.10.0 to ghcr.io/home-assistant/aarch64-hassio-cli:2023.11.0
23-12-12 22:32:58 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/aarch64-hassio-cli with tag 2023.11.0.
23-12-12 22:33:53 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/aarch64-hassio-cli:2023.11.0: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/aarch64-hassio-cli:2023.11.0/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-cli:2023.11.0")
23-12-12 22:33:53 ERROR (MainThread) [supervisor.plugins.cli] CLI update failed
23-12-12 22:33:53 ERROR (MainThread) [supervisor.plugins.manager] Can't update cli to 2023.11.0, the Supervisor healthy could be compromised!
23-12-12 22:33:53 INFO (MainThread) [supervisor.resolution.module] Create new suggestion execute_update - plugin / cli
23-12-12 22:33:53 INFO (MainThread) [supervisor.resolution.module] Create new issue update_failed - plugin / cli
23-12-12 22:33:53 INFO (MainThread) [supervisor.homeassistant.secrets] Loaded 0 Home Assistant secrets
23-12-12 22:33:53 INFO (MainThread) [supervisor.docker.interface] No version found for ghcr.io/home-assistant/yellow-homeassistant
23-12-12 22:33:53 INFO (MainThread) [supervisor.homeassistant.core] No Home Assistant Docker image ghcr.io/home-assistant/yellow-homeassistant found.
23-12-12 22:33:53 INFO (MainThread) [supervisor.docker.interface] Attaching to ghcr.io/home-assistant/yellow-homeassistant with version landingpage
23-12-12 22:33:53 INFO (MainThread) [supervisor.homeassistant.core] Using preinstalled landingpage
23-12-12 22:33:53 INFO (MainThread) [supervisor.homeassistant.core] Starting HomeAssistant landingpage
23-12-12 22:33:53 INFO (MainThread) [supervisor.homeassistant.module] Update pulse/client.config: /data/tmp/homeassistant_pulse
23-12-12 22:33:54 INFO (MainThread) [supervisor.docker.homeassistant] Starting Home Assistant ghcr.io/home-assistant/yellow-homeassistant with version landingpage
23-12-12 22:33:54 INFO (MainThread) [supervisor.os.manager] Detect Home Assistant Operating System 11.2 / BootSlot A
23-12-12 22:33:54 INFO (MainThread) [supervisor.store.git] Cloning add-on https://github.com/home-assistant/addons repository
23-12-12 22:33:54 INFO (MainThread) [supervisor.store.git] Cloning add-on https://github.com/esphome/home-assistant-addon repository
23-12-12 22:33:54 INFO (MainThread) [supervisor.store.git] Cloning add-on https://github.com/hassio-addons/repository repository
23-12-12 22:33:55 ERROR (MainThread) [supervisor.store.git] Can't clone https://github.com/hassio-addons/repository repository: Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v --recursive --depth=1 --shallow-submodules -- https://github.com/hassio-addons/repository /data/addons/git/a0d7b954
  stderr: 'Cloning into '/data/addons/git/a0d7b954'...
POST git-upload-pack (175 bytes)
POST git-upload-pack (244 bytes)
error: RPC failed; curl 56 OpenSSL SSL_read: OpenSSL/3.1.4: error:0A000119:SSL routines::decryption failed or bad record mac, errno 0
error: 1187 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
'.
23-12-12 22:33:55 ERROR (MainThread) [supervisor.store] Can't retrieve data from https://github.com/hassio-addons/repository due to 
23-12-12 22:33:55 INFO (MainThread) [supervisor.resolution.module] Create new suggestion execute_remove - store / a0d7b954
23-12-12 22:33:55 INFO (MainThread) [supervisor.resolution.module] Create new issue fatal_error - store / a0d7b954
23-12-12 22:33:56 INFO (MainThread) [supervisor.store] Loading add-ons from store: 31 all - 31 new - 0 remove
23-12-12 22:33:56 INFO (MainThread) [supervisor.addons] Found 0 installed add-ons
23-12-12 22:33:56 INFO (MainThread) [supervisor.backups.manager] Found 0 backup files
23-12-12 22:33:56 INFO (MainThread) [supervisor.discovery] Loaded 0 messages
23-12-12 22:33:56 INFO (MainThread) [supervisor.ingress] Loaded 0 ingress sessions
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state setup
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.check] System checks complete
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state setup
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
23-12-12 22:33:56 INFO (MainThread) [supervisor.jobs] 'ResolutionFixup.run_autofix' blocked from execution, system is not running - setup
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state setup
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
23-12-12 22:33:56 INFO (MainThread) [__main__] Running Supervisor
23-12-12 22:33:56 INFO (MainThread) [supervisor.os.manager] Rauc: A - marked slot kernel.0 as good
23-12-12 22:33:56 INFO (MainThread) [supervisor.addons] Phase 'initialize' starting 0 add-ons
23-12-12 22:33:56 INFO (MainThread) [supervisor.addons] Phase 'system' starting 0 add-ons
23-12-12 22:33:56 INFO (MainThread) [supervisor.addons] Phase 'services' starting 0 add-ons
23-12-12 22:33:56 INFO (MainThread) [supervisor.core] Skipping start of Home Assistant
23-12-12 22:33:56 INFO (MainThread) [supervisor.addons] Phase 'application' starting 0 add-ons
23-12-12 22:33:56 INFO (MainThread) [supervisor.misc.tasks] All core tasks are scheduled
23-12-12 22:33:56 INFO (MainThread) [supervisor.core] Supervisor is up and running
23-12-12 22:33:56 INFO (MainThread) [supervisor.homeassistant.core] Home Assistant setup
23-12-12 22:33:56 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/yellow-homeassistant:landingpage to ghcr.io/home-assistant/yellow-homeassistant:2023.12.1
23-12-12 22:33:56 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/yellow-homeassistant with tag 2023.12.1.
23-12-12 22:33:56 INFO (MainThread) [supervisor.host.info] Updating local host information
23-12-12 22:33:56 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state running
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.checks.base] Run check for free_space/system
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.checks.base] Run check for multiple_data_disks/system
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.checks.base] Run check for ipv4_connection_problem/system
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.checks.base] Run check for trust/supervisor
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_ipv6_error/dns_server
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.checks.base] Run check for pwned/addon
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.checks.base] Run check for docker_config/system
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.checks.base] Run check for no_current_backup/system
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.module] Create new suggestion create_full_backup - system / None
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.module] Create new issue no_current_backup - system / None
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_failed/dns_server
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.checks.base] Run check for security/core
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.check] System checks complete
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state running
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
23-12-12 22:33:56 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
23-12-12 22:33:57 INFO (MainThread) [supervisor.host.services] Updating service information
23-12-12 22:33:57 INFO (MainThread) [supervisor.host.network] Updating local network information
23-12-12 22:33:57 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information
23-12-12 22:33:57 WARNING (SyncWorker_0) [supervisor.host.sound] Can't update PulseAudio data: Failed to connect to pulseaudio server
23-12-12 22:33:57 INFO (MainThread) [supervisor.host.manager] Host information reload completed
23-12-12 22:36:49 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/yellow-homeassistant:2023.12.1: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/yellow-homeassistant:2023.12.1/json: Not Found ("No such image: ghcr.io/home-assistant/yellow-homeassistant:2023.12.1")
23-12-12 22:36:49 WARNING (MainThread) [supervisor.homeassistant.core] Error on Home Assistant installation. Retry in 30sec
23-12-12 22:37:19 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/yellow-homeassistant:landingpage to ghcr.io/home-assistant/yellow-homeassistant:2023.12.1
23-12-12 22:37:19 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/yellow-homeassistant with tag 2023.12.1.
23-12-12 22:39:49 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/yellow-homeassistant:2023.12.1: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/yellow-homeassistant:2023.12.1/json: Not Found ("No such image: ghcr.io/home-assistant/yellow-homeassistant:2023.12.1")
23-12-12 22:39:49 WARNING (MainThread) [supervisor.homeassistant.core] Error on Home Assistant installation. Retry in 30sec
23-12-12 22:40:19 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/yellow-homeassistant:landingpage to ghcr.io/home-assistant/yellow-homeassistant:2023.12.1
23-12-12 22:40:19 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/yellow-homeassistant with tag 2023.12.1.
23-12-12 22:43:06 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/yellow-homeassistant:2023.12.1: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/yellow-homeassistant:2023.12.1/json: Not Found ("No such image: ghcr.io/home-assistant/yellow-homeassistant:2023.12.1")
23-12-12 22:43:06 WARNING (MainThread) [supervisor.homeassistant.core] Error on Home Assistant installation. Retry in 30sec
23-12-12 22:43:36 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/yellow-homeassistant:landingpage to ghcr.io/home-assistant/yellow-homeassistant:2023.12.1
23-12-12 22:43:36 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/yellow-homeassistant with tag 2023.12.1.

@jensjakobandersen
Copy link
Author

jensjakobandersen commented Dec 13, 2023 via email

@agners
Copy link
Member

agners commented Dec 13, 2023

@GJSchroeder it seems like any container update fails (also the CLI update) 🤔

Can you connect via USB-C to see the serial console (see also https://yellow.home-assistant.io/guides/use-serial-console-windows/ or https://yellow.home-assistant.io/guides/use-serial-console-linux-macos/)?

What I would be interested is what happens when you manually pull using docker pull ghcr.io/home-assistant/yellow-homeassistant:2023.12.1. Also, can you check free disk space using df -h?

Do you have the NVMe SSD installed at this point?

@GJSchroeder
Copy link

GJSchroeder commented Dec 13, 2023

@agners I tried USB connection as described in your link, installed Putty, connected my Windows laptop to Yellow (with jumpers set appropriately), but in Device Manager I never see the "Ports" entry. So I couldn't get past that step. The guide says USB-C on Yellow to USB-A on computer, but my laptop only has USB-C. I can't imagine that would make a difference though.

UPDATE: I also tried to install on the SSD but could not get that to work. I tried the method of imaging the eMMC and during startup pushing the blue button which should cause it to install on the SSD, but nothing really happened. Also tried imaging on the SSD (which it was in a separate case outside of the Yellow box) but can't get Yellow to boot from the NVMe SSD. Finally got so frustrated that I gave up and went back to my Raspberry Pi.

Here's where it gets really weird. I restarted my original Pi, which was running 2023.10.5. When I first started it, I got the list of updates, so the first one I tried to apply was HA Supervisor 11.2. It failed in the same way that it was failing when I had the Yellow running a few weeks ago! I restarted it and tried a few more times and it continued to fail.

Finally I realized that before I originally tried to set up the Yellow several weeks ago, I replaced my network switch with one that supported POE because that was the Yellow kit that I had gotten. I had also noticed reading quite a few other posts on the various forums of other people having very similar problems with Yellow as I did, and it seemed like most of them were using the POE version. So I replaced the POE switch with my old non-POE one switch, restarted HA, and Eureka! All of the Supervisor, Core and OS updates applied fine.

So my theory is that the POE Yellow, or the POE switch, has some weird network issue that causes the installation/updates of HA to fail. To test this, I will go buy a regular power adaptor and try powering the Yellow with that and use a non-POE ethernet cable and my non-POE switch. It sounds like a long shot, but that's my last try. If that's not the issue, then I give up on Yellow and will stick with my Raspberry Pi, which is running fine now.

@GJSchroeder
Copy link

@agners A follow-up to my previous post. I tried using a regular power cable and non-POE switch on the Yellow. I did a reset (turn it on while pressing red button), then restarted with an image on USB SD card. Shockingly, it installed perfectly. That's the first time it's worked that way in dozens and dozens of attempts that I've done over the past 6 weeks when I was using POE ethernet with POE switch. Just to make sure it wasn't a random success, I reset it and then did the RPIBoot, write image to the Yellow and restart to install. Again it worked perfectly. After that I was able to restore a previous backup and all my devices, automations etc are working fine.

I then saw that I was running 2023.12.1 and that 12.2 was available. I attempted the update and it succeeded. That was the first time I was able to apply an update since my first install of the Yellow 6 weeks ago.

I then re-connected the Yellow to the POE cable and the POE switch and it's all still running fine. I guess the next test will be whether I can update to the next HA version using that POE switch.

I'm not a network guru, but it does seem like there's some problem with either the Yellow POE, or the POE switch that is causing certain activities like installation and updates to fail. Very, very strange. For reference, my switch is TP-Link 5-port 10/100Mbps model TL-SF1005LP.

But for now, I finally have the Yellow up and running again so I'm happy. Thank you for the help.

@github-actions github-actions bot locked and limited conversation to collaborators Jan 14, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests