proxmox 가 원격 접속시 간헐적으로 네트워크가 hang 걸릴때
Proxmox 서버나 해당 시스템에 생성된 VM으로 ssh 접속이든 뭐든 네트워크가 간혹 끊기는 느낌이나 현상이 일어날때가 있다.
평소엔 느끼지 못했는데 요즘 계속 접속해서 작업을 하다보니 순간 순간 ssh 터미널 화면에서 키가 입력되지 않는다거나 로그 보던게 멈춘다거나 하는 현상으로 불편했다.
네트워크 때문인가 싶어서 장소를 변경해보고 스마트폰 태더링도 이것 저것을 바꿔 가면서 해밨지만 여전히 끊기는 현상이 있었다.
로그를 살펴밨다.
Jan 10 10:26:37 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: NETDEV WATCHDOG: CPU: 2: transmit queue 0 timed out 6610 ms
Jan 10 10:26:37 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Reset adapter unexpectedly
Jan 10 10:26:37 ilikeafrica kernel: vmbr0: port 1(eno1) entered disabled state
Jan 10 10:26:40 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 10 10:26:40 ilikeafrica kernel: vmbr0: port 1(eno1) entered blocking state
Jan 10 10:26:40 ilikeafrica kernel: vmbr0: port 1(eno1) entered forwarding state
Jan 10 10:28:40 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
Jan 10 10:28:42 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
Jan 10 10:28:44 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
Jan 10 10:28:46 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
Jan 10 10:28:47 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: NETDEV WATCHDOG: CPU: 8: transmit queue 0 timed out 7118 ms
Jan 10 10:28:47 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Reset adapter unexpectedly
Jan 10 10:28:47 ilikeafrica kernel: vmbr0: port 1(eno1) entered disabled state
Jan 10 10:28:51 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 10 10:28:51 ilikeafrica kernel: vmbr0: port 1(eno1) entered blocking state
Jan 10 10:28:51 ilikeafrica kernel: vmbr0: port 1(eno1) entered forwarding state
Jan 10 10:29:23 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
Jan 10 10:29:25 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
Jan 10 10:29:27 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
Jan 10 10:29:29 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
Jan 10 10:29:30 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: NETDEV WATCHDOG: CPU: 5: transmit queue 0 timed out 9896 ms
Jan 10 10:29:30 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Reset adapter unexpectedly
Jan 10 10:29:30 ilikeafrica kernel: vmbr0: port 1(eno1) entered disabled state
Jan 10 10:29:34 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 10 10:29:34 ilikeafrica kernel: vmbr0: port 1(eno1) entered blocking state
Jan 10 10:29:34 ilikeafrica kernel: vmbr0: port 1(eno1) entered forwarding state
Jan 10 10:33:38 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
Jan 10 10:33:40 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
Jan 10 10:33:42 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Detected Hardware Unit Hang:
Jan 10 10:33:43 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: NETDEV WATCHDOG: CPU: 1: transmit queue 0 timed out 7256 ms
Jan 10 10:33:43 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: Reset adapter unexpectedly
Jan 10 10:33:43 ilikeafrica kernel: vmbr0: port 1(eno1) entered disabled state
Jan 10 10:33:46 ilikeafrica kernel: e1000e 0000:00:19.0 eno1: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jan 10 10:33:46 ilikeafrica kernel: vmbr0: port 1(eno1) entered blocking state
Jan 10 10:33:46 ilikeafrica kernel: vmbr0: port 1(eno1) entered forwarding state
root@ilikeafrica:~# 언젠가 부터 위와 같은 로그가 계속 발생하고 있었다.
검색해보고 chatgpt도 이용을 해보니 커널업데이트나 드라이버 업데이트등을 해보라고 하는데 모두 해소가 되지 않았다.
그러다 아래와 같은 방법으로 해결하게 됬었다.
vi /usr/lib/systemd/system/fix-e1000e.service
[Unit]
Description=Fix for ethernet hang errors
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/sbin/ethtool -K eno1 tso off gso off
[Install]
WantedBy=multi-user.target위와 같이 서비스 하나를 만들고
sudo systemctl daemon-reload
sudo systemctl enable fix-e1000e
sudo systemctl start fix-e1000e
이렇게 하면 문제는 해결된다.
이유는 아래와 같다.
문제 상황:
- Proxmox VE 환경에서 Intel 네트워크 어댑터 사용 시, 네트워크가 간헐적으로 중단되며 “Detected Hardware Unit Hang” 오류 메시지가 발생하는 문제가 보고되었습니다.
제시된 해결책:
- 세그멘테이션 오프로딩(Segmentation Offloading) 기능을 비활성화하여 문제를 완화하는 방법이 제안되었습니다. 이를 위해
ethtool을 사용하여 해당 기능을 끄는 systemd 서비스를 생성하는 방법이 소개되었습니다.
