wagnerjer

User Tools

Site Tools

Trace: • logs1
wiki:logs1

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
wiki:logs1 [2026/09/04 08:30] – wagner.jerwiki:logs1 [2026/09/09 11:55] (current) – wagner.jer
Line 1: Line 1:
-<code>root@localhost [ ~ ]# systemctl status applmgmt.service --no-pager +root@LX-VV-VC-001 [ /var/log/vmware/applmgmt ]# tail -f /var/log/vmware/applmgmt/applmgmt.log 
-● applmgmt.service +2026-09-09T11:51:44 AM UTC [7734]DEBUG:vmware.vherd.transport.xmlrpc:Looking for function get 
-     Loaded: masked (Reason: Unit applmgmt.service is masked.) +2026-09-09T11:51:44 AM UTC [7734]DEBUG:vmware.vherd.transport.xmlrpc:Method Signature = [['struct', 'string'], '[{"requiredRole": ["requiredRole", false, "str", false]}, {"requiredRole": ["str", "User role."]}, {}, ["requiredRole"], {"suppressCLI": true, "cliAlias": ""}]', 'This interface returns True if the logged in user has\n            the correct required role.', '[{"authorized": ["authorized", false, "bool", false]}, {"authorized": ["bool", "True if the user has the required role"]}, {}, ["authorized"], {"suppressCLI": true, "cliAlias": ""}]'] 
-     Active: failed (Result: timeout) since Fri 2026-09-04 08:17:18 UTC; 11min ago +2026-09-09T11:52:50 AM UTC [7734]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.recovery.restore, operation_id: validate 
-   Main PID: 9970 (code=killed, signal=KILL)+2026-09-09T11:52:50 AM UTC [7734]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ModifyConfiguration'] 
 +2026-09-09T11:52:50 AM UTC [7734]DEBUG:root:Validated user privileges in localstore or SSO 
 +2026-09-09T11:52:50 AM UTC [7734]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf 
 +2026-09-09T11:52:50 AM UTC [7734]INFO:plugins.SMBClientStorageIOLib:Running Command: ['/usr/bin/smbclient', '//10.231.192.55/ICT-SOFTWARE/', '-A', '/dev/shm/smbCredentialFilee715ltky', '-t', '140', '-c', 'cd VMWARE/VcenterBackup/vCenter/; ls'] 
 +2026-09-09T11:52:50 AM UTC [7734]ERROR:plugins.SMBClientStorageIOLib:Command failed with error: NT_STATUS_ACCOUNT_RESTRICTION 
 +2026-09-09T11:52:50 AM UTC [7734]ERROR:plugins.SMBClientStorageIOLib:smb cmd failed. RC: 1, Err: , Cmd: ['/usr/bin/smbclient', '//10.231.192.55/ICT-SOFTWARE/', '-A', '/dev/shm/smbCredentialFilee715ltky', '-t', '140', '-c', 'cd VMWARE/VcenterBackup/vCenter/; ls'] 
 +2026-09-09T11:52:50 AM UTC [7734]ERROR:vmware.appliance.vapi.impl:pint.Error('com.vmware.applmgmt.backup.plugin.internal_server_err', 'General system error reported by backup server.', **{'args': ['Plugin error occurred. ErrCode: 5, Args: ()']}) 
 +2026-09-09T11:54:14 AM UTC [7734]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.recovery.restore, operation_id: validate 
 +2026-09-09T11:54:14 AM UTC [7734]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ModifyConfiguration'] 
 +2026-09-09T11:54:14 AM UTC [7734]DEBUG:root:Validated user privileges in localstore or SSO 
 +2026-09-09T11:54:14 AM UTC [7734]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf 
 +2026-09-09T11:54:14 AM UTC [7734]INFO:plugins.SMBClientStorageIOLib:Running Command: ['/usr/bin/smbclient', '//lx-vv-fs-002/ICT-SOFTWARE/', '-A', '/dev/shm/smbCredentialFilez9k4dnoe', '-t', '140', '-c', 'cd VMWARE/VcenterBackup/vCenter/; ls'] 
 +2026-09-09T11:54:14 AM UTC [7734]ERROR:plugins.SMBClientStorageIOLib:Command failed with error: NT_STATUS_ACCOUNT_RESTRICTION 
 +2026-09-09T11:54:14 AM UTC [7734]ERROR:plugins.SMBClientStorageIOLib:smb cmd failed. RC: 1, Err: , Cmd: ['/usr/bin/smbclient', '//lx-vv-fs-002/ICT-SOFTWARE/', '-A', '/dev/shm/smbCredentialFilez9k4dnoe', '-t', '140', '-c', 'cd VMWARE/VcenterBackup/vCenter/; ls'] 
 +2026-09-09T11:54:14 AM UTC [7734]ERROR:vmware.appliance.vapi.impl:pint.Error('com.vmware.applmgmt.backup.plugin.internal_server_err', 'General system error reported by backup server.', **{'args': ['Plugin error occurred. ErrCode: 5, Args: ()']})
  
-Sep 04 08:17:18 localhost systemd[1]: applmgmt.service: State 'stop-sigterm' timed out. Killing. 
-Sep 04 08:17:18 localhost systemd[1]: applmgmt.service: Killing process 9970 (python) with signal SIGKILL. 
-Sep 04 08:17:18 localhost systemd[1]: applmgmt.service: Main process exited, code=killed, status=9/KILL 
-Sep 04 08:17:18 localhost systemd[1]: applmgmt.service: Failed with result 'timeout'. 
-Sep 04 08:17:18 localhost systemd[1]: applmgmt.service: Unit process 18301 (invoke_precheck) remains running after unit stopped. 
-Sep 04 08:17:18 localhost systemd[1]: applmgmt.service: Unit process 18345 (bash) remains running after unit stopped. 
-Sep 04 08:17:18 localhost systemd[1]: applmgmt.service: Unit process 18413 (python3) remains running after unit stopped. 
-Sep 04 08:17:18 localhost systemd[1]: applmgmt.service: Unit process 18414 (tee) remains running after unit stopped. 
-Sep 04 08:17:18 localhost systemd[1]: Stopped Appliance Management Service.. 
-Sep 04 08:17:18 localhost systemd[1]: applmgmt.service: Consumed 45.096s CPU time. 
-root@localhost [ ~ ]# tail -100 /var/log/vmware/vmon/vmon.log 
-2026-09-04T08:21:57.204Z In(05) host-20388 <vsm> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:21:57.204Z In(05) host-20388 <perfcharts> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:21:57.204Z In(05) host-20388 <hvc> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:21:57.204Z In(05) host-20388 <content-library> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:21:57.204Z In(05) host-20388 <rbd> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:21:57.204Z In(05) host-20388 Successfully executed service batch operation API_HEALTH. 
-2026-09-04T08:25:51.975Z In(05) host-20388 Client info Uid=0,Gid=0,Pid=21376,Comm=(vmon-cli),PPid=21364,Comm=(service-control),PPid=18238,Comm=(bash),PPid=13635,Comm=(vherdrunner),PPid=13611,Comm=(sshd),PPid=13505 
-2026-09-04T08:25:51.976Z In(05) host-20388 Client info Uid=0,Gid=0,Pid=21377,Comm=(vmon-cli),PPid=21364,Comm=(service-control),PPid=18238,Comm=(bash),PPid=13635,Comm=(vherdrunner),PPid=13611,Comm=(sshd),PPid=13505 
-2026-09-04T08:25:51.977Z In(05) host-20388 Client info Uid=0,Gid=0,Pid=21378,Comm=(vmon-cli),PPid=21364,Comm=(service-control),PPid=18238,Comm=(bash),PPid=13635,Comm=(vherdrunner),PPid=13611,Comm=(sshd),PPid=13505 
-2026-09-04T08:25:51.977Z In(05) host-20388 Received start request for applmgmt 
-2026-09-04T08:25:51.977Z In(05) host-20388 <applmgmt-prestart> Constructed command: /usr/lib/applmgmt/support/scripts/prestart-applmgmt.sh 
-2026-09-04T08:25:51.987Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: Removed /etc/systemd/system/applmgmt.service. 
-2026-09-04T08:25:51.987Z Wa(03)+ host-20388 
-2026-09-04T08:25:52.188Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: Created symlink /etc/systemd/system/applmgmt.service → /dev/null. 
-2026-09-04T08:25:52.188Z Wa(03)+ host-20388 
-2026-09-04T08:25:52.430Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: Error: Failed to open the store. 
-2026-09-04T08:25:52.430Z Wa(03)+ host-20388 vecs-cli failed. Error 1021: Cannot connect to vmafd service. 
-2026-09-04T08:25:52.430Z Wa(03)+ host-20388 
-2026-09-04T08:25:52.546Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: 2026-09-04 08:25:52,546 - 21472 - INFO - vcsa_network_base - 80 - StreamHandler added 
-2026-09-04T08:25:52.546Z Wa(03)+ host-20388 
-2026-09-04T08:25:52.546Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: 2026-09-04 08:25:52,546 - 21472 - INFO - vcsa_network_base - 66 - NetworkUtil; Doesn't exist creating one. 
-2026-09-04T08:25:52.546Z Wa(03)+ host-20388 
-2026-09-04T08:25:52.980Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: WARNING:root:Service account vsr file /etc/vmware/service-registry/vsr/service-accounts/vmware-applmgmtservice.yaml not present, we will go ahead and use the method parameters sent by the caller. 
-2026-09-04T08:25:52.980Z Wa(03)+ host-20388 
-2026-09-04T08:25:52.981Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: ERROR:root:Not found service account hash file: /var/cache/svcaccounts/vmware-applmgmtservice/vmware-applmgmtservice.hash 
-2026-09-04T08:25:52.981Z Wa(03)+ host-20388 
-2026-09-04T08:25:52.981Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: Traceback (most recent call last): 
-2026-09-04T08:25:52.981Z Wa(03)+ host-20388   File "/usr/lib/applmgmt/support/scripts/create_svc_account.py", line 80, in <module> 
-2026-09-04T08:25:52.981Z Wa(03)+ host-20388 
-2026-09-04T08:25:52.981Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr:     main() 
-2026-09-04T08:25:52.981Z Wa(03)+ host-20388   File "/usr/lib/applmgmt/support/scripts/create_svc_account.py", line 75, in main 
-2026-09-04T08:25:52.981Z Wa(03)+ host-20388 
-2026-09-04T08:25:52.981Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr:     setup_service_account(args.account, user_perm = args.userPerm, 
-2026-09-04T08:25:52.981Z Wa(03)+ host-20388   File "/usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py", line 573, in setup_service_account 
-2026-09-04T08:25:52.981Z Wa(03)+ host-20388 
-2026-09-04T08:25:52.981Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr:     machine_user, machine_password = loadMachineCredentials() 
-2026-09-04T08:25:52.981Z Wa(03)+ host-20388   File "/usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py", line 234, in loadMachineCredentials 
-2026-09-04T08:25:52.981Z Wa(03)+ host-20388 
-2026-09-04T08:25:52.981Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr:     name = vmafdClient.GetMachineName() 
-2026-09-04T08:25:52.981Z Wa(03)+ host-20388 RuntimeError: unidentifiable C++ exception 
-2026-09-04T08:25:52.981Z Wa(03)+ host-20388 
-2026-09-04T08:25:53.062Z Er(02) host-20388 <applmgmt> Service pre-start command failed with exit code 1. 
-2026-09-04T08:25:53.062Z Wa(03) host-20388 [ReadSvcSubStartupData] No startup information from applmgmt. 
-2026-09-04T08:26:57.205Z In(05) host-20388 Executing service batch op API_HEALTH. IgnoreFail=1, service count=17 
-2026-09-04T08:26:57.205Z In(05) host-20388 <envoy-sidecar> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <envoy-hgw> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <envoy> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <rhttpproxy> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <vcha> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <vmware-vpostgres> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <vc-ws1a-broker> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <vmware-postgres-archiver> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <lookupsvc> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <sts> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <vapi-endpoint> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <sps> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <vpxd-svcs> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <vpxd> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <pschealth> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <wcp> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 <rbd> Skip service health check. State STOPPED, Curr request 0 
-2026-09-04T08:26:57.205Z In(05) host-20388 Successfully executed service batch operation API_HEALTH. 
-2026-09-04T08:27:46.785Z In(05) host-20388 Client info Uid=0,Gid=0,Pid=21691,Comm=(vmon-cli),PPid=21679,Comm=(service-control),PPid=18238,Comm=(bash),PPid=13635,Comm=(vherdrunner),PPid=13611,Comm=(sshd),PPid=13505 
-2026-09-04T08:27:46.786Z In(05) host-20388 Client info Uid=0,Gid=0,Pid=21692,Comm=(vmon-cli),PPid=21679,Comm=(service-control),PPid=18238,Comm=(bash),PPid=13635,Comm=(vherdrunner),PPid=13611,Comm=(sshd),PPid=13505 
-2026-09-04T08:27:46.787Z In(05) host-20388 Client info Uid=0,Gid=0,Pid=21693,Comm=(vmon-cli),PPid=21679,Comm=(service-control),PPid=18238,Comm=(bash),PPid=13635,Comm=(vherdrunner),PPid=13611,Comm=(sshd),PPid=13505 
-2026-09-04T08:27:46.787Z In(05) host-20388 Received start request for applmgmt 
-2026-09-04T08:27:46.787Z In(05) host-20388 <applmgmt-prestart> Constructed command: /usr/lib/applmgmt/support/scripts/prestart-applmgmt.sh 
-2026-09-04T08:27:46.798Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: Removed /etc/systemd/system/applmgmt.service. 
-2026-09-04T08:27:46.798Z Wa(03)+ host-20388 
-2026-09-04T08:27:46.997Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: Created symlink /etc/systemd/system/applmgmt.service → /dev/null. 
-2026-09-04T08:27:46.997Z Wa(03)+ host-20388 
-2026-09-04T08:27:47.232Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: Error: Failed to open the store. 
-2026-09-04T08:27:47.232Z Wa(03)+ host-20388 
-2026-09-04T08:27:47.232Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: vecs-cli failed. Error 1021: Cannot connect to vmafd service. 
-2026-09-04T08:27:47.232Z Wa(03)+ host-20388 
-2026-09-04T08:27:47.337Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: 2026-09-04 08:27:47,337 - 21784 - INFO - vcsa_network_base - 80 - StreamHandler added 
-2026-09-04T08:27:47.337Z Wa(03)+ host-20388 
-2026-09-04T08:27:47.337Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: 2026-09-04 08:27:47,337 - 21784 - INFO - vcsa_network_base - 66 - NetworkUtil; Doesn't exist creating one. 
-2026-09-04T08:27:47.337Z Wa(03)+ host-20388 
-2026-09-04T08:27:47.749Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: WARNING:root:Service account vsr file /etc/vmware/service-registry/vsr/service-accounts/vmware-applmgmtservice.yaml not present, we will go ahead and use the method parameters sent by the caller. 
-2026-09-04T08:27:47.749Z Wa(03)+ host-20388 
-2026-09-04T08:27:47.751Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: ERROR:root:Not found service account hash file: /var/cache/svcaccounts/vmware-applmgmtservice/vmware-applmgmtservice.hash 
-2026-09-04T08:27:47.751Z Wa(03)+ host-20388 
-2026-09-04T08:27:47.751Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr: Traceback (most recent call last): 
-2026-09-04T08:27:47.751Z Wa(03)+ host-20388   File "/usr/lib/applmgmt/support/scripts/create_svc_account.py", line 80, in <module> 
-2026-09-04T08:27:47.751Z Wa(03)+ host-20388 
-2026-09-04T08:27:47.751Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr:     main() 
-2026-09-04T08:27:47.751Z Wa(03)+ host-20388   File "/usr/lib/applmgmt/support/scripts/create_svc_account.py", line 75, in main 
-2026-09-04T08:27:47.751Z Wa(03)+ host-20388 
-2026-09-04T08:27:47.752Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr:     setup_service_account(args.account, user_perm = args.userPerm, 
-2026-09-04T08:27:47.752Z Wa(03)+ host-20388   File "/usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py", line 573, in setup_service_account 
-2026-09-04T08:27:47.752Z Wa(03)+ host-20388 
-2026-09-04T08:27:47.752Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr:     machine_user, machine_password = loadMachineCredentials() 
-2026-09-04T08:27:47.752Z Wa(03)+ host-20388   File "/usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py", line 234, in loadMachineCredentials 
-2026-09-04T08:27:47.752Z Wa(03)+ host-20388 
-2026-09-04T08:27:47.752Z Wa(03) host-20388 <applmgmt> Service pre-start command's stderr:     name = vmafdClient.GetMachineName() 
-2026-09-04T08:27:47.752Z Wa(03)+ host-20388 RuntimeError: unidentifiable C++ exception 
-2026-09-04T08:27:47.752Z Wa(03)+ host-20388 
-2026-09-04T08:27:47.825Z Er(02) host-20388 <applmgmt> Service pre-start command failed with exit code 1. 
-2026-09-04T08:27:47.825Z Wa(03) host-20388 [ReadSvcSubStartupData] No startup information from applmgmt. 
-root@localhost [ ~ ]# systemctl start applmgmt.service 
-Failed to start applmgmt.service: Unit applmgmt.service is masked. 
-root@localhost [ ~ ]# vmon-cli --status applmgmt 
-Name: applmgmt 
-Starttype: AUTOMATIC 
-RunState: STOPPED 
-RunAsUser: root 
-CurrentRunStateDuration(ms): 810584 
-HealthState: UNHEALTHY 
-FailStop: TRUE 
-MainProcessId: N/A 
-root@localhost [ ~ ]# 
- 
-</code> 
wiki/logs1.1788510657.txt.gz · Last modified: by wagner.jer
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki