Workspot Agent Issues & Troubleshooting
Objective
This article is intended to enable and elevate Workspot administrators to identify and troubleshoot issues related to Workspot Templates, Desktops, and Servers.
Workspot Desktop Agent:
The Workspot Desktop Agents run on Workspot machines (Desktops or App Servers) and communicate with Workspot Control by opening TCP connections on port 443 and establishing a secure web socket connection with the Control. If, for some reason Agent stops communicating, then Workspot Control sets the status to Error, something like Agent is Offline.
The agent collects information about Users’ sessions, resources (CPU, RAM, Disks), Bandwidth and Latency, GPU Driver, etc., and sends it to the backend that eventually shows up in the Watch and Trends portals.
The agent needs to communicate with the following Workspot Components.
https://control.workspot.com:443 (Assign, configure, and monitor the user's Workspot resources). Also,
https://data.workspot.com:443 (Collects usage and event data for use by your IT). Also,
https://data.eu.workspot.com:4439644
https://sentry.io:443 (For uploading crash logs).
Please refer to the Internet Firewalls and Workspot Article: https://community.workspot.com/general-59/internet-firewalls-and-workspot-218
For GCP Templates, we have two additional sites with which the agent communicates
http://metadata.google.internal:80 (to communicate with the Google metadata service. Ip – 169.254.169.254) and
https://com-workspot-deploy.s3.amazonaws.com:443 (to download the agent installer in case of Build Template method of provisioning)
Workspot Agent States:
Depending on the selective features enabled on Customers’ accounts there are different states of Workspot Agents, they are listed below:
WS-Agent State | Description |
Configuring | (Only for on-premises) The desktop is running configuration tasks. |
Connected | A user is signed into the desktop and has an active Client session. |
Customizing | (Only for on-premises) Customization is getting done for the desktop. |
Deallocated | The desktop has been put in a power-saving state. (The desktop has been deleted from the Cloud and will be cold provisioned based on slot capacity) |
Deallocating | The desktop is being put in a power-saving state. (The desktop is being deleted from Cloud) |
Deleting | The desktop is being deleted in Control. |
Disconnected | A user is signed into the desktop but doesn’t have an active Client session. |
Error/Failed | The desktop agent detected a problem and is not available. |
Not Started | The provisioning has not yet started for the desktop. |
Offline | The provisioning is completed for the desktop but powering on failed on infrastructure. |
Online | The desktop is online but is not yet ready for user sessions. |
Paused | The desktop has been put in a power-saving hibernate state. |
Pausing | The desktop is being put in a power-saving hibernate state. |
Powering On | (Only for on-premises) Powering on the desktop. |
Provisioning | The desktop is being created. |
Ready | The desktop is ready for user sessions. |
Rebooting | The desktop is restarting. |
Redeploying | The desktop is being recreated. |
Refreshing | The desktop is being re-imaged with a new Template. |
Restoring | The desktop is getting restored from the backup |
Resuming | The desktop is being resumed from the hibernate/paused state. |
Resume Failed | The desktop detected a problem while resuming. |
Running | The desktop is online. |
Starting | The desktop is being brought online. |
Stopping | The desktop is being shut down. |
Suspended | The desktop has been put in a non-usable state due to user unassignment. |
Suspending | The desktop is being put in a power-saving state. |
Not Registered | The Control does not recognize the WS Agent/ VDI identity |
Waiting for IP | (Only for on-premises) Waiting for the IP address to get assigned to the desktop. |
Agent Installation and Configuration:
An administrator typically comes across the Workspot Agent for the first time when they need to build a template.
Please refer to the following article for details: https://community.workspot.com/workspot-agents-106/workspot-desktop-agent-installation-and-configuration-191
Template Configuration with the help of Workspot Config Editor
When you create a new desktop from the template, the Workspot Agent performs several tasks that are controlled by the WorkspotConfig.xml file. The Workspot Agent stores this information in an encrypted secure vault. The provided information is later used by the Workspot Agent during VM provisioning and gets deleted after successful provisioning of Virtual Desktop.
Workspot Configuration Editor tool provides an easy and simple way to update the WorkspotConfig.xml file in the Workspot Template and more importantly, it helps to validate the details provided related to Domain, OU, and credential validation.
To launch, go to the “C:\Program Files\WorkspotAgent” folder and run the WorkspotConfigEditor utility with the elevated Administrator privilege.
Note: Refer to the XML Template Configuration section in this document for more details. https://community.workspot.com/workspot-agents-106/workspot-desktop-agent-installation-and-configuration-191
In case the WorkspotConfig.xml file does not exist in the “c:\programdata/WorkspotAgent/Config” folder you will get the below warning prompt. Click on OK and Continue.
A Click on “OK” should copy the configured editor from Source: “C:\Program Files\WorkspotAgent” to Destination: “C:\ProgramData\WorkspotAgent\Config “.
Please Note: In case, you get the permission issue while copying the XML file to the destination folder.
Please navigate to the C:\ProgramData\WorkspotAgent\ folder and click on Continue. The window will set the permission to the folder, subfolders, and files.
Then, attempt to reopen the WorkspotConfigEditor utility from the source location using elevated Administrator privileges.
Sections of Workspot Config Editor:
Workspot Domain Join
Provide domain details to join the Virtual Desktop to the Domain. (Mandatory)
Note: Please run “Test Secure Credentials” to validate details provided are correct.
The following section will create a new domain join account for the customer to use with Workspot with the required permissions. (On a need basis)
Workspot Shell Setup (On a need basis):
Additionally, in the Domain Accounts list, add the Domain Users or group to the Remote Desktop Users group or any other local group of the VDI during VM provisioning.
Workspot Script Setup (On a need basis):
The Workspot Agent looks for these scripts under “C:\ProgramData\WorkspotAgent\Config”.
Setup Complete Script: This script runs immediately after cloning is complete and the VM is registered with Workspot Control, but before domain join. The script should be named as WsSetupComplete.ps1.
Run Once Script: This script runs after the new VM reboots post joining the domain successfully. The script should be named as WsRunOnce.ps1.
There are two parameters associated with this script:
Needs Reboot - the VM reboots after the script is run.
Stop On Failure - the Agent exits if the script fails, leaving the VM incompletely configured.
Please Note: If the script requires a reboot and the “needs a reboot” parameter is not checked the Agent flow will break and the VM will get stuck in the online state or fail to provision.
Click Okay to Submit and the details will get updated to the workspotconfig.xml file and the password will be securely stored in the encrypted password vault.
Note: Please do run “Test Secure Credentials” to validate that the details provided are correct before you click on submit.
How to update WorkspotConfig.xml in the Template
Clone the existing template, if the template is published. Otherwise, change the template to the draft mode from the Action menu in the Control, which enables an administrator to make the changes in the template.
Login to the Template with credentials.
Run the WorkspotConfigEditor from C:\ProgramFiles\WorkspotAgent" to edit the Workspotconfig.xml file.
Verify and update the Workspot Domain join details.
Test the secure connection and make sure it has succeeded. Click on Save the Editor to save the credentials.
If there is any WSsetupcomplete or WSRunOnce script, verify the same.
Finally shut down and Publish the Template from the Control.
Apply the Template to the Preview Pool or existing pool or create a new pool to test the template.
In case VMs were already in the failed state after provisioning a new template delete the VM.
If the pool configuration is set to “Auto-create on Delete”, the VM will be recreated automatically.
If not, increase the pool VM numbers to add VMs to the pool.
Observe the VDI successfully created and joined the domain without any issues.
Workspot Agent Log location:
The Workspot Agent logs will help in troubleshooting an issue with appropriate actions.
Here are the locations of the Agent logs:
C:\Program Files\WorkspotAgent\InstallLog: This log captures details related to the agent installation and initial template registration.
C:\ProgramData\WorkspotAgent\Log: This log captures all the activity of the Agent from Virtual Desktop provisioning.
C:\ProgramData\WorkspotMSIUpgrade: This log captures details related to Agent upgrade.
C:\ProgramData\WorkspotAgent\WorkspotScriptsOutput: This log captures the output of a script (if there is any).
C:\ProgramData\WorkspotAgent\ScriptLogs: This log captures the logs related to the Script initiated by the Agent.
Things to check before finalizing a Template
Here is the list of the things an Administrator must check before publishing a Template:
The template should be in the right network and have the right DNS address. You can check the details by running the “ipconfig/all” command from the command line.
Assumption: Template and Virtual Desktops should be in the same network (VNet/VPC).
Make sure the service account has appropriate permissions to add/modify and delete Computer objects in AD.
Workspot Config editor details must be correct. Please run “Test Secure Credentials” to validate the details provided related to Domain, OU, and credentials against the Domain controller.
Assumption: Template and Virtual Desktops should be in the same network.
If there is any script (run during setup or after domain Join) test properly before applying the Template to a pool.
Users and/or Groups should be added to the remote desktop group of the virtual machine. This can be configured through the Config Editor group policy or any other tool.
Problematic States:
There are two most undesirable states of Virtual Desktop where an Administrator needs attention to troubleshoot.
FAILED STATE
In 99% of cases, you will see a Virtual Desktop (VD) enter a failed state during provisioning. This occurs when Control attempts to provision a desktop, but the deployment process is incomplete, or the desktop fails to report back to Control. In the remaining 1% of cases, this issue arises when a user uninstalls the Workspot Agent from a desktop.
If a user uninstalls the Workspot Agent from a persistent desktop, follow the steps in the "Workspot Desktop Agent Re-registration" section to resolve the issue. For a non-persistent VM, you can delete and recreate the VD. The VM will be automatically recreated due to the non-persistent pool setting "Auto-create on Delete."
If a VD fails during provisioning, multiple causes could be responsible. The most likely cause is improper configuration of the Workspot Config XML in the template, resulting in the VM failing to join the Active Directory domain. Other potential issues include quota problems or various other causes.
One of the important logs to troubleshoot the domain join error is C:\Windows\Debug\netsetup.log.
Reference Article:
Failed VM Known Issues
We have tried to capture the different reasons why VMs fail to provision.
Quota Issue:
Errors:
On AZURE (not limited to the below errors):
The operation could not be completed as it resulted in exceeding the approved standardNVSv4Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus2, Current Limit: 120, Current Usage: 116, Additional Required: 8, (Minimum) New Limit Required: 124. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/?#create/Microsoft.Support/Parameters/%7B%22subId%22:%92ba-c657ae790603%22,%22pesId%22:%-5802-169c800dec89%22,%22supportTopicId%22:%af33-c6d0-3c50df9658a3%22%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests./n Error Code: OperationNotAllowed
CloneCustomizeAndPowerOnAfterDelete operation failed. Reason: Error Message: The operation couldn't be completed as it resulted in exceeding the quota limit of Core. Maximum allowed: 1050, Current in use: 1048, Additional requested: 4. Read more about quota limits at https://aka.ms/AzurePerVMQuotaLimits. Submit a request for Quota increase using the nk - Error Code: OperationNotAllowed
Status code 409, "{"error": {"code": "OperationNotAllowed", "message": "Operation could not be completed as it results in exceeding approved Standard NCASv3_T4 Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 0, Current Usage: 0, Additional Required: 8, (Minimum) New Limit Required: 8. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2291842da1-0dfc-4693-849c-... by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"} }"
On GCP (not limited to the below errors):
Unable to auto-scale within the time limit. This may be due to a lack of quota or a lack of resources.
Solution: Put a request to increase the Quota
Error Code 1331
Error: Domain joined failed with [Error Code:1331]. This user can’t sign in because this account is currently disabled.
Possible Cause: The Domain join Workspot Agent Service account used in the WorkspotConfigEditor has been disabled in the AD.
Possible Solution: Enabling the Workspot Agent Service account in the AD will fix the issue. The recommendation is to set the Workspot Agent Service account never to expire.
Error Code 2224
Error: Domain join failed with [Error Code:2224]. The computer account already exists in AD.
Possible Cause: The Agent service account is trying to create a computer account; however, the account already exists in a different OU.
Possible Solution: Delete the computer account from Active Directory and recreate the VM from Control or verify the WorkspotConfig.xml file details, the OU path might not be correct.
Error Code: 87
Error: Domain join Read Credentials failure (Error Code: 87). The parameter is incorrect.
Possible Cause: The most probable cause is Active Directory Organizational Unit (OU) path is not correct or check spaces & invalid characters in the OU path.
Possible Solution: Reconfiguring the Workspotconfig.xml file from WorkspotConfigEditor in Template. Delete the template.
Error Code 2732:
Error: Domain join failed with [Error Code: 2732]. An Account with the same name exists in Active Directory. Re-using the account was blocked by security policy.
Possible Cause: This issue may occur due to Microsoft hardening mentioned in the below article. Ref: https://support.microsoft.com/en-au/topic/kb5020276-netjoin-domain-join-hardening-changes-2b65a0f3-1f4c-42ef-ac0f-1caaf421baf8
Possible Solution: Two requirements deployed using GPO in the domain controller (DCs)
"Domain controller: Allow computer account re-use during domain join"
"Network access - Restrict clients allowed to make remote calls to SAM"
Error Code: 110
Error: Domain Join failed with [Error Code: 110]. The system cannot open the device or file specified.
Possible Cause: Workspot Agent software might have been uninstalled and installed again on the template.
Possible Solution: Please contact the Workspot Support Team to reregister the template with Control using a token. The Support Team will collaborate with the backend team to retrieve the token for that template from the database and execute the reregister agent with the token command to add the template back to Control.
Recommendation: Do not try to uninstall/ Re-install the WS Agent in the Workspot template (On Cloud Templates) after the template is registered successfully with Control once. Instead, the template could be cloned and updated with any custom changes as needed.
Error code 5
Error: Access is denied.
Possible Cause: Workspot Agent Service Account permissions issue.
Possible Solution:
Set up the appropriate recommended permissions to the domain join account will fix the issue.
The following permissions are recommended for the AD join service account to automate the process.
Allow Read permission to All AD Properties on OU
DSACLS $OU /I:S /G "$($user): GR;;computer"
Allow Permission to Reset Password of Computer object on OU
DSACLS $OU /I:S /G "$($user):CA;Reset Password;computer"
Allow Permission to Write DNS Host Name Attributes of Computer objects on OU
DSACLS $OU /I:S /G "$($user):WP;DNS Host Name Attributes;computer"
Allow Permission to Write computer Account Restrictions on OU
DSACLS $OU /I:S /G "$($user):WP;Account Restrictions;computer"
Allow Permission to Write Computer Service Principle Name on OU
DSACLS $OU /I:S /G "$($user):WP;servicePrincipalName;computer"
Allow Permission to Create and Delete Computer Objects on OU
DSACLS $OU /I:S /G "$($user):CCDC;computer;organizational
Please Refer: https://kb.workspot.support/configuring-the-active-directory-domain-join-account$setting%20permissions
Verify the Template VM and Virtual Desktops are getting the right DNS server to connect to AD.
Error Code: 53
Error: The network path was not found.
Possible Cause:
Virtual desktop not able to communicate to Active Directory.
Customers have installed security software on their DC. Ex: Palo Alto Cortex security software blocking domain joins.
A firewall rule to block SMB and RPC access.
Possible Solution: Remove the Security software, and any Firewall rules to resolve the issue.
Please note: In some cases, we also observed that this error code gets generated because the NTLM Compatibility level is set to 5. Try setting the value to 3 in the local client device and try to access the Workspot device. Reference Link: https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/security-policy-settings/network-security-lan-manager-authentication-level
Error Code: 2220
Error: Domain Join failed with [Error Code: 2220]. The group name could not be found.
Possible Cause: Incorrect details in Config editor XML file.
Possible Solution: Provide the correct details in the Workspot Config Editor.
Error Code: 2
Error: Domain join failed with [Error Code: 2]. The system cannot find the file specified.
Possible Cause: Domain details provided are not valid/misspelled in the WorkspotConfig Editor
The virtual desktop does not have the right DNS server address
Attempt to resolve the DNS name of a domain controller in the domain being joined has failed OU Path has unwanted spaces in it.
Possible Solution: Validate the DNS server address and Domain Controller reachability. Also, validate the domain, OU, and Credentials details through Workspot Config Editor.
Error Code: 64
Error: Domain join failed with [Error Code: 64]. The specified network name is no longer available.
Possible Cause: The Virtual machine is not able to resolve the DNS or Domain Controller address.
Possible Solution: Validate the DNS server address and Domain Controller reachability. Also, validate the domain, OU, and Credentials details through Workspot Config Editor.
In the GCP environment, the Compute metadata information in the GCP console is correct. Also, the Virtual Desktop gets the right DNS information from metadata. If not, make sure the firewall policy is in place to communicate with metadata “169.254.169.254”.
Error Code: 2700
Error: The machine attempted to join the domain but failed.
Possible Cause: The Template was already part of Azure AD.
Possible Solution: To join the Template machine to an Active Directory domain, you must first go to Settings > Accounts > Access Work or School and disconnect the device from your work or school. Then, re-configured the template through the config editor.
Error Code: 1326
Error: The username or password is incorrect.
Possible Cause: Check if the customer Admin might have changed the service account password which turned into VDI failing to join the domain or the Service Account password expired.
Possible Solution: Reconfiguring the config.xml file through the Workspot config Editor with an updated username and password will resolve the issue.
Error Code 1355
Error: Domain join failed with [Error Code: 1355]. The specified domain either does not exist or cannot be contacted.
Possible Cause: This could be because of the Network issue, or the wrong DNS configured.
Possible Solution: Reconfiguring the config.xml file through the Workspot config Editor with an updated username and password will resolve the issue.
Error Code: 8557
Error: Domain join failed with [Error Code: 8557]. Your computer could not be joined to the domain. You have exceeded the maximum number of computer accounts you are allowed to create in this domain. Contact your system administrator to have this limit reset or increased.
Possible Cause: Due to default AD limits the service account quota has been reached to add machines to the domain.
Possible Solution: Recommendation to modify the service account permissions to add more Computer accounts in the AD. Ref: https://community.workspot.com/active-directory-74/configuring-the-active-directory-domain-join-account-367
Required Domain join credentials
Error: Required Domain join credentials are not provided in the WorkspotConfig.xml file.
Possible Cause: Domain join credentials were not configured and validated through the WorkspotconfigEditor in the template VM.
Possible Solution: reconfigure the domain join credentials and validate through “Test Credentials”, after a successful test save the configuration and reprovision the Virtual Desktops.
Error/Status Code 409
Error: Status code 409, error code: OperationsNotAllowed. The specified disk size is smaller than the size of the corresponding disk in the VM Image. This is not allowed. Please choose equal or greater size or do not specify an explicit size.
Possible Cause: The template is 256GB in control and Azure, is 512 GB.
Possible Solution:
Manually created a new Template with a larger disk space using the disk from the parent Template.
Subsequently, register the new template with the Workspot control.
AAD Join exception. Install the WCD tool and configure it properly.
Error: Exception occurred while AAD Join: Template needs to be configured properly. Install the WCD tool.
Possible Cause: The WCD tool in the template would get corrupted.
Possible Solution: Update and reconfigure the WCD tool should resolve the issue.
Failed with provisioning Async Operation/ Azure
Error: CloneCustomizeAndPowerOnAfterDelete operation failed. Reason: Async operation failed with provisioning
Possible Cause: This is an Azure issue where the VM creation operation is somehow not executed successfully.
Possible Solution: Try deleting and re-creating the failed Virtual desktop.
Failed to Create desktop.
Error: Failed to create the desktop. Reason: Control Could not get a list of Valid VM sizes for region XYZ. Your original desktop has been deleted. Contact Workspot Support.
Possible Causes: The Azure API operation is taking longer than expected or the Service principal (SPN) Application secret expired as a result Control is not able to communicate with Azure Subscription.
Possible Solution: Validate the Service principal (SPN) Application secret, if expired update the new secret in the Control.
Ref: https://community.workspot.com/azure-117/creating-a-service-principal-in-azure-935?tid=935&fid=117
ERROR STATE
If a Virtual Desktop was communicating (at least once) to the Workspot Control and later Control did not receive any response from the Agent running in Virtual Desktop (VD), the Workspot Control puts the VM in the Error state. There could be multiple reasons for a VM to go to an error state. A few of them are:
The Virtual Desktop is not running
Virtual Desktop is running, however, NIC got disabled or not able to communicate to the internet because of Firewall, Routing
The VD is not able to resolve Service.workspot.com or Control.workspot.com.
Desktop Agent service not running or corrupted.
Workspot Desktop Agent Re-Registration:
If the Persistent Virtual Desktop is up and running, however, the Control or Watch still shows the Desktop status in Error “Agent is Offline”. Please follow the below steps:
Check Network Connectivity from the Virtual Desktop:
Test-NetConnection control.workspot.com -port 443
Test-NetConnection service.workspot.com -port 443
Test-NetConnection data.workspot.com -port 443
To re-register a VM Agent back to Control, please follow the below steps:
Go to the Workspot Control > Resources > VM’s Pool
Click on Agent Token
If the Agent Token is expired, click on regenerate to get a new token and copy the New Agent Token.
Login to the error VM with your admin credentials.
Uninstall any Workspot Agent from Add & Remove Programs
Download the latest Workspot Agent from the https://download.workspot.com/WorkspotAgentSetup.msi (DO NOT INSTALL)
Open the CMD prompt with elevated permissions and run the below command
msiexec /i "c:\users\%username%\downloads\WorkspotAgentSetup.msi" /qn NoregisterAgent=true
Note: This will install Workspot Agent in silent mode and force it to not automatically register with the Workspot Control. We only install Agent through UI mode in the new Corporate Template VM.
Navigate the path to "C:\Program Files\WorkspotAgent>" and then run the below command.
reregister_poolvm.bat [Agent Token]
For Newer Agents: reregister_poolvm.bat [Agent Token] [Company Identifier]
After the command is completed, refresh the control page and verify the VM status it should show the connected/ready status. Ref link: https://community.workspot.com/support-documentation-251/workspot-agent-state-and-troubleshooting-773
BIOS GUID address changed from when VM was registered (On-Prem Customers).
WS agent service is in a stopped state with the reason, “BIOS GUID address changed from when VM was registered”. In the WS Agent log.
Possible Cause: This issue occurred because of BIOS GUID changes. Desktop Agent uses BIOS GUID with MAC address to register a VM to the Control.
Practical Resolution: Change the Private IP address of the VM and Re-register the Virtual Desktop.
Azure Virtual Desktop showing no Internet on the Boot Diagnostic page
Possible Cause: The user could have tried to change the Azure Virtual Desktop Network properties manually.
Possible solution: Replace NIC when an Azure VM loses network Connectivity
Reference Link: https://community.workspot.com/azure-117/replace-nic-when-an-azure-vm-loses-network-connectivity-1134