Objective
This article aims to empower and enhance Workspot administrators in recognizing and resolving issues associated with Workspot Templates, Desktops, and Servers.
Workspot Desktop Agent:
The Workspot Desktop Agents run on Workspot machines (Desktops or App Servers) and communicate with Workspot Control by opening TCP connections on port 443 and establishing a secure websocket connection with the Control. If, for some reason, Agent stops communicating, then Workspot Control sets the status to Error, something like Agent is Offline.
The agent collects information about Users’ sessions, resources (CPU, RAM, Disks), Bandwidth and Latency, GPU Driver, etc., and sends it to the backend that eventually shows up in the Watch and Trends portals.
The agent needs to communicate with the following Workspot Components.
https://control.workspot.com:443 (Assign, configure, and monitor the user's Workspot resources). Also,
https://data.workspot.com:443 (Collects usage and event data for use by your IT). Also,
https://sentry.io:443 (For uploading crash logs).
Please refer to the Internet Firewalls and Workspot Article: Internet Firewalls and Workspot.
For GCP Templates, we have two additional sites with which the agent communicates
http://metadata.google.internal:80 (to communicate with the Google metadata service, IP – 169.254.169.254) and
https://com-workspot-deploy.s3.amazonaws.com:443 (to download the agent installer in case of the Build Template method of provisioning)
Workspot Agent States:
Depending on the selective features enabled on Customers’ accounts, there are different states of Workspot Agents, which are listed below:
WS-Agent State | Description |
Configuring | (Only for on-premises) The desktop is running configuration tasks. |
Connected | A user is signed into the desktop and has an active Client session. |
Customizing | (Only for on-premises) Customization is getting done for the desktop. |
Deallocated | The desktop has been put in a power-saving state. (The desktop has been deleted from the Cloud and will be cold provisioned based on slot capacity.) |
Deallocating | The desktop is being put in a power-saving state. (The desktop is being deleted from Cloud) |
Deleting | The desktop is being deleted in Control. |
Disconnected | A user is signed into the desktop but doesn’t have an active Client session. |
Error/Failed | The desktop agent detected a problem and is not available. |
Not Started | The provisioning has not yet started for the desktop. |
Offline | The provisioning is completed for the desktop, but powering on failed on the infrastructure. |
Online | The desktop is online but is not yet ready for user sessions. |
Paused | The desktop has been put in a power-saving hibernation state. |
Pausing | The desktop is being put in a power-saving hibernation state. |
Powering On | (Only for on-premises) Powering on the desktop. |
Provisioning | The desktop is being created. |
Ready | The desktop is ready for user sessions. |
Rebooting | The desktop is restarting. |
Redeploying | The desktop is being recreated. |
Refreshing | The desktop is being reimaged with a new Template. |
Restoring | The desktop is getting restored from the backup |
Resuming | The desktop is being resumed from the hibernated/paused state. |
Resume Failed | The desktop detected a problem while resuming. |
Running | The desktop is online. |
Starting | The desktop is being brought online. |
Stopping | The desktop is being shut down. |
Suspended | The desktop has been put in an unusable state due to user unassignment. |
Suspending | The desktop is being put in a power-saving state. |
Not Registered | The Control does not recognize the WS Agent/ VDI identity |
Waiting for IP | (Only for on-premises) Waiting for the IP address to be assigned to the desktop. |
Agent Installation and Configuration:
An administrator typically comes across the Workspot Agent for the first time when they need to build a template. Please refer to the following article for details: The Workspot Desktop Agent.
Template Configuration with the help of Workspot Config Editor
When you create a new desktop from the template, the Workspot Agent performs several tasks that are controlled by the WorkspotConfig.xml file. The Workspot Agent stores this information in an encrypted secure vault. The information given is subsequently utilized by the Workspot Agent during the provisioning of the VM and is removed after the Virtual Desktop has been successfully provisioned.
The Workspot Configuration Editor tool provides an easy and simple way to update the WorkspotConfig.xml file in the Workspot Template, and more importantly, it helps to validate the details provided related to Domain, OU, and credential validation.
To launch, go to the “C:\Program Files\WorkspotAgent” folder and run the WorkspotConfigEditor utility with elevated Administrator privileges.
Note: Refer to the XML Template Configuration section in this document for more details: The Workspot Desktop Agent.
In case the WorkspotConfig.xml file does not exist in the “c:\programdata/WorkspotAgent/Config” folder, you will get the following warning prompt. Click on OK and Continue.
A Click on “OK” should copy the configured editor from Source: “C:\Program Files\WorkspotAgent” to Destination: “C:\ProgramData\WorkspotAgent\Config “.
Note: In case you get the permission issue while copying the XML file to the destination folder.
Please navigate to the C:\ProgramData\WorkspotAgent\ folder and click on Continue. The window will set the permission to the folder, subfolders, and files.
Then, attempt to reopen the WorkspotConfigEditor utility from the source location using elevated Administrator privileges.
Sections of Workspot Config Editor:
Workspot Domain Join:
Provide domain details to join the Virtual Desktop to the Domain. (Mandatory)
Note: Please run “Test Secure Credentials” to validate details provided are correct.
The following section will create a new domain join account for the customer to use with Workspot with the required permissions. (On a need basis)
Workspot Shell Setup (On a need basis):
Additionally, in the Domain Accounts list, add the Domain Users or group to the Remote Desktop Users group or any other local group of the VDI during VM provisioning.
Workspot Script Setup (On a need basis):
The Workspot Agent looks for these scripts under “C:\ProgramData\WorkspotAgent\Config”.
- Setup Complete Script: This script runs immediately after cloning is complete and the VM is registered with Workspot Control, but before domain join. The script should be named as WsSetupComplete.ps1.
- Run Once Script: This script runs after the new VM reboots, post joining the domain successfully. The script should be named as WsRunOnce.ps1.
There are two parameters associated with this script:
Needs Reboot - the VM reboots after the script is run.
Stop On Failure - the Agent exits if the script fails, leaving the VM incompletely configured.
Please Note: If the script requires a reboot and the “needs a reboot” parameter is not checked Agent flow will break and the VM will get stuck in the online state or fail to provision.
Click Okay to Submit, and the details will get updated to the workspotconfig.xml file, and the password will be securely stored in the encrypted password vault.
Note: Please do run “Test Secure Credentials” to validate that the details provided are correct before you click on submit.
How to update WorkspotConfig.xml in the Template
Clone the existing template if the template is published. Otherwise, change the template to draft mode from the Action menu in the Control, which enables an administrator to make the changes in the template.
Log in to the Template with credentials.
Run the WorkspotConfigEditor from “C:\Program Files\WorkspotAgent" to edit the Workspotconfig.xml file.
Verify and update the Workspot Domain join details.
Verify the secure connection and confirm its success. Click on Save the Editor to store the credentials.
If there is any WSsetupcomplete or WSRunOnce script, verify the same.
Finally, shut down and publish the Template from the Control.
Apply the Template to the Preview Pool or existing pool, or create a new pool to test the template.
In case VMs were already in the failed state after provisioning a new template, delete the VM.
If the pool configuration is set to “Auto-create on Delete”, the VM will be recreated automatically.
If not, increase the pool VM numbers to add VMs to the pool.
Observe the VDI successfully created and joined the domain without any issues.
Workspot Agent Log location:
The Workspot Agent logs will help in troubleshooting an issue with appropriate actions.
Here are the locations of the Agent logs:
C:\Program Files\WorkspotAgent\InstallLog: This log captures details related to the agent installation and initial template registration.
C:\ProgramData\WorkspotAgent\Log: This log captures all the activity of the Agent from Virtual Desktop provisioning.
C:\ProgramData\WorkspotMSIUpgrade: This log captures details related to Agent upgrade.
C:\ProgramData\WorkspotAgent\WorkspotScriptsOutput: This log captures the output of a script (if there is any).
C:\ProgramData\WorkspotAgent\ScriptLogs: This log captures the logs related to the Script initiated by the Agent.
Things to check before finalizing a Template
Here is the list of the things an Administrator must check before publishing a Template:
The template should be in the right network and have the right DNS address. You can check the details by running the “ipconfig/all” command from the command line.
Make sure the service account has appropriate permissions to add/modify and delete Computer objects in AD. See Configuring the Active Directory Domain Join Account.
Workspot Config editor details must be correct. Please run “Test Secure Credentials” to validate the details provided related to Domain, OU, and credentials against the Domain controller.
If there is any script (run during setup or after domain Join) test properly before applying the Template to a pool.
Users and/or Groups should be added to the remote desktop group of the virtual machine. This can be configured through the Config Editor group policy or any other tool.
Problematic States:
There are two most undesirable states of Virtual Desktop where an Administrator needs to pay attention to troubleshoot.
FAILED STATE
In 99% of cases, you will see a Virtual Desktop (VD) enter a failed state during provisioning. This occurs when Control attempts to provision a desktop, but the deployment process is incomplete, or the desktop fails to report back to Control. In the remaining 1% of cases, this issue arises when a user uninstalls the Workspot Agent from a desktop.
If a user uninstalls the Workspot Agent from a persistent desktop, follow the steps in the "Workspot Desktop Agent Re-registration" section to resolve the issue. For a non-persistent VM, you can delete and recreate the VD. The VM will be automatically recreated due to the non-persistent pool setting "Auto-create on Delete."
If a VD fails during provisioning, multiple causes could be responsible. The most likely cause is improper configuration of the Workspot Config XML in the template, resulting in the VM failing to join the Active Directory domain. Other potential issues include quota problems or various other causes.
One of the important logs to troubleshoot the domain join error is C:\Windows\Debug\netsetup.log.
Reference Article:
Failed VM Known Issues
We have tried to capture the different reasons why VMs fail to provision.
Quota Issue:
Errors:
On AZURE (not limited to the below errors):
The operation could not be completed as it resulted in exceeding the approved standardNVSv4Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus2, Current Limit: 120, Current Usage: 116, Additional Required: 8, (Minimum) New Limit Required: 124. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/?#create/Microsoft.Support/Parameters/%7B%22subId%22:%92ba-c657ae790603%22,%22pesId%22:%-5802-169c800dec89%22,%22supportTopicId%22:%af33-c6d0-3c50df9658a3%22%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests./n Error Code: OperationNotAllowed
CloneCustomizeAndPowerOnAfterDelete operation failed. Reason: Error Message: The operation couldn't be completed as it resulted in exceeding the quota limit of Core. Maximum allowed: 1050, Current in use: 1048, Additional requested: 4. Read more about quota limits at https://aka.ms/AzurePerVMQuotaLimits. Submit a request for Quota increase using the nk - Error Code: OperationNotAllowed
Status code 409, "{"error": {"code": "OperationNotAllowed", "message": "Operation could not be completed as it results in exceeding approved Standard NCASv3_T4 Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 0, Current Usage: 0, Additional Required: 8, (Minimum) New Limit Required: 8. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2291842da1-0dfc-4693-849c-... by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"} }"
On GCP (not limited to the below errors):
Unable to auto-scale within the time limit. This may be due to a lack of quota or a lack of resources.
Solution: Put a request to increase the Quota
Error Code: 1331
Error: Domain joined failed with [Error Code:1331]. This user can’t sign in because this account is currently disabled.
Possible Cause: The Domain join Workspot Agent Service account used in the WorkspotConfigEditor has been disabled in the AD.
Possible Solution: Enabling the Workspot Agent Service account in the AD will fix the issue. The recommendation is to set the Workspot Agent Service account never to expire.
Error Code: 2224
Error: Domain join failed with [Error Code:2224]. The computer account already exists in AD.
Possible Cause: The Agent service account is trying to create a computer account; however, the account already exists in a different OU.
Possible Solution: Delete the computer account from Active Directory and recreate the VM from Control or verify the WorkspotConfig.xml file details, the OU path might not be correct.
Error Code: 87
Error: Domain join Read Credentials failure (Error Code: 87). The parameter is incorrect.
Possible Cause: The most probable cause is Active Directory Organizational Unit (OU) path is not correct or check spaces & invalid characters in the OU path.
Possible Solution: Reconfiguring the Workspotconfig.xml file from WorkspotConfigEditor in the Template. Delete the template.
Error Code: 2732
Error: Domain join failed with [Error Code: 2732]. An Account with the same name exists in Active Directory. Re-using the account was blocked by security policy.
Possible Cause: This issue may occur due to Microsoft's hardening mentioned in the article below. Ref: https://support.microsoft.com/en-au/topic/kb5020276-netjoin-domain-join-hardening-changes-2b65a0f3-1f4c-42ef-ac0f-1caaf421baf8
Possible Solution: Two requirements are deployed using GPO in the domain controller (DCs)
"Domain controller: Allow computer account re-use during domain join"
"Network access - Restrict clients allowed to make remote calls to SAM"
Error Code: 110
Error: Domain Join failed with [Error Code: 110]. The system cannot open the device or file specified.
Possible Cause: Workspot Agent software might have been uninstalled and installed again on the template.
Possible Solution: Please contact the Workspot Support Team to reregister the template with Control using a token. The Support Team will collaborate with the backend team to retrieve the token for that template from the database and execute the reregister agent with the token command to add the template back to Control.
Recommendation: Do not try to uninstall/reinstall the WS Agent in the Workspot template (On Cloud Templates) after the template is registered successfully with Control once. Instead, the template could be cloned and updated with any custom changes as needed.
Error code: 5
Error: Access is denied.
Possible Cause: Workspot Agent Service Account permissions issue.
Possible Solution: Setting up the appropriate recommended permissions for the domain join account will fix the issue. The following permissions are recommended for the AD join service account to automate the process.
Allow Read permission to All AD Properties on OU
DSACLS $OU /I:S /G "$($user): GR;;computer"
Allow Permission to Reset Password of Computer object on OU
DSACLS $OU /I:S /G "$($user):CA;Reset Password;computer"
Allow Permission to Write DNS Host Name Attributes of Computer objects on OU
DSACLS $OU /I:S /G "$($user):WP;DNS Host Name Attributes;computer"
Allow Permission to Write computer Account Restrictions on OU
DSACLS $OU /I:S /G "$($user):WP;Account Restrictions;computer"
Allow Permission to Write Computer Service Principle Name on OU
DSACLS $OU /I:S /G "$($user):WP;servicePrincipalName;computer"
Allow Permission to Create and Delete Computer Objects on OU
DSACLS $OU /I:S /G "$($user):CCDC;computer;organizational
Verify that the Template VM and Virtual Desktops are getting the right DNS server to connect to AD.
Error Code: 53
Error: The network path was not found.
Possible Cause: The virtual desktop is not able to communicate with Active Directory. Customers have installed security software on their DC. Ex: Palo Alto Cortex security software blocking domain joins. A firewall rule to block SMB and RPC access.
Possible Solution: Remove the Security software and any Firewall rules to resolve the issue.
Please note: In some cases, we also observed that this error code gets generated because the NTLM Compatibility level is set to 5. Try setting the value to 3 in the local client device and try to access the Workspot device. Reference Link: https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-10/security/threat-protection/security-policy-settings/network-security-lan-manager-authentication-level
Error Code: 2220
Error: Domain Join failed with [Error Code: 2220]. The group name could not be found.
Possible Cause: Incorrect details in the Config editor XML file.
Possible Solution: Provide the correct details in the Workspot Config Editor.
Error Code: 2
Error: Domain join failed with [Error Code: 2]. The system cannot find the file specified.
Possible Cause: Domain details provided are not valid/misspelled in the WorkspotConfig Editor. The virtual desktop does not have the right DNS server address. The attempt to resolve the DNS name of a domain controller in the domain being joined has failed OU Path has unwanted spaces in it.
Possible Solution: Validate the DNS server address and Domain Controller reachability. Also, validate the domain, OU, and Credentials details through Workspot Config Editor.
Error Code: 64
Error: Domain join failed with [Error Code: 64]. The specified network name is no longer available.
Possible Cause: The Virtual machine is not able to resolve the DNS or Domain Controller address.
Possible Solution: Validate the DNS server address and Domain Controller reachability. Also, validate the domain, OU, and Credentials details through Workspot Config Editor.
In the GCP environment, the Compute metadata information in the GCP console is correct. Also, the Virtual Desktop gets the right DNS information from metadata. If not, make sure the firewall policy is in place to communicate with metadata “169.254.169.254”.
Error Code: 2700
Error: The machine attempted to join the domain but failed.
Possible Cause: The Template was already part of Azure AD.
Possible Solution: To join the Template machine to an Active Directory domain, you must first go to Settings > Accounts > Access Work or School and disconnect the device from your work or school. Then, reconfigured the template through the config editor.
Error Code: 1326
Error: The username or password is incorrect.
Possible Cause: Check if the customer Admin might have changed the service account password which turned into VDI failing to join the domain or the Service Account password expired.
Possible Solution: Reconfiguring the config.xml file through the Workspot config Editor with an updated username and password will resolve the issue.
Error Code 1355
Error: Domain join failed with [Error Code: 1355]. The specified domain either does not exist or cannot be contacted.
Possible Cause: This could be because of a Network issue, or the wrong DNS configuration.
Possible Solution: Reconfiguring the config.xml file through the Workspot config Editor with an updated username and password will resolve the issue.
Error Code: 8557
Error: Domain join failed with [Error Code: 8557]. Your computer could not be joined to the domain. You have exceeded the maximum number of computer accounts you are allowed to create in this domain. Contact your system administrator to have this limit reset or increased.
Possible Cause: Due to default AD limits, the service account quota has been reached to add machines to the domain.
Possible Solution: Recommendation to modify the service account permissions to add more Computer accounts in the AD. See Configuring the Active Directory Domain Join Account.
Required Domain join credentials
Error: Required Domain join credentials are not provided in the WorkspotConfig.xml file.
Possible Cause: Domain join credentials were not configured and validated through the WorkspotconfigEditor in the template VM.
Possible Solution: reconfigure the domain join credentials and validate through “Test Credentials”, after a successful test, save the configuration and reprovision the Virtual Desktops.
Error/Status Code 409
Error: Status code 409, error code: OperationsNotAllowed. The specified disk size is smaller than the size of the corresponding disk in the VM Image. This is not allowed. Please choose equal or greater size, or do not specify an explicit size.
Possible Cause: The template is 256GB in control and Azure, is 512 GB.
Possible Solution: Manually created a new Template with a larger disk space using the disk from the parent Template. Subsequently, register the new template with the Workspot control.
AAD Join exception. Install the WCD tool and configure it properly.
Error: Exception occurred while AAD Join: Template needs to be configured properly. Install the WCD tool.
Possible Cause: The WCD tool in the template would get corrupted.
Possible Solution: Updating and reconfiguring the WCD tool should resolve the issue. See AAD VM Template Creation Process.
Failed with provisioning Async Operation/ Azure
Error: CloneCustomizeAndPowerOnAfterDelete operation failed. Reason: Async operation failed with provisioning
Possible Cause: This is an Azure issue where the VM creation operation is somehow not executed successfully.
Possible Solution: Try deleting and re-creating the failed Virtual desktop.
Failed to create desktop.
Error: Failed to create the desktop. Reason: Control could not get a list of Valid VM sizes for region XYZ. Your original desktop has been deleted. Contact Workspot Support.
Possible Causes: The Azure API operation is taking longer than expected, or the Service principal (SPN) Application secret has expired, as a result, Control is not able to communicate with the Azure Subscription.
Possible Solution: Validate the Service principal (SPN) Application secret, if expired, update the new secret in the Control. See Creating a Service Principal in Azure.
ERROR STATE
If a Virtual Desktop has communicated (at least once) with the Workspot Control and subsequently the Control fails to get a reply from the Agent operating in the Virtual Desktop (VD), the Workspot Control will place the VM into an Error state. There may be various factors that can cause a VM to enter an error state. A few of them are:
The Virtual Desktop is not running
Virtual Desktop is running; however, the NIC got disabled or is not able to communicate with the internet because of the Firewall and routing
The VD is not able to resolve Service.workspot.com or Control.workspot.com.
The Desktop Agent service is not running or corrupted.
Workspot Desktop Agent Re-Registration:
If the Persistent Virtual Desktop is up and running, however, the Control or Watch still shows the Desktop status in Error “Agent is Offline”. Please follow the steps below:
Check Network Connectivity from the Virtual Desktop:
Test-NetConnection control.workspot.com -port 443
Test-NetConnection service.workspot.com -port 443
Test-NetConnection data.workspot.com -port 443
To re-register a VM Agent back to Control, please follow the steps below:
Go to the Workspot Control > Resources > VMs Pool
Click on Agent Token
If the Agent Token is expired, click on regenerate to get a new token and copy the New Agent Token.
Log in to the error VM with your admin credentials.
Uninstall any Workspot Agent from Add & Remove Programs
Download the latest Workspot Agent from https://download.workspot.com/WorkspotAgentSetup.msi (DO NOT INSTALL)
Open the CMD prompt with elevated permissions and run the below command
msiexec /i "c:\users\%username%\downloads\WorkspotAgentSetup.msi" /qn NoregisterAgent=true
Note: This will install Workspot Agent in silent mode and force it not to automatically register with the Workspot Control. We only install the Agent through UI mode in the new Corporate Template VM.
Navigate the path to "C:\Program Files\WorkspotAgent>" and then run the below command.
reregister_poolvm.bat [Agent Token]
For Newer Agents: reregister_poolvm.bat [Agent Token] [Company Identifier]
After the command is completed, refresh the control page and verify the VM status it should show the connected/ready status. See Workspot Agent State and Troubleshooting.
BIOS GUID address changed from when the VM was registered (On-Prem Customers).
WS agent service is in a stopped state with the reason, “BIOS GUID address changed from when VM was registered”. In the WS Agent log.
Possible Cause: This issue occurred because of BIOS GUID changes. Desktop Agent uses BIOS GUID with MAC address to register a VM to the Control.
Practical Resolution: Change the Private IP address of the VM and Re-register the Virtual Desktop.
Azure Virtual Desktop showing no Internet on the Boot Diagnostic page
Possible Cause: The user could have tried to change the Azure Virtual Desktop Network properties manually.
Possible solution: Replace the NIC when an Azure VM loses network Connectivity. See Replace NIC when an Azure VM loses network Connectivity.