Overview

You are running ScaleArc v3.11.0.2.0-977 and have started experiencing a loss of load balancing in your ScaleArc environment after upgrading the SQL infrastructure to SQL Server 2019.

The following alert is appearing in the logs for all users:

2021-04-26 09:06:28 CID: 1, Client IP: 192.168.2.204, User: <username>, Debug_Code: 527, 
Message: Could not find per-user cached login information required for authenticating this user '<username>'. 
Bypassing authentication for this connection. Cluster restart may help resolve this issue 
SSL: No , State: 32, SSID: 819023, DB: <database name>, DB IP: NONE, Type: 37

Deleting and re-adding the user does not work and gives an error saying the user cannot be deleted. Adding a user also fails with ScaleArc alerting that it cannot connect to the primary server.

Error messages in api.error.* logs:

api.error.2021042608:2021-Apr-26 08:42:00 ClientIp=x.x.x.x 
HTTP_REFERER=https://x.x.x.x/api/cluster/1/user/18?apikey= DATA=" | Command sent to core = 
delete|user|1|T0xUUF9GaW5kQ2hpcHNNb3VzZXI=| | result = Data Not Found For Deletion | "

api.error.2021042608:2021-Apr-26 08:42:00 ClientIp=x.x.x.x 
HTTP_REFERER=https://x.x.x.x/api/cluster/1/user/18?apikey= DATA="dprohttps://support.scalearc.com/kb/articles/461 
2267 ERROR: kb2267 Failed to delete user. Data Not Found For Deletion"

Solution

ScaleArc v3.11.0.2.0-977 with SQL Server 2019 is an unsupported configuration according to the following extracts from the release notes:

• ScaleArc 3.11.0.2 only supports up to SQL Server 2016. 
• ScaleArc 2019.x and newer releases introduced support for SQL Server 2017 but with limited new features support. 
• SQL Server 2019 is not fully certified, hence not recommended for production environments, but is known to work correctly in ScaleArc 2020.x releases.

The following actions can be taken to bring ScaleArc load balancing back to normal when this issue is encountered after the SQL upgrade to 2019:

1. Create a new cluster pointing to the same SQL Availability Group (AG) with a different instance name and port and replicate all the configurations from the existing cluster to the new one.
2. Restart the cluster, which should bring back the core cluster process to sync with the cluster SQLite configuration database. This action will restore the user removal/modification feature to a working state and you should be able to remove and re-add the users as expected. If a large portion of users still needs to be re-created, execute a HA switch over to the newly created cluster since the replicated configuration from step 1 will also include users.
3. Stop the ScaleArc services, switch over the ports and instance names between the existing cluster and the new one, and start the ScaleArc cluster.
   Note: If the old cluster was configured as the HA fencing cluster, it will be automatically started as well hence you must subsequently switch over the HA fencing cluster to the new cluster, and thereafter the old cluster can be safely stopped.

As a general recommendation, ensure that you are running a supported configuration by referring to the release notes document for your version. Whenever possible, upgrade to the latest release to leverage new features and bug fixes for known issues affecting older ScaleArc releases.

See KB 2267 - Cluster Settings Error for some additional related information on the error code observed in the UI as well as the api.error.* logs.

Testing

Check the Live Monitor - Cluster Stats to confirm that Load balancing is distributing the read queries among all available servers as expected. Caching should also be predominant in the overall server traffic.