Overview

Memory consumption spiking during the normal operation of ScaleArc will take all the configured clusters offline. Invoking a failover brings up the same issues on the new primary in a few hours.

Solution

Diagnosis

This situation is also accompanied with replication not working and the Witness Server being rendered unreachable. These will show up as events in the events log as show here

You can also check for memory consumption in the terminal with the free command.

Steps to Fix

This issue has been fixed in ScaleArc 2021.1 and it is recommended to upgrade to this version if you're facing this issue. The steps to upgrade are provided in this KB article.