🧠 What Every DBA Must Know

In a clustered database environment like Oracle RAC, maintaining data consistency across nodes is critical. But what happens when nodes suddenly stop communicating with each other?

Welcome to one of the most critical scenarios in RAC:

⚠️ Split Brain Syndrome

🔍 What is Split Brain?

In an Oracle RAC cluster, nodes communicate using a private interconnect.

👉 If this interconnect fails:

Nodes cannot see each other
Each node assumes others are down
Each continues processing independently

This leads to:

❌ Multiple “brains” operating simultaneously
❌ No coordination
❌ High risk of data corruption

⚡ Real Problem Explained

Imagine a 2-node RAC:

Node 1 updates a data block
Node 2 updates the same block
No communication between them

👉 Result:

💥 Data inconsistency / corruption

🧩 Why Does This Happen?

Split brain occurs when:

Private interconnect fails
Network heartbeat is lost
Nodes are still physically UP
Database instances continue running

Each node thinks:

“I am the only surviving node.”

🔁 Types of Heartbeats in RAC

Oracle uses two mechanisms to detect node health:

1️⃣ Network Heartbeat

Via private interconnect
Fast communication between nodes

2️⃣ Disk Heartbeat

Via Voting Disk
Backup mechanism when network fails

🗳️ Role of Voting Disk

The Voting Disk is the brain behind conflict resolution.

👉 Each node:

Writes its presence
Checks connectivity with others

🔥 In Split Brain Scenario:

Nodes form sub-clusters
Each group tries to claim majority
Voting disk decides:

✅ Which nodes survive
❌ Which nodes get evicted

⚖️ Who Wins?

👉 Majority rule applies

Example:

10-node RAC cluster
6 nodes can communicate
4 nodes isolated

👉 Result:

6-node group survives
4-node group gets evicted

🚫 Node Eviction – Who Does It?

The eviction is handled by:

👉 CSSD (Cluster Synchronization Services Daemon)

🔧 CSSD Responsibilities:

Monitor node health
Check heartbeats
Detect communication failure
Evict problematic nodes

⚙️ How CSSD Monitors Nodes

Mechanism	Purpose
Network Heartbeat	Interconnect communication
Disk Heartbeat	Voting disk verification

⚡ Node Eviction Process

When a node is unhealthy:

CSSD detects heartbeat failure
Voting disk validation occurs
Node is forcibly evicted
Node is usually rebooted automatically
Cluster reconfigures

🚨 Common Error

ORA-29740: evicted by instance

👉 Indicates:

Node eviction occurred
Cluster protection mechanism triggered

🧪 Real-World Scenario

Situation:

4-node RAC
Node 3 loses interconnect

What happens:

Node 1 detects issue
Voting disk confirms
Node 3 is evicted
Remaining nodes continue

🔄 Why Eviction is Important

Eviction is NOT a failure.

👉 It is a protection mechanism

Without eviction:

Multiple nodes update same data
Corruption occurs

With eviction:

✅ Data integrity is preserved

🧠 Key DBA Takeaways

Split brain is network-related issue
Always ensure:
- Redundant interconnect
- Stable network
Monitor:
- CSSD logs
- Clusterware alerts

🎯 Interview Questions