When RMAN Goes Silent: Rescuing a Stalled ZDM Migration From a Half-Open TCP Connection

The Moment You Realize Something Is Wrong

There's a specific feeling every DBA knows: the one where you check on a long-running job that should be well underway by now, and the log timestamp hasn't moved in hours. You refresh. You check again, still nothing.

That's how this story started.

We were three days into a 16 TB Oracle physical migration from Exadata Cloud@Customer to Oracle Database@Azure using Zero Downtime Migration (ZDM 21.5). Everything had been running smoothly. RMAN was chugging along, restoring datafiles across a cross-region WAN. Then, silence.

The ZDM job status said EXECUTING. The current phase said ZDM_RESTORE_TGT. The RMAN process was still alive on the target. Datafiles were still being written… except they weren't.

For 35 hours.

This blog covers what happened, why it happened, and, more importantly, how to recover without losing 15 TB of restored data and starting from scratch.

The Setup

Source: On-premises Exadata Cloud@Customer, 16 TB database (PRODDB_ONP) in primary role

Target: Oracle Database@Azure ExaDB-D in another region (PRODDB_AZ)

Method: ZDM 21.5 physical online migration with DATA_TRANSFER_MEDIUM=DIRECT and ZDM_RMAN_DIRECT_METHOD=RESTORE_FROM_SERVICE

RMAN channels: 10 parallel channels

Network path: Enterprise WAN with firewalls between source datacenter and Azure region

The migration had already restored approximately 15 TB. RMAN was working on the tail end with a few large datafiles remaining. Then activity died.

What We Saw

The ZDM job log had gone quiet. But before assuming ZDM was hung, we needed to understand what was actually happening at every layer.

First Check: Is ZDM Alive?

ZDM believed it was still working. That was misleading.

Second Check: Is RMAN Alive?

On the target host, we mapped out the process tree:

RMAN was alive. Ten channel processes were alive. Everything looked normal at the process level.

Third Check: What Are the Channels Actually Doing?

This is where the picture changed. We queried the target's session view:

The output was ominous.

127,642 seconds = 35 hours and 27 minutes.

Six RMAN channels had been waiting on remote db file read for over 35 hours. The wait event remote db file read is what RMAN reports when a channel is reading data from a source database over SQL*Net — it's the source-server-side equivalent of physical reads for a RESTORE FROM SERVICE operation.

The four other channels were sitting on SQL*Net message from client — the RMAN coordinator's polite way of saying "I finished my work, waiting for the boss to give me more." But the boss couldn't hand out new work until the stuck channels either completed or failed.

The Smoking Gun

Here's where things got interesting. We connected to the source database and asked the reverse question: how many sessions are currently connected from the target?

The source had zero sessions from the target. But the target had six sessions actively waiting to receive data from the source. That's not a paradox; it's a definitive signature of a well-known problem.

What Actually Happened: Half-Open TCP Connections

Somewhere in the network path between the source and target, an intermediate device (a firewall, a NAT gateway, some stateful network appliance) silently terminated the TCP connections between RMAN and the source database.

Here's how it played out:

1. RMAN opened 10 TCP connections to the source database's listener.

2. Restore work flowed normally for hours.

3. At some point, a network device decided one or more connections had been idle too long, their state was purged from its tracking table, or some other opaque decision was made.

4. The connection was silently dropped.

5. Only one side of the conversation noticed.

The source database noticed because it had SQLNET.EXPIRE_TIME=10 set. This makes the Oracle server send periodic dead-connection-detection probes to clients. When those probes failed (because the connection was actually dead), the source cleanly terminated its side and released the session, which is why the source query showed zero sessions.

The target's RMAN process, however, was blocked on a socket read, waiting for data that would never come. And the target had no equivalent client-side detection mechanism kicking in aggressively enough. So it just waited. And waited… for 35 hours. And would have waited forever.

This is called a "half-open" TCP connection. One side is closed, the other side thinks it's still open. The application on the alive side is stuck waiting for the dead side to say something.

The Linux TCP keepalive mechanism was supposed to detect this eventually, but the defaults are cruel:

That's 2 hours of idle time before keepalive even starts probing, then 9 probes at 75-second intervals. Detection time in the best case: about 2 hours and 11 minutes. In the worst case, where an intermediate device is happily acknowledging keepalive probes on behalf of the vanished endpoint, never. However, we never wanted to change these parameters on either EXADATA system without proper consent from Oracle Support.

For a WAN migration running over an enterprise firewall, that's essentially useless.

The Tempting-But-Wrong Response

The obvious impulse is to abort the ZDM job and start over. But when you're 15 TB into a 16 TB restore, that impulse costs you three days of migration time. So before going nuclear, we needed to understand what safer options existed.

Two ideas worth trying before abort:

1. zdmcli suspend job: ZDM's built-in mechanism to gracefully pause a running job.
2. Manually killing the RMAN process: forcing RMAN to die so ZDM detects failure, then using resume to retry only the failed phase.

The critical insight is that in both cases, the already-restored data on the target remains intact. The target's ASM has all the datafiles that were successfully written. Only RMAN's in-memory state and the stuck connections need cleanup.

But before killing anything, we needed absolute certainty about which RMAN process to kill. On a shared Exadata host with multiple databases, other RMAN processes could be running backups. Killing the wrong one would be very bad.

Identifying the Exact Process to Kill (Paranoid Mode)

If we were going to escalate to killing RMAN, we needed forensic-level certainty. Below is the identification approach.

Trace the Process Ancestry from ZDM

This shows the exact tree spawned by ZDM. The RMAN process will be at the bottom, unambiguously parented by ZDM's sudo/su chain.

Cross-Reference with the Database Session View

The SPID values from this query are the OS process IDs of the RMAN channel processes. If they match the children of the RMAN parent PID from pstree, you have bulletproof confirmation.

Verify No Other RMAN Processes Exist

If only one RMAN process exists on the host (as was the case for us), the identification is trivial. If multiple exist (typical for hosts hosting backup jobs for other databases), each needs to be traced back to its parent to confirm which one is ZDM's.

We spent 15 minutes verifying the correct PID before touching anything. That's cheap insurance.

The Recovery: What Actually Worked

Step 1: Try zdmcli Suspend First

This is where the story turned. We ran:

Within about 10 minutes, the job status changed:

What happened under the hood: ZDM sent a signal that eventually made RMAN terminate. When RMAN died, its stuck socket reads unblocked (with errors). The stuck connections were torn down cleanly. Everything paused at a checkpoint.

Step 2: Resume the Job

Here's where it got beautiful. ZDM's resume didn't restart the restore from scratch. Instead, it triggered an incremental restore to catch up on the delta since the initial snapshot SCN, established fresh TCP connections that benefited from the new keepalive settings, and completed sync in a fraction of the time a full restore would have taken.

Watching the target session activity confirmed things were working correctly:

Compare those wait times to what we saw before. Instead of 127,000+ seconds stuck on one event, we saw 0-25 seconds cycling through work. Channels were actively transferring data. This is what a healthy RMAN restore looks like.

Within a few hours, source and target were in sync. DG apply lag: zero. The 35-hour stall was recovered without losing 15 TB of prior work.

Why Suspend/Resume Worked (And Why This Matters)

The zdmcli suspend command proved more capable than expected. It's not just a checkpoint marker; it actually forces the current RMAN operation to terminate cleanly. This is exactly what was needed to break the deadlock.

Combined with resume, which understands the incremental restore case, this pattern gives you a real recovery option that sits between "wait forever" and "abort and lose everything."

If suspend hadn't worked, the fallback plan was to manually kill the RMAN process (identified via the paranoid procedure above), let ZDM detect the failure, then resume from the failed phase- same eventual outcome, more manual work.

The Lessons

1. The Oracle Server-Side Setting Is Only Half the Fix

SQLNET.EXPIRE_TIME on the source (server side) helped the source clean up dead sessions. But there's no direct client-side equivalent that RMAN uses. For the target (client) to detect dead connections, you need aggressive OS TCP keepalive settings.

Set both:

• On both source and target: SQLNET.EXPIRE_TIME=10 (or lower) in sqlnet.ora, then reload the listener

• On both source and target OS: aggressive tcp_keepalive_* settings via sysctl

2. Enterprise Firewalls Are Hostile to Long-Lived TCP Connections

Firewalls maintain connection state tables with finite entries and timeouts. Long-lived Oracle SQL*Net connections that transfer data in bursts (like RMAN restore of a large section) can look "idle" enough to be culled. Worse, some devices continue ACKing keepalive probes on behalf of dropped connections, defeating the detection mechanism entirely.

Get your network team involved early. Ask specifically:

• What is the TCP idle timeout on firewalls in the path?

• Can it be extended for the duration of the migration window?

• Are there any devices proxying/state-preserving connections that might interfere with keepalive detection?

3. The Right Diagnostic Query Catches the Pattern Immediately

The single most useful query when RMAN restore appears stuck:

Long remote db file read waits on target + zero corresponding sessions on source = half-open TCP. That's the signature.

4. Suspend Before Abort

If you find yourself with a stuck ZDM restore, remember the recovery ladder:

1. Diagnose: confirm it's actually stuck and understand why.

2. Try zdmcli suspend job: surprisingly effective for breaking stuck states.

3. zdmcli resume job: takes advantage of incremental restore.

4. Kill RMAN manually (if suspend fails): but identify the process with paranoid care.

5. Abort: only as a last resort, after everything else has failed.

The difference between suspend/resume and abort/restart on a 16 TB migration is roughly three days of clock time. That's worth trying suspend even if you're not sure it'll work.

5. Section Size Is Not the Root Cause, But it Helps

We set ZDM_RMAN_SECTION_SIZE=16G (down from default 64G). Smaller sections mean shorter individual RMAN operations, so any single stall affects less work. But this is mitigation, not a fix. The underlying network problem needs to be addressed at the network layer.

The Human Part

Watching a 35-hour stall on a critical production migration is genuinely stressful. There's a temptation to do something, anything, to stop the sinking feeling.

Resist that temptation.

Take an hour to diagnose the problem. Cross-reference every data point you can. Verify assumptions before acting. The extra hour spent understanding what's happening is trivial compared to the days you could lose making the wrong recovery choice.

When we finally acted, we knew exactly:

• Which RMAN PID to kill if needed

• What state the data was in

• What suspend would do (in theory)

• What our fallback plans were

• Why the problem happened

That confidence made the recovery decision easy. Suspend, wait, resume, monitor. The system did the rest.

What I'd Do Differently

If I were starting this migration over from scratch:

1. Set OS TCP keepalive on all nodes before beginning any migration, not after the first stall.

   cat > /etc/sysctl.d/99-tcp-keepalive.conf <<'EOF' net.ipv4.tcp_keepalive_time = 120 net.ipv4.tcp_keepalive_intvl = 30 net.ipv4.tcp_keepalive_probes = 5 EOF sysctl -p /etc/sysctl.d/99-tcp-keepalive.conf

    

2. Engage the network team upfront. Get firewall timeouts documented and extended before RMAN starts.

3. Set ZDM_RMAN_SECTION_SIZE=16G or 8G by default. It reduces the blast radius of any single stall.

4. Test suspend/resume in a lower environment, so you know how it behaves when you need it.

5. Have the diagnostic queries ready. Script them, don't type them from memory at 2 AM.

Closing Thoughts

This particular failure mode isn't a ZDM bug or an Oracle bug. It's the natural result of running long-lived TCP connections across enterprise WAN infrastructure that was never designed to hold connections open for days at a time. It's a networking problem masquerading as a database problem.

The good news is that both Oracle and ZDM give you the tools to detect it, understand it, and recover. You just need to know where to look and what to trust.

Suspend/resume saved this migration. Adequate keepalive settings would have prevented the stall entirely. And network-layer changes would eliminate the root cause forever.

FIf you're planning a large cross-region Oracle migration over enterprise WAN infrastructure, do your future self a favor: don't rely on Oracle's server-side dead-connection detection alone. Configure aggressive OS TCP keepalives. Talk to your network team. Set section sizes conservatively. And know your recovery commands cold, before you need them.

For more information contact us today and one of our experts will be in touch.