Why is AWS Transform MGN or Elastic Disaster Recovery replication stalled on my Linux or Windows server?

Short description

After you install the AWS Replication Agent, AWS Transform MGN or Elastic Disaster Recovery runs initiation steps before it begins continual replication. If a step can't complete, then you receive the "Data replication stalled" error message that identifies the step that failed.

To resolve a stalled installation, complete the following steps.

Note: If the Pair or Connect step fails, then check the agent logs on the source server to identify errors. For Linux, check /var/lib/aws-replication-agent/agent.log.0, and for Windows, check C:\Program Files (x86)\AWS Replication Agent\agent.log.0.

Resolution

Note: In the following resolution, use a test Amazon Elastic Compute Cloud (Amazon EC2) instance. The Amazon EC2 instance must be in same subnet as your AWS Transform MGN replication settings or Elastic Disaster Recovery replication settings.

Troubleshoot network connectivity issues

For replication to work, you must allow the required traffic on TCP ports 443 and 1500.

Important: Don't apply SSL interception or deep packet inspection (DPI) to traffic between the source servers, the replication servers, and the AWS Transform MGN or Elastic Disaster Recovery endpoints.

Test TCP port 443

Your configuration must allow traffic on port 443 from the source and replication servers to the following endpoints:

• AWS Transform MGN or Elastic Disaster Recovery
• Amazon Simple Storage Service (Amazon S3)
• Amazon EC2

To test network connectivity for AWS Transform MGN, run the following commands from the replication server's subnet based on your operating system (OS).

Windows:

tnc mgn.us-east-1.amazonaws.com -port 443
tnc s3.us-east-1.amazonaws.com -port 443
tnc ec2.us-east-1.amazonaws.com -port 443

Note: Replace us-east-1 with your AWS Region. For Elastic Disaster Recovery, replace mgn with drs.

Linux:

curl -vvv https://mgn.us-east-1.amazonaws.com/
curl -vvv https://s3.us-east-1.amazonaws.com/
curl -vvv https://ec2.us-east-1.amazonaws.com/

Note: Replace us-east-1 with your Region. For Elastic Disaster Recovery, replace mgn with drs.

Test port 1500 connectivity

Your configuration must allow access from the source server to the replication server to allow block-level data transfer.

To test TCP reachability and the SSL handshake on port 1500, run the following commands based on your OS.

Windows:

tnc replication-server-IP -port 1500
netstat -ano | findstr "1500"
openssl s_client -connect replication-server-IP:1500

Note: Replace replication-server-IP with your replication server's IP address. To download openssl for Windows, see Win32/Win64 OpenSSL on the Shining Light Productions website. Download software only from a source that your organization approves.

Linux:

netcat -vz replication-server-IP 1500
telnet replication-server-IP 1500
netstat -an | grep "1500"
openssl s_client -connect replication-server-IP:1500

Note: Replace replication-server-IP with your replication server's IP address.

Check your configuration

To confirm that your configuration allows the required access, take the following actions:

• Confirm that the subnet in your replication settings still exists.
• Confirm that your source and replication server security groups and network access control lists (network ACLs) allow outbound traffic on port 443 to the required endpoints.
• Confirm that your network ACLs allow inbound traffic on the ephemeral port.
• If the TCP test for port 1500 passes but the openssl command fails, then check whether a firewall or DPI blocked SSL on port 1500.
• Confirm that your replication server security group allows inbound and outbound TCP traffic between the replication and source server on port 1500.
• Make sure that an antivirus or security software doesn't block the required traffic.
• Confirm that the virtual private cloud (VPC) can resolve the required endpoints. If you use custom DNS, then check the Dynamic Host Configuration Protocol (DHCP) option set. Also, check your VPC settings to confirm that enableDnsSupport and enableDnsHostnames are set to True, and that PrivateDnsEnabled is True on interface VPC endpoints.
• If you use AWS Network Firewall, then add the replication server subnet CIDRs to the allowlist.
• If the replication server subnet is public, then confirm that you activated Create public IP.
  Note: To find this setting, check Data routing in the replication settings for AWS Transform MGN or Elastic Disaster Recovery.

To troubleshoot an unreachable Amazon S3 endpoint, confirm that you associated the Amazon S3 gateway endpoint with the staging subnet's route table. Also, make sure that the VPC endpoint policy allows the required buckets, including the Amazon Linux 2023 (AL2023) repository bucket.

For the other endpoints, confirm that the endpoint policy allows access to the resources and that the security group allows inbound traffic on port 443. Also, check the staging subnet's route table to confirm that they route to the correct path, such as an internet gateway, NAT gateway, or VPC endpoint.

After you update your configuration, rerun the connectivity test to check whether you can now connect.

For more information, see Network requirements for AWS Transform MGN or Elastic Disaster Recovery network setting preparations.

Check for permission denials

Check your AWS CloudTrail Event history to search for the event name of the step that failed, such as CreateSecurityGroup, RunInstances, CreateVolume, or AttachVolume. Check for error codes such as "UnauthorizedOperation" or "AccessDenied". For all steps as of Authenticate with service, you can also filter by the replication server's instance ID. Use the denial reason to identify the policy that blocked the call. To resolve this issue, update your service control policy (SCP) to remove the deny that blocks access. Also, make sure that your permissions boundaries allow the required access.

If you use a customer managed AWS Key Management Service (AWS KMS) key for encryption, then check the AWS KMS key policy. Make sure that the policy doesn't block AWS Transform MGN or Elastic Disaster Recovery, and allows the required users.

Check your IAM permissions

The replication server assumes an AWS Identity and Access Management (IAM) role. If the role is missing, has the wrong policy, or has a restrictive permission boundary, then replication stalls.

To troubleshoot this issue, take the following actions:

• In the Amazon EC2 console, confirm that the replication server's IAM role exists and has the required AWS managed policy. For AWS Transform MGN, attach AWSApplicationMigrationReplicationServerPolicy. For Elastic Disaster Recovery, attach AWSElasticDisasterRecoveryReplicationServerPolicy

• Confirm that the role's trust relationship allows ec2.amazonaws.com to assume the role. Example policy:

  {
   "Version": "2012-10-17",
   "Statement": [
    {
     "Effect": "Allow",
     "Principal": { "Service": "ec2.amazonaws.com" },
     "Action": "sts:AssumeRole"
    }
   ]
  }
  

• Make sure that the role's permissions don't block permissions from the AWS managed policy.

Make sure that the AWS Replication Agent is running

If you receive the "Agent not seen" error message, then the service isn't receiving communication from the AWS Replication Agent. To troubleshoot this issue, confirm that the source server is booted, running, and has a running OS.

Then, check whether the AWS Replication Agent is running. Run the following command based on your OS.

Windows:

get-service | where-object name -like "*AWSR*"

Linux:

sudo systemctl status aws-replication

If the AWS Replication Agent isn't running, then run the following commands based on your OS to restart the agent.

Windows:

Start-Service AwsReplicationService

Linux:

sudo systemctl start aws-replication

Make sure that the agent can connect on port 443. For instructions, see Test TCP port 443.

If the agent is running and can connect on port 443, but you still get the "Agent not seen" error message, then the agent is missing on the server. To resolve this issue, reinstall the AWS Replication Agent on the source server. For instructions, see Installing the AWS Replication Agent for AWS Transform MGN or Installation instructions for the AWS Replication Agent for Elastic Disaster Recovery.

Important: Save a copy of the agent.log.0 file before you reinstall. A reinstall can start a new initial sync.

If you took the preceding actions and replication is still stalled, then contact AWS Support. In your support case, include the agent log, the name of the failed step, and your Region.

Related information

Replication questions (AWS Transform MGN)

Common replication errors (AWS Transform MGN)

Replication errors: agent communication (Elastic Disaster Recovery)

Why is my AWS Transform MGN or Elastic Disaster Recovery replication process stuck at 100% with the "Finalizing Initial Sync" message?