$ ssh admin@203.0.113.25
admin@203.0.113.25: Permission denied (publickey).

The instance is healthy. The security group is right. The disk is fine. You simply no longer hold the key that gets you in — the laptop was reimaged, the key was never committed anywhere, or it belonged to someone who left.

AWS cannot help you here, and that is by design. The fix is to make the instance add a key you do hold, on its next boot, using the one channel you still control: user data.

Same three parts as usual: how the lockout happens, what actually runs at boot, then the runbook.

Part 1 — How You Get Locked Out

The key pair model

When you launch an EC2 instance and pick a key pair, something less magical than it looks happens. AWS stores only the public half. At first boot, cloud-init reads that public key from the instance metadata service and writes it into ~/.ssh/authorized_keys for the distribution’s default user.

That is the entire mechanism. There is no agent, no callback, no key escrow.

The private half is generated once, offered to you once, and never stored by AWS. This is genuinely good design — it means an AWS compromise does not hand over SSH access to every instance in every account. It also means that when you lose that file, nobody can send it to you again, because nobody else ever had it.

Why the obvious fixes don’t work

Changing the key pair on the instance does nothing. The key pair association is metadata about the launch. Editing it does not touch authorized_keys on a running instance — the file was written once, at first boot, and has been an ordinary file on an ordinary disk ever since.

Creating an AMI and relaunching does not help either. The image captures the disk including the existing authorized_keys, so the new instance trusts exactly the keys the old one did.

The old way: swap the volume

For years the documented recovery was a volume transplant:

1. Stop the broken instance.
2. Detach its root EBS volume.
3. Attach that volume to a second instance you can log into.
4. Mount it, edit authorized_keys by hand, unmount.
5. Detach, reattach to the original as /dev/xvda, start it.

It works, and it still works when everything else fails. But it needs a second instance in the same availability zone, careful attention to device names, and a good ten minutes of clicking during which you can quite easily attach the wrong volume to the wrong machine.

The user data method does the same job in one reboot, and the machine edits its own file.

Part 2 — What Actually Runs at Boot

cloud-init, briefly

cloud-init is the service that turns a generic disk image into your instance: it sets the hostname, expands the root filesystem, configures users and SSH keys, and runs whatever you passed as user data. It runs in stages, and the stage a module runs in determines what already exists when it executes.

The subtlety is in that last box. User data is normally processed once per instance, keyed on the instance ID — cloud-init remembers it already ran and skips it. Pasting a plain #cloud-config with an SSH key into an existing instance therefore does nothing at all.

The recovery snippet works because it explicitly re-schedules the users-groups module into the final stage with a frequency of once, which makes cloud-init execute it on this boot regardless of what it did at launch.

Why the payload is MIME

The blob starts with Content-Type: multipart/mixed, which looks like overkill for a few lines of YAML. It is there because cloud-init accepts several user data formats — shell scripts, cloud-config, include files — and the MIME wrapper is how it tells them apart when you send more than one part. A single-part message works too, and the multipart form is what AWS documents, so it is the safest thing to paste.

Part 3 — The Runbook

The whole procedure is four steps. Read the caveats before you start, because step 1 involves stopping a production machine.

Before you begin

• Stopping is required. User data cannot be modified while an instance is running.
• The public IPv4 address will change unless the instance has an Elastic IP. Note it before you stop.
• Instance store volumes are wiped by a stop. If this instance has ephemeral local disks holding anything you care about, this method is not for you — use the volume-swap approach instead.
• The username must already exist, or match the distribution default: ec2-user on Amazon Linux, ubuntu on Ubuntu, admin on Debian, centos on CentOS.
• This is a Linux/cloud-init procedure. Windows uses EC2Launch and a different flow.

Step 0 — Have a key to install

If you do not already have one:

ssh-keygen -t ed25519 -f ~/.ssh/ec2-recovery -C "ec2 recovery"
cat ~/.ssh/ec2-recovery.pub

You need the public half for the next step. Nothing secret goes into user data.

Step 1 — Stop the instance

aws ec2 stop-instances --instance-ids i-0123456789abcdef0
aws ec2 wait instance-stopped --instance-ids i-0123456789abcdef0

Step 2 — Attach the recovery user data

Console: Actions → Instance settings → Edit user data. Or from the CLI, with the payload in a file:

aws ec2 modify-instance-attribute \
    --instance-id i-0123456789abcdef0 \
    --attribute userData --value file://recovery-user-data.txt

recovery-user-data.txt — replace the user name and the key with your own:

Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0

--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"

#cloud-config
cloud_final_modules:
- [users-groups, once]
users:
  - name: 'admin'
    ssh-authorized-keys:
     - 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDpiXpv9Gh5b00SmNoMgkN8NVZgTz4uPikiT7HDBgHGnniTCWmkzWI6o2SYOzPf2BV44ZeoIaFPNn2+P8wsJ4zo+AbmC2VgB4IXSjkYyZ/jTQV8EGKNYTJDWbOHmZZBnor2iIqGpDDPFtnPCqhSTPcRA0qyP7in0FBw+1Q2N/cyOypfzd7ko4CvD69QLjN1gxLNmZ5U9da/RuWFyk32kSAEW1tudk88/12LCRkHgkx7ep9Zjdg6W+hX2qb9e/yYUMYM1i4ZdfsVinv8bJYHiAXoePL2p6c9Y0dHG7iSzuP3Wtu3mENAjWbZATrzP6TJaU7U7o2a/dPQ24G0XY37+I50+lf/23kimSr3hgAnzayrKP6czX8n4xynGgnMRB9pUzqXeROH0+AD5ucjuLoDHW9Etw4SjzFJccS/pK3LCZlrCEszrHqYRINeiId5lmjpI1SzwYCA0zWv8Pb41VcjNLfVIaiIoNkOIzbOMVQneWFWBByaBybyzAxUsXJxXVH6bPOrK7Syy9m0508ESro+cPC5EYLV3x8K42SrT7oykB19LFE4/hZ/MLStpwqNKCww9kNJ8NncCSTykXJgMBIWl9kUc868id4tVYkTO6ZXEcRS7JhHkc47eX1TocOQWFtcsSzrUQcy9x/aQ7cKzx/yO3fByK/OzVZJ6KzgaTuXhxhsAQ== AWS 111122223333'

Two things people trip over:

• The users: key replaces the default user list for this boot. Name the account you actually want, and be aware that other users cloud-init would normally create are not created on this boot. Since you remove the user data immediately afterwards, that only affects the recovery boot.
• Overriding cloud_final_modules replaces the whole final-stage list, so modules like runcmd and scripts-user do not run on this boot either. For a recovery boot that is fine, and it is another reason not to leave this attached.

Step 3 — Start it and log in

aws ec2 start-instances --instance-ids i-0123456789abcdef0
aws ec2 wait instance-running --instance-ids i-0123456789abcdef0
aws ec2 describe-instances --instance-ids i-0123456789abcdef0 \
    --query 'Reservations[].Instances[].PublicIpAddress' --output text

Give cloud-init a moment after the instance reports running — the key is written a few seconds into boot, not at the instant the API says running.

$ ssh admin@203.0.113.25 -i ~/.ssh/ec2-recovery -o IdentitiesOnly=yes
The authenticity of host '203.0.113.25 (203.0.113.25)' can't be established.
ED25519 key fingerprint is SHA256:xNwem7G4KWa+rB+elk8B6MjCS5aWYtc7iGt/NioAmeA.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '203.0.113.25' (ED25519) to the list of known hosts.
Linux worker-node-1 6.1.0-32-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.129-1 (2025-03-06) x86_64

Last login: Sun Mar 23 14:48:52 2025 from 198.51.100.14
admin@worker-node-1:~$

-o IdentitiesOnly=yes is worth the habit. Without it, ssh offers every key your agent holds, and a server configured with a low MaxAuthTries will disconnect you before it reaches the one you actually passed with -i.

If it still refuses, the instance will tell you why:

sudo cat /var/log/cloud-init-output.log     # what the modules did
sudo cat /var/log/cloud-init.log            # the detail, including skips

A line saying the module was skipped because it had already run means the cloud_final_modules override did not take — usually a YAML indentation problem in the pasted block.

Step 4 — Remove the user data

Do not skip this. Stop the instance again and clear the attribute:

aws ec2 stop-instances --instance-ids i-0123456789abcdef0
aws ec2 wait instance-stopped --instance-ids i-0123456789abcdef0
aws ec2 modify-instance-attribute \
    --instance-id i-0123456789abcdef0 --user-data Value=
aws ec2 start-instances --instance-ids i-0123456789abcdef0

Three reasons this matters:

1. It re-applies on every boot, quietly reinstating that key even if someone later removes it from authorized_keys.
2. It suppresses the normal final-stage modules on every boot, which will confuse whoever debugs this machine next.
3. User data is readable from inside the instance by anyone who can reach the metadata service:

TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token \
    -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
    http://169.254.169.254/latest/user-data

A public key is not a secret, so this is not a credential leak. It is a signpost — it tells any process on that box exactly which key holds access, and that is information you were not obliged to publish.

Choosing the Right Method

The user data trick is not always the best option, and sometimes it is the only one.

The important column is the third one. Session Manager is the best answer to this problem and it is also the one you cannot reach for during the incident — either the agent and instance profile were already there, or they were not.

A security note worth internalising

Look again at what this procedure actually is: anyone who can modify user data can grant themselves root on that instance. No SSH key, no existing access, just one API call and a reboot.

That makes ec2:ModifyInstanceAttribute a privilege-escalation path, and it is very often granted casually — it looks like a boring configuration permission, and it is bundled into a lot of hand-rolled “developer” policies. If your IAM model assumes that shell access is controlled by SSH keys, that assumption is wrong wherever this permission is handed out.

Restrict it with a condition or a resource scope, and audit it the way you would audit the ability to ssh as root.

Not needing this again

• Install the SSM agent and give instances an instance profile. It is preinstalled on current Amazon Linux and Ubuntu AMIs, so this is frequently just the IAM role. It removes the entire class of problem, along with the need for port 22 and a bastion.
• Bake authorized keys into the AMI, or manage them with configuration management rather than relying on the launch-time key pair.
• Treat the launch key pair as a bootstrap credential, not the long-term access path. It is issued once and cannot be reissued, which makes it a poor thing to depend on a year later.
• Use an Elastic IP on anything you might need to stop, so recovery does not also change the address in everyone’s config.

Closing

The lockout feels like an AWS problem and it is not one. AWS never had your private key, which is exactly what you want from a cloud provider — the cost of that guarantee is that recovery has to come from somewhere else.

User data is that somewhere else: the one channel into a stopped instance that does not require already being inside it. Four steps, one reboot, and a note in your calendar to set up Session Manager so you never do it again.

References

• AWS re:Post — replace a lost key pair using user data
• cloud-init module reference
• Connect using EC2 Instance Connect
• AWS Systems Manager Session Manager