In this article14 sections
If SSH to your dedicated server stopped working after a firewall or configuration change, the server is almost certainly still running, and you can get back in through its IPMI console. IPMI is a separate management port with its own address that shows you the server’s screen and keyboard in a browser, exactly as if you were standing in front of it, whether or not the operating system’s network is working. Log in there, undo the change that locked you out, test, and SSH comes back. No reinstall is needed, and nothing on the disks is touched.
This guide covers the five causes behind nearly every lockout on Ubuntu and Debian servers, the exact commands to fix each from the console, what to do if you do not know a password to log in with, and four habits that stop it happening again.
What IPMI is, and why it still works
A dedicated server contains a small second computer called the baseboard management controller. It has its own network port, its own IP address and its own login, and it runs even when the main server is switched off. Through its web page you can open a remote console (also called KVM over IP), switch the power off and on, and attach a disk image to boot from. Because it does not pass through the server’s own network settings, firewall or SSH service, none of your mistakes there can lock you out of it.
Step 1: Open the console and log in
Sign in to the IPMI address from your server’s welcome details, find Remote Control or Console, and launch the HTML5 console. You will see a text login prompt. Log in with a user that has a password. SSH keys do not work here: the console is a keyboard and screen, not SSH.
If you do not know any password because you only ever used keys, skip to “If you cannot log in at the console” below.
Step 2: Find out what is wrong
Run these four commands. One of them will show the problem.
sudo systemctl status ssh --no-pager | head -5
sudo sshd -t
sudo ufw status
ip -br addr| What you see | Cause | Go to |
|---|---|---|
ssh is failed, or sshd -t prints an error |
A mistake in the SSH configuration | Fix A |
ufw is active and port 22 is not in the list |
The firewall is blocking SSH | Fix B |
| Everything looks right, but only your address is refused | Fail2ban banned you | Fix C |
| The interface has no IP address, or the wrong one | A network configuration error | Fix D |
| Commands fail with “No space left on device” | The disk is full | Fix E |
Fix A: A broken SSH configuration
sshd -t checks the configuration and names the file and line that is wrong:
/etc/ssh/sshd_config.d/10-custom.conf line 4: Bad configuration option: PermitRootLognOpen that file with sudo nano, correct or remove the line, and run sudo sshd -t again until it prints nothing. Then start the service:
sudo systemctl restart sshIf you changed the SSH port, check that the firewall allows the new port, not just the old one. That combination, a new port in the config and the old port in the firewall, is the single most common lockout.
Fix B: The firewall is blocking SSH
sudo ufw allow 22/tcp
sudo ufw status numberedUse your own port number if SSH is not on 22. If the rules are in a state you no longer understand, switch the firewall off, get back in over SSH, and rebuild the rules calmly:
sudo ufw disableTurn it back on as soon as the rules are right. A dedicated server with a public address and no firewall is being probed within minutes.
Fix C: Fail2ban banned your address
A few mistyped passwords, or a script that retried too quickly, are enough.
sudo fail2ban-client status sshd
sudo fail2ban-client set sshd unbanip 203.0.113.45Replace the address with your own public IP. To stop it happening again from your office, add that address to ignoreip in /etc/fail2ban/jail.local.
Fix D: A network configuration error
On Ubuntu the network is described in a file under /etc/netplan/. YAML is strict about indentation, and one misplaced space leaves the server with no address after the next reboot. Correct the file, then apply it with:
sudo netplan trynetplan try applies the new settings and waits two minutes for you to press Enter. If you do not, because the change cut you off, it puts the old settings back by itself. Use it every time, including over SSH, in place of netplan apply.
Fix E: The disk is full
A full disk stops logins in odd ways, because nothing can write a session or a log entry.
df -h
sudo journalctl --vacuum-size=200M
sudo apt clean
sudo du -xh / --max-depth=2 2>/dev/null | sort -rh | head -15The first command shows which filesystem is at 100%, the next two free space safely, and the last one shows where the space went. Log files and old backups are the usual answer.
If you cannot log in at the console
When every account uses SSH keys only, there is no password to type. You can still set one by starting the server in a minimal mode from the console. This needs a reboot, so the server’s services stop for a few minutes.
- In the IPMI page, choose Power, then Reset, and watch the console.
- When the GRUB boot menu appears, press
eto edit the first entry. If no menu appears, holdShiftor pressEscjust after the hardware screen. - Find the line that starts with
linux, go to its end, and addinit=/bin/bash. - Press
Ctrl+Xto boot. You arrive at a root prompt with no password asked. - Make the disk writable, set a password for your user, flush it to disk, and restart:
mount -o remount,rw / passwd yourusername sync reboot -f
The server boots normally, and you can now log in at the console with that password and continue from step 2. This works for anyone who reaches the console, which is exactly why the IPMI login itself must be protected.
Warning. If the disks are encrypted, or the boot menu is password-protected, this method does not apply. Boot a rescue image through the IPMI’s virtual media instead, or ask your provider’s support to help.
Four habits that prevent the next lockout
- Keep a second SSH session open while you change SSH, the firewall or the network. Test the change by opening a third. If it fails, the open session lets you undo it.
- Run
sudo sshd -tbefore every restart of SSH. A config with an error will not start, and the test takes a second. - Allow before you enable.
sudo ufw allow OpenSSHfirst,sudo ufw enablesecond, never the other way round. - Schedule your own rescue before a risky firewall change, and cancel it when the change works (install the tool once with
sudo apt install at):echo "ufw disable" | sudo at now + 10 minutes # make the change, confirm you can still log in, then: sudo atrm $(sudo atq | cut -f1)
The same audit that catches these problems on a new server is worth repeating after any change: see the 15-minute server audit, which applies to dedicated servers as much as to a VPS.
Protect the IPMI itself
- Change its default password on day one, and use one you do not use anywhere else.
- Do not leave it reachable by the whole internet if your provider lets you restrict it to your own addresses or a VPN.
- Give a user with a strong password sudo rights on the server, and store that password in a password manager. It is your way in at the console when keys are no use.
Common mistakes to avoid
- Reinstalling the server. It is almost never necessary, and it destroys the data a five-minute console session would have saved.
- Rebooting first. A reboot fixes none of these causes, and it turns an unsaved network change into a permanent one.
- Trying to paste an SSH key into the console. The console needs a password login.
- Leaving the firewall off after the rescue. Fix the rules and switch it back on.
- Fixing it and not writing down what happened. The same change will be made again in six months by someone who does not remember.
Dedicated servers with console access
Every SiteHarbour dedicated server comes with full root access and an IPMI console, so a mistake in a firewall rule or an SSH setting is something you can undo yourself in minutes, at any hour. Servers can be self-managed or fully managed, and our engineers are available around the clock if you would rather have someone else look.
Frequently asked questions
Is the IPMI console the same as SSH?
No. SSH is a service running inside the operating system. IPMI is separate hardware that shows the server’s screen, so it works when the operating system’s network or SSH does not.
Will I lose data by using it?
No. You are typing commands on the server as usual. Only the password reset method involves a reboot, and it changes nothing except the password.
The console shows a black screen. What now?
Press a key: the screen may only be blanked. If it stays black, the server may be hung or off. Check the power state in the IPMI page and use Reset only as a last resort.
Can I use this on a VPS?
A VPS has no IPMI, because it has no hardware of its own. Many providers offer an equivalent web console for virtual servers; if yours does not, their support can make the fix or boot the server into rescue mode for you.
How do I find my own IP address to unban it?
From the computer that is locked out, run curl -s ifconfig.me or search for “what is my IP”. Use that address in the Fail2ban command.
The short version: a lockout is a configuration problem, not a dead server. Open the IPMI console, run four commands to find which of five causes it is, apply the matching fix, and make sshd -t, netplan try and a second open session part of every future change.