Ubuntu Server Security Hardening

Platform: Pluralsight Hands-On Lab Environment: Ubuntu 24.04 (AWS EC2)

---

Overview §

This lab simulates a real-world pre-audit scenario: I’m part of an operations team at a regional logistics company, and our inventory server is about to face an external security audit. The server was intentionally left in a weak baseline state - SSH accepted password logins with no user restrictions, no firewall was active, there was no privileged action trail, and AppArmor was running but not yet tuned.

My goal was to harden the system across three layers:

1. SSH & brute-force defense - key-based auth for an auditor account, a locked-down sshd drop-in config, and a production-grade fail2ban jail
2. Firewall & audit logging - a deny-by-default UFW firewall, auditd rules targeting sensitive files and privileged commands, and multi-source event correlation
3. Mandatory access control - inspecting and toggling AppArmor profiles for nginx, observing enforcement in both complain and enforce modes, and practicing the aa-logprof refinement workflow

---

Objective 1 - Harden SSH Access and Configure fail2ban §

Baseline Environment §

The server was running Ubuntu 24.04 on AWS EC2 in a weakened state typical of a freshly provisioned instance. SSH was functional for the ubuntu account using key-based auth, but several critical controls were missing or inactive:

• fail2ban was installed but stopped - no brute-force protection in place
• UFW was disabled - all inbound ports open with no filtering
• auditd had no custom rules - no trail of privileged actions
• An auditor account existed with no SSH key configured
• AppArmor was running but profiles were untuned

Before making any changes, I audited the current state of each service to establish a clear before/after baseline.

---

Generating an SSH Key Pair for the Auditor User §

The auditor account existed but had no SSH key, only a password. The goal here was to move it to key-based authentication, which eliminates password guessing as an attack vector entirely.

I generated an ED25519 key pair (Ed25519 is faster and more secure than RSA for this use case) in my home directory so I could review it before deploying it:

ssh-keygen -t ed25519 -f ~/auditor_key -N '' -C 'auditor lab key'

Generating public/private ed25519 key pair.
Your identification has been saved in /home/ubuntu/auditor_key
Your public key has been saved in /home/ubuntu/auditor_key.pub
The key fingerprint is:
SHA256:6gVjBT7X+y5YZNprr4hFLXm7Vyt8vSuBfvNpZ/TDXHI auditor lab key
The key's randomart image is:
+--[ED25519 256]--+
|      .          |
|     . . .       |
|      o o .      |
|       + oo.     |
|      + S=+ .    |
|     . =.oo+ .o E|
|      . oooo..+*o|
|     . +..++++o=*|
|      o ..o=+o=*=|
+----[SHA256]-----+

I then created the .ssh directory with the exact ownership and permissions SSH requires, and installed the public key:

sudo install -d -o auditor -g auditor -m 700 /home/auditor/.ssh
sudo install -o auditor -g auditor -m 600 ~/auditor_key.pub /home/auditor/.ssh/authorized_keys

^install

Using install instead of mkdir + cp is a clean one-step way to set ownership and permissions atomically, something worth doing in scripts where you don’t want partial states.

Finally, I validated it worked with a live test:

ssh -i ~/auditor_key -o StrictHostKeyChecking=accept-new auditor@localhost 'whoami && hostname'

Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
auditor
ip-172-31-24-10

Key-based login confirmed, no password prompt.

---

Hardening the SSHD Configuration §

Rather than editing the main sshd_config, I used the drop-in directory /etc/ssh/sshd_config.d/. Any file placed here overrides the defaults without touching the main config, which is the Ubuntu-recommended approach and makes auditing config changes easier.

sudo tee /etc/ssh/sshd_config.d/70-hardening.conf > /dev/null <<'EOF'
PermitRootLogin no
PasswordAuthentication no
MaxAuthTries 3
AllowUsers ubuntu auditor
LoginGraceTime 30
EOF

What each directive does and why:

Directive	Value	Purpose
PermitRootLogin	no	Root logins are never needed over SSH; use sudo from a named account
PasswordAuthentication	no	Eliminates brute-force password attacks entirely
MaxAuthTries	3	Cuts off multi-attempt key fishing before fail2ban even needs to act
AllowUsers	ubuntu auditor	Explicit allow-list - any account not named here is denied, even with a valid key
LoginGraceTime	30	Reduces the unauthenticated connection window from the 120s default

Before reloading, I validated the syntax to avoid locking myself out:

sudo sshd -t && echo "config ok"

config ok

Then reloaded and confirmed the live runtime matched my config:

sudo systemctl reload ssh
sudo sshd -T | grep -E '^(permitrootlogin|passwordauthentication|maxauthtries|allowusers|logingracetime)'

logingracetime 30
maxauthtries 3
permitrootlogin no
passwordauthentication no
allowusers ubuntu
allowusers auditor

Note: sshd -T outputs each AllowUsers entry on its own line, both lines together represent the single AllowUsers ubuntu auditor directive.

---

Verifying Allow and Deny Behavior §

I tested both the allow path and the deny path explicitly. Good security configs should be proven, not assumed.

Allow - auditor with key (should succeed):

ssh -i ~/auditor_key -o StrictHostKeyChecking=accept-new auditor@localhost 'echo "auditor login ok"'

auditor login ok

Deny - root (should be blocked by AllowUsers):

ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new root@localhost 'whoami' || echo 'root login denied as expected'

root@localhost: Permission denied (publickey).
root login denied as expected

Deny - stranger (unlisted account, should be blocked):

sudo useradd -m -s /bin/bash stranger
ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new stranger@localhost 'whoami' || echo 'stranger login denied as expected'
sudo userdel -r stranger

stranger@localhost: Permission denied (publickey).
stranger login denied as expected

I then confirmed the denials were logged:

sudo tail -n 30 /var/log/auth.log | grep -E '(Accepted|Failed|not allowed|not listed)'

2026-05-25T15:44:16.574418+00:00 ip-172-31-24-10 sshd[4580]: Accepted publickey for auditor from 127.0.0.1 port 49464 ssh2: ED25519 SHA256:6gVjBT7X...
2026-05-25T15:45:30.477221+00:00 ip-172-31-24-10 sshd[4667]: User root from 127.0.0.1 not allowed because not listed in AllowUsers
2026-05-25T15:46:13.803257+00:00 ip-172-31-24-10 sshd[4680]: User stranger from 127.0.0.1 not allowed because not listed in AllowUsers

Both root and stranger were rejected by the AllowUsers check — as confirmed by the auth.log message not allowed because not listed in AllowUsers. Both PermitRootLogin no and AllowUsers would have blocked root independently; the log shows which one reported the denial.

---

Configuring fail2ban for SSH Defense §

fail2ban watches log sources and automatically bans IPs that exceed a failure threshold. Even with key-only auth, it’s a useful defense-in-depth layer, it can catch automated scanners that hammer connection attempts.

sudo tee /etc/fail2ban/jail.d/sshd.local > /dev/null <<'EOF'
[DEFAULT]
ignoreip = 127.0.0.1/8 ::1
bantime  = 3600
findtime = 600
maxretry = 3
 
[sshd]
enabled = true
port    = ssh
logpath = %(sshd_log)s
backend = systemd
EOF

These are production values: 3 failures within 10 minutes triggers a 1-hour ban. ignoreip exempts localhost so monitoring jobs and scheduled tasks are never accidentally banned, in a real environment, I’d also add the jump host IPs here.

sudo systemctl enable --now fail2ban
sudo fail2ban-client status

Status
|- Number of jail:      1
`- Jail list:   sshd

I confirmed the runtime values matched the policy:

sudo fail2ban-client get sshd maxretry   # → 3
sudo fail2ban-client get sshd findtime   # → 600
sudo fail2ban-client get sshd bantime    # → 3600

sudo fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- Journal matches:  _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

Armed and quiet, exactly what you want before any incidents.

---

Objective 2 - Enable UFW, Configure auditd, and Correlate Security Events §

Enabling UFW with a Deny-by-Default Inbound Policy §

A host firewall provides the first line of defense against unsolicited inbound connections. UFW (Uncomplicated Firewall) is a frontend for iptables/nftables that makes rule management straightforward.

I started by confirming UFW was inactive, then built the ruleset from scratch:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 80/tcp       # inventory dashboard
sudo ufw limit 22/tcp       # SSH with rate limiting
sudo ufw --force enable

sudo ufw default deny incoming Sets the default rule for all inbound traffic to drop it. sudo ufw default allow outgoing Lets the server initiate outbound connections freely. sudo ufw allow 80/tcp Punches a hole for HTTP traffic on port 80. sudo ufw limit 22/tcp Allows SSH, but with built-in rate limiting.

sudo ufw status verbose

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
80/tcp                     ALLOW IN    Anywhere
22/tcp                     LIMIT IN    Anywhere
80/tcp (v6)                ALLOW IN    Anywhere (v6)
22/tcp (v6)                LIMIT IN    Anywhere (v6)

Why ufw limit on SSH? limit rejects connections from a source IP if it makes six or more attempts in the last 30 seconds. This works at the firewall level, complementing the fail2ban application-layer defense set up in Objective 1.

I then verified persistence by reloading the firewall and confirming the rules were unchanged:

sudo ufw reload   # → Firewall reloaded
sudo ufw status verbose   # same rule table as above

UFW stores rules in /etc/ufw/, so they survive both a reload and a full reboot, there’s no window where the host is unprotected.

---

Defining Audit Rules for Sensitive Events §

auditd is the Linux kernel audit subsystem. By writing targeted watch rules, I can generate a forensic trail of exactly the events auditors care about: account file modifications, privileged command execution, and authentication log writes.

sudo tee /etc/audit/rules.d/security-hardening.rules > /dev/null <<'EOF'
# Watch sensitive account files for any write or attribute change
-w /etc/passwd -p wa -k account-changes
-w /etc/shadow -p wa -k account-changes
 
# Watch the SSH auth log for any write
-w /var/log/auth.log -p wa -k auth-log
 
# Audit every execution of sudo by tagging it with the 'privileged' key
-a always,exit -F arch=b64 -S execve -F path=/usr/bin/sudo -F key=privileged
EOF

The -k flag tags each event with a searchable key, which makes querying with ausearch straightforward later.

I loaded the rules into the running kernel:

sudo augenrules --load
sudo auditctl -l

-w /etc/passwd -p wa -k account-changes
-w /etc/shadow -p wa -k account-changes
-w /var/log/auth.log -p wa -k auth-log
-a always,exit -F arch=b64 -S execve -F path=/usr/bin/sudo -F key=privileged

All four rules active in the running audit subsystem.

---

Triggering and Querying Audit Events §

To prove the rules actually work, I generated one event of each type and then queried for it.

Trigger an account-changes event by modifying the auditor account’s comment field:

sudo usermod -c "Auditor Account" auditor

Trigger a privileged event by running a sudo command:

sudo ls /root   # → snap

Trigger an auth-log event with a deliberate failed login:

su - root -c whoami < /dev/null || echo 'failed su recorded'

Password: su: Authentication failure
failed su recorded

Query the account-changes trail:

sudo ausearch -k account-changes -ts recent | tail -n 25

time->Mon May 25 16:03:33 2026
type=PATH msg=audit(...): item=0 name="/etc/passwd" inode=33756 ... nametype=NORMAL
type=SYSCALL msg=audit(...): ... comm="usermod" exe="/usr/sbin/usermod" ... key="account-changes"
----
time->Mon May 25 16:03:33 2026
type=PATH msg=audit(...): item=0 name="/etc/shadow" inode=3086 ... nametype=NORMAL
type=SYSCALL msg=audit(...): ... comm="usermod" exe="/usr/sbin/usermod" ... key="account-changes"

The audit record shows the exact timestamp, the file touched (/etc/passwd, /etc/shadow), the process (usermod), and the calling user, exactly what an auditor would need.

Query the privileged sudo trail:

sudo ausearch -k privileged -ts recent | tail -n 25

time->Mon May 25 16:03:41 2026
type=EXECVE msg=audit(...): argc=3 a0="sudo" a1="ls" a2="/root"
type=SYSCALL msg=audit(...): ... exe="/usr/bin/sudo" ... key="privileged"

The EXECVE record captures the full command line (sudo ls /root), not just the binary, useful for reconstructing exactly what was run.

---

Real-Time Monitoring with journalctl §

In production you’d run journalctl -f in a second terminal to watch events live. Since this lab uses a single browser terminal, I backgrounded it with a 30-second timeout:

sudo timeout 30 journalctl -f _COMM=sudo --no-pager &
sudo ls /root

[1] 6045
May 25 16:09:33 ip-172-31-24-10 sudo[6049]:   ubuntu : TTY=pts/1 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/ls /root
May 25 16:09:33 ip-172-31-24-10 sudo[6049]: pam_unix(sudo:session): session opened for user root(uid=0) by ubuntu(uid=1000)

[1]+  Exit 124                sudo timeout 30 journalctl -f _COMM=sudo --no-pager

Exit 124 is expected, that’s timeout’s exit code when it terminates a process after the time limit.

For incident reconstruction after the fact, the retrospective query:

sudo journalctl _COMM=sudo --since "10 minutes ago" --no-pager | tail -n 20

---

Cross-Referencing Audit Sources §

A key audit skill is verifying that the same event appears across multiple independent log sources. Here I cross-referenced auditd, journalctl, and /var/log/auth.log for the same sudo events:

sudo tail -n 30 /var/log/auth.log | grep -E '(sudo|su:)'

2026-05-25T16:09:33.998807+00:00 ip-172-31-24-10 sudo: ubuntu : TTY=pts/1 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/ls /root
2026-05-25T16:09:34.000130+00:00 ip-172-31-24-10 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by ubuntu(uid=1000)

The same ls /root event appears in auditd (tagged privileged), journalctl (via _COMM=sudo), and auth.log. This overlap is deliberate, if a record exists in one source but not the others, that discrepancy is either a tampered log or a configuration gap, both of which are reportable audit findings.

I also generated a full summary of all audit activity since boot:

sudo aureport --summary

Summary Report
======================
Range of time in logs: 05/25/26 13:06:34 - 05/25/26 16:07:48
Number of changes in configuration: 263
Number of changes to accounts, groups, or roles: 12
Number of logins: 2
Number of failed logins: 2
Number of failed authentications: 2
Number of users: 4
Number of keys: 3
Number of events: 1358

The 12 account changes and 2 failed logins line up exactly with the test activity I ran.

---

Objective 3 - Manage AppArmor Profiles and Review Denial Logs §

Inspecting the AppArmor Posture §

AppArmor is a mandatory access control (MAC) system that confines processes to a defined set of file and capability permissions. Unlike discretionary access control (which relies on file ownership and chmod), MAC rules apply even to root processes.

We already have a pre-staged a custom nginx profile in enforce mode that explicitly denies writes to /etc/. I started by understanding what was already there.

sudo aa-status --json | python3 -c "
import json,sys; d=json.load(sys.stdin)
print('profiles loaded:', sum(len(v) for v in d['profiles'].values()))
print('processes confined:', sum(len(v) for v in d['processes'].values()))"

profiles loaded: 1295
processes confined: 4

Then I read the actual nginx profile to understand its rules:

sudo cat /etc/apparmor.d/usr.sbin.nginx

#include <tunables/global>

/usr/sbin/nginx {
    #include <abstractions/base>
    #include <abstractions/nis>

    capability dac_override,
    capability dac_read_search,
    capability net_bind_service,
    capability setgid,
    capability setuid,

    /etc/group r,
    /etc/nginx/** r,
    /etc/passwd r,
    /etc/ssl/openssl.cnf r,

    /run/nginx.pid rw,
    /run/nginx/** rw,

    /usr/sbin/nginx mr,

    /var/log/nginx/*.log w,
    /var/www/** r,

    # Explicitly DENY writes to /etc/ to make denials visible in dmesg
    deny /etc/** w,
}

^2b4ffe

The critical line is deny /etc/** w,, an explicit deny rule. This means even if the profile mode is switched to complain, this specific path will still block writes. Explicit denies override the mode setting by design.

sudo aa-status | grep -E '(nginx|profiles are in)'

51 profiles are in enforce mode.
   /usr/sbin/nginx
6 profiles are in complain mode.

nginx confirmed in enforce mode.

---

Switching to Complain Mode and Probing the Audit Trail §

Complain mode logs policy violations without blocking them, useful for profiling a new service or testing what a profile would block before enforcing it.

sudo aa-complain /etc/apparmor.d/usr.sbin.nginx

Setting /etc/apparmor.d/usr.sbin.nginx to complain mode.

I first ran the rogue demo script as unconfined root to establish a baseline, outside any profile, it can write anywhere:

sudo /usr/local/bin/rogue-demo.sh

rogue-demo.sh

#!/bin/bash
# rogue-demo.sh: attempts a forbidden write to /etc/ so learners can see
# AppArmor enforcement and auditd capture in action.
set +e
echo "rogue process running as $(whoami) at $(date -u +%FT%TZ)" > /etc/rogue-marker 2>/dev/null
if [ -f /etc/rogue-marker ]; then
    echo "WRITE SUCCEEDED: /etc/rogue-marker exists"
else
    echo "WRITE FAILED: AppArmor or filesystem permissions blocked the write"
fi

WRITE SUCCEEDED: /etc/rogue-marker exists

After cleaning up the marker file, I ran the same script under the nginx AppArmor profile in complain mode:

sudo rm -f /etc/rogue-marker
sudo aa-exec -p /usr/sbin/nginx -- /usr/local/bin/rogue-demo.sh

aa-exec is telling the kernel: “run this next command under nginx’s AppArmor profile.”

/usr/local/bin/rogue-demo.sh: line 5: /etc/rogue-marker: Permission denied
WRITE FAILED: AppArmor or filesystem permissions blocked the write

Key insight: The write was blocked even in complain mode. This is because the profile contains an explicit deny /etc/** w, rule. Explicit deny rules always block, they are not softened by complain mode. Without that explicit deny, complain mode would have logged the violation and allowed the write to proceed.

Checking the audit log for AVC records:

sudo ausearch -m AVC -ts recent | tail -n 30

<no matches>

This is actually a valid result on many Ubuntu 24.04 systems, the denial worked (the script reported WRITE FAILED), but the kernel on this AMI didn’t emit a type=AVC record. The events are captured in the audit log under a different record type, and aa-logprof will read them in the next section. In production, checking multiple sources (dmesg | grep apparmor, journalctl -k | grep apparmor, raw grep apparmor /var/log/audit/audit.log) covers all the bases.

---

Switching to Enforce Mode and Watching the Block §

sudo aa-enforce /etc/apparmor.d/usr.sbin.nginx

Setting /etc/apparmor.d/usr.sbin.nginx to enforce mode.

sudo aa-status | grep -E '/usr/sbin/nginx' | head -3

   /usr/sbin/nginx
   /usr/sbin/nginx//null-/usr/bin/date
   /usr/sbin/nginx//null-/usr/bin/whoami

Now enforce mode is stricter than complain: it blocks anything not explicitly allowed, not just the explicit deny rules. Running the rogue script again:

sudo aa-exec -p /usr/sbin/nginx -- /usr/local/bin/rogue-demo.sh

/bin/bash: /usr/local/bin/rogue-demo.sh: Permission denied

In enforce mode the shell itself can’t even read the script, because /usr/local/bin/rogue-demo.sh isn’t in the profile’s allowed paths. The error message changed: in complain mode the shell could execute the script but the write was blocked; in enforce mode execution is blocked before the script even starts.

ls -l /etc/rogue-marker 2>&1

ls: cannot access '/etc/rogue-marker': No such file or directory

No file created, the block worked at the kernel level.

---

Refining the Profile with aa-logprof §

Issue Encountered §

aa-logprof reads the audit log and presents interactive prompts for each AppArmor event recorded, asking whether to allow, deny, or abstract each access. Since ausearch -m AVC returned <no matches>, I wasn’t sure if the tool would find the events.

Resolution §

Despite the empty ausearch result, aa-logprof was still able to read the events from /var/log/audit/audit.log directly (it uses its own parser, not ausearch). The tool launched and presented prompts as expected.

Takeaway: ausearch -m AVC returning no results does not mean the audit log is empty, it means the events were stored under a different record type. aa-logprof reads the raw log and handles this transparently.

The decision table I applied to each prompt:

Rule	When the prompt shows…	Key to press
1	Anything mentioning /usr/local/bin/rogue-demo.sh	D (Deny)
2	A numbered include option (e.g. [1 - include <abstractions/consoles>])	1, then A
3	An Execute: line for a helper like /usr/bin/whoami or /usr/bin/date	I (Inherit)
4	A Path: line for any other system file	A (Allow)
5	A Network Family: line with no numbered include	A (Allow)

After working through all prompts, I saved with S. The tool wrote the refinements back to the profile file.

I then confirmed the critical deny rule survived the refinement session:

sudo grep -c 'deny /etc/' /etc/apparmor.d/usr.sbin.nginx

1

And ran the rogue script one final time to prove enforce mode still held:

sudo aa-exec -p /usr/sbin/nginx -- /usr/local/bin/rogue-demo.sh

/bin/bash: /usr/local/bin/rogue-demo.sh: Permission denied

The security boundary was intact after refinement, this is the point of aa-logprof: add the access paths a legitimate workload actually needs, without weakening the explicit deny rules that define its security boundary.

---

Skills Demonstrated §

Tool / Technique	What I Did
ssh-keygen, install	Generated ED25519 key pair and deployed it with correct permissions
sshd_config.d drop-in	Hardened SSH without touching the main config file
sshd -t, sshd -T	Validated config syntax and confirmed live runtime settings
fail2ban	Configured a production-grade SSH jail with maxretry, findtime, bantime
ufw	Deployed a deny-by-default firewall with rate-limited SSH
auditd / augenrules	Wrote and loaded kernel audit rules targeting sensitive files and sudo
ausearch, aureport	Queried audit logs by key, time range, and event type
journalctl	Watched events live and reconstructed them retrospectively
aa-status, aa-complain, aa-enforce	Inspected and toggled AppArmor profile modes
aa-exec	Tested a process under a specific AppArmor profile without restarting the service
aa-logprof	Interactively refined a profile against real audit events without weakening deny rules

---

Key Takeaways §

• Defense in depth is layered deliberately. SSH key auth, fail2ban, and ufw limit each operate at a different layer and cover each other’s gaps.
• Explicit deny rules in AppArmor override the profile mode. A deny path blocks in both complain and enforce modes, this is a feature, not a limitation.
• ausearch -m AVC returning <no matches> is not a failure. Ubuntu 24.04 AMIs may route AppArmor events under different record types; the denial still occurs, and aa-logprof will still find the events.
• Log correlation is the core audit skill. The same privileged event appearing in auditd, journalctl, and auth.log simultaneously is what gives you confidence the audit trail is trustworthy, and lets you spot gaps if one source is missing.