fix(syslog-server): rotate /var/log/remote so the disk can't fill (INFRA-11)

The collector's remote-syslog spool had no rotation, so each gateway's
/var/log/remote/<host>/<host>.log grew unbounded. Low risk at the old
~109 events/day, but the gateways now forward ~60k/day. CloudWatch (90d)
is the system of record; the local files are only a CW-agent spool, so
keep a short 7-day compressed window. copytruncate keeps rsyslog's open
dynaFile handles valid (truncate in place).

Applied live already; this codifies it so an instance replacement keeps it
(mirrors the existing netflow-retention timer). Deploying this user-data
change forces an instance replacement (userDataCausesReplacement) — the EIP
re-associates and the forwarding target is unchanged, so do it in a window.
This commit is contained in:
Adam Moussa 2026-06-10 12:59:36 -04:00
parent 97bc751782
commit 47f857a5b4

View file

@ -115,6 +115,22 @@ export class SyslogServerStack extends cdk.Stack {
"systemctl enable rsyslog",
"systemctl restart rsyslog",
"",
"# ── Rotate /var/log/remote so it can't grow unbounded ──",
"# CloudWatch (90d) is the system of record; these local files are just a",
"# spool for the CW agent, so keep only a short window. copytruncate keeps",
"# rsyslog's open dynaFile handles valid (truncate in place, same inode).",
"cat > /etc/logrotate.d/remote-syslog <<'EOF'",
"/var/log/remote/*/*.log {",
" daily",
" rotate 7",
" compress",
" delaycompress",
" missingok",
" notifempty",
" copytruncate",
"}",
"EOF",
"",
"# ── CloudWatch agent: ship /var/log/remote/**/*.log to unifi-syslog ──",
"dnf install -y amazon-cloudwatch-agent",
"cat > /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json <<'EOF'",