No description
Find a file
Adam Moussa c5839e23f4 fix(syslog-server): allow UDM tunnel source + pin collector private IP (INFRA-11)
Two durable fixes for the root causes that took the UniFi syslog feed dark
when the collector was recreated under IaC on 2026-06-09:

1. SecurityGroup: allow udp/tcp 514 from 192.168.0.0/24. The UDM gateways
   forward via syslog-ng / unifi-core sourced from the IPsec VTI tunnel
   inside-address (192.168.0.x), not their LAN IP, so the recreated SG
   silently denied all gateway-originated syslog at the ENI. (A LAN client
   keeps its 10.10.x source and was always allowed — which masked the issue.)

2. Pin the instance PrivateIpAddress to 10.20.10.221. The fleet forwards to
   this address over the site-to-site VPN; the recreation moved it .111->.221
   and orphaned every console's syslog target. Pinning keeps it stable across
   future replacements.

DEPLOY NOTES:
- (1) was applied live out-of-band to restore service; delete that live SG
  rule immediately before `cdk deploy` or CFN errors on InvalidPermission.Duplicate.
- (2) forces an instance replacement (cdk diff: PrivateIpAddress requires
  replacement; EIP re-associates). Deploy in a maintenance window — ~2-3 min
  log-collection gap while user-data re-runs; ends at the same .221 so no
  console reconfig needed.
2026-06-09 18:07:03 -04:00
.github/workflows feat: syslog-server under IaC (INFRA-12) 2026-06-09 13:51:17 -04:00
bin feat: syslog-server under IaC (INFRA-12) 2026-06-09 13:51:17 -04:00
lib fix(syslog-server): allow UDM tunnel source + pin collector private IP (INFRA-11) 2026-06-09 18:07:03 -04:00
.gitignore feat: syslog-server under IaC (INFRA-12) 2026-06-09 13:51:17 -04:00
cdk.context.json feat: syslog-server under IaC (INFRA-12) 2026-06-09 13:51:17 -04:00
cdk.json feat: syslog-server under IaC (INFRA-12) 2026-06-09 13:51:17 -04:00
package-lock.json feat: syslog-server under IaC (INFRA-12) 2026-06-09 13:51:17 -04:00
package.json feat: syslog-server under IaC (INFRA-12) 2026-06-09 13:51:17 -04:00
README.md feat: syslog-server under IaC (INFRA-12) 2026-06-09 13:51:17 -04:00
tsconfig.json feat: syslog-server under IaC (INFRA-12) 2026-06-09 13:51:17 -04:00

syslog-server

CDK stack for the syslog-server EC2 collector: receives remote syslog (UDP/TCP 514) from the office UniFi fleet over its Elastic IP and ships it to the unifi-syslog CloudWatch Logs group via the CloudWatch agent.

Brought under IaC for INFRA-12 (AWS audit L-6). Previously a console/CLI instance with no drift detection.

Architecture

office UniFi devices ──syslog/514──▶ EIP 184.72.154.32 ──▶ EC2 (rsyslog)
                                                              │
                                              /var/log/remote/<host>/*.log
                                                              │
                                              CloudWatch agent ──▶ unifi-syslog (90d)
                                                                        │
                                                       Syslog-NoIncomingLogs alarm ──▶ site-alerts
Resource Value
Instance syslog-server, t4g.nano, Amazon Linux 2023 (arm64), 30 GiB encrypted gp3
Subnet subnet-0eea820effe1b3ae5 (public, us-east-1a, vpc-0d3d4b67bd0cf8a68)
Elastic IP 184.72.154.32 (eipalloc-006bdefc9802f3285) — unmanaged, re-associated by ID
Security group syslog-server — 514 tcp/udp + 22 from office IPs + VPC/VPN CIDRs; 2055/2056 udp reserved (netflow/sflow)
IAM role syslog-server-role — AmazonSSMManagedInstanceCore + CloudWatchAgentServerPolicy
Log group unifi-syslog (90-day retention) — created/retained by the CW agent, not a CFN resource (holds history; see stack comment)
Alarm Syslog-NoIncomingLogs — IncomingLogEvents Sum < 1 over 2×1-day, ALARM-only → site-alerts

Access

SSM Session Manager (no key pair). SSH 22 is open from office/VPC for break-glass only.

Deploy

CI/CD via the org reusable workflows (ci-typescript-cdk.yaml, cd-cdk.yaml); merges to main deploy through the githubdeploy-syslog-server OIDC role. No Docker assets, so a local cdk deploy is also safe.

npm ci
npm run diff
npm run deploy

Notes

  • EIP is unmanaged. CloudFormation associates it but never releases it, so the public forwarding target survives any instance replacement.
  • AMI is pinned in cdk.context.json (cachedInContext). An AL2023 AMI change forces instance replacement — refresh deliberately with cdk context --reset <ami key> && cdk synth.
  • To widen device coverage of the forwarded syslog feed, see INFRA-11 (UniFi controller remote-logging config).