syslog-server

Vector collector that receives UniFi All Traffic syslog, CEF, and IPFIX over office IPsec, parses to JSON, and writes to S3 through Kinesis Data Firehose for 90-day Athena search.

Deploy path (PLAT-78 / PLAT-206): HCP Terraform in seahaven-prod (011934824531), workspace syslog-server-prod. CDK CD in mgmt is retired.

Architecture

Locust UDM 10.30  ──SD-WAN mesh──▶ Ronkonkoma UDM 10.10
                                         │
                         IPsec UDP 514 + IPFIX 2055/2056
                                         │
                                         ▼
                              Vector t4g.small (10.40)
                                         │
                                         ▼
                               Kinesis Data Firehose
                                         │
                                         ▼
                         S3 syslog-server-unifi-logs-* (90d)
                                         │
                                         ▼
                               Glue unifi + Athena
                                         │
                    Syslog-NoIncomingRecords ──▶ site-alerts
                    Syslog-FirehoseDeliveryFailed ──▶ site-alerts

Resource	Value
Account / region	seahaven-prod 011934824531 / us-east-1
HCP workspace	syslog-server-prod (project seahaven-prod; VCS main; working dir terraform; trigger terraform/**)
HCP plan/apply roles	hcptf-syslog-server-plan / hcptf-syslog-server
Instance	syslog-server, t4g.small, Amazon Linux 2023 (arm64), 20 GiB encrypted gp3, SSM only
VPC	dedicated 10.40.0.0/16, public subnet 10.40.10.0/24 (egress IP for Vector install / Firehose / SSM; not a syslog target)
IPsec	VGW + customer gateway on Ronkonkoma WAN 47.21.61.4; static routes 10.10.0.0/16 and 10.30.0.0/16
UniFi target	instance private IP:514 (syslog + CEF) and :2055/:2056 (IPFIX). No public 514.
Security group	syslog-server — UDP/TCP 514 and UDP 2055/2056 from 10.10.0.0/16 and 10.30.0.0/16 only
Instance IAM	/tf-managed/syslog-server-role with syslog-server-instance-boundary; AmazonSSMManagedInstanceCore + firehose:PutRecordBatch
Store	S3 syslog-server-unifi-logs-011934824531, prefixes format=cef|iptables|netflow/dt=YYYY-MM-DD/, 90-day expire
Query	Glue database unifi (cef, iptables, netflow) and Athena workgroup syslog-server
Alarms	Syslog-NoIncomingRecords, Syslog-FirehoseDeliveryFailed, EC2-StatusCheck-syslog-server, EC2-StatusCheckSystem-syslog-server-recover → site-alerts

The office IPsec tunnel that already reaches mgmt 10.20.0.0/16 does not land in this VPC. UniFi needs a second site-to-site peer for 10.40.0.0/16. Do not re-home this workspace in mgmt.

Access

SSM Session Manager. SSH 22 is closed. There is no Elastic IP forwarding target.

IAM bootstrap window

Instance-boundary document changes and apply-role inline policy changes need the hcptf-bootstrap window (DenySelfMutation plus deny on iam:CreatePolicyVersion). Sequence:

1. From seahaven-org-baseline: scripts/create-hcptf-bootstrap-roles.sh --account prod --allow-workspace syslog-server-prod
2. Point workspace TFC_AWS_APPLY_ROLE_ARN / TFC_AWS_PLAN_ROLE_ARN at hcptf-bootstrap / hcptf-bootstrap-plan. Keep TFC_AWS_PROVIDER_AUTH=true. Never TFC_AWS_RUN_ROLE_ARN.
3. One manual apply as bootstrap creates/updates the scoped hcptf-* inline policies and the instance boundary.
4. Retarget TFC_AWS_* to hcptf-syslog-server / hcptf-syslog-server-plan. Re-run the create script with no --allow-workspace.
5. Manual apply as the scoped role for the rest of the stack (instance, Firehose, S3, Glue, Athena, VGW). Auto-apply stays off until soak.

HCP outputs to copy: private_ip, vpn_connection_id, vpn_tunnel1_address, vpn_tunnel2_address, bucket_name, firehose_name, athena_workgroup. Read PSKs with terraform output -raw vpn_tunnel1_preshared_key after apply. Do not commit them.

AMI is pinned in var.ami_id. An AMI or user-data change replaces the instance. The box is stateless; archives live in S3.

Syslog-NoIncomingRecords defaults treat_missing_data to notBreaching until UniFi delivers over IPsec. After Firehose IncomingRecords is non-zero, set no_logs_treat_missing_data=breaching.

UniFi cutover

Do this after the HCP apply, not before. Apply drops public 514 and the CloudWatch unifi-syslog log group. Point UniFi immediately.

1. Ronkonkoma site-to-site VPN to the AWS tunnel addresses from HCP outputs. Remote network 10.40.0.0/16. Local network 10.10.0.0/16. IKEv2, AES-256, SHA-256, DH14 matches typical AWS defaults. Use the Terraform PSK outputs. This is a second child SA alongside the existing mgmt 10.20 tunnel. Do not replace the mgmt tunnel.
2. Locust SD-WAN mesh must already route AWS VPC CIDRs via Ronkonkoma (same as jumpbox SSH). Add 10.40.0.0/16 if it is missing.
3. Both controllers, Settings → CyberSecure / System Log:
   • SIEM server = collector private IP, port 514, UDP
   • Flow Logging = All Traffic
   • Activity Logging SIEM contents include firewall
   • Control Plane CEF to the same IP:514
4. Enable syslog on WAN and inter-VLAN firewall rules, or All Traffic stays silent.
5. NetFlow/IPFIX: Ronkonkoma → UDP 2055, Locust → UDP 2056, same private IP.
6. Prove the path: send a test syslog from Ronkonkoma; Athena SELECT on iptables and cef; confirm format=netflow objects for 2055/2056; confirm site-alerts does not fire while traffic is present.
7. Flip off any remaining public EIP / mgmt collector only after Firehose IncomingRecords is non-zero. PLAT-78 still owns deleting the mgmt syslog-server CloudFormation stack after soak.

Vector treats payloads as untrusted text. It parses fields and does not shell out. IPFIX datagrams are archived as base64 JSON with a site tag (Vector has no released IPFIX decoder).

Documentation

The canonical map of Sea Haven's AWS infrastructure lives in Confluence.

• AWS Architecture Map (Confluence, IT space, page 1540098)
• Syslog Server (page 67141633)

Tracked as PLAT-206. PLAT-78 remains the HCP move plus mgmt stack delete after this soak.