# syslog-server ![Terraform](https://img.shields.io/badge/Terraform-844FBA?logo=terraform&logoColor=white) ![AWS](https://img.shields.io/badge/AWS-FF9900?logo=amazonaws&logoColor=white) ![CI](https://github.com/Sea-Haven-Industries/syslog-server/actions/workflows/ci.yaml/badge.svg) Vector collector that receives UniFi All Traffic syslog, CEF, and IPFIX over office IPsec, parses to JSON, and writes to S3 through Kinesis Data Firehose for 90-day Athena search. Deploy path (PLAT-78 / PLAT-206): HCP Terraform in seahaven-prod (`011934824531`), workspace `syslog-server-prod`. CDK CD in mgmt is retired. ## Architecture ``` Locust UDM 10.30 ──SD-WAN mesh──▶ Ronkonkoma UDM 10.10 │ IPsec UDP 514 + IPFIX 2055/2056 │ ▼ Vector t4g.small (10.40) │ ▼ Kinesis Data Firehose │ ▼ S3 syslog-server-unifi-logs-* (90d) │ ▼ Glue unifi + Athena │ Syslog-NoIncomingRecords ──▶ site-alerts Syslog-FirehoseDeliveryFailed ──▶ site-alerts ``` | Resource | Value | |---|---| | Account / region | seahaven-prod `011934824531` / us-east-1 | | HCP workspace | `syslog-server-prod` (project `seahaven-prod`; VCS `main`; working dir `terraform`; trigger `terraform/**`) | | HCP plan/apply roles | `hcptf-syslog-server-plan` / `hcptf-syslog-server` | | Instance | `syslog-server`, t4g.small, Amazon Linux 2023 (arm64), 20 GiB encrypted gp3, SSM only | | VPC | dedicated `10.40.0.0/16`, public subnet `10.40.10.0/24` (egress IP for Vector install / Firehose / SSM; not a syslog target) | | IPsec | VGW + customer gateway on Ronkonkoma WAN `47.21.61.4`; static routes `10.10.0.0/16` and `10.30.0.0/16` | | UniFi target | instance **private IP**:514 (syslog + CEF) and :2055/:2056 (IPFIX). No public 514. | | Security group | `syslog-server` — UDP/TCP 514 and UDP 2055/2056 from `10.10.0.0/16` and `10.30.0.0/16` only | | Instance IAM | `/tf-managed/syslog-server-role` with `syslog-server-instance-boundary`; `AmazonSSMManagedInstanceCore` + `firehose:PutRecordBatch` | | Store | S3 `syslog-server-unifi-logs-011934824531`, prefixes `format=cef\|iptables\|netflow/dt=YYYY-MM-DD/`, 90-day expire | | Query | Glue database `unifi` (cef, iptables, netflow) and Athena workgroup `syslog-server` | | Alarms | `Syslog-NoIncomingRecords`, `Syslog-FirehoseDeliveryFailed`, `EC2-StatusCheck-syslog-server`, `EC2-StatusCheckSystem-syslog-server-recover` → `site-alerts` | The office IPsec tunnel that already reaches mgmt `10.20.0.0/16` does **not** land in this VPC. UniFi needs a second site-to-site peer for `10.40.0.0/16`. Do not re-home this workspace in mgmt. ## Access SSM Session Manager. SSH 22 is closed. There is no Elastic IP forwarding target. ## IAM bootstrap window Instance-boundary document changes and apply-role inline policy changes need the hcptf-bootstrap window (`DenySelfMutation` plus deny on `iam:CreatePolicyVersion`). Sequence: 1. From `seahaven-org-baseline`: `scripts/create-hcptf-bootstrap-roles.sh --account prod --allow-workspace syslog-server-prod` 2. Point workspace `TFC_AWS_APPLY_ROLE_ARN` / `TFC_AWS_PLAN_ROLE_ARN` at `hcptf-bootstrap` / `hcptf-bootstrap-plan`. Keep `TFC_AWS_PROVIDER_AUTH=true`. Never `TFC_AWS_RUN_ROLE_ARN`. 3. One manual apply as bootstrap creates/updates the scoped `hcptf-*` inline policies and the instance boundary. 4. Retarget `TFC_AWS_*` to `hcptf-syslog-server` / `hcptf-syslog-server-plan`. Re-run the create script with no `--allow-workspace`. 5. Manual apply as the scoped role for the rest of the stack (instance, Firehose, S3, Glue, Athena, VGW). Auto-apply stays off until soak. HCP outputs to copy: `private_ip`, `vpn_connection_id`, `vpn_tunnel1_address`, `vpn_tunnel2_address`, `bucket_name`, `firehose_name`, `athena_workgroup`. Read PSKs with `terraform output -raw vpn_tunnel1_preshared_key` after apply. Do not commit them. AMI is pinned in `var.ami_id`. An AMI or user-data change replaces the instance. The box is stateless; archives live in S3. `Syslog-NoIncomingRecords` defaults `treat_missing_data` to `notBreaching` until UniFi delivers over IPsec. After Firehose `IncomingRecords` is non-zero, set `no_logs_treat_missing_data=breaching`. ## UniFi cutover Do this after the HCP apply, not before. Apply drops public 514 and the CloudWatch `unifi-syslog` log group. Point UniFi immediately. 1. **Ronkonkoma site-to-site VPN** to the AWS tunnel addresses from HCP outputs. Remote network `10.40.0.0/16`. Local network `10.10.0.0/16`. IKEv2, AES-256, SHA-256, DH14 matches typical AWS defaults. Use the Terraform PSK outputs. This is a second child SA alongside the existing mgmt `10.20` tunnel. Do not replace the mgmt tunnel. 2. **Locust SD-WAN mesh** must already route AWS VPC CIDRs via Ronkonkoma (same as jumpbox SSH). Add `10.40.0.0/16` if it is missing. 3. Both controllers, **Settings → CyberSecure / System Log**: - SIEM server = collector **private IP**, port **514**, UDP - Flow Logging = **All Traffic** - Activity Logging SIEM contents include firewall - Control Plane **CEF** to the same IP:514 4. Enable syslog on WAN and inter-VLAN firewall rules, or All Traffic stays silent. 5. NetFlow/IPFIX: Ronkonkoma → UDP **2055**, Locust → UDP **2056**, same private IP. 6. Prove the path: send a test syslog from Ronkonkoma; Athena `SELECT` on `iptables` and `cef`; confirm `format=netflow` objects for 2055/2056; confirm `site-alerts` does not fire while traffic is present. 7. Flip off any remaining public EIP / mgmt collector **only after** Firehose `IncomingRecords` is non-zero. PLAT-78 still owns deleting the mgmt `syslog-server` CloudFormation stack after soak. Vector treats payloads as untrusted text. It parses fields and does not shell out. IPFIX datagrams are archived as base64 JSON with a site tag (Vector has no released IPFIX decoder). ## Documentation The canonical map of Sea Haven's AWS infrastructure lives in Confluence. - **[AWS Architecture Map](https://seahaven.atlassian.net/wiki/spaces/IT/pages/1540098)** (Confluence, IT space, page 1540098) - **[Syslog Server](https://seahaven.atlassian.net/wiki/spaces/IT/pages/67141633)** (page 67141633) Tracked as **PLAT-206**. PLAT-78 remains the HCP move plus mgmt stack delete after this soak.