mirror of
https://github.com/Sea-Haven-Industries/syslog-server.git
synced 2026-09-30 21:43:15 +00:00
2b12aba50e
Replace the public rsyslog-to-CloudWatch collector with Vector over a prod 10.40 IPsec VGW, Firehose, 90-day S3, Glue, and Athena.
132 lines
6.7 KiB
Markdown
132 lines
6.7 KiB
Markdown
# syslog-server
|
|
|
|
|
|
|
|
|
|
|
|
Vector collector that receives UniFi All Traffic syslog, CEF, and IPFIX over
|
|
office IPsec, parses to JSON, and writes to S3 through Kinesis Data Firehose
|
|
for 90-day Athena search.
|
|
|
|
Deploy path (PLAT-78 / PLAT-206): HCP Terraform in seahaven-prod
|
|
(`011934824531`), workspace `syslog-server-prod`. CDK CD in mgmt is retired.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Locust UDM 10.30 ──SD-WAN mesh──▶ Ronkonkoma UDM 10.10
|
|
│
|
|
IPsec UDP 514 + IPFIX 2055/2056
|
|
│
|
|
▼
|
|
Vector t4g.small (10.40)
|
|
│
|
|
▼
|
|
Kinesis Data Firehose
|
|
│
|
|
▼
|
|
S3 syslog-server-unifi-logs-* (90d)
|
|
│
|
|
▼
|
|
Glue unifi + Athena
|
|
│
|
|
Syslog-NoIncomingRecords ──▶ site-alerts
|
|
Syslog-FirehoseDeliveryFailed ──▶ site-alerts
|
|
```
|
|
|
|
| Resource | Value |
|
|
|---|---|
|
|
| Account / region | seahaven-prod `011934824531` / us-east-1 |
|
|
| HCP workspace | `syslog-server-prod` (project `seahaven-prod`; VCS `main`; working dir `terraform`; trigger `terraform/**`) |
|
|
| HCP plan/apply roles | `hcptf-syslog-server-plan` / `hcptf-syslog-server` |
|
|
| Instance | `syslog-server`, t4g.small, Amazon Linux 2023 (arm64), 20 GiB encrypted gp3, SSM only |
|
|
| VPC | dedicated `10.40.0.0/16`, public subnet `10.40.10.0/24` (egress IP for Vector install / Firehose / SSM; not a syslog target) |
|
|
| IPsec | VGW + customer gateway on Ronkonkoma WAN `47.21.61.4`; static routes `10.10.0.0/16` and `10.30.0.0/16` |
|
|
| UniFi target | instance **private IP**:514 (syslog + CEF) and :2055/:2056 (IPFIX). No public 514. |
|
|
| Security group | `syslog-server` — UDP/TCP 514 and UDP 2055/2056 from `10.10.0.0/16` and `10.30.0.0/16` only |
|
|
| Instance IAM | `/tf-managed/syslog-server-role` with `syslog-server-instance-boundary`; `AmazonSSMManagedInstanceCore` + `firehose:PutRecordBatch` |
|
|
| Store | S3 `syslog-server-unifi-logs-011934824531`, prefixes `format=cef\|iptables\|netflow/dt=YYYY-MM-DD/`, 90-day expire |
|
|
| Query | Glue database `unifi` (cef, iptables, netflow) and Athena workgroup `syslog-server` |
|
|
| Alarms | `Syslog-NoIncomingRecords`, `Syslog-FirehoseDeliveryFailed`, `EC2-StatusCheck-syslog-server`, `EC2-StatusCheckSystem-syslog-server-recover` → `site-alerts` |
|
|
|
|
The office IPsec tunnel that already reaches mgmt `10.20.0.0/16` does **not**
|
|
land in this VPC. UniFi needs a second site-to-site peer for `10.40.0.0/16`.
|
|
Do not re-home this workspace in mgmt.
|
|
|
|
## Access
|
|
|
|
SSM Session Manager. SSH 22 is closed. There is no Elastic IP forwarding
|
|
target.
|
|
|
|
## IAM bootstrap window
|
|
|
|
Instance-boundary document changes and apply-role inline policy changes need
|
|
the hcptf-bootstrap window (`DenySelfMutation` plus deny on
|
|
`iam:CreatePolicyVersion`). Sequence:
|
|
|
|
1. From `seahaven-org-baseline`:
|
|
`scripts/create-hcptf-bootstrap-roles.sh --account prod --allow-workspace syslog-server-prod`
|
|
2. Point workspace `TFC_AWS_APPLY_ROLE_ARN` / `TFC_AWS_PLAN_ROLE_ARN` at
|
|
`hcptf-bootstrap` / `hcptf-bootstrap-plan`. Keep `TFC_AWS_PROVIDER_AUTH=true`.
|
|
Never `TFC_AWS_RUN_ROLE_ARN`.
|
|
3. One manual apply as bootstrap creates/updates the scoped `hcptf-*` inline
|
|
policies and the instance boundary.
|
|
4. Retarget `TFC_AWS_*` to `hcptf-syslog-server` / `hcptf-syslog-server-plan`.
|
|
Re-run the create script with no `--allow-workspace`.
|
|
5. Manual apply as the scoped role for the rest of the stack (instance,
|
|
Firehose, S3, Glue, Athena, VGW). Auto-apply stays off until soak.
|
|
|
|
HCP outputs to copy: `private_ip`, `vpn_connection_id`,
|
|
`vpn_tunnel1_address`, `vpn_tunnel2_address`, `bucket_name`,
|
|
`firehose_name`, `athena_workgroup`. Read PSKs with
|
|
`terraform output -raw vpn_tunnel1_preshared_key` after apply. Do not commit
|
|
them.
|
|
|
|
AMI is pinned in `var.ami_id`. An AMI or user-data change replaces the
|
|
instance. The box is stateless; archives live in S3.
|
|
|
|
`Syslog-NoIncomingRecords` defaults `treat_missing_data` to `notBreaching`
|
|
until UniFi delivers over IPsec. After Firehose `IncomingRecords` is
|
|
non-zero, set `no_logs_treat_missing_data=breaching`.
|
|
|
|
## UniFi cutover
|
|
|
|
Do this after the HCP apply, not before. Apply drops public 514 and the
|
|
CloudWatch `unifi-syslog` log group. Point UniFi immediately.
|
|
|
|
1. **Ronkonkoma site-to-site VPN** to the AWS tunnel addresses from HCP
|
|
outputs. Remote network `10.40.0.0/16`. Local network `10.10.0.0/16`.
|
|
IKEv2, AES-256, SHA-256, DH14 matches typical AWS defaults. Use the
|
|
Terraform PSK outputs. This is a second child SA alongside the existing
|
|
mgmt `10.20` tunnel. Do not replace the mgmt tunnel.
|
|
2. **Locust SD-WAN mesh** must already route AWS VPC CIDRs via Ronkonkoma
|
|
(same as jumpbox SSH). Add `10.40.0.0/16` if it is missing.
|
|
3. Both controllers, **Settings → CyberSecure / System Log**:
|
|
- SIEM server = collector **private IP**, port **514**, UDP
|
|
- Flow Logging = **All Traffic**
|
|
- Activity Logging SIEM contents include firewall
|
|
- Control Plane **CEF** to the same IP:514
|
|
4. Enable syslog on WAN and inter-VLAN firewall rules, or All Traffic stays
|
|
silent.
|
|
5. NetFlow/IPFIX: Ronkonkoma → UDP **2055**, Locust → UDP **2056**, same
|
|
private IP.
|
|
6. Prove the path: send a test syslog from Ronkonkoma; Athena `SELECT` on
|
|
`iptables` and `cef`; confirm `format=netflow` objects for 2055/2056;
|
|
confirm `site-alerts` does not fire while traffic is present.
|
|
7. Flip off any remaining public EIP / mgmt collector **only after**
|
|
Firehose `IncomingRecords` is non-zero. PLAT-78 still owns deleting the
|
|
mgmt `syslog-server` CloudFormation stack after soak.
|
|
|
|
Vector treats payloads as untrusted text. It parses fields and does not
|
|
shell out. IPFIX datagrams are archived as base64 JSON with a site tag
|
|
(Vector has no released IPFIX decoder).
|
|
|
|
## Documentation
|
|
|
|
The canonical map of Sea Haven's AWS infrastructure lives in Confluence.
|
|
|
|
- **[AWS Architecture Map](https://seahaven.atlassian.net/wiki/spaces/IT/pages/1540098)** (Confluence, IT space, page 1540098)
|
|
- **[Syslog Server](https://seahaven.atlassian.net/wiki/spaces/IT/pages/67141633)** (page 67141633)
|
|
|
|
Tracked as **PLAT-206**. PLAT-78 remains the HCP move plus mgmt stack delete
|
|
after this soak.
|