mirror of
https://github.com/Sea-Haven-Industries/proposal-system.git
synced 2026-09-30 04:13:13 +00:00
aff38a11ae
Some checks failed
Deploy / Deploy to AWS (push) Has been cancelled
Replaces OpenSearch Serverless with Aurora PostgreSQL Serverless v2 + pgvector as the Bedrock Knowledge Base vector store (v1 PR3). Bedrock KB requires Aurora SSv2 (RDS Data API), not a plain RDS instance — so the DB engine moves to Aurora. - foundation: rds.DatabaseInstance (PG15) -> rds.DatabaseCluster Aurora SSv2 (0.5-4 ACU, enableDataApi). RDS alarms: free-storage -> freeable-memory. - compute: delete all AOSS (collection, policies, VPC endpoint, index-creator); add bedrock_user secret + KB role (scoped rds-data + secret read); repoint CfnKnowledgeBase to RDS storage (bedrock_integration.bedrock_kb, vector(1024)). - lambdas: oss-index-creator -> aurora-pgvector-init (bootstrap schema/table/ indexes/role via RDS Data API; transient-error retry; password guard). - ADR 0001 documents the decision. Eliminates the ~$175-350/mo AOSS OCU floor. NAT kept (egress still needed). GPT-4.1 cross-review: no BLOCK (FIX applied). tsc clean; foundation synth shows Aurora cluster with Data API enabled; 23 pytest pass.
68 lines
3.6 KiB
Markdown
68 lines
3.6 KiB
Markdown
# ADR 0001 — Bedrock Knowledge Base vector store: Aurora PostgreSQL + pgvector
|
||
|
||
- **Status:** Accepted (2026-06-12)
|
||
- **Decision owner:** Adam Moussa
|
||
- **Scope:** `proposal-system` infrastructure (foundation + compute stacks), v1 PR3
|
||
|
||
## Context
|
||
|
||
The proposal system's RAG pipeline uses an Amazon Bedrock Knowledge Base (Titan Embed
|
||
v2) over historical proposals. The original vector store was **OpenSearch Serverless
|
||
(AOSS)**, which:
|
||
|
||
- carries a minimum ~2-OCU billing floor (~$175–350/mo) even at near-zero query volume —
|
||
the dominant line item of the monthly bill for an internal tool;
|
||
- is VPC-only, which forces the NAT gateway and a `oss-index-creator` bootstrap Lambda
|
||
that exists purely to pre-create the vector index while an IAM access policy propagates.
|
||
|
||
The database is a small RDS PostgreSQL instance. The v1 assessment flagged AOSS as
|
||
over-built for the corpus size and recommended a pgvector store on the existing database.
|
||
|
||
## Decision
|
||
|
||
Migrate the database from **RDS PostgreSQL 15 → Aurora PostgreSQL Serverless v2** and use
|
||
**pgvector** as the Bedrock KB vector store.
|
||
|
||
Aurora is required because **Bedrock Knowledge Bases support Aurora PostgreSQL
|
||
Serverless v2 (with the RDS Data API) as a pgvector store, but not a plain RDS
|
||
instance.** A standard RDS instance cannot back a Bedrock KB, so "pgvector on the
|
||
existing RDS" was infeasible without the Aurora move.
|
||
|
||
Implementation:
|
||
|
||
- Aurora Serverless v2 (min 0.5 / max 4 ACU), `enableDataApi: true`, `defaultDatabaseName:
|
||
'proposals'`. The cluster also serves the .NET API's application data (one database).
|
||
- A bootstrap custom-resource Lambda (`aurora-pgvector-init`, via the RDS Data API)
|
||
enables `vector`, creates the `bedrock_integration.bedrock_kb` table
|
||
(`vector(1024)` for Titan v2 + HNSW cosine + GIN indexes) and a dedicated
|
||
`bedrock_user` role. This **replaces** `oss-index-creator` — the bootstrap is swapped,
|
||
not eliminated.
|
||
- `CfnKnowledgeBase.storageConfiguration` → `type: 'RDS'` with `rdsConfiguration`
|
||
(cluster ARN, `bedrock_user` secret, `bedrock_integration.bedrock_kb`, field mapping
|
||
`id`/`embedding`/`chunks`/`metadata`). KB role IAM swaps `aoss:APIAccessAll` →
|
||
scoped `rds-data` + secret read.
|
||
- All AOSS constructs (collection, policies, VPC endpoint, index-creator) are deleted.
|
||
|
||
## Consequences
|
||
|
||
- **Cost:** eliminates the AOSS OCU floor (~$175–350/mo). Aurora Serverless v2 at
|
||
0.5 ACU min is ~$43/mo and scales toward zero idle — a net reduction.
|
||
- **Simplification:** one data engine (Aurora) instead of RDS + AOSS; fewer constructs.
|
||
A bootstrap Lambda remains (now for pgvector schema rather than the AOSS index).
|
||
- **NAT:** kept for now — Lambdas and the KB's Data API path still need AWS-service
|
||
egress. Dropping NAT would require VPC interface endpoints; tracked separately.
|
||
- **Migration:** the RDS→Aurora swap is a CloudFormation replacement. The deployed stacks
|
||
are **test-only with no production data**, so this is a clean redeploy.
|
||
|
||
## Alternatives considered
|
||
|
||
- **Keep AOSS:** rejected — the cost floor is the single biggest waste for the scale.
|
||
- **S3 Vectors:** viable Bedrock backend, but Aurora unifies app data + vectors and was
|
||
the owner's preference (also cheaper than AOSS).
|
||
- **pgvector on the existing RDS instance:** infeasible — Bedrock KB does not support a
|
||
plain RDS instance as a vector store.
|
||
|
||
## References
|
||
|
||
- [Using Aurora PostgreSQL as a Bedrock Knowledge Base](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.VectorDB.html)
|
||
- [Bedrock KB vector-store prerequisites](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html)
|