AI on AWS · Guide ·
How to build a secure internal ChatGPT on AWS
A practical path to private enterprise AI using Amazon Bedrock — identity, data boundaries, logging, and cost controls without shipping prompts to public APIs.
Teams want ChatGPT-like productivity without sending proprietary data to a shared public API. On AWS, that usually means Amazon Bedrock plus your documents in S3 (or OpenSearch) with a retrieval layer — not a random EC2 box running an open-weight model without guardrails.
What “secure internal ChatGPT” actually means
Define success before picking models:
- Who can use it? SSO via IAM Identity Center; no shared API keys in Slack.
- What can it access? Explicit document corpus per team or workspace — not “the whole S3 account.”
- What leaves the boundary? Bedrock keeps inference in AWS; you still control prompts, logs, and egress.
- What gets logged? CloudTrail for API calls; optional prompt/response logging with retention limits and PII review.
- What does it cost? Token metering per team; kill switches when spend spikes.
Reference architecture (simplified)
| Layer | AWS building blocks |
|---|---|
| Identity | IAM Identity Center, per-group permissions |
| App | Internal web UI or Teams/Slack bot on Lambda |
| Models | Bedrock (Claude, Titan, etc.) via IAM-scoped access |
| Knowledge | S3 + Kendra or OpenSearch Serverless for RAG |
| Secrets | Secrets Manager; no keys in repos |
| Audit | CloudTrail, Config, optional Security Hub |
Start with one use case — HR policy Q&A, support playbook search, or engineering runbooks — not “every document we’ve ever written.”
Security mistakes we see in pilots
- Over-broad IAM —
bedrock:InvokeModelon*for “simplicity” - No data classification — confidential PDFs in the same index as public marketing
- Skipping human review for high-stakes answers (legal, medical, financial)
- Treating RAG as authorization — retrieval ≠ permission check
Rollout sequence that works
- 2-week pilot — one corpus, one model, 10–20 users, read-only docs
- Measure — answer quality, latency, cost per query, support tickets deflected
- Harden — VPC endpoints, KMS encryption, retention policies, red-team prompts
- Expand — additional corpora with separate indexes and IAM boundaries
We help New England and remote-first teams design Bedrock pilots with encryption, access policies, and cost caps leadership can approve. If you are comparing build vs. buy, read AWS Bedrock vs OpenAI for enterprises next, then AI governance basics before you scale users.