How to build a secure internal ChatGPT on AWS

Teams want ChatGPT-like productivity without sending proprietary data to a shared public API. On AWS, that usually means Amazon Bedrock plus your documents in S3 (or OpenSearch) with a retrieval layer — not a random EC2 box running an open-weight model without guardrails.

This guide covers the decisions that separate a useful pilot from a security and cost problem.

What “secure internal ChatGPT” actually means

Define success before picking models:

1. Who can use it? SSO via IAM Identity Center; no shared API keys in Slack.
2. What can it access? Explicit document corpus per team or workspace — not “the whole S3 account.”
3. What leaves the boundary? Bedrock keeps inference in AWS; you still control prompts, logs, and egress.
4. What gets logged? CloudTrail for API calls; optional prompt/response logging with retention limits and PII review.
5. What does it cost? Token metering per team; kill switches when spend spikes.
6. How do you know it works? Evaluation criteria before launch — not vibes after the demo.

If you cannot answer those six questions, you are not ready to choose a model.

Choose a use case that can win

Good first use cases share three traits: a clear user group, a bounded corpus, and a measurable outcome.

Use case	Why it works	Watch-outs
Internal policy / HR Q&A	Stable docs, low blast radius	Stale PDFs; legal review of answers
Support playbook search	Deflects repetitive tickets	Needs citation links; escalate when unsure
Engineering runbooks	High willingness to try tools	Secrets must never enter the index
Contract clause lookup	Clear ROI for professional services	Access control by matter or client
Meeting / document summarization	Easy to demo	Prompt injection via uploaded files

Avoid “search everything we have ever written” as the pilot. Wide corpora hide permission mistakes and make evaluation impossible.

Reference architecture

Layer	AWS building blocks	Design note
Identity	IAM Identity Center, permission sets	Map groups to corpora; no shared keys
App	Internal web UI or Teams/Slack bot on Lambda / App Runner	Keep the app thin; put policy in IAM and the retrieval layer
Models	Bedrock (Claude, Titan, etc.) via IAM-scoped access	Start with one model; change later with evidence
Knowledge	S3 + OpenSearch Serverless or Kendra for RAG	Separate indexes per sensitivity boundary
Secrets	Secrets Manager	Never commit keys or embed them in frontend code
Network	VPC endpoints for Bedrock/S3 where required	Reduce public egress; document what still leaves
Audit	CloudTrail, Config, optional Security Hub	Log admin changes; decide prompt-log retention deliberately
Cost	Budgets, usage metrics, per-team tags	Cap pilot spend before finance asks

RAG is not authorization

Retrieval finds relevant chunks. It does not prove the user is allowed to see them. Enforce permissions at:

• Identity — who can invoke the app
• Corpus membership — which indexes a group can query
• Document ACLs — when the source system already has finer permissions

If a contractor should not see executive compensation PDFs, those files must not live in the contractor-accessible index — full stop.

Security mistakes we see in pilots

• Over-broad IAM — bedrock:InvokeModel on * for “simplicity”
• No data classification — confidential PDFs in the same index as public marketing
• Skipping human review for high-stakes answers (legal, medical, financial)
• Treating RAG as authorization — retrieval ≠ permission check
• Unbounded spend — no token budgets, no kill switch, no owner for the bill
• Prompt logs forever — retaining user questions that contain customer data without a retention policy
• No evaluation set — expanding users before you know failure modes

Cost controls that finance will accept

Estimate rough monthly cost before the pilot:

1. Expected queries per day × average tokens in/out × model price
2. Retrieval and indexing cost for the corpus size
3. App hosting (usually small compared with inference at scale)

Then set:

• A hard monthly budget with alerting at 50% and 80%
• A per-user or per-team soft cap for the pilot cohort
• A model tiering plan (cheaper model for draft; stronger model only when needed)

Open-ended “we’ll see how people use it” is how AI pilots become finance emergencies.

Evaluation before you scale

Write ten to twenty representative questions with expected source documents. Score:

Dimension	Question
Groundedness	Does the answer cite the right docs?
Correctness	Would a subject-matter expert accept it?
Refusal	Does it decline when the corpus has no answer?
Safety	Does it resist prompt injection and data exfil attempts?
Latency / cost	Is the experience usable at the price you set?

Promote to production only when the pilot beats a baseline — usually “search the shared drive yourself” — on groundedness and time-to-answer.

Rollout sequence that works

1. Use-case workshop — one corpus, one success metric, named owner
2. 2-week pilot — 10–20 users, read-only docs, cost caps on
3. Measure — quality, latency, cost per query, tickets deflected, failure cases
4. Harden — VPC endpoints, KMS encryption, retention policies, red-team prompts
5. Expand — additional corpora with separate indexes and IAM boundaries
6. Operate — who owns index freshness, access reviews, and model changes

Build vs. buy (short version)

Path	Choose when
Bedrock + your app	You need AWS-native controls, custom UX, or tight corpus isolation
Packaged enterprise assistant	You want faster UI and connectors and can accept vendor boundaries
Public ChatGPT / consumer tools	Almost never for proprietary corpora

For a deeper comparison, read AWS Bedrock vs OpenAI for enterprises. For policy before technology, read AI governance basics.

---

We help product and IT teams design Bedrock pilots with encryption, access policies, evaluation, and cost caps leadership can approve. If you already have a use case in mind, discuss an AI use case or book a free discovery call.