AI on AWS · Insights · · 3 min read
When not to build generative AI on AWS
A practical filter for product and IT leaders — when a Bedrock pilot is worth it, and when a simpler workflow still wins.
Not every “we should do AI” request deserves a Bedrock project. The fastest way to waste a quarter is building a chat box for a problem that already has a search box, a report, or a better process.
Here is the filter we use with product and IT teams before anyone opens the AWS console.
Start with the job, not the model
Write one sentence: who needs what outcome, measured how, using which data.
If you cannot fill those four blanks, you are not ready to choose Claude vs Titan — you are still clarifying the product.
| Signal | Usually means |
|---|---|
| “Everyone should have ChatGPT” | No bounded use case yet |
| “Summarize this weekly report” | Often a template + script beats a model |
| “Answer questions from these 40 policies” | Strong RAG candidate |
| “Replace our support team” | Scope fantasy; start with deflection metrics |
Cases where generative AI is the wrong tool
1. The answer is already structured.
If the data lives in a CRM field, a BI dashboard, or a SQL query with a clear join, ship a report or API. LLMs shine when language and ambiguity are the hard part — not when the hard part is joining tables.
2. You need perfect accuracy every time.
Invoice totals, dose calculations, access grants, and payroll changes need deterministic systems. Use models for drafts and suggestions with a human (or a rules engine) in the loop — not as the system of record.
3. The corpus is a mess.
Generative AI amplifies whatever you index. If policies conflict, filenames are opaque, and nobody owns freshness, fix information hygiene first. Otherwise you will scale confident wrong answers.
4. You cannot name an owner.
Someone has to own cost, access reviews, evaluation, and “what happens when it is wrong.” If that role does not exist, the pilot will stall after the demo.
5. The only success metric is “people liked the demo.”
Likeability is not a business case. Prefer time-to-answer, tickets deflected, hours saved on a defined task, or error rate vs a baseline workflow.
A 30-minute go / no-go checklist
Ask these in order. One hard “no” is enough to pause.
- Is there a single user group for the pilot (not the whole company)?
- Is the corpus bounded and permissioned (not “all of S3”)?
- Is failure tolerable with human review, or is a wrong answer unacceptable?
- Do you have a cost cap leadership has already accepted?
- Can you evaluate quality with 10–20 labeled examples before wider rollout?
- Is a simpler path worse? Search, macros, RPA, or a better wiki often win on cost and risk.
If items 1–5 are yes and item 6 still points at generative AI, you have a real pilot — not a science fair.
What to do instead (when the answer is no)
| Need | Prefer |
|---|---|
| Find the right doc faster | Search with good metadata, or Kendra/OpenSearch without generation |
| Weekly status narrative | Template + metrics pull; optional LLM polish later |
| Classify tickets | Rules or classical ML with labeled data |
| “Something cool for the board” | One measured pilot later — not a production chatbot this sprint |
Saying no early is a delivery skill. It protects budget for the use cases that clear the bar.
When the answer is yes
Then the work looks like building a secure internal assistant on AWS: identity, corpus boundaries, evaluation, and cost controls before scale. Pair that with governance basics so legal and security are not bolted on after the demo.
If you have a use case in mind and want a frank go / no-go read, discuss an AI use case or book a free architecture review.