The AI infrastructure landscape has transformed dramatically. What was once the domain of specialized ML teams is now a core consideration for every application architect. Large language models, vector databases, and inference optimization have moved from research curiosities to production requirements.
For organizations building AI-powered applications on AWS, the decisions are no longer "if" but "how"—managed services versus self-hosted models, proprietary versus open-source, cost versus capability trade-offs that didn't exist two years ago. This article shares what we've learned building AI applications for clients in regulated industries, where these decisions carry additional weight.
The New AI Application Stack
AI applications require infrastructure layers that traditional web applications don't. Understanding this stack helps clarify where different AWS services fit.
AI Application Infrastructure Stack
Managed vs. Self-Hosted: The Strategic Decision
The first major architectural decision is whether to use managed AI services (primarily Amazon Bedrock) or host models yourself. This isn't a one-time choice—most production architectures use both, selecting the right approach for each use case.
Amazon Bedrock
You need rapid deployment, want to avoid infrastructure management, require access to frontier models (Claude, Llama), or need built-in guardrails. Best for most enterprise use cases where time-to-value matters more than per-token cost optimization.
Self-Hosted (vLLM/TGI)
You have high-volume, predictable workloads where per-token costs dominate. Need fine-tuned models with custom weights. Require complete control over model behavior. Have GPU expertise on your team.
SageMaker Endpoints
You need the flexibility of self-hosting with managed infrastructure. Running multiple model versions for A/B testing. Integrating with existing SageMaker ML pipelines. Want auto-scaling without managing Kubernetes.
Hybrid Approach
Different use cases have different requirements. Common pattern: Bedrock for complex reasoning tasks (Claude), self-hosted smaller models for high-volume classification or embedding generation.
The Cost Equation Has Changed
A year ago, self-hosting was almost always cheaper for high-volume workloads. That calculus has shifted with Bedrock's batch inference pricing and the operational overhead of maintaining GPU clusters.
When Self-Hosting Pays Off
Our analysis suggests self-hosting becomes cost-effective at roughly 10+ million tokens per day of sustained load with predictable patterns. Below that threshold, or with variable workloads, Bedrock's pay-per-use model typically wins when you factor in engineering time for GPU management, model updates, and infrastructure maintenance.
Open-Source LLMs in Production
The open-source model ecosystem has matured dramatically. Models like Llama 3, Mistral, and their derivatives now offer capabilities that rival proprietary options for many use cases—at a fraction of the per-token cost when self-hosted.
Model Selection for Enterprise Use Cases
| Model Family | Strengths | AWS Deployment Options | GPU Requirements |
|---|---|---|---|
| Llama 3.1 (8B/70B/405B) | General purpose, strong reasoning, large context (128K) | Bedrock, SageMaker, EC2 | 8B: Single A10G; 70B: 2-4x A100; 405B: 8x H100 |
| Mistral/Mixtral | Efficient inference, strong coding, MoE architecture | Bedrock, SageMaker, EC2 | 7B: Single A10G; 8x7B: 2x A100 |
| Phi-3 (Mini/Small/Medium) | Compact size, strong reasoning for size | SageMaker, EC2 | Mini: Single T4; Medium: Single A10G |
| Code Llama / DeepSeek Coder | Code generation, completion, review | SageMaker, EC2 | 7B: Single A10G; 34B: 2x A100 |
vLLM: The Production Inference Standard
For self-hosted deployment, vLLM has emerged as the de facto standard for production inference. Its PagedAttention mechanism dramatically improves throughput compared to naive implementations.
Key vLLM capabilities that matter for production:
- Continuous batching: New requests join in-progress batches, maximizing GPU utilization
- PagedAttention: Efficient memory management allowing larger batch sizes
- OpenAI-compatible API: Drop-in replacement for applications built on OpenAI SDKs
- Quantization support: Run INT8/INT4 quantized models for reduced memory footprint
- Speculative decoding: Use smaller draft models to accelerate generation
vLLM on AWS: Deployment Patterns
We typically deploy vLLM on EKS with GPU node pools (g5.xlarge for smaller models, p4d.24xlarge for larger ones). The NVIDIA device plugin handles GPU scheduling, and Karpenter manages node provisioning based on pending inference requests.
GPU Instance Selection
Choosing the right GPU instance is crucial for both performance and cost. AWS offers a range of options, each suited to different workloads.
G5 (A10G)
P4d (A100)
P5 (H100)
For most enterprise inference workloads, G5 instances (A10G GPUs) provide the best cost-performance ratio. They handle 7B-13B parameter models efficiently and are more readily available than P4d/P5 instances.
GPU Capacity Planning
GPU instance availability varies significantly by region and can have multi-week lead times for capacity reservations. For production workloads, we recommend capacity reservations in at least two regions, with fallback to Bedrock for overflow during capacity crunches.
Guardrails: Non-Negotiable for Regulated Industries
AI applications in banking, insurance, and healthcare require robust guardrails. Users will attempt—intentionally or accidentally—to elicit inappropriate responses. Without guardrails, a single screenshot of a problematic response can become a PR crisis.
Amazon Bedrock Guardrails
Bedrock's built-in guardrails provide a managed solution for common safety requirements:
- Content filters: Block harmful, inappropriate, or off-topic content
- Denied topics: Prevent discussion of specific subjects (competitors, legal advice, medical diagnoses)
- Word filters: Block specific terms or phrases
- PII detection: Identify and redact personal information
- Contextual grounding: Verify responses are grounded in provided context (reducing hallucination)
Guardrails Implementation Pattern
Custom Guardrails for Self-Hosted Models
When using self-hosted models without Bedrock, you need equivalent protections. Our typical stack includes:
- NVIDIA NeMo Guardrails: Programmable guardrails with custom policies
- Presidio: Microsoft's PII detection and anonymization library
- Custom classifiers: Fine-tuned models for domain-specific content moderation
- Prompt injection detection: Pattern matching and ML-based detection of manipulation attempts
Model Context Protocol (MCP): The Emerging Standard
MCP represents a significant shift in how AI applications integrate with external systems. Rather than building custom integrations for each tool or data source, MCP provides a standardized protocol for LLMs to interact with external capabilities.
Why MCP Matters
MCP transforms AI applications from isolated chat interfaces into connected systems that can query databases, call APIs, access file systems, and interact with enterprise tools—all through a standardized, auditable interface. For regulated industries, the standardization means consistent security controls across all integrations.
MCP Architecture on AWS
An MCP deployment consists of:
- MCP Servers: Services that expose tools, resources, or prompts to the LLM. Each server handles a specific capability (database access, API calls, file operations)
- MCP Client: The AI application orchestrator that communicates with servers and the LLM
- Transport Layer: Typically stdio for local servers, HTTP/SSE for remote servers
On AWS, we deploy MCP servers as containerized services on ECS or EKS, with IAM roles providing least-privilege access to downstream resources. This pattern allows the LLM to interact with S3, DynamoDB, RDS, or external APIs without exposing credentials or raw access.
Security Considerations for MCP
MCP introduces new attack surfaces that require careful consideration:
- Tool authorization: Which tools can the LLM invoke? Under what conditions?
- Input validation: LLM-generated tool arguments must be validated before execution
- Output sanitization: Data returned from tools may contain sensitive information
- Audit logging: Every tool invocation should be logged for compliance and debugging
- Rate limiting: Prevent runaway tool calls from overwhelming downstream systems
Vector Databases and RAG Infrastructure
Retrieval-Augmented Generation (RAG) remains the primary pattern for grounding LLM responses in enterprise data. AWS offers several options for the vector store component:
| Service | Best For | Considerations |
|---|---|---|
| Aurora PostgreSQL (pgvector) | Teams already using PostgreSQL, moderate scale (<10M vectors) | Familiar SQL interface, but vector search is not the primary use case |
| OpenSearch Serverless | Large-scale vector search, combined with full-text search | Higher cost, but scales well and integrates with Bedrock Knowledge Bases |
| Amazon Kendra | Enterprise search with built-in connectors | Higher-level abstraction, less flexibility, premium pricing |
| Pinecone / Weaviate (self-hosted) | Purpose-built vector DB features, advanced filtering | Additional operational overhead, but more specialized capabilities |
For most clients, we start with Aurora PostgreSQL with pgvector. It provides good-enough vector search for initial deployments, leverages existing PostgreSQL expertise, and avoids introducing another database to manage. As scale and requirements grow, migration to OpenSearch or a dedicated vector database is straightforward.
Building for Production
AI applications require the same production practices as any critical system, plus additional considerations unique to ML workloads:
- Model versioning: Track which model version generated each response for debugging and compliance
- Prompt versioning: System prompts are code—version them, test them, deploy them through CI/CD
- Response logging: Log inputs, outputs, and metadata for quality monitoring and incident investigation
- Latency monitoring: LLM response times vary significantly—track P50, P95, P99
- Cost tracking: Token usage can spike unexpectedly—monitor and alert on cost anomalies
- Fallback strategies: When primary model fails, what happens? Graceful degradation is essential
The AI infrastructure landscape continues to evolve rapidly. Services and best practices that are cutting-edge today may be superseded within months. The patterns in this article reflect our current understanding—we update our approaches continuously as the technology matures.
