Moving an AI application to a different LLM provider is more than replacing an endpoint and API key. This reusable migration checklist helps you inventory provider dependencies, compare behavior on representative workloads, test API compatibility, plan a staged cutover, and preserve a safe rollback path.
Overview
An LLM provider migration can affect every layer of an application: authentication, request formatting, model selection, prompt behavior, tool execution, structured output, streaming, observability, cost controls, and data handling. Two APIs may expose similar concepts while producing different results under the same prompt. A successful migration therefore measures both technical compatibility and application behavior.
Start by defining what must remain stable. For a customer-support chatbot, that may include answer quality, escalation decisions, citation format, response time, and tool-call accuracy. For an internal extraction workflow, the critical requirements may be schema validity, field-level accuracy, retry behavior, and processing cost. Write these acceptance criteria before testing the new provider so the migration is not judged by impressions alone.
Keep a versioned record of the current implementation. Capture the provider, model identifier, SDK version, request parameters, system instructions, tools, response schema, safety controls, retry policy, timeout values, and representative inputs. This inventory becomes the baseline for parity testing and makes it easier to identify whether a change came from the provider, the model, the prompt, or your integration code.
| Area | Questions to compare | Evidence to record |
|---|---|---|
| API and SDK | Are endpoints, SDK methods, authentication, and request fields compatible? | Working integration, dependency versions, migration notes |
| Model behavior | Does the replacement follow instructions and handle edge cases acceptably? | Evaluation results, failures, reviewer notes |
| Capabilities | Are streaming, tool calls, vision, structured responses, and embeddings supported as needed? | Capability tests and known limitations |
| Operations | Do latency, rate limits, retries, errors, and monitoring fit the workload? | Logs, timing samples, error classifications |
| Governance | Do data handling, retention, regional, and contractual requirements remain satisfied? | Reviewed provider documentation and internal approval |
Checklist by scenario
Before switching providers
- List every model endpoint, SDK, middleware component, proxy, and background job that calls the current provider.
- Record model identifiers and separate stable application settings from provider-specific defaults.
- Export a representative test set covering normal requests, long context, malformed input, refusal-sensitive requests, multilingual input, and known production failures.
- Save expected output properties rather than relying only on exact text. Examples include valid JSON, required fields, citation presence, tool selection, and maximum length.
- Document current latency, token usage, failure rates, retry counts, and timeout behavior using the same measurement method you will use after migration.
- Review the replacement provider’s current API documentation, model capabilities, limits, SDK examples, deprecation notices, and data terms before implementation.
When the APIs are structurally different
Build an adapter at your application boundary instead of spreading provider-specific conditionals through business logic. Convert your internal message format into the target provider’s format in one place. Do the same for tool definitions, response parsing, usage metadata, errors, and streaming events.
Check message roles and content parts carefully. A provider may distinguish text, images, tool calls, and tool results differently. Confirm how conversation history is ordered, how empty content is handled, how system instructions are represented, and whether unsupported fields are rejected or silently ignored. Test Unicode, large inputs, attachments, and truncated requests if they are part of your product.
When your application uses tools or structured output
- Run a function-calling test that verifies the model chooses the correct tool, produces valid arguments, and responds appropriately to tool results.
- Test missing, invalid, delayed, and partially successful tool responses.
- Validate structured output with a real parser and schema validator. Do not treat visually correct JSON as sufficient.
- Check whether the provider supports the same schema features, strictness controls, enums, nested objects, and refusal behavior that your application expects.
- Confirm how streamed tool calls are assembled before execution and how interrupted streams are recovered.
When migrating an agent or RAG workflow
Test the entire workflow, not only the generation step. For an AI agent, measure planning loops, tool selection, termination, duplicate actions, and recovery from tool errors. For retrieval-augmented generation, preserve the same retrieved documents and compare answer grounding, citation mapping, context usage, and behavior when the answer is absent. A provider change can expose weaknesses in prompts, retrieval, or orchestration that were previously hidden.
Related workflows can be reviewed in AI agent architecture patterns and the RAG knowledge base maintenance guide.
What to double-check
Authentication, limits, and errors
Confirm how credentials are created, scoped, rotated, and supplied across local, staging, and production environments. Replace hard-coded assumptions about status codes or error bodies with an internal error classification. Distinguish authentication failures, invalid requests, rate limits, timeouts, overloaded services, content-policy responses, and provider-side errors so retries do not amplify a failure.
Recalculate concurrency and retry behavior against the new provider’s documented limits. A retry policy that was safe for one service may create excessive load or duplicate side effects elsewhere. Use idempotency controls for tool actions and external writes, and set bounded timeouts for each stage of the request.
Streaming and user experience
Compare the complete streaming lifecycle: connection establishment, first-token timing, event names, partial content, finish signals, usage reporting, tool-call fragments, disconnects, and cancellation. Your application should not mark a response complete merely because text stopped arriving. Test a client disconnect and confirm that generation and downstream work are cancelled or safely contained.
Prompts and output behavior
Do not assume prompt portability. Differences in instruction following, tokenization, defaults, context handling, safety behavior, and sampling can change outputs. Keep the prompt stable for the first comparison, then optimize it separately if needed. Version any changes to system instructions, examples, schemas, and post-processing so you can attribute improvements or regressions.
Use a parity set with both automated checks and human review. Automated checks can validate schemas, required phrases, citations, tool arguments, and prohibited fields. Human review remains useful for tone, ambiguity, relevance, and nuanced refusals. The prompt testing and evaluation framework provides a broader structure for this process.
Privacy, security, and deployment
Review what data leaves your environment, which regions and environments are involved, how logs are configured, and whether sensitive fields need redaction. Recheck secrets management, network allowlists, audit trails, and vendor review requirements. For production deployment, follow a broader AI chatbot deployment checklist rather than treating the provider change as an isolated code release.
Common mistakes
- Testing only happy paths: Include malformed requests, long context, empty retrieval results, tool failures, timeouts, and partial responses.
- Comparing exact wording instead of behavior: Evaluate task-specific properties and acceptable ranges. Equivalent answers may use different wording.
- Changing the prompt during the API migration: Keep variables separate so you can identify the source of a behavior change.
- Ignoring hidden provider defaults: Explicitly record parameters such as temperature, token limits, response formats, safety settings, and stop conditions when available.
- Assuming a successful HTTP response means success: Parse and validate the content, tool arguments, citations, and schema before returning it to users.
- Cutting over all traffic at once: A small production slice can reveal workload-specific issues that a test set misses.
- Forgetting operational cost: Compare retries, token expansion, caching, observability, and engineering overhead—not just the nominal model call.
- Having no rollback trigger: Decide in advance which quality, latency, error, or cost thresholds require pausing or reversing the rollout.
For prompt and application changes that must remain traceable, pair the migration with a documented LLM prompt versioning workflow and, where appropriate, an LLM application CI/CD pipeline.
When to revisit
Revisit this checklist before each seasonal planning cycle, contract renewal, regional deployment, major model replacement, or significant prompt and workflow change. It should also be reviewed when a provider changes API requirements, model capabilities, pricing or usage limits, privacy terms, SDK behavior, or deprecation schedules. These inputs can change independently of your application code.
Maintain a short migration record with the review date, providers and model versions considered, test-set version, acceptance thresholds, unresolved differences, and rollback owner. Refresh the representative workload when user behavior changes; an old benchmark can make a migration appear stable while missing newer request types.
The practical sequence is straightforward: inventory the current stack, define invariants, build a provider adapter, run parity tests, review governance requirements, deploy to a limited traffic slice, monitor against predeclared thresholds, and keep the old path available until the new path has earned confidence. If quality declines, first isolate whether the cause is API translation, model behavior, prompt differences, retrieval, tools, or operations. That diagnosis is more valuable than treating the provider switch as a single pass-or-fail event.