Why a Checklist?
Multi-agent systems are more complex than classic software. Asking about "AI experience" is not enough. You want to know whether a provider solves architecture problems before they hit you in production.
The 5 Questions for Your First Meeting
1. Architecture: Which framework do you use for orchestration, and why?
Why it matters: Without a deliberate orchestration layer, multi-agent systems quickly become impossible to maintain.
Red flag: "We build everything ourselves" without a reason, or a single framework for every case.
Good answer: "LangGraph for strictly defined workflows, the OpenAI Agents SDK or Microsoft Agent Framework when handoffs between agents matter; tools connected via MCP."
2. Errors: How do you prevent endless loops and runaway costs?
Why it matters: An agent that fails and retries again and again can burn through a budget in hours.
Red flag: "That doesn't happen" or no concrete answer.
Good answer: "Step and cost limits per run, timeouts, retries with backoff and a supervisor that stops the run and alerts someone."
3. Quality: How do you measure whether the system gives correct results?
Why it matters: Language models still make things up. Without systematic evaluation nobody notices until a customer does.
Red flag: "The new models no longer hallucinate."
Good answer: "Test sets from real cases, automated evaluation in CI, tracing in production, grounding via RAG and structured outputs."
4. Memory: Where is the state of the agents stored?
Why it matters: Multi-step processes have to survive restarts and hand context between agents.
Red flag: "In the session" or no clear answer.
Good answer: "Persistent checkpoints in a database, separate long-term memory, clear retention periods."
5. Security: How do you stop agents from seeing data they shouldn't?
Why it matters: With access to your knowledge base, an agent must not reveal the CEO's salary to an intern.
Red flag: "We filter that in the prompt."
Good answer: "Access rights are enforced at retrieval level, every tool runs with the user's permissions, all actions are logged."
Bonus: Can I see a system in production?
Ask for concrete systems that run live, not demos. Ideally you can talk to a customer who has been operating the system for some time.
Conclusion
Providers who answer these questions concretely can build systems that go beyond a prototype. In our directory you can see for each entry whether and when we checked the basic details; the technical depth is something only you can check in conversation.