How to evaluate AI vendors as a non-technical founder (2026)

The 3 types of AI vendors you'll encounter

• Real AI engineering firms: They have shipped production AI systems. They can show you evaluation metrics, architecture decisions, and production incident postmortems. They ask hard questions about your data before promising outcomes.
• Prompt wrappers dressed as AI firms: Their “AI product” is an API call to OpenAI with a fancy UI. This isn't inherently bad — but they shouldn't be charging $50K for it.
• AI strategy consultants: They deliver slide decks, not software. They're useful for executive buy-in, useless for shipping.

10 questions to ask every AI vendor

1. Show me a production system you built and the metrics it drives. Real vendors have war stories. Fake vendors show you demos.
2. How do you evaluate whether the AI output quality is good? They should describe a test set, eval metrics, and regression testing. “We check it manually” is a red flag.
3. What happens when the LLM hallucinates? They should describe validation, fallbacks, and human-in-the-loop design. “We haven't had that problem” means they haven't shipped at scale.
4. How do you handle model version changes when OpenAI/Anthropic updates? Real vendors have versioned prompts and regression tests.
5. What model are you using and why? If they can only use GPT-4 and haven't evaluated alternatives, they're not sophisticated.
6. What will this cost to operate at 10x current scale? They should give you token estimates and cost projections. Vague answers mean they don't know.
7. Who owns the code and the models? You should always own your code and your fine-tuned models. Never sign a contract that transfers IP to a vendor.
8. What's your process for handling sensitive or PII data? They should describe data retention policies, encryption, and whether your data is used to train models.
9. How long will the first production version take? Honest answer: 4–12 weeks for a real production system. “We can demo in 2 days” means they'll demo, not ship.
10. Can I talk to 3 customers who used this in production? Any vendor worth hiring will have references. Refusal is a red flag.

Red flags to walk away from immediately

• They lead with the demo, not the problem they're solving
• They can't articulate what makes their approach different from calling the OpenAI API directly
• Their pricing is based on a percentage of “AI savings generated” — this is unverifiable and misleading
• They guarantee specific accuracy rates before seeing your data
• The demo is always internet-connected and cherry-picked inputs — ask to break it
• They recommend building fine-tuned models before establishing a baseline with prompting
• No mention of evaluation, monitoring, or incident response

What good AI vendor engagement looks like

A trustworthy AI engineering partner will:

• Start by understanding your data and use case before proposing a solution
• Recommend the simplest approach that solves the problem (often just prompting)
• Set up evaluation infrastructure before shipping anything
• Provide transparent cost projections with model alternatives
• Give you code ownership and full documentation from day one
• Show you production metrics, not just demo performance