How to Choose an AI Automation Agency

How to choose an AI automation agency comes down to proof, not promises. A good agency can show how it tests against your real examples, where a person approves high-impact steps, who monitors after launch, and what the product really is. Check those four things first. Walk away from vague answers.

This guide covers what to ask, what evidence to expect and what should worry you. Facts are as of Oct 2, 2026. Not legal advice.

Are you ready to talk to an agency yet?

Start on your side of the table. A vague brief gets a vague proposal. These are our suggestions, not a standard:

Volume. The task repeats often enough to matter. On our own service page we say that if it happens only a handful of times a year, doing it by hand is still the cheaper answer.
Pattern. Drafting, triage, classification, summarizing: work with a known "done" state.
Risk points. You can say where a person must sign off and what a bad output looks like.
Access. Your systems, tools and credentials can be opened for discovery.
Examples. Real samples, redacted if needed. Share as many as you can. Tests get written from them.
An owner. One person reviews pilot output, handles exceptions and makes the go or no-go call.

RAND's interview-based research on AI project failure (65 practitioners, published August 2024, not specific to LLM automation) lists a misunderstood problem and inadequate data among five root causes. It also describes leaders demanding AI where "a few simple if-then rules" would do. A good agency tells you when the answer is less AI, not more.

Readiness is mostly your homework: a clear task, open systems, real examples, one owner.

What should a good agency be able to show you?

Ask for evidence, not adjectives. Four things are worth seeing before you sign. Each has published guidance behind it. None of it binds a private buyer. Treat it as a model of good questions.

Testing on your real work. Anthropic's eval guide says to set success criteria that are specific and measurable, and to build evals that mirror your real task, edge cases included. OpenAI's eval guidance calls "vibe-based evals" an anti-pattern: trusting "it seems like it's working", or waiting until you ship to write any tests.
A plan for after launch. NIST's voluntary generative AI profile (AI 600-1, July 2024) separates pre-deployment testing from post-deployment monitoring. It also suggests "clauses in contracts which allow an organization to evaluate third-party GAI processes and standards."
Human approval where it counts. OWASP's LLM06 entry on excessive agency says to use "human-in-the-loop control to require a human to approve high-impact actions before they are taken." It also says to give extensions only the minimum permissions necessary.
Honest labels. Gartner uses the term "agent washing" for vendors rebranding existing assistants, RPA and chatbots as agents without substantial agentic capability. Ask what the thing actually is.

Ask to see the tests, the approval points and the monitoring plan before the demo.

What should you ask before you sign? A six-point scorecard

This is our own framework, built from the sources above and our view of the work. Run each agency through it. Mark every point clear, vague or missing. We do not add up a number on purpose. Several "missing" marks is your answer.

Scope. Ask: what exactly will you automate, and what counts as done? Listen for a written problem, success criteria and a named pilot boundary. Be wary of "AI strategy" with no deliverable.
Testing. Ask: how will you test it, on which of our examples, and when are the tests written? Listen for tests built alongside the work, with edge cases. Be wary of "we'll know it works when we see it."
Human control. Ask: where does a person approve before something high-impact happens, and what can the automation touch? Listen for approval points and least-privilege access. Be wary of an automation that can send, delete or pay on its own.
Data. Ask: which model provider, platform and endpoint will you use, under which retention setting, and is our data ever used for training? Listen for named settings and where they are written down. Provider terms differ and change. OpenAI's API documentation says API data is not used for training unless you opt in, and abuse-monitoring logs are kept up to 30 days by default. Zero Data Retention needs OpenAI's approval and does not cover every endpoint. Anthropic's documentation says that on Amazon Bedrock and Google Cloud the cloud provider is the data processor. Check the current pages.
Ownership. Ask: who owns the code, workflows, prompts, eval sets, accounts and hosting, and what may we use after the contract ends? Listen for answers that are written down. In US copyright law, a transfer of copyright ownership "is not valid unless... in writing and signed by the owner of the rights conveyed" (17 U.S.C. 204(a)). The UK government's IP guidance for its own contracts notes that the right to use IP is often more important than owning it. Ask counsel before you rely on any of this.
After launch. Ask: who monitors it, what happens when a model or API changes, and how are incidents handled? Listen for a named owner and a written plan. Also ask whether ongoing support is in this proposal or separate, and what exactly it covers.

How do you read a proposal?

Read it as if it will become the contract. Find the sentence that answers each of these. If you can't find it, ask.

Deliverables. What do you receive, in what form?
Exclusions. What is not included?
Pilot boundary. Which real, bounded tasks run first, and what must be true before it scales?
Ownership. Who owns and holds what, in writing?
Changes. How are mid-project requests estimated and approved?
Support. What happens after handoff?

This list is our own. Neither document below lists proposal contents: deliverables, exclusions, pilot boundary and change requests appear in neither. Both ask for related things. GOV.UK's 2020 guidelines for UK central-government buyers say to look for testing under a range of conditions and a defined acceptable model performance, to ask for knowledge transfer and training as part of the requirement, and not to expect a fully detailed plan up front. OMB's M-25-22 asks US federal agencies to favor terms on knowledge transfer, data and model portability and clear licensing, to reduce vendor lock-in. Neither binds you.

Cost sits inside every proposal, and we keep numbers out of this guide. For what drives AI automation cost, read our cost guide.

Which red flags should send you elsewhere?

No deliverable. A strategy deck with nothing you can test.
No tests. Performance claims with no account of how they were checked. The FTC's 2024 complaint against DoNotPay alleged it did not test whether its AI chatbot's output was equal to a human lawyer's.
Unlimited autonomy. No human approval before high-impact actions.
Guaranteed results. In March 2026 the FTC announced a settlement with Air AI, and a federal court entered the stipulated order that month. The FTC alleged Air AI made false or unsubstantiated claims about earnings and refund guarantees to small businesses, and the order bars earnings claims without adequate substantiation. Air AI sold business-opportunity packages, not typical agency work. The lesson still holds. See the FTC release.
Relabelled products. A chatbot sold as an "agent" (see Gartner above).
Layoffs first. Our view: if the pitch leads with headcount instead of guardrails and failure modes, keep looking.
Never says no. Our view: an agency that can't tell you when not to automate isn't advising you.

Agency, freelancer, no-code or in-house?

The scorecard works for all of them. A freelancer or an in-house hire should be able to answer the same six questions. Whether tools alone are enough depends on your volume, your risk and who can own the result. We cover that choice in full in Zapier or Make vs an agency.

How do we run projects at CD Sportswear Inc?

We work in five steps: find, design, build, pilot, scale. We build with evaluation from day one, and only what the pilot proved gets scaled. Every engagement is scoped in a written proposal. Requests that come up mid-project are estimated separately and added only with your sign-off. If it's not a fit, we'll say so. Details are on the page for how we run AI automation projects. Ask us the same six questions.

FAQ

How do I compare two agency proposals?

Run both through the same six questions: scope, testing, human control, data, ownership and after-launch support. Prefer the one that answers more of them in writing, not the one with the better demo.

What should I own after the build?

Decide it in writing before work starts, covering code, workflows, prompts, eval sets, accounts and hosting. In US copyright law a transfer of copyright ownership must be written and signed. We can't tell you what is typical, so ask counsel.

Do NIST, OWASP and the FTC bind my agency?

Not directly. NIST's profile is voluntary and OWASP's list is community guidance. The FTC enforces existing law: its Office of Technology said in January 2024 that "There is no AI exemption from the laws on the books."

Will AI automation replace my team?

No, not as we scope it. We target the repetitive load so your team keeps the judgment work. Treat that as our approach, not a forecast.

What should I put in a first brief?

The repetitive work and what "done" looks like, the systems involved, the risk points where a person must sign off, and as many real examples as you can share, redacted if needed. Add your timeline, who will review output, and a rough budget range if you have one.

Next step

If you have repetitive, high-volume work and a clear owner, see our AI automation service and tell us about it, or send a brief. If it's not a fit, we'll say so.