A field guide for dealers who are done buying "cool."
Paul, our COO, and I were driving to the airport, on our way to an AI conference in Atlanta. It was an unremarkable drive. Nothing about it suggested it would end up being the thing I remembered most from the whole trip.
At some point I mentioned a guy we both knew. "Well, he's nice," I said.
Paul paused. Then he said it: that's about the worst thing you can say about someone.
A person spends a lifetime building a hundred things worth saying about them. Sharp. Funny. Relentless. Kind in a way that costs them something. "Nice" is what's left when none of that came up.
I've thought about that sentence more than almost anything else from that trip.
Because I think that's AI right now.
Ask most people what their AI tool does and you get "it's cool." Or "it's new." Those are just "nice" wearing a different word. The vendors selling AI right now are mostly selling "nice." The ones worth paying for can tell you something sharper.
What does it actually do?
Before deciding whether a vendor's AI is any good, ask a simpler question: Which category does it belong to? Most AI tools fall into one of four categories, and knowing which one you're buying changes how you should evaluate it.
1. Replace. It does something a person used to do, full stop. A chatbot answering routine customer questions that used to tie up a rep in the business development center (BDC). An AI transcribing and structuring repair orders that a service writer used to type by hand.
2. Augment. It makes an existing person faster or sharper, without taking their job. An F&I manager who still runs the desk, but now walks in with a product recommendation already surfaced. A technician who still does the inspection, but dictates it instead of typing it.
3. Inform or report. It delivers information. It doesn't act, decide, or replace anyone. It just shows you something you couldn't easily see before.
Think of inform or report as AI that takes data you already have and turns it into something you can read and act on. A dashboard. A summary. A side-by-side comparison. A monthly report. The AI organizes the picture, but a person still makes the call. If it ever starts deciding things on its own, it's moved into a different bucket.
Here's what that looks like in practice, because it's easy to undersell. We use AI internally to build our 20 Group growth reports. Pulling the numbers, structuring them into something visual, applying the right logic to make the comparisons mean something. That took real setup. Understanding what the numbers actually meant. Fact-checking. A human still reviewing every version before it went out. It wasn't plug-and-play, and most tools that claim to do this won't fit your specific numbers or your specific rules without you doing real work to tell them what you actually want to see.
That's worth sitting with for a second, because "inform/report" sounds like the safest, lightest bucket. It isn't the AI you have to worry about deciding something wrong. But it's also not the AI you buy and walk away from. Someone still has to build the logic, still has to check the output, still has to know what "right" looks like well enough to catch it when the tool gets it wrong.
And here's the part that catches people off guard. AI doesn't say "I'm not sure." It will hand you a number that looks exactly like every other number on the page, formatted perfectly, stated with total confidence, and completely made up. It can misread a column, blend two data sets together, or invent a data point that never existed just to fill a gap. That's called a hallucination, and it isn't a rare glitch. It's a known behavior of how these tools work. A report that's mostly right and partly invented can be worse than no report at all, because nobody knows which part to distrust. That's why the human review isn't a nice extra. It's the job.
4. Save time or money, directly. The return is the whole point, not a byproduct. A pricing tool that tightens your inventory turn. A scheduling agent that fills cancellations before they cost you a service bay slot. If a vendor can't point to a specific number this tool moves, it doesn't belong in this bucket. It belongs back in "inform," dressed up to sound like more than it is.
Four buckets. Know which one you're buying before you sign anything, because the questions that matter change depending on the answer.
Could a human do this better, and is the time saved worth it?
This is the question almost nobody actually runs before signing a contract. It's really two questions stacked on top of each other.
First: is the AI actually better at this task, or just cheaper and faster at a worse version of it?
Second: even if it saves real time, does the time saved justify what it costs you, what it risks, and what it takes to keep an eye on it?
Sometimes the math is obvious. A compliance firm I heard about recently was set to hire two new associates. Instead, they brought in an AI legal research tool. A year in, it hasn't made an error they've caught, it doesn't get tired, it doesn't take a day off, and the two hires never happened. That's not a story about AI being better than people in general. It's a story about one specific, narrow, well-defined task that a machine turned out to do better than the alternative of hiring two junior people to do it by hand. The math worked, and it worked because someone ran it, instead of assuming.
Now the other side. Plenty of dealerships are buying automation for tasks a $15-an-hour employee already handles just fine. A scheduling tool that replaces a job nobody was struggling with. A chatbot answering questions your front desk person answered faster and better, because a customer wanted to talk to a person. In those cases, the AI isn't wrong, exactly. It's just solving a problem you didn't have, and the "time saved" is smaller than the setup cost, the monthly fee, and the new thing someone now must manage.
And there's a cost that never shows up on the vendor's ROI slide: the customer. Picture the service customer who has called your store for eight years and always gets the same person at the front desk. That person knows her car, knows her name, and gets her in on a Tuesday when the schedule says full. Then one day she calls and gets an AI avatar. She didn't ask for that. Nothing was broken. Her experience was already good, and you changed it anyway.
Some customers won't mind. Some will quietly take their next oil change, or their next vehicle purchase, somewhere that still answers with a person. You'll never see that loss in a report. It just shows up as a customer who stopped coming back. Before you automate anything customer-facing, ask the simplest question there is: did any customer ever ask for this? If the process works and your customers like it, replacing the person isn't an upgrade. It's a risk you took to fix a problem nobody had.
The tell is simple: if you can't say what specific, measurable thing got better, you didn't run the math. You just bought "cool."
Does it fix the problem, or clean up after it?
This is the question that separates a genuinely valuable AI tool from a well-marketed one, and almost nobody asks it, because almost nobody has thought to.
Most AI on the market right now is downstream. It watches for a problem after it already exists. A claim gets filed, and the AI flags it as suspicious. A customer walks off the lot, and the AI notices, later, that you should have sold them GAP insurance. That's real value. It's also, by definition, cleanup. The damage already happened. The AI just got there faster than a person would have.
Upstream is a different thing entirely, and it's rarer, because it's harder to build and less flashy to demo. Upstream means the problem never happens at all.
Here's a real version of that, from a story I heard at an industry conference. A grey market vehicle, one originally built for a different country's market and later imported and sold here, ends up with a warranty contract on it. Eighteen months later, a claim comes in, and it turns out the vehicle was never actually eligible for that coverage in the first place. Nobody caught it at the time of sale. When the case eventually went to arbitration, the question that mattered most wasn't what the customer did wrong. It was nothing. They bought a car from a dealer, signed a contract, and were told yes. The system that should have caught the eligibility issue didn't, and the mistake didn't surface until someone tried to use the coverage they'd paid for.
The fix people talked about wasn't a smarter way to review that claim after the fact. It was a check at the point of sale, before the contract is ever written, that flags an ineligible vehicle before anyone's money is on the table. Nobody has to catch the problem on the back end if it never gets the chance to happen in the first place.
That's the difference. Downstream AI makes you faster at catching problems. Upstream AI means there's nothing left to catch.
When a vendor pitches you their AI, ask them directly: does this stop something before it becomes a problem, or does it just help you clean up faster once it already is one? Most of them will tell you, honestly, that they're downstream. That's fine. It's still useful. Just don't let them sell you cleanup and call it prevention.
The questions to actually ask
Skip the sales deck. Ask these instead.
What does it actually do? Not "what can it do." What specific job is it doing, and which bucket does that fall into: replace, augment, inform, or save money directly?
What data is it trained on? A generic model trained on public data can't tell you anything about your specific dealership, your specific customers, or what happened in your finance office last Tuesday. Depth and specificity of training data is everything.
Predictive or generative, and does the vendor actually know the difference? Most "AI-powered" products are a thin layer on top of an existing tool. Genuine prediction requires a fundamentally different architecture, and a vendor who can't explain that difference clearly probably doesn't have it.
What is the vendor's data depth across the industry? A vendor with a few hundred dealers in their dataset can't see the patterns that show up across millions of transactions. Scale is the difference between a real pattern and a coincidence.
What does it say when it doesn't know? It will be wrong eventually. What happens then: does it guess with false confidence, or does it flag its own uncertainty?
Who authored the final decision? The system, a person, or a recommendation a person then approved? If you can't answer that clearly for a specific case, you don't have an audit trail. You have a black box with a nice interface.
How big was the sample it learned from? Four examples of anything is a fluke. A pattern needs real volume behind it before you trust it to make a call.
Does it fix the problem, or clean up after it? Ask this one directly. Most vendors will tell you honestly that they're downstream. That's fine to know going in.
What happens from day one to day ninety? Not the demo. The actual onboarding, the actual support response time, the actual point where something breaks and you need a human on the other end of a phone.
What comes off your plate, and what gets added to it? Every AI tool removes some work and adds some new work: monitoring it, reviewing it, correcting it. If nobody's honestly answered what that new work looks like, you haven't priced the tool correctly.
Could a person already do this well enough? Sometimes the honest answer is yes, and the tool is solving a problem you didn't have.
Is there a human in the loop for anything that matters? Before you sign anything, decide where your lines are. Which decisions will you let AI make on its own, which ones can it only recommend, and which ones does it never touch? There's no universal answer. It depends on how much risk your organization is willing to carry, and it should be decided before the demo, not during it. But more often than not, if a decision involves data transfer, credit, coverage, or money movement, letting AI make it without a person signing off isn't a feature. That's exposure.
What's the vendor's failure plan? Security practices, contract terms, what happens to your data and your workflow if the vendor disappears, gets acquired, or has a breach.
Ask what it does. Not just what it is.
"Cool" wears off. "New" wears off. In six months, your AI tool is just a tool, doing a job, in your business, and you'll have to answer for whether it was worth it.
So skip "nice." Skip "cool." Ask what it actually does. Ask if a person could do it better. Ask if it's solving something or just reporting on something that already went wrong.
The vendors who can answer those questions without flinching are the ones worth paying for. The rest are selling you new car smell.
Frequently Asked Questions
What's the most important question to ask an AI vendor?
What does it actually do. Not what it can do in theory, not what the demo shows, but which specific job it performs: replacing a task, augmenting a person, delivering information, or saving measurable time and money. If a vendor can't answer that plainly, that's the answer.
Is AI that only builds reports or dashboards a waste of money?
Not automatically. Reporting and analysis tools can be genuinely valuable, but they still require setup, human review, and someone who understands what the numbers mean well enough to catch mistakes. The risk isn't the tool, it's assuming it runs itself.
How do I know if AI is actually better than a person at a given task?
Ask what specific, measurable thing improves, and compare that improvement to the full cost: the subscription, the setup time, and the ongoing oversight the tool requires. If nobody can name the specific improvement, the tool probably isn't earning its cost yet.
What's the difference between upstream and downstream AI?
Downstream AI catches or flags a problem after it already happened, like fraud detection on a claim that's already been filed. Upstream AI prevents the problem from happening at all, like a check at the point of sale that stops a bad contract from ever being written. Both have value, but they solve different problems, and most AI on the market today is downstream.
Should there always be a human involved when AI touches customer data, credit, or money?
Yes. Anywhere a decision affects a customer's data, credit, coverage, or money, a human needs to be reviewing or authorizing the outcome, not just the AI. That's not a limitation of the technology, it's the difference between a defensible process and an unmanaged risk.
What should I ask about a vendor's data before I sign anything?
Ask what the model is trained on, how much of that data reflects your specific business versus generic public information, and how much depth the vendor has across the industry as a whole. A vendor with data from a handful of dealers can't see the patterns a vendor with data across millions of transactions can.
