AI Tools

Common AI Tools Mistakes

Mistaking Fluent Output for Verified Fact

A language model produces the most probable continuation, not a checked answer. A summary can read perfectly and still attribute a policy clause to the wrong section, invent a citation that resolves to nothing, or blend two similar figures into a third that appears nowhere in the source. The failure is quiet because the prose is confident and grammatical — nothing errors out when the model is wrong.

Treat every generated claim as a draft until something outside the model confirms it: the source document, a lookup in the system of record, or arithmetic you can redo yourself. The practical rule is to separate drafting from verification and give the checking step a named owner. A workflow where nobody owns verification eventually publishes a wrong figure to a customer.

Pasting Confidential Material Into a Hosted Tool

Consumer chat accounts and business workspaces frequently run under different data terms. On many consumer plans, submitted prompts may be retained and reviewed, and content may be used to improve the service unless you opt out; business tiers typically add retention controls and training exclusions. Dropping a contract, a merger memo, an unannounced product plan, or a customer list into the wrong tier is a disclosure, not a drafting shortcut.

Two habits prevent most of it. Keep sensitive material out of the prompt by replacing names, account identifiers, and figures with placeholders, then restoring them where they belong. And make the sanctioned path the fast one — an approved workspace with access to the same models people would otherwise reach for on personal accounts. Include meeting recorders and camera-based tools in that policy, since transcripts and screenshots carry the same content.

Reaching for a General Assistant on Work That Needs Grounding

A general assistant answers from everything it was trained on, which is the wrong knowledge base for a question whose answer lives in your own policy document, price list, or inventory record. Ask it to state your return window, or the current specification of a part you stock, and it has no way to look it up — so it returns a plausible answer shaped like your world.

Questions with a fixed answer need grounding: retrieve the relevant passages from your own corpus and pass them in the prompt, constrain the response to the fields you require, or route it to a rules engine when the logic is deterministic. Chat is a good surface for drafting and brainstorming and a poor substitute for a lookup. Match the mechanism to the question type before matching it to whoever is asking.

Ignoring Model Version Changes

Providers update models in place. A version that is topped up with a new release, or retired entirely, can shift tone, verbosity, formatting, refusal behaviour, and how literally a system prompt is followed — even when the request never changed. Automations that parse a fixed response shape are the most exposed: an extra preamble sentence or a renamed heading breaks the parser downstream.

Watch deprecation notices, pin a model version where the vendor offers one, and re-run a small regression set whenever the underlying version moves. Teams that assume yesterday’s result will reproduce on an updated endpoint learn about the change from a complaint rather than a changelog.

Automating a Task That Runs Twice a Year

Automation carries upkeep: a prompt to maintain, an integration to keep authenticated, an output format to re-verify after every model change, and one person who remembers how the thing works. For a job that runs a few times a year, that overhead can exceed the minutes it saves — and because the cadence is rare, nobody notices when it breaks.

Automate what is frequent, stable, and expensive in human time. Leave irregular, judgment-heavy work with a person who uses a model as a drafting aid. If you do automate a rare job, write down the manual fallback beside it, because a workflow exercised twice a year will be unfamiliar to everyone the first time it fails.

Skipping the Human Review Step

The tempting version of delegation is wiring the model straight to the output: the reply that reaches a customer, the commit that ships, the entry that updates a ledger. Models are non-deterministic by design — the same prompt can return a different answer on a second run — which makes unreviewed autopublishing a gamble on quality.

Place review where an error costs the most and sample where it costs the least. Anything public, financial, or touching health and legal standing, and anything that writes to a system of record, deserves a person at the gate. For lower-stakes flows, check a sample on a schedule and track the correction rate so you know whether the guardrail still earns its place. Send output to a draft state rather than to production until the evidence justifies more trust.

Storing Credentials Inside Prompts and Workspace Notes

An API key, a database password, or a session token pasted into a prompt becomes part of conversation history — logged, retained, and often visible to anyone with workspace access. Prompt libraries shared between teammates and connected integrations multiply the exposure. A credential living inside a prompt also cannot be scoped narrowly, rotated, or revoked independently of the tool that holds it.

Keep secrets in a secret manager or the platform’s credential store, inject them at runtime, and give every integration its own narrowly scoped key. Treat any credential that has appeared in a prompt, a saved chat, or a shared document as already exposed: rotate it and check what it was permitted to reach.

Assuming the Integration Exists

A tool that advertises integrations may still lack the one you need: a supported endpoint, write access rather than read-only, or a webhook that fires when a record changes. Vendor pages describe categories of connection while your system needs a specific authentication method, a field mapping, and a rate limit that survives peak volume.

Verify before committing a workflow. Make a real authenticated call against a sandbox, confirm which fields the tool can write back, and see what happens when a request exceeds the rate limit. Then hand-test the join between the two data models with a single record. The gap between “integrates with” and “integrates with yours” is where failed rollouts live; where no API exists, plan an export file and a manual step, and staff that step instead of pretending it does not exist.

FAQ

Is it safe to use AI tools with customer data?

It depends on the plan and the contract rather than the category of tool. Business and enterprise tiers generally provide configurable retention, training exclusions, and a signed agreement, while consumer accounts may keep prompts and use them to improve the service. Check the terms that attach to the plan in use, send the minimum data the task requires, and leave identifiers out when they are not needed.

How do I catch incorrect answers before they cause damage?

Confirm against a source outside the model: the original document, the system of record, or a calculation you can redo. Give that check a named owner instead of assuming whoever requested the draft performs it, and log the corrections so you can see which task types keep failing.

A workflow broke after a model update. What should we do first?

Re-run a representative sample of the affected tasks and compare the responses against the format your downstream code expects, looking for wording and structure drift before rewriting instructions. Pinning a version where the vendor supports it, plus a small regression set you rerun after every change, prevents the same surprise next time.

When is a chat assistant the wrong choice?

When the answer already exists in a document, database, or rule you control. A chat assistant replies from its training data, so it cannot know a value that changed this morning. Retrieval over your own content, structured extraction, or a deterministic rule handles those cases with something you can trace back to a source.