To evaluate reconciliation software, run every candidate on the same set of your own statements and ledgers, with errors you planted and already know about. Count which errors each tool catches, which it misses and how many false alarms it raises. Then time how long a reviewer still spends finishing the work. Demos with vendor data prove little.
Building a test set to evaluate reconciliation software
A test set is a fixed bundle of real statements and matching ledger data that every candidate tool gets in the same form. It should look like your hardest month, not your easiest. Pick enough variety to expose weak spots, but keep it small enough that you know the right answer for every line.
What to put in the test set
Choose from your own files, with permission and with client data handled as you would in production:
- Three to five accounts that reflect your mix: a busy operating account, a quiet savings account, a credit card, and one account from a bank with an awkward layout.
- At least one scanned statement, preferably a poor one, such as a photo or a fax-quality copy.
- A statement with a year-end break, where the period runs from December into January.
- A multi-page statement with wrapped descriptions and a page break in the middle of the transaction list.
- The matching ledger export for each account and period.
Record the answer key first
Before any tool sees the data, reconcile it yourself and write down the answer key: the matched pairs, the reconciling items, the closing difference and every anomaly you know of. Store it where the people running the trial cannot see it until scoring. Without an answer key, you end up judging the tool by whether its output looks plausible.
How big is big enough?
A few hundred lines across all accounts is usually enough. More lines add time without adding insight, unless your real volume is very high and speed is part of the question. If speed matters, run a separate volume test with clean data.
Seeding known errors into a reconciliation software trial
Seeding means planting specific errors in the test data so you can see whether each tool finds them. Real data contains some errors already, but you rarely know them all. Planted errors give you a fixed list of things a tool should catch and a fair way to compare tools.
Errors worth planting
Use a copy of the data, never the original, and keep a record of each change:
- A duplicate line. Copy a payment so it appears twice in the ledger.
- A missing line. Delete one statement transaction from the ledger.
- A transposed amount. Change 1,540.00 to 1,450.00 in the ledger.
- A wrong-period entry. Move a ledger entry from the last day of one month into the next.
- A check number gap. Remove a check from the ledger so the sequence skips a number.
- Two equal amounts on different days. Make sure the pair can only be matched correctly by date or reference.
- A statement that does not foot. On a copy of one statement's data, change a closing balance so opening plus activity no longer equals it.
What each one tests
Each planted error tests a different ability. The duplicate and the check gap test anomaly detection. The missing line tests completeness. The transposed amount tests whether the tool proposes a near match or forces one. The wrong-period entry tests date windows. The equal amounts test whether matching is careful or greedy. The unfooted statement tests whether the tool checks the statement's own arithmetic before trusting it.
How do you score reconciliation software results?
Score reconciliation software by comparing its output with your answer key, line by line, in three counts: errors caught, errors missed and false alarms. Weight a missed error far more heavily than a false alarm. A false alarm costs a minute of review. A missed error can end up in signed financial statements.
A scoring sheet
The table is an example scoring sheet for two candidate tools on the seven seeded errors plus false alarms and reviewer time. Tool names and results are illustrative.
| Check | Tool A | Tool B | Weight |
|---|---|---|---|
| Duplicate line | Caught | Caught | 3 |
| Missing line | Caught | Caught | 3 |
| Transposed amount | Flagged as near match | Force-matched, not flagged | 3 |
| Wrong-period entry | Caught | Caught | 2 |
| Check number gap | Caught | Missed | 2 |
| Equal amounts, different days | Matched correctly | Matched to the wrong partner | 3 |
| Statement does not foot | Marked out of balance | Not checked | 3 |
| False alarms on clean lines | 9 | 2 | −0.25 each |
| Reviewer minutes to finish | 48 | 31 | for comparison |
Reading the result
Tool B looks better on false alarms and on time. It also failed four of the seven seeded errors: it forced a transposed amount into a match, missed the check number gap, paired equal amounts wrongly, and never checked whether the statement footed. Tool A raised more noise but missed nothing. In most accounting settings, Tool A is the safer choice, and its false alarms are a tuning question to raise with the vendor.
Scoring scanned statements separately
Score the scanned statement on its own row. Count how many extracted amounts and dates were wrong before any matching. A tool that matches well on clean data but misreads scans will fail exactly where you most need help.
Re-running after vendor tuning
Vendors often ask to adjust settings after a first run. That is fair, since thresholds and date windows are meant to be tuned. Allow one tuning round for every candidate, under the same rules, and score the second run as well as the first. Keep both results. A tool that only passes after heavy tuning by the vendor's own staff tells you how much setup work you will carry when your statements change, and whether your team could do that work without help. Write down every setting that changed between runs, so the final configuration can be reproduced if you buy.
Comparing reviewer time across reconciliation software candidates
Reviewer time is how long a person spends after the tool finishes, clearing flags, checking matches and signing off. It is the cost that remains once you own the tool, so it deserves its own measurement. Time it with a stopwatch, with the same reviewer, on the same test set for each tool.
How to measure fairly
- Use the same reviewer for every tool, or rotate reviewers so each tool gets each person once.
- Practice first. Give the reviewer a short session with each tool on separate data, so the timed run measures the tool, not the learning curve.
- Start the clock when the tool finishes, and stop it when the reviewer would sign.
- Note what took the time. False alarms, unclear explanations, missing source links and awkward screens each tell you something different.
Turning minutes into a decision
Reviewer minutes, multiplied by your monthly volume and a cost per hour, give the running cost of each tool beyond its price. That figure feeds directly into a payback estimate. The method for building one is laid out in reconciliation software ROI in hours, errors and risk.
Which reconciliation software evaluation shortcuts mislead you?
The shortcuts that mislead are the ones that let the vendor choose the conditions. A demo on prepared data, a trial on your cleanest account, or a comparison of feature lists all tell you what a tool can do on a good day. None tells you how it behaves on your awkward statements.
Shortcuts to avoid
- Vendor sample data. It was chosen because the tool handles it well.
- Clean accounts only. A quiet savings account reconciles in any tool.
- Feature checklists. Two tools can both claim "duplicate detection" and catch very different things.
- Judging by the dashboard. A tidy screen says nothing about whether the matches under it are right.
- One person's impression. The person who ran the trial may prefer the tool they learned first.
- Skipping the ledger side. If your books are in QuickBooks, test with a real export from it, and read reconciliation software for QuickBooks users to see where the ledger's own screen already does the work.
When an evaluation is not worth running
If you reconcile a handful of accounts a month and the work is already on time and reviewed, a full evaluation may cost more than it saves. A short trial on one awkward account is enough to see whether anything is gained.
Security and control checks before the reconciliation software decision
Before choosing reconciliation software, confirm how it protects the statements you upload and whether it fits your own control design. Security covers storage, access and retention of client data. Control fit covers who can prepare, who can approve and whether the tool records both.
Security questions
Ask where statements are stored, how they are encrypted in transit and at rest, who at the vendor can see them, and how long originals are kept. Ask how access is removed when someone leaves. A fuller list is in client data security questions for bank reconciliation apps.
Control questions
Map the tool against your existing cash controls. Can the preparer also approve? Can anyone edit after sign-off without a trace? Does the tool force a reason for overrides? The internal control questionnaire for cash gives a structured way to check that a new tool does not open a gap the old process had closed.
PreciseBankFlow is opening to CPA and audit firms in small groups, so a trial with your own statements is the way to test it against the scoring sheet above. You can request a place in early access.