Evaluation method
How We Evaluate CRM and Marketing Tools
Our method turns each regulatory or operational requirement into a test that a vendor must demonstrate in a trial account, and records the result on a four-level evidence scale. We publish the tests, not scores for named products.
In short
- Start from requirements, not features: each test traces to a rule or an operational need.
- Ask vendors to show, not tell: a live demonstration in a trial account beats a slide.
- Grade evidence, not marketing claims: demonstrated, documented, stated or not available.
- Weight the safeguarding tests highest and treat a failure there as disqualifying.
Scope and limits. This method is for buyers running their own evaluation. We have not run it against named products, and nothing here is a recommendation or a ranking.
From requirement to test
Each test on the criteria pages has three parts: the requirement, the test, and what passing looks like.
Example, illustrative only:
| Part | Content |
|---|---|
| Requirement | Self-excluded customers must not receive marketing (Gambling Commission SR code 3.5.3 for Great Britain) |
| Test | In a trial account, exclude a seeded customer who is in a running flow, then trigger the next step |
| Pass | The step is suppressed, the reason is logged and no message reaches the test address |
The criteria pages apply this to CRM platforms, marketing automation platforms and consent and preference platforms.
The evidence scale
| Level | Meaning |
|---|---|
| Demonstrated | Shown working in a trial account you controlled, with your own test data |
| Documented | Described in current product documentation, with a date, but not yet shown |
| Stated | Claimed by the vendor in a call, slide or questionnaire |
| Not available | The vendor confirms the capability does not exist |
Only "demonstrated" should count for safeguarding tests. A stated capability is a reason to ask for a demonstration, not a pass.
Weighting
We suggest three tiers. Tier 1 covers safeguarding and consent: exclusions, objections, product and channel permissions and send-time checks. We suggest treating a tier 1 failure as disqualifying unless you can place your own control in front of the tool. Tier 2 covers evidence and integration: audit trails, event delivery and data minimization. Tier 3 covers everything else, such as segmentation, templates and reporting.
Keeping demos honest
- Send the test script in advance and ask for each test to be run live.
- Use your own seeded accounts, not the vendor's prepared ones.
- Ask to see the log entry for every suppressed send.
- Record the product version and date of the demonstration.
- Write down anything the vendor says it will build, and treat it as not available until it ships.
What we never do
We do not accept payment, commission or free licenses from vendors, publish scores or rankings, or describe tests we have not designed and checked against a source. Our wider standards are on the editorial policy page.
Building your own test sheet
Keep one row per test with these columns: requirement and source, test steps, expected result, observed result, evidence level, product version, date and the name of the person who ran it. A shared sheet makes it easy to compare vendors on the same tests and to show later why a choice was made.
Example, illustrative only:
| Column | Example entry |
|---|---|
| Requirement | Opt-out honored across all tools |
| Steps | Opt out by email footer; check the text message tool ten minutes later |
| Expected | Text message tool shows the customer as opted out of the email product only |
| Observed | As expected |
| Evidence | Demonstrated, product version noted |
Re-testing after releases
Vendors change their products. Rerun the tier 1 tests after each major release and whenever a rule changes in one of your markets, and record the new result next to the old one. A tool that passed last year has not necessarily passed this year. All criteria pages and the explorer are in the tools and evaluation section.
Questions
Why not score products on a 10-point scale?
A score implies we tested each product the same way. We have not, so we publish the tests and let buyers produce their own results.
Can a tool that fails a tier 1 test still be used?
Only if you put your own control in front of it, such as a decision service that every send must pass, and test that control.
Sources
- Gambling Commission, LCCP Condition 5.1.12 - Direct marketing preferences, no date shown.
- Gambling Commission, LCCP 3.5.3 Self-exclusion - remote SR code, no date shown.
- Gambling Commission, LCCP Condition 5.1.11 - Direct electronic marketing consent, no date shown.
Every source was read on October 28, 2026. How we choose sources: data sources.
Safer gambling support is listed on the safer gambling page.