How to detect AI hallucinations inside n8n — RagMetrics node walkthrough

Source: Dev.to

Introduction

If you’re running LLM outputs through n8n workflows, you probably have no systematic way to verify what the model actually produced.

• Did it hallucinate?
• Did it stay grounded in your source data?
• Was the answer accurate?

We just launched a native n8n node for RagMetrics that solves this.

How it works

Three nodes are all you need to evaluate every AI output in your workflow:

Trigger → Edit Fields → RagMetrics Evaluation

RagMetrics node inputs

• question – the original user query
• answer – the model‑generated response
• ground_truth – the correct expected answer
• context – source documents for grounding evaluation
• conversation – session ID for grouping evaluations
• evaluation_group – your RagMetrics criteria configuration

RagMetrics node outputs

The node returns structured JSON containing:

• Criteria name (e.g., Accuracy, Hallucination, Grounding)
• Score – 1 to 5
• Detailed reasoning for the score
• Token usage for cost tracking

What you can do with the score

• Route outputs below a threshold to a human‑review queue
• Trigger Slack or email alerts when hallucination is detected
• Log every evaluation to your RagMetrics dashboard automatically
• Block downstream actions when quality is too low

Evaluation methods

Live AI Evaluation

Uses a pre‑configured Evaluation Group for consistent scoring across multiple evaluations. Ideal for production monitoring and batch processing.

Direct Evaluation API

Submit single question‑answer pairs for immediate scoring without an Evaluation Group. Perfect for ad‑hoc evaluations and quick testing.

Quick setup

1. Create a RagMetrics account at
2. Configure your judge model API key in the dashboard
3. Create an Evaluation Group and select your criteria
4. Add your RagMetrics API key to n8n credentials
5. Add the RagMetrics Evaluation node to your workflow
6. Map your fields and connect to downstream logic

Get started

• 📄 Node documentation:
• ⚡ Starter workflow (ready to import):

Contact

• Email:
• Phone: +1 917 767 4075