Name: ashtonfeliciano/n8n-evaluations-runner
Author: n8n

n8n Evaluations Runner

When to trigger

Adding or modifying filter logic, keyword lists, validators, or resume maps
Refactoring a Code node's pure-function logic
Before merging a workflow change — regression guard
After a production incident — codify the failure into a test
"How do I unit-test n8n code?" / "How do I test without spending API credits?"

What this is (and isn't)

This is	This is NOT
A logic-only n8n workflow with mock inputs	An integration test against live APIs
No external calls except the final report send — runs in ~3 seconds	A performance test
Pass/fail/pending counts pushed to Telegram	A replacement for manual smoke tests of end-to-end flow
A regression net for pure JS in Code nodes	Coverage for HTTP / Gmail / Telegram node behavior

Cost: no API credits. Mock inputs, in-memory only. The Telegram report is the one outside call.

Architecture

[Manual Trigger] → [Run All Tests (Code node)] → [Format Report (Code node)] → [Send Report (Telegram)]

One workflow per logic surface. Re-run on demand.

Standard test runner Code node

const results = [];

function assert(suite, testName, condition, detail) {
  results.push({ suite, test: testName, pass: condition, detail: detail || '' });
}

// ============================================================
// Suite 1 — <name>
// ============================================================

// Copy the production function VERBATIM here (don't import — Code nodes don't share scope)
function passesJs10Filter(job) {
  // ... same code as production node ...
}

// Tests
assert('js-10', 'Clearance job rejected',
  !passesJs10Filter({
    title: 'Technical Writer',
    description: 'Clearance required.',
    location: 'remote'
  }),
  'clearance required must trigger rejection');

assert('js-10', 'Valid remote tech writer passes',
  passesJs10Filter({
    title: 'Technical Writer',
    description: 'Write API documentation for cloud platform.',
    location: 'remote'
  }),
  'Valid remote technical writer should pass all filters');

// For not-yet-implemented features:
results.push({
  suite: 'js-10',
  test: '30-day age cutoff (PENDING — backlog)',
  pass: null,
  detail: 'Feature not yet implemented. Add postedAt date check, then re-run.'
});

// ============================================================
// Summarize
// ============================================================

const passed  = results.filter(r => r.pass === true);
const failed  = results.filter(r => r.pass === false);
const pending = results.filter(r => r.pass === null);

return [{ json: {
  summary: { total: results.length, passed: passed.length, failed: failed.length, pending: pending.length },
  results, failed, pending
}}];

Report formatter

const { summary, results } = $input.first().json;

const passIcon = '✅';
const failIcon = '❌';
const pendIcon = '⏳';
const topEmoji = summary.failed === 0 ? '🟢' : (summary.failed <= 3 ? '🟡' : '🔴');

const lines = [
  `<b>${topEmoji} <NAME> — Logic Eval</b>`,
  `${passIcon} ${summary.passed} passed  ${failIcon} ${summary.failed} failed  ${pendIcon} ${summary.pending} pending`,
  ''
];

const suites = {};
for (const r of results) (suites[r.suite] = suites[r.suite] || []).push(r);

for (const [suite, tests] of Object.entries(suites)) {
  const suitePass = tests.every(t => t.pass !== false);
  lines.push(`<b>${suitePass ? passIcon : failIcon} ${suite}</b>`);
  for (const t of tests) {
    const icon = t.pass === true ? passIcon : (t.pass === false ? failIcon : pendIcon);
    lines.push(`  ${icon} ${t.test}`);
    if (t.pass !== true) lines.push(`    <i>→ ${(t.detail || '').substring(0, 140)}</i>`);
  }
  lines.push('');
}

const now = new Date().toLocaleString('en-US', {
  timeZone: 'UTC', dateStyle: 'short', timeStyle: 'short'
});
lines.push(`<i>Run: ${now} UTC</i>`);

return [{ json: { message: lines.join('\n') } }];

Telegram send node (env-var pattern, no n8n credential)

{
  "method": "POST",
  "url": "=https://api.telegram.org/bot{{ $env.TELEGRAM_BOT_TOKEN }}/sendMessage",
  "sendBody": true,
  "specifyBody": "keypair",
  "bodyParameters": {
    "parameters": [
      { "name": "chat_id", "value": "={{ $env.TELEGRAM_CHAT_ID }}" },
      { "name": "text", "value": "={{ $json.message }}" },
      { "name": "parse_mode", "value": "HTML" }
    ]
  }
}

What to test (priority order)

Reject path coverage — every reason a record can be filtered out
Boundary conditions — at-threshold, just-below, just-above
Invalid-input fallbacks — undefined, null, 0, 'None', wrong type → expected default
Known-good real data — at least one example from a past confirmed-working run
Known-bad real data — at least one example that caused a past incident
Substring vs exact matchers — e.g., 'HR Dept' vs 'Human Resources Department'
Pending features — written as pass: null so they show up in reports as backlog

Test naming convention

<suite>: <human-readable assertion> with the detail field naming the underlying rule or past-bug reference:

assert('js-22', 'Human Resources Department rejected',
  !isValidContact('Human Resources Department'),
  'BAD_CONTAINS_CONTACT substring match must catch this (Session 5 bug fix)');

The "convert past bugs to tests" rule

Every entry in your Lessons Learned Library that names a specific input → wrong behavior is a test waiting to be written.

Example from a real lesson:

"Aggregate Sector Quotes was computing 1-MONTH change instead of 1-DAY because meta.previousClose returned chart-start price on range=1mo."

→ Test:

assert('aggregate', '1-day change computed from closes[], not meta.previousClose',
  computeChange({
    chart: { result: [{ meta: { previousClose: 100 }, indicators: { quote: [{ close: [100, 98, 99, 105] }] } }] }
  }) === ((105 - 99) / 99) * 100,
  'meta.previousClose can be unreliable; must use closes[-2]');

When to run

Before any change to filter logic, keyword lists, validators, resume maps
After the change → confirm no regressions
After any LL Library entry is added → write the test that would have caught it
Pre-deploy of any new workflow version

Status icon meaning

Icon	Meaning
🟢 (green)	0 failures — this suite passes; deploy only if your other checks pass too
🟡 (yellow)	1-3 failures — deploy blocked until fixed
🔴 (red)	4+ failures — deploy blocked; likely a broader break

Pending (⏳) does not count toward red/yellow — it's tracked separately as backlog. Pending tests are coverage gaps: list them in the deploy note.

Anti-patterns

Anti-pattern	What to do instead
Test against live APIs	Mock inputs only — eval suite is logic-only
Skip pending features (lose visibility)	Add as `pass: null` with detail explaining what's missing
Import production code	Copy verbatim — Code nodes don't share scope
One giant test function	One `assert()` per assertion — readable report
Drop the test after a bug is fixed	Keep it forever — that's the regression guard
Only test happy path	Reject path coverage is the highest-value tests
Run only after a change	Run before AND after — confirms you didn't introduce regressions
Hardcode chat ID / token in workflow JSON	Use `$env.TELEGRAM_BOT_TOKEN` / `$env.TELEGRAM_CHAT_ID`

Pre-deploy checklist

Eval workflow runs in < 10 seconds
All suites grouped in report
Production code copied verbatim (not paraphrased)
At least 1 test per reject path
At least 1 "known-good real data" test per suite
Pending features represented as pass: null with detail
Report fits in one Telegram message (< 4000 chars)

Reference

n8n Code node docs: https://docs.n8n.io/code/code-node/
Telegram Bot API formatting: https://core.telegram.org/bots/api#formatting-options

ashtonfeliciano/n8n-evaluations-runner

What is n8n-evaluations-runner?

Ask in your favorite AI

Documentation