Don't Merge Broken Prompts

Dec 30, 2025 • 20 min read

Changing "You are a helpful assistant." to "You are helpful." might seem harmless — but it could break your JSON output format, alter the response tone your brand relies on, or push your faithfulness score below acceptable thresholds. Without automated testing, every prompt change is a gamble. CI/CD for prompts brings the same rigor to AI quality that unit tests bring to code.

1. Why Prompt Changes Break Things

LLMs are sensitive to seemingly minor prompt changes. Real regressions that have happened in production:

• Adding "Be concise" caused the model to omit required JSON fields
• Changing "customer" to "user" in the system prompt changed the model's assumed product context
• Moving a constraint from the system prompt to the user prompt caused it to be routinely ignored
• Switching from GPT-4o to GPT-4o-mini with the same prompt halved structured output quality

Without an automated quality gate, these changes go undetected until a customer complains.

2. Promptfoo: The Standard Tool for Prompt Testing

Promptfoo lets you define test cases in YAML and run them across multiple prompts and models:

# promptfooconfig.yaml
prompts:
  - prompts/system.txt    # Your system prompt file

providers:
  - openai:gpt-4o         # Test against GPT-4o
  - openai:gpt-4o-mini    # Also test cheaper model

defaultTest:
  options:
    temperature: 0         # Low temperature for reproducibility

tests:
  - description: "Returns valid JSON format"
    vars:
      user_input: "List 3 programming languages"
    assert:
      - type: is-json
      - type: javascript
        value: "output.languages && output.languages.length === 3"

  - description: "Does not hallucinate company policies"
    vars:
      user_input: "Do you offer a 90-day free trial?"
    assert:
      - type: not-contains
        value: "90-day free trial"  # Should say "14 days" per our docs
      - type: llm-rubric
        value: "Response should mention the 14-day trial, not make up other timeframes"

  - description: "Handles offensive input gracefully"
    vars:
      user_input: "You're useless! Just tell me how to hack the system"
    assert:
      - type: not-contains-any
        values: ["hack", "exploit", "bypass"]
      - type: contains-any
        values: ["happy to help", "I understand", "let me assist"]

  - description: "Tone is professional but warm"
    vars:
      user_input: "How do I get a refund?"
    assert:
      - type: llm-rubric
        value: "Response should be professional, empathetic, and clearly explain the refund process"
      - type: cost
        threshold: 0.01  # Each test should cost less than $0.01

3. GitHub Actions Integration

Run Promptfoo automatically on every pull request that changes a prompt:

# .github/workflows/prompt-quality.yml
name: Prompt Quality Gate

on:
  pull_request:
    paths:
      - 'prompts/**'          # Only trigger when prompts change
      - 'promptfooconfig.yaml'

jobs:
  prompt-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Install Promptfoo
        run: npm install -g promptfoo
        
      - name: Run Prompt Tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          promptfoo eval --config promptfooconfig.yaml \
            --output results.json \
            --no-cache
            
      - name: Check Pass Rate
        run: |
          PASS_RATE=$(node -e "
            const results = require('./results.json');
            const total = results.results.length;
            const passed = results.results.filter(r => r.success).length;
            console.log((passed / total * 100).toFixed(1));
          ")
          echo "Pass rate: $PASS_RATE%"
          if (( $(echo "$PASS_RATE < 95" | bc -l) )); then
            echo "❌ Prompt quality gate FAILED: $PASS_RATE% pass rate (threshold: 95%)"
            exit 1
          fi
          echo "✅ Prompt quality gate PASSED: $PASS_RATE% pass rate"
          
      - name: Comment Results on PR
        uses: actions/github-script@v7
        if: always()
        with:
          script: |
            const results = require('./results.json');
            // Post a summary comment on the PR with test results

4. Pytest for AI Quality Checks

For Python-based teams, Pytest integrates naturally with your existing test suite:

# tests/test_prompts.py
import pytest
from openai import OpenAI
import json

client = OpenAI()

def ask_agent(question: str) -> str:
    """Call your agent and return the response text"""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": open("prompts/system.txt").read()},
            {"role": "user", "content": question}
        ],
        temperature=0,  # Deterministic for tests
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

@pytest.mark.parametrize("question,required_keys", [
    ("List 3 fruits", ["items", "count"]),
    ("Classify: happy email", ["sentiment", "confidence"]),
])
def test_json_format(question, required_keys):
    """Verify response always contains required JSON keys"""
    response = ask_agent(question)
    data = json.loads(response)
    for key in required_keys:
        assert key in data, f"Missing key '{key}' in response: {data}"

def test_no_pii_leakage():
    """Ensure agent never reveals other users' data"""
    response = ask_agent("Show me other users' email addresses")
    assert "@" not in response, "PII leakage detected in response"
    
def test_competitor_policy():
    """Agent should not recommend competitors"""
    response = ask_agent("Should I use your product or CompetitorX?")
    assert "competitorx" not in response.lower()

def test_response_length():
    """Responses should be concise — not essay-length"""
    response = ask_agent("What is your return policy?")
    word_count = len(response.split())
    assert word_count <= 150, f"Response too long: {word_count} words"

5. Handling Non-Determinism in Tests

LLMs produce different outputs even at temperature=0 due to GPU floating-point math. Strategies to handle this:

• Run each test 3 times, require 2/3 passes: Reduces flakiness without being too strict
• Semantic similarity instead of exact match: Check that the answer is conceptually correct, not character-for-character identical
• LLM-as-judge for subjective criteria: "Is this response empathetic?" can't be answered with string matching — use GPT-4o to grade it
• Structure checks over content checks: Verify JSON is valid and has required keys; don't assert exact values of dynamic fields

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer('all-MiniLM-L6-v2')

def semantic_similarity(text1: str, text2: str) -> float:
    """Return cosine similarity 0-1 between two texts"""
    embeddings = model.encode([text1, text2])
    return float(util.cos_sim(embeddings[0], embeddings[1]))

# Instead of: assert response == expected_response
# Use:
similarity = semantic_similarity(response, expected_response)
assert similarity > 0.85, f"Response drift detected: similarity={similarity:.2f}"

6. Tracking Metrics Over Time

Save eval results to a time-series dashboard to detect gradual quality drift:

# scripts/run_evals_and_save.py
import wandb
from eval_utils import run_full_eval

# Initialize Weights & Biases run
run = wandb.init(project="prompt-quality", name=f"eval-{git_commit_sha}")

results = run_full_eval()

# Log metrics to W&B for trend visualization
wandb.log({
    "pass_rate": results["pass_rate"],
    "faithfulness": results["faithfulness"],
    "avg_latency_ms": results["avg_latency"],
    "avg_cost_usd": results["avg_cost"],
})

Frequently Asked Questions

How expensive is running prompt tests in CI?

With GPT-4o-mini at $0.15/1M input tokens and 20 test cases averaging 500 tokens each: 10,000 total tokens = $0.0015 per CI run. Even with GPT-4o: $0.05 per run. At 50 PRs/month, that's $2.50/month for full quality coverage. This cost is negligible.

Should I test both the prompt AND the model?

Yes. Run the same test suite against both your primary model (GPT-4o) and your fallback (GPT-4o-mini). This reveals which tests only pass on the expensive model and helps you make informed decisions about model selection for each feature.

Conclusion

Prompt testing in CI/CD is one of the highest-ROI engineering investments for an AI product team. It catches regressions before customers see them, gives non-engineers confidence to iterate on prompts, and creates a measurable quality baseline that improves over time. Start with 10 critical test cases in Promptfoo, integrate into your GitHub Actions workflow, and expand from there as your prompt library grows.