Building a Support Ticket Classifier from Scratch

Support teams face a daily challenge: tickets arrive without proper categorization. A billing question gets routed to technical support. A technical issue lands in sales. This manual work is slow and error-prone.

Most teams try one of two extremes: either they manually triage every ticket (expensive and inconsistent), or they implement a rigid rule-based system (fast but brittle). Neither scales well as the ticket volume grows.

The real problem isn't that automatic classification is hard. It's that most implementations conflate multiple concerns. They mix file I/O, parsing, schema validation, classification logic, API calls, and evaluation into one monolithic script. When something breaks, you don't know if it's your rules, your API call, or your parsing.

A support ticket classifier that separates concerns cleanly. Here's how it works:

• Reads from JSON and validates structure with Pydantic
• Routes through a rule baseline using keyword matching and ranked precedence
• Calls OpenAI API for cases where simple rules don't work
• Measures accuracy against labeled examples so you know if it actually works

The classifier handles the boundary between my Python code and external APIs cleanly. I own the schema, the rules, the decision logic. The API handles the heavy lifting for edge cases. And the evaluation gates tell me when the hybrid approach is paying off.

The system flows through six clear stages:

1. Load and validate: Read JSON ticket, parse with Pydantic, catch schema errors early
2. Rule baseline: Extract keywords, apply ranked matching, get first classification
3. API fallback: If confidence is low, ask OpenAI with a structured prompt
4. Log results: Record both predictions with timing and token usage
5. Compare with labels: Measure accuracy against human-labeled truth
6. Analyze errors: Understand where rules fail and where API adds value

Each stage is independent. You can test the rule baseline without ever calling the API. You can evaluate API performance on a dev set before running it on production data. The separation of concerns makes the system easier to debug and improve.

Building this classifier teaches four things that transfer to any AI application:

I built a full walkthrough showing the classifier handling real ticket scenarios. The video covers project setup, the rule baseline, API integration, and how to measure what actually works:

The full source code is on GitHub. It includes starter templates, test cases, and a reference implementation:

This classifier is more than a toy example. It teaches the real patterns you'll use in production:

• Hybrid systems (rules + AI) are how production apps actually classify data
• Schema validation catches errors before they hit your API
• Evaluation metrics turn guesses into facts
• Separating concerns makes debugging and improvement possible

If you can build this classifier, you can build any customer-facing AI system that needs to be reliable, fast, and cost-efficient.