The LLM platform that cut building an extractor from a week to under an hour

An NLP/LLM platform for EvolutionIQ's product teams.

Problem: Back in 2023, creating a new LLM extractor of production-grade quality, with custom data labeling and evaluation, took up to a week of engineering time for us. From 2023 we were building LLM extractors and entity recognition and serving them in production, and each new extractor needed its own labeling and evaluation.
User: 3 product teams and 30+ engineers.
Decision: I spotted the problem, got leadership buy-in to fix it, and founded AI Labs, an internal R&D team that I hired and led. Among other areas of research, it built and ran the NLP/LLM platform, the tooling the company uses to build LLM extractors, with our own labeling and evaluation pipelines built in. We followed typical agile development, with sprints, planning and quarterly planning, and I set up how the team works with other teams. I acted as research product manager for those teams and promoted the tools and results so they would adopt them.
Outcome: Building a new extractor, labeling and evaluation metrics included, went from over a week to under an hour. It is simple enough that engineers and non-technical users, in particular product managers and subject-matter experts, can build one themselves, with just a 10 to 15 minute Q&A session about the extraction task. Everything else is automated: labeling, building extractor prompts, evaluating label quality, prompt iteration and publishing extractors. Product teams adopted the platform, it opened up new work, and AI Labs went on to more research, prototyping, and tooling projects.