AI solutions engineering · LLM evaluation · agent infrastructure
Joe Laskowski
I build the systems that decide whether AI output is any good, and then I build the things that deliver it. 25 years in IT. Right now I'm the human in the loop for a pair of production LLM assistants, which means I wrote the rubrics, coded the failure taxonomy, shipped the chatbots the technicians actually use, and then measured whether any of it moved.
Outside the day job I've been doing the same work unsupervised for years. Retrieval infrastructure, orchestration layers, evaluation tooling, and a lot of code nobody asked me to write. Nearly all of it built with AI in the loop, which I'd rather say out loud than bury. Getting these systems to produce tested, production-grade work is the skill, and it happens to be the one I get paid to evaluate.
- Now
- AI Quality Analyst, Aunalytics
- Based
- South Bend, Indiana
- In IT
- 25 years
Experience
AI Quality Analyst
March 2026 — Present- I'm the human in the loop for 2 LLM-powered ticket enrichment assistants. I wrote the 5-dimension rubric we score them against, plus the coded failure taxonomy that turns a bad answer into something the data science team can act on. The finding that's held up for 5 months: the model finds the right documentation and then stops short. Synthesis is the problem, not retrieval.
- I designed and built the Microsoft Teams chatbots the technicians use to reach both systems. Azure Functions and Copilot Studio over an in-house LLM, with live third-party data wired in through Power Automate. Two near-identical bots got refactored down into one shared core at 95% statement coverage.
- Technician documentation quality is up 29% across 18 weekly coaching cycles, and the share of tickets carrying a usable resolution entry has more than doubled. Those notes are what the model trains on, so it compounds.
- Built the parser and reporting that measured a new automated remediation feature through its first week in production: 493 tickets, 96% script success, 61% of them closing with zero technician touch, 3.45 TB of disk reclaimed. The automation was another team's build. The instrumentation and the analysis were mine.
- Caught a data-export defect that had been quietly misstating a reporting population by roughly 10x. I suspended the trend comparisons, rebuilt the baseline, and published the corrected numbers. The old numbers looked better.
- Kicked off our standard-resolutions program with a discovery script I wrote that mined 134,000+ tickets across 15 months into 115 recurring types, then worked with a small team to turn the highest-volume ones into published resolutions. I also built the docx-to-SQL parser that loads them into the Azure SQL knowledge base the assistants read at enrichment time.
- Wrote the operating standards for the whole function: scoring rubrics, failure codes, documentation style guides, and a 12-KPI measurement framework. Also the Azure architecture proposal for turning the function into a product.
Internal Support Engineer
August 2023 — March 2026- Built the company's user onboarding and offboarding automation end to end. 9 PowerShell modules covering Active Directory, Microsoft 365, and provisioning driven straight off the ticket, with certificate-based auth from Key Vault, schema-validated config, separate prod and test environments, and a full Pester suite.
- Built zero-touch Windows Autopilot deployment: certificate-only auth, a resumable checkpoint state machine with exponential backoff, and automated cleanup of stale Intune, Autopilot, and Entra records.
- Wrote the quarterly privileged-access review tooling. Exporter, risk flagger, and a quarter-over-quarter diff, plus a Microsoft 365 tenant audit module carrying CIS and vendor baselines.
- Linux subject-matter expert for the Centralized Services team. Also worked part-time with the security team as a SOC analyst on internal and client work.
- Tier 1 and Tier 2 support for internal staff, and a hand in most organization-wide rollouts.
Service Desk Location Lead — Kalamazoo
January 2022 — August 2023- Led the Kalamazoo team through Tier 1 and Tier 2 delivery: incident management, escalation, SLA maintenance. Owned staff performance, mentoring, and training.
- Co-lead engineer for internal departments. User access administration, Microsoft 365 licensing, and whatever engineering support came up.
Service Desk Lead
October 2019 — January 2022- Led a team of technicians and engineers on the service desk's largest accounts, including environments north of 1,500 users.
- Owned the customer relationship through their IT point of contact. Service delivery, escalation communication, ticket follow-through.
Managed Services Technician
March 2017 — September 2019- Closed 70 to 100 tickets a week on remote first-level support, working complex hardware and software faults and escalating what needed it.
Field Technician Manager
July 2007 — March 2017- Ran day-to-day operations for break/fix technicians nationwide. Recruited and certified more than 200 techs doing warranty repair for Dell, Sony, IBM, HP, and Lexmark.
- Managed dispatch, parts logistics, and vendor relations. Kept the company intranet and public site running.
Before that — field and deployment technician work through the 2000s. Warranty repair for Dell and the rest of the OEMs, and a multi-site enterprise rollout where I supervised a small team through 6,000 machines. It's on this page because knowing what the tech on the other end of the ticket is looking at is why the AI work lands.
Selected work
Personal projects, built on my own time. Several ship under AutoScript Studio LLC, an Indiana company I started to hold the work that outgrew being a hobby.
cowork-rag + cowork-graph
Retrieval infrastructure · MCPTwo search systems over the same pile of my own documents. One is semantic, one is structured, and both are wired into Claude, Codex, Grok Build, and Hermes over MCP. The host brings the model. These bring the evidence. It all runs on my own hardware, so there's no API key and no monthly bill.
- Retrieval
- Nomic embed-text-v1.5 (768-dim) + MiniLM-L-6-v2 cross-encoder, both in-process via FastEmbed ONNX · ChromaDB · SQLite FTS5
- Generation
- Provider-neutral, defaulting to local llama.cpp Qwen3-8B on a nightly systemd timer
- Index
- 2,651 files · 98,242 chunks · 2,573 summaries · 38,345 graph entities in 150 communities
- Surface
- 15 MCP tools · ~6,900 lines of Python, 1,800 of it tests
Read more
Hybrid retrieval. A disk-backed FTS5 keyword lane runs next to the vector lane, and the two get fused by reciprocal-rank fusion. Embeddings are famously bad at exact identifiers, so case numbers and filenames and account IDs stay findable. A cross-encoder then reranks the fused pool in-process.
GraphRAG sits on top. An LLM pulls entities and relationships out of each document, cached by content hash, merges them into a graph, clusters that into communities, and summarizes each one. Relational questions and big "what's going on across the whole corpus" questions get their own tools.
It knows about time and domain. Chunks carry the date ranges they mention, so asking about a specific year filters automatically. Every chunk is tagged with its top-level folder, and a scope argument keeps fiction from contaminating a factual answer.
Sync respects my compute. Change detection runs in 2 passes, a quick mtime scan and then a content-hash confirmation, because a folder syncing between machines rewrites timestamps constantly and I got tired of re-embedding files that hadn't changed. Embedding reuse is chunk-level. Edit one paragraph of a big document and exactly one chunk gets re-embedded.
Everything else is local too. Tesseract pulls text out of scanned documents while Florence-2 writes a description of standalone images. There's an optional PII detector that scans at ingest and masks identifiers in tool output before they reach a host's context. It also refuses to save a memory note containing a raw one.
cowork-graph handles what embeddings handle badly. It's a SQLite knowledge graph parsed out of the same markdown, and it answers structural questions: what's active, what's blocked, what links to what, what got decided and when. Markdown stays canonical in both systems. Every index is derived state I can throw away and rebuild.
PixelDrops production pipeline
Automated documentary studio · 5 toolsI hand it a research question and it hands back a finished, published documentary. Five tools pass work down a chain, a runner carries a per-video manifest between them, and an agent team drives the whole thing, stopping at 7 human approval gates where I actually need to look at something. Everywhere else it runs unattended.
- Chain
- pixeldrops-research-mcp → scriptcast-cli → shotlist → framecast-cli → yt-publish-cli
- Orchestration
- Manifest-as-baton runner · 7 gate types with policies · stub-skip, manual-halt and auto-pass advancer · spend mirror
- Render
- Remotion · 1920×1080 H.264 at 30fps · roughly 1–2 minutes of render per minute of video
- Economics
- ~$0.38 of narration per 30 minutes · ~150 image generations per episode · 4 channels · 5 episodes shipped
Read more
pixeldrops-research-mcp conducts. It owns the per-video manifest, the stage machinery, and the gates: packaging, angle, script, shot plan, asset, QC, final cut. Each gate has its own policy, confidence, and exception path. An advancer walks the chain, skips stubs, halts for me where the policy says to, auto-passes where it doesn't, and mirrors spend as it goes. The research itself wraps Gemini, Perplexity, and OpenAI.
scriptcast-cli turns an approved script into narration and timing. A 7-rule priority chunker splits text for synthesis without ever breaking mid-sentence or inside an emotion tag. Emotion tags get passed through to the voice model, production markers get stripped before anything can read them out loud. WhisperX then force-aligns the rendered audio back against the script and produces sentence-level beat timestamps. Everything downstream keys off those.
The shot planner reads the script and the beats and emits a plan for every cue: a primary layer that's either a generated image or a scene authored in code, plus overlays. Data motif, citation lower third, kinetic type, or code composited over image. That plan fans out into the artifacts each tool consumes, and I sign off once instead of approving 150 decisions one at a time.
shotlist is where I spend most of my time. It parses a markdown prompt sheet into a grid of assignable slots. Click a slot and the prompt lands on the clipboard, paste it into an image model, drop the result back, click to assign, and the file gets renamed and tracked. No database, no auth, the filesystem is the source of truth. One more click deploys config, beats, images, and audio into the renderer and registers the composition.
framecast-cli assembles in Remotion, and it takes 2 completely different inputs through the same path: generated stills, or scenes written in code. Ken Burns motion in 4 patterns, crossfade and zoom-dissolve transitions, spring-animated title cards, lower thirds fired on beat boundaries, kinetic typography, and a per-era color grade and film texture so 1995 material doesn't look like 2025 material. An LLM classifies the era way upstream and the value rides through the entire chain.
The data motifs are my favorite thing in here. An authoring step pulls figures out of the researched script, and framecast turns them into animated overlays bound to specific beats. Counters, node graphs, dials, with per-cue overrides. The components are a discriminated union, and the architecture made a promise: adding a new motif variant should cost 1 union member and 1 switch case, with nothing existing getting reshaped. That's held 3 times now. The node-graph variant went in and its sibling components came out byte-identical.
Citations get the same treatment. On-screen source attributions authored per episode, resolved by ID instead of position, rendered as lower thirds that coexist with motifs on the same slide. yt-publish-cli closes it out with caption generation and the publish push. All 4 channels carry a full kit of themes, motion profiles, signature motifs, and transitions.
Nexus Signals
Market analysis platformA market analysis platform for crypto and stocks. I got about halfway into the first version, saw exactly how I should have built it, and started over. This is the second one, and it's the cleanest architecture I've written.
- Backend
- Python 3.12 · FastAPI · SQLAlchemy 2 async · PostgreSQL 16 · Redis 7
- Frontend
- React 19 · TypeScript · Vite · TradingView lightweight-charts
- Architecture
- DDD · Clean Architecture · CQRS · 6 bounded contexts · 43 versioned REST endpoints
- Rigor
- 3,764 passing tests (3,043 backend, 721 frontend) · mypy --strict, 0 Any types
Read more
Seven market data providers sit behind circuit-breaker fallback chains. When a vendor rate-limits or falls over, the breaker opens and the next one picks it up. Looks like overkill right up until the day a provider goes dark mid-session.
Twenty-three technical indicators feed a normalized −10 to +10 signal score. LLM sentiment and directional analysis run alongside the quantitative signals.
There's a full paper-trading context with portfolios, orders, positions, and live P&L, so a strategy can run against streaming data over WebSocket without any money on the table.
It's parked. I pivoted to side projects that had some chance of generating revenue. The code's still there and I still like it.
Claude Sentient
Agent orchestration layer · publicAn autonomous development loop for coding agents, built back when the tooling didn't ship one. Seven stages with hard quality gates between them, language-aware profiles, and a mechanism that watched me override it and proposed edits to its own rules. I ran it for months and then took it back out of everything.
- The loop
- INIT → UNDERSTAND → PLAN → EXECUTE → VERIFY → COMMIT → EVALUATE
- Surface
- 19 slash commands · 15 hooks · 9 agent roles · 9 language profiles · 12 JSON schemas
- Rigor
- 897 tests across profiles, hooks, commands, schemas and integration, all passing
- Where
- Public — github.com/thebiglaskowski/claude-sentient
Read more
Quality gates are the spine. Lint, test, build, and git state get checked between stages, with auto-fix where auto-fix is safe, and the loop won't advance past a red gate. Nine language profiles detect their own environment, including Python across conda, venv, poetry, and pdm, so the gates run the right commands without being told.
Twelve JSON schemas enforce cross-module integrity, with a suite asserting that profiles, agents, and gates stay structurally consistent with each other. With this many moving parts, those tests are what stop a rename in one place from quietly breaking 3 others.
It learned from being corrected. A self-improvement mechanism watched for cases where I overrode its behavior and proposed amendments to its own rule files.
Then I retired it. The platform kept shipping native features that did what my abstraction layer did, and my abstraction layer stopped helping and started getting in the way. I pulled it out of every project and went vanilla. Building it taught me how these systems behave under load. Knowing when it had stopped earning its keep was the harder call, and I've left the repo public because both halves are worth reading.
NexusLedger
Multi-entity finance systemOne ledger covering my personal accounts and 2 companies, where the entities can't bleed into each other and a closed month can't quietly move underneath a report that already quoted it. Plaid in, rules-based categorization, reconciliation, reports out. Built assuming I'll eventually make a mistake.
- Stack
- Python CLI · Plaid · SQLite · rules in YAML · workbook and Sheets output
- Scope
- 3 legal entities — personal, an Indiana operating company, a Wyoming holding company
- Correctness
- Checksummed migrations · period locks · pending-to-posted carryover · synthetic transfer pairs · integrity-check doctor command
- Rigor
- 662-test Python suite
Read more
Period locks mean a closed month can't change under a report that already quoted it. Checksummed migrations mean a schema change that didn't apply cleanly fails loudly. Pending-to-posted carryover handles the window where one transaction exists twice in 2 different states. Synthetic pairs keep a transfer between entities from reading as income on one side and an expense on the other.
A doctor command runs integrity checks across the whole ledger and reports drift before it lands in a report, which is well before somebody makes a decision off a bad number.
Categorization learns. Rules live in YAML, so adding a pattern re-categorizes matching historical rows on the spot, with a full recategorize pass available when the rules change in a big way.
This replaced an earlier Flask and Plaid budgeting app I built, used for a while, and outgrew. That one taught me what the problem actually was.
PromptVault
Prompt compiler and semantic libraryA compiler for prompts. Rough idea in, structured prompt out, and everything it generates lands in a searchable library that gets more useful as it fills up. I built it because large parts of prompt engineering are clearly a compilation problem that people keep treating as a craft.
- Stack
- Node 22 · pnpm monorepo · Fastify · Next.js · PostgreSQL 16 + pgvector · Redis 7 · Drizzle ORM
- Retrieval
- HNSW vector indexes for semantic search across the library
- Content
- 92 system cards (28 image, 32 LLM, 32 code) · 27 templates
- Rigor
- 30-case regression suite diffing compiler output against stored baselines
Read more
A card and template engine composes prompts out of reusable structured pieces instead of string concatenation, and a multi-pass critic reviews and revises the assembled result before it ships.
The regression suite is the load-bearing piece. Thirty cases run against stored baselines, so an edit to the compiler or the cards can't silently degrade output. That's the failure mode that makes most prompt tooling untrustworthy after a few months of tinkering.
Single user by design. The API key gets passed per request from the browser and never persists server-side, and I reach it across devices over a tunnel.
Honest status: I don't reach for it as much as I expected to, partly because the models got better at the thing it compensates for. It's still genuinely handy when a prompt has to be exactly right.
SEO content pipeline
13-package monorepoA whole content operation as software, from seed keyword discovery through published pages. It ranks candidate keywords by expected revenue before a word gets written, and the last stage grades finished pages on whether an AI search engine would actually cite them.
- Stack
- TypeScript · pnpm workspace · 13 packages
- Chain
- Seed discovery → gap analysis → revenue-per-keyword scoring → competitive briefs → content and image generation → static build → GEO scoring
- Providers
- OpenAI · Anthropic · DALL·E 3 · Stability AI · Serper.dev SERP data
- Status
- Shelved. The working name is taken and it needs a rebrand before it goes anywhere
Read more
Revenue-per-keyword scoring is what keeps it from being a content mill. Volume and difficulty are cheap to get. What matters is expected value per page, and the pipeline ranks candidates on that first.
GEO scoring (generative engine optimization) grades finished pages on citation-readiness across ChatGPT, Perplexity, Google's AI surfaces, and Claude. The bet is that getting quoted by an answer engine matters more every year than ranking on a page of blue links. I built that stage before the acronym was in common use, which was either good timing or a lucky guess.
Copy and imagery get generated in the same pass, copy from a mix of models and illustration from 2 image providers, then assembled into a static site with a web dashboard front-ending the toolchain.
DropTok
TikTok commerce platformEnd-to-end operations for TikTok Shop, running against the live marketplace APIs. Product research, listing generation, creative, creator outreach, and fulfillment, all in one place. It handles the parts of that business that are actually hard.
- Stack
- TypeScript end to end · npm monorepo · React 18 / Vite 6 / Tailwind / Zustand · Express 4 / Drizzle / better-sqlite3
- Integrations
- TikTok Shop Seller API · TikTok for Business · TikTok Ads · CJ Dropshipping · AutoDS
- AI
- Anthropic SDK for listing generation, the content pipeline, and opportunity scoring
- Ops
- node-cron scheduling · Vitest · token-spend dashboard
Read more
An AI-scored opportunity board and a trend radar rank products against each other, so what I open in the morning is a ranked shortlist.
The listing generator writes titles, hooks, and descriptions, and a separate content pipeline produces video hooks, scripts, and shot lists. That's the creative bottleneck, automated far enough that I'm editing.
Fulfillment is an explicit state machine. Order states in dropshipping have real transitions and real dead ends, and modeling them as a status column is how orders go missing.
A token-usage dashboard tracks my own model spend against the operation. That number is the difference between a tool that's profitable and one that just looks busy.
SKYNET backup
Encrypted backup and alertingEncrypted, deduplicated, snapshot-preserving backups across 3 redundancy layers on 2 machines, with alerting on every single run. I built it after admitting that I'd never once tested a restore, which meant I didn't really have backups.
- Stack
- restic · systemd timers · Windows Task Scheduler · Slack webhooks · Bash
- Layers
- Live working copy → local encrypted repository → off-site cloud
- Alerting
- Green/yellow/red to a dedicated channel on every run, fail-only on prune and verify
- Verification
- Restore round-trips confirmed byte-identical by SHA-256
Read more
Two machines, two schedulers, one alert channel. A systemd timer covers the Linux side and Task Scheduler covers Windows, and both report into the same place, so a silent failure on either surface shows up the next morning.
It alerts on success too, at least while I was still learning to trust it. A backup system that only speaks up when it breaks looks identical to one that quietly stopped running 3 months ago.
The restores are tested. Both surfaces have been round-tripped and verified byte-identical by hash. Almost everybody skips that step, and it's the only one that proves the rest of it works.
xrp-ticker
Terminal portfolio tracker · publicA keyboard-driven terminal dashboard that reads balances straight off the XRP ledger over WebSocket, aggregates them across multiple self-custody wallets, and prices them live. Small, finished, and the direct descendant of a decade of ricing my terminal.
- Stack
- Python 3.11+ · Textual · uv · XRPL WebSocket · Coinbase API
- Features
- Multi-wallet aggregation · USD valuation · 24h market stats · auto-reconnect
- Interface
- 4 cyclable sparkline renderers (blocks, braille, dots, line) · 2 themes · fully keyboard-driven
- Where
- Public on GitHub
Read more
It only works against self-custody addresses. Exchange deposit addresses sit inside shared omnibus wallets, so querying one tells you about the exchange's pooled balance and nothing about your holdings. The app tells you so. I learned that by building it and watching it lie to me once.
Live on both axes. Price comes from an exchange API, balances come off the ledger over a WebSocket subscription, and the two reconcile into a valuation that updates without a refresh key.
There's also a novel I drafted with AI tooling under a pen name, with an Amazon launch pending. Turned out to be a pipeline problem in a different costume.
Toolset
- AI & LLM
- LLM output evaluation and rubric design · prompt and agent-instruction engineering · MCP server development · RAG and GraphRAG pipelines · embedding, reranking, hybrid retrieval · Microsoft Copilot Studio · Anthropic and OpenAI APIs · agent orchestration
- Languages
- PowerShell · Python · TypeScript · JavaScript · SQL · Bash
- Cloud & platform
- Azure (Functions, Bot Service, SQL, Key Vault, App Insights) · Entra ID · Microsoft 365 · Intune · Active Directory · Power Automate
- Systems
- Linux (RHEL, Ubuntu, CentOS) daily since 2007 and exclusively for 16 years · Proxmox and virtualization · systemd · shell tooling · self-hosted infrastructure, backup, monitoring
- Data & storage
- PostgreSQL · SQLite · Redis · ChromaDB · pgvector · pandas · FTS5
- Engineering
- Pester · pytest · Vitest · REST and GraphQL · circuit breakers and retry design · FastAPI · React · Remotion · Git
- How I work
- Spec before implementation · test-driven where it counts · dry-run defaults on anything destructive · schema-drift guards · documentation as a deliverable
More
Co-founder, Michiana InfoSec, 2019. I helped start the regional information security community here in South Bend and ran it through the early years, which given the timing meant a few months in a room and the rest of them not. I handed the reins to a new admin a few years back and it kept going without me. That's the only real test of whether a thing like that got built right.
Linux, exclusively, for 16 straight years, and an obsession considerably older than that. The terminal is where I think clearest, and most of what I know about systems I learned by breaking my own first.
I'm working toward the Claude Certified Architect — Foundations credential, and I mentor in my employer's internal program. Most of this eventually shows up on the blog, which has been running in one form or another since 2021.
Download as PDF for the systems that insist on one.