Joe Laskowski

AI solutions engineering · LLM evaluation · agent infrastructure

I build the systems that decide whether AI output is any good — and then I build the things that deliver it. 25 years in IT, the last stretch spent as the human in the loop for production LLM assistants: authoring the rubrics, coding the failure taxonomy, shipping the chatbots technicians actually use, and measuring whether any of it moved. Outside the day job I've been doing the same work unsupervised for years — orchestration layers, evaluation harnesses, knowledge graphs, and a great deal of code nobody asked me to write. Nearly all of it built with AI in the loop, which is the point rather than the asterisk: getting these systems to produce tested, production-grade work is itself the skill, and it happens to be the one I'm paid to evaluate.

South Bend, Indiana · Get in touch · GitHub · LinkedIn

Selected Work

Personal projects. Built on my own time, for my own reasons, and the reason the rest of this page reads the way it does.

cowork-rag + cowork-graph

Private retrieval infrastructure · MCP

The retrieval layer my agents actually run on — two complementary systems over the same corpus, both exposed as Model Context Protocol servers and shared across Claude, Codex, Grok Build, and Hermes. The host supplies the answer model; these supply the evidence.

cowork-rag is a fully local stack: Nomic embed-text-v1.5 at 768 dimensions and a MiniLM-L-6-v2 cross-encoder both running in-process through FastEmbed ONNX, over ChromaDB — no GPU server, no API key, no Ollama. Retrieval is hybrid, fusing a disk-backed SQLite FTS5 keyword lane with the vector lane by reciprocal-rank fusion so exact identifiers stay findable, then reranking the fused pool. On top of that sit map-reduce document summaries, a GraphRAG layer that extracts entities and relationships per document and clusters them into summarized communities, date-aware filtering, per-domain scoping, local OCR and image captioning through Tesseract and Florence-2, cross-session persistent memory, and an optional PII detector that masks identifiers before they ever reach a host's context. Sync is two-level — mtime scan, then content-hash confirmation — with chunk-level embedding reuse, so editing one paragraph of a large file re-embeds one chunk instead of the file. Generation is deliberately separated behind a provider-neutral interface, defaulting to a local llama.cpp Qwen3-8B that a nightly systemd timer starts, uses for summaries and graph extraction, and shuts down again.

cowork-graph is the structured counterpart: a SQLite knowledge graph parsed from the same Markdown, answering the questions embeddings are bad at — what is active, what is blocked, what links to what, what got decided and when. Markdown stays canonical in both; every index is derived state, rebuildable from scratch.

2,651 files · 98,242 chunks · 38,345 graph entities in 150 communities · 15 MCP tools · ~6,900 lines Python, 1,800 in tests

Nexus Signals

Market analysis platform

Python 3.12 / FastAPI / PostgreSQL 16 / Redis 7 behind a React 19 and TypeScript front end. Domain-driven design across six bounded contexts with CQRS throughout. Seven market-data providers chained behind circuit-breaker fallbacks so a dead vendor degrades instead of failing, 23 technical indicators feeding a −10 to +10 signal score, LLM-driven sentiment and directional analysis, paper trading with portfolios, orders, positions and live P&L, and WebSocket streaming across 43 versioned REST endpoints. I got deep into v1, realized I could build it properly, and threw it away to start over — the version that exists is the second one.

3,764 passing tests · mypy --strict, zero Any types

Claude Sentient

Agent orchestration layer — public

An autonomous development loop for coding agents, structured as INIT → UNDERSTAND → PLAN → EXECUTE → VERIFY → COMMIT → EVALUATE with hard quality gates on lint, test, build, and git between stages. Nineteen slash commands, fifteen hooks, nine agent roles, nine language profiles with Python environment detection across conda, venv, poetry and pdm, twelve JSON schemas enforcing cross-module integrity, and a self-improvement mechanism that proposes its own rule updates after it gets corrected. I ran it for months, then watched the platform ship native features that turned my abstraction layer into drag — and pulled it back out of every project. Building it and retiring it were both the right call.

924 tests · public on GitHub

Documentary production pipeline

Three chained tools

Script in, finished 1080p video out. Stage one synthesizes narration through a TTS model using a seven-rule chunker that will not split mid-sentence or inside an emotion tag, then aligns the audio with WhisperX into sentence-level beat timestamps the downstream stages key off. Stage two is a browser-based asset manager — React and Express, filesystem as the source of truth, no database — that turns a Markdown prompt sheet into an assignable grid for the roughly 150 individual image generations a single episode consumes, then deploys config, beats, images, and audio into the renderer in one click. Stage three assembles it in Remotion: Ken Burns motion, crossfade and zoom-dissolve transitions, lower thirds fired on beat boundaries, and a per-era color grade and film-texture overlay whose era value is classified upstream by an LLM and carried through the whole chain.

90% test coverage on stage one · ~$0.38 of narration per 30 minutes · five episodes shipped

NexusLedger

Multi-entity finance system

Plaid ingestion into SQLite, categorized against learned rules, reconciled, and pushed out to reporting workbooks — across three separate legal entities that must never bleed into each other. Checksummed migrations, period locks so a closed month cannot silently move, pending-to-posted carryover, synthetic transaction pairs for transfers, and a doctor command that runs integrity checks and catches drift before it reaches a report rather than after.

662-test Python suite

Also built

PromptVault — A deterministic prompt compiler and semantic library — raw idea in, structured and provider-optimized prompt out, through a card/template engine and a multi-pass critic. Ships with 92 system cards across image, LLM, and code generation plus 27 templates, and a 30-case regression harness that diffs output against stored baselines so a change to the compiler cannot silently degrade it. Single-user by design: the API key is passed per request from the browser and never persisted server-side.

Node 22 · pnpm monorepo · Fastify · Next.js · PostgreSQL 16 + pgvector with HNSW indexes · Redis 7 · Drizzle ORM

SEO content pipeline — A thirteen-package monorepo running the whole chain: seed keyword discovery, gap analysis, revenue-per-keyword scoring, competitive analysis into generated content briefs, AI copy and image generation, static site build, and finally GEO scoring — grading each page for citation-readiness across generative search engines rather than for blue links. Built before most people had a name for that problem.

TypeScript · pnpm workspace · OpenAI + Anthropic · DALL·E 3 + Stability AI · Serper.dev SERP data

DropTok — A TikTok commerce platform wired to live marketplace APIs rather than mock data — multi-shop management, an AI listing generator for titles, hooks and descriptions, a content pipeline producing video hooks and shot lists, an AI-scored opportunity board, creator outreach with a CRM behind it, a fulfillment queue built as an explicit state machine, and a dashboard tracking my own model token spend.

TypeScript end to end · React 18 / Vite 6 / Tailwind / Zustand · Express 4 / Drizzle / better-sqlite3 · TikTok Shop Seller, Ads, CJ Dropshipping and AutoDS integrations · Vitest

BudgetLedger — A budgeting application on live Plaid account sync, with CSRF protection, rate limiting, and optional TOTP multi-factor. Its one genuinely good idea was pay-period planning: bills auto-assigned to the specific paycheck that covers them by due date, leftover computed per paycheck, and negative paychecks flagged before they happen. I built it, used it, and went back to my spreadsheet — which had absorbed enough of the good ideas by then to win.

Flask 3.1 · SQLAlchemy 2.0 · SQLite (Postgres-ready) · Chart.js · Plaid

xrp-ticker — A terminal portfolio tracker that reads balances straight off the XRP ledger over WebSocket and aggregates them across multiple self-custody wallets, with live pricing from a separate exchange API. Four cyclable sparkline renderers, two themes, auto-reconnect, entirely keyboard-driven. It only works against self-custody addresses, because exchange deposit addresses sit in shared omnibus wallets — which is exactly the sort of detail you only learn by building the thing.

Python 3.11+ · Textual · uv · XRPL WebSocket · Coinbase API · public on GitHub

SKYNET backup — Encrypted, deduplicated, snapshot-preserving backup across three redundancy layers — live working copy, local restic repository, off-site cloud — driven by both systemd timers and Windows Task Scheduler, with green/yellow/red Slack alerts on every single run and fail-only alerts on prune and verify. Restores have been round-tripped and confirmed byte-identical by SHA-256, which is the part almost nobody actually does.

restic · systemd · Task Scheduler · Slack webhooks · Bash

And a novel, drafted with AI tooling under a pen name, with an Amazon launch pending — which turned out to be a pipeline problem wearing a different hat.

Experience

AI Quality Analyst

March 2026 — Present

Aunalytics

The human in the loop for two LLM-powered ticket enrichment assistants — scoring model output against a five-dimension rubric I authored, with a coded failure taxonomy that turns individual bad answers into diagnosable model weaknesses. The finding that has held for five months: the failure is in synthesis, not retrieval.
Designed and built the Microsoft Teams chatbots technicians use to reach both systems — Azure Functions and Copilot Studio over an in-house LLM, with live third-party data wired in through Power Automate. Two near-identical bots were refactored into a single shared core at 95% statement coverage.
Raised technician documentation quality 29% across eighteen weekly coaching cycles, more than doubling the share of tickets carrying a usable resolution entry. Because those notes are the model’s training corpus, the improvement compounds.
Measured an automated remediation program to 96% script success with 61% of tickets resolving without a technician ever touching them, and drove script failures to zero.
Caught and corrected a data-export defect that had inflated a reporting population by an order of magnitude. Suspended trend comparisons, rebuilt the baseline from scratch, and published the corrected figure rather than letting a flattering artifact stand.
Authored the operating standards for the function — scoring rubrics, failure codes, documentation style guides, and a twelve-KPI measurement framework — plus the architecture proposal for productizing the whole thing on Azure.

Internal Support Engineer

August 2023 — March 2026

Aunalytics

Designed and built the company’s user lifecycle automation end to end — nine PowerShell modules covering Active Directory, Microsoft 365, and ticket-driven provisioning, with certificate-based authentication from Key Vault, schema-validated configuration, separate production and test environments, and a full Pester test suite.
Built zero-touch Windows Autopilot deployment: certificate-only auth, a resumable checkpoint state machine with exponential backoff, and automated cleanup of stale Intune, Autopilot, and Entra records.
Wrote the quarterly privileged-access review tooling — a privileged-access exporter, risk flagger, and quarter-over-quarter diff — along with a Microsoft 365 tenant audit module carrying CIS and vendor baselines.
Served as Linux subject-matter expert for the Centralized Services team, and worked part-time with the Security team as a SOC analyst on internal and client security needs.
Provided Tier 1 and Tier 2 support to internal staff and contributed to organization-wide rollouts.

Service Desk Location Lead — Kalamazoo

January 2022 — August 2023

Aunalytics

Led the Kalamazoo team through Tier 1 and Tier 2 delivery — incident management, escalation, and SLA maintenance — while managing staff performance, mentoring, and training.
Co-lead engineer for internal departments: user access administration, Microsoft 365 licensing, and general engineering support.

Service Desk Lead

October 2019 — January 2022

Aunalytics

Led a team of technicians and engineers supporting the service desk’s largest accounts, including environments north of 1,500 users.
Owned customer relationships through the IT point of contact — service delivery, escalation communication, and ticket follow-through.

Managed Services Specialist

March 2017 — September 2019

MicroIntegration

Resolved 70 to 100 tickets a week across remote first-level support, investigating complex hardware and software faults and escalating what needed escalating.

Field Technician Manager

July 2007 — March 2017

Realm Technologies

Ran day-to-day operations for break/fix technicians nationwide — recruiting and certifying more than 200 techs delivering warranty repair for Dell, Sony, IBM, HP, and Lexmark.
Managed service call dispatch, parts logistics, and vendor relations, and maintained the company intranet and public site.

Earlier — field and deployment technician work through the 2000s: warranty repair for Dell and the other OEMs, and a multi-site enterprise hardware rollout where I supervised a small team through six thousand machines. It is not on this page because it is recent; it is on this page because knowing what the tech on the other end of the ticket is actually looking at is why the AI work lands.

Toolset

AI & LLM: LLM output evaluation and rubric design · prompt and agent-instruction engineering · Model Context Protocol (MCP) server development · Microsoft Copilot Studio · Anthropic and OpenAI APIs · retrieval and knowledge-gap analysis · agent orchestration and multi-agent design
Languages: PowerShell · Python · TypeScript · JavaScript · SQL · Bash
Cloud & platform: Azure (Functions, Bot Service, SQL, Key Vault, App Insights) · Entra ID · Microsoft 365 · Intune · Active Directory · Power Automate
Systems: Linux — RHEL, Ubuntu, CentOS — as a daily driver since 2007 and exclusively for sixteen years · Proxmox and virtualization · Bash and shell tooling · self-hosted infrastructure, backup, and monitoring
Engineering: Pester · pytest · Vitest · REST and GraphQL integration · circuit breakers and retry design · pandas · FastAPI · React · Remotion · Git
Practice: Spec before implementation · test-driven development · dry-run defaults on destructive operations · schema-drift guards · documentation as a first-class deliverable

Also

Co-founder — Michiana InfoSec, 2019. I helped start the regional information-security community here in South Bend and ran it through its early years — which, given the timing, meant its first few months in a room and the rest of them not. Handed the reins to new leadership a few years back. It kept going without me, which is the only real test of whether a thing like that got built properly.

Linux, exclusively, for sixteen straight years — and an obsession considerably older than that. The terminal is where I think clearest, and most of what I know about systems I learned by breaking my own first.

Pursuing the Claude Certified Architect — Foundations credential. Mentor in my employer's internal mentoring program. I write about most of this at the blog, which has been running in one form or another since 2021 and covers the AI tooling side back when that was still a strange thing to be interested in.

Download as PDF — for the systems that insist on one.