Projects / Built · Case study
The AI engine behind a customer-intelligence platform turns CRM notes, call transcripts and emails into a few prioritized insights. I owned it, and the work that matters is what I measured, what I undid, and what I traced to root cause.
Early warning signs in a B2B account, such as a deal going cold or a support pattern repeating, are scattered across CRM notes, call transcripts and emails. Nobody reads all of it, so the signals surface late or not at all. The platform reads that data continuously and turns it into a small number of prioritized insights people can trust.
A person gets an early warning plus a next step, without reading every note and transcript. Every insight arrives with its severity, the evidence it was built from, and its history, so it can be checked rather than taken on trust. These are illustrative examples modeled on the pipeline’s documented output, with invented evidence. They are not customer data.
Critical insight
Customers are weighing a switch because of the rival’s reporting and unlimited users. It came up in emails and calls across several accounts, so it reads as a pattern, not one unhappy customer.
5 sources · 2 systems · renewal at risk
Key insight
New users struggle with setup and adoption is slipping. The pipeline recognises it as the same theme as earlier runs, so it extends an existing thread instead of being reported as new.
Match reason kept: “same underlying onboarding complexity theme”
Signal
Pricing pushback in enterprise deals is tied to perceived ROI rather than headline cost. It spans pricing, renewal and expansion conversations, which points to a positioning gap.
Next step: tighten enterprise ROI messaging
How each insight stays traceable is covered under Trusting the output.
My own investigation found the pipeline reprocessing 80–90% of already-covered data on every run. That wasted model spend and resurfaced the same insight as if it were new. I built a check that runs before generation: if an earlier insight with an overlapping window already covers a cluster’s sources, regeneration is skipped.
The same cost instinct cut the weekly digest from seven model calls to four by merging two prompts, while I built its four sections: critical signals, emerging themes, executive summary and focus areas.
There is no fixed correct answer to unit-test an insight against, and reading every insight by hand does not scale. So I built a judge, and validated the judge before trusting it. Each check was tuned on a labeled development set of 25–30 observations, re-checked on a held-out set of 10–15, then sampled continuously in production with alerts on regressions around new prompt releases. Both built checks agreed with hand labels above 90%. The sets are small, and I say so.
| Check | Asks | Status |
|---|---|---|
| Groundedness | Is the insight backed by the source data? | Built, validated |
| Data sufficiency | Was there enough evidence to generate it at all? | Built, validated |
| Quality | Is it well written? | Specified |
| Actionability | Does it tell the team what to do? | Specified |
| Categorization | Is it correctly tagged? | Specified |
| Relevance | Is it relevant to that account? | Specified |
| Calibration | Is the urgency neither over- nor under-stated? | Specified |
Citations
Each insight stores the IDs of the source texts it was built from. The groundedness judge sees exactly those snippets, so a claim is checked against its own evidence and nothing else.
Lineage
New insights are matched to the previous run’s, and linked only above a 0.6 confidence. The score and the model’s written reason are stored with every link, so a trend can be questioned.
Audit trail
Each call is traced with the prompt version that produced it, so a bad output can be tied to a specific prompt change and rolled back.
I applied the same standard to my own work. I shipped a rule-based signal-strength score, measured it in production, and found it over-inflated significance in specific patterns: single-account insights, and narrative framing without evidence. I replaced it with the model’s own judgment rather than keep two competing systems.
Timeouts and SSL failures traced to the hosting platform throttling CPU to near zero once the HTTP response was sent, starving background uploads. Tuning timeouts could not fix it; the fix was at the deployment level. Switching serialization to orjson also cut peak memory in the upload path from about 3.8× to 2× the payload, for payloads up to 66 MB.
A webhook handler returned 400 for permanently unprocessable messages, so the queue retried them forever and drained the database connection pool. In the same investigation I fixed two N+1 query patterns and a misread pool metric, and found that worker threads equaled pool size, leaving no headroom under load.
An unindexed multi-column query forced full scans of about a minute for large organizations. I added a composite index through a live backfill, then hardened the fix against two Postgres traps: autovacuum conflicting with concurrent index builds, and invalid indexes silently blocking retries.
Peak memory in the upload path, as a multiple of payload size
| Decision | Why | Outcome |
|---|---|---|
| Reverted incremental sync for a CRM (Monday.com) | Updates could be filtered by exact date, but items only accept relative presets. A mixed sync risked silent gaps, because two downstream tables hard-delete rows missing from a payload. | Back to a full sync. Other providers got per-provider watermarks in place of fixed 180-day pulls. |
| Plain index over trigram index | The filter was an exact match on a field with few distinct values, which a B-tree index handles directly. | Trigram kept only for free-text search. |
| Throttle before backoff | Retrying after a 429 only reacts once a request is already wasted. | Limit of 2 requests per second, with backoff as the fallback. |
| Rule-based score to model judgment | Production data showed the rule-based score over-rated certain patterns. | Deleted it, rather than run two scoring systems. |
I also reviewed the AI engine against SOLID principles (it scored 5 out of 10, with file-level citations), wrote a five-phase refactor plan, and executed the first phases: one shared background-task runner in place of three copies, pooled database connections, and one shared layer for events, logging and errors.
Before committing to a planned feature that would group insights by theme, I ran a short experiment to see whether it was feasible, with no production writes. The verdict: grouping by theme looks feasible, but not fully automatic. The feature itself is not built.
The experiment turned each insight into an embedding, a list of numbers where similar text gets similar numbers, then measured how well insights separate into groups.
Insights on the same topic are more alike than insights on different topics, but the ranges overlap. That is enough to group insights, not to separate them cleanly.
0.30: below this a clustering is considered weak
| Question | Finding |
|---|---|
| Can similarity deduplicate? | Treating pairs above 0.93 similarity as duplicates wrongly merged 0 of 435 synthetic pairs and 2 of 1,225 real pairs. |
| Which methods lost? | t-SNE (gives different results each run and only works in 2–3 dimensions) and k-means (needs the number of groups chosen up front, forces every insight into a group, and assumes round-shaped groups). UMAP with HDBSCAN fit instead. |
| Can the model step out of the loop? | No. The model must still make the final call where meaning can flip, such as risk rising versus risk falling. |