We rebuild the intent graph from scratch every day

Ask an intent vendor how fresh their scores are and you get a refresh cadence: weekly, sometimes biweekly. That is usually presented as an operational detail. It is not. Intent is a decaying signal. In our last measurement post we found the median payoff of a display impression lands 77 hours out; research behavior moves on the same clock. A score computed last Tuesday is not a stale answer to the question, it is an answer to a different question.

So we made freshness a structural property instead of an operational one. The intent graph that serves our API is not updated. It is replaced, in full, every day.

What gets rebuilt

A word on scope first, because the intent system has two halves. The pipeline that scores raw behavior into topics runs hourly, all day, and accumulates as it goes; that system has its own war stories and will get its own post. This post is about what serves: the graph our intent API answers from when a partner asks who is researching what.

That serving graph is person-grain, keyed to durable hashed-email identities. As of this writing it carries over 1.2 billion person-topic signals across roughly 27 million people and 2 million companies, spanning tens of thousands of topics, with per-topic reverse indexes so you can walk it from either direction: person to topics, or topic to the people and companies researching it.

Every one of those numbers is different tomorrow, because tomorrow it is a different graph. Once a day, the consolidation step joins the accumulated signals against the identity spine and emits a complete new serving set. Yesterday's graph is not patched to agree with it. It is retired.

The mechanics: never update, only replace

We looked at the standard answer first: an always-on analytics cluster serving live queries over the scored data. Then we looked at our actual access pattern and it was point lookups and bounded scans, not analytics. That reframing let us pick a much duller architecture. The daily build exports the graph to object storage, bulk-imports it into a brand new key-value table named for its data date, validates it, and then performs the entire cutover with a single pointer write. The API reads the pointer, caches it briefly, and follows it to whichever table is current.

Everything interesting about the design falls out of that one property. Imports are boring because nothing serves from a table until the pointer moves. Rollback is the same operation as rollout: move the pointer back. And a failed build changes nothing at all; the pointer keeps naming yesterday, the API keeps answering, and the failure is an operations page instead of an outage. Old tables linger for a grace window and are then deleted.

pointer  ->  { table: 'intent-<date>', as_of: '<date>' }

build day N:   export -> import intent-N -> validate
cutover:       write pointer to intent-N        (one operation)
failure:       pointer still names intent-N-1   (nothing happened)
retire:        drop tables older than the grace window

What full rebuilds buy, and what they cost

No drift, by construction. There are no incremental updates to corrupt, no tombstones, no compaction debt, no migration that half-applied. Any question about the graph on a given day is answerable by the artifact built that day, which also makes the pipeline reproducible in a way mutable stores never quite are.

The bill for that discipline is real. You pay for the whole build every day whether one signal changed or a billion did, and the rebuild has to fit inside the day. That second constraint is the one to respect: the morning your build takes 25 hours, your freshness story is over. We watch build duration the way other teams watch error budgets, because for this architecture, build time is the error budget.

And it bounds freshness at the cadence, no better. This is batch, refreshed daily, and the API says so: every response carries the as_of date of the graph that answered it. If a vendor cannot tell you the as_of of the score they just sold you, that is worth sitting with.

Why this shape is rare

Rebuilding daily only makes sense if you control the whole path: the signal collection, the identity spine that keys it, the scoring pipeline, and the serving layer. Miss any piece and you are stitching feeds you do not control into a store you dare not throw away, and incremental updates become the only option. The reason most intent products refresh weekly is not that their engineers prefer it. It is that their architecture accumulated, and accumulated systems are updated, not replaced.

Ours gets thrown away every morning and comes back bigger. We think that is the most honest thing about it.