Google Search Console
Source Analytics

Google Search Console

Every query and page your site ranks for — past the 1,000-row cap and the 16-month window, in a database you own.

Insightlytics ingests your Google Search Console performance data — clicks, impressions, CTR and position for every query and page — through the native bulk export, the Search Analytics API or both. We clean it, enrich it with your own grouping rules and model it into data models that answer SEO questions directly, rebuilt daily. You get the models and the accumulated history; we run the ingestion, the refreshes and the monitoring. Everything is written to a database you own — BigQuery today, with more warehouses on the roadmap.

What we ingest

Search Console exposes its performance data two ways, and the tables differ between them. We land the raw tables first, then build the data models on top — so you keep the source rows, not just our interpretation of them.

  • searchdata_site_impression Bulk export

    One row per property per day per dimension combination. Anonymized queries are included here as flagged, query-less rows, which is what makes property totals reconcilable at all.

  • searchdata_url_impression Bulk export

    One row per URL per day, carrying the search-appearance flags — rich results, AMP, video — that Google attaches to the impression.

  • ExportLog Bulk export

    Google’s own record of which tables landed on which day. We monitor it so a missing export day is caught on our side rather than found in your report.

  • Search Analytics API API

    Paged pulls across query, page, country, device, search appearance and date. This is the route that backfills the trailing 16 months and reaches depth the 1,000-row report never shows.

Which of these you end up with depends on the route — and that decision is covered further down, in how we work with Search Console data.

The problems we solve

Search Console data has sharp edges that only show up once you work with it daily. These are the ones every serious team hits — and the ones this pipeline is built around.

Why is Search Console data two to three days behind?

Search Console finalizes performance data roughly two to three days after the fact; the freshest rows are preliminary and still move. Any report built directly on the raw feed either shows numbers that later change or silently mixes final and preliminary days.

We treat the finalization window as a first-class fact: recent days are re-synced until they settle, and the data models mark where ‘final’ ends — so nobody screenshots a number that was about to move.

Why don’t query rows add up to the property totals?

Google withholds anonymized queries — rare or privacy-sensitive searches — from the API’s query-level results, while still counting them in totals. On many sites that gap is a double-digit share of impressions, and if you only store query rows it looks like missing data or a broken join.

We store totals alongside query rows so the gap becomes a measurable, explainable number. The bulk export goes one better: it includes anonymized queries as flagged, query-less rows, so on that path the gap is itself a column you can chart.

Why does Search Console history stop at 16 months?

The API’s retention window rolls forward daily: every day you wait, a day of history is permanently deleted at the far end. Year-over-year comparisons need at least 24 months, which Search Console alone can never give you.

The only fix is structural — land the data in a database you own before you need it. The bulk export accumulates forward without caps from the day it’s enabled, the API backfills the trailing 16 months, and from then on the window stops rolling for you.

How do you get more than 1,000 rows out of Search Console?

The interface caps every report at 1,000 rows regardless of how many queries or pages actually have data. Head terms fill the visible rows; the long tail — the thousands of specific, low-competition phrases you almost rank for — is exactly what the cap hides.

Both routes surface it: paged API pulls return up to 25,000 rows per request and reach depths of up to 50,000 rows per property per search type per day, and the bulk export has no row caps at all. Depth is the first thing this pipeline fixes.

Why does average position mislead you?

Position is an impression-weighted average of averages. Combine two queries, two pages or two date ranges and the blended number can move in the opposite direction of every part — a classic aggregation paradox. Raw exports make this worse by inviting naive AVG() over rows at mixed grain.

The data models recompute position weighted by impressions at each grain, so the number means what it appears to mean.

How we work with Search Console data

Two routes exist, and choosing between them is a consulting decision, not a checkbox.

The native bulk export writes complete, uncapped daily tables straight into BigQuery — anonymized queries included as flagged rows — but it starts the day it’s enabled and can’t backfill. The Search Analytics API backfills the 16 months Google exposes and pages far past the 1,000-row report cap, but it’s quota-metered and omits anonymized queries from query-level rows. Which route fits — or whether to run both — depends on how much history you need, how many properties you run and how exact your totals have to be.

We settle that together on the discovery call. Most properties end up on both routes — API backfill for the trailing 16 months, export accumulation from day one — feeding the same data models. Either way, running the pipeline is our job: no connector to configure, no quota to babysit.

Delivered to a database you own

Every table — raw and modeled — is written to your own database, under your billing and your access controls. Our access is read-only, and if we part ways, everything stays with you.

  • BigQuery BigQuery Your data models are built in your own BigQuery project — our access is read-only, and the tables are always yours.

BigQuery is the supported database today. If you run a different warehouse, tell us — the roadmap is driven by requests.

How current the data stays

Data models rebuild every day. Where a platform restates recent numbers after the fact, we re-read those days rather than freezing the first version we saw.

Refresh

Daily on both routes. API syncs trail Search Console’s finalization window and re-pull the last few days; the bulk export lands each day’s rows as Google finalizes them.

Backfill

The full 16 months the API exposes, loaded on day one — the maximum anyone can offer. The bulk export accumulates forward, uncapped, from the day it’s enabled.

Monitoring

Every sync is checked for freshness, volume and schema drift. When Google changes something, we adjust the pipeline — you find out from a note, not from a broken chart.

What lands in your database

Not raw API responses — data models: named, documented tables at a stated grain, rebuilt every day and ready to query. Each one has its own page.

The shipped data models are the starting point, not the ceiling. When your questions need a different shape — another grain, another split, a join against another source — we build additional models for your stack as part of the engagement.

Stop rebuilding the same Search Console export.

Tell us how many properties you run and how far back you need to go. We’ll map the route, the data models and the history you can still recover before it rolls out of Google’s window.

Free discovery call · No commitment — leave with a starting point