Source Analytics

Google Search Console Data, Delivered to Your Database

Q: How do you get more than 1,000 rows?

The 1,000-row limit is a display cap in the Search Console interface, not a data cap — and both routes go past it. The Search Analytics API returns up to 25,000 rows per request, and paging reaches the API’s export depth of up to 50,000 rows per property per search type per day. The native bulk export has no row caps at all — Search Console writes complete daily tables straight into BigQuery. The long tail the interface hides is where most of the value lives.

Q: How much history can you backfill?

On the API path, 16 months — the most the API exposes; data older than that is permanently gone from Google’s side, and no vendor can recover it. The native bulk export can’t backfill at all: it starts the day it’s enabled and accumulates forward without caps. The strongest setup combines them — API backfill for the trailing 16 months, export accumulation from day one — and either way history lives in your database from then on.

Q: Why don’t your numbers match the Search Console interface exactly?

Three documented reasons: anonymized queries (the API omits rare and privacy-filtered searches from query-level rows, while the bulk export includes them only as flagged, query-less rows), preliminary versus final data on recent days, and average position being an average of averages that shifts with how rows are grouped. The marts store totals and query-level rows separately so the gaps are visible instead of confusing.

Q: Do you change anything in my Search Console account?

No. API access is read-only — the pipeline calls the read endpoints and nothing else; no settings change, no sitemaps are submitted. If you use the bulk-export path, you enable it once in your own property and Search Console writes the tables itself — we still touch nothing.

Cleaned, enriched and modeled into ready-to-query mart tables. Currently on BigQuery. We run the pipeline, you own the data.

Insightlytics ingests your Google Search Console performance data — clicks, impressions, CTR and position for every query and page — through the native bulk export, the Search Analytics API, or both. We clean it, enrich it with your own grouping rules and model it into mart tables that answer SEO questions directly, refreshed daily. You get the marts and the accumulated history; we run the ingestion, the refreshes and the monitoring. Everything is written to a database you own — BigQuery today, with more warehouses on the roadmap.

Book an architecture review Ask for a sample schema

The problems we solve

Search Console data has sharp edges that only show up once you work with it daily. These are the ones every serious team hits — and the ones this pipeline is built around.

Why is Search Console data two to three days behind?

Search Console finalizes performance data roughly two to three days after the fact; the freshest rows are preliminary and still move. Any report built directly on the raw feed either shows numbers that later change or silently mixes final and preliminary days.

We treat the finalization window as a first-class fact: recent days are re-synced until they settle, and the marts mark where ‘final’ ends — so nobody screenshots a number that was about to move.

Why don’t query rows add up to the property totals?

Google withholds anonymized queries — rare or privacy-sensitive searches — from the API’s query-level results, while still counting them in totals. On many sites that gap is a double-digit share of impressions, and if you only store query rows it looks like missing data or a broken join.

We store totals alongside query rows so the gap becomes a measurable, explainable number. The bulk export goes one better: it includes anonymized queries as flagged, query-less rows, so on that path the gap is itself a column you can chart.

Why does Search Console history stop at 16 months?

The API’s retention window rolls forward daily: every day you wait, a day of history is permanently deleted at the far end. Year-over-year comparisons need at least 24 months, which Search Console alone can never give you.

The only fix is structural — land the data in a database you own before you need it. The bulk export accumulates forward without caps from the day it’s enabled, the API backfills the trailing 16 months, and from then on the window stops rolling for you.

How do you get more than 1,000 rows out of Search Console?

The interface caps every report at 1,000 rows regardless of how many queries or pages actually have data. Head terms fill the visible rows; the long tail — the thousands of specific, low-competition phrases you almost rank for — is exactly what the cap hides.

Both routes surface it: paged API pulls return up to 25,000 rows per request and reach depths of up to 50,000 rows per property per search type per day, and the bulk export has no row caps at all. Depth is the first thing this pipeline fixes.

Why does average position mislead you?

Position is an impression-weighted average of averages. Combine two queries, two pages or two date ranges and the blended number can move in the opposite direction of every part — a classic aggregation paradox. Raw exports make this worse by inviting naive AVG() over rows at mixed grain.

The marts recompute position weighted by impressions at each grain, so the number means what it appears to mean.

How we work with Search Console data

Two routes exist, and choosing between them is a consulting decision, not a checkbox.

The native bulk export writes complete, uncapped daily tables straight into BigQuery — anonymized queries included as flagged rows — but it starts the day it’s enabled and can’t backfill. The Search Analytics API backfills the 16 months Google exposes and pages far past the 1,000-row report cap, but it’s quota-metered and omits anonymized queries from query-level rows. Which route fits — or whether to run both — depends on how much history you need, how many properties you run and how exact your totals have to be.

We settle that together at the architecture review. Most properties end up on both routes — API backfill for the trailing 16 months, export accumulation from day one — feeding the same marts. Either way, running the pipeline is our job: no connector to configure, no quota to babysit.

Refresh, backfill and monitoring

Refresh

Daily on both routes. API syncs trail Search Console’s finalization window and re-pull the last few days; the bulk export lands each day’s rows as Google finalizes them.

Backfill

The full 16 months the API exposes, loaded on day one — the maximum anyone can offer. The bulk export accumulates forward, uncapped, from the day it’s enabled.

Monitoring

Every sync is checked for freshness, volume and schema drift. When Google changes something, we adjust the pipeline — you find out from a note, not from a broken chart.

Delivered to a database you own

Every table — raw and mart — is written to your own database, under your billing and your access controls. Our access is read-only, and if we part ways, everything stays with you.

BigQuery Marts are built in your own BigQuery project — our access is read-only, and the tables are always yours.

BigQuery is the supported database today. If you run a different warehouse, tell us — the roadmap is driven by requests.

What lands in your database

Not raw API responses — named mart tables at documented grain, refreshed daily and ready to query.

mart_gsc_queries Every query with clicks, impressions, CTR and impression-weighted position — past the 1,000-row cap — joined to your brand and intent labels.
mart_gsc_pages Performance by page, carrying the page-type and content-group labels your rules assign.

The standard marts are the starting point, not the ceiling. When your questions need a different shape — another grain, another split, a join against another source — we define additional marts for your stack as part of the engagement.

The SEO answers built on these marts

Each enrichment applied to your Search Console data is a shipped data model with its own page — the rules are written for your site and live in your database.

What you can run on Search Console data

Every capability we ship works on these marts from day one.

Frequently asked questions

If something is still unclear, a discovery call clears it up fast.

Bulk export or API — which one do we choose?

Usually both, and it’s a decision we make together at the architecture review. The native bulk export writes complete, uncapped daily tables straight into BigQuery but starts the day it’s enabled; the Search Analytics API backfills the 16 months of history Google exposes and pages far past the interface’s 1,000-row cap, but is quota-metered and omits anonymized queries from query-level rows. The strongest setup combines them — API backfill for the trailing history, export accumulation from day one — and either way we run it.

How do you get more than 1,000 rows?

How much history can you backfill?

Why don’t your numbers match the Search Console interface exactly?

Do you change anything in my Search Console account?

Your search data, past the caps — in a database you own.