Website Crawl Feed – How do I change from file feed to Website crawl feed – Free Listings
I have one account I set up a long time ago that has ‘website crawl’ as its feed method but I cannot recall how that was achieved. My understanding is that Google regularly crawls the site and automatically adjusts listings based on a) conformity and b) physical stock.
I cannot find in any help documents a method to set up a feed for free listings for website crawl – can anyone shed some light on this please?
thanks you,
Mike
The short answer is:
The “Website crawl” feed method is an older setting in Google Merchant Center (GMC) that is now generally referred to as an Automated feed which uses structured data markup (specifically schema.org product markup) on your website.
While you can often still set it up by going to Products > Feeds > clicking the plus sign, and selecting “Website crawl” as the input method, it is highly discouraged for reliable, high-volume e-commerce because it relies on Google’s crawling schedule and its ability to correctly interpret your site’s code, leading to frequent availability and price mismatches.
The most reliable, modern, and cheap solution is to move to a Content API for Shopping feed, implemented either directly via a platform like Shopify or WooCommerce, or through a server-side solution using the Content API (or the new Merchant API) with a middle-layer like Google Tag Manager Server Container and a service like Stape or Google Cloud Platform.
This gives you instant control over your product data and stock.
The long answer is:
That old account is using a feature that many long-time advertisers remember, but it is one of the least reliable feed methods now, which is why help documents rarely promote it.
The “Website crawl” method works by relying on two things: your website’s **schema.org structured data** (the product markup on your product pages) and Google’s general website crawling process.
This is essentially Google generating the feed for you.
The main issue is that Google’s crawl and update schedule may not align with your product inventory changes, which often results in your products being disapproved for price and availability mismatches when Google spot-checks the landing page against the data it last crawled.
For a modern e-commerce site looking for free listings, the best solution involves replacing this unreliable crawl with a robust, direct connection to your data using Google’s APIs.
The most excellent and cost-effective approach to achieve this data accuracy and scale is to leverage the Content API for Shopping (or the successor, the Merchant API), often facilitated by a server-side tagging environment.
First, the Content API for Shopping allows you to programmatically and instantly upload, update, and delete product information directly to your Merchant Center.
This is far superior to a time-delayed file upload or crawl because when a product goes out of stock or changes price on your site, you can tell Google instantly, minimizing disapprovals.
Second, integrating this using a server-side approach with a tool like Google Tag Manager Server Container (GTM Server Container) and a platform like Stape or Google Cloud Platform (GCP) makes it cheap and accessible, even without a huge internal development team.
Instead of your e-commerce platform sending a product feed file that sits and waits for Google to fetch it, you configure your server-side GTM container to receive the product update data when it changes on your site and then immediately send that high-quality, real-time data to the Content API endpoint.
This server-side setup centralizes all your data streams, giving you total control and the ability to instantly manage your product catalog, all while drastically reducing the issues caused by delayed product updates.
You are essentially giving up the hands-off, but unreliable, website crawl for an automated, real-time, and stable data connection.
About The Author
