The crawler behind
a cooking search engine.

Weeknight Dinners is a cooking search engine. Our users type a dish into a search box and expect to find good, cookable recipes — their own saved ones, and public ones from around the web. RouxBot is the crawler that builds the public half of that index.

Think of it as Googlebot, but narrow: a small, polite crawler that only cares about recipe content, indexes it for search, and links back to the original source with attribution. The content still lives on your server. We store enough to make a search result useful — title, image, author, ingredient and step text where marked up — and a link pointing back to you.

Every request RouxBot makes carries a User-Agent header in this form:

RouxBot/1.0 (+https://roux.app/bot)

The version number before the slash may change as the crawler evolves. The parenthesised URL will always point back here. The name RouxBot is stable and is what you should match against in robots.txt and access-log filters.

RouxBot does not rotate User-Agents, does not impersonate browsers, and does not pretend to be something else. If a request claims to be RouxBot but doesn’t behave like what’s described on this page, it isn’t us — please let us know.

RouxBot operates in two modes.

Refresh — the common case. Once a recipe URL from your site is in our index (either because a Weeknight Dinners user saved it, or because we discovered it via the mode below), we periodically re-fetch that one URL to keep the cached title, image, and attribution current. One GET on the recipe URL, one GET on robots.txt (cached per-host), and an optional HEAD on the hero image. No link-following, no site walk, no sitemap traversal. The refresh interval is on the order of weeks, not minutes.

Discovery — for a small set of sites we’ve manually added to our index (see the Partners section), we follow same-host links from a seed URL up to a depth of three to find additional recipes. Discovery only runs for hosts we’ve explicitly enabled in our configuration — it is not triggered by arbitrary user activity. Discovery still respects robots.txt, still applies per-host rate limiting, and still stops at your domain boundary.

Either way, we hard-cap our batch sizes and the total queue depth, so RouxBot’s aggregate footprint on any given host stays small.

If you write a food blog and you’d like your work to be indexed and surfaced on Weeknight Dinners, we want to talk to you. Our creator partnership is still being built, but the intent is simple: incentives aligned with the people making the content.

When the program is fully launched it will include:

• Ingestion of your recipes via RSS and native HTML, with canonical links back to your site on every result.
• A share of ad revenue on pages attributed to you — the details (rate, payout, attribution rules) are still being worked out, and we'll publish them here before any money changes hands.
• A complimentary Weeknight Dinners subscription so you can use the calendar, sharing, and social features yourself.
• No exclusivity, no content license transfer — we're an index pointing back to you, not a republisher. You keep everything.

In the meantime, if you’d like your site added to our index manually, reach out via the feedback form and we’ll follow up directly.

RouxBot rate-limits itself on a per-host basis. We honour the Crawl-delay directive in robots.txt, and apply a minimum per-host delay in addition to whatever you specify, so two requests to the same host are never fired back-to-back.

If you see RouxBot generating meaningful load on your server, the most likely cause is that a lot of your recipes are in our index and due for refresh around the same time — which is something we’re happy to help you throttle. Reach out via the feedback form and we’ll adjust our pacing for your host.

RouxBot reads and obeys robots.txt. To block it from your entire site, add this to /robots.txt:

User-agent: RouxBot
Disallow: /

To block specific paths but allow the rest:

User-agent: RouxBot
Disallow: /drafts/
Disallow: /admin/

To set a minimum delay between requests (in seconds):

User-agent: RouxBot
Crawl-delay: 10

For discovery traffic, we cache robots.txt per host for up to seven days to avoid re-fetching it on every request, so a fresh Disallow rule can take up to a week to take effect there. Refresh traffic re-reads robots.txt on each run and honours new rules immediately. If you need a faster opt-out for discovery, reach out via the feedback form below and we’ll disable fetching for your host immediately; the cache will then expire on its own.

• Submit forms, execute JavaScript, or log into anything. RouxBot is a plain HTTP client that reads static HTML.
• Bypass paywalls, authentication, or anti-bot challenges. If a page requires login, RouxBot sees the login page and gives up.
• Cross a domain boundary. RouxBot only ever requests URLs on the same host as the page we indexed; it will never follow an outbound link to someone else's site.
• Use your pages to train AI models. We use a small classifier to decide whether a URL is a recipe, but your content is never used as training data — ours or anyone else's.
• Republish your content. Search results on Weeknight Dinners show a title, an image, and a snippet; the full recipe lives on your site and we always link back.

For the broader picture of how Weeknight Dinners handles recipe content, see the Content Policy. For how we use AI (and what we commit to never doing with it), see the AI Policy.

If RouxBot is misbehaving, ignoring your robots.txt, or fetching more than what’s described above, we want to know. The fastest way to reach us is the feedback form.

When reporting an issue, a few lines from your access log — timestamps, the User-Agent string, the path, and the response code — are enormously helpful. We’ll respond within a few days.