AI agent workflows for Web research, scraping & price monitoring

Build spec

2 agents — 1 economy extractor (runs once per URL via the Loop node) + 1 premium arbiter (runs only on the Condition node's ambiguous branch). Loop, Parallel, Code, Condition and Approval are workflow nodes, not agents. · Pattern: Loop (per URL) → economy extract → Code/SpEL diff → Condition escalates only ambiguous records → premium arbiter → human approves any repricing

System prompt (paste-ready)

You extract ONE product page into strict JSON: {price (number|null), currency (ISO code|null), inStock (true|false|null), title, promoText|null, confidence (0-1), ambiguityReason|null}. Read ONLY the content supplied to you — never guess a price, never carry a value over from a previous page, and never fill a field from general knowledge about the brand. If the page shows several numbers (was/now, bundle, EMI, per-unit) and you cannot be certain which is the single sell price, do NOT pick one: set confidence below 0.6 and describe the ambiguity — the workflow escalates it. If the fetch returned a block page, a consent wall or a login wall, return {blocked: true} and stop; do not attempt to work around it.

Agent roles & model tiers

• Page extractor (per URL) — Economy tier — this is the volume step; it runs once per watched URL, so this single choice sets the cost of the whole sweep

  Turn one fetched page into schema-valid JSON. Nulls for anything not visibly present. Self-report low confidence rather than guessing between two numbers. Never work around a block/consent/login wall — report it.

• Ambiguity arbiter (only the flagged minority) — Premium tier — justified precisely because it runs on a handful of records, not the whole list

  Given one low-confidence extraction plus its page context, decide the single actual sell price and true availability, and explain the reasoning in one line. If it is still genuinely undecidable, return NEEDS_HUMAN — an honest escalation beats a confident wrong price.

Guardrails

• robots.txt and the site's Terms of Service are respected — the watchlist only contains pages you are permitted to fetch, and a disallowed path is dropped from the sweep rather than fetched anyway
• Rate-limited and backed off: a capped concurrency in the Parallel node, a delay between requests to the same host, exponential backoff on 429/5xx, and an honest identifying user agent — the sweep must never be indistinguishable from an attack
• Public data only. No page behind a login you do not own, no bot-detection bypass, and CAPTCHAs are NEVER solved — a block page, consent wall or login wall ends that item as {blocked: true} and is reported, not worked around
• No personal data harvesting: product/price fields only. Seller names, reviewer names and any other individual's details are dropped at the Code node — this workflow never builds a profile of a person
• No credentials, ever: no agent types a password, card number or government ID anywhere in this flow. There is nothing to log into here; if a source ever requires auth it is handled by a secrets-managed integration under your own account, not by an agent typing into a box
• Read-only by construction. This sweep submits no form and clicks no irreversible control; any proposed repricing — the one action with consequences — is held by the Approval node for a category manager
• Every reported figure carries its source URL and capture timestamp; an extraction below the confidence floor is escalated or marked NEEDS_HUMAN, never quietly published as fact
• The SpEL user-defined tools are data-transform expressions over the tool's JSON input — no method calls, no constructors, no bean or type references, and no loop in the grammar. A threshold the analyst wrote themselves therefore cannot become an execution risk; it can only compute a value

Cost strategy

This is the clearest cost-tiering story in the guide. Cost = N × (economy extractor) + a small k × (premium arbiter), where k is the ambiguous minority, not N. Put the premium model on the per-page step and a 600-URL sweep is expensive for no benefit — the per-page job is mechanical field-lifting that an economy model does well. Keep the premium tier for the genuinely undecidable record (three prices on one page, EMI vs sell price) and you pay for reasoning exactly where reasoning exists. Cap the extractor's output tokens (the schema is small), run the Parallel batch for wall-clock speed — which changes latency, not per-item cost — and set a hard per-run budget so a site redesign that breaks parsing fails loudly instead of expensively.

Output & delivery

The Loop maps the economy extractor across the watchlist → a Code node joins to last week's values and the SpEL flags fire → the Condition node escalates only the ambiguous records to the premium arbiter → any repricing proposal waits at the Approval node → an Output node posts the changed-only digest to Slack or Email with old → new, flag reason, source URL and capture timestamp per SKU.

Build spec

2 agents — 1 economy screener (once per listing, via the Loop node) + 1 standard-tier summariser (only on the shortlist). Knowledge, Condition and Approval are workflow nodes. · Pattern: Loop (per source) → economy screen → Knowledge-grounded relevance gate → standard-tier summary of the shortlist only → human bid/no-bid

System prompt (paste-ready)

(Screener) Read ONE public tender listing and return strict JSON {reference, title, buyer, estimatedValue|null, closingDate|null, url, relevant (true|false), relevanceReason}. Judge relevance ONLY against the capability profile and eligibility criteria supplied from Knowledge — not against your own assumptions about the industry. Never invent a closing date, a value or a reference number; null is always the correct answer for a field the listing does not show. If the listing is behind a login or a document paywall, return {gated: true} with the public URL and stop — do not attempt access.

Agent roles & model tiers

• Listing screener (per listing) — Economy tier — runs across every posting on every portal every day; the volume step

  One listing → schema-valid JSON plus a relevant/not verdict with a one-line reason grounded in the supplied capability profile. Nulls, not guesses. Gated content is reported, never accessed.

• Shortlist summariser — Standard tier — runs only on what survives the relevance gate, typically a handful a day

  For each shortlisted tender, write a two-line scope summary, list the eligibility flags that need checking, state the closing date, and note what would need assembling to bid. Use only the listing's own content; mark unknowns as unknown.

Guardrails

• robots.txt and each portal's Terms of Service are respected — a portal that forbids automated retrieval is watched by a human or not at all, and is removed from the source list rather than fetched quietly
• Rate-limited and honestly identified: one polite pass per source per day, delays between requests, backoff on 429/5xx — a daily watch has no reason to ever look like a crawl
• Public listings only. No login you do not own, no bot-detection bypass, and CAPTCHAs are NEVER solved — a gated document set is reported as {gated: true} with its public URL for a human to open
• No personal data harvesting: buyer organisation and public contact-of-record only, never named individuals compiled across postings into a profile
• No credentials, ever: the workflow does not hold or type portal passwords. Where a portal account exists, a human logs in — a secrets-managed integration under your own account is the only alternative, never an agent typing into a login box
• No registration, no expression of interest, no submission — the workflow reads and reports. Any consequential step, starting with bid/no-bid, is held by the Approval node for a human
• Every shortlisted item carries its source URL and the date it was seen; a null field stays null rather than being filled by inference
• The SpEL user-defined tools are pure data-transform expressions over the tool's JSON input — no method calls, no constructors, no bean/type references and no loops in the grammar, so a bids lead editing their own urgency band cannot introduce an execution risk

Cost strategy

Volume lives entirely in the screener, so it belongs on the economy tier — most postings on most days are irrelevant and only need a cheap yes/no with a reason. The standard-tier summariser sees only the survivors, which on a normal day is a handful. That ordering is the whole point: screen cheap, reason expensive, and never reason over the rejects. Cap the daily run budget so a portal that changes layout and starts returning junk fails visibly instead of billing quietly.

Output & delivery

The Loop screens each source with the economy model → the Knowledge-grounded Condition node keeps only relevant postings → the standard-tier summariser writes up the shortlist → the Approval node holds bid/no-bid → an Output node delivers the dated digest to Slack or Email, explicitly stating when a day genuinely had nothing.

Build spec

2 agents — 1 economy reader (runs once per source, fanned out by the Parallel node) + 1 premium synthesiser (runs exactly once). Parallel, Code and Condition are workflow nodes. · Pattern: Parallel fan-out (cheap readers) → Code merge/de-dupe → single premium synthesis → unsourced-claim gate

System prompt (paste-ready)

(Reader) Read ONE source and extract only what it actually says: {claims: [{claim, verbatimQuote, url, publishedDate|null}], sourceType}. Every claim MUST carry a verbatim quote from this source and this source's URL. Do not add context from your own knowledge, do not merge in what another source said, and do not infer a date the page does not state. If the page is a paywall, a consent wall or a login wall, return {blocked: true} — do not attempt to get around it. An empty claims list is a perfectly good answer for a page with nothing relevant.

Agent roles & model tiers

• Source reader (one per source, in parallel) — Economy tier — the fan-out step; N of these run per brief, so this tier sets the run's cost

  One source → claims with verbatim quotes and URLs. No outside knowledge, no cross-source merging, no inferred dates. Blocked pages are reported, never bypassed.

• Brief synthesiser (once) — Premium tier — one call over pre-digested, quoted material is exactly where a premium model earns its price

  Write the brief from the merged claims only. Name the themes, surface where sources disagree rather than averaging them, and state plainly what remains unknown. Every assertion keeps its URL. Never introduce a fact that is not in the merged input.

Guardrails

• robots.txt and each publisher's Terms of Service are respected; sources that disallow automated retrieval are read by a human, and paywalled content is neither fetched nor reproduced
• Rate-limited and backed off, with capped Parallel concurrency and honest identification — a research brief is never a reason to hammer a publisher
• Public data only: no login you do not own, no bot-detection bypass, and CAPTCHAs are NEVER solved. A paywall, consent wall or login wall ends that source as {blocked: true} and is listed as not-read in the brief
• No personal data harvesting — the brief covers organisations, products and policy, and does not compile information about named individuals across sources
• No credentials, ever: no agent types a password or subscription login into any site. Publisher subscriptions you hold are accessed by the human who holds them
• Nothing is published, posted or submitted by this workflow — it fetches and reports. If a brief is ever wired to an external publishing step, that submit goes through an Approval node first
• Unsourced-claim gate: the Condition node fails the brief if any claim lacks a URL, and sources that were blocked or empty are named as such rather than silently dropped
• The SpEL freshness tool is a data-transform expression over the tool's JSON input — no method calls, no constructors, no bean/type refs, no loops in the grammar — so an analyst-authored recency rule is a calculation and can never become executable behaviour

Cost strategy

Cost is N cheap reads plus ONE expensive synthesis, and the ordering is the entire savings. Running the premium model across thirty raw pages costs many times more and reads worse, because the strong model spends its budget on boilerplate navigation text instead of on reasoning. Let economy readers strip each page down to quoted claims, then spend the premium call on the merged digest. Parallel affects wall-clock only — per-source cost is unchanged — and a hard per-brief cap keeps an accidentally huge source list from becoming an accidentally huge bill.

Output & delivery

The Parallel node fans economy readers across the sources → a Code node merges and de-duplicates while preserving every URL → one premium pass writes the brief → the Condition node's unsourced-claim gate blocks any unattributed assertion → an Output node delivers the brief plus full citation list to Slack or Email and files it via a Tool (MCP) node.

Build spec

1 agent — 1 change-explainer agent, and it runs ONLY on pages the Code-node diff proved had changed. Loop, Code, Condition, Knowledge and Approval are workflow nodes. · Pattern: Loop (per page) → deterministic Code diff → Condition drops cosmetic churn → agent explains only real changes → owner acknowledges

System prompt (paste-ready)

You explain what changed on ONE watched page. You are given the previous and current text plus the computed diff. Return {changed (true|false), materiality (COSMETIC|MINOR|MATERIAL), plainSummary, beforeQuote, afterQuote, affectedProcesses[] (from the process register in Knowledge only), effectiveDate|null}. Quote verbatim — never paraphrase inside a quote field. Do not speculate about intent or about what the publisher will do next. If the diff is only formatting, dates or navigation, say materiality=COSMETIC and stop. You never decide the response; you describe the change so an accountable human can.

Agent roles & model tiers

• Change explainer (only on changed pages) — Standard tier — the diff has already removed the volume, so what is left genuinely needs careful reading; a page with a heavy legal diff can escalate to premium via the same Condition pattern

  Explain one real diff: plain summary, verbatim before/after quotes, materiality, and which registered processes it touches. No speculation, no paraphrased quotes, no recommended action.

Guardrails

• robots.txt and each site's Terms of Service are respected — only pages you are permitted to fetch are watched, and a disallowed page is removed from the list rather than fetched anyway
• Rate-limited and backed off with honest identification; a monthly or weekly cadence per page means there is never a reason to fetch a site aggressively
• Public pages only. No login you do not own, no bot-detection bypass, and CAPTCHAs are NEVER solved — a page that becomes gated is reported as no-longer-watchable so a human takes it over
• No personal data harvesting: the watch covers published policy, terms and register text, not individuals
• No credentials, ever — no agent types a password or ID into any site. A source that moves behind an account you hold is read by the human who holds that account, or through a secrets-managed integration, never by an agent typing a login
• The workflow submits nothing and clicks no irreversible control; the one consequential step, the owner's acknowledgement of a material change, is held by the Approval node
• Quotes are verbatim and every notice carries the source URL, capture timestamp and both snapshot ids — the evidence trail is the deliverable, not a nicety
• The SpEL materiality and urgency tools are data-transform expressions over the tool's JSON input, with no method calls, constructors, bean/type references or loops in the grammar — the officer's own threshold is a formula that cannot become executable behaviour

Cost strategy

The deterministic Code-node diff runs at zero model cost and eliminates the majority of items on a typical sweep, so most runs invoke no agent at all. Only genuinely changed pages reach the standard-tier explainer, and only a heavy legal diff needs to escalate above it. This inverts the usual failure mode of change monitoring — summarising every page every cycle — where the model bill scales with the page count instead of with the change count. Cap per-run spend so a site redesign that makes every page look 'changed' fails loudly rather than expensively.

Output & delivery

The Loop fetches each watched page → a Code node diffs it against the stored snapshot at no model cost → the Condition node discards cosmetic churn → the agent explains only real changes and maps them to processes from Knowledge → the accountable owner acknowledges at the Approval node → an Output node sends the change notice with before/after quotes while a Tool (MCP) node stores the new baseline snapshot.