Ecommerce Server Log Monitoring: What to Log, Alert On and Keep.

An online store does not have a server log. It has six of them, and each one records a different part of the same customer request. So “what should we log” is really the question “which failures do we need to be able to explain afterwards?”

The failure that makes the point is an ordinary one. A customer says the payment page hung and their card was never charged. The site was up all afternoon and the synthetic uptime probe stayed green. Because no single file contains the whole journey, and no shared identifier links the six records to each other, nobody can say at which source the request stopped.

Ecommerce server log monitoring is the continuous collection, centralisation and evaluation of the records a store’s systems produce, so that a failure raises an alert rather than a support ticket. Four things get confused under that name. Logging writes the record. Aggregation moves records from many systems into one place. Monitoring evaluates them continuously and alerts. Log analysis investigates a pattern or an incident afterwards. A store can write excellent logs and still have no monitoring, if those records sit on separate hosts with inconsistent clocks and nothing to join them by.

This article names the six sources a store writes logs at, which of them your platform lets you read, the fields that decide what you can reconstruct later, and how records become alerts. It does not recommend a tool. What you log is settled before where you send it.

What counts as a server log on an ecommerce site?

An ecommerce site writes logs at six sources, and each one records a different slice of the same customer request: the CDN or edge, the web server, the application, the database, outbound calls to payment and fulfilment services, and internal site search. Each source writes its own record with its own timestamp. A request that fails at one source can therefore be absent from the next.

Follow one request. It arrives at the edge, where a CDN answers it from cache, blocks it, or passes it to the origin. Nginx or Apache receives what survives and writes one access log line. The application validates the basket and records what it decided, then queries the database. The database reports slow statements in a slow query log if one is switched on. The application calls a payment gateway and a fulfilment service, and those replies are recorded only because the application chose to record them. Internal site search runs inside the application and captures what shoppers typed.

Six is the useful minimum, not the ceiling. Depending on architecture a store also writes load balancer, WAF, authentication, queue, webhook and container logs, and on a headless build the application is two sources, the storefront and the commerce API. The number matters less than the principle. Each system records only what it received and returned, so no single log is complete: an origin access log cannot show a request answered at the edge, and an application log cannot show a request that failed before reaching the application. Anything you cannot join to the others is a fragment.

Two sources go missing most often. Application logs record business events rather than HTTP traffic, so a store can log every request and still not know which order failed. Internal site search logs and authentication logs are usually available and rarely collected.

Which of these can you see on your platform?

How many of these records you can read depends on who runs the infrastructure, and the answer varies more than “hosted versus self-hosted” suggests. This is what each vendor’s own documentation offers, as of September 2026, for the platforms most stores run on.

Shopify, Shopify Plus

Edge

No. A proxy or CDN in front of the store is unsupported

Web server

No

Application

No. App developers see their own function and webhook logs only

Database

No

Gateway

Transaction status, gateway and error code on the order’s transactions; the orders/create and orders/paid webhooks deliver the order itself

Site search

Reports only: searches with no results, no clicks

BigCommerce

Edge

Not documented

Web server

No

Application

Store Logs: orders, payment transactions, shipping quotes, kept 365 days

Database

No

Gateway

Payment transaction entries in Store Logs; webhooks carry ids only

Site search

In-Store Search report, including keywords with no results

Salesforce Commerce Cloud

Edge

Yes: eCDN Logpush with client IP, URL, user agent and firewall events; eCDN error logs in Log Center

Web server

No origin access log file; the request record lives at the eCDN

Application

Yes: system and custom logs, Log Center, log streaming, alerts

Database

No

Gateway

Whatever your payment code writes to the custom log

Site search

Not listed among the log files

Adobe Commerce on cloud infrastructure (Pro)

Edge

Fastly and WAF data via New Relic

Web server

Yes: Nginx access and error logs

Application

Yes

Database

MySQL error and slow logs, seven-day lifetime, production and staging only

Gateway

Whatever the application records

Site search

Elasticsearch logs

Adobe Commerce, WooCommerce or PrestaShop on your own hosting

Edge

Yes, if you front it with a CDN that streams logs

Web server

Yes

Application

Yes: Magento’s system and exception logs, WooCommerce > Status > Logs, PrestaShop’s admin Logs page

Database

Yes

Gateway

Whatever the application or extension records

Site search

Yes, if you run the search service

Headless (Next.js on Vercel, Hydrogen on Oxygen)

Edge

Vercel log drains on Pro and Enterprise carry client IP, status, cache result and WAF action

Web server

Storefront host runtime logs: all Vercel plans, one hour to thirty days by plan and add-on; Oxygen request logs on Shopify Plus only

Application

Split between the storefront and the commerce API, and the Shopify half stays opaque

Database

Depends on what you run

Gateway

Only what your checkout integration logs

Site search

Yes, if you run it

On Shopify or BigCommerce you are handed the gateway’s decision and the browser’s behaviour, and very little in between. Shopify’s help pages say a Cloudflare proxy in front of the store, including Orange-to-Orange, is not supported, which closes the workaround most people reach for. Push an order or basket identifier into your webhook payloads, your analytics events and your support tickets, because those are the only records you own. Shopify’s order object does carry the shopper’s IP address and user agent, so the browser side of a failed checkout is recoverable from the order even though the request side is not.

Salesforce Commerce Cloud is hosted but gives you custom logs from your own code, a Log Center with streaming and alerts, and request-level logs from its Cloudflare-based edge. Adobe Commerce on Adobe’s cloud infrastructure gives you the Nginx and PHP logs and, on Pro production and staging, the MySQL slow log, with application logs archived for thirty days by default.

On self-hosted Magento, WooCommerce or PrestaShop you have every source listed above, and will usually find that several of them are switched off or writing the default format. Availability is not the constraint. Configuration is.

One WordPress caution. Turning on WP_DEBUG_LOG in production writes PHP errors, and whatever request data they carry, to wp-content/debug.log, which the WordPress handbook itself describes as a publicly accessible location and a security risk. WooCommerce’s own logger has the same shape: by default it writes to wp-content/uploads/wc-logs/, protected by an .htaccess file that, in WooCommerce’s words, not all web servers recognise. Move both above the web root before a payment extension writes anything there.

Which access log fields actually matter?

Three fields decide what you can reconstruct after a failed checkout: how long the request took, what the upstream returned, and an identifier that survives across sources. Combined Log Format, the default on most Nginx and Apache installations, records none of the three. It holds the Common Log Format’s seven fields plus two more: the referrer and the User-Agent string.

A single anonymised line looks like this:

203.0.113.4 - - [14/Nov/2025:13:42:07 +0000] "POST /checkout/payment HTTP/1.1" 502 1043 "https://example.com/checkout" "Mozilla/5.0 (iPhone; CPU iPhone OS 17_4)"

Reading left to right: the client address, two fields for identity and authenticated user that sit empty on almost every store, the timestamp with its offset, the request line’s method, path and protocol, the status code, the bytes sent, the referrer, and the User-Agent.

Behind a CDN, that first address is your own edge rather than the shopper, unless you configure the web server to read X-Forwarded-For. Trust only the portion your edge appends, because a client can send the rest itself. On Nginx that trust boundary is the realip module: set_real_ip_from lists the edge addresses you believe, real_ip_header X-Forwarded-For names the header, and real_ip_recursive on walks the chain back to the last address that is not itself trusted. Anything to the left of that in the header is whatever the client chose to write. Log the original connecting address too ($realip_remote_addr), so you still know which edge node handed you the request.

Add the three missing fields explicitly. Log request duration and upstream response time separately, so you can tell your own work from a slow gateway. Log the upstream status next to the status sent to the client, because an edge or an error_page rule can turn a 502 into a custom error page that records as a 200. Generate a request identifier at the edge and write it at every source.

On Nginx that is one log_format in the http block:

log_format store '$remote_addr "$http_x_forwarded_for" [$time_iso8601] '
                 '"$request" $status "$upstream_status" $body_bytes_sent '
                 '$request_time "$upstream_response_time" '
                 '"$http_referer" "$http_user_agent" $request_id';

access_log /var/log/nginx/access.log store;

And, inside the location that proxies to your application, next to the proxy_set_header lines you already have:

proxy_set_header X-Request-ID $request_id;

Nginx only inherits proxy_set_header directives into a block that defines none of its own, so a lone X-Request-ID line at the top of the file silently disappears in any location that already sets Host or X-Forwarded-For.

What the fields mean, per the Nginx documentation. $request_time runs from the first bytes read from the client to the log write after the last bytes were sent back, so it includes reading a slow phone’s request and sending the response over a slow connection, not only your server’s work. $upstream_response_time is the slice spent waiting on your application or, through it, the gateway. The gap between the two is everything that was not the upstream: your own processing, and the client’s connection. $upstream_connect_time and $upstream_header_time split the upstream slice further. $upstream_status is what the application returned, which can differ from $status once an error page rewrites it. $request_id is a 32-character hexadecimal identifier Nginx generates per request. The three quoted fields can hold comma-separated lists, the forwarded-for chain always and the upstream pair whenever Nginx tries more than one upstream, which is why they are quoted: unquoted, a retry adds columns to the line and shifts every parser downstream.

Two details bite. $time_iso8601 is the server’s local time in ISO 8601 form, so it is only UTC if the server’s timezone is UTC. Set the timezone, then log ISO 8601, and sorting across sources becomes arithmetic rather than interpretation. Apache’s equivalents are LogFormat and CustomLog, with %D for the request time in microseconds; Apache has no built-in upstream timing field.

The access log tells you a 502 happened. The error log tells you why. Nginx’s error_log carries the upstream diagnosis in plain words, upstream prematurely closed connection, connect() failed (111: Connection refused), upstream timed out, no live upstreams, each with the client, the request and the upstream address it was talking to. Apache’s is the ErrorLog directive. Ship both. An access log without its error log is a list of symptoms with the diagnosis left behind on the server.

The web server writes one line per request from a configurable field list. A field missing from the format is missing from the record permanently.

Structured logging, request IDs and trace context

The web server’s request identifier is one of five identifiers a checkout request accumulates, and they answer different questions:

• Request ID: one HTTP request.
• Trace ID: one operation spanning several services, which on a store means the storefront, the checkout API and the gateway call.
• Session ID: one browsing session, which shopper.
• Order or basket ID: the commercial transaction.
• Payment attempt ID: one authorisation attempt against that basket.

A session identifier answers which shopper, and a request identifier answers which attempt. A payment attempt can span several requests.

This is the problem distributed tracing exists to solve. Whether you adopt OpenTelemetry and propagate the W3C traceparent header, or hand-roll an X-Request-ID, the mechanism is identical: one value, written by every source, queryable in one place. That last clause is the one people skip. Six logs on six hosts sharing an identifier still cost you six SSH sessions. The identifier only pays off once the sources land in a single searchable store.

The W3C Trace Context Recommendation (Level 1, November 2021) defines traceparent as four hyphen-separated fields: a version, a 32-character trace ID, a 16-character parent ID and a flags byte. The OpenTelemetry Collector receives traces, metrics and logs, and the OpenTelemetry log data model carries the trace ID and span ID on a log record wherever the emitter can supply them, which is how a log line and a trace end up joined without anyone grepping.

Do not trust an identifier supplied by the public client. The Trace Context spec itself warns that an attacker can forge trace ID collisions that make monitoring data unusable, and recommends different behaviour for unauthenticated requests. A client-supplied value written straight into a log is also the textbook log injection case: newlines in the value forge extra log entries. Validate or replace the identifier at the edge. Nginx’s default log_format escaping neutralises quotes and control characters in variables, so the exposure is mostly in application logs that write raw headers.

Prefer structured JSON over positional fields where your tooling allows it. A named-key line survives field reordering, and it lets you add a field next quarter without breaking the parser that consumed last quarter’s logs. A hypothetical application log line for the failure at the top of this article, carrying identifiers and not credentials, might read:

{
  "timestamp": "2025-11-14T13:42:07.381Z",
  "service": "checkout-api",
  "environment": "production",
  "request_id": "7f31c0e2a9d84b6f8e2d1c4a5b6e7f80",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "route": "/checkout/payment",
  "method": "POST",
  "status": 502,
  "upstream_service": "payment-gateway",
  "upstream_status": 504,
  "duration_ms": 1843,
  "order_id": "ord_82c1",
  "payment_attempt_id": "pay_49a2",
  "error_code": "GATEWAY_TIMEOUT",
  "release": "2025.11.14-3"
}

The keys are the point, not the JSON. upstream_status and duration_ms are what the default access log lacks, payment_attempt_id is what the gateway’s dashboard will ask you for, and release is what tells you whether the 502s started with a deploy.

Server logs and JavaScript analytics answer different questions

JavaScript analytics such as GA4 record what a browser successfully executed. Server logs record what each system received and returned. The requests that matter most during an incident are precisely the ones analytics never sees.

Client-side collection needs the page to render and the script to run. A request produces no GA4 event if it returns an HTTP 5xx, times out, or is cut off mid-response. Neither does a shopper who declines the consent banner or blocks the script. Safari’s Intelligent Tracking Prevention caps cookies written by JavaScript at seven days, and at twenty-four hours on a landing page reached through a link decorated by a domain ITP has classified as a tracker, so returning Safari shoppers fragment into new users in analytics while the access log sees one address returning all month.

Logs also hold traffic no browser generated: crawlers, scrapers, API clients, mobile apps, autonomous shopping agents, and inbound webhooks such as payment callbacks. GA4 goes the other way on purpose. It excludes traffic from known bots and spiders automatically, using its own research and the IAB’s International Spiders and Bots List, and that exclusion cannot be turned off or its volume seen. A crawler that never executes the tag sends nothing in the first place. Apart from the app SDKs, the only route by which non-browser traffic enters GA4 is the server-side Measurement Protocol, which Google describes as supplementing tagged collection rather than replacing it.

Analytics reads intent and sequence: which products were viewed, which step was abandoned. Logs read delivery and failure: what was requested, what came back, and how long it took.

Those shoppers never fired the events that would have shown the drop. Their sessions are absent rather than visibly failing, which is why an incident looks like a quiet afternoon.

How records become monitoring

Five stages separate a log file from an alert, and a store can be missing any of them.

1. Generate structured records at the edge, the web server, the application, the database and every outbound call, with the shared identifier in each.
2. Collect and ship them: a log agent on each host, syslog, an OpenTelemetry Collector, or the native log streaming your CDN and hosting platform offer.
3. Centralise and index them in one place: a log management platform, an observability platform, or a SIEM if security is the primary consumer.
4. Enrich and correlate: environment, service, release version, request ID, route group and region attached to every record, so a question can be asked across sources rather than per file.
5. Dashboard and alert on error ratio, latency percentiles, payment outcomes and ingestion health.

Stage five fails quietly when stage two does, so the pipeline’s own health is a signal to alert on; the section on alerts below treats it as one.

Each category of tool answers one question, which is why the categories can be named without naming a product. Structured logging and log aggregation answer where the records go. Distributed tracing answers how they join. A SIEM answers who reads the security half. A WAF, rate limiting and bot management answer what to do about the abuse patterns the logs reveal. Real user monitoring answers what the browser saw, which the origin cannot. Synthetic monitoring answers whether the path works right now. Naming categories is coverage. Naming winners is a buying guide.

Which patterns deserve an alert?

Absolute error counts are close to useless on a store, because traffic is seasonal and a hundred errors means nothing without the denominator. Alert instead on the ratio of errors to total requests, and on how fast that ratio is changing, scoped to the paths that take money.

On the checkout, cart or payment callback path, a 5xx is a different class of event from a 5xx on a product listing page. Grouping them together hides the first inside the second. The specific code narrows the cause: a 502 means the upstream replied with something unusable, a 503 means the upstream or the server itself refused the work, and a 504 means the upstream was too slow to answer.

Some 4xx patterns are worth reading closely. A burst of 404s following a catalogue change usually points at withdrawn products or rewritten URLs rather than at shoppers. A burst of 403s immediately after an edge rule change is normally the rule, not an attack. Sustained 401 volume against the login endpoint is an abuse signal. A 429 has two opposite meanings depending on which log it sits in: in your own access log it is your rate limiter working, and in your outbound call log it means the gateway is rate limiting you, which is a different emergency entirely. The absence of 429s proves nothing, since a server is not required to use the code when it limits, and Nginx’s return 444 drops the connection without any status at all.

Client-aborted requests are more often a latency symptom than an error, because the shopper left before the server replied. Nginx records these as 499, its own code for a client that closed the connection before Nginx sent the response headers. That code is a web server convention rather than a standard HTTP status, so say so when you report it, and Apache has no equivalent at all, only an X in the connection-status field. Rising 499 volume can indicate latency, but it can also be ordinary navigation, a cancelled fetch, a bot or a dropped mobile connection. Read it alongside response times and the route group’s own baseline: rising 499s with rising p95 on the same route is a latency story, and rising 499s with flat latency usually is not.

Redirect chains on faceted category URLs cost crawler time and page speed without ever registering as failures, since every hop returns a healthy 301. Retired products deserve a 410 rather than a 404 where you know they are not coming back: the HTTP specification reserves 410 for a condition that is likely to be permanent and says the server wants links to it removed, and says to use 404 when you do not know. A 200 that carries an error body, an empty search result page or a “connection to the database failed” message, is what Google calls a soft 404, and it is invisible to a status-code alert by definition. Log the upstream status and the response size, and the empty 200s stand out.

Status codes are one of four alert families. The other three are where stores lose money.

Latency. Alert on the 95th or 99th percentile per route group, not the average, because averages hide the shoppers who left. Google’s SRE book puts it bluntly: if you are not measuring the distribution, the idea that most requests sit near the mean is hopeful thinking, and 99th-percentile latency over a short window can give a very early signal of saturation. Time to first byte, the number people search for, is a browser measurement that includes DNS, TCP and TLS, so it belongs to real user monitoring; the nearest thing an origin log can give you is $upstream_header_time, the wait for the application’s first response bytes.

Business outcomes. These are the alerts vendors never write about:

• Attempted payments against authorised payments, per gateway and per method.
• An order created with no payment result.
• A payment confirmed with no order created.
• Duplicate authorisation attempts against one basket.
• Zero-result site searches as a share of all searches.
• Orders per session falling while the error ratio stays flat.

Each of these is what the SRE book calls an implicit error: a 200 with the wrong outcome. None of them is visible in a status-code dashboard.

Ingestion health. A log source that goes quiet looks identical to a source with no errors. Alert on the volume from each source reaching zero, and on it dropping below its own baseline.

Express every threshold per route group, measured against that group’s own baseline rather than a site-wide number.

How do you trace one failed checkout across sources?

You reconstruct a failed order by carrying one identifier through every source the request touched, then reading those sources in the order the request travelled. The source that logged the error is rarely the source that caused it. Tracing one failed checkout is the test that shows why the six sources matter: the request crosses the edge, the application and the payment gateway, and each one holds a fragment of the answer.

Start from something the customer already has. An order reference, a basket identifier, a payment attempt identifier, or a timestamp accurate to the minute is enough to find one application log line. If support starts from an email address, resolve it to an internal identifier in an authorised system first. Do not grep operational logs for an email, because the moment that works, the email was in a file it should not have been in.

That line gives you the request identifier, and the identifier gives you every other source. If no identifier exists, this is the moment you learn that you need one. The fix belongs in the log format rather than in the investigation.

Then read forward rather than backward. Begin at the edge record, move to the web server line, then the application’s own account of what it decided, then the database, then the outbound call. Working backwards from the error message stops at the source that reported the symptom.

The gateway’s reply carries information the application’s exception does not. A decline names a reason the issuer supplied, and a timeout says only that no answer arrived. The two demand opposite responses. Gateways return that reason in a structured form and give the request an identifier of their own, and both belong in your log next to your request ID, because the provider’s dashboard is where a chargeback dispute is eventually argued. Stripe returns a Request-Id response header on every call and puts the decline detail in the charge’s outcome, with its own decline code and the network’s. Adyen returns a sixteen-character pspReference and, for a refused authorisation, a refusalReason and refusalReasonCode inside an HTTP 200, so a status-code alert will never see an Adyen decline. Braintree returns a processor response code and text and asks you to log the code in case the customer calls. Access Worldpay returns a WP-CorrelationId header it asks you to keep. Log the gateway’s identifier, not its raw request and response bodies, which is where card data leaks into logs.

A 3D Secure challenge is a pending step, not a failure. Adyen reports it as a ChallengeShopper result code and Stripe records the flow as challenge on the charge’s three_d_secure property. An alert that treats a challenge as a declined attempt fires on every shopper whose bank asked a question.

An idempotency key is a value the application sends so that the gateway treats a repeated request as the same attempt rather than a new one. If you send one, its reuse tells you whether a retry created a second authorisation attempt against the same basket. Stripe takes it as an Idempotency-Key header on POST requests, up to 255 characters, and keeps it for at least 24 hours. Adyen takes an idempotency-key header up to 64 characters and recommends a UUID. PayPal’s REST API takes PayPal-Request-Id, on the endpoints that support it. Worldpay’s WPG gateway takes Idempotency-Key for card, wallet and network token payments, and tells you to speak to your relationship manager before integrating it. Braintree documents no client-supplied key and runs duplicate transaction checking on its side instead, so on Braintree the duplicate shows up as a gateway rejection rather than a replayed response. Stripe’s documentation adds one line that belongs in every store’s runbook: do not use email addresses or other personal identifiers as idempotency keys.

Timestamp-only correlation fails quietly. Clock skew between hosts and mixed local timezones can present events in the reverse of the order they actually happened. Keep every host on NTP, log in UTC, and correlate on the identifier.

Log patterns reveal bot traffic, credential stuffing and card testing

Abuse against a store shows up as request shape rather than request content: many attempts against one endpoint, thin sessions that skip the pages a human would load, and payment attempts whose failure pattern no genuine customer produces.

Credential stuffing looks like a high volume of POSTs to the login endpoint, spread across many distinct usernames, with a failure ratio far above the store’s baseline. The client side of the pattern is either very few addresses or a great many addresses sharing one fingerprint, such as the same User-Agent and the same header order.

Card testing looks like repeated low-value authorisation attempts with a decline ratio no real checkout produces, and no browsing beforehand. The tell is a request to the payment endpoint with none of the catalogue, product and cart requests that precede a genuine purchase.

Scraping looks like a sequential walk through the catalogue, with no requests for static assets and no session continuity between requests. A browser fetches CSS, images and scripts while a script does not. That is a heuristic, not proof: a headless browser fetches the CSS too, and a legitimate API client never does.

Declared crawlers announce themselves in the User-Agent, which is a claim rather than an identification. Google’s list runs to Googlebot Smartphone and Desktop, Storebot-Google for Shopping, and GoogleOther. Bing’s is bingbot. OpenAI runs GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for fetches a person asked for. Anthropic runs ClaudeBot, Claude-SearchBot and Claude-User. Perplexity runs PerplexityBot and Perplexity-User. Google-Extended is not a crawler at all: it has no user agent string of its own and appears only as a robots.txt token that controls whether content Googlebot already fetched may be used for Gemini training and grounding, with no effect, Google says, on Search inclusion or ranking. OpenAI and Perplexity both document that their user-triggered fetchers may ignore robots.txt, since a person asked for the page, and Google’s do the same.

Verify a claim before you count it. Google and Bing both document a reverse DNS lookup on the requesting address followed by a forward lookup back to it, against googlebot.com, google.com and googleusercontent.com for Google and search.msn.com for Bing, and both publish JSON lists of their address ranges. OpenAI, Anthropic and Perplexity publish address-range JSON files and no verifiable hostnames, so for those three an IP match is the only check the vendor supports, and the files carry a creation date you should read before trusting them. Then compare the verified traffic against robots.txt to see who ignored it. Robots.txt is advisory: Google’s own documentation says it cannot enforce crawler behaviour, and Shopify’s says its rules are directional. If you need something enforced, enforce it at the edge. Separating the crawlers you want from the ones you don’t is the same exercise as making a storefront legible to AI crawlers, read from the other end.

All four signatures sit in logs the store already writes. You find them by grouping requests by client, endpoint and outcome over a time window, then comparing the sequence against the sequence a real purchase produces. The response is rate limiting at the edge, a WAF rule or a bot management product, and the log is what tells you whether the rule caught the intended traffic or produced a wave of 403s from real shoppers. As more of that traffic becomes agent-driven rather than human, arriving through an API or an MCP server, the shape of a legitimate session stops matching the browser sessions your baseline was built from.

What the same logs say about search crawling

The access log that shows a card-testing burst also shows where Googlebot spends its time, and on a store that allocation is rarely where you would put it. Group verified Googlebot Smartphone requests by template: product pages, category pages, filtered and sorted variants, internal search results, pagination, cart and checkout. Google indexes with the smartphone agent, so that is the crawler whose allocation matters.

The common finding is that filter and sort combinations take a large share of crawler requests while a portion of the product URLs in the sitemap go unrequested for weeks. Google’s own year-end crawling report for 2025, as Gary Illyes described it in February 2026, put about three quarters of the crawling problems Google saw down to faceted navigation and action parameters. That figure describes the problems Google saw across the web, not any one store’s allocation, which is what your log measures. Products no verified crawler has requested in thirty days are products Google cannot have seen recently, whatever the site-level Page Indexing report says.

Compare the log figures against the Crawl Stats report in Search Console. They should roughly agree, and Google says the report counts most requests but not all. Where they disagree, the direction tells you why. Logs higher than the report usually means you counted user agents you never verified. Logs lower usually means the edge answered the crawler before the origin saw it, so the request exists only in the CDN log. The report far higher than your logs has one documented cause: when your robots.txt file was unavailable, Google counts the crawls it would have made and does not make them.

The alerts section recommended a 410 for retired products. Google’s current documentation treats all 4xx codes except 429 the same and recommends either for a permanently removed page, so the gain is small: older statements from Google in 2012 and 2018 said a 410 sometimes falls out of the index a couple of days faster, and one small 2023 experiment found 404s recrawled about half again as often as 410s. The 410 is the honest code, and a log full of 404s for products you retired on purpose is noise you chose. The redirect chains on faceted URLs mentioned earlier show up here as requests that never reach a product.

A log entry proves a crawler requested a URL. It does not prove the URL was indexed, and it says nothing at all about ranking. Crawl budget is a concern Google reserves for large or fast-changing sites: its guide is addressed to sites with over a million pages that change weekly, sites with over ten thousand pages that change daily, or sites with a large share of URLs stuck at “Discovered, currently not indexed”. A three-hundred-SKU store does not have a crawl budget problem. It may still have a discovery problem, and the log is the only place that shows it.

Your own alerting feeds back into crawling. Google documents that 5xx responses, 429s and rising latency all lower the rate at which it crawls a host, and that returning 503 or 429 for more than two or three days can reduce crawling in the longer term. A rate limiter that catches Googlebot Smartphone is a search problem you created with a security control.

What are you allowed to keep, and for how long?

Two constraints decide how long a UK store can keep its server logs. Online identifiers in logs, an IP address first among them, are treated as personal data where they relate to an identifiable person, so keeping them requires a stated purpose and a stated limit. Card data must be kept out of log files by design.

The first constraint is UK GDPR. Article 4(1) defines personal data by reference to identifiers including “an online identifier”, and Recital 30 names internet protocol addresses as the first example. The ICO’s guidance lists IP addresses first among online identifiers and says identifiability is assessed on the identifier alone or in combination with other information you hold. An IP address plus a timestamp plus a request path is that combination. The EU GDPR text is identical on both points, and California’s CCPA lists “Internet Protocol address” in its statutory definition of personal information; other US state laws vary and are not covered here. What follows is written for a UK store answering to the ICO, but the mechanics transfer.

In practice that means naming the purpose each log serves, setting a retention period per source, restricting who can read the files and auditing that access, and deciding whether truncating or pseudonymising the address still leaves the log useful. The ICO does not set time limits; it expects you to justify yours, write them into a retention schedule and review them. Article 32 lists pseudonymisation and encryption as example security measures “as appropriate”, and the ICO calls encryption an appropriate measure and recommends it for stored and transmitted personal data, so logs are encrypted in transit to the aggregator and at rest inside it. A log management vendor that receives your access logs is a processor, and Article 28(3) requires a written contract covering, among eight terms, sub-processors and what happens to the data when the contract ends. Log rotation is the operational twin of retention: the file that rotated away last Tuesday is the one the investigation needs, and rotation set for disk space rather than for purpose is how evidence disappears on schedule.

The second constraint is PCI DSS, and the current version is v4.0.1, the only active version since v4.0 was retired on 31 December 2024. Requirement 3.3.1 says sensitive authentication data is not stored after authorisation, “even if encrypted”; its sub-requirements name the full track contents, the card verification code (the three or four digits the brands call CVV2, CVC2 or CID) and the PIN. Requirement 3.5.1 says the primary account number is rendered unreadable anywhere it is stored, by one-way hash, truncation, index token or strong cryptography. A log file is storage. The previous version of the standard, v3.2.1, said so explicitly, listing audit logs among the places a PAN must be unreadable, and its testing procedures had assessors examine “all logs, history files, trace files” for stored verification codes. Requirement 3.4.1 caps what may be displayed at the BIN and the last four digits. Use the payment token, the provider’s reference and the last four as your operational identifiers, and exclude the rest at the point of logging rather than in a redaction job that runs afterwards. Once a full card number reaches a log line, it exists in backups, replicas and anything that shipped the file onward.

On retention, Requirement 10.5.1 says to retain audit log history for at least twelve months, with at least the most recent three months immediately available for analysis. Under v3.2.1 that was Requirement 10.7, “at least one year”, so check which version you are assessed against before quoting a number. The requirement covers audit logs of in-scope system components, not every CDN object log your business happens to produce. It cuts the other way too: PCI’s own guidance names logging servers among the connected systems that must be brought into scope, so the platform that receives logs from your checkout hosts is in scope with them.

Three levers control volume: sample low-value paths such as static assets, shorten the hot storage window, and aggregate before archiving. Short retention protects the customer and destroys the evidence a slow-burning fraud pattern needs. Name that tension rather than pretending it away. It has a legal shape on both sides. The Court of Justice’s Breyer judgment, the case that made dynamic IP addresses personal data for a website operator, also held that an operator may have a legitimate interest in storing visitor data to protect itself against attacks. And Visa’s merchant dispute guidelines let you defend a card-absent fraud dispute with two or more earlier undisputed transactions that share at least two of device ID, device fingerprint and IP address with the disputed one, settled between 120 and 365 days before the dispute. A store that keeps its raw access log for a year is over-retaining. A store that cannot join an IP address and a device fingerprint to an order for a year is throwing away its defence.

How should log monitoring change before a peak sales event?

Every threshold and baseline worth having during a peak sales event has to exist before the event starts, because a baseline captured on the day describes the incident rather than normal. Watch business outcomes and technical leading indicators together. Completed orders against attempted orders tells you whether shoppers are getting through. Latency percentiles, queue depth and database connection pressure tell you that they are about to stop.

Capture a normal-week baseline per route group first: error ratio, response time percentiles and request volume for checkout, cart, product and search, and log volume per source. A threshold means nothing on the day unless it was derived from a week when nothing was wrong. Check retention and volume headroom in advance too, since peak traffic is exactly when log writes get dropped. Dropped logs remove the post-mortem.

During the event, watch the ratio of completed orders to attempted orders. CPU utilisation tells you how the machines feel, and that ratio tells you whether shoppers are getting through.

Synthetic probes and log alerts each fail in known ways. A synthetic uptime probe passes while checkout is broken for one payment method or one region, because the probe tests the path it was told to test. A log alert fires only if the request reached a source that writes logs, so a failure at the edge or in a third-party script stays invisible. Run both and know which blind spot belongs to which.

Write down the times of every change, deployment and alert as the event runs. Correlation afterwards depends on that list existing, and it is the same discipline that keeps a bulk catalogue import from failing silently in the middle of a trading peak.

Where to start with server log monitoring

Take the last failed order anyone can name. Ask whoever owns the platform to reconstruct it from logs alone, from the edge request through to the gateway response. Give them one working day. The source that goes silent during that exercise is the source to fix first. The fix is usually a field that was never in the log format rather than a product that was never bought. Do it before the next peak trading date, because the baseline has to exist before the day it is needed.

The checklist, in the order the work usually goes:

• Inventory every edge, server, application, database and third-party log source, including the ones your platform will never show you.
• Set every host’s clock by NTP and its timezone to UTC, then log timestamps in ISO 8601.
• Prefer structured JSON to free text where the tooling allows.
• Generate a request ID at the edge, replace any the client sent, and propagate it to every source.
• Log route, outcome, total duration, upstream duration, upstream status and release version on every request.
• Use order, basket and payment attempt identifiers as your keys, never an email address or card data.
• Centralise, and alert when any source stops sending.
• Build route-level dashboards for cart, checkout, login and payment callbacks.
• Alert on ratios and percentiles against each route group’s baseline, never on absolute counts.
• Segment payment monitoring by gateway, method and region.
• Baseline a normal week before the peak.
• Document retention and purpose per source, restrict access, audit it, and put the log platform inside your PCI scope if checkout hosts feed it.
• Test the whole thing by reconstructing one known failed order end to end.

Decide what to log by naming the failures you cannot currently explain, then work backwards to the source that would have recorded them.