Server Log Monitoring for Ecommerce: What to Log and Why.

An online store does not have a server log. It has six of them, and each one records a different part of the same customer request. So "what should we log" is really the question "which failures do we need to be able to explain afterwards?"

The failure that makes the point is an ordinary one. A customer says the payment page hung and their card was never charged. The site was up all afternoon and the synthetic uptime probe stayed green. Because no single file contains the whole journey, and no shared identifier links the six records to each other, nobody can say at which tier the request stopped.

This article names the six tiers a store writes logs at, the fields inside them that decide what you can reconstruct later, and the question each tier is the only one able to answer. It does not recommend a tool. What you log is settled before where you send it.

What counts as a server log on an ecommerce site?

An ecommerce site writes logs at six tiers, and each one records a different slice of the same customer request: the CDN or edge, the web server, the application, the database, outbound calls to payment and fulfilment services, and internal site search. Each tier writes its own record with its own timestamp. A request that fails at one tier can therefore be absent from the next.

Follow one request. It arrives at the edge, where a CDN answers it from cache, blocks it, or passes it to the origin. Nginx or Apache receives what survives and writes one access log line. The application validates the basket and records what it decided, then queries the database. The database reports slow statements in a slow query log if one is switched on. The application calls a payment gateway and a fulfilment service, and those replies are recorded only because the application chose to record them. Internal site search runs inside the application and captures what shoppers typed.

Two tiers go missing most often. Application logs record business events rather than HTTP traffic, so a store can log every request and still not know which order failed. Internal site search logs and authentication logs are usually available and rarely collected. Which tiers you get by default depends on the platform the store runs on, since a hosted checkout hands you the gateway response and nothing underneath it.

Which access log fields actually matter?

Three fields decide what you can reconstruct after a failed checkout: how long the request took, what the upstream returned, and an identifier that survives across tiers. Combined Log Format, the default on most Nginx and Apache installations, records none of the three. It holds the Common Log Format's seven fields plus two more: the referrer and the User-Agent string.

A single anonymised line looks like this:

Reading left to right: the client address, two fields for identity and authenticated user that sit empty on almost every store, the timestamp with its offset, the request line's method, path and protocol, the status code, the bytes sent, the referrer, and the User-Agent.

Behind a CDN, that first address is your own edge rather than the shopper, unless you configure the web server to read X-Forwarded-For. Trust only the portion your edge appends, because a client can send the rest itself. Write timestamps in UTC in ISO 8601 order, so that sorting across tiers is arithmetic rather than interpretation.

Add the three missing fields explicitly. Log request duration and upstream response time separately, so you can tell your own queue from a slow gateway. Log the upstream status next to the status sent to the client, because an edge can turn a 502 into a custom error page that records as a 200. Generate an X-Request-ID at the edge and write it at every tier.

Keep the two kinds of identity distinct. A session identifier answers which shopper, and a request identifier answers which attempt. The web server writes one line per request from a configurable field list. A field missing from the format is missing from the record permanently.

Server logs and JavaScript analytics answer different questions

JavaScript analytics such as GA4 record what a browser successfully executed. Server logs record what the server was actually asked to do. The requests that matter most during an incident are precisely the ones analytics never sees.

Client-side collection needs the page to render and the script to run. A request produces no GA4 event if it returns an HTTP 5xx, times out, or is cut off mid-response. Neither does a shopper who declines the consent banner or blocks the script. Logs also hold traffic no browser generated: crawlers, scrapers, API clients, mobile apps, autonomous shopping agents, and inbound webhooks such as payment callbacks.

Analytics reads intent and sequence: which products were viewed, which step was abandoned. Logs read delivery and failure: what was requested, what came back, and how long it took.

Those shoppers never fired the events that would have shown the drop. Their sessions are absent rather than visibly failing, which is why an incident looks like a quiet afternoon.

Which status code patterns deserve an alert?

Absolute error counts are close to useless on a store, because traffic is seasonal and a hundred errors means nothing without the denominator. Alert instead on the ratio of errors to total requests, and on how fast that ratio is changing, scoped to the paths that take money.

On the checkout, cart or payment callback path, a 5xx is a different class of event from a 5xx on a product listing page. Grouping them together hides the first inside the second. The specific code narrows the cause: a 502 means the upstream replied with something unusable, a 503 means the upstream or the server itself refused the work, and a 504 means the upstream was too slow to answer.

Some 4xx patterns are worth reading closely. A burst of 404s following a catalogue change usually points at withdrawn products or rewritten URLs rather than at shoppers. A burst of 403s immediately after an edge rule change is normally the rule, not an attack. Sustained 401 volume against the login endpoint is an abuse signal.

Client-aborted requests are a latency symptom rather than an error, because the shopper left before the server replied. Nginx records these as 499. That code is a web server convention rather than a standard HTTP status code, so say so when you report it. Redirect chains on faceted category URLs cost crawl budget and page speed without ever registering as failures, since every hop returns a healthy 301.

Express every threshold per path group, measured against that group's own baseline rather than a site-wide number.

How do you trace one failed checkout across tiers?

You reconstruct a failed order by carrying one identifier through every tier the request touched, then reading those tiers in the order the request travelled. The tier that logged the error is rarely the tier that caused it. Tracing one failed checkout is the test that shows why tiering matters: the request crosses the edge, the application and the payment gateway, and each one holds a fragment of the answer.

Start from something the customer already has. Any one of four things is enough to find one application log line: an order reference, an email address, a basket identifier, or a timestamp accurate to the minute.

That line gives you the request identifier, and the identifier gives you every other tier. If no identifier exists, this is the moment you learn that you need one. The fix belongs in the log format rather than in the investigation.

Then read forward rather than backward. Begin at the edge record, move to the web server line, then the application's own account of what it decided, then the database, then the outbound call. Working backwards from the error message stops at the tier that reported the symptom.

The gateway's reply carries information the application's exception does not. A decline names a reason the issuer supplied, and a timeout says only that no answer arrived. The two demand opposite responses. An idempotency key is a value the application sends so that the gateway treats a repeated request as the same attempt rather than a new one. If you send one, its reuse tells you whether a retry created a second authorisation attempt against the same basket.

Timestamp-only correlation fails quietly. Clock skew between hosts and mixed local timezones can present events in the reverse of the order they actually happened. Keep every host on NTP, log in UTC, and correlate on the identifier.

Log patterns reveal bot traffic, credential stuffing and card testing

Abuse against a store shows up as request shape rather than request content: many attempts against one endpoint, thin sessions that skip the pages a human would load, and payment attempts whose failure pattern no genuine customer produces.

Credential stuffing looks like a high volume of POSTs to the login endpoint, spread across many distinct usernames, with a failure ratio far above the store's baseline. The client side of the pattern is either very few addresses or a great many addresses sharing one fingerprint, such as the same User-Agent and the same header order.

Card testing looks like repeated low-value authorisation attempts with a decline ratio no real checkout produces, and no browsing beforehand. The tell is a request to the payment endpoint with none of the catalogue, product and cart requests that precede a genuine purchase.

Scraping looks like a sequential walk through the catalogue, with no requests for static assets and no session continuity between requests. A browser fetches CSS, images and scripts while a script does not.

Declared crawlers such as Googlebot and GPTBot announce themselves in the User-Agent, which is a claim rather than an identification. Verify a claim with a reverse DNS lookup on the requesting address followed by a forward lookup back to it. Compare the result against robots.txt to see who ignored it. Separating the crawlers you want from the ones you don't is the same exercise as making a storefront legible to AI crawlers, read from the other end.

All four signatures sit in logs the store already writes. You find them by grouping requests by client, endpoint and outcome over a time window, then comparing the sequence against the sequence a real purchase produces. As more of that traffic becomes agent-driven rather than human, the shape of a legitimate session stops matching the browser sessions your baseline was built from.

What are you allowed to keep, and for how long?

Two constraints decide how long a UK store can keep its server logs. IP addresses in logs are personal data under UK GDPR, so keeping them requires a stated purpose and a stated limit. Cardholder data must never reach a log file.

An IP address plus a timestamp plus a request path can identify a person, which brings log files inside data minimisation and storage limitation. In practice that means naming the purpose each log serves, setting a retention period per tier, restricting who can read the files, and deciding whether truncating or pseudonymising the address still leaves the log useful. The store is the data controller, and the ICO is the authority that asks these questions.

PCI DSS goes further on payment data. Sensitive authentication data must never be stored after authorisation, and card numbers must not be readable. Once a full card number reaches a log line, it exists in backups, replicas and anything that shipped the file onward. Exclude both at the point of logging rather than in a redaction job that runs afterwards. The standard also sets a minimum retention period for audit logs plus a shorter window in which recent logs stay immediately available. Read the current version of the standard for those figures rather than trusting a number quoted second hand.

Three levers control volume: sample low-value paths such as static assets, shorten the hot storage window, and aggregate before archiving. Short retention protects the customer and destroys the evidence a slow-burning fraud pattern needs. Name that tension rather than pretending it away.

How should log monitoring change before a peak sales event?

Every threshold and baseline worth having during a peak sales event has to exist before the event starts, because a baseline captured on the day describes the incident rather than normal. The one signal worth watching live is the conversion path, not the servers.

Capture a normal-week baseline per path group first: error ratio, response time and request volume for checkout, cart, product and search. A threshold means nothing on the day unless it was derived from a week when nothing was wrong. Check retention and volume headroom in advance too, since peak traffic is exactly when log writes get dropped. Dropped logs remove the post-mortem.

During the event, watch the ratio of completed orders to attempted orders. CPU utilisation tells you how the machines feel, and that ratio tells you whether shoppers are getting through.

Synthetic probes and log alerts each fail in known ways. A synthetic uptime probe passes while checkout is broken for one payment method or one region, because the probe tests the path it was told to test. A log alert fires only if the request reached a tier that writes logs, so a failure at the edge or in a third-party script stays invisible. Run both and know which blind spot belongs to which.

Write down the times of every change, deployment and alert as the event runs. Correlation afterwards depends on that list existing, and it is the same discipline that keeps a bulk catalogue import from failing silently in the middle of a trading peak.

Where to start with server log monitoring

Take the last failed order anyone can name. Ask whoever owns the platform to reconstruct it from logs alone, from the edge request through to the gateway response. Give them one working day. The tier that goes silent during that exercise is the tier to fix first. The fix is usually a field that was never in the log format rather than a product that was never bought. Do it before the next peak trading date, because the baseline has to exist before the day it is needed.

Decide what to log by naming the failures you cannot currently explain, then work backwards to the tier that would have recorded them.