Duplicate submissions and idempotency

There are only a handful of causes, and knowing which one you have determines the fix.

A visitor presses submit twice, usually because nothing appeared to happen. A visitor uses the browser back button and resubmits. A network hiccup means the response never arrived and the page retried. A delivery system retried a write that had actually succeeded. Or the same person genuinely enquired twice, a week apart, about different things.

The last one is not a duplicate at all, and any system that treats it as one is worse than the problem it is solving, because it discards a real enquiry in order to tidy a report.

That distinction is the whole subject: repeats are legitimate and duplicates are not, and they can look identical in the data.

Duplicates are treated as untidiness, which underrates them. They cost in three separate ways.

Two people on your team work the same enquiry, which wastes time and occasionally produces two different answers to the same customer. Your counts inflate, so conversion rates and campaign performance are quietly wrong. And the person on the other end receives two replies to one message, which reads as disorganisation.

The inflation is the most insidious because it is invisible. A campaign that produced forty submissions of which six were duplicates looks better than one that produced thirty-six clean ones, and budget follows the number.

It compounds where the duplicate rate is uneven. If one landing page has a slower form, or one device type produces more double clicks, then that page or that segment gets systematically flattered in every report you run. You end up spending more on the thing that is worse, and the data appears to support it.

Most double submissions come from a page that gives no feedback, and the fix is on the page rather than the backend.

Disable the button on click and change its label to something that indicates work in progress. Restore it if the request fails. Do not clear the fields until success. That combination removes the impatient-click case almost entirely.

The back-button case is handled by the post-redirect-get pattern, which has been standard practice for two decades and still gets omitted: redirect to a thank-you page after a successful post, so the browser has no form submission to repeat.

These two fixes are cheap and together they address the majority of real-world duplicates.

An operation is idempotent when performing it twice has the same effect as performing it once. It is the property that makes retrying safe.

For a form pipeline it means one submission produces one row, no matter how many times any part of the delivery is attempted. That is achieved by giving each submission an identity when it is accepted and tracking whether its delivery has succeeded.

A retry then asks a different question. Not send this again, but has this been delivered, and if not, deliver it. The difference sounds subtle and is the difference between a duplicate-free sheet and a messy one.

Without it, every reliability feature you add makes duplicates more likely, because retries are the mechanism.

Some cases cannot be resolved by the system, and it is worth being honest about which.

The same person submitting the same form twice in ten seconds is almost certainly a double click. The same person submitting an identical message four days later is probably chasing you because you did not reply. The same email address with a different message is a new enquiry.

A short time window plus content comparison covers the clear cases. Beyond that window, rejecting on the basis of similarity risks discarding a real follow-up, which is a much more expensive mistake than an extra row.

So the sensible default is to be aggressive within seconds and permissive after minutes.

When you do detect a likely duplicate, marking it beats deleting it.

A row flagged as a probable repeat of an earlier submission gives your team the context they need without destroying evidence. If the detection was wrong, nothing was lost. If it was right, the second row is a signal in itself: someone was anxious enough to submit twice, which is worth knowing when you reply.

This is the same principle as quarantine in spam handling. Automated judgement about ambiguous cases should narrow the human decision rather than replace it.

Most existing sheets have accumulated duplicates. Cleaning them is a one-time job worth doing before you build any reporting on the data.

Sort by email address and timestamp, and look for pairs within a few minutes of each other with matching content. Those are safe to collapse. Pairs days apart are almost always real repeat contacts and should be left alone.

Keep the earlier row when collapsing, because it carries the original attribution. The second submission often arrives without the campaign parameters that were on the first, since the visitor has since navigated away from the landing page.

That last detail catches people out and is a good reason to prefer flagging over deleting in the first place.

If you report on submissions at all, decide explicitly whether your numbers are deduplicated, and be consistent.

A count of rows is easy and wrong. A count of distinct people per period is harder and more useful. The gap between them is itself worth watching, because a sudden rise in duplicates usually means something on the page has broken, most often the feedback on submit.

That makes the duplicate rate a rather good canary for front-end problems you would otherwise never hear about.

Whether submissions carry a stable identity from acceptance onwards. Whether delivery state is tracked so a retry cannot write twice. Whether repeats are flagged rather than silently dropped.

A service that cannot answer the second question honestly is one whose retry behaviour is a liability, which matters because retries are exactly what you want during a downstream outage.

Reliability and duplicate control are the same design problem viewed from two sides. Systems that get one right usually get both.

Fix the page first, because it causes most duplicates and costs nothing. Require delivery to be idempotent, because it is what makes retrying safe. Flag ambiguous repeats instead of deleting them, because a wrongly deleted follow-up is a lost customer and an extra row is an inconvenience.

Then clean the history once, and start counting people rather than rows.

None of this needs a project. The page changes are an afternoon, the deduplication is an hour with a sorted sheet, and the counting change is a decision rather than a build. What it buys is a set of numbers you can trust and a team that stops answering the same enquiry twice.

What is the simplest fix for double submissions?

Should I block the same email address twice?

How short should the deduplication window be?

Do retries cause duplicates?

How do I clean up existing duplicates?

Why does the second submission often lack attribution?

Is a rising duplicate rate meaningful?

Should duplicates count in my reporting?

What about someone who submits from two different addresses?