005·75

watermark

/ˈwɔː.tər.mɑːrk/ - n.

1 [colloq.] A single stored number that is the difference between an incremental load and a four-hour incident.Keep. Punchy.This is the problem.

Working definition

2. A tracked marker, often a timestamp or sequence value, recording how far a pipeline has processed so the next run knows where to resume.

Filed

See also

change data captureCapturing every change the source makes, including the ones it would prefer you had not noticed.
idempotencyThe property everyone assumed the job had until the day it ran twice.
incremental loadProcessing only what changed, defined by a watermark everyone trusts and no one can explain.
late-arriving dataData that shows up after the meeting that needed it, with a perfectly good explanation no one waited to hear.