Run keys stop duplicate runs, but the sensor still lists and checks everything on every tick. When the source is large, keep a cursor: a string Dagster stores for the sensor between ticks.
Read it with context.cursor (None on the first tick) and save a new one with context.update_cursor("..."), or return dg.SensorResult(run_requests=[...], cursor="..."). The cursor is only saved when the tick succeeds, so a failing tick retries the same window.
A cursor must be a string. Store a number with str(...), or several values with json.dumps. Pick something that only moves forward: a count of files in an append-only list, the latest modified time, or the newest row id. In the UI you can evaluate a tick with a different cursor, for example to reprocess from the start. Run keys still protect you then, so keep both.
Only files after the stored position are requested; the cursor moves to the end of the list.