Skip to content

Commit

Permalink
[SPARK-2362] Fix for newFilesOnly logic in file DStream
Browse files Browse the repository at this point in the history
The newFilesOnly logic should be inverted: the logic should be that if the flag newFilesOnly==true then only start reading files older than current time. As the code is now if newFilesOnly==true then it will start to read files that are older than 0L (that is: every file in the directory).

Author: Gabriele Nizzoli <mail@nizzoli.net>

Closes apache#1077 from gabrielenizzoli/master and squashes the following commits:

4f1d261 [Gabriele Nizzoli] Fix for newFilesOnly logic in file DStream
  • Loading branch information
gabrielenizzoli authored and tdas committed Jul 8, 2014
1 parent 32516f8 commit e6f7bfc
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: NewInputFormat[K,V] : Clas
// Files with mod time earlier than this is ignored. This is updated every interval
// such that in the current interval, files older than any file found in the
// previous interval will be ignored. Obviously this time keeps moving forward.
private var ignoreTime = if (newFilesOnly) 0L else System.currentTimeMillis()
private var ignoreTime = if (newFilesOnly) System.currentTimeMillis() else 0L

// Latest file mod time seen till any point of time
@transient private var path_ : Path = null
Expand Down

0 comments on commit e6f7bfc

Please sign in to comment.