Add a Reader/Writer Interface for Streaming

[marmbrus]

In this PR I add a new interface for opening new streams (as Dataframes) and starting a new streaming query. These are modeled after the DataFrame reader/writer interface.

val df =
  sqlContext
    .streamFrom
    .format("text")
    .open("/michael/logs")

val filtered = df.filter($"value" contains "ERROR")

val runningQuery =
  filtered
    .streamTo
    .format("text")
    .start("/michael/errors")

runningQuery.stop()

Sources and Sinks are created by a StreamSourceProvider or StreamSinkProvider, which are similar to a RelationProvider in the Data Source API (and in fact a single class can be all of the above if desired).

I include a throwaway implementation of a text file source/sink for demonstration / testing.

TODO:

• Improve tests
• Improve comments

[AmplabJenkins]

Merged build finished. Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/21/
Test FAILed.

[AmplabJenkins]

Merged build finished. Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/22/
Test FAILed.

[AmplabJenkins]

Merged build finished. Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/23/
Test FAILed.

[AmplabJenkins]

Merged build finished. Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/24/
Test FAILed.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/25/
Test PASSed.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/26/
Test PASSed.

[zsxwing]

why use wait(100) instead of wait()?

[marmbrus]

Thats a really good question. When I had it as wait(), it was hanging non-deterministically. I think its okay to spin occasionally?

[tdas]

That sounds like a sign of corner cases we are missing.

[marmbrus]

From the docs interrupts and spurious wakeups are possible, and this method should always be used in a loop. Do my guess is that we some times wake up spuriously. There is then a race to check the done condition / finish the batch (which is why it would hang with very low probability (3/1000)). So, this actually does seem like the best solution. We don't spin a ton wastefully. In most takes we awake immediately. In very rare cases we sleep 100ms too long.

[tdas]

awaitBatchCompletion? awaitNextBatch doesnt signify whether to wait for next batch to start or end.

[marmbrus]

changed

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/29/
Test PASSed.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/30/
Test PASSed.

[tdas]

Writing reader.format("kafka") is a quite weird, and will be weird for most non-fs streaming sources. Rather I propose having an alias called source, which works nice for both batch and streaming - source("text"), source("parquet"), source("kafka") all make sense.

[marmbrus]

/cc @rxin

[rxin]

I think this depends on what other methods are available on the reader/writer interfaces.

[rxin]

ah never mind -- i misunderstood it. your proposal makes sense

[AmplabJenkins]

Merged build finished. Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/35/
Test FAILed.

[AmplabJenkins]

Merged build finished. Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/36/
Test FAILed.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/spark-streaming-df-test/37/
Test PASSed.