Skip to content

[SPARK-4196][SPARK-4602][Streaming] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles - #3457

Closed
tdas wants to merge 2 commits into
apache:masterfrom
tdas:savefiles-fix
Closed

[SPARK-4196][SPARK-4602][Streaming] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles#3457
tdas wants to merge 2 commits into
apache:masterfrom
tdas:savefiles-fix

Conversation

@tdas

@tdas tdas commented Nov 25, 2014

Copy link
Copy Markdown
Contributor

Solves two JIRAs in one shot

  • Makes the ForechDStream created by saveAsNewAPIHadoopFiles serializable for checkpoints
  • Makes the default configuration object used saveAsNewAPIHadoopFiles be the Spark's hadoop configuration

@tdas tdas changed the title [SPARK-4196][SPARK-4602] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles [SPARK-4196][SPARK-4602][Streaming] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles Nov 25, 2014
srowen
srowen reviewed Nov 25, 2014
View reviewed changes
Comment thread streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala Outdated

@srowen srowen Nov 25, 2014

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah you're already on it, yep, that looks good.

@srowen

srowen commented Nov 25, 2014

Copy link
Copy Markdown
Member

Just had a thought -- will saveAsHadoopFiles and JobConf need the same treatment?

@tdas

tdas commented Nov 25, 2014

Copy link
Copy Markdown
Contributor Author

That's exactly I am testing now. Adding an unit test at the very least.

@SparkQA

SparkQA commented Nov 25, 2014

Copy link
Copy Markdown

Test build #23842 has started for PR 3457 at commit b382ea9.

  • This patch merges cleanly.

@tdas

tdas commented Nov 25, 2014

Copy link
Copy Markdown
Contributor Author

@srowen Yep, saveAsHadoopFiles had the same issue. Fixed it. Thanks for catching this bug!

@AmplabJenkins

AmplabJenkins commented Nov 25, 2014

Copy link
Copy Markdown

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23842/
Test FAILed.

@tdas

tdas commented Nov 25, 2014

Copy link
Copy Markdown
Contributor Author

Jenkins, test this please.

@SparkQA

SparkQA commented Nov 25, 2014

Copy link
Copy Markdown

Test build #23843 has started for PR 3457 at commit bb4729a.

  • This patch merges cleanly.

@SparkQA

SparkQA commented Nov 25, 2014

Copy link
Copy Markdown

Test build #23843 has finished for PR 3457 at commit bb4729a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Nov 25, 2014

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23843/
Test PASSed.

mengxr pushed a commit to mengxr/spark that referenced this pull request Nov 26, 2014
@tdas
[SPARK-4196][SPARK-4602][Streaming] Fix serialization issue in PairDS…
a9944c8 
…treamFunctions.saveAsNewAPIHadoopFiles

Solves two JIRAs in one shot
- Makes the ForechDStream created by saveAsNewAPIHadoopFiles serializable for checkpoints
- Makes the default configuration object used saveAsNewAPIHadoopFiles be the Spark's hadoop configuration

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes apache#3457 from tdas/savefiles-fix and squashes the following commits:

bb4729a [Tathagata Das] Same treatment for saveAsHadoopFiles
b382ea9 [Tathagata Das] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles.

(cherry picked from commit 8838ad7)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
@asfgit asfgit closed this in 8838ad7 Nov 26, 2014
asfgit pushed a commit that referenced this pull request Nov 26, 2014
@tdas
[SPARK-4196][SPARK-4602][Streaming] Fix serialization issue in PairDS…
7aa592c 
…treamFunctions.saveAsNewAPIHadoopFiles

Solves two JIRAs in one shot
- Makes the ForechDStream created by saveAsNewAPIHadoopFiles serializable for checkpoints
- Makes the default configuration object used saveAsNewAPIHadoopFiles be the Spark's hadoop configuration

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #3457 from tdas/savefiles-fix and squashes the following commits:

bb4729a [Tathagata Das] Same treatment for saveAsHadoopFiles
b382ea9 [Tathagata Das] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles.

(cherry picked from commit 8838ad7)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tdas @srowen @SparkQA @AmplabJenkins