Repository navigation

[SPARK-23723] New charset option for json datasource - #20849

Closed

MaxGekk wants to merge 37 commits into

apache:masterfrom

MaxGekk:json-charset

Closed

[SPARK-23723] New charset option for json datasource#20849
MaxGekk wants to merge 37 commits into
apache:masterfrom
MaxGekk:json-charset

Commits on Mar 17, 2018
- Test for reading json in UTF-16 with BOM
  MaxGekk
  committed
  b2e92b4
- Use user's charset or autodetect it if the charset is not specified
  MaxGekk
  committed
  cb2f27b
- Added a type and a comment for charset
  MaxGekk
  committed
  0d45fd3
- Replacing the monadic chaining by matching because it is more readable
  MaxGekk
  committed
  1fb9b32
- Keeping the old method for backward compatibility
  MaxGekk
  committed
  c3b04ee
- testFile is moved into the test to make more local because it is used only in the test
  MaxGekk
  committed
  93d3879
- Adding the charset as third parameter to the text method
  MaxGekk
  committed
  15798a1
- Removing whitespaces at the end of the line
  MaxGekk
  committed
  cc05ce9
- Fix the comment in javadoc style
  MaxGekk
  committed
  74f2026
- Simplifying of the UTF-16 test
  MaxGekk
  committed
  4856b8e
- A hint to the exception how to set the charset explicitly
  MaxGekk
  committed
  084f41f
- Fix for scala style checks
  MaxGekk
  committed
  31cd793
- Run tests again
  MaxGekk
  committed
  6eacd18
- Improving of the exception message
  MaxGekk
  committed
  3b4a509
- Appended the original message to the exception
  MaxGekk
  committed
  cd1124e
- Multi-line reading of json file in utf-32
  MaxGekk
  committed
  ebf5390
- Autodetect charset of jsons in the multiline mode
  MaxGekk
  committed
  c5b6a35
- Test for reading a json in UTF-16LE in the multiline mode by using user's charset
  MaxGekk
  committed
  ef5e6c6
- Fix test: rename the test file - utf32be -> utf32BE
  MaxGekk
  committed
  f9b6ad1
- Fix code style
  MaxGekk
  committed
  3b7714c
- Appending the create verb to the method for readability
  MaxGekk
  committed
  edb9167
- Making the createParser as a separate private method
  MaxGekk
  committed
  5ba2881
- Fix code style
  MaxGekk
  committed
  1509e10
- Checks the charset option is supported
  MaxGekk
  committed
  e3184b3
- Support charset as a parameter of the json method
  MaxGekk
  committed
  87d259c
- Test for charset different from utf-8
  MaxGekk
  committed
  76c1d08
- Description of the charset option of the json method
  MaxGekk
  committed
  88395b5
- Minor changes in comments: added . at the end of a sentence
  MaxGekk
  committed
  f2f8ae7
- Added a test for wrong charset name
  MaxGekk
  committed
  b451a03
- Testing that charset in any case is acceptable
  MaxGekk
  committed
  c13c159
- Test: user specified wrong (but supported) charset
  MaxGekk
  committed
  1cb3ac0
- Set charset as an option
  MaxGekk
  committed
  108e8e7
- Test: saving to json in UTF-32BE
  MaxGekk
  committed
  0d20cc6
- Taking user's charset for saved json
  MaxGekk
  committed
  54baf9f
- Test: output charset is UTF-8 by default
  MaxGekk
  committed
  1d50d94
- Changing the readJsonFiles method for readability
  MaxGekk
  committed
  bb53798
- The test checks that json written by Spark can be read back
  MaxGekk
  committed
  961b482

Commits

Commits on Mar 17, 2018