The index of TarFileSystem is stored as space separated values in parent file system for later reuse. When we try to read the index back it fails if file name contains spaces.
17/04/22 00:20:32 INFO tar.TarFileSystem: *** Using Tar file system ***
17/04/22 00:20:33 ERROR tar.TarIndex: Invalid Index File: /jd/systems.tar.index
We should either use properly quoted CSV file. Or use some structured format like JSON.
If we support hierarchical index in future to support directories, structured format like JSON seems to be a better choice.
The index of TarFileSystem is stored as space separated values in parent file system for later reuse. When we try to read the index back it fails if file name contains spaces.
We should either use properly quoted CSV file. Or use some structured format like JSON.
If we support hierarchical index in future to support directories, structured format like JSON seems to be a better choice.