[US_Urban_Schools_Teachers_And_Staff] Modified run.sh script to downlaod complete data - #2163
[US_Urban_Schools_Teachers_And_Staff] Modified run.sh script to downlaod complete data#2163Krishnam24maheshwari wants to merge 7 commits into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the run.sh script to use comma-separated file lists instead of space-separated ones for the --input_data argument. It also quotes the variables referencing these lists to prevent word splitting issues. There are no review comments, so I have no additional feedback to provide.
Krishnam24maheshwari
changed the title
Krishnam24maheshwari
left a comment
Krishnam24maheshwari
left a comment
There was a problem hiding this comment.
Made all changes
2 participants
In this PR import has intermittent failures caused due to non-deterministic file discovery and improper handling of CLI flags in run.sh.
Issues :
1. Identified unquoted variables and lack of comma-delimiting caused only partial data processing.
2. Issue caused by "find" keyword that would output 2018_Teachers.csv, 2021_Teachers.csv, or 2022_Teachers.csv first due to which there is irregular deletions happening during different runs.
The issue was resolved by forcing deterministic file sorting, implementing comma-separated lists, and quoting input arguments -
1.Added sort and paste -sd : Guarantees deterministic alphabetical order and formats files as a proper comma-separated string (.../2018.csv,.../2021.csv,.../2022.csv).
2. Quoted all
--input_dataarguments: Ensured Bash does not perform word-splitting.3. Comma-delimited legacy XLSX files : Ensured 2010, 2012, and 2014 files are all passed together to --input_data.