Problem

1. Only the GCS fetch script benefits from CSV file initialization. The arXiv script shouldn't have copied this pattern. It should be removed:

• quantifying/scripts/1-fetch/arxiv_fetch.py

  Lines 128 to 135 in 34c1caa

  	def initialize_data_file(file_path, headers):
  	"""Initialize CSV file with headers if it doesn't exist."""
  	if not os.path.isfile(file_path):
  	with open(file_path, "w", encoding="utf-8", newline="\n") as file_obj:
  	writer = csv.DictWriter(
  	file_obj, fieldnames=headers, dialect="unix"
  	)
  	writer.writeheader()

2. The various fetch scripts duplicate a lot of code between them when they save their data:

   • quantifying/scripts/1-fetch/arxiv_fetch.py

     Lines 475 to 485 in 34c1caa

     	def rows_to_csv(args, fieldnames, rows, file_path):
     	if not args.enable_save:
     	return args
     	with open(file_path, "w", encoding="utf-8", newline="\n") as file_handle:
     	writer = csv.DictWriter(
     	file_handle, fieldnames=fieldnames, dialect="unix"
     	)
     	writer.writeheader()
     	for row in rows:
     	writer.writerow(row)

   • quantifying/scripts/1-fetch/gcs_fetch.py

     Lines 181 to 185 in 34c1caa

     	with open(file_path, "a", encoding="utf-8", newline="\n") as file_obj:
     	writer = csv.DictWriter(
     	file_obj, fieldnames=fieldnames, dialect="unix"
     	)
     	writer.writerow(row)

   • quantifying/scripts/1-fetch/github_fetch.py

     Lines 92 to 98 in 34c1caa

     	with open(FILE1_COUNT, "w", encoding="utf-8", newline="\n") as file_obj:
     	writer = csv.DictWriter(
     	file_obj, fieldnames=HEADER1_COUNT, dialect="unix"
     	)
     	writer.writeheader()
     	for row in tool_data:
     	writer.writerow(row)

   • quantifying/scripts/1-fetch/openverse_fetch.py

     Lines 195 to 207 in 34c1caa

     	def write_data(args, data):
     	if not args.enable_save:
     	return
     	os.makedirs(PATHS["data_phase"], exist_ok=True)
     	with open(FILE_PATH, "w", encoding="utf-8", newline="") as file_obj:
     	writer = csv.DictWriter(
     	file_obj,
     	fieldnames=OPENVERSE_FIELDS,
     	dialect="unix",
     	)
     	writer.writeheader()
     	for row in data:
     	writer.writerow(row)

   • quantifying/scripts/1-fetch/smithsonian_fetch.py

     Lines 98 to 121 in 34c1caa

     	def write_data(args, data_metrics, data_units):
     	if not args.enable_save:
     	return args
     	# Create data directory for this phase
     	os.makedirs(PATHS["data_phase"], exist_ok=True)
     	with open(FILE_1_METRICS, "w", encoding="utf-8", newline="\n") as file_obj:
     	writer = csv.DictWriter(
     	file_obj, fieldnames=HEADER_1_METRICS, dialect="unix"
     	)
     	writer.writeheader()
     	for row in data_metrics:
     	writer.writerow(row)
     	with open(FILE_2_UNITS, "w", encoding="utf-8", newline="\n") as file_obj:
     	writer = csv.DictWriter(
     	file_obj, fieldnames=HEADER_2_UNITS, dialect="unix"
     	)
     	writer.writeheader()
     	for row in data_units:
     	writer.writerow(row)
     	return args

   • quantifying/scripts/1-fetch/wikipedia_fetch.py

     Lines 78 to 91 in 34c1caa

     	def write_data(args, tool_data):
     	if not args.enable_save:
     	return args
     	LOGGER.info("Saving fetched data")
     	os.makedirs(PATHS["data_phase"], exist_ok=True)
     	with open(FILE_LANGUAGES, "w", encoding="utf-8", newline="\n") as file_obj:
     	writer = csv.DictWriter(
     	file_obj, fieldnames=HEADER_LANGUAGES, dialect="unix"
     	)
     	writer.writeheader()
     	for row in tool_data:
     	writer.writerow(row)
     	return args

Description

1. Add rows_to_csv() function to shared library (shared.py)
   • New function should check args.enable_save
   • New function should "Create data directory for this phase"
   • New function shoudn't return args
     • None of the curernt functions that return args modify args--there's no reason to return it
   • GCS fetch script only rights a single row, but it can send a list with a single row
   • Update fetch scripts to use new function
   • Test fetch scripts to verify they behave as intended
2. Rename data_to_csv() function to dataframe_to_csv()
   • Update process scripts to use new name

Additional context

• Abstraction principle (computer programming) - Wikipedia