Myrient DAT Downloader Optimized (dltool_optimized)

🔱 Fork Chain:

• Original: dltool by kosmosnautti
• First Fork: dltool_optimized by f3rs3n
• This Fork: wget integration + performance enhancements

A Python script to automatically download ROMs/files from Myrient based on DAT files (such as those generated by No-Intro, Redump, etc.).

Why This Fork?

The previous versions use Python's requests library for downloads, which is not officially supported by Myrient and results in automatic speed throttling to 10 KB/s. This fork replaces the download engine with wget (officially supported), enabling full-speed downloads while building upon the improvements from f3rs3n's optimized version.

Quick Comparison

Feature	Original (kosmosnautti)	f3rs3n's Fork	This Fork
Download Speed	⚠️ Limited to 10 KB/s	⚠️ Limited to 10 KB/s	✅ Full speed
Download Tool	Python requests	Python requests	wget (supported)
Resume Downloads	Manual/unreliable	Improved but limited	✅ Automatic (wget -c)
Auto Retry on Error	❌ No	❌ No	✅ Yes (3 attempts)
Game Name Periods Bug	❌ Broken	✅ Fixed	✅ Fixed
Output Path Optional	❌ No (-o required)	✅ Yes	✅ Yes
Progress Bar	Basic	progressbar2	wget native
Python Dependencies	Unknown	3 packages	2 packages
Code Complexity	~620 lines	~620 lines	~525 lines

Main Features

• DAT File Analysis: Reads XML-formatted DAT files to extract the list of desired games/sets.
• Interaction with Myrient:
  • Navigates Myrient's directories to identify catalogs (e.g., No-Intro, Redump).
  • Searches for and selects the specific system collection (e.g., "Nintendo - Game Boy Advance", "Commodore - Amiga").
• Automatic and Manual Selection:
  • Attempts to automatically identify the catalog and collection based on information in the DAT file header and Myrient's structure.
  • Provides command-line options (-c, -s) to force manual selection if automatic selection fails or is not desired.
• Advanced Download Logic:
  • Downloads missing files to the specified output folder using wget (officially supported by Myrient, avoiding speed throttling).
  • Existing File Management:
    • --skip-existing option: If a file with the same name already exists in the destination folder, the download is skipped.
    • Default behavior (without --skip-existing): wget automatically handles resume/continue for partially downloaded files with its -c flag.
• Output Folder Creation:
  • The output folder parameter (-o) is optional.
  • If omitted, the script automatically creates a subfolder named after the Myrient collection (sanitized) in the same directory as the script.
• User Interface:
  • Colored and timestamped output for better readability and operation tracking.
  • Detailed progress bar provided by wget for each download, showing percentage, transferred data, speed, and estimated time remaining (ETA).
  • "List only" mode (-l): Allows checking which files are missing without downloading anything.
• Robustness:
  • Management of network errors (timeouts, HTTP errors) and parsing errors.
  • Automatic retry on download failures (wget retries up to 3 times).
  • Controlled exit and clear messages in case of user interruption (Ctrl+C).

Prerequisites

• Python 3.x
• wget (command-line download utility - must be installed on your system)
• The following Python libraries (installable via pip):
  • requests (for HTTP requests to navigate Myrient directories)
  • beautifulsoup4 (for parsing HTML of Myrient pages)

Dependency Installation

Python Libraries

You can install the necessary Python libraries using pip (it is recommended to do this within a Python virtual environment):

pip install requests beautifulsoup4

wget Installation

This script requires wget to be installed on your system for downloading files (wget is officially supported by Myrient, avoiding download speed throttling).

Linux (Debian/Ubuntu):

sudo apt-get install wget

Linux (Fedora/RHEL/CentOS):

sudo dnf install wget

macOS:

brew install wget

Windows: Download from GNU Wget for Windows or use WSL (Windows Subsystem for Linux)

Usage

Run the script from a terminal or command prompt:

python dltool_optimized.py -i <file.dat> [OPTIONAL_PARAMETERS]

Command-Line Parameters

• Required Parameters:

  • -i <file.dat>: Specifies the path to your input DAT file (e.g., Nintendo - Nintendo Entertainment System.dat).

• Optional Parameters:

  • -o <output_folder>: Specifies the folder where downloaded files will be saved.
    • Default Behavior if Omitted: The script will automatically create a subfolder named after the system/collection (e.g., ./Nintendo - Nintendo Entertainment System/) in the same directory where the script is located.
  • -c: Force manual catalog choice. Prompts the user to manually choose the catalog (e.g., "No-Intro", "Redump") from the presented list, even if the script believes it has identified it automatically from the DAT file.
  • -s: Force manual system/collection choice. Prompts the user to manually choose the specific collection (e.g., "Nintendo - Game Boy Advance") from the presented list, even if the script believes it has identified it automatically.
  • -l: List missing only (No download). Performs a scan and comparison of files in the DAT against those on the Myrient server, but instead of downloading files, it only lists which ROMs/files from the DAT were not found (or do not match) in the selected Myrient collection.
  • --skip-existing: Skip existing files. If this option is specified, the script will not attempt to re-download a file if it already exists in the destination folder, regardless of its size. If not specified, the script will compare sizes to decide whether to resume, skip, or re-download (see "Advanced Download Logic").
  • -h or --help: Show Help. Displays a detailed help message with a description of all parameters and their usage.

Usage Examples

1. Basic download, specifying the output folder (compares sizes for existing files):

   python dltool_optimized.py -i "Sony - PlayStation.dat" -o "/media/roms/PlayStation"

2. Download with automatic output folder creation, skipping any already existing files in the destination:

   python dltool_optimized.py -i "Commodore - Amiga.dat" --skip-existing

   (If the chosen collection is "Commodore - Amiga", files will be downloaded to a subfolder named ./Commodore - Amiga/. If a file with the same name is already present in that folder, it will be skipped without checking the size).

3. Only check for missing files for a given DAT, without downloading anything:

   python dltool_optimized.py -i "Sega - Mega Drive - Genesis.dat" -l

4. Force manual selection of the system/collection:

   python dltool_optimized.py -i "My Custom System.dat" -s

Detailed Improvements & Changelog

🆕 What's New in This Fork (wget Integration)

Main contribution of this fork:

• 🚀 wget Integration (Performance & Compatibility): Replaced Python requests library for downloads with wget, which is officially supported by Myrient. This eliminates the 10 KB/s speed throttling that affects unsupported download methods, resulting in significantly faster downloads. wget also provides native resume capability and automatic retry on failures.

• 🔄 Automatic Resume & Retry:

  • Leverages wget's -c flag for automatic resume of partial downloads
  • Automatic retry on download failures (wget retries up to 3 times with --tries=3)
  • Comprehensive wget error code handling with descriptive messages

• 📦 Simplified Codebase:

  • Removed complex manual download logic (~95 lines removed)
  • Removed progressbar2 Python dependency
  • Delegated download management to wget (more reliable and battle-tested)

• 📊 Native Progress Display:

  • Uses wget's built-in progress bar (more reliable than Python implementation)
  • Shows percentage, speed, and ETA natively

🔧 Inherited from f3rs3n's dltool_optimized

These improvements were already present in the f3rs3n fork that this builds upon:

• Game Name Handling (Bug Fix): Corrected a critical issue where game names from the DAT file containing periods (e.g., in version numbers like "v1.02" or abbreviations like "S.T.U.N.") were erroneously truncated.

• Optional Output Path (-o): The output directory is optional. If not specified, a directory named after the selected Myrient collection is automatically created in the script's location.

• --skip-existing Flag: Straightforward way to avoid re-downloading any file that already exists locally.

• Enhanced User Interface:

  • Timestamped and colored log messages for better clarity
  • Log line rewriting for cleaner status updates

• Code Structure Improvements:

  • Refactored helper functions (logger, inputter, scale1024, sanitize_filename)
  • Directory name sanitization for cross-platform compatibility
  • Robust path handling with os.path.join, os.path.abspath, os.path.normpath

Known Limitations

• Dependency on Myrient's HTML Structure: The script relies on the current HTML structure of the Myrient website to find links to files and directories. Significant changes to the site's layout could break the script's functionality until it is updated to reflect those changes.
• Reliability of Automatic Selection: Automatic determination of the catalog and system/collection is based on heuristics (string matching in names and URLs). It might not work correctly for DAT files with unusual formatting or names, or if the names on Myrient differ significantly. In such cases, using the manual selection options (-c and -s) is recommended.

---

Contributing & Issues

Found a bug or have a feature request? Please open an issue on this repository.

Pull requests are welcome! If you're planning major changes, please open an issue first to discuss.

Testing Checklist

Before submitting a PR, please ensure:

• wget is installed and the script runs without errors
• Downloads work at full speed (not throttled to 10 KB/s)
• Resume functionality works correctly
• Code follows the existing style (helper functions, logging, etc.)

---

Credits & License

This project has evolved through multiple contributors:

🔗 Fork Lineage:

1. Original: dltool by kosmosnautti - Initial implementation
2. First Fork: dltool_optimized by f3rs3n - Bug fixes and usability improvements
3. This Fork: wget integration for full-speed downloads and code simplification

Special Thanks:

• Myrient Team: https://myrient.erista.me/ - ROM preservation project and file hosting
• All previous contributors to the dltool ecosystem

License: This project is provided as-is for personal use. Please respect Myrient's terms of service and bandwidth limitations. When using this tool, be considerate of Myrient's resources.