Skip to content

feat: seatplus:sde-import command + weekly SdeImportJob schedule#677

Merged
herpaderpaldent merged 4 commits into
4.xfrom
feat/sde-import-scheduled
May 26, 2026
Merged

feat: seatplus:sde-import command + weekly SdeImportJob schedule#677
herpaderpaldent merged 4 commits into
4.xfrom
feat/sde-import-scheduled

Conversation

@herpaderpaldent
Copy link
Copy Markdown
Contributor

New: seatplus:sde-import artisan command

Downloads and imports EVE Online static data from the official CCP JSONL SDE.

Usage

# Download and import from CCP servers
php artisan seatplus:sde-import

# Use a pre-downloaded local zip (skips HTTP download)
php artisan seatplus:sde-import --source=/path/to/sde.zip

What gets imported

Respects FK dependency order:

  1. Categories → universe_categories
  2. Groups → universe_groups (requires categories)
  3. Types → universe_types (requires groups)
  4. Regions → universe_regions
  5. Constellations → universe_constellations (requires regions)
  6. Systems → universe_systems (requires constellations)

All imports use chunked upsert() (500 rows per batch). Re-running is safe (idempotent).

New: SdeImportJob (scheduled weekly)

  • ShouldBeUnique queued job with a 600s timeout
  • Delegates to Artisan::call('seatplus:sde-import')
  • Seeded in the schedules table with 0 0 * * 0 (every Sunday at midnight)

- Add SdeImportCommand: downloads CCP JSONL SDE zip and upserts
  categories, groups, types, regions, constellations and systems in
  FK-dependency order (500-row chunks, idempotent).
  --source option skips the HTTP download for local-file imports.

- Add SdeImportJob: thin ShouldBeUnique queued job that delegates to
  Artisan::call('seatplus:sde-import'). timeout=600s.

- Seed Schedules with '0 0 * * 0' (every Sunday midnight) so the
  import runs automatically once a week without manual intervention.

- Register SdeImportCommand in EveapiServiceProvider.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@what-the-diff
Copy link
Copy Markdown

what-the-diff Bot commented May 26, 2026

PR Summary

  • Establishing New Scheduling Feature in Database
    Ye will be finding a fresh migration file in our trove. This be added to forge an SDE import schedule within our database, helping us sync and track EVE Online Static Data.

  • Implementing Command to Import Static Data
    We've unleashed a new command on the seven seas, known as the 'SdeImportCommand'. This scurvy dog allows us to import EVE Online Static Data from .zip files like a sturdy and dependable ship.

  • Creation of a Job Class to Handle Import Process
    A new job class, named 'SdeImportJob', has been summoned from the depths. This class wields the power of queuing jobs to handle the SDE import process, making the operation run as smooth as calm waters.

  • Registration of the New Command through EveapiServiceProvider
    The EveapiServiceProvider has been updated, on par with the latest charts. It now registers our new import command, 'SdeImportCommand', ensuring it's ready for the next voyage.

  • Unit Tests for Ensuring Correct Functionality
    The cannons of Unit Tests have been fired for 'SdeImportCommand' and 'SdeImportJob'. This will make sure that our new systems are shipshape and Bristol fashion, sailing through the code seas without hitting any unexpected rocks.

herpaderpaldent and others added 3 commits May 26, 2026 16:01
Coverage fixes in SdeImportCommand:
- Mark fopen(false) and scandir(false) guards with @codeCoverageIgnore
  (OS-level edge cases, not reasonably testable)
- Add test: corrupt ZIP triggers fail() exit code 1
- Add test: empty + non-JSON lines in JSONL are silently skipped
- Add test: 501 records triggers mid-stream chunk flush (CHUNK_SIZE=500)
- Add test: ZIP with subdirectory exercises removeDirectory() recursion

Auto-dispatch on first install:
- New migration dispatches SdeImportJob when universe_categories is empty
- Idempotent: re-running migrate on existing install is a no-op
- Safe in tests: LazilyRefreshDatabase runs migrations after setUp()
  so Queue::fake() is already active when the migration fires

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Consolidate two migrations into one: the schedule-seeding migration now
also dispatches SdeImportJob immediately after seeding the weekly cron.
Removes the separate first-install migration and its empty-table guard —
migrations only run once, so the unconditional dispatch is correct.

Update tests: replace two conditional-dispatch tests with a single test
verifying the schedule migration dispatches SdeImportJob.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
FORK_MEM_PER_PROC=107374182400 (100 GB) makes pokio calculate
maxByMemory=0, resulting in max(1, min(cpuBased, 0))=1 process.
This eliminates the race condition where parallel workers corrupt
.temp/v3.php, causing ParseError on the Formats/ci (push) workflow.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@herpaderpaldent herpaderpaldent merged commit 08de2ed into 4.x May 26, 2026
6 checks passed
@herpaderpaldent herpaderpaldent deleted the feat/sde-import-scheduled branch May 26, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant