Allow uploading and storing cracker binaries on the server by s3inlc · Pull Request #2499 · hashtopolis/server

s3inlc · 2026-09-17T09:51:39Z

Cracker binaries can now be stored on the server itself instead of only referencing an external download URL. Existing URL-based binaries are fully unaffected.

Upload (apiv2)

POST /api/v2/ui/crackers accepts an alternative to downloadUrl: sourceType/sourceData with three sources:
- import: consume an archive uploaded through the existing chunked (TUS) upload to the import directory — the path for arbitrarily large archives, no PHP request-size limits involved
- url: the server fetches the archive itself (http/https only)
- inline: base64 archive data for small uploads
Only .7z archives are accepted (7z magic-byte validation); the archive filename is server-composed ({crackerType}-{version}.7z), and the downloadUrl is set automatically to the server's download endpoint — no client-supplied filenames or URLs
New nullable filename column on CrackerBinary doubles as the local-storage flag (NULL = external URL binary, as before)

Download

New generic endpoint GET /api/download.php/{kind}/{id} (kind registry, extensible to files/hashlists later) — authenticated with either an agent token (?token=) or an apiv2 Bearer JWT; streams with HTTP Range and ETag support
Agents require zero changes: the downloadBinary action appends the requesting agent's token to the URL of locally hosted binaries, so the existing fetch-and-extract flow just works
Deleting a binary (or a whole cracker type) also removes its stored archive; the downloadUrl of local binaries is server-owned and cannot be patched

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate issues affect storage permissions, URL handling, validation, authentication, and download correctness.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 5 · 2

Open (7)

Include crackers directory in Docker startup ownership handling · New
Do not persist download URLs from untrusted Host headers · New
Keep legacy URL creation independent of server-side caching · New
Limit decoded inline upload size before allocation · New
Prevent SSRF through private destinations and redirects · New
Reject null downloadUrl updates for external binaries · New
Correctly parse suffix and open-ended HTTP ranges · New

What changed in this PR

Adds server-hosted cracker binary uploads and an authenticated download endpoint while preserving agent integration.

Changes:

Supports inline, imported, and URL-based archive uploads with local storage.
Adds authenticated streaming downloads with range and ETag support.
Updates models, migrations, OpenAPI documentation, Docker setup, and tests.

File	Summary
`src/migrations/postgres/20260917114300_cracker-binary-local-upload.sql`	Adds local filename storage.
`src/migrations/mysql/20260917114300_cracker-binary-local-upload.sql`	Adds filename storage and expands URL capacity.
`src/inc/utils/DownloadUtils.php`	Provides download streaming and range handling. Moderate (3 votes): suffix and open-ended ranges are parsed incorrectly. Moderate (1 vote): 416 responses use an incorrect `Content-Range`.
`src/inc/utils/CrackerUtils.php`	Handles archive upload, caching, lifecycle, and cleanup. Critical (3 votes): URL-only creation now requires server-side fetching, breaking existing external URLs. Moderate (1 vote): URL edits have the same compatibility break. Critical (3 votes): scheme-only URL validation permits SSRF. Critical (1 vote): inline base64 input is unbounded before size checks.
`src/inc/Util.php`	Adds URL downloading and backend URL generation. Critical (1 vote): request `Host` can be persisted into agent download URLs.
`src/inc/StartupConfig.php`	Configures cracker storage.
`src/inc/startup/setup.php`	Initializes the cracker directory.
`src/inc/startup/setup.json`	Updates initial cracker data.
`src/inc/downloadapi/DownloadRegistry.php`	Registers download handlers.
`src/inc/downloadapi/DownloadAuthMiddleware.php`	Adds agent and JWT authentication.
`src/inc/downloadapi/DownloadApp.php`	Defines download routes. Moderate (1 vote): IDs accept nonnumeric prefixes. Moderate (1 vote): bearer authentication failures can become HTTP 500 responses.
`src/inc/downloadapi/CrackerBinaryDownloadHandler.php`	Serves local cracker archives.
`src/inc/defines/DDirectories.php`	Adds the cracker directory key.
`src/inc/apiv2/openapi/SpecOverrides.php`	Documents cracker upload fields.
`src/inc/apiv2/openapi/ModelApiPathBuilder.php`	Applies request descriptions.
`src/inc/apiv2/model/CrackerBinaryAPI.php`	Implements upload creation and update rules. Moderate (2 votes): updates allow `downloadUrl: null` despite the database constraint.
`src/inc/apiv2/common/AbstractHelperAPI.php`	Reuses download utilities.
`src/inc/apiv2/auth/JwtAuthenticationFactory.php`	Shares JWT configuration.
`src/inc/agentapi/model/DownloadBinaryAction.php`	Adds agent-token URLs for local archives.
`src/dba/models/generator.php`	Updates model metadata.
`src/dba/models/CrackerBinaryFactory.php`	Maps the filename field.
`src/dba/models/CrackerBinary.php`	Adds local archive metadata. Moderate (1 vote): nullable API validation conflicts with the non-null database column.
`src/api/v2/index.php`	Uses shared JWT authentication.
`src/api/download.php`	Adds the download API entry point.
`openapi.json`	Updates the generated API specification. Moderate (1 vote): the schema lacks conditional requirements for upload sources.
`Dockerfile`	Creates and configures cracker storage. Critical (2 votes): existing Docker volumes may leave the directory owned by root and unwritable by `www-data`.
`ci/phpunit/TestBase.php`	Updates test model construction.
`ci/phpunit/inc/UtilTest.php`	Tests backend URL generation.
`ci/phpunit/inc/utils/CrackerUtilsTest.php`	Tests archive lifecycle behavior.
`ci/phpunit/inc/utils/CrackerBinaryUtilsTest.php`	Updates cracker fixtures.
`ci/phpunit/inc/apiv2/openapi/SpecOverridesTest.php`	Tests schema descriptions.
`ci/phpunit/fixtures/openapi/crackerbinarytype.spec.json`	Updates OpenAPI fixtures.
`ci/phpunit/downloadapi/DownloadAppTest.php`	Tests download authentication and streaming.
`ci/phpunit/dba/MassUpdateSetTest.php`	Updates model fixtures.
`ci/phpunit/dba/AbstractModelFactoryTest.php`	Updates model fixtures.
`ci/apiv2/utils.py`	Adds upload and archive test helpers.
`ci/apiv2/test_taskwrapper.py`	Uses valid downloadable archives.
`ci/apiv2/test_cracker.py`	Tests upload, caching, and deletion flows.
`ci/apiv2/test_agent_protocol.py`	Tests agent downloads and token handling.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

 ENV HASHTOPOLIS_LOG_PATH=${HASHTOPOLIS_PATH}/log
 ENV HASHTOPOLIS_CONFIG_PATH=${HASHTOPOLIS_PATH}/config
 ENV HASHTOPOLIS_BINARIES_PATH=${HASHTOPOLIS_PATH}/binaries
+ENV HASHTOPOLIS_CRACKERS_PATH=${HASHTOPOLIS_PATH}/crackers


+    }
+    return rtrim(Util::buildServerUrl() . SConfig::getInstance()->getVal(DConfig::BASE_URL), '/');


+    try {
+      CrackerUtils::storeLocalCopy($binary);
+    }
+    catch (HttpError $e) {
+      Factory::getCrackerBinaryFactory()->delete($binary);


+      case "inline":
+        $archiveData = base64_decode($sourceData, true);
+        if ($archiveData === false) {
+          throw new HttpError("sourceData not valid base64 encoding");
+        }


+      case "url":
+        $scheme = parse_url($sourceData, PHP_URL_SCHEME);
+        if ($scheme != "http" && $scheme != "https") {
+          throw new HttpError("Only http and https URLs are supported as sourceData!");
+        }


+    $refreshLocalCopy = $binary->getFilename() === null
+      && ($data[CrackerBinary::DOWNLOAD_URL] ?? null) !== null
+      && $data[CrackerBinary::DOWNLOAD_URL] != $binary->getDownloadUrl();


+      $range = explode('-', $range);
+      $c_start = (int)$range[0];
+      if ((isset($range[1]) && is_numeric($range[1]))) {
+        $c_end = (int)$range[1];
+      }
+      else {
+        $c_end = $size;
+      }
+    }


s3inlc added this pull request to stack #2501 September 17, 2026 12:36

s3inlc force-pushed the 886-stack-1-upload branch from 44630aa to 7d0dd1b Compare September 18, 2026 12:59

s3inlc added 2 commits September 22, 2026 14:17

adding local file and upload of cracker binaries

5a94364

updated openapi.json

910b4b7

s3inlc force-pushed the 886-stack-1-upload branch from 7d0dd1b to 910b4b7 Compare September 22, 2026 12:17

s3inlc requested a review from jessevz September 22, 2026 12:17

s3inlc marked this pull request as ready for review September 22, 2026 12:17

added crackers binary path to dockerfile env

e67eb08

s3inlc requested a lite review from Copilot September 22, 2026 14:28

Copilot started reviewing on behalf of s3inlc September 22, 2026 14:29 View session

Copilot AI reviewed Sep 22, 2026

View reviewed changes

Navigation Menu

Allow uploading and storing cracker binaries on the server - #2499

Allow uploading and storing cracker binaries on the server#2499
s3inlc wants to merge 3 commits into
masterfrom
886-stack-1-upload

s3inlc commented Sep 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		}
		return rtrim(Util::buildServerUrl() . SConfig::getInstance()->getVal(DConfig::BASE_URL), '/');

Conversation

s3inlc commented Sep 17, 2026

Upload (apiv2)

Download

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants