Sync emeraldhost/wings with Rene-Roscher fork features#1
Merged
Conversation
- Updated import paths across multiple files to reflect the new module name `github.com/Rene-Roscher/wings`. - This change affects files in the environment, events, internal, router, server, and sftp packages. - Ensured consistency in the import statements for better maintainability and clarity.
… and immediate updates
…ved logging and estimation
…nce progress tracking
…up and restore operations
… for accurate progress tracking
… and callback handling
…progress estimation
- Added BackupOperation and BackupOperationRegistry to manage backup and restore operations. - Implemented methods for registering, cancelling, retrieving, listing, and completing operations. - Introduced cleanup mechanism for stale operations running longer than a specified duration. - Added global instance and a background goroutine for periodic cleanup of stale operations.
- Introduced a new `Format` field in the configuration to specify the compression format for backups, supporting "gzip" (default), "zstd", and "none". - Refactored the archive creation logic to accommodate the new compression format, including methods for creating ZSTD and GZIP writers. - Implemented automatic format detection during backup restoration, allowing seamless handling of both GZIP and ZSTD compressed backups. - Updated the `Path()` method to generate appropriate file extensions based on the selected compression format. - Enhanced S3 backup handling to prevent self-deletion on failure and improved context management. - Added comprehensive unit tests for compression format detection and decompression functionality. - Documented the changes and provided a migration strategy to ensure backward compatibility with existing GZIP backups.
…performance feat(backup): enhance backup state management and context awareness feat(s3): improve S3 backup handling with context checks and orphaned part cleanup feat(archive): add security measures for compression format detection and decompression
- Added SftpSecurityConfiguration to manage security settings. - Introduced SmartSecurityProtector for tracking login attempts and blocking IPs based on thresholds. - Enhanced SFTP server to utilize smart protection during authentication. - Updated backup and restore processes to support context-aware operations. - Improved logging for security events and actions. - Added comprehensive configuration guide for optimized backup settings.
…g and error recovery
…etection and directory preservation
…ring backup restoration
… integrity validation
…nhance restore path handling
…n and update S3 upload handling
…ogress is always sent
…and enhancing debug logging for initial and final progress events
- Introduced a detailed WORK.md outlining backup requirements and integrity guarantees. - Developed a compression package with support for GZIP, ZSTD, TAR, and uncompressed formats, including a registry for extensibility. - Created extensive unit tests for backup cleanup, progress tracking, retry logic, and queue management to ensure robustness and reliability. - Implemented logic for handling backup retries with exponential backoff and context management. - Ensured backward compatibility and minimal invasiveness in changes to the existing backup system. - Validated progress tracking for S3 uploads with an 80/20 split between archiving and uploading phases.
…curacy during archive and upload phases
…rogress tracking and real-time event management
…ad size from context for S3 backups
…atibility and performance
…nhanced detection
…ved security and consistency feat(archive): update ZSTD writer and reader to use default settings for maximum compatibility feat(tests): remove integrity test file as it is no longer needed
…nt binary file corruption
…with compression options
… prevent binary corruption
…revent corruption
…ted compression formats refactor(archive): eliminate ZSTD support and related functions to prevent binary corruption refactor(compress): remove system tar extraction logic to streamline decompression process delete(tests): remove ZSTD compression tests as the feature is no longer supported
…phase to prevent written bytes exceeding total
…e progress tracking
…wnload size to prevent path traversal and DoS attacks
…ling for TAR archives
Pulls in upstream commit ac81409 ("clearer error messaging on downloads, don't log as a 500-level error"). Brings our update branch fully in sync with pterodactyl/wings:develop (0 commits behind). Conflict resolution: - router/downloader/downloader.go: kept our redirect-loop refactor; IsDownloadError() and the 30s dialer / 2h client timeouts from upstream applied cleanly outside the conflict zone. - router/router_server_files.go: adopted upstream's IsDownloadError-gated error logging and the 400+JSON response for download errors; preserved our SaveActivity() call so file-download activity logging remains intact.
The release workflow built `github.com/${{ github.repository_owner }}/wings`,
which on the emeraldhost/wings fork mirror resolves to
`github.com/emeraldhost/wings` — but go.mod declares
`github.com/Rene-Roscher/wings` as the module path. Building against a
non-matching import path fails. Hardcode to the actual module path so
release builds succeed on any mirror.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| // If not found as Local backup, check if it's an S3 backup file that exists locally | ||
| // S3 backups may leave local files behind after failed uploads or for debugging | ||
| s3Backup := backup.NewS3(client, backupID, "") | ||
| if _, err := os.Stat(s3Backup.Path()); err == nil { |
| } | ||
|
|
||
| // Basic file existence and size check | ||
| stat, err := os.Stat(backupPath) |
| } | ||
|
|
||
| // Quick magic bytes check for GZIP | ||
| f, err := os.Open(backupPath) |
| serverChecksum := "unknown" | ||
|
|
||
| // Get backup file SHA1 (reuse existing checksum method) | ||
| if backupFile, err := os.Open(backupPath); err == nil { |
| func (s *Server) countBackupEntries(backupPath string) (*fileStats, error) { | ||
| stats := &fileStats{} | ||
|
|
||
| f, err := os.Open(backupPath) |
| if os.IsNotExist(err) { | ||
| logger.Debug("backup directory does not exist, nothing to clean up") | ||
| return nil | ||
| } |
| failedRemovals = append(failedRemovals, fileName) | ||
| } else { | ||
| logger.WithField("file", fileName).Info("removed backup file") | ||
| removedFiles = append(removedFiles, fileName) |
| }() | ||
|
|
||
| // Check if backup archive already exists (S3 two-phase backup) | ||
| if _, err := os.Stat(s.Path()); os.IsNotExist(err) { |
| }) | ||
|
|
||
| // List all files in backup directory | ||
| files, err := os.ReadDir(backupDir) |
|
|
||
| logger.WithField("file", fileName).Debug("found backup file, attempting removal") | ||
|
|
||
| if err := os.Remove(filePath); err != nil { |
13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bringt den
update-Branch (69 Commits ahead vonpterodactyl:develop, 0 behind nach dem heutigen upstream-Merge) inemeraldhost/wings:develop.Inhalt:
backupundrestoremit automatischer Recoverypterodactyl/wings:develop@ac81409(clearer download error messaging)release.yamlBuild-Pfad hardcoded aufgithub.com/Rene-Roscher/wings(Module-Pfad)Test plan
PushWorkflow: build + test + race auf Go 1.24.11 und 1.25.5, beide GOARCHsCodeQLWorkflow: keine neuen Security-Findingsghcr.io/emeraldhost/wings:developgepushtwings --debug --config config.ymlstartet sauber