diff --git a/.github/workflows/gosdk.yml b/.github/workflows/gosdk.yml index 45403195..d576178b 100644 --- a/.github/workflows/gosdk.yml +++ b/.github/workflows/gosdk.yml @@ -43,5 +43,5 @@ jobs: 0Chain GoSDK `${{ github.event.inputs.gosdk }}` is released. see full changelog on https://github.com/0chain/gosdk/releases/tag/${{ github.event.inputs.gosdk }} draft: false - reviewers: cnlangzi + reviewers: guruhubb labels: GoSDK diff --git a/README.md b/README.md index 45a3cde4..0d9a238c 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,7 @@ zbox is a command line interface (CLI) tool to understand the capabilities of Z - [Create new allocation](#create-new-allocation) - [Free storage allocation](#free-storage-allocation) - [Update allocation](#update-allocation) + - [Transfer allocation ownership](#transfer-allocation-ownership) - [Forbid Allocation](#forbid-allocation) - [Cancel allocation](#cancel-allocation) - [Finalise allocation](#finalise-allocation) @@ -25,14 +26,16 @@ zbox is a command line interface (CLI) tool to understand the capabilities of Z - [List all files](#list-all-files) - [List owner's allocations](#list-owners-allocations) - [Update blobber settings](#update-blobber-settings) - - [Update Validator Settings](#update-validator-settings) + - [Update Validator Settings](#update-validator-settings) - [List All Validators](#list-all-validators) - [Get Validator Configuration](#get-validator-configuration) + - [Shutdown Blobber](#shutdown-blobber) + - [Shutdown Validator](#shutdown-validator) - [Kill Blobber](#kill-blobber) - [Kill Validator](#kill-validator) - [Uploading and Managing files](#uploading-and-managing-files) + - [Create Directory](#create-directory) - [Upload](#upload) - - [Stream](#stream) - [Feed](#feed) - [Download](#download) - [Update](#update) @@ -42,18 +45,15 @@ zbox is a command line interface (CLI) tool to understand the capabilities of Z - [List](#list) - [Copy](#copy) - [Move](#move) - - [Sync](#sync) - - [Get differences](#get-differences) - [Get wallet](#get-wallet) - [Get](#get) - [Get metadata](#get-metadata) - [Rename](#rename) - [Stats](#stats) - [Repair](#repair) + - [Decrypt](#decrypt) - [Sign data](#sign-data) - - [Streaming](#streaming) - - [How it works:](#how-it-works) - - [Usage](#usage) + - [Lock and Unlock Tokens](#lock-and-unlock-tokens) - [Challenge pool information](#challenge-pool-information) - [Create read pool](#create-read-pool) @@ -129,65 +129,60 @@ Usage: zbox [command] Available Commands: - add Adds free storage assigner - alloc-cancel Cancel an allocation - alloc-fini Finalize an expired allocation - bl-info Get blobber info - bl-update Update blobber settings by its delegate_wallet owner - collect-reward Collect accrued rewards for a stake pool. - completion Generate the autocompletion script for the specified shell - copy copy an object(file/folder) to another folder on blobbers - cp-info Challenge pool information. - createdir Create directory - decrypt Decrypt text with passphrase - delete delete file from blobbers - download download file from blobbers - feed download segment files from remote live feed, and upload - get-diff Get difference of local and allocation root - get-download-cost Get downloading cost - get-mpt Directly view blockchain data - get-upload-cost Get uploading cost - getallocation Gets the allocation info - getwallet Get wallet information - help Help about any command - kill-blobber punitively deactivate a blobber - kill-validator punitively deactivate a validator - list list files from blobbers - list-all list all files from blobbers - listallocations List allocations for the client - ls-blobbers Show active blobbers in storage SC. - ls-validators Show active Validators. - meta get meta data of files from blobbers - move move an object(file/folder) to another folder on blobbers - newallocation Creates a new allocation - recent-refs get list of recently added refs - rename rename an object(file/folder) on blobbers - rp-create Create read pool if missing - rp-info Read pool information. - rp-lock Lock some tokens in read pool. - rp-unlock Unlock some expired tokens in a read pool. - sc-config Show storage SC configuration. - share share files from blobbers - shutdown-blobber deactivate a blobber - shutdown-validator deactivate a validator - sign-data Sign given data - sp-info Stake pool information. - sp-lock Lock tokens lacking in stake pool. - sp-unlock Unlock tokens in stake pool. - sp-user-info Stake pool information for a user. - start-repair start repair file to blobbers - stats stats for file from blobbers - stream capture video and audio streaming form local devices, and upload - sync Sync files to/from blobbers - transferallocation Transfer an allocation from one account to another - update update file to blobbers - updateallocation Updates allocation's expiry and size - upload upload file to blobbers - validator-info Get validator info - validator-update Update validator settings by its delegate_wallet owner - version Prints version information - wp-lock Lock some tokens in write pool. - wp-unlock Unlock some expired tokens in a write pool. + alloc-cancel Cancel an allocation + alloc-fini Finalize an expired allocation + bl-info Get blobber info + bl-update Update blobber settings by its delegate_wallet owner + collect-reward Collect accrued rewards for a stake pool. + completion Generate the autocompletion script for the specified shell + copy copy an object(file/folder) to another folder on blobbers + cp-info Challenge pool information. + createdir Create directory + decrypt Decrypt text with passphrase + delete delete file from blobbers + download download file from blobbers + feed download segment files from remote live feed, and upload + get-download-cost Get downloading cost + get-upload-cost Get uploading cost + getallocation Gets the allocation info + getwallet Get wallet information + help Help about any command + kill-blobber punitively deactivate a blobber + kill-validator punitively deactivate a validator + list list files from blobbers + list-all list all files from blobbers + listallocations List allocations for the client + ls-blobbers Show active blobbers in storage SC. + ls-validators Show active Validators. + meta get meta data of files from blobbers + move move an object(file/folder) to another folder on blobbers + newallocation Creates a new allocation + rename rename an object(file/folder) on blobbers + rp-create Create read pool if missing + rp-info Read pool information. + rp-lock Lock some tokens in read pool. + rp-unlock Unlock some expired tokens in a read pool. + sc-config Show storage SC configuration. + share share files from blobbers + shutdown-blobber deactivate a blobber + shutdown-validator deactivate a validator + sign-data Sign given data + sp-info Stake pool information. + sp-lock Lock tokens lacking in stake pool. + sp-unlock Unlock tokens in stake pool. + sp-user-info Stake pool information for a user. + start-repair start repair file to blobbers + stats stats for file from blobbers + sync Sync files to/from blobbers + transferallocation Transfer an allocation from one account to another + update update file to blobbers + updateallocation Updates allocation's expiry and size + upload upload file to blobbers + validator-info Get validator info + validator-update Update validator settings by its delegate_wallet owner + version Prints version information + wp-lock Lock some tokens in write pool. + wp-unlock Unlock some expired tokens in a write pool. Flags: --config string config file (default is config.yaml) @@ -195,11 +190,13 @@ Flags: --fee float transaction fee for the given transaction (if unset, it will be set to blockchain min fee) -h, --help help for zbox --network string network file to overwrite the network details (if required, default is network.yaml) - --silent Do not show interactive sdk logs (shown by default) + --silent (default false) Do not show interactive sdk logs (shown by default) --wallet string wallet file (default is wallet.json) --wallet_client_id string wallet client_id --wallet_client_key string wallet client_key --withNonce int nonce that will be used in transaction (default is 0) + +Use "zbox [command] --help" for more information about a command. ``` ### Global Flags @@ -249,24 +246,27 @@ and `free_storage` parameters. | allocationFileName | local file to store allocation information | allocation.txt | file path | | cost | returns the cost of the allocation, no allocation created | | flag | | data | number of data shards, effects upload and download speeds | 2 | int | -| expire | duration to allocation expiration | 720h | duration | -| free_storage | free storage marker file. | | file path | +| free_storage | free storage marker file. | | file path to json marker file | | owner | owner's id, use for funding an allocation for another | | string | | owner_public_key | public key, use for funding an allocation for another | | string | -| lock | lock write pool with given number of tokens | | float | -| parity | number of parity shards, effects availability | 2 | int | +| lock\* | lock write pool with given number of tokens | | float | +| parity | number of parity shards, effects availability (has to be more than 1 and less than the number of available blobbers on the chain (upper capped to 30)) | 2 | int | | read_price | filter blobbers by read price range | 0-inf | range | | size | size of space reserved on blobbers | 2147483648 | bytes | | usd | give token value in USD | | flag | | write_price | filter blobbers by write price range | 0-inf | range | -| false | bool | | third_party_extendable | specify if the allocation can be extended by users other than the owner | false | bool | -| forbid_upload | specify if users cannot upload to this allocation | false | bool | -| forbid_delete | specify if the users cannot delete objects from this allocation | false | bool | -| forbid_update | specify if the users cannot update objects in this allocation | false | bool | -| forbid_move | specify if the users cannot move objects from this allocation | false | bool | -| forbid_copy | specify if the users cannot copy object from this allocation | false | bool | -| forbid_rename | specify if the users cannot rename objects in this allocation | false | bool | +| forbid_upload | specify if users cannot upload to this allocation | false | bool | +| forbid_delete | specify if the users cannot delete objects from this allocation | false | bool | +| forbid_update | specify if the users cannot update objects in this allocation | false | bool | +| forbid_move | specify if the users cannot move objects from this allocation | false | bool | +| forbid_copy | specify if the users cannot copy object from this allocation | false | bool | +| forbid_rename | specify if the users cannot rename objects in this allocation | false | bool | +| blobber_auth_tickets | comma separated list of blobber auth tickets | | comma separated list of strings | +| force | force to get blobbers even if required number of blobbers are not available (should be passed true in case of restricted blobbers) | false | bool | +| name | allocation name | | string | +| preferred_blobbers | comma separated list of preferred blobber ids | | comma separated list of strings | +`*` - only required if free_storage not set.
newallocation @@ -351,25 +351,25 @@ Further we can add a blobber to the allocation, adding a blobber will allow a blobber to be removed. An increase in blobber count will increment the parity shards. -| Parameter | Required | Description | Valid Values | -| -------------- | -------- | -------------------------------------------------------------------- | ------------ | -| allocation | yes | allocation id | string | -| expiry | | adjust storage expiration time | duration | -| free_storage | | free storage marker file | string | -| lock | yes\* | lock additional tokens in write pool | int | -| size | | adjust allocation size | bytes | -| add_blobber | | add a new blobber to the allocation, required for remove_blobber | string | -| remove_blobber | | remove a blobber from the allocation, requires an add_blobber option | string2 | - +| Parameter | Required | Description | Default| Valid Values | +| -------------- | -------- | -------------------------------------------------------------------- | | ------------ | +| allocation | yes | allocation id | | string | +| name | | allocation name | | string | +| extend | | adjust storage expiration time | | duration | +| free_storage | | free storage marker file | | string | +| lock | yes\* | lock additional tokens in write pool | | int | +| size | | adjust allocation size | | bytes | +| add_blobber | | add a new blobber to the allocation, required for remove_blobber | | string | +| add_blobber_auth_ticket | | Auth ticket of blobber to add to the allocation | | string | +| remove_blobber | | remove a blobber from the allocation, requires an add_blobber option | | string | +| third_party_extendable | | specify if the allocation can be extended by users other than the owner | false | bool +| forbid_upload | |specify if users cannot upload to this allocation | false | bool +| forbid_delete | |specify if the users cannot delete objects from this allocation | false | bool +| forbid_update | |specify if the users cannot update objects in this allocation | false | bool +| forbid_move | |specify if the users cannot move objects from this allocation | false | bool +| forbid_copy | |specify if the users cannot copy object from this allocation | false | bool +| forbid_rename | |specify if the users cannot rename objects in this allocation | false | bool `*` only required if free_storage not set. -| third_party_extendable | specify if the allocation can be extended by users other than the owner | false | bool -| forbid_upload | specify if users cannot upload to this allocation |false | bool -| forbid_delete | specify if the users cannot delete objects from this allocation | false | bool -| forbid_update | specify if the users cannot update objects in this allocation |false | bool -| forbid_move | specify if the users cannot move objects from this allocation |false | bool -| forbid_copy | specify if the users cannot copy object from this allocation |false | bool -| forbid_rename | specify if the users cannot rename objects in this allocation |false | bool -
updateallocation @@ -398,6 +398,27 @@ Output: Allocation updated with txId : fb84185dae620bbba8386286726f1efcd20d2516bcf1a448215434d87be3b30d ``` +#### Transfer allocation ownership + +`transferallocation` transfers an allocation from one account to another. +This operation needs to be run by the current owner of the allocation. + +| Parameter | Required | Description | Valid Values | +| ---------- | -------- | ------------- | ------------ | +| allocation | yes | allocation id | string | +| new_owner | yes | new owner id | string | +| new_owner_key | yes | new owner public key | string | + +```shell +./zbox transferallocation --allocation d0939e912851959637257573b08c748474f0dd0ebbc8e191e4f6ad69e4fdc7ac --new_owner e61b1d5f081c4dfa4d45c852ca8abbfcdc3023ed4ffe2402ba7e9b2ebc56b129 --new_owner_key 421d9f68e121884a02587c1d5aad0ca81a4df2358abe1acb0952efdd5a6afc0ed198ec897a848890bd36a74f3dfd178a1ed9dcd2fab969b6ed073f98d795759d +``` + +Output: + +``` +transferred ownership of allocation : d0939e912851959637257573b08c748474f0dd0ebbc8e191e4f6ad69e4fdc7ac to e61b1d5f081c4dfa4d45c852ca8abbfcdc3023ed4ffe2402ba7e9b2ebc56b129 +``` + #### Forbid Allocation There are various operations which you can forbid on an allocation. Forbid flag works with [update allocation](#update-allocation) command. Check its working first. @@ -415,11 +436,11 @@ Here are the operations: Here is a sample command for --forbid_upload .Other parameters can be done the same way. -``` +```sh ./zbox updateallocation --allocation $ALLOC --forbid_upload ``` Sample Response : -``` +```sh Allocation Updated with txID : b84185dae620bbba8386286726f1efcd20d2516bcf1a448215434d87be3b30d ``` To test functionality try uploading file to allocation. You should get the following response : @@ -449,9 +470,13 @@ Cancelling an allocation can only occur if the amount of failed challenges excee Example -``` +```sh ./zbox alloc-cancel --allocation ``` +Sample Response : +```sh +Allocation canceled with txId : 501df5a8e2a6b8ebced1d1e7dc4ed17f653012dd49834801b4413786ec031cd2 +``` #### Finalise allocation @@ -478,17 +503,24 @@ An allocation can be finalised by the owner or one of the allocation blobbers. Example -``` +```sh ./zbox alloc-fini --allocation ``` + +Sample Response: + +```sh +Allocation finalized with txId : 2a6c031ced1d1e7dc4ed17801b4413786ecd2501df5ab8ebf653012dd498348e +``` #### List blobbers Use `ls-blobbers` command to show active blobbers. | Parameter | Required | Description | Valid Values | | --------- | -------- | ------------------------------------ | ------------ | -| all | no | shows active and non active blobbers | flag | -| json | no | display result in .json format | flag | +| all | no | shows active and non active blobbers | bool | +| json | no | display result in .json format | bool | +| stakable | no | gets only stakable list of blobbers if set to true | bool |
ls-blobbers @@ -525,7 +557,7 @@ Use `bl-info` command to get detailed blobber information. | Parameter | Required | Description | default | Valid values | | ---------- | -------- | ----------------------------------- | ------- | ------------ | -| blobber id | yes | blobber on which to get information | | string | +| blobber_id | yes | blobber on which to get information | | string | | json | no | print result in json format | false | boolean |
@@ -570,10 +602,16 @@ settings: | ---------- | -------- | ---------------------------------------------- | ------- | ------------ | | allocation | yes | allocation id, sender must be allocation owner | | string | -```shell +Sample Request : +```sh ./zbox list-all --allocation 4ebeb69feeaeb3cd308570321981d61beea55db65cbeba4ba3b75c173c0f141b ``` +Sample Response : +```sh +[{"size":0,"mimetype":"","actual_size":0,"hash":"","type":"d","encrypted_key":"","lookup_hash":"55e45925760e33f113642569a539e17a531285d2bf53dc78fde954017c8b27b3","created_at":1714335152,"updated_at":1714335152,"name":"abcd","path":"/abcd"}] +``` + #### List owner's allocations `listallocations` provides a list of all allocations owned by the user. @@ -613,7 +651,9 @@ on the blockchain not the blobber. | read_price | no | update read price | | float | | service_charge | no | update service charge | | float | | write_price | no | update write price | | float | - +| url | no | update the url of the blobber | | string | +| is_restricted | no | update whether blobber is restricted | false | bool | +| not_available | no | set blobber's availability for new allocations | false | bool |
bl-update @@ -625,20 +665,48 @@ Example Update blobber read price -``` +```sh ./zbox bl-update --blobber_id 0ece681f6b00221c5567865b56040eaab23795a843ed629ce71fb340a5566ba3 --read_price 0.1 ``` -Get Version +Response : +```sh +blobber settings updated successfully +``` -The version of Zbox and Gosdk can be fetched using the ./zbox version command. +#### Update validator settings + +Use `./zbox validator-update ` to update a validator's configuration settings. This updates the settings +on the blockchain not the validator. + +| Parameter | Required | Description | default | Valid values | +| ------------------ | -------- | ----------------------------------------- | ------- | ------------ | +| validator_id | yes | id of validator of which to update settings | | string | +| max_stake | no | update maximum stake | | float | +| min_stake | no | update minimum stake | | float | +| num_delegates | no | update maximum number of delegates | | int | +| service_charge | no | update service charge | | float | +
+ validator-update + +Example + +Update validator service charge +```sh +./zbox validator-update --validator_id 0ece681f6b00221c5567865b56040eaab23795a843ed629ce71fb340a5566ba3 --service_charge 0.1 ``` + +#### Get Version + +The version of Zbox and Gosdk can be fetched using the ./zbox version command. + +```sh ./zbox version ``` Sample Response: -``` +```sh zbox....: v1.4.3 gosdk...: v1.8.14 ``` @@ -648,11 +716,17 @@ gosdk...: v1.8.14 List all active validators on the network Command: -``` +```sh ./zbox ls-validators ``` + +| Parameter | Required | Description +| ------------------ | -------- | ----------------------------------------- +| --stakable | no | Gets only validators that can be staked if set to true +| --json | no | Print Response as json data + Response : -``` +```sh id: b9f4f244e2e483548795e42dad0c5b5bb8f5c25d70cadeafc202ce6011b7ff8c url: https://demo.zus.network/validator03/ settings: @@ -679,14 +753,13 @@ settings: | ------------------ | -------- | ----------------------------------------- | --validator_id | yes | id of validator whose configuration has to be fetched | --json | optional | Print Response as json data -| --help | no | Provide information about the command Sample Command : -``` +```sh ./zbox validator-info --validator_id f82ab34a98406b8757f11513361752bab9cb679a5cb130b81 ``` Sample Response : -``` +```sh id: f82ab34a98406b8757f11513361752bab9cb679a5cb130b81a4e86cec50eefc3 url: https://demo2.zus.network/validator01 last_health_check: 2023-05-12 20:09:15 +0530 IST @@ -701,14 +774,51 @@ settings: num_delegates: 50 service_charge: 10 % ``` + +#### Shutdown Blobber +`./zbox shutdown-blobber` command deactivates a blobber to avoid storage of data . Required parameters are : + +| Parameter | Required | Description +| ------------------ | -------- | ----------------------------------------- +| --id | yes | Blobber Id to kill a specific blobber.Can be retrieved using [List blobbers](#list-blobbers). +| --fee | no | Custom fee for transaction + + Sample Command : +```sh +./zbox shutdown-blobber --id $BLOBBER_ID --wallet $CHAIN_OWNER_WALLET +``` +Note : Shutdown Blobber command should be invoked from chain owner wallet only + +Sample Response : +```sh +shutdown blobber $BLOBBER_ID with txId : 2a6c031ced1d1e7dc4ed17801b4413786ecd2501df5ab8ebf653012dd498348e +``` + +#### Shutdown Validator +`./zbox shutdown-validator` command deactivates a validator. Required parameters are : + +| Parameter | Required | Description +| ------------------ | -------- | ----------------------------------------- +| --id | yes | Validator Id to kill a specific validator.Can be retrieved using [List all Validators](#list-all-validators). +| --fee | no | Custom fee for transaction + + Sample Command : +```sh +./zbox shutdown-validator --id $VALIDATOR_ID --wallet $CHAIN_OWNER_WALLET +``` +Note : Shutdown Validator command should be invoked from chain owner wallet only + +Sample Response : +```sh +shutdown validator $VALIDATOR_ID with txId : 2a6c031ced1d1e7dc4ed17801b4413786ecd2501df5ab8ebf653012dd498348e +``` + #### Kill Blobber `./zbox kill-blobber` command deactivates a blobber to avoid storage of data . Required parameters are : | Parameter | Required | Description | ------------------ | -------- | ----------------------------------------- -| --blobber_id | yes | Blobber Id to kill a specific blobber.Can be retrieved using [List blobbers](#list-blobbers). -| --json | optional | Print Response as json data -| --help | no | Provide information about the command +| --id | yes | Blobber Id to kill a specific blobber.Can be retrieved using [List blobbers](#list-blobbers). Sample Command : ``` @@ -727,21 +837,40 @@ killed blobber $BLOBBER_ID | Parameter | Required | Description | ------------------ | -------- | ----------------------------------------- -| --validator_id | yes | Validator Id to kill a specific blobber.Can be retrieved using [List all Validators](#list-all-validators). -| --json | optional | Print Response as json data -| --help | no | Provide information about the command +| --id | yes | Validator Id to kill a specific blobber.Can be retrieved using [List all Validators](#list-all-validators). Sample Command : -``` +```sh ./zbox kill-validator --id $VALIDATOR_ID --wallet $CHAIN_OWNER_WALLET ``` Sample Response : -``` +```sh killed validator, id: $VALIDATOR_ID ``` ### Uploading and Managing files +#### Create Directory + +Use `createdir` command to create a directory in the specified allocation + +The user must be the owner of the allocation. + +| Parameter | Required | Description | Default | Valid values | +| ------------- | -------- | ------------------------------------------------------- | ------- | ------------ | +| allocation | yes | allocation id, sender must be allocation owner | | string | +| dirname | yes | path to directory | | string | + +Sample Request : +```sh +./zbox createdir --allocation {ALLOC_ID} --dirname /abcd/ +``` + +Sample Response : +```sh +/abcd/ directory created +``` + #### Upload Use `upload` command to upload file(s). @@ -759,8 +888,10 @@ The user must be the owner of the allocation.You can request the file be encrypt | web-streaming | no | transcode file before upload to fragmented mp4 | false | boolean | | localpath | yes | local path of the file to upload | | file path | | remotepath | yes | remote path to upload file to, use to access file later | | string | -| thumbnailpath | no | local path of thumbnaSil | | file path | +| thumbnailpath | no | local path of thumbnail | | file path | | chunknumber | no | how many chunks should be uploaded in a http request | 1 | int | +| attr-who-pays-for-reads | no | Who pays for reads: owner or 3rd_party | owner | owner / 3rd_party| +| multiuploadjson | no | A JSON file containing multiupload options | | file path |
upload @@ -817,29 +948,6 @@ Response: Status completed callback. Type = video/fmp4. Name = raw.samplevideo.mp4 ``` -#### Stream - -Use `stream` to capture video and audio streaming form local devices, and upload - -The user must be the owner of the allocation.You can request the file be encrypted before upload, and can send thumbnails with the file. - -| Parameter | Required | Description | Default | Valid values | -| ------------- | -------- | ------------------------------------------------------------ | ------- | ------------ | -| allocation | yes | allocation id, sender must be allocation owner | | string | -| encrypt | no | encrypt file before upload | false | boolean | -| localpath | yes | local path of segment files to download, generate and upload | | file path | -| remotepath | yes | remote path to upload file to, use to access file later | | string | -| thumbnailpath | no | local path of thumbnaSil | | file path | -| chunknumber | no | how many chunks should be uploaded in a http request | 1 | int | -| delay | no | set segment duration to seconds. | 5 | int | - -
- stream - -![image](https://github.com/0chain/blobber/wiki/uml/usecase/live_upload_live.png) - -
- #### Feed Use `feed` command to automatically download segment files from remote live feed with `--downloader-args "-q -f best"` @@ -861,7 +969,8 @@ The user must be the owner of the allocation.You can request the file be encrypt | feed | no | set remote live feed to url. | false | url | | downloader-args | no | pass args to youtube-dl to download video. default is \"-q -f best\". | -q -f best | [youtube-dl](https://github.com/ytdl-org/youtube-dl/blob/master/README.md#options) | | ffmpeg-args | no | pass args to ffmpeg to build segments. | -loglevel warning | [ffmpeg](https://www.ffmpeg.org/ffmpeg.html) | - +| attr-who-pays-for-reads | no | Who pays for reads: owner or 3rd_party | owner | owner / 3rd_party| +enum
feed @@ -922,11 +1031,14 @@ Use `download` command to download your own or a shared file. | endblock | no | download until specified block number | | int | | localpath | yes | local path to which to download the file to | | file path | | remotepath | yes | remote path to which the file was uploaded | | string | -| rx_pay | no | `authticket` must be valid, true = sender pays, false = allocation owner pays | false | boolean | | startblock | no | start download from specified block | | int | | thumbail | no | only download the thumbnail | false | boolean | | live | no | start m3u8 downloader,and automatically generate media playlist(m3u8) on --localpath | false | boolean | | delay | no | pass segment duration to generate media playlist(m3u8). only works with --live. default duration is 5s. | 5 | int | +| lookuphash | no | The remote lookuphash of the object retrieved from the list | | string | +| multidownloadjson | no | A JSON file containing multi download options | | string | +| verifydownload | no | pass this option to verify downloaded blocks | false | boolean | +
download @@ -935,12 +1047,20 @@ Use `download` command to download your own or a shared file.
-Example +Example (for owner) ``` ./zbox download --allocation 3c0d32560ea18d9d0d76808216a9c634f661979d29ba59cc8dafccb3e5b95341 --remotepath /myfiles/horse.jpeg --localpath ../horse.jpeg ``` +Example (for non-owner) with authticket + +``` +./zbox download --allocation 3c0d32560ea18d9d0d76808216a9c634f661979d29ba59cc8dafccb3e5b95341 --authticket eyJjbGllbnRfaWQiOiIiLCJvd25lcl9pZCI6IiIsImFsbG9jYXRpb24iOiIzYzBkMzI1NjBlYTE4ZDlkMGQ3NjgwODIxNmE5YzYzNGY2NjE5NzlkMjliYTU5Y2M4ZGFjYmMzZTViOTUzNDEiLCJyZW1vdGVwYXRoIjoiL215ZmlsZXMvaG9yc2UuanBlZyIsImV4cG9ydCI6IjIwMjEtMDctMjFUMDk6MjA6MjAuMDAwWiJ9 --localpath ../horse.jpeg +``` +If the authticket is for a directory, not a file, then you need to provide the lookup hash of the file you want to download. You can get the lookup hash from the [list](#list) command. + + Response: ``` @@ -978,6 +1098,16 @@ can update a file. To add collaborators to an allocation, use
+Sample Command : +```sh +./zbox update --allocation {ALLOC_ID} --localpath ./zbox_commands.txt --remotepath /abcd/ --encrypt +``` + +Sample Response : +```sh +Status completed callback. Type = application/octet-stream. Name = zbox_commands.txt +``` + #### Delete Use `delete` command to delete your file on the allocation. Only the owner @@ -1015,14 +1145,6 @@ File successfully deleted (Can be verified using [list](https://github.com/0chai Use share command to generate an authtoken that provides authorization to the holder to the specified file on the remotepath. -- --allocation string Allocation ID -- --clientid string ClientID of the user to share with. Leave blank for public share -- --encryptionpublickey string Encryption public key of the client you want to share with (from [getwallet](#Get-wallet) command ) -- --remotepath string Remote path to share -- --expiration-seconds number The seconds after which the ticket will expire(defaults to number of seconds in 90 days - if option not provided) -- --available-after Timelock for private file that makes the file available for download at certain time. 4 input formats are supported: +1h30m, +30, 1647858200 and 2022-03-21 10:21:38 - `auth ticket` can be used with [download](#download), and [list](#list), [meta](#get-metadata) and [get_download_cost](#download-cost), but only for files in the pre-defined remote path. @@ -1030,7 +1152,7 @@ the pre-defined remote path. | Parameter | Required | Description | Valid values | | ------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | allocation | yes | allocation id | string | -| clientid | no | id of user to share file with, leave blank for public share | string | +| clientid | no | id of user to share file with (or revoke share from), leave blank for public share (or to revoke public sharing) | string | | encryptionpublickey | no | public key of the client to share file with, required if clientId | string | | expiration-seconds | no | seconds before `auth ticket` expires | int | | remotepath | yes | remote path of file to share | string | @@ -1176,19 +1298,27 @@ contents.
-Example +Example (for owner) ``` ./zbox list --allocation 8695b9e7f986d4a447b64de020ba86f53b3b5e2c442abceb6cd65742702067dc --remotepath / ``` -Response: +Example (for non-owner with auth ticket) ``` -auth ticket :eyJjbGllbnRfaWQiOiJiNmRlNTYyYjU3YTBiNTkzZDA0ODA2MjRmNzlhNTVlZDQ2ZGJhNTQ0NDA0NTk1YmVlMDI3MzE0NGUwMTAzNGFlIiwib3duZXJfaWQiOiJiNmRlNTYyYjU3YTBiNTkzZDA0ODA2MjRmNzlhNTVlZDQ2ZGJhNTQ0NDA0NTk1YmVlMDI3MzE0NGUwMTAzNGFlIiwiYWxsb2NhdGlvbl9pZCI6Ijg2OTViOWU3Zjk4NmQ0YTQ0N2I2NGRlMDIwYmE4NmY1M2IzYjVlMmM0NDJhYmNlYjZjZDY1NzQyNzAyMDY3ZGMiLCJmaWxlX3BhdGhfaGFzaCI6IjIwZGM3OThiMDRlYmFiMzAxNTgxN2M4NWQyMmFlYTY0YTUyMzA1YmFkNmY3NDQ5YWNkMzgyOGM4ZDcwYzc2YTMiLCJmaWxlX25hbWUiOiIxLnR4dCIsInJlZmVyZW5jZV90eXBlIjoiZiIsImV4cGlyYXRpb24iOjE2MjY0MjA1NzQsInRpbWVzdGFtcCI6MTYxODY0NDU3NCwicmVfZW5jcnlwdGlvbl9rZXkiOiJ7XCJyMVwiOlwiOUpnci9aVDh6VnpyME1BcWFidlczdnhoWEZoVkdMSGpzcVZtVUQ1QTJEOD1cIixcInIyXCI6XCIrVEk2Z1pST3JCR3ZURG9BNFlicmNWNXpoSjJ4a0I4VU5SNTlRckwrNUhZPVwiLFwicjNcIjpcInhySjR3bENuMWhqK2Q3RXU5TXNJRzVhNnEzRXVzSlZ4a2N6YXN1K0VqQW89XCJ9Iiwic2lnbmF0dXJlIjoiZTk3NTYyOTAyODU4OTBhY2QwYTcyMzljNTFhZjc0YThmNjU2OTFjOTUwMzRjOWM0ZDJlMTFkMTQ0MTk0NmExYSJ9 +./zbox list --allocation 39c41dcc1f3fd5154e92e6285dd18ed869b72662198839a862d8f1fa627ec256 --authticket eyJjbGllbnRfaWQiOiIiLCJvd25lcl9pZCI6ImJhNmFiNDI2NjQ0ZjJiMzI2YjFkYTNkZDQyNjIyOWZlZDE5ZWVkODUzODdkZWFmNzJlMDVjYjBmYTRjNWFiYmIiLCJhbGxvY2F0aW9uX2lkIjoiMzljNDFkY2MxZjNmZDUxNTRlOTJlNjI4NWRkMThlZDg2OWI3MjY2MjE5ODgzOWE4NjJkOGYxZmE2MjdlYzI1NiIsImZpbGVfcGF0aF9oYXNoIjoiNDIzYjRhZTE0Y2YxZWNhMDViYzg1M2E0ZmJiMWVmZGU5ZWU4NzJkYmQxM2M0M2RmZjY4MmVlMTI3ZWU4ZjQ1NyIsImFjdHVhbF9maWxlX2hhc2giOiIiLCJmaWxlX25hbWUiOiJmaWxlcyIsInJlZmVyZW5jZV90eXBlIjoiZCIsImV4cGlyYXRpb24iOjAsInRpbWVzdGFtcCI6MTcxNDY0MTcxNiwiZW5jcnlwdGVkIjpmYWxzZSwic2lnbmF0dXJlIjoiMTNhNzk0YTNjZTczZDFmNjE2ZmY3Njc0MjNhNzBlMzZlNDMwZWYyYTA1MWQyOGViZjQ1YTkzMjIzNjliNTMwYyJ9 ``` -Response will be a list with information for each file/folder in the given path. The information includes lookuphash which is require for download via authticket. +Example Response: + +``` + TYPE | NAME | PATH | SIZE | NUM BLOCKS | ACTUAL SIZE | ACTUAL NUM BLOCKS | LOOKUP HASH | IS ENCRYPTED +-------+--------+---------------+------+------------+-------------+-------------------+------------------------------------------------------------------+--------------- + f | one.js | /files/one.js | 5 | 1 | 19 | 1 | 566669a0240db1ca5d48ab0e013f125faf3beb34f01ef065ffb31255d9e63f43 | NO +``` + +Response will be a list with information for each file/folder in the given path. **The information includes lookuphash which is require for download via authticket**. #### Copy @@ -1252,84 +1382,6 @@ Response: /file.txt moved ``` -#### Sync - -`sync` command syncs all files from the local folder recursively to the remote. -Only the allocation's owner can successfully run `sync`. - -| Parameter | Required | Description | default | Valid values | -| ----------- | -------- | --------------------------------------------------------------------------------------------- | ------- | ------------ | -| allocation | yes | allocation id | | string | -| encryptpath | no | local directory path to be uploaded as encrypted | false | boolean | -| excludepath | no | paths to exclude from sync | | string array | -| localchache | no | local chache of remote snapshot. Used for comparsion with remote. After sync will be updated. | | string | -| localpath | yes | local directory to which to sync | | file path | -| uploadonly | no | only upload and update files | false | boolean | -| chunknumber | no | how many chunks should be uploaded in a http request | 1 | int | - -
- sync - -![image](https://user-images.githubusercontent.com/6240686/127884376-a95c4f27-4b2a-4d9b-91b6-c7e7919f88bc.png) - -
- -Example - -``` -./zbox sync --allocation 8695b9e7f986d4a447b64de020ba86f53b3b5e2c442abceb6cd65742702067dc --localpath /home/dung/Desktop/alloc --localcache /home/dung/Desktop/localcache.json -``` - -Response: - -``` - OPERATION | PATH -+-----------+----------------+ - Download | /1.txt - Download | /afolder/1.txt - Download | /d2.txt - - 4 / 4 [===========================================================================] 100.00% 0s -Status completed callback. Type = application/octet-stream. Name = 1.txt - 4 / 4 [===========================================================================] 100.00% 0s -Status completed callback. Type = application/octet-stream. Name = 1.txt - 7 / 7 [===========================================================================] 100.00% 0s -Status completed callback. Type = application/octet-stream. Name = d2.txt - -Sync Complete -Local cache saved. -``` - -It will sync your localpath with the remote and do all the required CRUD operations. - -#### Get differences - -`./zbox get-diff` command returns the differences between the local files specified by `localpath` and the files stored -on the root remotepath of the allocation.`localcache` flag can also be specified to use the local cache of remote snapshot created during [Sync](#sync) for file comparison. - -| Parameter | Required | Description | default | Valid values | -| ----------- | -------- | --------------------------------------------- | ------- | ------------ | -| allocation | yes | allocation id | | string | -| excludepath | no | remote folder paths to exclude during syncing | | string array | -| localcache | no | local cache of remote snapshot | | string | -| localpath | yes | local directory to sync | | string | - -Example - -``` -./zbox get-diff --allocation $ALLOC --localpath $local -``` - -Response: - -``` -[{"operation":"Upload","path":"/file1.txt","type":"f","attributes":{}}, -{"operation":"Upload","path":"/file2.txt","type":"f","attributes":{}}, -{"operation":"Upload","path":"/file3.txt","type":"f","attributes":{}}, -{"operation":"Download","path":"/myfiles/file1.txt","type":"f","attributes":{}}, -{"operation":"Download","path":"/myfiles/file2.txt","type":"f","attributes":{}}] -``` - #### Get wallet Use `getwallet` command to get additional wallet information including Encryption @@ -1355,7 +1407,7 @@ Response: Response will give details for current selected wallet (or wallet file specified by optional --wallet parameter) -#### Get +#### Get Allocation Information Use `getallocation` command to get the information about the allocation such as total size , used size, number of challenges and challenges passed/failed/open/redeemed. @@ -1363,6 +1415,7 @@ and challenges passed/failed/open/redeemed. | Parameter | Required | Description | default | Valid values | | ---------- | -------- | ----------------------------- | ------- | ------------ | | allocation | yes | allocation id | | string | +| blocks-per-marker | no | print response in json format | 10 | int | | json | no | print response in json format | false | boolean |
@@ -1382,29 +1435,129 @@ Response: ``` allocation: - id: 8695b9e7f986d4a447b64de020ba86f53b3b5e2c442abceb6cd65742702067dc - tx: 026c9d331e9c93aee4f3247507c20bdd4b7429bd81d27845bfab83f9c9c082e6 (latest create/update allocation transaction hash) + id: 2813684263f6a28de4d1301255bba21c6cf9429cc02286b56ef38db7b068d8f1 + tx: 2813684263f6a28de4d1301255bba21c6cf9429cc02286b56ef38db7b068d8f1 (latest create/update allocation transaction hash) data_shards: 4 parity_shards: 2 - size: 6.0 GiB - expiration_date: 2021-05-24 00:27:23 +0700 +07 + size: 2.0 GiB + expiration_date: 2024-05-27 15:42:23 +0200 EET + third_party_extendable: false + file_options: 00111111 + write pool 1.000 ZCN blobbers: - - blobber_id: dea18e3f3c308666cb489877b9b2c7e2babf797d8b8c322fa9d074105787a9e9 - base URL: http://demo.zus.network:31304 - size: 1.0 GiB - min_lock_demand: 0.0012333333 - spent: 0.0000244839 (moved to challenge pool or to the blobber) - penalty: 0 (blobber stake slash) - read_reward: 0.000024414 - returned: 0.0000000233 (on challenge failed) - challenge_reward: 0 (on challenge passed) - final_reward: 0 (if finalized) + min_lock_demand: 0 % + - blobber_id: 6f895dfc20b5e55df6b3084eeb69ad604eb232e5853cf054983d7495bdbc9ed7 + base URL: https://dev1.zus.network/blobber02/ + size: 512.0 MiB + min_lock_demand: 0 SAS + spent: 0 SAS (moved to challenge pool or to the blobber) + penalty: 0 SAS (blobber stake slash) + read_reward: 0 SAS + returned: 0 SAS (on challenge failed) + challenge_reward: 0 SAS (on challenge passed) + final_reward: 0 SAS (if finalized) + terms: (allocation related terms) + read_price: 0 SAS / GB (by 64KB chunks) + write_price: 10.000 mZCN / GB + min_lock_demand: 0 SAS % + max_offer_duration: 0s + - blobber_id: 98f14362f075caf467653044cf046eb9e8a5dfee88dc8b78cad1891748245003 + base URL: https://dev1.zus.network/blobber01/ + size: 512.0 MiB + min_lock_demand: 0 SAS + spent: 0 SAS (moved to challenge pool or to the blobber) + penalty: 0 SAS (blobber stake slash) + read_reward: 0 SAS + returned: 0 SAS (on challenge failed) + challenge_reward: 0 SAS (on challenge passed) + final_reward: 0 SAS (if finalized) + terms: (allocation related terms) + read_price: 0 SAS / GB (by 64KB chunks) + write_price: 10.000 mZCN / GB + min_lock_demand: 0 SAS % + max_offer_duration: 0s + - blobber_id: 0e2fa9abc5a14231a1e7dc27b129480b732222e8e864d3b4e62d60a8b8ae617b + base URL: https://dev2.zus.network/blobber02/ + size: 512.0 MiB + min_lock_demand: 0 SAS + spent: 0 SAS (moved to challenge pool or to the blobber) + penalty: 0 SAS (blobber stake slash) + read_reward: 0 SAS + returned: 0 SAS (on challenge failed) + challenge_reward: 0 SAS (on challenge passed) + final_reward: 0 SAS (if finalized) terms: (allocation related terms) - read_price: 0.0099999999 tok / GB (by 64KB chunks) - write_price: 0.0099999999 tok / GB - min_lock_demand: 10 % - max_offer_duration: 744h0m0s - challenge_completion_time: 2m0s + read_price: 0 SAS / GB (by 64KB chunks) + write_price: 10.000 mZCN / GB + min_lock_demand: 0 SAS % + max_offer_duration: 0s + - blobber_id: 06166f3dfd72a90cd0b51f4bd7520d4434552fc72880039b1ee1e8fe4b3cd7ea + base URL: https://dev2.zus.network/blobber01/ + size: 512.0 MiB + min_lock_demand: 0 SAS + spent: 0 SAS (moved to challenge pool or to the blobber) + penalty: 0 SAS (blobber stake slash) + read_reward: 0 SAS + returned: 0 SAS (on challenge failed) + challenge_reward: 0 SAS (on challenge passed) + final_reward: 0 SAS (if finalized) + terms: (allocation related terms) + read_price: 0 SAS / GB (by 64KB chunks) + write_price: 10.000 mZCN / GB + min_lock_demand: 0 SAS % + max_offer_duration: 0s + - blobber_id: 5c61f8d3e63528dfe45db89a598bd5c42c71b3994f7639d4647268ba75269d9a + base URL: https://dev3.zus.network/blobber02/ + size: 512.0 MiB + min_lock_demand: 0 SAS + spent: 0 SAS (moved to challenge pool or to the blobber) + penalty: 0 SAS (blobber stake slash) + read_reward: 0 SAS + returned: 0 SAS (on challenge failed) + challenge_reward: 0 SAS (on challenge passed) + final_reward: 0 SAS (if finalized) + terms: (allocation related terms) + read_price: 0 SAS / GB (by 64KB chunks) + write_price: 10.000 mZCN / GB + min_lock_demand: 0 SAS % + max_offer_duration: 0s + - blobber_id: 8d19a8fd7147279d1dfdadd7e3ceecaf91c63ad940dae78731e7a64b104441a6 + base URL: https://dev3.zus.network/blobber01/ + size: 512.0 MiB + min_lock_demand: 0 SAS + spent: 0 SAS (moved to challenge pool or to the blobber) + penalty: 0 SAS (blobber stake slash) + read_reward: 0 SAS + returned: 0 SAS (on challenge failed) + challenge_reward: 0 SAS (on challenge passed) + final_reward: 0 SAS (if finalized) + terms: (allocation related terms) + read_price: 0 SAS / GB (by 64KB chunks) + write_price: 10.000 mZCN / GB + min_lock_demand: 0 SAS % + max_offer_duration: 0s + read_price_range: 0 SAS-0 SAS (requested) + write_price_range: 0 SAS-25.000 mZCN (requested) + challenge_completion_time: 0s (max) + start_time: 2024-04-27 15:42:23 +0200 EET + finalized: false + canceled: false + moved_to_challenge: 0 SAS + moved_back: 0 SAS + moved_to_validators: 0 SAS + stats: + total size: 2.0 GiB + used size: 0 B + number of writes: 0 + total challenges: 0 + passed challenges: 0 + failed challenges: 0 + open challenges: 0 + last challenge redeemed: + price: + time_unit: 720h0m0s + read_price: 0 SAS / GB (by 64KB) + write_price: 15.000 mZCN / GB / 720h0m0s ``` #### Get metadata @@ -1444,7 +1597,7 @@ Response: f | 1.txt | /1.txt | 20dc798b04ebab3015817c85d22aea64a52305bad6f7449acd3828c8d70c76a3 | 4 | application/octet-stream | 03cfd743661f07975fa2f1220c5194cbaff48451 ``` -**With authticket ** +**With authticket** ``` ./zbox meta --lookuphash 20dc798b04ebab3015817c85d22aea64a52305bad6f7449acd3828c8d70c76a3 --authticket eyJjbGllbnRfaWQiOiJiNmRlNTYyYjU3YTBiNTkzZDA0ODA2MjRmNzlhNTVlZDQ2ZGJhNTQ0NDA0NTk1YmVlMDI3MzE0NGUwMTAzNGFlIiwib3duZXJfaWQiOiJiNmRlNTYyYjU3YTBiNTkzZDA0ODA2MjRmNzlhNTVlZDQ2ZGJhNTQ0NDA0NTk1YmVlMDI3MzE0NGUwMTAzNGFlIiwiYWxsb2NhdGlvbl9pZCI6Ijg2OTViOWU3Zjk4NmQ0YTQ0N2I2NGRlMDIwYmE4NmY1M2IzYjVlMmM0NDJhYmNlYjZjZDY1NzQyNzAyMDY3ZGMiLCJmaWxlX3BhdGhfaGFzaCI6IjIwZGM3OThiMDRlYmFiMzAxNTgxN2M4NWQyMmFlYTY0YTUyMzA1YmFkNmY3NDQ5YWNkMzgyOGM4ZDcwYzc2YTMiLCJmaWxlX25hbWUiOiIxLnR4dCIsInJlZmVyZW5jZV90eXBlIjoiZiIsImV4cGlyYXRpb24iOjE2MjY0MjA1NzQsInRpbWVzdGFtcCI6MTYxODY0NDU3NCwicmVfZW5jcnlwdGlvbl9rZXkiOiJ7XCJyMVwiOlwiOUpnci9aVDh6VnpyME1BcWFidlczdnhoWEZoVkdMSGpzcVZtVUQ1QTJEOD1cIixcInIyXCI6XCIrVEk2Z1pST3JCR3ZURG9BNFlicmNWNXpoSjJ4a0I4VU5SNTlRckwrNUhZPVwiLFwicjNcIjpcInhySjR3bENuMWhqK2Q3RXU5TXNJRzVhNnEzRXVzSlZ4a2N6YXN1K0VqQW89XCJ9Iiwic2lnbmF0dXJlIjoiZTk3NTYyOTAyODU4OTBhY2QwYTcyMzljNTFhZjc0YThmNjU2OTFjOTUwMzRjOWM0ZDJlMTFkMTQ0MTk0NmExYSJ9 @@ -1461,6 +1614,20 @@ TYPE | NAME | LOOKUP HASH Response will be metadata for the given filepath/lookuphash (if using authTicket) +**For a directory** + +``` +./zbox meta --allocation 8695b9e7f986d4a447b64de020ba86f53b3b5e2c442abceb6cd65742702067dc --remotepath /files +``` + +Response: + +``` + TYPE | NAME | PATH | LOOKUP HASH +-------+-------+--------+------------------------------------------------------------------- + d | files | /files | 9184e5f2634bd7b2cdaec97de9c3eb8f60192640d5e6f32bb3271f094ef7cc7a +``` + #### Rename `rename` command renames a file existing already on dStorage. Only the @@ -1530,7 +1697,7 @@ Response: #### Repair -Use `start-repair` command to repair a file on dStorage. +Use `start-repair` command to repair an allocation. Sometimes, your operations on the allocation files may not be successful on some blobbers (since the consensus of the operations is `data_shards + 1`) leading to incosistency of the allocation data on its hosting blobbers. In such cases, you can use the repair command to repair the allocation. The repair command will repair the allocation by downloading the files from the blobbers with the latest version and re-uploading them to the blobbers that failed before. The repair command will also update the allocation metadata on the blobbers. Only the owner of the allocation can repair the allocation. ![repair](https://user-images.githubusercontent.com/65766301/120052600-b364c680-c043-11eb-9bf2-038ab244fed6.png) \ @@ -1559,91 +1726,32 @@ Response: Repair file completed, Total files repaired: 0 ``` -#### Sign data +#### Decrypt -`sign-data` uses the information from your wallet to sign the input data string +`decrypt` is used to decrypt data with a passphrase | Parameter | Required | Description | default | Valid values | | --------- | -------- | -------------- | ------- | ------------ | -| data | yes | string to sign | | string | +| text | yes | string to decrypt | | string | +| passphrase | yes | passphrase to use to decrypt | | string | -```shell -./zbox sign-data "data to sign" -Signature : 9432ab2ee602062afaf48c4016b373a65db48a8546a81c09dead40e54966399e +```sh +./zbox decrypt --text {encryted_data} --passphrase {passphrase} ``` ---- - -#### Streaming - -Video streaming with Zbox CLI can be implemented with players for different operating platforms(iOS, Android Mac).Zbox CLI does not have a player itself and use the the downloadFileByBlocks helper function to properly returns file-chunks with correct byte range. - -![streaming-android](https://user-images.githubusercontent.com/65766301/120052635-ce373b00-c043-11eb-94a5-a9711078ee54.png) - -##### How it works: - -When the user starts the video player (ExoPlayer for Android or AVPlayer for iOS), A ZChainDataSource starts chunked download and requests chunks of video from the buffer(a Middleman between streaming player and Zbox). - -After the arrival of the first chunk, the player starts requesting more chunks from the buffer, which requests the Zbox SDK. Zbox SDK, which is built using GO, makes use of the downloadFileByBlocks method to reliably download large files by chunking them into a sequence of parts that can be downloaded individually. Once the blocks are downloaded, they are read into input streams and added to the media source of the streaming player. - -The task of downloading files and writing them to buffer using Zbox SDK happens constantly, and If players request random bits of video, they are delivered instantly by a buffer. - -In a case, if the player didn't receive chunks (for example, it's still not downloaded), then the player switches to STALE state, and the video stream will pause. During the STALE state, a player tries to make multiple requests for chunks; if didn't receive a response, the video stream stops. - -##### Usage - -To understand how Zbox CLI provides downloading of files by blocks. Let's consider an allocation that has `audio. mp3 ` file stored on dStorage. Make sure the file has a large size(more than 64 kB(64000 bytes)) to download the file by blocks. The size and other attributes of the sample `audio. mp3` file can be viewed using - -``` -./zbox list --allocation $ALLOC --remotepath /myfiles -``` - -Response: - -``` - TYPE | NAME | PATH | SIZE | NUM BLOCKS |LOOKUP HASH | IS ENCRYPTED | DOWNLOADS PAYER -+------+-----------+--------------------+---------+------------+---------------- - f | audio.mp3 | /myfiles/audio.mp3 | 5992396 | 92 | 3cea39505cc30fb9f6fc5c6045284188feb14eac8ff3a19577701c4f6d973239 | NO | owner - -``` -Here we can see the `audio.mp3` file of size (5993286) bytes having 92 blocks.If we want to download a certain number of blocks for the `audio.mp3` file we can use the `--endblock` or `--startblock` flag with `./zbox download` command. Other flags for download can be viewed using `./zbox download --help` - -``` -Flags: - - -b, --blockspermarker int pass this option to download multiple blocks per marker (default 10) - -e, --endblock int pass this option to download till specific block number - -h, --help help for download - --localpath string Local path of file to download - --remotepath string Remote path to download - -s, --startblock int Pass this option to download from specific block number -``` - -For only downloading three blocks of `audio.mp3` file, we specify `--startblock` and`--endblock` with integer value of 1 and 3. `--blockspermarker` flag can also be specified to download multiple blocks at a time(default is 10). - -Sample command for downloading till 3rd block of the `audio.mp3` file would be: - -``` -./zbox download --localpath /root --remotepath /myfiles/audio.mp3 --allocation $ALLOC --startblock 1 --endblock 3 -``` +#### Sign data -Response: +`sign-data` uses the information from your wallet to sign the input data string -``` - 393216 / 2996198 [====================>-----------------------------------------------------------------------------------------------------------------------------------------] 13.12% 1s -Status completed callback. Type = audio/mpeg. Name = audio.mp3 +| Parameter | Required | Description | default | Valid values | +| --------- | -------- | -------------- | ------- | ------------ | +| data | yes | string to sign | | string | +```sh +./zbox sign-data "data to sign" +Signature : 9432ab2ee602062afaf48c4016b373a65db48a8546a81c09dead40e54966399e ``` -As we can see, the downloaded file size(393216) is less than the original(2996198), which means zbox has downloaded some blocks of the file. - - - - - - - - ### Lock and Unlock Tokens #### Challenge pool information @@ -1700,12 +1808,14 @@ Use `collect-reward` to transfer reward tokens from a stake pool in which you ha invested to your wallet. You earn rewards for: + Blobbers - File space used by allocation owners and associates. - A min lock demand for each allocation. - Block rewards. Each block a reward gets paid out to blobber stakeholders in the form of a random lottery. - Validators + +Validators - Payment for validating blobber challenge responses. The stake pool keeps an account for all stakeholders to maintain accrued rewards. @@ -1713,10 +1823,11 @@ These rewards can be accessed using this `collect-reward` command. | Parameter | Required | Description | default | Valid values | | ------------- | -------- | -------------------- | ------- | ------------ | -| provider_type | no | blobber or validator | blobber | string | +| provider_type | no | blobber or validator | blobber | "blobber" \| "validator" | +| provider_id | no | id of blobber or validator | | string | ```bash -./zbox colect-reward --provider_type blobber +./zbox collect-reward --provider_type blobber ``` #### Read pool info @@ -1770,11 +1881,10 @@ Unlocked tokens get returned to the original wallet balance. #### Storage SC configurations -Show storage SC configuration. +Use `sc-config` to show storage SC configuration. | Parameter | Required | Description | default | Valid values | | ---------- | -------- | --------------------------- | ------- | ------------ | -| allocation | yes | allocation id | | string | | json | no | print result in json format | false | boolean |
@@ -1784,18 +1894,27 @@ Show storage SC configuration.
-``` +```sh ./zbox sc-config ``` +```sh +{"blobber_slash":"0.1","block_reward.block_reward":"0.0063","block_reward.gamma.a":"10","block_reward.gamma.alpha":"0.2","block_reward.gamma.b":"9","block_reward.qualifying_stake":"1","block_reward.zeta.i":"1","block_reward.zeta.k":"0.9","block_reward.zeta.mu":"0.2","cancellation_charge":"0.2","challenge_enabled":"true","challenge_generation_gap":"1","cost.add_blobber":"266","cost.add_free_storage_assigner":"124","cost.add_validator":"348","cost.blobber_health_check":"97","cost.cancel_allocation":"1163","cost.challenge_response":"728","cost.collect_reward":"181","cost.commit_connection":"743","cost.commit_settings_changes":"56","cost.finalize_allocation":"1091","cost.free_allocation_request":"2132","cost.generate_challenge":"600","cost.kill_blobber":"651","cost.kill_validator":"277","cost.new_allocation_request":"1919","cost.pay_blobber_block_rewards":"100","cost.read_pool_lock":"170","cost.read_pool_unlock":"104","cost.read_redeem":"664","cost.shutdown_blobber":"597","cost.shutdown_validator":"227","cost.stake_pool_lock":"187","cost.stake_pool_unlock":"119","cost.update_allocation_request":"2692","cost.update_blobber_settings":"338","cost.update_settings":"143","cost.update_validator_settings":"247","cost.write_pool_lock":"186","free_allocation_settings.data_shards":"4","free_allocation_settings.parity_shards":"2","free_allocation_settings.read_pool_fraction":"0","free_allocation_settings.read_price_range.max":"0","free_allocation_settings.read_price_range.min":"0","free_allocation_settings.size":"2147483648","free_allocation_settings.write_price_range.max":"0.025","free_allocation_settings.write_price_range.min":"0","health_check_period":"1h30m0s","max_blobber_select_for_challenge":"5","max_blobbers_per_allocation":"30","max_challenge_completion_rounds":"1200","max_charge":"0.5","max_delegates":"100","max_file_size":"549755813888","max_individual_free_allocation":"1","max_read_price":"0","max_stake":"2e+06","max_total_free_allocation":"1e+07","max_write_price":"0.025","min_alloc_size":"1073741824","min_blobber_capacity":"10737418240","min_stake":"1","min_stake_per_delegate":"10","min_write_price":"0.001","num_validators_rewarded":"10","owner_id":"1746b06bb09f55ee01b33b5e2e055d6cc7a900cb57c0a3a5eaabb8a0e7745802","readpool.min_lock":"0","stakepool.kill_slash":"0.5","stakepool.min_lock_period":"0s","time_unit":"8760h0m0s","validator_reward":"0.025","validators_per_challenge":"2","writepool.min_lock":"0.1"} +``` + #### Stake pool info Use `sp-info` to get your stake pool information and settings. | Parameter | Required | Description | default | Valid values | | ---------- | -------- | --------------------------- | -------------- | ------------ | -| blobber_id | | id of blobber | current client | string | +| blobber_id | | id of blobber | | string | +| authorizer_id | | id of authorizer | | string | +| miner_id | | id of miner | | string | +| sharder_id | | id of sharder | | string | +| validator_id | | id of validator | | string | | json | no | print result in json format | false | boolean | + \* - one of the above ids is required
sp-info @@ -1807,6 +1926,29 @@ Use `sp-info` to get your stake pool information and settings. ``` ./zbox sp-info --blobber_id ``` +Sample Response : + +```sh +pool id: 98f14362f075caf467653044cf046eb9e8a5dfee88dc8b78cad1891748245003 +balance: 12.000 ZCN +total stake: 12.000 ZCN +unclaimed rewards: 0 SAS +total rewards: 0 SAS +delegate_pools: +- id: ba6ab426644f2b326b1da3dd426229fed19eed85387deaf72e05cb0fa4c5abbb + balance: 12.000 ZCN + delegate_id: ba6ab426644f2b326b1da3dd426229fed19eed85387deaf72e05cb0fa4c5abbb + unclaimed reward: 0 SAS + total_reward: 0 SAS + total_penalty: 0 SAS + status: active + round_created: 2074 + unstake: false + staked_at: 2024-04-27 16:29:47 +0200 EET +settings: + delegate_wallet: fdaa2b74e666a3f609ea714a649d44edd9c46ff468e094b797e1811c533d0b2b + num_delegates: 50 +``` #### Lock tokens into stake pool @@ -1822,6 +1964,9 @@ to stake pool later. | ------------ | -------- | --------------- | ------- | ------------ | | blobber_id | | id of blobber | n/a | string | | validator_id | | id of validator | n/a | string | +| miner_id | | id of miner | n/a | string | +| sharder_id | | id of blobber | n/a | string | +| authorizer_id| | id of authorizer| n/a | string | | fee | no | transaction fee | 0 | float | | tokens | yes | tokens to lock | | float | @@ -1846,15 +1991,16 @@ To stake tokens for validators: #### Unlock tokens from stake pool -Unlock a stake pool by pool owner. If the stake pool cannot be unlocked as -it would leave insufficient funds for opened offers, then `sp-unlock` tags -the stake pool to be unlocked later. This tag prevents the stake pool affecting +Unlock a stake pool by pool owner. This tag prevents the stake pool affecting blobber allocation for any new allocations. | Parameter | Required | Description | default | Valid values | | ------------ | -------- | --------------- | ------- | ------------ | | blobber_id | | id of blobber | n/a | string | | validator_id | | id of validator | n/a | string | +| miner_id | | id of miner | n/a | string | +| sharder_id | | id of blobber | n/a | string | +| authorizer_id| | id of authorizer| n/a | string | | fee | no | transaction fee | 0 | float |
@@ -1876,12 +2022,18 @@ To unstake validator tokens: ./zbox sp-unlock --validator_id --pool_id ``` +Same for the other providers. + #### Stake pools info of user Get information about all stake pools of current user. | Parameter | Required | Description | default | Valid values | | --------- | -------- | --------------------------- | ------- | ------------ | +| all | no | get all the pools | false | boolean | +| client_id | no | client_id of the user | | string | +| limit | no | limit the number of records returned | 20 | int | +| offset | no | skip the number of rows before beginning | | int | | json | no | print result in json format | false | boolean |
@@ -1895,32 +2047,20 @@ Get information about all stake pools of current user. ./zbox sp-user-info ``` -#### Write pool info - -Write pool information. Use allocation id to filter results to a singe allocation. - -| Parameter | Required | Description | default | Valid values | -| ------------- | -------- | --------------------------- | ------- | ------------ | -| allocation id | no | allocation id | | string | -| json | no | print result in json format | false | boolean | - -
- wp-info - -![image](https://user-images.githubusercontent.com/6240686/124603444-d9ab2c80-de61-11eb-82f2-900d540ba63f.png) - -
- -For all write pools. - -``` -./zbox wp-info -``` - -Filtering by allocation. +Sample Response : ``` -./zbox wp-info --allocation +- blobber_id: 98f14362f075caf467653044cf046eb9e8a5dfee88dc8b78cad1891748245003 + - id: ba6ab426644f2b326b1da3dd426229fed19eed85387deaf72e05cb0fa4c5abbb + balance: 12.000 ZCN + delegate_id: ba6ab426644f2b326b1da3dd426229fed19eed85387deaf72e05cb0fa4c5abbb + unclaimed reward: 0 SAS + total rewards: 0 SAS + total penalty: 0 SAS + status: active + round_created: 2074 + unstake: false + staked_at: 2024-04-27 16:29:47 +0200 EET ``` #### Lock tokens into write pool @@ -1933,24 +2073,22 @@ All tokens will be divided between allocation blobbers depending on their write - If the user does not have a pre-existing read pool, then the smart-contract creates one. -Anyone can lock tokens with a write pool attached an allocation. These tokens can +Anyone can lock tokens with a write pool attached to an allocation. These tokens can be used to pay for the allocation updates and min lock demand as needed. Any tokens moved into the challenge pool to underwrite blobbers' min lock demands return to the -allocation's owner on closing the allocation. +allocation's owner on closing the allocation either by cancelation or expiry. | Parameter | Required | Description | default | Valid values | | ------------- | -------- | --------------------------------- | ------- | ------------ | -| allocation id | no | allocation id | | string | -| blobber | no | blobber id | | string | -| duration | yes | duration for which to lock tokens | | duration | +| allocation | no | allocation id | | string | | fee | no | transaction fee | 0 | float | | tokens | yes | number of tokens to lock | | float |
- rp-lock with a specific blobber + wp-lock with a specific blobber ```shell -./zbox rp-lock --allocation --duration 40m --tokens 1 --blobber f65af5d64000c7cd2883f4910eb69086f9d6e6635c744e62afcfab58b938ee25 +./zbox wp-lock --allocation --tokens 1 ``` ![image](https://user-images.githubusercontent.com/6240686/123988183-b4c93c00-d9bf-11eb-825c-9a5849fedbbf.png) @@ -1958,22 +2096,16 @@ allocation's owner on closing the allocation.
- rp-lock spread across all blobbers - -Tokens are spread between the blobber pools weighted by -each blobber's Terms.ReadPrice. + wp-lock spread across all blobbers ```shell -./zbox rp-lock --allocation --duration 40m --tokens 1 +./zbox wp-lock --allocation --tokens 1 ``` ![image](https://user-images.githubusercontent.com/6240686/123979735-e5f23e00-d9b8-11eb-8232-339a4a3374d0.png)
-``` -./zbox wp-lock --allocation --duration 40m --tokens 1 -``` #### Unlock tokens from write pool @@ -1982,17 +2114,18 @@ An expired write pool, associated with an allocation, can be locked until alloca | Parameter | Required | Description | default | Valid values | | --------- | -------- | --------------- | ------- | ------------ | +| allocation | no | allocation id | | string | | fee | no | transaction fee | 0 | float |
- rp-unlock + wp-unlock ![image](https://user-images.githubusercontent.com/6240686/123980742-b09a2000-d9b9-11eb-8987-c18ff90ee705.png)
``` -./zbox wp-unlock +./zbox wp-unlock --allocation ``` #### Download cost @@ -2004,6 +2137,7 @@ owner, collaborator, or using an auth ticket to determine the download cost of t | ---------- | -------- | ----------------------------------------- | ------- | ------------ | | allocation | yes | allocation id | | string | | authticket | no | auth ticket to use if not the owner | | string | +| blocks-per-marker | no | blocks signed per Read Marker | 10 | int | | lookuphash | no | hash of remote file, use with auth ticket | | string | | remotepath | no | file of which to get stats, use if owner | | string | @@ -2013,6 +2147,7 @@ owner, collaborator, or using an auth ticket to determine the download cost of t ![image](https://user-images.githubusercontent.com/6240686/124497750-41ef0500-ddb3-11eb-99ea-115a4e234eda.png)
+ Command: ``` ./zbox get-download-cost --allocation --remotepath /path/file.ext diff --git a/cmd/add.go b/cmd/add.go index 6bf51887..917d1e55 100644 --- a/cmd/add.go +++ b/cmd/add.go @@ -12,6 +12,7 @@ var addCmd = &cobra.Command{ Short: "Adds free storage assigner", Long: "Adds free storage assigner", Args: cobra.MinimumNArgs(0), + Hidden: true, Run: func(cmd *cobra.Command, args []string) { var err error var flags = cmd.Flags() diff --git a/cmd/fileRefs.go b/cmd/fileRefs.go index 61d4f3b4..d63b19f0 100644 --- a/cmd/fileRefs.go +++ b/cmd/fileRefs.go @@ -29,6 +29,7 @@ var fileRefsCmd = &cobra.Command{ Short: "get list of recently added refs", Long: `get list of recently added refs`, Args: cobra.MinimumNArgs(0), + Hidden: true, Run: func(cmd *cobra.Command, args []string) { page, err := cmd.Flags().GetUint("page") checkError(err) diff --git a/cmd/getmpt.go b/cmd/getmpt.go index 5328325f..1799e442 100644 --- a/cmd/getmpt.go +++ b/cmd/getmpt.go @@ -16,6 +16,7 @@ var getMptKeyCommand = &cobra.Command{ Short: "Directly view blockchain data", Long: `Directly view blockchain data from MPT key`, Args: cobra.MinimumNArgs(0), + Hidden: true, Run: func(cmd *cobra.Command, args []string) { if cmd.Flags().Changed("key") == false { log.Fatal("Required Mpt key missing\n") diff --git a/cmd/newallocation.go b/cmd/newallocation.go index 10e24d33..c640d27f 100755 --- a/cmd/newallocation.go +++ b/cmd/newallocation.go @@ -23,6 +23,7 @@ var ( size *int64 allocationFileName *string preferred_blobbers []string + blobber_auth_tickets []string ) func getPriceRange(val string) (pr sdk.PriceRange, err error) { @@ -119,6 +120,17 @@ var newallocationCmd = &cobra.Command{ } } + if flags.Changed("blobber_auth_tickets") { + b, err := flags.GetString("blobber_auth_tickets") + if err != nil { + log.Fatal("invalid read_price value: ", err) + } + blobber_auth_tickets = strings.Split(b, ",") + for i, id := range blobber_auth_tickets { + blobber_auth_tickets[i] = strings.TrimSpace(id) + } + } + if flags.Changed("read_price") { rps, err := flags.GetString("read_price") if err != nil { @@ -171,6 +183,7 @@ var newallocationCmd = &cobra.Command{ } thirdPartyExtendable, _ := flags.GetBool("third_party_extendable") + force, _ := flags.GetBool("force") // Read the file options flags var fileOptionParams sdk.FileOptionsParameters @@ -230,17 +243,19 @@ var newallocationCmd = &cobra.Command{ ParityShards: *parityshards, Size: *size, ReadPrice: sdk.PriceRange{ - Min: uint64(readPrice.Min), - Max: uint64(readPrice.Max), + Min: readPrice.Min, + Max: readPrice.Max, }, WritePrice: sdk.PriceRange{ - Min: uint64(writePrice.Min), - Max: uint64(writePrice.Max), + Min: writePrice.Min, + Max: writePrice.Max, }, Lock: uint64(lock), BlobberIds: preferred_blobbers, + BlobberAuthTickets: blobber_auth_tickets, FileOptionsParams: &fileOptionParams, ThirdPartyExtendable: thirdPartyExtendable, + Force: force, } allocationID, _, _, err = sdk.CreateAllocationWith(options) if err != nil { @@ -258,7 +273,7 @@ var newallocationCmd = &cobra.Command{ } allocationID, _, _, err = sdk.CreateAllocationForOwner(owner, ownerPublicKey, *datashards, *parityshards, - *size, readPrice, writePrice, lock, preferred_blobbers, thirdPartyExtendable, &fileOptionParams) + *size, readPrice, writePrice, lock, preferred_blobbers, blobber_auth_tickets, thirdPartyExtendable, force, &fileOptionParams) if err != nil { log.Fatal("Error creating allocation: ", err) } @@ -327,7 +342,9 @@ func init() { newallocationCmd.Flags().String("name", "", "allocation name") newallocationCmd.Flags().String("preferred_blobbers", "", "coma seperated list of preferred blobbers") + newallocationCmd.Flags().String("blobber_auth_tickets", "", "coma seperated list of blobber auth tickets") + newallocationCmd.Flags().Bool("force", false, "(default false) force to get blobbers even if required number of blobbers are not available (should be passed true in case of restricted blobbers)") newallocationCmd.Flags().Bool("third_party_extendable", false, "(default false) specify if the allocation can be extended by users other than the owner") newallocationCmd.Flags().Bool("forbid_upload", false, "(default false) specify if users cannot upload to this allocation") newallocationCmd.Flags().Bool("forbid_delete", false, "(default false) specify if the users cannot delete objects from this allocation") diff --git a/cmd/rolllback.go b/cmd/rolllback.go index 6d32c414..b620757b 100644 --- a/cmd/rolllback.go +++ b/cmd/rolllback.go @@ -14,6 +14,7 @@ var rollbackCmd = &cobra.Command{ Short: "rollback file to previous version", Long: `rollback file to previous version`, Args: cobra.MinimumNArgs(0), + Hidden: true, Run: func(cmd *cobra.Command, args []string) { fflags := cmd.Flags() if !fflags.Changed("allocation") { diff --git a/cmd/share.go b/cmd/share.go index 537ee427..a4f6f279 100644 --- a/cmd/share.go +++ b/cmd/share.go @@ -124,7 +124,7 @@ func init() { rootCmd.AddCommand(shareCmd) shareCmd.PersistentFlags().String("allocation", "", "Allocation ID") shareCmd.PersistentFlags().String("remotepath", "", "Remote path to share") - shareCmd.PersistentFlags().String("clientid", "", "ClientID of the user to share with. Leave blank for public share") + shareCmd.PersistentFlags().String("clientid", "", "ClientID of the user to share with (or revoke share from). Leave blank for public share (or revoking public access).") shareCmd.PersistentFlags().String("encryptionpublickey", "", "Encryption public key of the client you want to share with. Can be retrieved by the getwallet command") shareCmd.PersistentFlags().Int64("expiration-seconds", 0, "Authticket will expire when the seconds specified have elapsed after the instant of its creation") shareCmd.PersistentFlags().String("available-after", "", "Timelock for private file that makes the file available for download at certain time. 4 input formats are supported: +1h30m, +30, 1647858200 and 2022-03-21 10:21:38. Default value is current local time.") diff --git a/cmd/stakepool.go b/cmd/stakepool.go index e43305d9..3ed00bc0 100644 --- a/cmd/stakepool.go +++ b/cmd/stakepool.go @@ -36,8 +36,8 @@ func printStakePoolInfo(info *sdk.StakePoolInfo) { // settings fmt.Println("settings:") fmt.Println(" delegate_wallet: ", info.Settings.DelegateWallet) - fmt.Println(" min_stake: ", info.Settings.MinStake.String()) - fmt.Println(" max_stake: ", info.Settings.MaxStake.String()) + //fmt.Println(" min_stake: ", info.Settings.MinStake.String()) + //fmt.Println(" max_stake: ", info.Settings.MaxStake.String()) fmt.Println(" num_delegates: ", info.Settings.NumDelegates) } diff --git a/cmd/storage.go b/cmd/storage.go index 2f398907..7da2cf4a 100644 --- a/cmd/storage.go +++ b/cmd/storage.go @@ -66,13 +66,16 @@ var lsBlobers = &cobra.Command{ Run: func(cmd *cobra.Command, args []string) { doJSON, _ := cmd.Flags().GetBool("json") doAll, _ := cmd.Flags().GetBool("all") - + isStakable, err := cmd.Flags().GetBool("stakable") + if err != nil { + log.Fatalf("err parsing in stakable flag: %v", err) + } // set is_active=true to get only active blobbers isActive := true if doAll { isActive = false } - var list, err = sdk.GetBlobbers(isActive) + list, err := sdk.GetBlobbers(isActive, isStakable) if err != nil { log.Fatalf("Failed to get storage SC configurations: %v", err) } @@ -133,14 +136,15 @@ var blobberInfoCmd = &cobra.Command{ fmt.Println("capacity_used: ", blob.Allocated) fmt.Println("total_stake: ", blob.TotalStake) fmt.Println("not_available: ", blob.NotAvailable) + fmt.Println("is_restricted: ", blob.IsRestricted) fmt.Println("terms:") fmt.Println(" read_price: ", blob.Terms.ReadPrice, "/ GB") fmt.Println(" write_price: ", blob.Terms.WritePrice, "/ GB") fmt.Println(" max_offer_duration:", blob.Terms.MaxOfferDuration) fmt.Println("settings:") fmt.Println(" delegate_wallet:", blob.StakePoolSettings.DelegateWallet) - fmt.Println(" min_stake: ", blob.StakePoolSettings.MinStake) - fmt.Println(" max_stake: ", blob.StakePoolSettings.MaxStake) + //fmt.Println(" min_stake: ", blob.StakePoolSettings.MinStake) + //fmt.Println(" max_stake: ", blob.StakePoolSettings.MaxStake) fmt.Println(" num_delegates: ", blob.StakePoolSettings.NumDelegates) fmt.Println(" service_charge: ", blob.StakePoolSettings.ServiceCharge*100, "%") }, @@ -221,32 +225,6 @@ var blobberUpdateCmd = &cobra.Command{ stakePoolSettings := &blockchain.UpdateStakePoolSettings{} var stakePoolSettingChanged bool - if flags.Changed("min_stake") { - var minStake float64 - if minStake, err = flags.GetFloat64("min_stake"); err != nil { - log.Fatal(err) - } - stake, err := common.ToBalance(minStake) - if err != nil { - log.Fatal(err) - } - stakePoolSettings.MinStake = &stake - stakePoolSettingChanged = true - } - - if flags.Changed("max_stake") { - var maxStake float64 - if maxStake, err = flags.GetFloat64("max_stake"); err != nil { - log.Fatal(err) - } - stake, err := common.ToBalance(maxStake) - if err != nil { - log.Fatal(err) - } - stakePoolSettings.MaxStake = &stake - stakePoolSettingChanged = true - } - if flags.Changed("num_delegates") { var nd int if nd, err = flags.GetInt("num_delegates"); err != nil { @@ -274,11 +252,26 @@ var blobberUpdateCmd = &cobra.Command{ } if flags.Changed("not_available") { + var na bool + if na, err = flags.GetBool("not_available"); err != nil { + log.Fatal(err) + } + if !na { + na = false + } + updateBlobber.NotAvailable = &na + } + + if flags.Changed("is_restricted") { var ia bool - if ia, err = flags.GetBool("not_available"); err != nil { + // Check if the flag is set to true + if ia, err = flags.GetBool("is_restricted"); err != nil { log.Fatal(err) } - updateBlobber.NotAvailable = &ia + if !ia { + ia = false + } + updateBlobber.IsRestricted = &ia } if termsChanged { @@ -301,16 +294,17 @@ var resetBlobberStatsCmd = &cobra.Command{ Short: "Reset blobber stats", Long: `Reset blobber stats`, Args: cobra.MinimumNArgs(0), + Hidden: true, Run: func(cmd *cobra.Command, args []string) { var ( flags = cmd.Flags() - blobberID string + blobberID string prevAllocated int64 prevSavedData int64 - newAllocated int64 - newSavedData int64 - err error + newAllocated int64 + newSavedData int64 + err error ) if !flags.Changed("blobber_id") { @@ -348,12 +342,12 @@ var resetBlobberStatsCmd = &cobra.Command{ log.Fatal("error in 'new_saved_data' flag: ", err) } - resetBlobberStatsDto := &sdk.ResetBlobberStatsDto { - BlobberID: blobberID, + resetBlobberStatsDto := &sdk.ResetBlobberStatsDto{ + BlobberID: blobberID, PrevAllocated: prevAllocated, PrevSavedData: prevSavedData, - NewAllocated: newAllocated, - NewSavedData: newSavedData, + NewAllocated: newAllocated, + NewSavedData: newSavedData, } fmt.Println(*resetBlobberStatsDto) @@ -375,6 +369,7 @@ func init() { scConfig.Flags().Bool("json", false, "(default false) pass this option to print response as json data") lsBlobers.Flags().Bool("json", false, "(default false) pass this option to print response as json data") lsBlobers.Flags().Bool("all", false, "(default false) shows active and non active list of blobbers on ls-blobbers") + lsBlobers.Flags().Bool("stakable", false, "(default false) gets only stakable list of blobbers if set to true") blobberInfoCmd.Flags().String("blobber_id", "", "blobber ID, required") blobberInfoCmd.Flags().Bool("json", false, @@ -392,6 +387,7 @@ func init() { buf.Int("num_delegates", 0, "update num_delegates, optional") buf.Float64("service_charge", 0.0, "update service_charge, optional") buf.Bool("not_available", true, "(default false) set blobber's availability for new allocations") + buf.Bool("is_restricted", true, "(default false) set is_restricted") buf.String("url", "", "update the url of the blobber, optional") blobberUpdateCmd.MarkFlagRequired("blobber_id") diff --git a/cmd/stream.go b/cmd/stream.go index 54743ee4..2212d74f 100644 --- a/cmd/stream.go +++ b/cmd/stream.go @@ -16,6 +16,7 @@ var streamCmd = &cobra.Command{ Short: "capture video and audio streaming form local devices, and upload", Long: "capture video and audio streaming form local devices, and upload", Args: cobra.MinimumNArgs(0), + Hidden: true, Run: func(cmd *cobra.Command, args []string) { fflags := cmd.Flags() // fflags is a *flag.FlagSet if !fflags.Changed("allocation") { // check if the flag "path" is set @@ -114,7 +115,6 @@ func init() { streamCmd.PersistentFlags().String("allocation", "", "Allocation ID") streamCmd.PersistentFlags().String("remotepath", "", "Remote path to upload") streamCmd.PersistentFlags().String("localpath", "", "Local path of file to upload") - streamCmd.PersistentFlags().String("thumbnailpath", "", "Local thumbnail path of file to upload") streamCmd.PersistentFlags().String("attr-who-pays-for-reads", "owner", "Who pays for reads: owner or 3rd_party") streamCmd.Flags().Bool("encrypt", false, "(default false) pass this option to encrypt and upload the file") diff --git a/cmd/sync.go b/cmd/sync.go index 3089f9e3..6c4bf64c 100644 --- a/cmd/sync.go +++ b/cmd/sync.go @@ -3,6 +3,7 @@ package cmd import ( "fmt" "os" + "path" "runtime" "strings" "sync" @@ -101,7 +102,7 @@ func startMultiUploadUpdate(allocationObj *sdk.Allocation, argsSlice []chunkedUp return err } - mimeType, err := zboxutil.GetFileContentType(fileReader) + mimeType, err := zboxutil.GetFileContentType(path.Ext(args.localPath), fileReader) if err != nil { return err } @@ -151,6 +152,7 @@ var syncCmd = &cobra.Command{ Short: "Sync files to/from blobbers", Long: `Sync all files to/from blobbers from/to a localpath`, Args: cobra.MinimumNArgs(0), + Hidden: true, Run: func(cmd *cobra.Command, args []string) { fflags := cmd.Flags() // fflags is a *flag.FlagSet if fflags.Changed("localpath") == false { @@ -332,6 +334,7 @@ var getDiffCmd = &cobra.Command{ Short: "Get difference of local and allocation root", Long: `Get difference of local and allocation root`, Args: cobra.MinimumNArgs(0), + Hidden: true, Run: func(cmd *cobra.Command, args []string) { fflags := cmd.Flags() // fflags is a *flag.FlagSet diff --git a/cmd/updateallocation.go b/cmd/updateallocation.go index 1e777acc..af49b350 100755 --- a/cmd/updateallocation.go +++ b/cmd/updateallocation.go @@ -26,12 +26,15 @@ var updateAllocationCmd = &cobra.Command{ log.Fatal("invalid 'allocation_id' flag: ", err) } - var addBlobberId, removeBlobberId string + var addBlobberId, addBlobberAuthTicket, removeBlobberId string if flags.Changed("add_blobber") { addBlobberId, err = flags.GetString("add_blobber") if err != nil { log.Fatal("invalid 'add_blobber' flag: ", err) } + + addBlobberAuthTicket, _ = flags.GetString("add_blobber_auth_ticket") + if flags.Changed("remove_blobber") { removeBlobberId, err = flags.GetString("remove_blobber") if err != nil { @@ -130,6 +133,7 @@ var updateAllocationCmd = &cobra.Command{ extend, lock, addBlobberId, + addBlobberAuthTicket, removeBlobberId, setThirdPartyExtendable, &fileOptionParams, @@ -151,6 +155,7 @@ var updateAllocationCmd = &cobra.Command{ allocID, lock, addBlobberId, + addBlobberAuthTicket, removeBlobberId, setThirdPartyExtendable, &fileOptionParams, @@ -169,6 +174,8 @@ func init() { "Allocation ID") updateAllocationCmd.PersistentFlags().String("add_blobber", "", "ID of blobber to add to the allocation") + updateAllocationCmd.PersistentFlags().String("add_blobber_auth_ticket", "", + "Auth ticket of blobber to add to the allocation") updateAllocationCmd.PersistentFlags().String("remove_blobber", "", "ID of blobber to remove from the allocation") updateAllocationCmd.PersistentFlags().Float64("lock", 0.0, diff --git a/cmd/validator.go b/cmd/validator.go index 958eb4da..0d4607e4 100644 --- a/cmd/validator.go +++ b/cmd/validator.go @@ -40,9 +40,12 @@ var lsValidators = &cobra.Command{ Args: cobra.MinimumNArgs(0), Run: func(cmd *cobra.Command, args []string) { doJSON, _ := cmd.Flags().GetBool("json") + stakable, err := cmd.Flags().GetBool("stakable") + if err != nil { + log.Fatalf("error parsing stakable flag: %v", err) + } - var list, err = sdk.GetValidators() - + list, err := sdk.GetValidators(stakable) if err != nil { log.Fatalf("Failed to get storage SC configurations: %v", err) } @@ -193,4 +196,5 @@ func init() { validatorUpdateCmd.MarkFlagRequired("validator_id") lsValidators.Flags().Bool("json", false, "(default false) pass this flag to get response as json object") + lsValidators.Flags().Bool("stakable", false, "(default false) Gets only validators that can be staked if set to true") } diff --git a/go.mod b/go.mod index dd037d31..34b103a8 100644 --- a/go.mod +++ b/go.mod @@ -4,7 +4,7 @@ go 1.21 require ( github.com/0chain/errors v1.0.3 - github.com/0chain/gosdk v1.13.13 + github.com/0chain/gosdk v1.14.0-RC7 github.com/icza/bitio v1.1.0 github.com/olekukonko/tablewriter v0.0.5 github.com/spf13/cobra v1.6.0 @@ -30,6 +30,7 @@ require ( github.com/gorilla/websocket v1.5.0 // indirect github.com/h2non/filetype v1.1.4-0.20231228185113-6469358c2bcb // indirect github.com/hashicorp/golang-lru v0.5.5-0.20210104140557-80c98217689d // indirect + github.com/hashicorp/golang-lru/v2 v2.0.1 // indirect github.com/hashicorp/hcl v1.0.0 // indirect github.com/herumi/bls-go-binary v1.31.0 // indirect github.com/hitenjain14/fasthttp v0.0.0-20240229173600-722723e15e17 // indirect diff --git a/go.sum b/go.sum index 7156a81f..8dc372d6 100644 --- a/go.sum +++ b/go.sum @@ -40,8 +40,12 @@ github.com/0chain/common v0.0.6-0.20230127095721-8df4d1d72565 h1:z+DtCR8mBsjPnEs github.com/0chain/common v0.0.6-0.20230127095721-8df4d1d72565/go.mod h1:UyDC8Qyl5z9lGkCnf9RHJPMektnFX8XtCJZHXCCVj8E= github.com/0chain/errors v1.0.3 h1:QQZPFxTfnMcRdt32DXbzRQIfGWmBsKoEdszKQDb0rRM= github.com/0chain/errors v1.0.3/go.mod h1:xymD6nVgrbgttWwkpSCfLLEJbFO6iHGQwk/yeSuYkIc= -github.com/0chain/gosdk v1.13.13 h1:qgxKZkHZdshsri2i7lqE5Z2acuIsZZqgQKkvaM86g5Y= -github.com/0chain/gosdk v1.13.13/go.mod h1:tgAiVAuIy+Vs1tGfKCPEuuWWARwNQBEw32y950LrqrU= +github.com/0chain/gosdk v1.14.0-RC2 h1:OIpjj7mDKxVqJVlxJKm8/fPzdJRN5roXO7/plBCtmTg= +github.com/0chain/gosdk v1.14.0-RC2/go.mod h1:tgAiVAuIy+Vs1tGfKCPEuuWWARwNQBEw32y950LrqrU= +github.com/0chain/gosdk v1.14.0-RC5 h1:3/K++2jreD9jaZDAYkGXQ41ywANUq6KlIO9vjVZ3d3A= +github.com/0chain/gosdk v1.14.0-RC5/go.mod h1:tgAiVAuIy+Vs1tGfKCPEuuWWARwNQBEw32y950LrqrU= +github.com/0chain/gosdk v1.14.0-RC7 h1:hrxt3IGqUBMlZp3QrT1Tx5hTilt1biAAdh9V1SJOTc0= +github.com/0chain/gosdk v1.14.0-RC7/go.mod h1:tgAiVAuIy+Vs1tGfKCPEuuWWARwNQBEw32y950LrqrU= github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= github.com/Luzifer/go-openssl/v3 v3.1.0 h1:QqKqo6kYXGGUsvtUoCpRZm8lHw+jDfhbzr36gVj+/gw= @@ -209,6 +213,8 @@ github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.5-0.20210104140557-80c98217689d h1:dg1dEPuWpEqDnvIw251EVy4zlP8gWbsGj4BsUKCRpYs= github.com/hashicorp/golang-lru v0.5.5-0.20210104140557-80c98217689d/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= +github.com/hashicorp/golang-lru/v2 v2.0.1 h1:5pv5N1lT1fjLg2VQ5KWc7kmucp2x/kvFOnxuVTqZ6x4= +github.com/hashicorp/golang-lru/v2 v2.0.1/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4= github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= github.com/herumi/bls-go-binary v1.31.0 h1:L1goQ2tMtGgpXCg5AwHAdJQpLs/pfnWWEc3Wog6OhmI= diff --git a/internal.md b/internal.md new file mode 100644 index 00000000..7344f31b --- /dev/null +++ b/internal.md @@ -0,0 +1,213 @@ +Contents : +[Sync](#sync) +[Get Differences](#get-differences) +[Stream](#stream) +[Get MPT](#get-mpt) +[Rollback Allocation](#rollback-allocation) +[Streaming](#streaming) + - [How it works:](#how-it-works) + - [Usage](#usage) +#### Sync + +`sync` command syncs all files from the local folder recursively to the remote. +Only the allocation's owner can successfully run `sync`. + +| Parameter | Required | Description | default | Valid values | +| ----------- | -------- | --------------------------------------------------------------------------------------------- | ------- | ------------ | +| allocation | yes | allocation id | | string | +| encryptpath | no | local directory path to be uploaded as encrypted | false | boolean | +| excludepath | no | paths to exclude from sync | | string array | +| localchache | no | local chache of remote snapshot. Used for comparsion with remote. After sync will be updated. | | string | +| localpath | yes | local directory to which to sync | | file path | +| uploadonly | no | only upload and update files | false | boolean | +| chunknumber | no | how many chunks should be uploaded in a http request | | int | +| remotepath | no | Remote dir path from where it sync | "/" | string | +| verifydownload | no | how many chunks should be uploaded in a http request | true | boolean | + +
+ sync + +![image](https://user-images.githubusercontent.com/6240686/127884376-a95c4f27-4b2a-4d9b-91b6-c7e7919f88bc.png) + +
+ +Example + +``` +./zbox sync --allocation 8695b9e7f986d4a447b64de020ba86f53b3b5e2c442abceb6cd65742702067dc --localpath /home/dung/Desktop/alloc --localcache /home/dung/Desktop/localcache.json +``` + +Response: + +``` + OPERATION | PATH ++-----------+----------------+ + Download | /1.txt + Download | /afolder/1.txt + Download | /d2.txt + + 4 / 4 [===========================================================================] 100.00% 0s +Status completed callback. Type = application/octet-stream. Name = 1.txt + 4 / 4 [===========================================================================] 100.00% 0s +Status completed callback. Type = application/octet-stream. Name = 1.txt + 7 / 7 [===========================================================================] 100.00% 0s +Status completed callback. Type = application/octet-stream. Name = d2.txt + +Sync Complete +Local cache saved. +``` + +It will sync your localpath with the remote and do all the required CRUD operations. + + +#### Get differences + +`get-diff` command returns the differences between the local files specified by `localpath` and the files stored +on the root remotepath of the allocation.`localcache` flag can also be specified to use the local cache of remote snapshot created during [Sync](#sync) for file comparison. + +| Parameter | Required | Description | default | Valid values | +| ----------- | -------- | --------------------------------------------- | ------- | ------------ | +| allocation | yes | allocation id | | string | +| excludepath | no | remote folder paths to exclude during syncing | | string array | +| localcache | no | local cache of remote snapshot | | string | +| localpath | yes | local directory to sync | | string | + +Example + +```sh +./zbox get-diff --allocation $ALLOC --localpath $local +``` + +Response: + +```sh +[{"operation":"Upload","path":"/file1.txt","type":"f","attributes":{}}, +{"operation":"Upload","path":"/file2.txt","type":"f","attributes":{}}, +{"operation":"Upload","path":"/file3.txt","type":"f","attributes":{}}, +{"operation":"Download","path":"/myfiles/file1.txt","type":"f","attributes":{}}, +{"operation":"Download","path":"/myfiles/file2.txt","type":"f","attributes":{}}] +``` + +#### Stream + +Use `stream` to capture video and audio streaming form local devices, and upload + +The user must be the owner of the allocation.You can request the file be encrypted before upload, and can send thumbnails with the file. + +| Parameter | Required | Description | Default | Valid values | +| ------------- | -------- | ------------------------------------------------------------ | ------- | ------------ | +| allocation | yes | allocation id, sender must be allocation owner | | string | +| encrypt | no | encrypt file before upload | false | boolean | +| localpath | yes | local path of segment files to download, generate and upload | | file path | +| remotepath | yes | remote path to upload file to, use to access file later | | string | +| thumbnailpath | no | local path of thumbnaSil | | file path | +| chunknumber | no | how many chunks should be uploaded in a http request | 1 | int | +| delay | no | set segment duration to seconds. | 5 | int | +| attr-who-pays-for-reads | no | Who pays for reads: owner or 3rd_party | owner | owner / 3rd_party| + +
+ stream + +![image](https://github.com/0chain/blobber/wiki/uml/usecase/live_upload_live.png) + +
+ + +#### Get MPT + +`get-mpt` is used to directly get blockchain data from the MPT key + +| Parameter | Required | Description | default | Valid values | +| --------- | -------- | -------------- | ------- | ------------ | +| key | yes | Key in MPT datastore | | string | + +Sample Command : +```sh +./zbox get-mpt --key {MPT_KEY} +``` + +Sample Response : +```sh + +``` + +#### Rollback Allocation + +`rollback` is used to directly get blockchain data from the MPT key + +| Parameter | Required | Description | default | Valid values | +| --------- | -------- | -------------- | ------- | ------------ | +| allocation | yes | Allocation ID | | string | + +Sample Command : +```sh +./zbox rollback --allocation $ALLOC +``` + +Sample Response : +```sh +Rollback successful +``` + +#### Streaming + +Video streaming with Zbox CLI can be implemented with players for different operating platforms(iOS, Android Mac).Zbox CLI does not have a player itself and use the the downloadFileByBlocks helper function to properly returns file-chunks with correct byte range. + +![streaming-android](https://user-images.githubusercontent.com/65766301/120052635-ce373b00-c043-11eb-94a5-a9711078ee54.png) + +##### How it works: + +When the user starts the video player (ExoPlayer for Android or AVPlayer for iOS), A ZChainDataSource starts chunked download and requests chunks of video from the buffer(a Middleman between streaming player and Zbox). + +After the arrival of the first chunk, the player starts requesting more chunks from the buffer, which requests the Zbox SDK. Zbox SDK, which is built using GO, makes use of the downloadFileByBlocks method to reliably download large files by chunking them into a sequence of parts that can be downloaded individually. Once the blocks are downloaded, they are read into input streams and added to the media source of the streaming player. + +The task of downloading files and writing them to buffer using Zbox SDK happens constantly, and If players request random bits of video, they are delivered instantly by a buffer. + +In a case, if the player didn't receive chunks (for example, it's still not downloaded), then the player switches to STALE state, and the video stream will pause. During the STALE state, a player tries to make multiple requests for chunks; if didn't receive a response, the video stream stops. + +##### Usage + +To understand how Zbox CLI provides downloading of files by blocks. Let's consider an allocation that has `audio. mp3 ` file stored on dStorage. Make sure the file has a large size(more than 64 kB(64000 bytes)) to download the file by blocks. The size and other attributes of the sample `audio. mp3` file can be viewed using + +``` +./zbox list --allocation $ALLOC --remotepath /myfiles +``` + +Response: + +``` + TYPE | NAME | PATH | SIZE | NUM BLOCKS |LOOKUP HASH | IS ENCRYPTED | DOWNLOADS PAYER ++------+-----------+--------------------+---------+------------+---------------- + f | audio.mp3 | /myfiles/audio.mp3 | 5992396 | 92 | 3cea39505cc30fb9f6fc5c6045284188feb14eac8ff3a19577701c4f6d973239 | NO | owner + +``` +Here we can see the `audio.mp3` file of size (5993286) bytes having 92 blocks.If we want to download a certain number of blocks for the `audio.mp3` file we can use the `--endblock` or `--startblock` flag with `./zbox download` command. Other flags for download can be viewed using `./zbox download --help` + +``` +Flags: + + -b, --blockspermarker int pass this option to download multiple blocks per marker (default 10) + -e, --endblock int pass this option to download till specific block number + -h, --help help for download + --localpath string Local path of file to download + --remotepath string Remote path to download + -s, --startblock int Pass this option to download from specific block number +``` + +For only downloading three blocks of `audio.mp3` file, we specify `--startblock` and`--endblock` with integer value of 1 and 3. `--blockspermarker` flag can also be specified to download multiple blocks at a time(default is 10). + +Sample command for downloading till 3rd block of the `audio.mp3` file would be: + +``` +./zbox download --localpath /root --remotepath /myfiles/audio.mp3 --allocation $ALLOC --startblock 1 --endblock 3 +``` + +Response: + +``` + 393216 / 2996198 [====================>-----------------------------------------------------------------------------------------------------------------------------------------] 13.12% 1s +Status completed callback. Type = audio/mpeg. Name = audio.mp3 + +``` + +As we can see, the downloaded file size(393216) is less than the original(2996198), which means zbox has downloaded some blocks of the file.