Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 20 additions & 9 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,26 +9,37 @@ jobs:
deploy_docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Install dependencies
uses: borales/actions-yarn@v3.0.0
- uses: actions/checkout@v3

- name: Install pnpm
uses: pnpm/action-setup@v2
with:
cmd: install # will run `yarn install` command
- name: Build docs
uses: borales/actions-yarn@v3.0.0
version: 7
run_install: true

- name: Setup Node.js
uses: actions/setup-node@v3
with:
cmd: docs:build -d docs_dist # will run `docs:build -d docs_dist` command
node-version: 18
cache: pnpm

- name: Build docs
run: |-
pnpm docs:build

- name: Setup git
run: |
git config --global lfs.allowincompletepush true

- name: Deploy
uses: peaceiris/actions-gh-pages@v3
with:
personal_token: ${{ secrets.PERSONAL_TOKEN }}
external_repository: taskiq-python/taskiq-python.github.io
publish_dir: ./docs_dist
user_name: 'github-actions[bot]'
user_email: 'github-actions[bot]@users.noreply.github.com'
user_name: "github-actions[bot]"
user_email: "github-actions[bot]@users.noreply.github.com"

deploy:
runs-on: ubuntu-latest
steps:
Expand Down
51 changes: 51 additions & 0 deletions docs/.vuepress/config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
import { defineUserConfig } from "vuepress";
import { searchProPlugin } from "vuepress-plugin-search-pro";
import { hopeTheme } from "vuepress-theme-hope";

export default defineUserConfig({
lang: "en-US",
title: "Taskiq",
description: "Distributed task queue with full async support",
head: [
[
"meta",
{
property: "og:image",
content: "https://taskiq-python.github.io/logo.svg",
},
],
],

theme: hopeTheme({
hostname: "https://taskiq-python.github.io",
logo: "/logo.svg",

repo: "taskiq-python/taskiq",
docsBranch: "master",
docsDir: "docs",

navbarAutoHide: "none",
sidebar: "structure",

pure: true,
backToTop: false,

plugins: {
copyCode: {
showInMobile: true,
},

mdEnhance: {
tabs: true,
mermaid: true,
},

sitemap: {
changefreq: "daily",
sitemapFilename: "sitemap.xml",
},
},
}),

plugins: [searchProPlugin({ indexContent: true })],
});
8 changes: 4 additions & 4 deletions docs/available-components/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
order: 1
dir:
order: 3
order: 3
---

# Available components

In this section, you can find a list of officially supported plugins for the taskiq.

* [Available brokers](./brokers.md)
* [Available result backends](./result-backends.md)
* [Available schedule sources](./scheduler-sources.md)
- [Available brokers](./brokers.md)
- [Available result backends](./result-backends.md)
- [Available schedule sources](./schedule-sources.md)
22 changes: 12 additions & 10 deletions docs/available-components/brokers.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,37 +10,38 @@ In this section we'll list officially supported brokers.

This is a special broker for local development. It uses the same functions to execute tasks,
but all tasks are executed locally in the current thread.
By default it uses `InMemoryResultBackend` but this can be overiden.

By default it uses `InMemoryResultBackend` but this can be overriden.

## ZeroMQBroker

This broker uses [ZMQ](https://zeromq.org/) to comunicate between worker and client processes.
This broker uses [ZMQ](https://zeromq.org/) to communicate between worker and client processes.
It's suitable for small projects with only ONE worker process, because of the ZMQ architecture.

It publishes messages on the local port. All worker processes are reading messages from this port.
If you run many worker processes, all tasks will be executed `N` times, where `N` is the total number of worker processes.

::: danger Be careful!
If you choose this type of broker, please run taskiq with `-w 1` parameter,
otherwise you may encounter undefined behaviour.
otherwise you may encounter undefined behavior.
:::


To run this broker please install the [pyzmq](https://pypi.org/project/pyzmq/) lib. Or you can taskiq with `zmq` extra.

::: tabs

@tab Only PyZMQ

```bash
pip install pyzmq
```

@tab Taskiq with ZMQ

```bash
pip install "taskiq[zmq]"
```
:::

:::

## Async shared broker and shared tasks

Expand All @@ -56,26 +57,28 @@ def my_task() -> bool:
```

To kiq this task you have to options:
* Explicitly define broker using kicker for this kiq;
* Add default broker for all shared tasks.

- Explicitly define broker using kicker for this kiq;
- Add default broker for all shared tasks.

::: tabs

@tab Defining default broker

```python
from taskiq.brokers.shared_broker import async_shared_broker

async_shared_broker.default_broker(broker)
```

@tab using kicker

```python
await my_task.kicker().with_broker(broker).kiq()
```

:::


## AioPikaBroker (for RabbitMQ)

This broker is not part of the core taskiq lib. You can install it as a separate package [taskiq-aio-pika](https://pypi.org/project/taskiq-aio-pika/).
Expand All @@ -86,7 +89,6 @@ pip install taskiq-aio-pika

You can read more about parameters and abilities of this broker in [README.md](https://github.com/taskiq-python/taskiq-aio-pika).


## Redis broker

This broker is not part of the core taskiq lib. You can install it as a separate package [taskiq-redis](https://pypi.org/project/taskiq-redis/).
Expand Down
7 changes: 3 additions & 4 deletions docs/available-components/result-backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ order: 3

Result backends are used to store execution results.
This includes:
* Captured logs;
* return value;
* Execution time in seconds.

- Captured logs;
- return value;
- Execution time in seconds.

## DummyResultBackend

Expand All @@ -18,7 +18,6 @@ where you need actual results.

This broker will always return `None` for any return_value. Please be careful.


## Redis result backend

This result backend is not part of the core taskiq library. You can install it as a separate package [taskiq-redis](https://pypi.org/project/taskiq-redis/).
Expand Down
12 changes: 5 additions & 7 deletions docs/available-components/schedule-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,11 @@
order: 4
---


# Available schedule sources

These objects are used to fetch current schedule for tasks.
Currently we have only one schedule source.


## LabelScheduleSource

This source parses labels of tasks, and if it finds a `schedule` label, it considers this task as scheduled.
Expand All @@ -31,10 +29,11 @@ async def my_task():
```

Parameters:
* `cron` - crontab string when to run the task.
* `args` - args to use, when invoking the task.
* `kwargs` - key-word arguments to use when invoking the task.
* `labels` - additional labels to use wehn invoking the task.

- `cron` - crontab string when to run the task.
- `args` - args to use, when invoking the task.
- `kwargs` - key-word arguments to use when invoking the task.
- `labels` - additional labels to use when invoking the task.

Usage:

Expand All @@ -50,7 +49,6 @@ scheduler = TaskiqScheduler(
)
```


::: warning Cool notice!

In order to resolve all labels correctly, don't forget to import
Expand Down
14 changes: 6 additions & 8 deletions docs/extending-taskiq/README.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,19 @@
---
dir:
order: 2
order: 2
---

# Extending taskiq


Taskiq is super extendable. The core library comes with different abstract classes.
You can implement these abstract classes to extend functionality.

All abstract classes can be found in `taskiq.abc` package.


## Contents:

* [Brokers](./broker.md)
* [Brokers](./broker.md)
* [Result backends](./resutl-backend.md)
* [CLI](./cli.md)
* [Schedule sources](./schedule-sources.md)
- [Brokers](./broker.md)
- [Brokers](./broker.md)
- [Result backends](./resutl-backend.md)
- [CLI](./cli.md)
- [Schedule sources](./schedule-sources.md)
4 changes: 2 additions & 2 deletions docs/extending-taskiq/broker.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,6 @@ For brokers, we have several conventions. It's good if your broker implements th
These rules are optional, and it's ok if your broker doesn't implement them.

1. If the message has the `delay` label with int or float number, this task's `execution` must be delayed
with the same number of seconds as in the delay label.
with the same number of seconds as in the delay label.
2. If the message has the `priority` label, this message must be sent with priority. Tasks with
higher priorities are executed faster.
higher priorities are executed faster.
1 change: 0 additions & 1 deletion docs/extending-taskiq/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,6 @@ demo = "my_project.cmd:MyCommand"
You can read more about entry points in [python documentation](https://packaging.python.org/en/latest/specifications/entry-points/).
The subcommand name is the same as the name of the entry point you've created.


```bash
$ taskiq demo --help
usage: demo [-h] [--test TEST]
Expand Down
1 change: 0 additions & 1 deletion docs/extending-taskiq/resutl-backend.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ order: 3
Result backends are used to store information about task execution.
To create new `result_backend` you have to implement `taskiq.abc.result_backend.AsyncResultBackend` class.


Here's a minimal example of a result backend:

@[code python](../examples/extending/result_backend.py)
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
order: 1
dir:
order: 1
order: 1
---

# Introduction
Expand Down
26 changes: 14 additions & 12 deletions docs/guide/architecture-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,16 +31,16 @@ end
Result backend ->> Your code: Receive the result
```


Let's discuss every component.

## Broker

Brokers are the most critical element of the taskiq. Every broker **must** implement the `AsyncBroker` abstract class from [taskiq.abc.broker](https://github.com/taskiq-python/taskiq/blob/master/taskiq/abc/broker.py) to make things work.

`AsyncBroker` class has two main methods to implement:
* kick
* listen

- kick
- listen

The `kick` method puts the message in the external system.
For example, it may call the `PUB` command in Redis.
Expand Down Expand Up @@ -162,24 +162,26 @@ class MyMiddleware(TaskiqMiddleware):
```

Here are methods you can implement in the order they are executed:
* `pre_send` - executed on the client side before the message is sent. Here you can modify the message.
* `post_send` - executed right after the message was sent.
* `pre_execute` - executed on the worker side after the message was received by a worker and before its execution.
* `on_error` - executed after the task was executed if the exception was found.
* `post_execute` - executed after the message was executed.
* `post_save` - executed after the result was saved in the result backend.

You can use sync or async hooks without changing aything, but adding async to the hook signature.
- `pre_send` - executed on the client side before the message is sent. Here you can modify the message.
- `post_send` - executed right after the message was sent.
- `pre_execute` - executed on the worker side after the message was received by a worker and before its execution.
- `on_error` - executed after the task was executed if the exception was found.
- `post_execute` - executed after the message was executed.
- `post_save` - executed after the result was saved in the result backend.

You can use sync or async hooks without changing anything, but adding async to the hook signature.

::: warning important note
If excetion happens in middlewares it won't be caught. Please ensure that you have try\except for all edge cases of your middleware.

If exception happens in middlewares it won't be caught. Please ensure that you have try\except for all edge cases of your middleware.

:::

Middlewares can store information in `message.labels` for
later use. For example `SimpleRetryMiddleware` uses labels
to remember number of failed attempts.


## Messages

Every message has labels. You can define labels
Expand Down
Loading