pikatasks

pikatasks is a minimalistic library that allows you to run remote tasks easily. There's also a Django integration.

Requirements

• pip install pika
• RabbitMQ as message broker

How-to

Import:

import pikatasks

Configure:

pikatasks.settings.BROKER_HOST = "localhost"
pikatasks.settings.BROKER_PORT = "5671"
pikatasks.settings.SSL_ENABLED = False
pikatasks.settings.VIRTUAL_HOST = "foo"
pikatasks.settings.USERNAME = "lancelot"
pikatasks.settings.PASSWORD = "swalloWcoc0nut"

Or in Django settings:

PIKATASKS_BROKER_HOST = "localhost"
PIKATASKS_BROKER_PORT = "5671"
PIKATASKS_SSL_ENABLED = False
PIKATASKS_VIRTUAL_HOST = "foo"
PIKATASKS_USERNAME = "lancelot"
PIKATASKS_PASSWORD = "swalloWcoc0nut"

Implement a task (server):

@pikatasks.task(name="hello")
def hello(something):
    msg = "Hello, " + something + "!"
    print(msg)
    return msg

Note: you will need a queue with exactly the same name as the task. See section: Queues and Permissions.

Start a server:

pikatasks.worker.start(tasks=[hello])

Run a task (client):

To simply run a task:

pikatasks.run("hello", something="World")

Run a task and get its reult:

result = pikatasks.rpc("hello", something="World")
print(result)
#  >>> "Hello, World!"

Catch exceptions:

if a task raises an exception on the server, pikatask.rpc() call on the client will also raise an exception. Full exception message is not sent for security/isolation reasons.

try:
  pikatasks.run("hello", something=42)
except pikatasks.RPCError as e:
  print(e)
  # >>> Task hello raised TypeError (see worker log for details).
  # Note: TypeError was raised when the server was running: "Hello, " + 42 + "!"

Queues and Permissions

Queues and exchanges:

With AMQ, messages first arrive to exchanges, then broker distributes them to to queues using routing keys. If you are not sure what it is all about, read this tutorial first and further RabbitMQ documentation if needed.

Queues and Tasks:

• pikatasks requires a separate queue for each task.
• queue name == task name
• You need to create these queues by yourself.

Developent setup:

You are done after creating queues for each of your tasks. Don't need anything else for the development. Note: exchange amq.default will be used.

Client:

• Create a new exchange for your client. Let's call it client-out, and its type should be direct. This exchange will be used for sending tasks.
• Decide which tasks should the client use. Let's say these are task1 and task2 (you should have the corresponding queues already).
• For each of the tasks, create a new binding for the exchange client-out, with routing key == queue name == task name
  • e.g. exchange = client-out, routing key = task1, queue = task1
• User permissions:
  • Configure: empty string (no config permissions)
  • Write: ^client-out$ (replace with the name of your exchange)
  • Read: empty string (no read permissions, RPC results/replies will still work)

Worker:

• User permissions:
  • Configure: empty string (no config permissions)
  • Write: .* (everything) or ^amq.default$ (amq.default is required to send "direct reply-to")
  • Read: ^(task1|task2)$, replace taskN with whatever your task names are