-
Notifications
You must be signed in to change notification settings - Fork 27
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
f33b2f9
commit 9eeb5a1
Showing
7 changed files
with
169 additions
and
0 deletions.
There are no files selected for viewing
27 changes: 27 additions & 0 deletions
27
cheatsheets/python/libraries/queue-messaging/amazon-sqs.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
# Amazon SQS / boto3 | ||
|
||
Amazon Simple Queue Service (SQS) is a fully managed message queuing service. | ||
|
||
## Key Features | ||
|
||
- Scalable and fully managed by AWS | ||
- Supports standard and FIFO queues | ||
- Integrates well with other AWS services | ||
|
||
## Basic Usage with boto3 | ||
|
||
```python | ||
import boto3 | ||
|
||
|
||
sqs = boto3.client('sqs') | ||
|
||
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/MyQueue' | ||
|
||
response = sqs.send_message( | ||
QueueUrl=queue_url, | ||
MessageBody='Hello, SQS!' | ||
) | ||
|
||
print(f"Message sent. ID: {response['MessageId']}") | ||
``` |
19 changes: 19 additions & 0 deletions
19
cheatsheets/python/libraries/queue-messaging/apache-kafka.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
# Apache Kafka | ||
|
||
Kafka is a distributed streaming platform that can be used for building real-time data pipelines and streaming apps. | ||
|
||
## Key Features | ||
- High-throughput, fault-tolerant, publish-subscribe messaging system | ||
- Scalable and durable storage | ||
- Stream processing | ||
|
||
## Basic Usage with kafka-python | ||
|
||
```python | ||
from kafka import KafkaProducer | ||
|
||
producer = KafkaProducer(bootstrap_servers=['localhost:9092']) | ||
|
||
producer.send('my-topic', b'Hello, Kafka!') | ||
producer.flush() | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
# Celery | ||
|
||
Celery is a distributed task queue system for Python. | ||
|
||
## Key Features | ||
|
||
- Asynchronous task execution | ||
- Distributed architecture | ||
- Multiple broker support (RabbitMQ, Redis, etc.) | ||
- Periodic task scheduling | ||
|
||
## Basic Usage | ||
|
||
```python | ||
from celery import Celery | ||
|
||
app = Celery('tasks', broker='pyamqp://guest@localhost//') | ||
|
||
|
||
@app.task | ||
def add(x, y): | ||
return x + y | ||
|
||
|
||
# To call the task | ||
result = add.delay(4, 4) | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
# Queues and messaging | ||
|
||
|
||
### Choosing a queuing approach | ||
|
||
The choice between these depends on your specific requirements, such as: | ||
|
||
- Scalability needs | ||
- Integration with existing systems | ||
- Performance requirements | ||
- Complexity of your task processing logic | ||
|
||
Here's a quick guide: | ||
|
||
- Celery is great for distributed task processing in Python applications. | ||
- RabbitMQ is a robust message broker that can be used with Celery or standalone. | ||
- Redis Queue (RQ) is simpler than Celery and good for smaller applications. | ||
- Apache Kafka is excellent for high-throughput, real-time data streaming applications. | ||
- ZeroMQ is lightweight and fast, good for direct inter-process communication. | ||
- Amazon SQS is a fully managed solution, ideal for AWS-based applications. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
# RabbitMQ | ||
|
||
RabbitMQ is a message broker that can be used with Celery or independently. | ||
|
||
## Key Features | ||
- Supports multiple messaging protocols | ||
- Clustering for high availability | ||
- Message persistence | ||
- Flexible routing | ||
|
||
## Basic Usage with Pika (Python client) | ||
|
||
```python | ||
import pika | ||
|
||
|
||
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost')) | ||
channel = connection.channel() | ||
|
||
channel.queue_declare(queue='hello') | ||
|
||
channel.basic_publish(exchange='', | ||
routing_key='hello', | ||
body='Hello World!') | ||
|
||
print(" [x] Sent 'Hello World!'") | ||
|
||
connection.close() | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
# Redis Queue (RQ) | ||
|
||
RQ is a simple Python library for queueing jobs and processing them in the background with workers. | ||
|
||
## Key Features | ||
- Simple to use and deploy | ||
- Uses Redis as its backend | ||
- Supports job prioritization | ||
|
||
## Basic Usage | ||
|
||
```python | ||
from redis import Redis | ||
from rq import Queue | ||
|
||
q = Queue(connection=Redis()) | ||
|
||
|
||
def my_job(param1, param2): | ||
return param1 + param2 | ||
|
||
|
||
job = q.enqueue(my_job, 1, 2) | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
# ZeroMQ (ZMQ) | ||
|
||
ZeroMQ is a high-performance asynchronous messaging library. | ||
|
||
## Key Features | ||
- Supports various messaging patterns (pub/sub, request/reply, etc.) | ||
- No central broker, can be used for direct inter-process communication | ||
- Very fast and lightweight | ||
|
||
## Basic Usage | ||
|
||
```python | ||
import zmq | ||
|
||
context = zmq.Context() | ||
socket = context.socket(zmq.REP) | ||
socket.bind("tcp://*:5555") | ||
|
||
while True: | ||
message = socket.recv() | ||
print(f"Received request: {message}") | ||
socket.send(b"World") | ||
``` |