so i've been looking building application relies on similar messaging bus. idea extremely fault tolerant. have queue of tasks need performed, , here steps believe end goal in queue based system.
- a/b server adds item queue worked on.
- c server listening on queue sees new item on queue, , begins work on it. believe should lock item timeout, (if server crashes, etc..., need other workers able work on it.)
one of 2 things happens now:
- c server fails respond said task, or has taken long past timeout, , queue unlocks , passes server d processing
- or -
- c server completes task , removes entry queue.
i looking different solutions, , see lot of people using redis backend performing operation, it's queue simplistic. example rpoplpush remove key queue. happens if server crashes? queue thinks processed item, , have lost task.
what steps recommended ensuring task completions, , noting task failures able reproccessed server? intend write tasks in go , i'm open using cloud services such aws.
redis basic component on can build queuing system. said, implementing true guaranteed delivery system on top of redis not trivial, if need transactional behavior.
here queuing systems implemented redis in various languages:
- http://python-rq.org/
- https://github.com/resque/resque
- https://github.com/ask/celery
- https://pypi.python.org/pypi/rpqueue
similar things developed in go, when comes true guaranteed delivery semantic, devil in details.
you better served dedicated queuing system, such rabbitmq or activemq. while more complex, offer more features, , better guarantees.
here go client rabbitmq: https://github.com/streadway/amqp
you might interested looking @ disque (a dedicated queuing solution redis author), , corresponding go client @ https://github.com/everythingme/go-disque
finally, beanstalkd lightweight solution; can find go client at: https://github.com/kr/beanstalk
Comments
Post a Comment