i have large codebase parallelise. can avoid rewriting method signatures of hundreds of functions using single global queue. know it's messy; please don't tell me if i'm using globals i'm doing wrong in case easiest choice. code below works don't understand why. declare global multiprocessing.queue() don't declare should shared between processes (by passing parameter worker). python automatically place queue in shared memory? safe on larger scale?
note: can tell queue shared between processes: worker processes start doing work on empty queues , idle 1 second before main queue pushes work onto queues.
import multiprocessing import time outqueue = none class workerprocess(multiprocessing.process): def __init__(self): multiprocessing.process.__init__(self) self.exit = multiprocessing.event() def dowork(self): global outqueue ob = outqueue.get() ob = ob + "!" print ob time.sleep(1) #simulate more hard work outqueue.put(ob) def run(self): while not self.exit.is_set(): self.dowork() def shutdown(self): self.exit.set() if __name__ == '__main__': global outqueue outqueue = multiprocessing.queue() procs = [] x in range(10): procs.append(workerprocess()) procs[x].start() time.sleep(1) x in range(20): outqueue.put(str(x)) time.sleep(10) p in procs: p.shutdown() p in procs: p.join() try: while true: x = outqueue.get(false) print x except: print "done"
assuming you're using linux, answer in way os creates new process.
when process spawns new 1 in linux, forks parent one. result child process properties of parent one. clone.
in example instantiating queue , creating new processes. therefore children processes have copy of same queue , able use it.
to see things broken try first create processes , creating queue object. you'll see children having global variable still set none while parent have queue.
it safe, yet not recommended, share queue global variable on linux. on windows, due different process creation approach, sharing queue through global variable won't work.
as mentioned in programming guidelines
explicitly pass resources child processes
on unix using fork start method, child process can make use of shared resource created in parent process using global resource. however, better pass object argument constructor child process.
apart making code (potentially) compatible windows , other start methods ensures long child process still alive object not garbage collected in parent process. might important if resource freed when object garbage collected in parent process.
for more info linux forking can read man page.
Comments
Post a Comment