wiQddlZddlZddlZddlZddlZddlZddlmZm Z m Z m Z m Z m Z mZddlmZddlmZddlmZdZdZdZd Zd Zd Zd ZGd dZdS)N)AnyCallableDictListOptionalTupleUnion)RUNNING_WINDOWS)WorkerMapParams)DelayedKeyboardInterruptc eZdZdZdZdejjdede ddfdZ de fd Z d\d Z d\d Z d\d Z d]dededeede deeeff dZdeeeffdZd\dZd\dZd\dZd\dZd\dZd\dZd\dZde fdZd^deededeeddfdZd_ded e d!ed"e!fd#Z"d^deedefd$Z#dedefd%Z$deddfd&Z%ded'e ddfd(Z&dedej'fd)Z(dede fd*Z)dededdfd+Z*dedefd,Z+deed-e,eeee efddfd.Z-d`d0e d1eedefd2Z.deddfd3Z/deddfd4Z0d5e1ddfd6Z2deddfd7Z3de fd8Z4d1eede fd9Z5defd:Z6d\d;Z7de fd<Z8d\d=Z9d\d>Z:d\d?Z;deddfd@Zd\dCZ?deddfdDZ@deddfdEZAdede fdFZBdadGe ddfdHZCdadGe ddfdIZDdJeEjFddfdKZGd\dLZHdbdMejIdNe ddfdOZJeKdbdMejIdNe ddfdPZLdeddfdQZMdeddfdRZNdeddfdSZOdeddfdTZPdeddfdUZQdeddfdVZRded1ede fdWZSded1ede fdXZTded1ede fdYZUeKdZed1ede fd[ZVdS)c WorkerCommsa Class that contains all the inter-process communication objects (locks, events, queues, etc.) and functionality to interact with them, except for the worker insights comms. Contains: - Progress bar comms - Tasks & (exit) results comms - Exception handling comms - Terminating and restarting comms General overview of how the comms work: - When ``map`` or ``imap`` is used, the workers need to return the ``idx`` of the task they just completed. This is needed to return the results in order. This is communicated by using the ``_keep_order`` boolean value. - The main process assigns tasks to the workers by using their respective task queue (``_task_queues``). When no tasks have been completed yet, the main process assigns tasks in order. To determine which worker to assign the next task to, the main process uses the ``_task_idx`` counter. When tasks have been completed, the main process assigns the next task to the worker that completed the last task. This is communicated by using the ``_last_completed_task_worker_id`` deque. - Each worker keeps track of whether it is running a task by using the ``_worker_running_task`` boolean value. This is used by the main process in case a worker needs to be interrupted (due to an exception somewhere else). When a worker is not busy with any task at the moment, the worker will exit itself because of the ``_exception_thrown`` event that is set in such cases. However, when it is running a task, we want to interrupt it. The RLock object is used to ensure that there are no race conditions when accessing the ``_worker_running_task`` boolean value. E.g., when getting and setting the value without another process doing something in between. - Each worker also keeps track of which job it is working on by using the ``_worker_working_on_job`` array. This is needed to assess whether a certain task times out, and we need to know which job to set to failed. - The workers communicate their results to the main process by using the results queue (``_results_queue``). Each worker keeps track of how many results it has added to the queue (``_results_added``), and the main process keeps track of how many results it has received from each worker (``_results_received``). This is used by the workers to know when they can safely exit. - Workers can request a restart when a maximum lifespan is configured and reached. This is done by setting the ``_worker_restart_array`` boolean array. The main process listens to this array and restarts the worker when needed. The ``_worker_restart_condition`` is used to signal the main process that a worker needs to be restarted. - The ``_workers_dead`` array is used to keep track of which workers are alive and which are not. Sometimes, a worker can be terminated by the OS (e.g., OOM), which we want to pick up on. The main process checks regularly whether a worker is still alive according to the OS and according to the worker itself. If the OS says it's dead, but the value in ``_workers_dead`` is still False, we know something went wrong. - The ``_workers_time_task_started`` array is used to keep track of when a worker started a task. This is used by the main process to check whether a worker times out. - Exceptions are communicated by using the ``_exception_thrown`` event. Both the main process as the workers can set this event. The main process will set this when, for example, a timeout has been reached when running a ``map`` task. The source of the exception is stored in the ``_exception_job_id`` value, which is used by the main process to obtain the exception and raise accordingly. - The workers communicate every 0.1 seconds how many tasks they have completed. This is used by the main process to update the progress bar. The workers communicate this by using the ``_tasks_completed_array`` array. The ``_progress_bar_last_updated`` datetime object is used to keep track of when the last update was sent. The ``_progress_bar_shutdown`` boolean value is used to signal the progress bar handler thread to shut down. The ``_progress_bar_complete`` event is used to signal the main process and workers that the progress bar is complete and that it's safe to exit. g?ctxn_jobs order_tasksreturnNc||_||_||_d|_|jt jdd|_g|_d|_ g|_ tj |_ d|_d|_g|_d|_d|_|j|j|_d|_d|_|j|_|j|_d|_|jt jdd|_d|_d|_d|_d|_ dS)z :param ctx: Multiprocessing context :param n_jobs: Number of workers :param order_tasks: Whether to provide tasks to the workers in order, such that worker 0 will get chunk 0, worker 1 will get chunk 1, etc. FTlockN)!rrr _initializedValuectypesc_bool _keep_order _task_queues _task_idx_worker_running_task collectionsdeque_last_completed_task_worker_id_worker_working_on_job_results_queue_results_added_results_received_worker_restart_array ConditionLock_worker_restart_condition _workers_dead_workers_time_task_startedexception_lockEvent_exception_thrown_exception_job_id_kill_signal_received_tasks_completed_array_progress_bar_last_updated_progress_bar_shutdown_progress_bar_complete)selfrrrs c/var/www/development/aibuddy-work/election-extract/venv/lib/python3.11/site-packages/mpire/comms.py__init__zWorkerComms.__init__ZsE &! 8>>&-T>JJ57(,46!.9.?.A.A+:># ;?)+59:>")-););DHMMOO)L)L&26 ?C'#hmmoo!%!1!159%)X^^FM5t^%T%T";?#;?':>#:>###c|jS)zB :return: Whether the comms have been initialized rr;s r<is_initializedzWorkerComms.is_initializeds   r>cd|_dS)zm Resets initialization state. Note: doesn't actually reset the comms, just resets the state. FNr@rAs r<resetzWorkerComms.resets"r>c.fdtjD_fdtjD_jdjd_j_dtjD_ jdjj _ jtj jd_jtj jd_dgjzjdd<jd jd zd_jjdd d_d j_jdjj _t1j_jtj d d_j_d_dS) a Initialize/Reset comms containers. Threading doesn't have a JoinableQueue, so the threading context returns a multiprocessing.JoinableQueue instead. However, in the (unlikely) scenario that threading does get one, we explicitly switch to a multiprocessing.JoinableQueue for both the exception queue and progress bar tasks completed queue, because the progress bar handler needs process-aware objects. cBg|]}jS)r JoinableQueue.0_r;s r< z*WorkerComms.init_comms..s'RRR!TX3355RRRr>cg|]@}jtjdjAS)Fr)rrrr RLockrIs r<rLz*WorkerComms.init_comms..sH% % % LMDHNN6=%dhnn6F6FN G G% % % r>iTrcg|]}dS)rrG)rJrKs r<rLz*WorkerComms.init_comms..s===Qq===r>LNdrF)rangerr"r$rArrayr(rHr)r*rNr+rr r,r0r1r4clearrr5r6valuer7timer8r9r3r:reset_progressrrAs`r< init_commszWorkerComms.init_commss0SRRRuT[?Q?QRRR% % % % QVW[WbQcQc% % % !'+hnnS$+Dn&Q&Q##h4466==% *<*<===!%T[tx~~GWGW!X!X&*X^^FM4;UY^%Z%Z"!X^^FM4;T^RR!% 4111*.(..dkAoTX.*Y*Y' $$&&&!%QT!B!B+0"('+hnnS$+DHNNL\L\n&]&]#*.)++'&*hnnV]EPTn&U&U#&*hnn&6&6#  r>cd|_|jdg|jz|jdd<||dS)zG Resets the task_idx and last_completed_task_worker_id rN)r#r'rVrr7clear_progress_bar_shutdownclear_progress_bar_completerAs r<rYzWorkerComms.reset_progresssd +11333*+t{):#AAA& ((*** ((*****r>F worker_idprogress_bar_last_updatedprogress_bar_n_tasks_completed force_updatec|s|dz }tj}|s||z |jkrJ|j5|j|xx|z cc<dddn #1swxYwY|}d}||fS)a Signal that we've completed a task every 0.1 seconds, for the progress bar :param worker_id: Worker ID :param progress_bar_last_updated: Last time the progress bar update was send :param progress_bar_n_tasks_completed: Number of tasks completed since last update :param force_update: Whether to force an update :return: Tuple containing new last updated time and number of tasks completed since last update Nr)rXprogress_bar_update_intervalr7get_lock)r;r^r_r`ranows r<task_completed_progress_barz'WorkerComms.task_completed_progress_bars 0 *a / *ikk  /C";;t?```,5577 Y Y+I666:XX666 Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y(+ %-. *(*HHHsA&&A*-A*cXtj|jz }||jkrtj|j|z |sN|s:|jjs.t|j }tj|_|StS)z Obtain the number of tasks completed by the workers. As the progress bar handler lives inside a thread we don't poll continuously, but every 0.1 seconds. :return: The number of tasks done or a poison pill ) rXr8rdsleepexception_thrownkill_signal_receivedr9rWsumr7 POISON_PILL)r; time_diffn_tasks_completeds r< get_tasks_completed_progress_barz,WorkerComms.get_tasks_completed_progress_barsIKK$"AA t8 8 8 Jt89D E E E((** %43L3L3N3N %.4 % #D$? @ @ .2ikkD +$ $r>cd|j_dS)zH Signals the progress bar handling process to shut down TNr9rWrAs r<signal_progress_bar_shutdownz(WorkerComms.signal_progress_bar_shutdowns-1#)))r>c0|jd|j_dSdS)z9 Clears the progress bar shutdown signal NFrrrAs r<r\z'WorkerComms.clear_progress_bar_shutdown s&  & 205D ' - - - 3 2r>c8|jdS)z: Signal that the progress bar is complete N)r:setrAs r<signal_progress_bar_completez(WorkerComms.signal_progress_bar_completes #'')))))r>cJ|j|jdSdS)z9 Clear that the progress bar is complete N)r:rVrAs r<r]z'WorkerComms.clear_progress_bar_completes0  & 2  ' - - / / / / / 3 2r>cJ|j|jdSdS)z; Waits until the progress bar is completed N)r:waitrAs r<#wait_until_progress_bar_is_completez/WorkerComms.wait_until_progress_bar_is_complete!s0  & 2  ' , , . . . . . 3 2r>cd|j_dS)z8 Set that we need to keep order in mind TNr!rWrAs r<signal_keep_orderzWorkerComms.signal_keep_order,s"&r>cd|j_dS)z; Forget that we need to keep order in mind FNr}rAs r<clear_keep_orderzWorkerComms.clear_keep_order2s"'r>c|jjS)z@ :return: Whether we need to keep order in mind r}rAs r< keep_orderzWorkerComms.keep_order8s%%r>job_idtaskc||}t5|||fn|}|j||dddddS#1swxYwYdS)a Add a task to the queue so a worker can process it. :param job_id: Job ID or None :param task: A tuple of arguments to pass to a worker, which acts upon it :param worker_id: If provided, give the task to the worker ID NTblock)_get_task_worker_idr r"put)r;rrr^s r<add_taskzWorkerComms.add_taskBs,,Y77 % ' ' ? ?%+%7FD>>TD  i ( , ,T , > > > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?s+AA #A rGfuncargskwargsc|i}|}|dt||||||ff|dS)ao Add a task to the queue so a worker can process it. First though, add an APPLY_PILL such that the worker knows it needs to treat this task differently. :param func: Function to apply :param job_id: Job ID :param args: Arguments to pass to the function :param kwargs: Keyword arguments to pass to the function N)rr APPLY_PILL)r;rrrrr^s r<add_apply_taskzWorkerComms.add_apply_taskOsZ >F,,..  dJ 222 ftdF^4i@@@@@r>c|G|js|js |j|jz}|xjdz c_n|j}|S)a^ Get the worker ID for the next task. When a worker ID is not present, we first check if we need to pass on the tasks in order. If not, we check whether we got results already. If so, we give the next task to the worker who completed that task. Otherwise, we decide based on order :return: Worker ID Nrc)rr'r#rpopleftr;r^s r<rzWorkerComms._get_task_worker_id`s[   Jt'J J NT[8 !# ?GGII r>c|sL |j|ddS#tj$rYnwxYw|LdS)z Obtain new chunk of tasks. Occasionally we check if an exception has been thrown. If so, we should quit. :param worker_id: Worker ID :return: Chunk of tasks or None when an exception was thrown T{Gz?rtimeoutN)rjr"getqueueEmptyrs r<get_taskzWorkerComms.get_taskss}''))  (377dD7QQQ;    ''))  ts!8A  A cD|j|dS)zY Signal that we've completed a task :param worker_id: Worker ID N)r" task_doners r<rzWorkerComms.task_dones$ )$..00000r>runningc*||j|_dS)z Set the task the worker is currently running :param worker_id: Worker ID :param running: Whether the worker is running a task Nr$rW)r;r^rs r<set_worker_running_taskz#WorkerComms.set_worker_running_tasks6=!),222r>c@|j|S)zy Obtain the lock for the worker running task :param worker_id: Worker ID :return: RLock )r$rers r<get_worker_running_task_lockz(WorkerComms.get_worker_running_task_locks(3<<>>>r>c&|j|jS)z Obtain whether the worker is running a task :param worker_id: Worker ID :return: Whether the worker is running a task rrs r<get_worker_running_taskz#WorkerComms.get_worker_running_tasks(399r>c||j|<dS)z Signal that the worker is working on the job ID :param worker_id: Worker ID :param job_id: Job ID Nr()r;r^rs r<signal_worker_working_on_jobz(WorkerComms.signal_worker_working_on_jobs28#I...r>c|j|S)zy Obtain the job ID the worker is working on :param worker_id: Worker ID :return: Job ID rrs r<get_worker_working_on_jobz%WorkerComms.get_worker_working_on_jobs*955r>resultscl||j|xxdz cc<|j||fdS)z Add results to the results queue :param worker_id: Worker ID :param results: A list of tuples of job ID, success bool, and output from the worker Nrc)r*r)r)r;r^rs r< add_resultszWorkerComms.add_resultssK    * * *a / * * * G 455555r>Trrc t5|j||\}}|j|`|j5|j|xxdz cc<dddn #1swxYwY|j||cdddS#1swxYwYdS#t$rddtfgcYSwxYw)a$ Obtain the next result from the results queue :param block: Whether to block (wait for results) :param timeout: How long to wait for results in case ``block==True`` :return: The next result from the queue, which is the result of calling the function rNrc) r r)rrr+rer'appendEOFErrorrm)r;rrr^rs r< get_resultszWorkerComms.get_resultss /)++  %)%8%<%<5RY%<%Z%Z" 7#--///(/88::??.y999Q>999???????????????7>>yIII                   / / /4-. . . . /sYCAB9$B: B9B B9 B B9, C9B==CB=CCCcd|j|<dS)zi Reset the number of results received from a worker :param worker_id: Worker ID rN)r+rs r<reset_results_receivedz"WorkerComms.reset_results_receiveds -.y)))r>c|j||j|kr2tjd|j||j|k0dSdS)z Wait for the main process to receive all the results from a specific worker :param worker_id: Worker ID rN)r+r*rXrirs r<wait_for_all_results_receivedz)WorkerComms.wait_for_all_results_receivedsX $Y/43Fy3QQQ Jt   $Y/43Fy3QQQQQQQr> map_paramsct|jD]5}|dt||d||6dS)zc Submits new map params for each worker :param map_params: New map params N)rTrrNEW_MAP_PARAMS_PILL)r;rr^s r<add_new_map_paramszWorkerComms.add_new_map_paramssW t{++ 7 7I MM$ 3Y ? ? ? MM$ I 6 6 6 6 7 7r>cP||j_|jdS)zf Set the exception event :param job_id: Job ID which triggered the exception N)r5rWr4rv)r;rs r<signal_exception_thrownz#WorkerComms.signal_exception_throwns* (.$ ""$$$$$r>c4|jS)zP :return: Whether an exception was thrown by one of the workers )r4is_setrAs r<rjzWorkerComms.exception_throwns%,,...r>c8|j|S)z Waits until the exception thrown event is set :param timeout: How long to wait before giving up :return: True when exception was thrown, False if timeout was reached r)r4rz)r;rs r<wait_for_exception_thrownz%WorkerComms.wait_for_exception_throwns%**7*;;;r>c|jjS)z? :return: Job ID which triggered the exception )r5rWrAs r<get_exception_thrown_job_idz'WorkerComms.get_exception_thrown_job_id s%++r>cd|j_dS)z4 Set the kill signal received event TNr6rWrAs r<signal_kill_signal_receivedz'WorkerComms.signal_kill_signal_receiveds,0"(((r>c|jjS)zS :return: Whether a kill signal was received in one of the workers rrAs r<rkz WorkerComms.kill_signal_receiveds)//r>clt|jD]}|dt|dS)z7 'Tell' the workers their job is done. N)rTrrrmrs r<insert_poison_pillzWorkerComms.insert_poison_pill#s@t{++ 8 8I MM$ Y 7 7 7 7 8 8r>cD|dddtfgdS)zF 'Tell' the apply results listener their job is done. NT)rrmrAs r<#insert_poison_pill_results_listenerz/WorkerComms.insert_poison_pill_results_listener*s* t[ 9:;;;;;r>clt|jD]}|dt|dS)z When ``keep_alive=True``, the workers should stay alive, but they need to wrap up their work (like sending the latest progress bar update) N)rTrrNON_LETHAL_POISON_PILLrs r<insert_non_lethal_poison_pillz)WorkerComms.insert_non_lethal_poison_pill0sE t{++ C CI MM$ 6 B B B B C Cr>cd|j|<|j5|jddddS#1swxYwYdS)zx Signal to the main process that this worker needs to be restarted :param worker_id: Worker ID TN)r,r/notifyrs r<signal_worker_restartz!WorkerComms.signal_worker_restart8s 15"9-  + 4 4  * 1 1 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4s 9==cx|j5|jddddS#1swxYwYdS)z Signal the condition primitive, such that the worker restart handler thread can continue. This is useful when an exception has been thrown and the thread needs to exit. N)r/rrAs r<signal_worker_restart_conditionz+WorkerComms.signal_worker_restart_conditionBs  + 4 4  * 1 1 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4s /33cfd}j5|}|s#j|}|cdddS#1swxYwYdS)a Obtain the worker IDs that need to be restarted. Blocks until at least one worker needs to be restarted. It returns an empty list when an exception has been thrown (which also notifies the worker_done_condition) :return: List of worker IDs c@dtjDS)Ncg|] \}}|| SrGrG)rJr^restarts r<rLzQWorkerComms.get_worker_restarts.._get_worker_restarts..Rs#iii"4)WahiIiiir>) enumerater,rAsr<_get_worker_restartsz=WorkerComms.get_worker_restarts.._get_worker_restartsQs"ii $B\8]8]iii ir>N)r/rz)r;r worker_idss` r<get_worker_restartszWorkerComms.get_worker_restartsJs j j j j j +  .-//J 4.335551133                   s1A  AAcd|j|<dS)z_ Worker has been restarted, reset signal. :param worker_id: Worker ID FN)r,rs r<reset_worker_restartz WorkerComms.reset_worker_restart^s 16"9---r>cd|j|<dS)zV Indicate that a worker is alive :param worker_id: Worker ID FNr0rs r<signal_worker_alivezWorkerComms.signal_worker_alivefs ).9%%%r>cd|j|<dS)zU ` Indicate that a worker is dead :param worker_id: Worker ID TNrrs r<signal_worker_deadzWorkerComms.signal_worker_deadns )-9%%%r>c|j| S)z Check whether the worker is alive :param worker_id: Worker ID :return: Whether the worker is alive rrs r<is_worker_alivezWorkerComms.is_worker_alivevs%i000r> keep_alivec|j|s4|j|jdSdS)zs Join results and exit results queues :param keep_alive: Whether to keep the queues alive N)r)joinclose join_threadr;rs r<join_results_queueszWorkerComms.join_results_queuess[   """ .   % % ' ' '   + + - - - - - . .r>ctd|jD|s$d|jDd|jDdSdS)z_ Join task queues :param keep_alive: Whether to keep the queues alive c6g|]}|SrG)rrJqs r<rLz0WorkerComms.join_task_queues..s ---a---r>c6g|]}|SrG)rrs r<rLz0WorkerComms.join_task_queues..s 2 2 21QWWYY 2 2 2r>c6g|]}|SrG)rrs r<rLz0WorkerComms.join_task_queues..s 8 8 8Q]]__ 8 8 8r>N)r"rs r<join_task_queueszWorkerComms.join_task_queuessa .-4,---- 9 2 2 1 2 2 2 2 8 8d&7 8 8 8 8 8 8 9 9r>dont_wait_eventcd} |d|d}-#tjtf$r|r|YdSYdSwxYw)a Drain the results queue without blocking. This is done when terminating workers, while they could still be busy putting something in the queues. This function will always be called from within a thread. :param dont_wait_event: Event object to indicate whether other termination threads should continue. I.e., when we set it to False, threads should wait. FTrN)rrVrrOSErrorrv)r;r got_resultss r<$drain_results_queue_terminate_workerz0WorkerComms.drain_results_queue_terminate_workers  & #  u ---%%'''"  # W% & & & &##%%%%%% & & & &s.2,A%$A%cbfdjDjdS)z0 Drain tasks and results queues c:g|]}|SrG)drain_and_join_queue)rJrr;s r<rLz,WorkerComms.drain_queues..s'AAA! " "1 % %AAAr>N)r"rr)rAs`r< drain_queueszWorkerComms.drain_queuess@ BAAAt/@AAAA !!$"566666r>rrctr|||dS |j|j||f}||d|r(|||r*|| dSdS#t$rYdSwxYw)ay Drains a queue completely, such that it is joinable. If a timeout is reached, we give up and terminate. So far, I've only seen it happen when an exception is thrown when using spawn as start method, and even then it only happens once every 1000 runs or so. :param q: Queue to join :param join: Whether to join the queue or not )targetrrN) r _drain_and_join_queuerProcessstartris_alive terminaterrr)r;rrprocesss r<rz WorkerComms.drain_and_join_queues    & &q$ / / / / / (**$2LTUW[S\*]]  Q '''##%%#%%'''LLNNN$GGIIIMMOOOOO$$    sB3C C&%C&c|dSd} ||dz }#ttf$rYnwxYw |r|dkr6|dd|dz}|0|dk6n(#tt t jtf$rYnwxYw|rW || | dS#ttf$rYdSwxYwdS)z Drains a queue completely, such that it is joinable :param q: Queue to join :param join: Whether to join the queue or not NrTrcg?r) rr ValueErroremptyrrrrrrr)rrns r<rz!WorkerComms._drain_and_join_queuesP 9 F    Q $    D  ggii 166D#...Qggii 1665; ;    D      Z(       s0#77AB "B10B17' A 666r>cHtj|j|dzdz<dS)zl Sets the task started timestamp for a specific worker :param worker_id: Worker ID rSrcNr rs r<signal_worker_task_startedz&WorkerComms.signal_worker_task_started' >BY[[' A (9:::r>cHtj|j|dzdz<dS)zs Sets the worker_exit started timestamp for a specific worker :param worker_id: Worker ID rSNr rs r<signal_worker_exit_startedz&WorkerComms.signal_worker_exit_startedr r>c d|j|dz<dS)zu Resets the worker_init started timestamp for a specific worker :param worker_id: Worker ID rrSNr1rs r<signal_worker_init_completedz(WorkerComms.signal_worker_init_completeds :;' A 666r>c&d|j|dzdz<dS)zn Resets the task started timestamp for a specific worker :param worker_id: Worker ID rrSrcNrrs r<signal_worker_task_completedz(WorkerComms.signal_worker_task_completed >?' A (9:::r>c&d|j|dzdz<dS)zu Resets the worker_exit started timestamp for a specific worker :param worker_id: Worker ID rrSrNrrs r<signal_worker_exit_completedz(WorkerComms.signal_worker_exit_completedrr>cN|j|dz}|||S)z Checks whether a worker_init takes longer than the timeout value :param worker_id: Worker ID :param timeout: Timeout in seconds :return: True when time has expired, False otherwise rSr1_has_worker_timed_outr;r^r started_times r<has_worker_init_timed_outz%WorkerComms.has_worker_init_timed_out's+6y1}E )),@@@r>cT|j|dzdz}|||S)z Checks whether a worker task takes longer than the timeout value :param worker_id: Worker ID :param timeout: Timeout in seconds :return: True when time has expired, False otherwise rSrcrrs r<has_worker_task_timed_outz%WorkerComms.has_worker_task_timed_out206y1}q7HI )),@@@r>cT|j|dzdz}|||S)z Checks whether a worker_exit takes longer than the timeout value :param worker_id: Worker ID :param timeout: Timeout in seconds :return: True when time has expired, False otherwise rSrrrs r<has_worker_exit_timed_outz%WorkerComms.has_worker_exit_timed_out=r!r>rcF|dkrdntj|z |kS)z Checks whether time has passed beyond the timeout :param started_time: Timestamp :param timeout: Timeout in seconds :return: True when time has expired, False otherwise gF)rX)rrs r<rz!WorkerComms._has_worker_timed_outHs)%++uu$)++ 2LQX1XXr>)rN)NF)N)rGN)TN)F)T)W__name__ __module__ __qualname____doc__rdmpcontext BaseContextintboolr=rBrDrZrYfloatrrrgr strrprsr\rwr]r{r~rrrrrrrrrrrrNrrrrrrrrrr rrrjrrrrkrrrrrrrrrrrr threadingr3rrrHr staticmethodrr r rrrrrr r#rrGr>r<rr s33l$' 5?BJ25?C5?d5?W[5?5?5?5?v!!!!! """" '!'!'!'!R++++UY9>IISIUZIDLSMI26ICHPSCTIIII4%S/*1111 6666**** 0000////&&&& '''' &D&&&& ? ?x} ?C ?HSM ?]a ? ? ? ?AASAAATXAAAA"Xc]c& # #    13141111==t=====?c?bh????::::::8c8384888863636666 6Xc] 6T%QT W[]`H`Ba=b 6gk 6 6 6 6///x/RU////*......st7_77777%c%d%%%%/$//// <%r)r%rmultiprocessingr)rr0rXtypingrrrrrrr mpire.contextr mpire.paramsr mpire.signalr rmrrr MAIN_PROCESS INIT_FUNC EXIT_FUNCrrGr>r<r:s%   DDDDDDDDDDDDDDDDDD))))))((((((111111     q Yq Yq Yq Yq Yq Yq Yq Yq Yq Yr>