widdlZddlZddlZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z mZmZmZ ddlZdZn#e$rdZdZYnwxYwddlmZmZmZmZmZddlmZmZmZmZm Z ddl!m"Z"m#Z#ddl$m%Z%dd l&m'Z'dd l(m)Z)dd l*m+Z+m,Z,m-Z-m.Z.dd l/m0Z0dd l1m2Z2ddl3m4Z4m5Z5ddl6m7Z7m8Z8m9Z9ddl:m;Z;mZ?GddZ@dS)N) AnyCallableDict GeneratorIterableListOptionalSizedUnionTupleTF) AsyncResultAsyncResultTypeAsyncResultWithExceptionGetter UnorderedAsyncExitResultIteratorUnorderedAsyncResultIterator) EXIT_FUNC INIT_FUNC MAIN_PROCESS POISON_PILL WorkerComms)DEFAULT_START_METHODRUNNING_WINDOWS) get_dashboard_connection_details)populate_exception)WorkerInsights)check_map_parametersCPUListWorkerMapParamsWorkerPoolParams)ProgressBarHandler)DisableKeyboardInterruptSignal)get_tqdm TqdmManager)apply_numpy_chunking chunk_tasksset_cpu_affinity) MP_CONTEXTSworker_factoryc$eZdZdZddddddeddddf deededede d ed ed e d ed edededdfdZ dGdeddfdZ dHde ddfdZ dGd eddfdZdGd eddfdZdGdeddfdZdIdZdeddfdZdIdZdIdZdIdZdIdZdefdZdJd Zd!e ddfd"Z dKd#ed$eeefd%eed&eed'eed(eed)eed*ed+ed,eed-eed.eed/eed0eed1ee e e fd2ee de f"d3Z! dLd#ed$eeefd%eed&eed'eed(eed)eed*ed,eed-eed.eed/eed0eed1ee e e fd2ee de f d4Z" dLd#ed$eeefd%eed&eed'eed(eed)eed*ed,eed-eed.eed/eed0eed1ee e e fd2ee de#e ddff d5Z$ dLd#ed$eeefd%eed&eed'eed(eed)eed*ed,eed-eed.eed/eed0eed1ee e e fd2ee de#e ddff d6Z% dMd#ed8e d9e d:eed;eed,eed-eed.eed/eed0eede fd<Z& dMd#ed8e d9e d:eed;eed,eed-eed.eed/eed0eede'fd=Z(dId>Z)dNd eddfd?Z*e*Z+dId@Z,dedAe-j.ddfdBZ/deddfdCZ0dIdDZ1dIdEZ2de fdFZ3dS)O WorkerPoolzw A multiprocessing worker pool which acts like a ``multiprocessing.Pool``, but is faster and has more options. NTFn_jobsdaemoncpu_idsshared_objectspass_worker_iduse_worker_state start_method keep_aliveuse_dillenable_insights order_tasksreturnc t||||||||| | | |_d|_t|| |_|dkrt d|_nt | rdnd||_i|_t|jtt|jtt|jg|_ t|j|jj|jj|_d|_d|_d|_d|_d|_t-j|_d|_t5|j|jj|jj|_dS)a :param n_jobs: Number of workers to spawn. If ``None``, will use ``mpire.cpu_count()`` :param daemon: Whether to start the child processes as daemon :param cpu_ids: List of CPU IDs to use for pinning child processes to specific CPUs. The list must be as long as the number of jobs used (if ``n_jobs`` equals ``None`` it must be equal to ``mpire.cpu_count()``), or the list must have exactly one element. In the former case, element `i` specifies the CPU ID(s) to use for child process `i`. In the latter case the single element specifies the CPU ID(s) for all child processes to use. A single element can be either a single integer specifying a single CPU ID, or a list of integers specifying that a single child process can make use of multiple CPU IDs. If ``None``, CPU pinning will be disabled :param shared_objects: Objects to be passed on as shared objects to the workers once. It will be passed on to the target, ``worker_init``, and ``worker_exit`` functions. ``shared_objects`` is only passed on when it's not ``None``. Shared objects will be copy-on-write when using ``fork`` as start method. When enabled, functions receive the shared objects as second argument, depending on other settings. The order is: ``worker_id``, ``shared_objects``, ``worker_state``, and finally the arguments passed on from ``iterable_of_args`` :param pass_worker_id: Whether to pass on a worker ID to the target, ``worker_init``, and ``worker_exit`` functions. When enabled, functions receive the worker ID as first argument, depending on other settings. The order is: ``worker_id``, ``shared_objects``, ``worker_state``, and finally the arguments passed on from ``iterable_of_args`` :param use_worker_state: Whether to let a worker have a worker state. The worker state will be passed on to the target, ``worker_init``, and ``worker_exit`` functions. When enabled, functions receive the worker state as third argument, depending on other settings. The order is: ``worker_id``, ``shared_objects``, ``worker_state``, and finally the arguments passed on from ``iterable_of_args`` :param start_method: Which process start method to use. Options for multiprocessing: ``'fork'`` (default, if available), ``'forkserver'`` and ``'spawn'`` (default, if ``'fork'`` isn't available). For multithreading use ``'threading'``. See https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods for more information and https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods for some caveats when using the ``'spawn'`` or ``'forkserver'`` methods :param keep_alive: When ``True`` it will keep workers alive after completing a map call, allowing to reuse workers :param use_dill: Whether to use dill as serialization backend. Some exotic types (e.g., lambdas, nested functions) don't work well when using ``spawn`` as start method. In such cased, use ``dill`` (can be a bit slower sometimes) :param enable_insights: Whether to enable worker insights. Might come at a small performance penalty (often neglible) :param order_tasks: Whether to provide tasks to the workers in order, such that worker 0 will get chunk 0, worker 1 will get chunk 1, etc. N threadingmp_dillmpF)r pool_params map_paramsr(Workerr'ctx_cacherrrr_workersrr+r5 _worker_comms _map_running_results_handler_thread_restart_handler_thread_timeout_handler_thread _unexpected_death_handler_threadr8Event_handler_threads_stop_event_progress_bar_handlerrr3_worker_insights) selfr+r,r-r.r/r0r1r2r3r4r5s b/var/www/development/aibuddy-work/election-extract/venv/lib/python3.11/site-packages/mpire/pool.py__init__zWorkerPool.__init__&sZX,FGV^Uceu,8*hP_alnn%\8<<  ; & &";/DHH"#B99dCLQDH35 &t{LAAA&t{I>>>(555 (43C3JDL\Lhii!(,$'+$'+$04-+4?+<+<(&*"!/tx9I9PRVRbRk l lpass_oncp||jjkr|j||j_dS)a Set whether to pass on the worker ID to the function to be executed or not (default= ``False``). :param pass_on: Whether to pass on a worker ID to the target, ``worker_init``, and ``worker_exit`` functions. When enabled, functions receive the worker ID depending on other settings. The order is: ``worker_id``, ``shared_objects``, ``worker_state``, and finally the arguments passed on using ``iterable_of_args`` N)r;r/rAreset)rKrOs rLpass_on_worker_idzWorkerPool.pass_on_worker_idws; d&5 5 5   $ $ & & &*1'''rNcp||jjkr|j||j_dS)ay Set shared objects to pass to the workers. :param shared_objects: Objects to be passed on as shared objects to the workers once. It will be passed on to the target, ``worker_init``, and ``worker_exit`` functions. ``shared_objects`` is only passed on when it's not ``None``. Shared objects will be copy-on-write when using ``fork`` as start method. When enabled, functions receive the shared objects depending on other settings. The order is: ``worker_id``, ``shared_objects``, ``worker_state``, and finally the arguments passed on using ``iterable_of_args``` N)r;r.rArQ)rKr.s rLset_shared_objectszWorkerPool.set_shared_objectss; T-< < <   $ $ & & &*8'''rNcp||jjkr|j||j_dS)aI Set whether or not each worker should have its own state variable. Each worker has its own state, so it's not shared between the workers. :param use_worker_state: Whether to let a worker have a worker state. The worker state will be passed on to the target, ``worker_init``, and ``worker_exit`` functions. When enabled, functions receive the worker state depending on other settings. The order is: ``worker_id``, ``shared_objects``, ``worker_state``, and finally the arguments passed on using ``iterable_of_args`` N)r;r0rArQ)rKr0s rLset_use_worker_statezWorkerPool.set_use_worker_states; t/@ @ @   $ $ & & &,<)))rNc||j_dS)z Set whether workers should be kept alive in between consecutive map calls. :param keep_alive: When True it will keep workers alive after completing a map call, allowing to reuse workers N)r;r2)rKr2s rLset_keep_alivezWorkerPool.set_keep_alives '1###rNc||j_dS)a4 Set whether to provide tasks to the workers in order, such that worker 0 will get chunk 0, worker 1 will get chunk 1, etc. :param order_tasks: Whether to provide tasks to the workers in order, such that worker 0 will get chunk 0, worker 1 will get chunk 1, etc. N)r;r5)rKr5s rLset_order_taskszWorkerPool.set_order_taskss(3$$$rNc|jt|jt|jt|j|j|j j dg|j j z|_ t|j j D]}|||jt#j|jd|_t#j|jd|_t#j|jd|_t#j|jd|_|j|j|j|jdS)zl Spawns the workers and starts them so they're ready to start reading from the tasks queue. NT)targetr,)r?rrQrrrA init_commsrJreset_insightsr;r4r+r@range _start_workerrHclearr8Thread_results_handlerrC_restart_handlerrD_timeout_handlerrE_unexpected_death_handlerrFstartrK worker_ids rL_start_workerszWorkerPool._start_workerss L!''))) I$$&&& I$$&&& %%''' ,,T-=-MNNN!1!88 t/677 * *I   y ) ) ) ) (..000'0'7t?T]a'b'b'b$'0'7t?T]a'b'b'b$'0'7t?T]a'b'b'b$090@Hfos0t0t0t- $**,,, $**,,, $**,,, -3355555rNric :t5|||j|j|j|jt jttj |j |<|jj |j |_ d||j |_ |j | dddn #1swxYwY|jjr2t|j |j|jj|dSdS)z Creates and starts a single worker :param worker_id: ID of the worker :return: Worker instance Worker-N)r!r=r;r<rArJr#get_connection_detailsrtimer@r,namergr-r&pidrhs rLr`zWorkerPool._start_workersG, - - - -'+{{4+T_d>PRVRg2446V6X6XZ^ZcZeZe((DM) $/3.>.EDM) $ +,Ai,A,ADM) $ ) M) $ * * , , , - - - - - - - - - - - - - - -   # ` T]9594;K;ST];^ _ _ _ _ _ ` `sB7CCCc |jd}|D]\}}}t|tr|tkrdS |r#|j|d|n}t|\}}||_|tkr5t|j tthz n|h}|D]$}|j|d|%#t$rYwxYw)z Listen for results from the workers and add it to the cache. Note that when ``set`` is called on a result object, the result is automatically removed from the cache. T)blockNsuccessresultF)rA get_results isinstancestrrr?_setr __cause__rsetkeysrrKeyError) rK results_batchjob_idrtruerr traceback_errjob_ids_job_ids rLrczWorkerPool._results_handlers=   .:::FFM+8  'fc**v/D/DFF Q F+00f0MMMM-?-H*](5 ]cfo\o\oC (8(8(:(:$;$;|Y>W$W$W$*8 '.QQG K055eC5PPPPD + s B"C,, C98C9c<|js|js|jD]}|js|jrdS |j|n#t$rYnwxYw|j|| ||js|jdSdSdSdS)zH Listen for worker restarts and restart them if needed. N) rAexception_thrownrHis_setget_worker_restartsr@joinOSErrorreset_worker_restartr`rhs rLrdzWorkerPool._restart_handlersM$5577 .@`@g@g@i@i .!/CCEE . . %6688D<\>vFFF&'N'N'N'NOOC"$+"2"2"4"455FG")LL F+00s0KKKKFF  JsOOO3$5577 @`@g@g@i@i         s:B B#"B#c  dtttttt tgt ffffd }dtttttt tgt ffffd }dtttttt tgt fff fd }jsDj s,j j cj j WtdjDrt%jdt)jjD]W}j| t0kr|\}}}n) t2kr|\}}}n| \}}}||||r t0t2hvpt5j t6}|rj |t=d|d |d |d } t0kr5t?j tBt2hz n h} | D]$ j "d | %|rdSYt%jdjs j (dSdSdSdS)zA Check for worker_init/task/worker_exit timeouts r6c6djjjjfS)N worker_init)r<worker_init_timeoutrAhas_worker_init_timed_outrKsrL_get_init_configz5WorkerPool._timeout_handler.._get_init_configF $/"EtGYGss srNc6djjjjfS)N worker_exit)r<worker_exit_timeoutrAhas_worker_exit_timed_outrsrL_get_exit_configz5WorkerPool._timeout_handler.._get_exit_configIrrNc djjjjfS#t$rddjjfcYSwxYw)Ntask)r?_timeoutrAhas_worker_task_timed_outr})rrrKs rL_get_task_configz5WorkerPool._timeout_handler.._get_task_configLs^ Rt{62;T=O=iii R R RtT%7%QQQQQ Rs ">>Nc3(K|] }|jduVdSN)r).0jobs rL z.WorkerPool._timeout_handler..Xs)TT ,TTTTTTrNrrl z timed out (timeout=)Frs)#r rxr floatrintboolrArrHrr<rrallr?copyvaluesrnrr_r;r+rrrrwrr_send_kill_signal_to_worker TimeoutErrorr{r|rry) rKrrrritimeout_func_name timeout_varhas_timed_out_func kill_poolrrrs ` @rLrezWorkerPool._timeout_handlerBs t%Xe_hU|UYGY>Z(Z"[ t t t t t t t%Xe_hU|UYGY>Z(Z"[ t t t t t t RsHUOXsTYl\`N`Ea/a)b R R R R R R R $5577. @`@g@g@i@i. 3;O7?TT 8H8H8J8J8Q8Q8S8STTTTT@ 3"4#3#:;;   +EEiPPY&&IYIYI[I[F%{4F4Fy((IYIYI[I[F%{4F4FIYIYZ`IaIaF%{4F*/A/A)[/Y/Y* 9i"88V"4;v#68TUU!K*BB6JJJ44Y???''r'r'r=N'r'rdo'r'r'rssCW]ajWjWjs4;#3#3#5#566, 9RRRqwpxG")LL F+00s0KKKK  JsOOO]$5577. @`@g@g@i@i. . . . . . . . . rNcJ|jtS)zu Obtain a list of exit results when an exit function is defined. :return: Exit results list )r?rrvrs rLget_exit_resultszWorkerPool.get_exit_resultss {9%11333rNc|S)z; Enable the use of the ``with`` statement. rs rL __enter__zWorkerPool.__enter__s  rN_c.|dS)zk Enable the use of the ``with`` statement. Gracefully terminates workers, if there are any N) terminate)rKrs rL__exit__zWorkerPool.__exit__s rNfunciterable_of_args iterable_lenmax_tasks_active chunk_sizen_splitsworker_lifespan progress_barconcatenate_numpy_outputrr task_timeoutrrprogress_bar_optionsprogress_bar_stylec\|jtr.s+VV$Hd#VVVVVVrNcg|] }|d S)r)rrus rL z"WorkerPool.map..s```&)```rNc|dS)Nrr)rus rLz WorkerPool.map..s U[\]U^rN)keyr)rAsignal_keep_orderNUMPY_INSTALLEDrwnpndarrayr$r;r+hasattrr map_unordered enumerateclear_keep_ordersorted concatenate)rKrrrrrrrrrrrrrrrrresultssorted_resultss rLmapzWorkerPool.mapssd ,,...  qz*:BJGG qCWXhjvXbdlX\XhXoDqDq @ lJ  G,!,bj99N~...?M OrNcbt|||||||||| | | | | ||S)a Same as ``multiprocessing.map()``, but unordered. Also allows a user to set the maximum number of tasks available in the queue. :param func: Function to call each time new task arguments become available. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param iterable_of_args: A numpy array or an iterable containing tuples of arguments to pass to a worker, which passes it to the function ``func`` :param iterable_len: Number of elements in the ``iterable_of_args``. When chunk_size is set to ``None`` it needs to know the number of tasks. This can either be provided by implementing the ``__len__`` function on the iterable object, or by specifying the number of tasks :param max_tasks_active: Maximum number of active tasks in the queue. If ``None`` it will be converted to ``n_jobs * chunk_size * 2`` :param chunk_size: Number of simultaneous tasks to give to a worker. When ``None`` it will use ``n_splits``. :param n_splits: Number of splits to use when ``chunk_size`` is ``None``. When both ``chunk_size`` and ``n_splits`` are ``None``, it will use ``n_splits = n_jobs * 64``. :param worker_lifespan: Number of tasks a worker can handle before it is restarted. If ``None``, workers will stay alive the entire time. Use this when workers use up too much memory over the course of time :param progress_bar: When ``True`` it will display a progress bar :param worker_init: Function to call each time a new worker starts. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param worker_exit: Function to call each time a worker exits. Return values will be fetched and made available through :obj:`mpire.WorkerPool.get_exit_results`. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param task_timeout: Timeout in seconds for a single task. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). Note: the timeout doesn't apply to ``worker_init`` and ``worker_exit`` functions, use `worker_init_timeout` and `worker_exit_timeout` for that, respectively :param worker_init_timeout: Timeout in seconds for the ``worker_init`` function. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). :param worker_exit_timeout: Timeout in seconds for the ``worker_exit`` function. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). :param progress_bar_options: Dictionary containing keyword arguments to pass to the ``tqdm`` progress bar. See ``tqdm.tqdm()`` for details. The arguments ``total`` and ``leave`` will be overwritten by MPIRE. :param progress_bar_style: The progress bar style to use. Can be one of ``None``, ``'std'``, or ``'notebook'`` :return: List with unordered results )listimap_unordered)rKrrrrrrrrrrrrrrrs rLrzWorkerPool.map_unorderedsRfD''.> N^`j(0/\}} ||vr| |V|dz }nn#||kr |V|dz }9|||<?t|D]}| |V|jdS)a Same as ``multiprocessing.imap_unordered()``, but ordered. Also allows a user to set the maximum number of tasks available in the queue. :param func: Function to call each time new task arguments become available. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param iterable_of_args: A numpy array or an iterable containing tuples of arguments to pass to a worker, which passes it to the function ``func`` :param iterable_len: Number of elements in the ``iterable_of_args``. When chunk_size is set to ``None`` it needs to know the number of tasks. This can either be provided by implementing the ``__len__`` function on the iterable object, or by specifying the number of tasks :param max_tasks_active: Maximum number of active tasks in the queue. If ``None`` it will be converted to ``n_jobs * chunk_size * 2`` :param chunk_size: Number of simultaneous tasks to give to a worker. When ``None`` it will use ``n_splits``. :param n_splits: Number of splits to use when ``chunk_size`` is ``None``. When both ``chunk_size`` and ``n_splits`` are ``None``, it will use ``n_splits = n_jobs * 64``. :param worker_lifespan: Number of tasks a worker can handle before it is restarted. If ``None``, workers will stay alive the entire time. Use this when workers use up too much memory over the course of time :param progress_bar: When ``True`` it will display a progress bar :param worker_init: Function to call each time a new worker starts. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param worker_exit: Function to call each time a worker exits. Return values will be fetched and made available through :obj:`mpire.WorkerPool.get_exit_results`. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param task_timeout: Timeout in seconds for a single task. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). Note: the timeout doesn't apply to ``worker_init`` and ``worker_exit`` functions, use `worker_init_timeout` and `worker_exit_timeout` for that, respectively :param worker_init_timeout: Timeout in seconds for the ``worker_init`` function. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). :param worker_exit_timeout: Timeout in seconds for the ``worker_exit`` function. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). :param progress_bar_options: Dictionary containing keyword arguments to pass to the ``tqdm`` progress bar. See ``tqdm.tqdm()`` for details. The arguments ``total`` and ``leave`` will be overwritten by MPIRE. :param progress_bar_style: The progress bar style to use. Can be one of ``None``, ``'std'``, or ``'notebook'`` :return: Generator yielding ordered results rNrc3$K|] \}}||fV dSrrrs rLrz"WorkerPool.imap..YsF=]=]R`RZ\`h=M=]=]=]=]=]=]rNTr)rArrrwrrr$r;r+rrrrpoprr|r)rKrrrrrrrrrrrrrrrnext_result_idx tmp_results result_idxrus rLimapzWorkerPool.imapsb ,,...  qz*:BJGG qCWXhjvXbdlX\XhXoDqDq @ lJ   G,t|tjr$t |||||jj\}}}}d}t|j|||||||||| | | \}}}}}t|| | ||| | | }|st||||}t|}| }d}d} |j rg|j t|jtdt%d|d|_ |rt)j|jj}|jrI|j s0t2d|d|jr,|j|kr!||_|j ||js||_|t?|j|| }|j }tC|j|j||||j |j"5|_# d }d } tI|}|tK|z }n#tL$rYnwxYw|j 'sz|tK|z|krd |$dd V|d z}n#tPj)$rYnwxYw|j 's|tK|z|kd|j 'rn/|j *|||tK|z }|j 's/|+||j#,||j 'sT |$dd Vn #tPj)$rYntL$rYnwxYw|j 'T|j 'r|||jj-|r|j .n$#t^$r|YnwxYwdddn #1swxYwY|r(|0|t)j1||2d|_#d|_ |j 3nl#|r(|0|t)j1||2d|_#d|_ |j 3wxYw|jj4r3t25|j"6dSdS)a Same as ``multiprocessing.imap_unordered()``. Also allows a user to set the maximum number of tasks available in the queue. :param func: Function to call each time new task arguments become available. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param iterable_of_args: A numpy array or an iterable containing tuples of arguments to pass to a worker, which passes it to the function ``func`` :param iterable_len: Number of elements in the ``iterable_of_args``. When chunk_size is set to ``None`` it needs to know the number of tasks. This can either be provided by implementing the ``__len__`` function on the iterable object, or by specifying the number of tasks :param max_tasks_active: Maximum number of active tasks in the queue. If ``None`` it will be converted to ``n_jobs * chunk_size * 2`` :param chunk_size: Number of simultaneous tasks to give to a worker. When ``None`` it will use ``n_splits``. :param n_splits: Number of splits to use when ``chunk_size`` is ``None``. When both ``chunk_size`` and ``n_splits`` are ``None``, it will use ``n_splits = n_jobs * 64``. :param worker_lifespan: Number of tasks a worker can handle before it is restarted. If ``None``, workers will stay alive the entire time. Use this when workers use up too much memory over the course of time :param progress_bar: When ``True`` it will display a progress bar :param worker_init: Function to call each time a new worker starts. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param worker_exit: Function to call each time a worker exits. Return values will be fetched and made available through :obj:`mpire.WorkerPool.get_exit_results`. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param task_timeout: Timeout in seconds for a single task. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). Note: the timeout doesn't apply to ``worker_init`` and ``worker_exit`` functions, use `worker_init_timeout` and `worker_exit_timeout` for that, respectively :param worker_init_timeout: Timeout in seconds for the ``worker_init`` function. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). :param worker_exit_timeout: Timeout in seconds for the ``worker_exit`` function. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). :param progress_bar_options: Dictionary containing keyword arguments to pass to the ``tqdm`` progress bar. See ``tqdm.tqdm()`` for details. The arguments ``total`` and ``leave`` will be overwritten by MPIRE. :param progress_bar_style: The progress bar style to use. Can be one of ``None``, ``'std'``, or ``'notebook'`` :return: Generator yielding unordered results FTNz0Cannot call 'map' while another 'map' is runningrszHWorkerPool parameters changed while keep_alive=True. Restarting workers.r2timeoutr{Gz?)rrrrr)7rrwrrr$r;r+rrr%r"get_lockrBrArrr?ryr_handle_exceptionr# start_managerr3r@is_initializedloggerwarning stop_and_joinr<add_new_map_paramsrjrrr rJrInextr StopIterationrqueueEmptyadd_task set_length set_new_totalr2#wait_until_progress_bar_is_completeKeyboardInterruptset_lock stop_managerremove_from_cachereset_progressr4debugget_insights_string)rKrrrrrrrrrrrrrrriterator_of_chunked_argsnumpy_chunkingn_tasksnew_map_paramstqdmoriginal_tqdm_locktqdm_manager_owner imap_iteratorrn_activechunk_of_taskss rLrzWorkerPool.imap_unorderedxsf$&   "z*:BJGG "K_ , HdFVF]LL H $lJ"NUi  . >NPZ\dfu .0BLRe U U Q!:|=Q ){KZfht)<>QSS d'23CWjZb'c'c $*++!]]__" c 0  )"::<HHH L)..u6BCu6v6v/xxx&&((( $D  Z%0%>t?O?X%Y%Y"} 5T%7%F%F%H%H 5ijjj""e"444} F$/^"C"C"0"55nEEE= &"0##%%%9gWcdddM")F$D$4do|Ui$68J$($9;;8 ->B>X5- HG8"-12J-K-KN#s>':'::GG,"""!E"$(#5#F#F#H#H%'#n*=*==@PPP%&3&8&8tT&8&R&R R R R (A #(;%%% $% $(#5#F#F#H#H%'#n*=*==@PPP ->>@@"!*33FNKKK C$7$77/84 ->>@@J%009992@@III"0AACC"""/"4"44"4"M"MMMMM${!!! D,"""!E" #0AACC")::<<1..000&&$2B2M&NNN$Q*NNPPP(---**,,,,,-o8 -8 -8 -8 -8 -8 -8 -8 -8 -8 -8 -8 -8 -8 -8 -v" + 0111(***(//111)-D & %D    - - / / / /" + 0111(***(//111)-D & %D    - - / / / /   + F LL.BBDD E E E E E F FsFT!R. Q>!I43Q>4 J>Q>J2Q>4KQ>K%"Q>$K%%CQ>OQ>O9+Q>- O96Q>8O99BQ>=R.>RR.RR." T!.R22T!5R26T!!A)V rrkwargscallbackerror_callbackc b|||||||||| | S)a Apply a function to a single task. This is a blocking call. :param func: Function to apply to the task. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param args: Arguments to pass to a worker, which passes it to the function ``func`` as ``func(*args)`` :param kwargs: Keyword arguments to pass to a worker, which passes it to the function ``func`` as ``func(**kwargs)`` :param callback: Callback function to call when the task is finished. The callback function receives the output of the function ``func`` as its argument :param error_callback: Callback function to call when the task has failed. The callback function receives the exception as its argument :param worker_init: Function to call each time a new worker starts. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param worker_exit: Function to call each time a worker exits. Return values will be fetched and made available through :obj:`mpire.WorkerPool.get_exit_results`. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param task_timeout: Timeout in seconds for a single task. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). Note: the timeout doesn't apply to ``worker_init`` and ``worker_exit`` functions, use `worker_init_timeout` and `worker_exit_timeout` for that, respectively :param worker_init_timeout: Timeout in seconds for the ``worker_init`` function. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). :param worker_exit_timeout: Timeout in seconds for the ``worker_exit`` function. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). :return: Result of the function ``func`` applied to the task ) apply_asyncget) rKrrrrrrrrrrs rLapplyzWorkerPool.apply0sDDdFHnk[f ,.ACVXXX[X[X]X] ^rNc |js/t|||dd|| | |_|t |j|||} |j| j|||| S)a Apply a function to a single task. This is a non-blocking call. :param func: Function to apply to the task. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param args: Arguments to pass to a worker, which passes it to the function ``func`` as ``func(*args)`` :param kwargs: Keyword arguments to pass to a worker, which passes it to the function ``func`` as ``func(**kwargs)`` :param callback: Callback function to call when the task is finished. The callback function receives the output of the function ``func`` as its argument :param error_callback: Callback function to call when the task has failed. The callback function receives the exception as its argument :param worker_init: Function to call each time a new worker starts. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param worker_exit: Function to call each time a worker exits. Return values will be fetched and made available through :obj:`mpire.WorkerPool.get_exit_results`. When passing on the worker ID the function should receive the worker ID as its first argument. If shared objects are provided the function should receive those as the next argument. If the worker state has been enabled it should receive a state variable as the next argument :param task_timeout: Timeout in seconds for a single task. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). Note: the timeout doesn't apply to ``worker_init`` and ``worker_exit`` functions, use `worker_init_timeout` and `worker_exit_timeout` for that, respectively :param worker_init_timeout: Timeout in seconds for the ``worker_init`` function. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). :param worker_exit_timeout: Timeout in seconds for the ``worker_exit`` function. When the timeout is exceeded, MPIRE will raise a ``TimeoutError``. Use ``None`` to disable (default). :return: Result of the function ``func`` applied to the task NFr) r@rr<rjr r?rAadd_apply_taskr) rKrrrrrrrrrrrus rLrzWorkerPool.apply_asyncUsH} "-dKdTY[g.ACVXXDO    ! ! !T[(NLYYY ))&-tVLLL rNc|jr>|j|j}|j}n/|jtt}|}|j M|j |p||j j %|j j d|_ | |j|)zM Handles exceptions thrown by workers and KeyboardInterrupts N)rArr?get_exception_thrown_job_id get_exceptionrzrrrrI set_exceptionthreadrrr)rK exceptioncauses rLrzWorkerPool._handle_exceptions   . . 0 0  D$6$R$R$T$TUcceeI'EE   6 6| D D D)++IE  % 1  & 4 4U5Gi H H H)0<*166888-1*  ++---rNc|jrC|r|jn|jt j|jj|f}d|_||j sD| d| sn|j D|j r| |st|jD]\}} | n#t$rwxYwt|drud}| r9|dkr3t!jd|dz}| r|dk3 |#t$rYwxYwg|_|jd |j r| |s3||jd d Sd Sd S) a When ``keep_alive=False``: inserts a poison pill, grabs the exit results, waits until the tasks/results queues are done, and waits until all workers are finished. When ``keep_alive=True``: inserts a non-lethal poison pill, and waits until the tasks/results queues are done. ``join``and ``stop_and_join`` are aliases. :param keep_alive: Whether to keep the workers alive r\rTrrcloserrrFN)r@rAinsert_non_lethal_poison_pillinsert_poison_pillr8rbjoin_task_queuesr,rgrrrrrrrrnrr+join_results_queues_stop_handler_threads)rKr2twidworker_process try_counts rLrzWorkerPool.stop_and_joins =8 I 8"@@BBBB"55777 (:(KS]R_```AAH GGIII(99;; t$$$zz||(99;; !2244 )&&((( #+4T]+C+C!!'C&++----% ~w77!$% ,5577+IMM Jt,,,%NI-5577+IMM!*002222)!!! D!!!#   2 2d 2 C C C!2244 )&&((( I**,,,"66%6HHHHHq8 I8 Il I Is D55 E$F99 GGcxjsdSjsSjtjtdtdj'j tdj j dkrj}ng}tj }|tj jD]H}tjj||f}|||I|D]}|jg_fdtt0t2fD_dS)z Tries to do a graceful shutdown of the workers, by interrupting them. In the case processes deadlock it will send a sigkill. NFzPool was terminatedrsr8r*c,i|]}|j|Sr)r?)rrrKs rL z(WorkerPool.terminate..s"]]]sDK,]]]rN)r@rArrrr?ryrrIr%r;r1r8rGr{r_r+rb_terminate_workerrgappendrr1 drain_queuesrr)rKthreadsdont_wait_eventrir2s` rLrzWorkerPool.terminates }  F!2244 f   6 6| D D D K % * *5NcAdAd * e e e  % 1  & 4 4\BW5X5X Y Y Y   (K 7 7mGGG'o//O    ! ! !"4#3#:;; " " $D,B)UdIefff q!!!!  A FFHHHH ""$$$ ''))) ]]]]yR[8\]]] rNr=c|j||j|jdS||d}|dkr |j|d|j|snjn#t $rYdSt $rYnwxYw|j||dz}| s| |dk |j|r>|j| |j|n#t $rYdSt $rYnwxYwt|j|dr? |j| dS#t $rYdSt $rYdSwxYwdS)a Terminates a single worker process. When a process.join() raises an AssertionError, it means the worker hasn't started yet. In that case, we simply return. A ValueError can be raised on Windows systems. :param worker_id: Worker ID :param dont_wait_event: Event object to indicate whether other termination threads should continue. I.e., when we set it to False, threads should wait. N rrrrr+)r@rprrrAssertionErrorrrA$drain_results_queue_terminate_workerrwaitrrr+)rKrir=r5s rLr9zWorkerPool._terminate_workers- = # +t}Y/G/K/S F ((333 !mm  i(--c-:::}Y/88::!          C CO T T T NI"))++ '$$&&&!mm& }Y'0022 0 i(22444 i(--///    FF    D  4=+W 5 5   i(..00000!          sJAB B BB,AE E# E#"E#F## F=0 F=<F=cts|jjdkr|j|5|j|rc|j|d tj|j |j tj n#ttf$rYnwxYwddddS#1swxYwYdSdSdS)z Sends a kill signal to a worker process, but only if we know it's running a task. :param worker_id: Worker ID r8FN)rr;r1rAget_worker_running_task_lockget_worker_running_taskset_worker_running_taskoskillr@rpsignalSIGUSR1ProcessLookupErrorrrhs rLrz&WorkerPool._send_kill_signal_to_workerMs$ 4#3#@K#O#O#@@KK  %==iHH&>>y%PPP i 8 > @ @ @  ( - - / / /+/D (  ' 38T8]8]8_8_ 3  ( - - / / /+/D (  1 =5>>@@ >  1 6 6 8 8 848D 1 1 1 > = = =rNcRt|jdS)z, Prints insights per worker N)printrJrrs rLprint_insightszWorkerPool.print_insights~s' d#7799:::::rNc4|jS)zt Creates insights from the raw insight data :return: Dictionary containing worker insights )rJ get_insightsrs rLrSzWorkerPool.get_insightss $11333rN)Tr)r6N)r6r*)NNNNNFTNNNNNNN) NNNNNFNNNNNNN) rNNNNNNNN)F)4__name__ __module__ __qualname____doc__rr rrrrrxrMrRrTrVrXrZrjr`rcrdrfrerrrrrr r rrrrrrrrrr rrrrrr8rGr9rr1rQrSrrNrLr*r*!s 04D]a'+Edi%9efk).EOmOmx}OmTOmSZOm!$Om=AOm]aOm"OmGKOm_cOm#'Om>BOmOSOmOmOmOmb 2 2 2 2 2 2 2 9 9 9 9 9 9 9 = =T =T = = = =111111133434333366668`s`t````,@....2####J>>>>@4$4444 34 koptptVZY]jn04 KOKOKOE%/4JKOZbcfZgKO&smKO@H KO`hil`mKO%c]KOAEKOimKO"(+KOBJ(ASKO#5/ KOHPPU KO "*% KO PXX\]`be]eXfOg KO !) KO := KOKOKOKO\]aRVRW`dcg=AGK:>6=6=(6=eE8O>T6=$,SM6=LTUXM6="*3-6=BJ3-6=(0}6=LP6=$,H#5 6=LTT\K] 6= %-UO 6= RZZ_Q` 6= ,4E? 6=-5T#s(^,D6=+33-6=DG6=6=6=6=plpqurvUYae>B15 [.[.[.U5(?5K[.[cdg[h[.'}[.AI#[.aijman[.&sm[.BF[.]efn]o[.#8,[.DLE?[.#+5/ [.PXX] [. $,DcN#; [. "*# [. ;DCtO:T [.[.[.[.|^bSWSXaedh>BHL;?vFvF8vFuUH_?UvF%-c]vFMUVY]vF#+C=vFCKC=vF)1 vFMQvF%-X$6 vFMUU]L^ vF &.e_ vF S[[`Ra vF -5UO vF.6d38n-EvF,4C=vFENcSWY]oD^vFvFvFvFpim[_VZbf#^#^(#^##^D#^S[\dSe#^&x0#^FNxFX#^#H-#^DLUO#^$,E?#^QYY^P_#^lo#^#^#^#^Josae\`;?;? ,,,,$,YabjYk,$,X$6,LTU]L^,!)(!3,JRSX/,*2%,*2% ,EP ,,,,\:BIBIBIBIBIBIBIH D+^+^+^+^Z8388UY8888tST(9999:;;;; 4d444444rNr*)AloggingrGrrIr8rntypingrrrrrrr r r r numpyrr ImportErrormpire.async_resultr rrrr mpire.commsrrrrr mpire.contextrr mpire.dashboard.connection_utilsrmpire.exceptionrmpire.insightsr mpire.paramsrrrrmpire.progress_barr mpire.signalr!mpire.tqdm_utilsr"r# mpire.utilsr$r%r& mpire.workerr'r( getLoggerrTrr*rrNrLrisw  ````````````````````````OO BOOO``````````````TTTTTTTTTTTTTT????????MMMMMM......))))))YYYYYYYYYYYY11111177777722222222KKKKKKKKKK44444444  8 $ $i4i4i4i4i4i4i4i4i4i4s9 AA