wiG;#8ddlZddlZddlZddlZddlmZmZddlm Z m Z m Z m Z m Z mZmZmZmZmZddlmZddlmZmZddlmZe eee efZedd Gd d Zedd Gd d Zdedeee fdeedeedeeeefdeedeededee e e fdee deedeedeedeeeeeeee e e fffdZ!deee fdeedeefdZ" d'de de d eed!fd"ed#eeddf d$Z#dee e e fd%eedee de e e ffd&Z$dS)(N) dataclassfield) AnyCallableDictIterableListOptionalSizedTupleTypeUnion) TqdmKeyError)DEFAULT_START_METHOD RUNNING_MACOS)get_tqdmTF)initfrozenceZdZUdZeeed<eddZeed<ee ed<eddZ e ed<dZ e ed <d Z eed <dZe ed <dZe ed <eZeed<dZe ed<dZe ed<dZe ed<dZe ed<edeefdZejdeedd fdZede fdZejdee dd fdZdee de fdZd S)WorkerPoolParamsz@ Data class for all :obj:`mpire.WorkerPool` parameters. n_jobsF)rrepr_n_jobscpu_ids_cpu_idsTdaemonNshared_objectspass_worker_iduse_worker_state start_method keep_aliveuse_dillenable_insights order_tasksreturnc|jSN)rselfs d/var/www/development/aibuddy-work/election-extract/venv/lib/python3.11/site-packages/mpire/params.pyrzWorkerPoolParams.n_jobs$s |c:|ptj|_dSr')mp cpu_countr)r)rs r*rzWorkerPoolParams.n_jobs(s/ r+c|jSr')rr(s r*rzWorkerPoolParams.cpu_ids,s }r+c:|||_dSr')_check_cpu_idsr)r)rs r*rzWorkerPoolParams.cpu_ids0s++G44 r+cg}|rtrtjdtt |dkr>t ||jkr&t dt ||jfzd}d}|D]}t|trd| |t|td|D}t|td|D}{t|tr7| |gt||}t||}td|tjkr(t d|tjdz fz|dkrt d t |dkr-tt!j|d|j}|S) ai Checks the cpu_ids parameter for correctness :param cpu_ids: List of CPU IDs to use for pinning child processes to specific CPUs. The list must be as long as the number of jobs used (if ``n_jobs`` equals ``None`` it must be equal to ``mpire.cpu_count()``), or the list must have exactly one element. In the former case, element x specifies the CPU ID(s) to use for child process x. In the latter case the single element specifies the CPU ID(s) for all child processes to use. A single element can be either a single integer specifying a single CPU ID, or a list of integers specifying that a single child process can make use of multiple CPU IDs. If ``None``, CPU pinning will be disabled. Note that CPU pinning may only work on Linux based systems :return: cpu_ids zJSetting CPU affinity is not supported on MacOS. Ignoring cpu_ids parameterz9Number of CPU IDs (%d) does not match number of jobs (%d)rc3K|]}|VdSr'.0cpus r* z2WorkerPoolParams._check_cpu_ids..S"4K4KSS4K4K4K4K4K4Kr+c3K|]}|VdSr'r5r6s r*r9z2WorkerPoolParams._check_cpu_ids..Tr:r+z3CPU ID(s) must be either a list or a single integerzACPU ID %d exceeds the maximum CPU ID available on your system: %dzCPU IDs cannot be negative)rwarningswarnRuntimeWarninglenr ValueError isinstancelistappendmaxminint TypeErrorr-r. itertoolsrepeat)r)rconverted_cpu_ids max_cpu_id min_cpu_idcpu_ids r*r1zWorkerPoolParams._check_cpu_ids4s   ? . j,...7||q  S\\T[%@%@ !\"%g,, !<"=>>>JJ! [ [fd++ [%,,V444!$Z4K4KF4K4K4K1K1K!L!LJ!$Z4K4KF4K4K4K1K1K!L!LJJ,,[%,,fX666!$Z!8!8J!$Z!8!8JJ#$YZZZR\^^++ !d",blnnq.@!A"BCCCA~~ !=>>>  ! !Q & & $Y%56G6JDK%X%X Y Y   r+)__name__ __module__ __qualname____doc__r rF__annotations__rrCPUListrrboolrrrrrr strr!r"r#r$propertyrsetterrr1r5r+r*rrs SM5e%000GS000 g 5u555Hg555FDNC ND   "d""",L#,,,JHd!OT!!!K  X ]0Xc]0t000]0X ^5x05T555^55!hw&75!G5!5!5!5!5!5!r+rceZdZUdZeed<dZeeed<dZeeed<dZ ee ed<dZ e ed<dZ eeed <dZeeed <dZeeed <d dd e fdZdS)WorkerMapParamszk Data class for all :meth:`mpire.WorkerPool.map` parameters that need to be passed on to a worker. funcN worker_init worker_exitworker_lifespanF progress_bar task_timeoutworker_init_timeoutworker_exit_timeoutotherr%cz|j|jks|j|jkrtjdtd|j|jkoo|j|jko_|j|jkoO|j|jko?|j|jko/|j|jko|j |j ko|j |j kS)zg :param other: Other WorkerMapConfig :return: Whether the configs are the same aYou're changing either the worker_init and/or worker_exit function while keep_alive is enabled. Be aware this can have undesired side-effects as worker_init functions are only executed when a worker is started and worker_exit functions when a worker is terminated. stacklevel) r[r\r<r=r>rZr]r^r_r`ra)r)rbs r*__eq__zWorkerMapParams.__eq__s   0 0 0E4EIY4Y4Y Mu)Q 8 8 8 8  di'F!T%55F!T%55F%)==F"d&77 F "d&77 F )T-EE F)T-EE Gr+)rNrOrPrQrrRr[r r\r]rFr^rTr_floatr`rargr5r+r*rYrYls NNN&*K(#***&*K(#***&*OXc])))L$%)L(5/(((+/%///+/%///G-G$GGGGGGr+rY pool_paramsiterable_of_args iterable_lenmax_tasks_active chunk_sizen_splitsr]r^progress_bar_optionsprogress_bar_styler_r`rar%c bt||} t|dttfddt|dtfdd|8|| | |z }n.| t jdt d d }n | |jd zz }|-|jttj |zdz}nt|d tfddt|d tfddt|| | }| df| df| dffD]%\}}t||ttfdd&| ||||fS)aV Check the parameters provided to any (i)map function. Also extracts the number of tasks and can modify the ``chunk_size`` and ``progress_bar`` parameters. :param pool_params: WorkerPool config :param iterable_of_args: A numpy array or an iterable containing tuples of arguments to pass to a worker :param iterable_len: Number of elements in the ``iterable_of_args`` :param max_tasks_active: Maximum number of active tasks in the queue. Use ``None`` to not limit the queue :param chunk_size: Number of simultaneous tasks to give to a worker. If ``None`` it will generate ``n_jobs * 4`` number of chunks :param n_splits: Number of splits to use when ``chunk_size`` is ``None`` :param worker_lifespan: Number of chunks a worker can handle before it is restarted. If ``None``, workers will stay alive the entire time. Use this when workers use up too much memory over the course of time :param progress_bar: When ``True`` it will display a progress bar :param progress_bar_options: Dictionary containing keyword arguments to pass to the ``tqdm`` progress bar. See ``tqdm.tqdm()`` for details. The arguments ``total`` and ``leave`` will be overwritten by MPIRE. :param progress_bar_style: Style to use for the progress bar :param task_timeout: Timeout in seconds for a single task :param worker_init_timeout: Timeout in seconds for the ``worker_init`` function :param worker_exit_timeout: Timeout in seconds for the ``worker_exit`` function :return: Number of tasks, max tasks active, chunk size, progress bar, progress bar options rmTr3) allowed_types none_allowedmin_rnNzFailed to obtain length of iterable when chunk size or number of splits is None. Chunk size is set to 4. Remedy: either provide an iterable with a len() function or specify iterable_len in the function callrdre@rlr]r_r`rag:0yE>) get_number_of_tasks check_numberrFrhr<r=r>rmathceilcheck_progress_bar_options)rirjrkrlrmrnr]r^rorpr_r`ran_tasks timeout_vartimeout_var_names r*check_map_parametersrs>""2LAAG\#uTX_`aaaa:cV$UVWWWW  G$7 8+JJ BCQ^_aaaa $ (:R(?@ &-DIj4I4I0J0JJQN%'9#^bijkkkk"3C6X\cdeeee66JGUghh,8*H+>@U*V+>@U*V*Xnn% % ["23,]ahlmmmmm $j,@T TTr+cL||St|drt|SdS)a Get the number of tasks to process. If iterable_len is provided, it will be used. Otherwise, if iterable_of_args is a Sized object, len(iterable_of_args) will be used. Otherwise, None will be returned. :param iterable_of_args: A numpy array or an iterable containing tuples of arguments to pass to a worker :param iterable_len: Number of elements in the ``iterable_of_args`` :return: Number of tasks to process N__len__)hasattrr?)rjrks r*rwrws7++%#$$$ 4r+varvar_namerr.rsrtc|r|dSt||st|d||||krt|d|dSdS)aq Check that a variable is of the correct type and within the allowed range :param var: Variable to check :param var_name: Name of the variable :param allowed_types: Allowed types for the variable :param none_allowed: Whether None is allowed for the variable :param min_: Minimum value for the variable. If None, no minimum value is checked Nz should be of type z should be >= )rArGr@)rrrrrsrts r*rxrxsz  c= ) )I8GG GGHHH C$JJH::D::;;;JJr+r|c|pi}t|tstdd|vrtjdt dd|vrtjdt d|dkrd |vrt d d d d dd||d d}t|} ||n)#ttf$r}|tdd}~wwxYw|S)a Check that the progress bar options are properly formatted and set some defaults :param progress_bar_options: Dictionary containing keyword arguments to pass to the ``tqdm`` progress bar. See ``tqdm.tqdm()`` for details. The arguments ``total`` and ``leave`` will be overwritten by MPIRE. :param n_tasks: Number of tasks to process :param progress_bar_style: Progress bar style to use :return: Dictionary containing the progress bar options z+progress_bar_options should be a dictionarytotalzThe 'total' keyword argument is overwritten by MPIRE. Set the total number of tasks to process using the iterable_len parameterrdreleavez9The 'leave' keyword argument will be overwritten by MPIRErichpositionzJThe 'position' parameter is currently not supported for rich progress barsrTg?g?)r dynamic_ncols mininterval maxinterval)rrzThere's an error in progress_bar_options. Either one of the parameters doesn't exist or it's not properly formatted. See tqdm.tqdm() for details.N) rAdictrGr<r=r>NotImplementedErrorr check_optionsrr@)ror|rptqdmes r*r{r{sj052 *D 1 1GEFFF&&& 9:HUV X X X X&&& QSanoppppV## 6J(J(J!"nooo        & ' 'D ` /0000 ) $```Z!_`` `` s B66CCCr')%rHrymultiprocessingr-r< dataclassesrrtypingrrrrr r r r r rrr mpire.contextrrmpire.tqdm_utilsrrFrSrrYrhrTrUrrwrxr{r5r+r*rs (((((((([[[[[[[[[[[[[[[[[[[[[[[[========%%%%%% uS$s)^$ % U###W!W!W!W!W!W!W!$#W!t T"""&G&G&G&G&G&G&G#"&GRBU&6BU%PUW_P_J`BU'/}BUHPQT BU%-eCJ.?%@BULTUXMBU+33-BUHLBU08S#X/G BU^ffi]j BU (0 BU NVV[_ BU /7uo BU #Xc]D$sCx.H IBUBUBUBUJ%x*@PXY\P]bjknbo&*.<