wil/ ddlZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZ ddlZdZn#e$rdZdZYnwxYwdd lmZmZmZer8 ddlZddlZddl Z dZ!dZ"n&#e$rZ#dZ!e#Z"e"xj$d z c_$YdZ#[#ndZ#[#wwxYwd e%d ee%d dfdZ& d"dedee%deee%e'fdee%d eeddff dZ( d#dedee%dee%dee%dee%d eee%e%dff dZ) d#dedee%dee%dee%dee%d e%f dZ*d$dede+d eeeffdZ,deee%e'fde+d e-fdZ.GddZ/de+d e fdZ0Gd d!Z1dS)%N) timedelta) cpu_count) SyncManager)SynchronizedArray)Callable Collection GeneratorIterableListOptionalTupleUnionTF) RUNNING_MACOSRUNNING_WINDOWSmp_dillzZ If you're using Conda, you can run `conda install pywin32` to install the missing module.pidmaskreturnctrTtstd}|D] }|d|zz} tjt jd|}tj||dStrdStj ||dS)a> Sets the CPU affinity for a given process. On Windows-based systems with more than 64 processors, I'm not sure if this will work. See https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-setprocessaffinitymask#parameters. :param pid: Process ID :param mask: List of CPU IDs rTN) rWIN32API_AVAILABLEWIN32API_ERRORwin32api OpenProcesswin32conPROCESS_ALL_ACCESS win32processSetProcessAffinityMaskrossched_setaffinity)rr windows_maskcpu_idhandles c/var/www/development/aibuddy-work/election-extract/venv/lib/python3.11/site-packages/mpire/utils.pyset_cpu_affinityr%$s(" !   ( (F AK 'LL%h&A4MM+FLAAAAA (  S$'''''iterable_of_args iterable_len chunk_sizen_splitsc #K||td|9||}n/t|drt|}ntd||z }t|}|}d} trHt |t jr.|||tdtj |z}nBttj |tdtj |}t|dkrdS|+|t|z|kr|d||z }|r|VdS|V||ztj |z }|t|z })a Chunks tasks such that individual workers will receive chunks of tasks rather than individual ones, which can speed up processing drastically. :param iterable_of_args: A numpy array or an iterable containing tuples of arguments to pass to a worker, which passes it to the function :param iterable_len: Number of tasks available in ``iterable_of_args``. Only needed when ``iterable_of_args`` is a generator :param chunk_size: Number of simultaneous tasks to give to a worker. If ``None``, will use ``n_splits`` to determine the chunk size :param n_splits: Number of splits to use when ``chunk_size`` is ``None`` :return: Generator of chunked task arguments Nz+chunk_size and n_splits cannot both be None__len__zqEither iterable_len or an iterable with a len() function should be provided when chunk_size and n_splits are NonerT) ValueErrorhasattrleniterNUMPY_INSTALLED isinstancenpndarraymaxmathceiltuple itertoolsislice) r'r(r)r*n_tasks args_itercurrent_chunk_sizen_elements_returnedchunks r$ chunk_tasksrACs h.FGGG  #"GG %y 1 1 A*++GG@AA Ax' %&&I#*  ^z*:BJGG ^$%89LsSTVZV_`rVsVsOtOt9t%tuEE)*9c!TYGY=Z=Z6[6[\\]]E u::?? F  #(;c%jj(H<(W(W=<*===>E  F 0:=K]A^A^^s5zz))*r&n_jobsc | |d|}t|||||}tt|t|||p||dznd}d}d}||||fS)a If we're dealing with numpy arrays, chunk them using numpy slicing and return changed map parameters :param iterable_of_args: A numpy array or an iterable containing tuples of arguments to pass to a worker, which passes it to the function :param iterable_len: When chunk_size is set to ``None`` it needs to know the number of tasks. This can either be provided by implementing the ``__len__`` function on the iterable object, or by specifying the number of tasks :param chunk_size: Number of simultaneous tasks to give to a worker. If ``None``, will generate ``n_jobs * 4`` number of chunks :param n_splits: Number of splits to use when ``chunk_size`` is ``None`` :param n_jobs: Number of workers to spawn. If ``None``, will use ``cpu_count()``. :return: Chunked ``iterable_of_args`` with updated ``iterable_len``, ``chunk_size`` and ``n_splits`` Nr-) get_n_chunksmake_single_argumentsrAr0)r'r(r)r*rBs r$apply_numpy_chunkingrGs +M\M: 0, HV\]]L,[9I3O_K`K`bl9A9qTZTffqjjlp.s.sttJH \:x ??r&cB|0t|drt|t|n|}n/t|drt|}ntd|||p|p t dzz }t|t j||z S)a Get number of chunks :param iterable_of_args: A numpy array or an iterable containing tuples of arguments to pass to a worker, which passes it to the function :param iterable_len: Number of tasks available in ``iterable_of_args``. Only needed when ``iterable_of_args`` is a generator :param chunk_size: Number of simultaneous tasks to give to a worker. If ``None``, will use ``n_splits`` to determine the chunk size :param n_splits: Number of splits to use when ``chunk_size`` is ``None`` :param n_jobs: Number of workers to spawn. If ``None``, will use ``cpu_count()`` :return: Number of chunks that will be created by the chunker Nr,zFailed to obtain length of iterable. Remedy: either provide an iterable with a len() function or specify iterable_len in the function callrD)r/minr0r.rr7r8)r'r(r)r*rBr<s r$rErEs >EFVXa>b>bt#lC(8$9$9:::ht !9 - -R&''QRR R GV-By{{a,GH w 'J"677 8 88r& generatorc@d|D}|r|nt|S)a Converts an iterable of single arguments to an iterable of single argument tuples :param iterable_of_args: A numpy array or an iterable containing tuples of arguments to pass to a worker, which passes it to the function :param generator: Whether or not to return a generator, otherwise a materialized list will be returned :return: Iterable of single argument tuples c3K|]}|fVdSN).0args r$ z(make_single_arguments..s$ . .cC6 . . . . . .r&)list)r'rJgens r$rFrFs- / .- . . .C *33c*r&secondswith_millisecondsc|dStt|dd}|r/t|dkr|dd|ddd}n|d}|S)z Format seconds to a string, optionally with or without milliseconds :param seconds: Number of seconds :param with_milliseconds: Whether to display milliseconds as well :return: String formatted time N)rT.r-r)strrrsplitr0)rTrUdurations r$format_secondsr]sr9W---..55c1==HS]]Q..qk55HQKO55A; Or&c heZdZdZ d deededeedeeddf dZd d Z d d Z dS) TimeItz Simple class that provides a context manager for keeping track of task duration and adds the total number of seconds in a designated output array Ncum_time_array array_idxmax_time_arrayformat_args_funcrcL||_||_||_||_d|_dS)a :param cum_time_array: Optional array to store cumulative time in :param array_idx: Index of cum_time_array to store the time value to :param max_time_array: Optional array to store maximum time duration in. Note that the array_idx doesn't apply to this array. The entire array is used for heapq :param format_args_func: Optional function which should return the formatted args corresponding to the function called within this context manager N)r`rarbrc start_time)selfr`rarbrcs r$__init__zTimeIt.__init__s.-", 0r&c6tj|_dSrM)timere)rfs r$ __enter__zTimeIt.__enter__s)++r&c0tj|jz }|j|j|jxx|z cc<|jP||jddkr;t j|j||j|ndfdSdSdS)Nr)rirer`rarbheapq heappushpoprc)rfexc_typeexc_valexc_tbr\s r$__exit__zTimeIt.__exit__s9;;0   *   / / /8 ; / / /   *x$:Ma:PQR:S/S/S  d1'DDYDe)>)>)@)@)@kop r r r r r + */S/Sr&)NN)rN) __name__ __module__ __qualname____doc__r rintrrgrjrqrNr&r$r_r_s11@D8<x0A'Bs!)*;!<#+H#5AE"&&&&rrrrrrr&r_use_dillctjd}|r tj|nt |S)z Create a SyncManager instance :param use_dill: Whether dill is used as serialization library :return: SyncManager instance )authkey)rurandomrmanagersr)rwrzs r$create_sync_managerr}sAjnnGrs   %%%%%%000000::::::ZZZZZZZZZZZZZZZZZZZZOO BOOOBAAAAAAAAA ( (! ((("( (((#(T#Y(4((((>KOZ^9*9*(9*(3-9*$U3:%679*JRSV-9* Zt+ ,9*9*9*9*xTXUY15@@8@8C=@%-c]@EMc]@!)#@:?#sTX@X:Y@@@@6nrIM99898C=9]efi]j9#C=99A#9RU9999> + +H + +QVW[]fWfQg + + + +HU3:%67DUX*rrrrrrrrDg$g;ggggs*A AA%A66B;BB