i]>ddlmZddlZddlZddlmZddlmZmZm Z ddl m Z ddl m Z ddlmZddlZddlmZe r ddlZddlZddlZiZ d Z d'dZd(dZ d)d*dZGd d!Z d+d,d&ZdS)-) annotationsN) bisect_left)CallableSequence TYPE_CHECKING)cache) accumulate)suppress)tqdm).?!*;,()[]u“u”u‘u’'"`:u—u…/\u–&-textstrreturntuple[str, bool, list[str]]cd}d|vsd|vr#ttjd|}nd|vr#ttjd|}n\tjd|r#ttjd|}n$tD] }||vrd }n d |t |fS||||fS) zDSplit text using the most semantically meaningful splitter possible.T  z[\r\n]+ z\t+z\sz\s+F)maxrefindallsearch"_NON_WHITESPACE_SEMANTIC_SPLITTERSlistsplit)rsplitter_is_whitespacesplitters i/var/www/development/aibuddy-work/election-extract/venv/lib/python3.11/site-packages/semchunk/semchunk.py _split_textr1s" t||tt||rz*d3344 rz&$//00 5$   :rz&$//00; : :H4).& -tDzz9 9 +TZZ-A-A AAsplits list[str] chunk_sizeintr/ token_counterrtuple[int, str]cd}d}t|dz}ttd|Dd}||d||krt |||dz||z}t ||z|dz } |||d| } || r| dkr || | z n|}| |kr| }n| dz}||k|dz ||d|dz fS)zMerge splits until a chunk size is reached, returning the index of the last split included in the merged chunk along with the merged chunk itself.g?rc,g|]}t|Slen).0r-s r0 z merge_splits..Fs)I)I)I#e**)I)I)Ir2)initialN)r>r,r appendrminjoin) r3r5r/r7averagelowhighcumulative_lengthsimidpointtokenss r0 merge_splitsrM@sDG C v;;?Dj)I)I&)I)I)ISTUUUVV04555 ** *3>:J

??@@;Mh;Ws\bef\f\f$X.77ls J  DDQ,C ** 7HMM&#'"233 33r2TFCallable[[str], int]memoizebool_recursion_depth_reattach_whitespace_splittersc `|s*|r(t|t|}t|\}}}|rd}g} t  t |D](\} } | vr || |kr,| t| ||||dz|n[t|| d|||\} }  t| dz| | z| | |s| t|dz ksotfdt| dzt|Ds6|| d|zx}|kr|| d<| |*|sttd| } | S)aOSplit a text into semantically meaningful chunks of a specified size as determined by the provided token counter. Args: text (str): The text to be chunked. chunk_size (int): The maximum number of tokens a chunk may contain. token_counter (Callable[[str], int]): A callable that takes a string and returns the number of tokens in it. memoize (bool, optional): Whether to memoize the token counter. Defaults to `True`. Returns: list[str]: A list of chunks up to `chunk_size`-tokens-long, with any whitespace used to split the text removed.Fr:)r7rOrQrRNc3 K|]}|vV dSNr<)r?jskipss r0 zchunk..s)JyJyZ[1PU:JyJyJyJyJyJyr2rB)_memoized_token_counters setdefaultrr1set enumerateextendchunkrMupdaterangerCr>allr,filter)rr5r7rOrQrRr/r.r3chunksrJr-final_split_in_chunk_i new_chunklast_chunk_with_splitterrWs @r0r^r^Xs%( aa0;;M5Q^K_K_`` 0;4/@/@,H$f%E'= F EEE^f%%((5 ::  =  * , , MM%z=\cyILMyMpNOOO P P P P 1=VABBZU]_l0m0m - "I LLq1ua*@&@AA B B B MM) $ $ $& (qCKK!O/C/CsJyJyJyJy_defijejlopvlwlw_x_xJyJyJyGyGy/C}h9NN5OOS]]]5r  h''' ,fT6**++ Mr2c,eZdZddZdd Z dddZdS)Chunkerr5r6r7rNr Nonec"||_||_dSrU)r5r7)selfr5r7s r0__init__zChunker.__init__s$*r2rrr4c<t||j|jdS)z Chunk a text.F)rO)r^r5r7)rkrs r0r^z Chunker.chunks T4?D,>%PPPPr2r:F text_or_textsstr | Sequence[str] processesprogressrPlist[str] | list[list[str]]cHt|tr|S|r|dkrt|}|dkrfd|DSt j|d5}|j||cdddS#1swxYwYdS)a{Split text or texts into semantically meaningful chunks of a specified size as determined by the provided tokenizer or token counter. Args: text_or_texts (str | Sequence[str]): The text or texts to be chunked. Returns: list[str] | list[list[str]]: If a single text has been provided, a list of chunks up to `chunk_size`-tokens-long, with any whitespace used to split the text removed, or, if multiple texts have been provided, a list of lists of chunks, with each inner list corresponding to the chunks of one of the provided input texts. processes (int, optional): The number of processes to use when chunking multiple texts. Defaults to `1` in which case chunking will occur in the main process. progress (bool, optional): Whether to display a progress bar when chunking multiple texts. Defaults to `False`.r:c:g|]}|Sr<)r^)r?rrks r0r@z$Chunker.__call__..s%???DJJt$$???r2T)use_dill) progress_barN) isinstancerr^r mpire WorkerPoolmap)rkrnrprqpools` r0__call__zChunker.__call__s  mS ) ) -::m,, ,  0 Q //M >>??????? ?  iD 9 9 9 PT88DJ h8OO P P P P P P P P P P P P P P P P P Ps-BBBN)r5r6r7rNr ri)rrr r4)r:F)rnrorpr6rqrPr rr)__name__ __module__ __qualname__rlr^r|r<r2r0rhrhsh++++QQQQ PPPPPPPr2rhtokenizer_or_token_counterhstr | tiktoken.Encoding | transformers.PreTrainedTokenizer | tokenizers.Tokenizer | Callable[[str], int] int | Nonemax_token_charsc ttr ddl} |j}n #t$r|j}YnwxYwnQ#t$rD ddl}|j}n!#t$rtddwxYwYnwxYw|dD]}t|r{tt|r^t|}t|dr6|r4td|Drtd|Dntd rtjt rijtd rQt#t5t%d zdddn #1swxYwYntd td r.d t)jjjvrdfd } n dfd } n} dz | d fd } | } |r(t.| t3| } t5| S)aI Construct a chunker that splits one or more texts into semantically meaningful chunks of a specified size as determined by the provided tokenizer or token counter. Args: tokenizer_or_token_counter (str | tiktoken.Encoding | transformers.PreTrainedTokenizer | tokenizers.Tokenizer | Callable[[str], int]): Either: the name of a `tiktoken` or `transformers` tokenizer (with priority given to the former); a tokenizer that possesses an `encode` attribute (eg, a `tiktoken`, `transformers` or `tokenizers` tokenizer); or a token counter that returns the number of tokens in a input. chunk_size (int, optional): The maximum number of tokens a chunk may contain. Defaults to `None` in which case it will be set to the same value as the tokenizer's `model_max_length` attribute (deducted by the number of tokens returned by attempting to tokenize an empty string) if possible otherwise a `ValueError` will be raised. max_token_chars (int, optional): The maximum numbers of characters a token may contain. Used to significantly speed up the token counting of long inputs. Defaults to `None` in which case it will either not be used or will, if possible, be set to the numbers of characters in the longest token in the tokenizer's vocabulary as determined by the `token_byte_values` or `get_vocab` methods. memoize (bool, optional): Whether to memoize the token counter. Defaults to `True`. Returns: Callable[[str | Sequence[str], bool, bool], list[str] | list[list[str]]]: A chunker that takes either a single text or a sequence of texts and returns, if a single text has been provided, a list of chunks up to `chunk_size`-tokens-long with any whitespace used to split the text removed, or, if multiple texts have been provided, a list of lists of chunks, with each inner list corresponding to the chunks of one of the provided input texts. The resulting chunker can be passed a `processes` argument that specifies the number of processes to be used when chunking multiple texts. It is also possible to pass a `progress` argument which, if set to `True` and multiple texts are passed, will display a progress bar. Technically, the chunker will be an instance of the `semchunk.Chunker` class to assist with type hinting, though this should have no impact on how it can be used.rNrz" was provided to `semchunk.chunkerify` as the name of a tokenizer but neither `tiktoken` nor `transformers` have a tokenizer by that name. Perhaps they are not installed or maybe there is a typo in that name?)token_byte_values get_vocab__iter__c36K|]}t|dVdS)__len__N)hasattrr?tokens r0rXzchunkerify..s-?m?m^cy@Y@Y?m?m?m?m?m?mr2c34K|]}t|VdSrUr=rs r0rXzchunkerify..s()H)H#e**)H)H)H)H)H)Hr2model_max_lengthencoder&a!Your desired chunk size was not passed to `semchunk.chunkerify` and the provided tokenizer either lacks an attribute named 'model_max_length' or that attribute is not an integer. Either specify a chunk size or provide a tokenizer that has a 'model_max_length' attribute that is an integer.add_special_tokensrrr r6cLt|dS)NF)rr>rrrs r0r7z!chunkerify..token_counter s&5<.token_counters 5<.faster_token_counters`"QI4yy9$$)?)?EaiRaFaEa@b)c)cfp)p)p!A~%))$// /r2)rrr r6)rwrtiktokenencoding_for_model Exception get_encoding transformers AutoTokenizerfrom_pretrained ValueErrorrcallablegetattrrar'rr6r r>rinspect signature parametersrYrZrrh) rr5rrOr tokenizerr$potential_vocabulary_getter_functionvocabr7rrs ``` @r0 chunkerifyrs2,c22/ T OOO N7H78RSS  N N N1H12LMM  N T T T T####(6FFGabb  T T T "S%?"S"S"STTT T  T&/"5   013WXX ]efmoIKogpgp^q^q a :<`aacc5*--%C?m?mgl?m?m?mrs"""""" 4444444444  ;;;;;;;;;;;;;9&" SBBBBB44448+0 ?????D#P#P#P#P#P#P#P#PP""& j.j.j.j.j.j.j.r2