ZjC_ 6SrSSKrSSKrSSKrSSKrSSKJr SSKJr \R"\ 5r SSS.r S r "S S \5rS rS r\S\SSSSSS4 r\R$"SSR'\5-\R(\R*-\R,-5r\R$"S5r\R$"\\R(\R*-\R,-5r\R$"S5rS SjrS!Sjr"SS5rSrSrS"Sjr S /r!g)#z!Tokenization classes for BERTweetN)PreTrainedTokenizer)logging vocab.txt bpe.codes) vocab_file merges_filecz[5nUSnUSSHnURX#45 UnM [U5nU$)zy Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rN)setadd)wordpairs prev_charchars ڃ/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/models/bertweet/tokenization_bertweet.py get_pairsr#sH EEQIQR 9#$  JE Lc ^\rSrSrSr\rSU4Sjjr\S5r Sr Sr Sr Sr S rS rS rS rSS\S\S -S\\S44SjjrSrSrU=r$)BertweetTokenizer3a8 Constructs a BERTweet tokenizer, using Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. normalization (`bool`, *optional*, defaults to `False`): Whether or not to apply a normalization preprocess. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. cls_token (`str`, *optional*, defaults to `""`): The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. mask_token (`str`, *optional*, defaults to `""`): The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. c  >SSKJn XlXlX l0UlSUR[U5'SUR[U 5'SUR[U5'SUR[U5'URU5 URR5V Vs0sHupX_M snn Ul [USS9nUR5RS 5SS nSSS5 WVs/sHn[!UR5SS 5PM! nn[#[%U['[)U5555Ul0UlX0l[15UlS S S .Ul[6TU]p"SUUUUUUU U SSSS. U D6 g![a [R S5 SUlGNf=fs snn f!,(df  N=fs snf)Nr)demojizezsemoji is not installed, thus not converting emoticons or emojis into text. Install emoji: pip3 install emoji==0.6.0r rutf-8encoding 'z...)u’u… all_zerosTcls_double_sep) normalization bos_token eos_token sep_token cls_token unk_token pad_token mask_tokentoken_type_ids_pattern%token_type_ids_include_special_tokensspecial_tokens_pattern)emojir demojizer ImportErrorloggerwarningrr encoderstr add_from_fileitemsdecoderopenreadsplittupledictziprangelen bpe_rankscacher#TweetTokenizertweetPreprocessorspecial_punctssuper__init__)selfrr r#r$r%r&r'r(r)r*kwargsrkv merges_handlemergesmerge __class__s rrGBertweetTokenizer.__init__hs " &%N%& '( S^$'( S^$'( S^$'( S^$ :&)-););)=>)=)=> + 0M"'')//5cr:F19?@% cr*+@c&%F *<=> *!/!1&)%8  '!#.26#3  = " NN( "DN  "$? 0 0@s) F0G#G &G&GG Gc,[UR5$N)r@r4rHs r vocab_sizeBertweetTokenizer.vocab_sizes4<<  rcB[UR40URD6$rR)r=r4added_tokens_encoderrSs r get_vocabBertweetTokenizer.get_vocabsDLL>D$=$=>>rc6^UTR;aTRU$[U5n[[USS5USS-/-5n[U5nU(dU$[ UU4SjS9nUTR ;aOUupV/nSnU[ U5:aURXX5n URX(U 5 U nX(U:Xa6U[ U5S- :a$X(S-U:XaURXV-5 US- nOURX(5 US- nU[ U5:aM[U5nUn[ U5S:XaO [U5nMSRU5nUSS nUTRU'U$![a URX(S5 Mqf=f) NrzcN>TRRU[S55$)Ninf)rAgetfloat)pairrHs r'BertweetTokenizer.bpe..s1C1CD%PU,1Wrkeyrr r@@ ) rBr<listrminrAr@indexextend ValueErrorappendjoin) rHtokenrrbigramfirstsecondnew_wordijs ` rbpeBertweetTokenizer.bpes DJJ ::e$ $U|T$s)_R6(9'::;$L$WXFT^^+"MEHAc$i- 5,A OOD1I.A7e#CIM(9dq5kV>SOOEN3FAOODG,FAc$i- XHD4yA~!$9:zz$CRy  5 -"OODH-sE77FFcUR(aURU5n/n[R"SU5nUH;nUR [ UR U5RS555 M= U$)zTokenize a string.z\S+\n? )r#normalizeTweetrefindallrirfrtr;)rHtext split_tokenswordsrms r _tokenizeBertweetTokenizer._tokenizesf   &&t,D  9d+E   TXXe_%:%:3%? @ Arc.URH!nURX RU5nM# URRU5nSR UVs/sHo@R U5PM sn5nURSS5RSS5RSS5RSS5RS S 5nURS S 5RS S5RSS5RSS5RSS5RSS5nURSS5RSS5RSS5RSS5nSR UR 55$s snf)z Normalize a raw Tweet rwzcannot zcan not zn't z n't zn 't zca n'tzcan'tzai n'tzain'tz'm z 'm z're z 're z's z 's z'll z 'll z'd z 'd z've z 've z p . m .z p.m.z p . m z p.m z a . m .z a.m.z a . m z a.m )rEreplacerDtokenizerlnormalizeTokenr;)rHtweetpuncttokensrm normTweets rrx BertweetTokenizer.normalizeTweets^((EMM%)<)XR;aURU$URbUR U5$U$U$)z Normalize tokens in a Tweet @z@USERhttpwwwHTTPURLr )lower startswithr@rEr0)rHrmlowercased_tokens rr BertweetTokenizer.normalizeTokens!;;=   C   ( ( 0 04D4O4OPU4V4V Z1_+++**511~~)~~e,, Lrc~URRXRRUR55$)z0Converts a token (str) in an id using the vocab.)r4r]r()rHrms r_convert_token_to_id&BertweetTokenizer._convert_token_to_ids*||||'7'7'GHHrcLURRXR5$)z=Converts an index (integer) in a token (str) using the vocab.)r8r]r()rHrhs r_convert_id_to_token&BertweetTokenizer._convert_id_to_tokens||~~66rcdSRU5RSS5R5nU$)z:Converts a sequence of tokens (string) in a single string.rwrd)rlrstrip)rHr out_strings rconvert_tokens_to_string*BertweetTokenizer.convert_tokens_to_string#s,XXf%--eR8>>@ rNsave_directoryfilename_prefixreturn.c "[RRU5(d[R SUS35 g[ US05nU(aUS3OSn[RR XURSS5-5n[US S S 9n[URR5S S 9H$upxUS:dM URUSUS35 M& SSS5 [RR XURSS5-5n [U S S S 9n U RS[URR5SS 955 SSS5 XY4$!,(df  N=f!,(df  XY4$=f)z6 Save the vocabulary and merges files to a directory. zVocabulary path (z) should be a directoryr.vocab_files_names-rrrwrrc US$Nr r.kvs rr`3BertweetTokenizer.save_vocabulary..=s rRSurrbrwrNr rc3N# UHupSRU5S-v M g7f)rwrN)rl).0 bpe_tokens token_indexs r 4BertweetTokenizer.save_vocabulary..Es(/c+J$t+/cs#%c US$rr.rs rr`rGs ]_`a]br)ospathisdirr2errorgetattrrlr]r9sortedr4r7write writelinesrA) rHrrrprefixrfrmtoken_id merge_filewriters rsave_vocabulary!BertweetTokenizer.save_vocabulary.seww}}^,, LL,^,<DT#Uq=GGugQxj34$V5WW\\.;L;P;PQ^`k;l2lm *cG 4   /5dnn6J6J6LRb/c 5 ''5 45 4 ''s-E-E-(:E>- E;> Fc[U[5(a'[USSS9nURU5 SSS5 gUR5nUH\nUR5nURS5nUS:Xa [S 5eUSUn[UR5URU'M^ g!,(df  g=f![anUeSnAf[ a [ SUS35ef=f) zY Loads a pre-existing dictionary from a text file and adds its symbols to this instance. rrrNzIncorrect encoding detected in z, please rebuild the datasetrwrz5Incorrect dictionary format, expected ' ') isinstancer5r9r6FileNotFoundError UnicodeError Exception readlinesrrfindrjr@r4) rHrfdfnfelineslineTmplineidxrs rr6BertweetTokenizer.add_from_fileLs a   c!S73r&&r*4  G==?D**S/Cby !XYY:D!$T\\!2DLL  43  %   c"A!D` abb cs3 CB0C0 B>:C>C C* C  C*) rArBr8r0r4r r#rErDr)FrrzzzrR)__name__ __module__ __qualname____firstlineno____doc__VOCAB_FILES_NAMESrrGpropertyrTrXrtr~rxrrrrr5r<rr6__static_attributes__ __classcell__)rOs@rrr3s0d* > @!!?*X  +D&I7(c(C$J(Z_`ceh`hZi(<33rrac (?: [<>]? [:;=8] # eyes [\-o\*\']? # optional nose [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth | [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth [\-o\*\']? # optional nose [:;=8] # eyes [<>]? | <3 # heart )u # Capture 1: entire matched URL (?: https?: # URL protocol and colon (?: /{1,3} # 1-3 slashes | # or [a-z0-9%] # Single letter or digit or '%' # (Trying not to match e.g. "URI::Escape") ) | # or # looks like domain name followed by a slash: [a-z0-9.\-]+[.] (?:[a-z]{2,13}) / ) (?: # One or more: [^\s()<>{}\[\]]+ # Run of non-space, non-()<>{}[] | # or \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...) | \([^\s]+?\) # balanced parens, non-recursive: (...) )+ (?: # End with: \([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...) | \([^\s]+?\) # balanced parens, non-recursive: (...) | # or [^\s`!()\[\]{};:'".,<>?«»“”‘’] # not a space or one of these punct chars ) | # OR, the following to match naked domains: (?: (?\s]+>z [\-]+>|<[\-]+z (?:@[\w_]+)z(?:\#+[\w_]+[\w\'_\-]*[\w_]+)z#[\w.+-]+@[\w-]+\.(?:[\w-]\.?)+[\w-]a (?:[^\W\d_](?:[^\W\d_]|['\-_])+[^\W\d_]) # Words with apostrophes or dashes. | (?:[+\-]?\d+[,/.:-]\d+[+\-]?) # Numbers, including fractions, decimals. | (?:[\w_]+) # Words without apostrophes or dashes. | (?:\.(?:\s*\.){1,}) # Ellipsis dots. | (?:\S) # Everything else that isn't whitespace. z(%s)|z([^a-zA-Z0-9])\1{3,}z&(#?(x?))([^&;\s]+);c\UcSn[U[5(aURX5$U$)Nr)rbytesdecode)r{rerrorss r_str_to_unicoders.${{8,, KrcR^^UU4Sjn[RU[X55$)u Remove entities from text by converting them to their corresponding unicode character. Args: text: A unicode string or a byte string encoded in the given *encoding* (which defaults to 'utf-8'). keep (list): List of entity names which should not be replaced. This supports both numeric entities (`&#nnnn;` and `&#hhhh;`) and named entities (such as ` ` or `>`). remove_illegal (bool): If `True`, entities that can't be converted are removed. Otherwise, entities that can't be converted are kept "as is". Returns: A unicode string with the entities removed. See https://github.com/scrapy/w3lib/blob/master/w3lib/html.py Examples: ```python >>> from nltk.tokenize.casual import _replace_html_entities >>> _replace_html_entities(b"Price: £100") 'Price: \xa3100' >>> print(_replace_html_entities(b"Price: £100")) Price: £100 ```c,>URS5nURS5(a\URS5(a [US5nO [US5nSUs=::aS::aO O\[U45RS5$O@UT;aURS 5$[ R RRU5nUb [U5$T(aS $URS 5$![a SnN7f=f![[4a N>f=f) Nrr r cp1252rr) groupintrrrjhtmlentitiesname2codepointr]chr OverflowError)match entity_bodynumberkeepremove_illegals r_convert_entity/_replace_html_entities.._convert_entity;skk!n ;;q>> ;;q>> b1F b1F 6)T) &+228<<* d"{{1~%5599+F   6{"$r7Q7   .  s)>C.)C. D. C=<C=DD)ENT_REsubr)r{rrrrs `` r_replace_html_entitiesrs :8: ::ot'F GGrc(\rSrSrSrSSjrSrSrg)rCi^a Examples: ```python >>> # Tokenizer for tweets. >>> from nltk.tokenize import TweetTokenizer >>> tknzr = TweetTokenizer() >>> s0 = "This is a cooool #dummysmiley: :-) :-P <3 and some arrows < > -> <--" >>> tknzr.tokenize(s0) ['This', 'is', 'a', 'cooool', '#dummysmiley', ':', ':-)', ':-P', '<3', 'and', 'some', 'arrows', '<', '>', '->', '<--'] >>> # Examples using *strip_handles* and *reduce_len parameters*: >>> tknzr = TweetTokenizer(strip_handles=True, reduce_len=True) >>> s1 = "@remy: This is waaaaayyyy too much for you!!!!!!" >>> tknzr.tokenize(s1) [':', 'This', 'is', 'waaayyy', 'too', 'much', 'for', 'you', '!', '!', '!'] ```c(XlX lX0lgrR preserve_case reduce_len strip_handles)rHrrrs rrGTweetTokenizer.__init__rs*$*rc[U5nUR(a [U5nUR(a [ U5n[ R SU5n[RU5nUR(d<UVs/sH/n[RU5(aUOUR5PM1 nnU$s snf)z Args: text: str Returns: list(str) A tokenized list of strings; concatenating this list returns the original string if `preserve_case=False` \1\1\1) rrremove_handlesrreduce_lengtheningHANG_RErWORD_RErzr EMOTICON_REsearchr)rHr{ safe_textr}xs rrTweetTokenizer.tokenizews&d+   !$'D ??%d+DKK 40  *!!HMN1+,,Q//QQWWY>EN Os6B>rNTFF)rrrrrrGrrr.rrrCrC^s &+ rrCcR[R"S5nURSU5$)zY Replace repeated character sequences of length 3 or greater with sequences of length 3. z (.)\1{2,}rregexcompilerr{patterns rrrs#mmL)G ;;y$ ''rcR[R"S5nURSU5$)z, Remove Twitter username handles from text. zv(?r sG( 6   H %  n3+n3x L $)\  (. A+ ` -- chhw&779PSXS`S`9` a --/ 0mmIu}}uww'>'NO  . /:H@00p("   r