ZjoSSKrSSKrSSKJrJr SSKrSSKJr SSK J r J r J r SSK JrJrJrJr \ "5(a SSKrSSKJr "S S \5r"S S \ 5r\ "\"S S9S5"SS\55r\rg)N)Anyoverload)BasicTokenizer) ExplicitEnumadd_end_docstringsis_torch_available)ArgumentHandler ChunkPipelineDatasetbuild_pipeline_init_args),MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING_NAMESc2\rSrSrSrS\\\-4SjrSrg)"TokenClassificationArgumentHandlerz- Handles arguments for token classification. inputsc hURSS5nURS5nUbA[U[[45(a&[ U5S:a[U5n[ U5nOf[U[ 5(aU/nSnOK[ b[U[ 5(d[U[R5(aXSU4$[S5eURS5nU(aJ[U[5(a[US[5(aU/n[ U5U:wa [S5eXXd4$) Nis_split_into_wordsF delimiterrr zAt least one input is required.offset_mappingz;offset_mapping should have the same batch size as the input) get isinstancelisttuplelenstrr types GeneratorType ValueError)selfrkwargsrr batch_sizers |/root/GenerationalWealth/GenerationalWealth/venv/lib/python3.13/site-packages/transformers/pipelines/token_classification.py__call__+TokenClassificationArgumentHandler.__call__s $jj)>FJJ{+  *VdE]"C"CF VW&\FVJ  $ $XFJ  Z%@%@JvW\WjWjDkDki? ?>? ?$45 .$//J~a?PRW4X4X"0!1>"j0 !^__NEEN) __name__ __module__ __qualname____firstlineno____doc__rrr%__static_attributes__r(r'r$rrsFsT#YFr'rc,\rSrSrSrSrSrSrSrSr Sr g ) AggregationStrategy3zDAll the valid aggregation strategies for TokenClassificationPipelinenonesimplefirstaveragemaxr(N) r)r*r+r,r-NONESIMPLEFIRSTAVERAGEMAXr.r(r'r$r0r03sN D F EG Cr'r0T) has_tokenizera ignore_labels (`list[str]`, defaults to `["O"]`): A list of labels to ignore. stride (`int`, *optional*): If stride is provided, the pipeline is applied on all the text. The text is split into chunks of size model_max_length. Works only with fast tokenizers and `aggregation_strategy` different from `NONE`. The value of this argument defines the number of overlapping tokens between chunks. In other words, the model will shift forward by `tokenizer.model_max_length - stride` tokens each step. aggregation_strategy (`str`, *optional*, defaults to `"none"`): The strategy to fuse (or not) tokens based on the model prediction. - "none" : Will simply not do any aggregation and simply return raw results from the model - "simple" : Will attempt to group entities following the default schema. (A, B-TAG), (B, I-TAG), (C, I-TAG), (D, B-TAG2) (E, B-TAG2) will end up being [{"word": ABC, "entity": "TAG"}, {"word": "D", "entity": "TAG2"}, {"word": "E", "entity": "TAG2"}] Notice that two consecutive B tags will end up as different entities. On word based languages, we might end up splitting words undesirably : Imagine Microsoft being tagged as [{"word": "Micro", "entity": "ENTERPRISE"}, {"word": "soft", "entity": "NAME"}]. Look for FIRST, MAX, AVERAGE for ways to mitigate that and disambiguate words (on languages that support that meaning, which is basically tokens separated by a space). These mitigations will only work on real words, "New york" might still be tagged with two different entities. - "first" : (works only on word based models) Will use the `SIMPLE` strategy except that words, cannot end up with different tags. Words will simply use the tag of the first token of the word when there is ambiguity. - "average" : (works only on word based models) Will use the `SIMPLE` strategy except that words, cannot end up with different tags. scores will be averaged first across tokens, and then the maximum label is applied. - "max" : (works only on word based models) Will use the `SIMPLE` strategy except that words, cannot end up with different tags. Word entity will simply be the token with the maximum score.c^\rSrSrSrSrSrSrSrSr \ "54U4Sjjr S)S\ S-S \ \\\4S-S \S \S-S \S-4 S jjr\S\S\S\ \\\44Sj5r\S\ \S\S\ \ \\\44Sj5rS\\ \-S\S\ \\\4\ \ \\\4-4U4SjjrS*SjrSr\ R2S4SjrSrS+S\S\R:S\R:S \ \\\4S-S\R:S\ S\ \S-S-S\ \\\4S-S\ \4SjjrS\ \S\ S\ \4S jrS!\ \S\ S\4S"jr S!\ \S\ S\ \4S#jr!S!\ \S\4S$jr"S%\S\\\44S&jr#S!\ \S\ \4S'jr$S(r%U=r&$),TokenClassificationPipeline=u Named Entity Recognition pipeline using any `ModelForTokenClassification`. See the [named entity recognition examples](../task_summary#named-entity-recognition) for more information. Example: ```python >>> from transformers import pipeline >>> token_classifier = pipeline(model="Jean-Baptiste/camembert-ner", aggregation_strategy="simple") >>> sentence = "Je m'appelle jean-baptiste et je vis à montréal" >>> tokens = token_classifier(sentence) >>> tokens [{'entity_group': 'PER', 'score': 0.9931, 'word': 'jean-baptiste', 'start': 12, 'end': 26}, {'entity_group': 'LOC', 'score': 0.998, 'word': 'montréal', 'start': 38, 'end': 47}] >>> token = tokens[0] >>> # Start and end provide an easy way to highlight words in the original text. >>> sentence[token["start"] : token["end"]] ' jean-baptiste' >>> # Some models use the same idea to do part of speech. >>> syntaxer = pipeline(model="vblagoje/bert-english-uncased-finetuned-pos", aggregation_strategy="simple") >>> syntaxer("My name is Sarah and I live in London") [{'entity_group': 'PRON', 'score': 0.999, 'word': 'my', 'start': 0, 'end': 2}, {'entity_group': 'NOUN', 'score': 0.997, 'word': 'name', 'start': 3, 'end': 7}, {'entity_group': 'AUX', 'score': 0.994, 'word': 'is', 'start': 8, 'end': 10}, {'entity_group': 'PROPN', 'score': 0.999, 'word': 'sarah', 'start': 11, 'end': 16}, {'entity_group': 'CCONJ', 'score': 0.999, 'word': 'and', 'start': 17, 'end': 20}, {'entity_group': 'PRON', 'score': 0.999, 'word': 'i', 'start': 21, 'end': 22}, {'entity_group': 'VERB', 'score': 0.998, 'word': 'live', 'start': 23, 'end': 27}, {'entity_group': 'ADP', 'score': 0.999, 'word': 'in', 'start': 28, 'end': 30}, {'entity_group': 'PROPN', 'score': 0.999, 'word': 'london', 'start': 31, 'end': 37}] ``` Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) This token recognition pipeline can currently be loaded from [`pipeline`] using the following task identifier: `"ner"` (for predicting the classes of tokens in a sequence: person, organisation, location or miscellaneous). The models that this pipeline can use are models that have been fine-tuned on a token classification task. See the up-to-date list of available models on [huggingface.co/models](https://huggingface.co/models?filter=token-classification). sequencesFTc x>[TU]"S0UD6 UR[5 [ SS9UlXlg)NF) do_lower_caser()super__init__check_model_typerr_basic_tokenizer _args_parser)r! args_parserr" __class__s r$rD$TokenClassificationPipeline.__init__s5 "6" JK .U C'r'Naggregation_strategyrrstriderch0nXGS'U(a UcSOUUS'UbX7S'0nUb[U[5(a[UR5nU[R[R [R 1;a&URR(d [S5eX(S'UbXS'UbxXPRR:a [S5eU[R:Xa[S US 35eURR(a S S US .n XS 'O [S5eU0U4$)Nr rrz{Slow tokenizers cannot handle subwords. Please set the `aggregation_strategy` option to `"simple"` or use a fast tokenizer.rK ignore_labelszl`stride` must be less than `tokenizer.model_max_length` (or even lower if the tokenizer adds special tokens)zI`stride` was provided to process all the text but `aggregation_strategy="z&"`, please select another one instead.T)return_overflowing_tokenspaddingrLtokenizer_paramszm`stride` was provided to process all the text but you're using a slow tokenizer. Please use a fast tokenizer.) rrr0upperr9r;r: tokenizeris_fastr model_max_lengthr7) r!rOrKrrrLrpreprocess_paramspostprocess_paramsrRs r$_sanitize_parameters0TokenClassificationPipeline._sanitize_parametersso3F/0 4=4ES9 k *  %2@. /  +.44':;O;U;U;W'X$$'--/B/F/FH[HcHcde.. >:N5 6  $2? /  888 C$':'?'?? ,--SU >>))59#'"(($ =M&89$8!"&888r'rr"returnc gNr(r!rr"s r$r%$TokenClassificationPipeline.__call__sLOr'c gr]r(r^s r$r%r_sX[r'c >UR"U40UD6up4pVXBS'XbS'U(a)[SU55(d[TU] "U/40UD6$U(aXRS'[TU] "U40UD6$)ak Classify each token of the text(s) given as inputs. Args: inputs (`str` or `List[str]`): One or several texts (or one list of texts) for token classification. Can be pre-tokenized when `is_split_into_words=True`. Return: A list or a list of list of `dict`: Each result comes as a list of dictionaries (one for each token in the corresponding input, or each entity if this pipeline was instantiated with an aggregation_strategy) with the following keys: - **word** (`str`) -- The token/word classified. This is obtained by decoding the selected tokens. If you want to have the exact string in the original sentence, use `start` and `end`. - **score** (`float`) -- The corresponding probability for `entity`. - **entity** (`str`) -- The entity predicted for that token/word (it is named *entity_group* when *aggregation_strategy* is not `"none"`. - **index** (`int`, only present when `aggregation_strategy="none"`) -- The index of the corresponding token in the sentence. - **start** (`int`, *optional*) -- The index of the start of the corresponding entity in the sentence. Only exists if the offsets are available within the tokenizer - **end** (`int`, *optional*) -- The index of the end of the corresponding entity in the sentence. Only exists if the offsets are available within the tokenizer rrc3B# UHn[U[5v M g7fr])rr).0inputs r$ 7TokenClassificationPipeline.__call__..s*WPVu:eT+B+BPVsr)rGallrCr%)r!rr"_inputsrrrrIs r$r%r_s6CGBSBSTZBe^dBe?n(;$%'{ s*WPV*W'W'W7#VH77 7 '5# $w1&11r'c +# URS05nURR=(a URRS:nSnUSnU(aUSn[U[5(d [ S5eUn UR U 5n/n[U5n Sn U H2n URX[U 5-45 U [U 5U -- n M4 U n SUS'O"[U[5(d [ S5eUn UR"U 4SUSURRS .UD6nU(a&URR(d [ S 5eURS S5 [US 5n[U5H{nUR5VVs0sHunnUUURS5_M nnnUbUUS 'US:XaUOSUS'UUS- :HUS'UbURU5US'UUS'Uv M} gs snnf7f)NrRrrrzEWhen `is_split_into_words=True`, `sentence` must be a list of tokens.TzKWhen `is_split_into_words=False`, `sentence` must be an untokenized string.pt)return_tensors truncationreturn_special_tokens_maskreturn_offsets_mappingz@is_split_into_words=True is only supported with fast tokenizers.overflow_to_sample_mapping input_idsrsentencer is_lastword_idsword_to_chars_map)poprTrVrrr joinrappendrrUrangeitems unsqueezers)r!rqrrWrRrlrtrrwords delimiter_len char_offsetwordtext_to_tokenizer num_chunksikv model_inputss r$ preprocess&TokenClassificationPipeline.preprocesss,001CRH^^44\9X9X[\9\  /0EF )+6Ih-- !hiiE ~~e,H "  NMK!((+SY7N)OPs4y=88  % 6: 2 3h,, !noo'   !'+#'>>#9#9     t~~'='=_` ` /6 ,- z"A=C\\^L^TQAqt~~a00^LL)1? -.346xtL $&':>&9L # ,+1??1+= Z(4E 01 #LsFH "H>AH cNURS5nURSS5nURS5nURS5nURSS5nURSS5nUR"S 0UD6n[U[5(aUSOUSn U UUUUUUS .UE$) Nspecial_tokens_maskrrqrrrsrtlogitsr)rrrrqrrrsrtr()rumodelrdict) r!rrrrqrrrsrtoutputrs r$_forward$TokenClassificationPipeline._forward.s*../DE%))*:DA##J/""9-##J5(,,-@$G+l+%/%=%=!6!9#6,  !2    r'c UcS/n/nUSRS5nUGHnUSSR[R[R4;a4USSR [R 5R5nOUSSR5nUSSnUSSn USbUSSOSn USSR5n URS 5n [R"US S S 9n [R"X}- 5nXRS S S 9- nURUU UU U UU US 9nURUU5nUVs/sH5nURSS5U;dMURSS5U;dM3UPM7 nnURU5 GM [U5nUS:aUR!U5nU$s snf)NOrrtrrqrprrrsT)axiskeepdims)rsrtentity entity_groupr )rdtypetorchbfloat16float16tofloat32numpynpr6expsumgather_pre_entities aggregateextendraggregate_overlapping_entities)r! all_outputsrKrO all_entitiesrt model_outputsrrqrprrrsmaxes shifted_expscores pre_entitiesgrouped_entitiesrentitiesrs r$ postprocess'TokenClassificationPipeline.postprocessEs    EM (N../BC(MX&q)//ENNEMM3RR&x0366u}}EKKM&x0399;"1~j1H%k215I6CDT6U6a ./2gk #00E"Fq"I"O"O"Q $((4HFF6T:E&&0K ??T?#JJF33#$!"34 L $~~l>>|LLsG 2G  G c@[U5S:XaU$[USS9n/nUSnUHcnUSUSs=::a US:a7O O4USUS- nUSUS- nXV:dXV:XaUSUS:aUnMLMNMPURU5 UnMe URU5 U$)Nrc US$)Nstartr()xs r$LTokenClassificationPipeline.aggregate_overlapping_entities..zs!G*r'keyrendscore)rsortedrw)r!raggregated_entitiesprevious_entityrcurrent_lengthprevious_lengths r$r:TokenClassificationPipeline.aggregate_overlapping_entitiesws x=A O((<= "1+Fw'6'?S_U=SS!'!@"1%"8?7;S"S"4%8w/'*BB&,OC9 $**?;"( ""?3""r'rqrprrrsrtc /n [U5GHupXZ(aMURR[X*55n UGbRXJupUbUbXznUbXunnU U- n UU- n[ U [5(d U R 5n UR 5nXUn[ URSS5(aH[ URRRSS5(a[U 5[U5:gnOgU[R[R[R1;a[R"S[ 5 U S:=(a SXS- U S-;n[X*5URR":XaUn SnOSn SnSnU U U UU US .nU R%U5 GM U $) zTFuse various numpy arrays into dicts with all the information needed for aggregationN _tokenizercontinuing_subword_prefixz?Tokenizer does not support real words, using fallback heuristicrrNr F)r~rrrindex is_subword) enumeraterTconvert_ids_to_tokensintritemgetattrrrrr0r9r:r;warningswarn UserWarning unk_token_idrw)r!rqrprrrrKrsrtridx token_scoresr~ start_indend_ind word_index start_char_word_refr pre_entitys r$r/TokenClassificationPipeline.gather_pre_entitiess !*6!2 C"'>>77IN8KLD)%3%8" ',=,I!)J!-(9(E A!Z/ :-!)S11 ) 0I%llnG#g64>><>>7NN--335PRVDD "%Tc(m!;J,+11+33+//0 ! ]'"+Q!e3hST}W`cdWd>e3eJy~&$..*E*EE#D!&J " &"( J    +q"3rr'rcU[R[R1;an/nUHenUSR5nUSUnURR R UUUSUSUSUSS.nURU5 Mg OURX5nU[R:XaU$URU5$)Nrrr~rr)rrrr~rr) r0r7r8argmaxrconfigid2labelrwaggregate_wordsgroup_entities)r!rrKrr entity_idxrrs r$r%TokenClassificationPipeline.aggregates $7$<$<>Q>X>X#Y YH* '188: "8,Z8"jj//88D"'0&v.'0%e, '+++LOH #6#;#; ;O""8,,r'rc(URRUVs/sHo3SPM sn5nU[R:Xa@USSnUR 5nXVnUR R RUnOU[R:XaH[USS9nUSnUR 5nXVnUR R RUnOU[R:Xav[R"UVs/sHo3SPM sn5n[R"USS9n U R 5n UR R RU nXnO [S5eUUUUSSUS S S .n U $s snfs snf) Nr~rrc(USR5$)Nr)r6)rs r$r.s&:J:N:N:Pr'r)rzInvalid aggregation_strategyrrr)rrr~rr)rTconvert_tokens_to_stringr0r9rrrrr;r6r:rstacknanmeanr ) r!rrKrr~rrr max_entityaverage_scoresr new_entitys r$aggregate_word*TokenClassificationPipeline.aggregate_words|~~66U]7^U]6vU]7^_ #6#<#< <a[*F--/CKEZZ&&//4F !%8%<%< <X+PQJ)F--/CKEZZ&&//4F !%8%@%@ @XXhGhFh/hGHFZZQ7N'..0JZZ&&// ;F".E;< <a[)B<&  78_Hs F FcTU[R[R1;a [S5e/nSnUHKnUcU/nM US(aUR U5 M(UR UR XB55 U/nMM Ub UR UR XB55 U$)z Override tokens from a given word that disagree to force agreement on word boundaries. Example: micro|soft| com|pany| B-ENT I-NAME I-ENT I-ENT will be rewritten with first strategy as microsoft| company| B-ENT I-ENT z;NONE and SIMPLE strategies are invalid for word aggregationNr)r0r7r8r rwr)r!rrK word_entities word_grouprs r$r+TokenClassificationPipeline.aggregate_words s  $ $  & &$  Z[ [  F!$X  %!!&)$$T%8%8%Z[$X   !  !4!4Z!V Wr'cLUSSRSS5Sn[R"UVs/sHo"SPM sn5nUVs/sHo"SPM nnW[R"U5URR U5USSUSS S .nU$s snfs snf) z Group together the adjacent tokens with the same entity predicted. Args: entities (`dict`): The entities predicted by the pipeline. rr-r rrr~rr)rrr~rr)splitrrmeanrTr)r!rrrtokensrs r$group_sub_entities.TokenClassificationPipeline.group_sub_entities%s!X&,,S!4R88D8G_8DE/78xV.x8#WWV_NN;;FCa[)B<&  E8s B B! entity_namecURS5(a SnUSSnX#4$URS5(a SnUSSnX#4$SnUnX#4$)NzB-BrzI-I) startswith)r!rbitags r$get_tag#TokenClassificationPipeline.get_tag:sk  ! !$ ' 'Bab/Cw # #D ) )Bab/C wBCwr'c/n/nUHnU(dURU5 MURUS5upVURUSS5upxXh:XaUS:waURU5 MjURURU55 U/nM U(a URURU55 U$)z Find and group together the adjacent tokens with the same entity predicted. Args: entities (`dict`): The entities predicted by the pipeline. rrr)rwrr) r!r entity_groupsentity_group_disaggrrrlast_bilast_tags r$r*TokenClassificationPipeline.group_entitiesHs  F&#**62 ll6(#34GB $ -@-DX-N O G29#**62$$T%<%<=P%QR'-h#'(   !8!89L!M Nr')rGrF)NNNFNNr])NN)'r)r*r+r,r-default_input_names_load_processor_load_image_processor_load_feature_extractor_load_tokenizerrrDr0rrrboolrrYrrrr%rrr7rrrndarrayrrrrrrrr. __classcell__)rIs@r$r>r>=s@"H&O!#O#E#G(;?7;$)! $992D899U38_-4 99 " 99 d 99:99vOsOcOd4S>6JOO [tCy[C[Dd3PS8nAUFF::F F U38_-4 F  ZZ F2FsTz"T)F c3h047F dFP-d4j-H[-`dei`j-,tDzI\ae<T J]bfgkbl84:$* 3 5c? #tDz#d4j##r'r>)rrtypingrrrr$models.bert.tokenization_bert_legacyrutilsrrr baser r r rrmodels.auto.modeling_autorrr0r> NerPipeliner(r'r$rs  A TSXFF:,40n>O-O?>Od* r'