관리-도구
편집 파일: _tokenizer.cpython-35.pyc
��Re�, � @ s+ d d l m Z m Z m Z d d l m Z d d l m Z m Z d d l m Z d d l m Z d d l m Z d d l m Z m Z d d l m Z m Z m Z d d l m Z m Z d d l m Z d d l m Z d d l m Z e e � Z e d k r e Z n e Z Gd d � d e � Z d S)� )�absolute_import�division�unicode_literals)�unichr)�deque� OrderedDict)� version_info� )�spaceCharacters)�entities)� asciiLetters�asciiUpper2Lower)�digits� hexDigits�EOF)� tokenTypes� tagTokenTypes)�replacementCharacters)�HTMLInputStream)�Trie� � c s� e Z d Z d Z d � f d d � Z d d � Z d d � Z d d d d � Z d d � Z d d � Z d d � Z d d � Z d d � Z d d � Z d d � Z d d � Z d d � Z d d � Z d d! � Z d" d# � Z d$ d% � Z d& d' � Z d( d) � Z d* d+ � Z d, d- � Z d. d/ � Z d0 d1 � Z d2 d3 � Z d4 d5 � Z d6 d7 � Z d8 d9 � Z d: d; � Z d< d= � Z d> d? � Z! d@ dA � Z" dB dC � Z# dD dE � Z$ dF dG � Z% dH dI � Z& dJ dK � Z' dL dM � Z( dN dO � Z) dP dQ � Z* dR dS � Z+ dT dU � Z, dV dW � Z- dX dY � Z. dZ d[ � Z/ d\ d] � Z0 d^ d_ � Z1 d` da � Z2 db dc � Z3 dd de � Z4 df dg � Z5 dh di � Z6 dj dk � Z7 dl dm � Z8 dn do � Z9 dp dq � Z: dr ds � Z; dt du � Z< dv dw � Z= dx dy � Z> dz d{ � Z? d| d} � Z@ d~ d � ZA d� d� � ZB d� d� � ZC d� d� � ZD d� d� � ZE d� d� � ZF d� d� � ZG d� d� � ZH d� d� � ZI d� d� � ZJ d� d� � ZK d� d� � ZL � S)�� HTMLTokenizera This class takes care of tokenizing HTML. * self.currentToken Holds the token that is currently being processed. * self.state Holds a reference to the method to be invoked... XXX * self.stream Points to HTMLInputStream object. Nc sb t | | � | _ | | _ d | _ g | _ | j | _ d | _ d | _ t t | � j � d S)NF) r �stream�parser� escapeFlag� lastFourChars� dataState�state�escape� currentToken�superr �__init__)�selfr r �kwargs)� __class__� ��/builddir/build/BUILDROOT/alt-python35-pip-20.2.4-5.el8.x86_64/opt/alt/python35/lib/python3.5/site-packages/pip/_vendor/html5lib/_tokenizer.pyr" ( s zHTMLTokenizer.__init__c c s{ t g � | _ xe | j � rv x4 | j j rT d t d d | j j j d � i Vq! Wx | j rr | j j � VqX Wq Wd S)z� This is where the magic happens. We do our usually processing through the states and when we have a token to return we yield the token which pauses processing until the next token is requested. �type� ParseError�datar N)r � tokenQueuer r �errorsr �pop�popleft)r# r& r& r' �__iter__7 s ( zHTMLTokenizer.__iter__c % C s� t } d } | r t } d } g } | j j � } x8 | | k rm | t k rm | j | � | j j � } q6 Wt d j | � | � } | t k r� t | } | j j d t d d d d d | i i � n�d | k o� d k n s� | d k r(d } | j j d t d d d d d | i i � nld | k o?d k n sd | k o[d k n sd | k owd k n sd | k o�d k n s| t d d d d d d d d d d d d! d" d# d$ d% d& d' d( d) d* d+ d, d- d. d/ d0 d1 d2 d3 d4 d5 d6 d7 d g# � k r?| j j d t d d d d d | i i � y t | � } WnB t k r�| d8 } t d | d ?B� t d9 | d: @B� } Yn X| d; k r�| j j d t d d d< i � | j j | � | S)=z�This function returns either U+FFFD or the character based on the decimal or hexadecimal representation. It also discards ";" if present. If not present self.tokenQueue.append({"type": tokenTypes["ParseError"]}) is invoked. � � � r( r) r* z$illegal-codepoint-for-numeric-entity�datavars� charAsInti � i�� i�� u �r � � � � � i� i� � i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i i � i� �;z numeric-entity-without-semicolon)r r r �charr �append�int�joinr r+ r � frozenset�chr� ValueError�unget) r# �isHex�allowed�radix� charStack�cr4 r< �vr& r& r' �consumeNumberEntityG sb + z!HTMLTokenizer.consumeNumberEntityFc C s� d } | j j � g } | d t k s] | d t d d f k s] | d k rt | | d k rt | j j | d � n�| d d k rkd } | j | j j � � | d d k r� d } | j | j j � � | r� | d t k s� | r | d t k r | j j | d � | j | � } q5| j j d t d d d i � | j j | j � � d d j | � } n�xC | d t k r�t j d j | � � s�P| j | j j � � qnWy2 t j d j | d d � � � } t | � } Wn t k r�d } Yn X| d k r�| d d k r:| j j d t d d d i � | d d k r�| r�| | t k s�| | t k s�| | d k r�| j j | j � � d d j | � } q5t | } | j j | j � � | d j | | d � � 7} nI | j j d t d d d i � | j j | j � � d d j | � } | rW| j d d d | 7<n; | t k rld } n d } | j j d t | d | i � d S) N�&r �<�#Fr �x�XTr( r) r* zexpected-numeric-entityr2 r; z named-entity-without-semicolon�=zexpected-named-entity�SpaceCharacters� Characters���)rN rO rS rS rS rS rS rS rS rS )r r< r r rC r= r r rJ r+ r r- r? � entitiesTrie�has_keys_with_prefix�longest_prefix�len�KeyErrorr r r ) r# � allowedChar� fromAttribute�outputrG �hex� entityName� entityLength� tokenTyper& r& r' � consumeEntity� sf ) " zHTMLTokenizer.consumeEntityc C s | j d | d d � d S)zIThis method replaces the need for "entityInAttributeValueState". rY rZ TN)r` )r# rY r& r& r' �processEntityInAttribute� s z&HTMLTokenizer.processEntityInAttributec C s | j } | d t k r� | d j t � | d <| d t d k r� | d } t | � } t | � t | � k r� | j | d d d � � | | d <| d t d k r� | d r� | j j d t d d d i � | d r� | j j d t d d d i � | j j | � | j | _ d S) z�This method is a generic handler for emitting the tags. It also sets the state to "data" because that's what's needed after a token has been emitted. r( �name�StartTagr* Nr �EndTagr) zattributes-in-end-tag� selfClosingz self-closing-flag-on-end-tagrS ) r r � translater r � attributeMaprW �updater+ r= r r )r# �token�rawr* r& r& r' �emitCurrentToken� s$ z HTMLTokenizer.emitCurrentTokenc C s | j j � } | d k r* | j | _ n� | d k rE | j | _ n� | d k r� | j j d t d d d i � | j j d t d d d i � n� | t k r� d S| t k r� | j j d t d d | | j j t d � i � n6 | j j d � } | j j d t d d | | i � d S) NrK rL � r( r) r* zinvalid-codepointrR FrQ T)rK rL rl ) r r< �entityDataStater � tagOpenStater+ r= r r r � charsUntil)r# r* �charsr& r& r' r s& zHTMLTokenizer.dataStatec C s | j � | j | _ d S)NT)r` r r )r# r& r& r' rm s z HTMLTokenizer.entityDataStatec C s | j j � } | d k r* | j | _ n� | d k rE | j | _ n� | t k rU d S| d k r� | j j d t d d d i � | j j d t d d d i � nx | t k r� | j j d t d d | | j j t d � i � n6 | j j d � } | j j d t d d | | i � d S)NrK rL Frl r( r) r* zinvalid-codepointrR u �rQ T)rK rL rl ) r r< �characterReferenceInRcdatar �rcdataLessThanSignStater r+ r= r r ro )r# r* rp r& r& r' � rcdataState" s& zHTMLTokenizer.rcdataStatec C s | j � | j | _ d S)NT)r` rs r )r# r& r& r' rq ? s z(HTMLTokenizer.characterReferenceInRcdatac C s� | j j � } | d k r* | j | _ n� | d k ry | j j d t d d d i � | j j d t d d d i � nF | t k r� d S| j j d � } | j j d t d d | | i � d S) NrL rl r( r) r* zinvalid-codepointrR u �FT)rL rl ) r r< �rawtextLessThanSignStater r+ r= r r ro )r# r* rp r& r& r' � rawtextStateD s zHTMLTokenizer.rawtextStatec C s� | j j � } | d k r* | j | _ n� | d k ry | j j d t d d d i � | j j d t d d d i � nF | t k r� d S| j j d � } | j j d t d d | | i � d S) NrL rl r( r) r* zinvalid-codepointrR u �FT)rL rl ) r r< �scriptDataLessThanSignStater r+ r= r r ro )r# r* rp r& r& r' �scriptDataStateV s z HTMLTokenizer.scriptDataStatec C s� | j j � } | t k r d S| d k rn | j j d t d d d i � | j j d t d d d i � n0 | j j d t d d | | j j d � i � d S) NFrl r( r) r* zinvalid-codepointrR u �T)r r< r r+ r= r ro )r# r* r& r& r' �plaintextStateh s z HTMLTokenizer.plaintextStatec C s� | j j � } | d k r* | j | _ nc| d k rE | j | _ nH| t k r� d t d d | d g d d d d i | _ | j | _ n| d k r� | j j d t d d d i � | j j d t d d d i � | j | _ n� | d k r1| j j d t d d d i � | j j | � | j | _ n\ | j j d t d d d i � | j j d t d d d i � | j j | � | j | _ d S)N�!�/r( rc rb r* re F�selfClosingAcknowledged�>r) z'expected-tag-name-but-got-right-bracketrR z<>�?z'expected-tag-name-but-got-question-markzexpected-tag-namerL T)r r< �markupDeclarationOpenStater �closeTagOpenStater r r � tagNameStater+ r= r rC �bogusCommentState)r# r* r& r& r' rn w s6 zHTMLTokenizer.tagOpenStatec C s1 | j j � } | t k rO d t d d | d g d d i | _ | j | _ n� | d k r� | j j d t d d d i � | j | _ n� | t k r� | j j d t d d d i � | j j d t d d d i � | j | _ nH | j j d t d d d d d | i i � | j j | � | j | _ d S)Nr( rd rb r* re Fr| r) z*expected-closing-tag-but-got-right-bracketz expected-closing-tag-but-got-eofrR z</z!expected-closing-tag-but-got-charr3 T) r r< r r r r� r r+ r= r r rC r� )r# r* r& r& r' r � s( z HTMLTokenizer.closeTagOpenStatec C s� | j j � } | t k r* | j | _ n� | d k rC | j � n� | t k r~ | j j d t d d d i � | j | _ nl | d k r� | j | _ nQ | d k r� | j j d t d d d i � | j d d 7<n | j d | 7<d S) Nr| r( r) r* zeof-in-tag-namerz rl zinvalid-codepointrb u �T) r r< r �beforeAttributeNameStater rk r r+ r= r r �selfClosingStartTagStater )r# r* r& r& r' r� � s" zHTMLTokenizer.tagNameStatec C ss | j j � } | d k r3 d | _ | j | _ n< | j j d t d d d i � | j j | � | j | _ d S)Nrz r2 r( rR r* rL T) r r< �temporaryBuffer�rcdataEndTagOpenStater r+ r= r rC rs )r# r* r& r& r' rr � s z%HTMLTokenizer.rcdataLessThanSignStatec C sy | j j � } | t k r9 | j | 7_ | j | _ n< | j j d t d d d i � | j j | � | j | _ d S)Nr( rR r* z</T) r r< r r� �rcdataEndTagNameStater r+ r= r rC rs )r# r* r& r& r' r� � s z#HTMLTokenizer.rcdataEndTagOpenStatec C s� | j o( | j d j � | j j � k } | j j � } | t k r� | r� d t d d | j d g d d i | _ | j | _ n� | d k r� | r� d t d d | j d g d d i | _ | j | _ n� | d k r | r d t d d | j d g d d i | _ | j � | j | _ na | t k r=| j | 7_ nC | j j d t d d d | j i � | j j | � | j | _ d S) Nrb r( rd r* re Frz r| rR z</T)r �lowerr� r r< r r r� r r� rk r r r+ r= rC rs )r# � appropriater* r& r& r' r� � s2 + z#HTMLTokenizer.rcdataEndTagNameStatec C ss | j j � } | d k r3 d | _ | j | _ n< | j j d t d d d i � | j j | � | j | _ d S)Nrz r2 r( rR r* rL T) r r< r� �rawtextEndTagOpenStater r+ r= r rC ru )r# r* r& r& r' rt � s z&HTMLTokenizer.rawtextLessThanSignStatec C sy | j j � } | t k r9 | j | 7_ | j | _ n< | j j d t d d d i � | j j | � | j | _ d S)Nr( rR r* z</T) r r< r r� �rawtextEndTagNameStater r+ r= r rC ru )r# r* r& r& r' r� s z$HTMLTokenizer.rawtextEndTagOpenStatec C s� | j o( | j d j � | j j � k } | j j � } | t k r� | r� d t d d | j d g d d i | _ | j | _ n� | d k r� | r� d t d d | j d g d d i | _ | j | _ n� | d k r | r d t d d | j d g d d i | _ | j � | j | _ na | t k r=| j | 7_ nC | j j d t d d d | j i � | j j | � | j | _ d S) Nrb r( rd r* re Frz r| rR z</T)r r� r� r r< r r r� r r� rk r r r+ r= rC ru )r# r� r* r& r& r' r� s2 + z$HTMLTokenizer.rawtextEndTagNameStatec C s� | j j � } | d k r3 d | _ | j | _ nw | d k rn | j j d t d d d i � | j | _ n< | j j d t d d d i � | j j | � | j | _ d S) Nrz r2 ry r( rR r* z<!rL T) r r< r� �scriptDataEndTagOpenStater r+ r= r �scriptDataEscapeStartStaterC rw )r# r* r& r& r' rv , s z)HTMLTokenizer.scriptDataLessThanSignStatec C sy | j j � } | t k r9 | j | 7_ | j | _ n< | j j d t d d d i � | j j | � | j | _ d S)Nr( rR r* z</T) r r< r r� �scriptDataEndTagNameStater r+ r= r rC rw )r# r* r& r& r' r� : s z'HTMLTokenizer.scriptDataEndTagOpenStatec C s� | j o( | j d j � | j j � k } | j j � } | t k r� | r� d t d d | j d g d d i | _ | j | _ n� | d k r� | r� d t d d | j d g d d i | _ | j | _ n� | d k r | r d t d d | j d g d d i | _ | j � | j | _ na | t k r=| j | 7_ nC | j j d t d d d | j i � | j j | � | j | _ d S) Nrb r( rd r* re Frz r| rR z</T)r r� r� r r< r r r� r r� rk r r r+ r= rC rw )r# r� r* r& r& r' r� E s2 + z'HTMLTokenizer.scriptDataEndTagNameStatec C sj | j j � } | d k rJ | j j d t d d d i � | j | _ n | j j | � | j | _ d S)N�-r( rR r* T) r r< r+ r= r � scriptDataEscapeStartDashStater rC rw )r# r* r& r& r' r� a s z(HTMLTokenizer.scriptDataEscapeStartStatec C sj | j j � } | d k rJ | j j d t d d d i � | j | _ n | j j | � | j | _ d S)Nr� r( rR r* T) r r< r+ r= r � scriptDataEscapedDashDashStater rC rw )r# r* r& r& r' r� k s z,HTMLTokenizer.scriptDataEscapeStartDashStatec C s | j j � } | d k rJ | j j d t d d d i � | j | _ n� | d k re | j | _ n� | d k r� | j j d t d d d i � | j j d t d d d i � nQ | t k r� | j | _ n6 | j j d � } | j j d t d d | | i � d S) Nr� r( rR r* rL rl r) zinvalid-codepointu �T)rL r� rl ) r r< r+ r= r �scriptDataEscapedDashStater �"scriptDataEscapedLessThanSignStater r ro )r# r* rp r& r& r' �scriptDataEscapedStateu s" z$HTMLTokenizer.scriptDataEscapedStatec C s | j j � } | d k rJ | j j d t d d d i � | j | _ n� | d k re | j | _ n� | d k r� | j j d t d d d i � | j j d t d d d i � | j | _ nG | t k r� | j | _ n, | j j d t d d | i � | j | _ d S) Nr� r( rR r* rL rl r) zinvalid-codepointu �T) r r< r+ r= r r� r r� r� r r )r# r* r& r& r' r� � s" z(HTMLTokenizer.scriptDataEscapedDashStatec C s: | j j � } | d k r> | j j d t d d d i � n� | d k rY | j | _ n� | d k r� | j j d t d d d i � | j | _ n� | d k r� | j j d t d d d i � | j j d t d d d i � | j | _ nG | t k r | j | _ n, | j j d t d d | i � | j | _ d S) Nr� r( rR r* rL r| rl r) zinvalid-codepointu �T) r r< r+ r= r r� r rw r� r r )r# r* r& r& r' r� � s& # z,HTMLTokenizer.scriptDataEscapedDashDashStatec C s� | j j � } | d k r3 d | _ | j | _ n� | t k r{ | j j d t d d d | i � | | _ | j | _ n< | j j d t d d d i � | j j | � | j | _ d S)Nrz r2 r( rR r* rL T) r r< r� � scriptDataEscapedEndTagOpenStater r r+ r= r � scriptDataDoubleEscapeStartStaterC r� )r# r* r& r& r' r� � s $ z0HTMLTokenizer.scriptDataEscapedLessThanSignStatec C ss | j j � } | t k r3 | | _ | j | _ n< | j j d t d d d i � | j j | � | j | _ d S)Nr( rR r* z</T) r r< r r� � scriptDataEscapedEndTagNameStater r+ r= r rC r� )r# r* r& r& r' r� � s z.HTMLTokenizer.scriptDataEscapedEndTagOpenStatec C s� | j o( | j d j � | j j � k } | j j � } | t k r� | r� d t d d | j d g d d i | _ | j | _ n� | d k r� | r� d t d d | j d g d d i | _ | j | _ n� | d k r | r d t d d | j d g d d i | _ | j � | j | _ na | t k r=| j | 7_ nC | j j d t d d d | j i � | j j | � | j | _ d S) Nrb r( rd r* re Frz r| rR z</T)r r� r� r r< r r r� r r� rk r r r+ r= rC r� )r# r� r* r& r& r' r� � s2 + z.HTMLTokenizer.scriptDataEscapedEndTagNameStatec C s� | j j � } | t t d � Bk rx | j j d t d d | i � | j j � d k ri | j | _ q� | j | _ nZ | t k r� | j j d t d d | i � | j | 7_ n | j j | � | j | _ d S) Nrz r| r( rR r* �scriptT)rz r| )r r< r r@ r+ r= r r� r� � scriptDataDoubleEscapedStater r� r rC )r# r* r& r& r' r� � s z.HTMLTokenizer.scriptDataDoubleEscapeStartStatec C s3 | j j � } | d k rJ | j j d t d d d i � | j | _ n� | d k r� | j j d t d d d i � | j | _ n� | d k r� | j j d t d d d i � | j j d t d d d i � n[ | t k r| j j d t d d d i � | j | _ n | j j d t d d | i � d S) Nr� r( rR r* rL rl r) zinvalid-codepointu �zeof-in-script-in-scriptT) r r< r+ r= r � scriptDataDoubleEscapedDashStater �(scriptDataDoubleEscapedLessThanSignStater r )r# r* r& r& r' r� � s$ z*HTMLTokenizer.scriptDataDoubleEscapedStatec C sK | j j � } | d k rJ | j j d t d d d i � | j | _ n� | d k r� | j j d t d d d i � | j | _ n� | d k r� | j j d t d d d i � | j j d t d d d i � | j | _ ng | t k r| j j d t d d d i � | j | _ n, | j j d t d d | i � | j | _ d S) Nr� r( rR r* rL rl r) zinvalid-codepointu �zeof-in-script-in-scriptT) r r< r+ r= r �$scriptDataDoubleEscapedDashDashStater r� r� r r )r# r* r& r& r' r� s( z.HTMLTokenizer.scriptDataDoubleEscapedDashStatec C sz | j j � } | d k r> | j j d t d d d i � n8| d k ry | j j d t d d d i � | j | _ n� | d k r� | j j d t d d d i � | j | _ n� | d k r| j j d t d d d i � | j j d t d d d i � | j | _ ng | t k rJ| j j d t d d d i � | j | _ n, | j j d t d d | i � | j | _ d S) Nr� r( rR r* rL r| rl r) zinvalid-codepointu �zeof-in-script-in-scriptT) r r< r+ r= r r� r rw r� r r )r# r* r& r& r' r� % s, # z2HTMLTokenizer.scriptDataDoubleEscapedDashDashStatec C ss | j j � } | d k rS | j j d t d d d i � d | _ | j | _ n | j j | � | j | _ d S)Nrz r( rR r* r2 T) r r< r+ r= r r� � scriptDataDoubleEscapeEndStater rC r� )r# r* r& r& r' r� > s z6HTMLTokenizer.scriptDataDoubleEscapedLessThanSignStatec C s� | j j � } | t t d � Bk rx | j j d t d d | i � | j j � d k ri | j | _ q� | j | _ nZ | t k r� | j j d t d d | i � | j | 7_ n | j j | � | j | _ d S) Nrz r| r( rR r* r� T)rz r| )r r< r r@ r+ r= r r� r� r� r r� r rC )r# r* r& r& r' r� I s z,HTMLTokenizer.scriptDataDoubleEscapeEndStatec C s� | j j � } | t k r1 | j j t d � nt| t k rf | j d j | d g � | j | _ n?| d k r | j � n&| d k r� | j | _ n | d k r� | j j d t d d d i � | j d j | d g � | j | _ n� | d k rD| j j d t d d d i � | j d j d d g � | j | _ na | t k r| j j d t d d d i � | j | _ n& | j d j | d g � | j | _ d S)NTr* r2 r| rz �'�"rP rL r( r) z#invalid-character-in-attribute-namerl zinvalid-codepointu �z#expected-attribute-name-but-got-eof)r� r� rP rL )r r< r ro r r r= �attributeNameStater rk r� r+ r r r )r# r* r& r& r' r� Y s6 z&HTMLTokenizer.beforeAttributeNameStatec C sc | j j � } d } d } | d k r6 | j | _ n�| t k rw | j d d d | | j j t d � 7<d } nA| d k r� d } n,| t k r� | j | _ n| d k r� | j | _ n� | d k r| j j d t d d d i � | j d d d d 7<d } n� | d k r^| j j d t d d d i � | j d d d | 7<d } nZ | t k r�| j j d t d d d i � | j | _ n | j d d d | 7<d } | r_| j d d d j t � | j d d d <xa | j d d d � D]H \ } } | j d d d | k r| j j d t d d d i � PqW| r_| j � d S) NTFrP r* r r r| rz rl r( r) zinvalid-codepointu �r� r� rL z#invalid-character-in-attribute-namezeof-in-attribute-namezduplicate-attributerS rS )r� r� rL rS rS rS rS rS rS )r r< �beforeAttributeValueStater r r ro r �afterAttributeNameStater� r+ r= r r r rf r rk )r# r* �leavingThisState� emitTokenrb �_r& r& r' r� w sT -$ z HTMLTokenizer.attributeNameStatec C s� | j j � } | t k r1 | j j t d � n�| d k rL | j | _ nt| d k re | j � n[| t k r� | j d j | d g � | j | _ n&| d k r� | j | _ n | d k r | j j d t d d d i � | j d j d d g � | j | _ n� | d k r_| j j d t d d d i � | j d j | d g � | j | _ na | t k r�| j j d t d d d i � | j | _ n& | j d j | d g � | j | _ d S)NTrP r| r* r2 rz rl r( r) zinvalid-codepointu �r� r� rL z&invalid-character-after-attribute-namez expected-end-of-tag-but-got-eof)r� r� rL )r r< r ro r� r rk r r r= r� r� r+ r r r )r# r* r& r& r' r� � s: z%HTMLTokenizer.afterAttributeNameStatec C s� | j j � } | t k r1 | j j t d � n�| d k rL | j | _ n�| d k rw | j | _ | j j | � n\| d k r� | j | _ nA| d k r� | j j d t d d d i � | j � n| d k r | j j d t d d d i � | j d d d d 7<| j | _ n� | d k rs| j j d t d d d i � | j d d d | 7<| j | _ n` | t k r�| j j d t d d d i � | j | _ n% | j d d d | 7<| j | _ d S)NTr� rK r� r| r( r) r* z.expected-attribute-value-but-got-right-bracketrl zinvalid-codepointr u �rP rL �`z"equals-in-unquoted-attribute-valuez$expected-attribute-value-but-got-eofrS )rP rL r� rS rS )r r< r ro � attributeValueDoubleQuotedStater �attributeValueUnQuotedStaterC � attributeValueSingleQuotedStater+ r= r rk r r r )r# r* r& r& r' r� � s> z'HTMLTokenizer.beforeAttributeValueStatec C s� | j j � } | d k r* | j | _ n� | d k rF | j d � n� | d k r� | j j d t d d d i � | j d d d d 7<nd | t k r� | j j d t d d d i � | j | _ n) | j d d d | | j j d � 7<d S)Nr� rK rl r( r) r* zinvalid-codepointr u �z#eof-in-attribute-value-double-quoteTrS rS )r� rK rl ) r r< �afterAttributeValueStater ra r+ r= r r r r ro )r# r* r& r& r' r� � s z-HTMLTokenizer.attributeValueDoubleQuotedStatec C s� | j j � } | d k r* | j | _ n� | d k rF | j d � n� | d k r� | j j d t d d d i � | j d d d d 7<nd | t k r� | j j d t d d d i � | j | _ n) | j d d d | | j j d � 7<d S)Nr� rK rl r( r) r* zinvalid-codepointr u �z#eof-in-attribute-value-single-quoteTrS rS )r� rK rl ) r r< r� r ra r+ r= r r r r ro )r# r* r& r& r' r� s z-HTMLTokenizer.attributeValueSingleQuotedStatec C sa | j j � } | t k r* | j | _ n3| d k rF | j d � n| d k r_ | j � n� | d k r� | j j d t d d d i � | j d d d | 7<n� | d k r� | j j d t d d d i � | j d d d d 7<nn | t k r*| j j d t d d d i � | j | _ n3 | j d d d | | j j t d � t B� 7<d S)NrK r| r� r� rP rL r� r( r) r* z0unexpected-character-in-unquoted-attribute-valuer rl zinvalid-codepointu �z eof-in-attribute-value-no-quotesT)r� r� rP rL r� rS rS rS )rK r| r� r� rP rL r� rl )r r< r r� r ra rk r+ r= r r r r ro r@ )r# r* r& r& r' r� s, z)HTMLTokenizer.attributeValueUnQuotedStatec C s� | j j � } | t k r* | j | _ n� | d k rC | j � n� | d k r^ | j | _ n� | t k r� | j j d t d d d i � | j j | � | j | _ n< | j j d t d d d i � | j j | � | j | _ d S) Nr| rz r( r) r* z$unexpected-EOF-after-attribute-valuez*unexpected-character-after-attribute-valueT) r r< r r� r rk r� r r+ r= r rC r )r# r* r& r& r' r� . s" z&HTMLTokenizer.afterAttributeValueStatec C s� | j j � } | d k r5 d | j d <| j � n� | t k r� | j j d t d d d i � | j j | � | j | _ n< | j j d t d d d i � | j j | � | j | _ d S) Nr| Tre r( r) r* z#unexpected-EOF-after-solidus-in-tagz)unexpected-character-after-solidus-in-tag) r r< r rk r r+ r= r rC r r r� )r# r* r& r& r' r� B s z&HTMLTokenizer.selfClosingStartTagStatec C sa | j j d � } | j d d � } | j j d t d d | i � | j j � | j | _ d S)Nr| rl u �r( �Commentr* T) r ro �replacer+ r= r r< r r )r# r* r& r& r' r� T s z HTMLTokenizer.bogusCommentStatec C s | j j � g } | d d k rt | j | j j � � | d! d k r�d t d d d i | _ | j | _ d SnW| d" d# k r d } x; d* D]3 } | j | j j � � | d+ | k r� d } Pq� W| r�d t d d d d d d d d d i | _ | j | _ d Sn� | d, d k r�| j d k r�| j j j r�| j j j d- j | j j j k r�d } xM d d d d d d g D]3 } | j | j j � � | d. | k r~d } Pq~W| r�| j | _ d S| j j d t d d d i � x | r | j j | j � � q�W| j | _ d S)/Nr r� r( r� r* r2 T�d�D�o�OrH �C�t�T�y�Y�p�P�e�EF�Doctyperb �publicId�systemId�correct�[�Ar) zexpected-dashes-or-doctyperS rS rS )r� r� �r� r� �rH r� �r� r� �r� r� �r� r� �r� r� )r� r� r� r� r� r� rS rS rS rS )r r< r= r r �commentStartStater � doctypeStater �tree� openElements� namespace�defaultNamespace�cdataSectionStater+ rC r- r� )r# rG �matched�expectedr& r& r' r~ c sR % z(HTMLTokenizer.markupDeclarationOpenStatec C s' | j j � } | d k r* | j | _ n� | d k rj | j j d t d d d i � | j d d 7<n� | d k r� | j j d t d d d i � | j j | j � | j | _ nk | t k r| j j d t d d d i � | j j | j � | j | _ n | j d | 7<| j | _ d S) Nr� rl r( r) r* zinvalid-codepointu �r| zincorrect-commentzeof-in-commentT) r r< �commentStartDashStater r+ r= r r r r � commentState)r# r* r& r& r' r� � s( z HTMLTokenizer.commentStartStatec C s+ | j j � } | d k r* | j | _ n� | d k rj | j j d t d d d i � | j d d 7<n� | d k r� | j j d t d d d i � | j j | j � | j | _ no | t k r| j j d t d d d i � | j j | j � | j | _ n! | j d d | 7<| j | _ d S) Nr� rl r( r) r* zinvalid-codepointu -�r| zincorrect-commentzeof-in-commentT) r r< �commentEndStater r+ r= r r r r r� )r# r* r& r& r' r� � s( z#HTMLTokenizer.commentStartDashStatec C s� | j j � } | d k r* | j | _ n� | d k rj | j j d t d d d i � | j d d 7<no | t k r� | j j d t d d d i � | j j | j � | j | _ n! | j d | | j j d � 7<d S) Nr� rl r( r) r* zinvalid-codepointu �zeof-in-commentT)r� rl ) r r< �commentEndDashStater r+ r= r r r r ro )r# r* r& r& r' r� � s zHTMLTokenizer.commentStatec C s� | j j � } | d k r* | j | _ n� | d k rv | j j d t d d d i � | j d d 7<| j | _ no | t k r� | j j d t d d d i � | j j | j � | j | _ n! | j d d | 7<| j | _ d S) Nr� rl r( r) r* zinvalid-codepointu -�zeof-in-comment-end-dashT) r r< r� r r+ r= r r r� r r )r# r* r& r& r' r� � s z!HTMLTokenizer.commentEndDashStatec C s� | j j � } | d k r= | j j | j � | j | _ nV| d k r� | j j d t d d d i � | j d d 7<| j | _ n | d k r� | j j d t d d d i � | j | _ n� | d k r| j j d t d d d i � | j d | 7<n� | t k rR| j j d t d d d i � | j j | j � | j | _ nA | j j d t d d d i � | j d d | 7<| j | _ d S)Nr| rl r( r) r* zinvalid-codepointu --�ry z,unexpected-bang-after-double-dash-in-commentr� z,unexpected-dash-after-double-dash-in-commentzeof-in-comment-double-dashzunexpected-char-in-commentz--T) r r< r+ r= r r r r r� �commentEndBangStater )r# r* r& r& r' r� � s6 z HTMLTokenizer.commentEndStatec C s( | j j � } | d k r= | j j | j � | j | _ n� | d k ri | j d d 7<| j | _ n� | d k r� | j j d t d d d i � | j d d 7<| j | _ no | t k r| j j d t d d d i � | j j | j � | j | _ n! | j d d | 7<| j | _ d S) Nr| r� r* z--!rl r( r) zinvalid-codepointu --!�z eof-in-comment-end-bang-stateT) r r< r+ r= r r r r� r r� r )r# r* r& r& r' r� s( z!HTMLTokenizer.commentEndBangStatec C s� | j j � } | t k r* | j | _ n� | t k r� | j j d t d d d i � d | j d <| j j | j � | j | _ n< | j j d t d d d i � | j j | � | j | _ d S) Nr( r) r* z!expected-doctype-name-but-got-eofFr� zneed-space-after-doctypeT) r r< r �beforeDoctypeNameStater r r+ r= r r r rC )r# r* r& r& r' r� s zHTMLTokenizer.doctypeStatec C s9 | j j � } | t k r n| d k ry | j j d t d d d i � d | j d <| j j | j � | j | _ n� | d k r� | j j d t d d d i � d | j d <| j | _ nt | t k r | j j d t d d d i � d | j d <| j j | j � | j | _ n | | j d <| j | _ d S)Nr| r( r) r* z+expected-doctype-name-but-got-right-bracketFr� rl zinvalid-codepointu �rb z!expected-doctype-name-but-got-eofT) r r< r r+ r= r r r r �doctypeNameStater )r# r* r& r& r' r� * s. z$HTMLTokenizer.beforeDoctypeNameStatec C sk | j j � } | t k rG | j d j t � | j d <| j | _ n | d k r� | j d j t � | j d <| j j | j � | j | _ n� | d k r� | j j d t d d d i � | j d d 7<| j | _ n� | t k rV| j j d t d d d i � d | j d <| j d j t � | j d <| j j | j � | j | _ n | j d | 7<d S) Nrb r| rl r( r) r* zinvalid-codepointu �zeof-in-doctype-nameFr� T)r r< r r rf r �afterDoctypeNameStater r+ r= r r r� r )r# r* r& r& r' r� D s, z HTMLTokenizer.doctypeNameStatec C s� | j j � } | t k r n�| d k rL | j j | j � | j | _ ny| t k r� d | j d <| j j | � | j j d t d d d i � | j j | j � | j | _ n| d! k rd } x0 d' D]( } | j j � } | | k r� d } Pq� W| rp| j | _ d Sn[ | d( k rpd } x0 d. D]( } | j j � } | | k r.d } Pq.W| rp| j | _ d S| j j | � | j j d t d d d d d | i i � d | j d <| j | _ d S)/Nr| Fr� r( r) r* zeof-in-doctyper� r� T�u�U�b�B�l�L�i�IrH r� �s�Sr� r� r� r� r� r� �m�Mz*expected-space-or-right-bracket-in-doctyper3 )r� r� �r� r� �r� r� �r� r� �r� r� �rH r� )r� r� r� r� r� )r� r� �r� r� �r� r� �r� r� �r� r� �r� r� )r� r� r� r� r� )r r< r r+ r= r r r r rC r � afterDoctypePublicKeywordState� afterDoctypeSystemKeywordState�bogusDoctypeState)r# r* r� r� r& r& r' r� ] sT z#HTMLTokenizer.afterDoctypeNameStatec C s� | j j � } | t k r* | j | _ n� | d k ru | j j d t d d d i � | j j | � | j | _ nw | t k r� | j j d t d d d i � d | j d <| j j | j � | j | _ n | j j | � | j | _ d S) Nr� r� r( r) r* zunexpected-char-in-doctypezeof-in-doctypeFr� T)r� r� ) r r< r �"beforeDoctypePublicIdentifierStater r+ r= r rC r r r )r# r* r& r& r' r� � s" z,HTMLTokenizer.afterDoctypePublicKeywordStatec C sa | j j � } | t k r n?| d k rF d | j d <| j | _ n| d k rn d | j d <| j | _ n� | d k r� | j j d t d d d i � d | j d <| j j | j � | j | _ n� | t k r$| j j d t d d d i � d | j d <| j j | j � | j | _ n9 | j j d t d d d i � d | j d <| j | _ d S)Nr� r2 r� r� r| r( r) r* zunexpected-end-of-doctypeFr� zeof-in-doctypezunexpected-char-in-doctypeT) r r< r r �(doctypePublicIdentifierDoubleQuotedStater �(doctypePublicIdentifierSingleQuotedStater+ r= r r r r� )r# r* r& r& r' r� � s4 z0HTMLTokenizer.beforeDoctypePublicIdentifierStatec C s5 | j j � } | d k r* | j | _ n| d k rj | j j d t d d d i � | j d d 7<n� | d k r� | j j d t d d d i � d | j d <| j j | j � | j | _ nl | t k r | j j d t d d d i � d | j d <| j j | j � | j | _ n | j d | 7<d S)Nr� rl r( r) r* zinvalid-codepointr� u �r| zunexpected-end-of-doctypeFr� zeof-in-doctypeT) r r< �!afterDoctypePublicIdentifierStater r+ r= r r r r )r# r* r& r& r' r� � s* z6HTMLTokenizer.doctypePublicIdentifierDoubleQuotedStatec C s5 | j j � } | d k r* | j | _ n| d k rj | j j d t d d d i � | j d d 7<n� | d k r� | j j d t d d d i � d | j d <| j j | j � | j | _ nl | t k r | j j d t d d d i � d | j d <| j j | j � | j | _ n | j d | 7<d S)Nr� rl r( r) r* zinvalid-codepointr� u �r| zunexpected-end-of-doctypeFr� zeof-in-doctypeT) r r< r� r r+ r= r r r r )r# r* r& r& r' r� � s* z6HTMLTokenizer.doctypePublicIdentifierSingleQuotedStatec C s� | j j � } | t k r* | j | _ nR| d k rX | j j | j � | j | _ n$| d k r� | j j d t d d d i � d | j d <| j | _ n� | d k r� | j j d t d d d i � d | j d <| j | _ n� | t k rC| j j d t d d d i � d | j d <| j j | j � | j | _ n9 | j j d t d d d i � d | j d <| j | _ d S)Nr| r� r( r) r* zunexpected-char-in-doctyper2 r� r� zeof-in-doctypeFr� T)r r< r �-betweenDoctypePublicAndSystemIdentifiersStater r+ r= r r r �(doctypeSystemIdentifierDoubleQuotedState�(doctypeSystemIdentifierSingleQuotedStater r� )r# r* r& r& r' r� � s6 z/HTMLTokenizer.afterDoctypePublicIdentifierStatec C s4 | j j � } | t k r n| d k rL | j j | j � | j | _ n� | d k rt d | j d <| j | _ n� | d k r� d | j d <| j | _ n� | t k r� | j j d t d d d i � d | j d <| j j | j � | j | _ n9 | j j d t d d d i � d | j d <| j | _ d S)Nr| r� r2 r� r� r( r) r* zeof-in-doctypeFr� zunexpected-char-in-doctypeT) r r< r r+ r= r r r r� r� r r r� )r# r* r& r& r' r� s. z;HTMLTokenizer.betweenDoctypePublicAndSystemIdentifiersStatec C s� | j j � } | t k r* | j | _ n� | d k ru | j j d t d d d i � | j j | � | j | _ nw | t k r� | j j d t d d d i � d | j d <| j j | j � | j | _ n | j j | � | j | _ d S) Nr� r� r( r) r* zunexpected-char-in-doctypezeof-in-doctypeFr� T)r� r� ) r r< r �"beforeDoctypeSystemIdentifierStater r+ r= r rC r r r )r# r* r& r& r' r� ) s" z,HTMLTokenizer.afterDoctypeSystemKeywordStatec C sa | j j � } | t k r n?| d k rF d | j d <| j | _ n| d k rn d | j d <| j | _ n� | d k r� | j j d t d d d i � d | j d <| j j | j � | j | _ n� | t k r$| j j d t d d d i � d | j d <| j j | j � | j | _ n9 | j j d t d d d i � d | j d <| j | _ d S)Nr� r2 r� r� r| r( r) r* zunexpected-char-in-doctypeFr� zeof-in-doctypeT) r r< r r r� r r� r+ r= r r r r� )r# r* r& r& r' r� = s4 z0HTMLTokenizer.beforeDoctypeSystemIdentifierStatec C s5 | j j � } | d k r* | j | _ n| d k rj | j j d t d d d i � | j d d 7<n� | d k r� | j j d t d d d i � d | j d <| j j | j � | j | _ nl | t k r | j j d t d d d i � d | j d <| j j | j � | j | _ n | j d | 7<d S)Nr� rl r( r) r* zinvalid-codepointr� u �r| zunexpected-end-of-doctypeFr� zeof-in-doctypeT) r r< �!afterDoctypeSystemIdentifierStater r+ r= r r r r )r# r* r& r& r' r� Z s* z6HTMLTokenizer.doctypeSystemIdentifierDoubleQuotedStatec C s5 | j j � } | d k r* | j | _ n| d k rj | j j d t d d d i � | j d d 7<n� | d k r� | j j d t d d d i � d | j d <| j j | j � | j | _ nl | t k r | j j d t d d d i � d | j d <| j j | j � | j | _ n | j d | 7<d S)Nr� rl r( r) r* zinvalid-codepointr� u �r| zunexpected-end-of-doctypeFr� zeof-in-doctypeT) r r< r� r r+ r= r r r r )r# r* r& r& r' r� r s* z6HTMLTokenizer.doctypeSystemIdentifierSingleQuotedStatec C s� | j j � } | t k r n� | d k rL | j j | j � | j | _ n� | t k r� | j j d t d d d i � d | j d <| j j | j � | j | _ n, | j j d t d d d i � | j | _ d S) Nr| r( r) r* zeof-in-doctypeFr� zunexpected-char-in-doctypeT) r r< r r+ r= r r r r r r� )r# r* r& r& r' r� � s z/HTMLTokenizer.afterDoctypeSystemIdentifierStatec C s | j j � } | d k r= | j j | j � | j | _ n> | t k r{ | j j | � | j j | j � | j | _ n d S)Nr| T) r r< r+ r= r r r r rC )r# r* r& r& r' r� � s z HTMLTokenizer.bogusDoctypeStatec C s\ g } x� | j | j j d � � | j | j j d � � | j j � } | t k rZ Pq | d k sl t � | d d d � d k r� | d d d � | d <Pq | j | � q Wd j | � } | j d � } | d k r&x4 t | � D]& } | j j d t d d d i � q� W| j d d � } | rL| j j d t d d | i � | j | _ d S)N�]r| r � z]]r2 rl r r( r) r* zinvalid-codepointu �rR TrS ���rS r� rS )r= r ro r< r �AssertionErrorr? �count�ranger+ r r� r r )r# r* r< � nullCountr� r& r& r' r� � s0 z HTMLTokenizer.cdataSectionState)M�__name__� __module__� __qualname__�__doc__r" r/ rJ r` ra rk r rm rs rq ru rw rx rn r r� rr r� r� rt r� r� rv r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r~ r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r& r& )r% r' r s� HP # 6 " - 3 r N)r r ) � __future__r r r Zpip._vendor.sixr rA � collectionsr r �sysr � constantsr r r r r r r r r r � _inputstreamr �_trier rT �dictrg �objectr r& r& r& r' �<module> s