관리-도구
편집 파일: _tokenizer.cpython-39.pyc
a ��Re�, � @ s� d dl mZmZmZ d dlmZ d dlmZm Z d dl m Z ddl m Z ddl mZ ddl mZmZ dd l mZmZmZ dd l mZmZ dd l mZ dd lmZ dd lmZ ee�Ze dkr�e Z ne Z G dd� de �Z dS )� )�absolute_import�division�unicode_literals)�unichr)�deque� OrderedDict)� version_info� )�spaceCharacters)�entities)� asciiLetters�asciiUpper2Lower)�digits� hexDigits�EOF)� tokenTypes� tagTokenTypes)�replacementCharacters)�HTMLInputStream)�Trie)� � c sd e Zd ZdZd�� fdd� Zdd� Zdd� Zd�d d �Zd d � Zdd� Z dd� Z dd� Z dd� Z dd� Z dd� Zdd� Zd d � Zd d � Zd d!� Zd"d#� Zd$d%� Zd&d'� Zd(d)� Zd*d+� Zd,d-� Zd.d/� Zd0d1� Zd2d3� Zd4d5� Z d6d7� Z d8d9� Z d:d;� Z d<d=� Z d>d?� Z!d@dA� Z"dBdC� Z#dDdE� Z$dFdG� Z%dHdI� Z&dJdK� Z'dLdM� Z(dNdO� Z)dPdQ� Z*dRdS� Z+dTdU� Z,dVdW� Z-dXdY� Z.dZd[� Z/d\d]� Z0d^d_� Z1d`da� Z2dbdc� Z3ddde� Z4dfdg� Z5dhdi� Z6djdk� Z7dldm� Z8dndo� Z9dpdq� Z:drds� Z;dtdu� Z<dvdw� Z=dxdy� Z>dzd{� Z?d|d}� Z@d~d� ZAd�d�� ZBd�d�� ZCd�d�� ZDd�d�� ZEd�d�� ZFd�d�� ZGd�d�� ZHd�d�� ZId�d�� ZJd�d�� ZKd�d�� ZL� ZMS )�� HTMLTokenizera This class takes care of tokenizing HTML. * self.currentToken Holds the token that is currently being processed. * self.state Holds a reference to the method to be invoked... XXX * self.stream Points to HTMLInputStream object. Nc sJ t |fi |��| _|| _d| _g | _| j| _d| _d | _t t | �� � d S )NF) r �stream�parser� escapeFlag� lastFourChars� dataState�state�escape� currentToken�superr �__init__)�selfr r �kwargs�� __class__� ��/builddir/build/BUILDROOT/alt-python39-pip-21.3.1-2.el8.x86_64/opt/alt/python39/lib/python3.9/site-packages/pip/_vendor/html5lib/_tokenizer.pyr" ( s zHTMLTokenizer.__init__c c sP t g �| _| �� rL| jjr6td | jj�d�d�V q| jr | j�� V q6q dS )z� This is where the magic happens. We do our usually processing through the states and when we have a token to return we yield the token which pauses processing until the next token is requested. � ParseErrorr ��type�dataN)r � tokenQueuer r �errorsr �pop�popleft�r# r' r' r( �__iter__7 s zHTMLTokenizer.__iter__c C s� t }d}|rt}d}g }| j�� }||v rH|turH|�|� | j�� }q"td�|�|�}|tv r�t| }| j �t d dd|id�� �n d| kr�d ks�n |d kr�d }| j �t d dd|id�� n�d | kr�d k�s>n d| kr�dk�s>n d| k�rdk�s>n d| k�r*dk�s>n |t g d��v �rZ| j �t d dd|id�� z t |�}W n< t �y� |d }t d|d? B �t d|d@ B � }Y n0 |dk�r�| j �t d dd�� | j�|� |S )z�This function returns either U+FFFD or the character based on the decimal or hexadecimal representation. It also discards ";" if present. If not present self.tokenQueue.append({"type": tokenTypes["ParseError"]}) is invoked. � � � r) z$illegal-codepoint-for-numeric-entity� charAsInt�r+ r, �datavarsi � i�� � � �r � � � � � i� i� )#� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� r9 i i � i� �;z numeric-entity-without-semicolonr* )r r r �charr �append�int�joinr r- r � frozenset�chr� ValueError�unget) r# �isHex�allowed�radix� charStack�cr6 rB �vr' r' r( �consumeNumberEntityG sn � � � �� � � � � � � & � z!HTMLTokenizer.consumeNumberEntityFc C s� d}| j �� g}|d tv sB|d tddfv sB|d urV||d krV| j �|d � �n |d dk�r d}|�| j �� � |d dv r�d}|�| j �� � |r�|d tv s�|s�|d tv r�| j �|d � | �|�}n4| j �t d d d �� | j �|� � � dd � |� }�nf|d tu�rDt �d � |���s0�qD|�| j �� � �q z$t �d � |d d� ��}t|�}W n t�y� d }Y n0 |d u�r>|d d k�r�| j �t d dd �� |d d k�r|�r|| tv �s�|| tv �s�|| dk�r| j �|� � � dd � |� }n.t| }| j �|� � � |d � ||d � �7 }n4| j �t d dd �� | j �|� � � dd � |� }|�r�| jd d d |7 < n*|tv �r�d}nd}| j �t | |d �� d S )N�&r �<�#F���)�x�XTr) zexpected-numeric-entityr* r5 rA z named-entity-without-semicolon�=zexpected-named-entityr, r �SpaceCharacters� Characters)r rB r r rI rC r r rP r- r r/ rE � entitiesTrie�has_keys_with_prefix�longest_prefix�len�KeyErrorr r r ) r# � allowedChar� fromAttribute�outputrM �hex� entityName� entityLength� tokenTyper' r' r( � consumeEntity� s~ � � � � � � � � � � zHTMLTokenizer.consumeEntityc C s | j |dd� dS )zIThis method replaces the need for "entityInAttributeValueState". T)r_ r` N)rf )r# r_ r' r' r( �processEntityInAttribute� s z&HTMLTokenizer.processEntityInAttributec C s� | j }|d tv r�|d �t�|d< |d td krp|d }t|�}t|�t|�krh|�|ddd� � ||d< |d td kr�|d r�| j� td d d �� |d r�| j� td d d �� | j� |� | j | _ dS ) z�This method is a generic handler for emitting the tags. It also sets the state to "data" because that's what's needed after a token has been emitted. r+ �name�StartTagr, NrT �EndTagr) zattributes-in-end-tagr* � selfClosingz self-closing-flag-on-end-tag) r r � translater r � attributeMapr] �updater- rC r r )r# �token�rawr, r' r' r( �emitCurrentToken� s( � � z HTMLTokenizer.emitCurrentTokenc C s� | j �� }|dkr | j| _n�|dkr.| j| _n�|dkrd| j�td dd�� | j�td dd�� n`|tu rpdS |t v r�| j�td || j � t d � d�� n&| j � d �}| j�td || d�� d S ) NrQ rR � r) �invalid-codepointr* rY FrX T�rQ rR rr ) r rB �entityDataStater � tagOpenStater- rC r r r � charsUntil�r# r, �charsr' r' r( r s. � � � �zHTMLTokenizer.dataStatec C s | � � | j| _dS �NT)rf r r r1 r' r' r( ru s z HTMLTokenizer.entityDataStatec C s� | j �� }|dkr | j| _n�|dkr.| j| _n�|tkr:dS |dkrp| j�td dd�� | j�td d d�� nT|t v r�| j�td || j � t d � d�� n&| j � d �}| j�td || d�� d S ) NrQ rR Frr r) rs r* rY r: rX Trt ) r rB �characterReferenceInRcdatar �rcdataLessThanSignStater r- rC r r rw rx r' r' r( � rcdataState" s. � � � �zHTMLTokenizer.rcdataStatec C s | � � | j| _dS rz )rf r} r r1 r' r' r( r{ ? s z(HTMLTokenizer.characterReferenceInRcdatac C s� | j �� }|dkr | j| _nh|dkrR| j�td dd�� | j�td dd�� n2|tkr^dS | j �d �}| j�td || d�� d S � NrR rr r) rs r* rY r: F)rR rr T) r rB �rawtextLessThanSignStater r- rC r r rw rx r' r' r( � rawtextStateD s" � � �zHTMLTokenizer.rawtextStatec C s� | j �� }|dkr | j| _nh|dkrR| j�td dd�� | j�td dd�� n2|tkr^dS | j �d �}| j�td || d�� d S r~ ) r rB �scriptDataLessThanSignStater r- rC r r rw rx r' r' r( �scriptDataStateV s" � � �z HTMLTokenizer.scriptDataStatec C sr | j �� }|tkrdS |dkrL| j�td dd�� | j�td dd�� n"| j�td || j �d� d�� dS ) NFrr r) rs r* rY r: T)r rB r r- rC r rw �r# r, r' r' r( �plaintextStateh s � � �z HTMLTokenizer.plaintextStatec C s | j �� }|dkr | j| _n�|dkr.| j| _n�|tv rVtd |g ddd�| _| j| _n�|dkr�| j � td dd �� | j � td d d �� | j | _nt|d kr�| j � td d d �� | j � |� | j | _n@| j � td dd �� | j � td dd �� | j � |� | j | _dS )N�!�/ri F)r+ rh r, rk �selfClosingAcknowledged�>r) z'expected-tag-name-but-got-right-bracketr* rY z<>�?z'expected-tag-name-but-got-question-markzexpected-tag-namerR T)r rB �markupDeclarationOpenStater �closeTagOpenStater r r � tagNameStater- rC r rI �bogusCommentStater� r' r' r( rv w s> � � � � zHTMLTokenizer.tagOpenStatec C s� | j �� }|tv r0td |g dd�| _| j| _n�|dkrX| j�td dd�� | j | _nn|t u r�| j�td dd�� | j�td d d�� | j | _n0| j�td d d |id �� | j � |� | j | _dS )Nrj F�r+ rh r, rk r� r) z*expected-closing-tag-but-got-right-bracketr* z expected-closing-tag-but-got-eofrY �</z!expected-closing-tag-but-got-charr, r7 T) r rB r r r r� r r- rC r r rI r� r� r' r' r( r� � s0 � � � � z HTMLTokenizer.closeTagOpenStatec C s� | j �� }|tv r | j| _n�|dkr.| �� n~|tu rV| j�t d dd�� | j | _nV|dkrh| j | _nD|dkr�| j�t d dd�� | j d d 7 < n| j d |7 < d S ) Nr� r) zeof-in-tag-namer* r� rr rs rh r: T) r rB r �beforeAttributeNameStater rq r r- rC r r �selfClosingStartTagStater r� r' r' r( r� � s& � �zHTMLTokenizer.tagNameStatec C sP | j �� }|dkr"d| _| j| _n*| j�td dd�� | j �|� | j | _dS �Nr� r5 rY rR r* T) r rB �temporaryBuffer�rcdataEndTagOpenStater r- rC r rI r} r� r' r' r( r| � s z%HTMLTokenizer.rcdataLessThanSignStatec C sX | j �� }|tv r*| j|7 _| j| _n*| j�td dd�� | j � |� | j | _dS �NrY r� r* T) r rB r r� �rcdataEndTagNameStater r- rC r rI r} r� r' r' r( r� � s z#HTMLTokenizer.rcdataEndTagOpenStatec C s | j o | j d �� | j�� k}| j�� }|tv rT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | � � | j | _nH|t v r�| j|7 _n0| j �td d| j d �� | j�|� | j| _d S � Nrh rj Fr� r� r� rY r� r* T)r �lowerr� r rB r r r� r r� rq r r r- rC rI r} �r# � appropriater, r' r' r( r� � s: � � � � z#HTMLTokenizer.rcdataEndTagNameStatec C sP | j �� }|dkr"d| _| j| _n*| j�td dd�� | j �|� | j | _dS r� ) r rB r� �rawtextEndTagOpenStater r- rC r rI r� r� r' r' r( r � s z&HTMLTokenizer.rawtextLessThanSignStatec C sX | j �� }|tv r*| j|7 _| j| _n*| j�td dd�� | j � |� | j | _dS r� ) r rB r r� �rawtextEndTagNameStater r- rC r rI r� r� r' r' r( r� s z$HTMLTokenizer.rawtextEndTagOpenStatec C s | j o | j d �� | j�� k}| j�� }|tv rT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | � � | j | _nH|t v r�| j|7 _n0| j �td d| j d �� | j�|� | j| _d S r� )r r� r� r rB r r r� r r� rq r r r- rC rI r� r� r' r' r( r� s: � � � � z$HTMLTokenizer.rawtextEndTagNameStatec C sx | j �� }|dkr"d| _| j| _nR|dkrJ| j�td dd�� | j| _n*| j�td dd�� | j � |� | j | _dS ) Nr� r5 r� rY z<!r* rR T) r rB r� �scriptDataEndTagOpenStater r- rC r �scriptDataEscapeStartStaterI r� r� r' r' r( r� , s z)HTMLTokenizer.scriptDataLessThanSignStatec C sX | j �� }|tv r*| j|7 _| j| _n*| j�td dd�� | j � |� | j | _dS r� ) r rB r r� �scriptDataEndTagNameStater r- rC r rI r� r� r' r' r( r� : s z'HTMLTokenizer.scriptDataEndTagOpenStatec C s | j o | j d �� | j�� k}| j�� }|tv rT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | � � | j | _nH|t v r�| j|7 _n0| j �td d| j d �� | j�|� | j| _d S r� )r r� r� r rB r r r� r r� rq r r r- rC rI r� r� r' r' r( r� E s: � � � � z'HTMLTokenizer.scriptDataEndTagNameStatec C sJ | j �� }|dkr2| j�td dd�� | j| _n| j �|� | j| _dS �N�-rY r* T) r rB r- rC r � scriptDataEscapeStartDashStater rI r� r� r' r' r( r� a s z(HTMLTokenizer.scriptDataEscapeStartStatec C sJ | j �� }|dkr2| j�td dd�� | j| _n| j �|� | j| _dS r� ) r rB r- rC r � scriptDataEscapedDashDashStater rI r� r� r' r' r( r� k s z,HTMLTokenizer.scriptDataEscapeStartDashStatec C s� | j �� }|dkr2| j�td dd�� | j| _n�|dkrD| j| _nn|dkrz| j�td dd�� | j�td dd�� n8|tkr�| j | _n&| j � d �}| j�td || d�� d S ) Nr� rY r* rR rr r) rs r: )rR r� rr T) r rB r- rC r �scriptDataEscapedDashStater �"scriptDataEscapedLessThanSignStater r rw rx r' r' r( �scriptDataEscapedStateu s( � � �z$HTMLTokenizer.scriptDataEscapedStatec C s� | j �� }|dkr2| j�td dd�� | j| _n�|dkrD| j| _nn|dkr�| j�td dd�� | j�td dd�� | j| _n0|t kr�| j | _n | j�td |d�� | j| _d S ) Nr� rY r* rR rr r) rs r: T) r rB r- rC r r� r r� r� r r r� r' r' r( r� � s& � � z(HTMLTokenizer.scriptDataEscapedDashStatec C s� | j �� }|dkr*| j�td dd�� n�|dkr<| j| _n�|dkrd| j�td dd�� | j| _nn|dkr�| j�td dd�� | j�td d d�� | j| _n0|t kr�| j | _n | j�td |d�� | j| _d S ) Nr� rY r* rR r� rr r) rs r: T) r rB r- rC r r� r r� r� r r r� r' r' r( r� � s* � � z,HTMLTokenizer.scriptDataEscapedDashDashStatec C s� | j �� }|dkr"d| _| j| _n\|tv rT| j�td d| d�� || _| j | _n*| j�td dd�� | j � |� | j | _dS r� ) r rB r� � scriptDataEscapedEndTagOpenStater r r- rC r � scriptDataDoubleEscapeStartStaterI r� r� r' r' r( r� � s z0HTMLTokenizer.scriptDataEscapedLessThanSignStatec C sP | j �� }|tv r"|| _| j| _n*| j�td dd�� | j � |� | j | _dS r� ) r rB r r� � scriptDataEscapedEndTagNameStater r- rC r rI r� r� r' r' r( r� � s z.HTMLTokenizer.scriptDataEscapedEndTagOpenStatec C s | j o | j d �� | j�� k}| j�� }|tv rT|rTtd | jg dd�| _ | j| _n�|dkr�|r�td | jg dd�| _ | j | _n||dkr�|r�td | jg dd�| _ | � � | j | _nH|t v r�| j|7 _n0| j �td d| j d �� | j�|� | j| _d S r� )r r� r� r rB r r r� r r� rq r r r- rC rI r� r� r' r' r( r� � s: � � � � z.HTMLTokenizer.scriptDataEscapedEndTagNameStatec C s� | j �� }|ttd�B v rR| j�td |d�� | j�� dkrH| j | _ q�| j | _ nB|t v r�| j�td |d�� | j|7 _n| j � |� | j | _ dS �N)r� r� rY r* �scriptT)r rB r rF r- rC r r� r� � scriptDataDoubleEscapedStater r� r rI r� r' r' r( r� � s z.HTMLTokenizer.scriptDataDoubleEscapeStartStatec C s� | j �� }|dkr2| j�td dd�� | j| _n�|dkrZ| j�td dd�� | j| _nt|dkr�| j�td dd�� | j�td dd�� n>|tkr�| j�td d d�� | j | _n| j�td |d�� d S � Nr� rY r* rR rr r) rs r: �eof-in-script-in-scriptT) r rB r- rC r � scriptDataDoubleEscapedDashStater �(scriptDataDoubleEscapedLessThanSignStater r r� r' r' r( r� � s* � � � z*HTMLTokenizer.scriptDataDoubleEscapedStatec C s� | j �� }|dkr2| j�td dd�� | j| _n�|dkrZ| j�td dd�� | j| _n�|dkr�| j�td dd�� | j�td dd�� | j| _nF|t kr�| j�td d d�� | j | _n | j�td |d�� | j| _d S r� ) r rB r- rC r �$scriptDataDoubleEscapedDashDashStater r� r� r r r� r' r' r( r� s. � � � z.HTMLTokenizer.scriptDataDoubleEscapedDashStatec C s | j �� }|dkr*| j�td dd�� n�|dkrR| j�td dd�� | j| _n�|dkrz| j�td dd�� | j| _n�|dkr�| j�td dd�� | j�td d d�� | j| _nF|t kr�| j�td d d�� | j | _n | j�td |d�� | j| _d S ) Nr� rY r* rR r� rr r) rs r: r� T) r rB r- rC r r� r r� r� r r r� r' r' r( r� % s2 � � � z2HTMLTokenizer.scriptDataDoubleEscapedDashDashStatec C sP | j �� }|dkr8| j�td dd�� d| _| j| _n| j �|� | j | _dS )Nr� rY r* r5 T) r rB r- rC r r� � scriptDataDoubleEscapeEndStater rI r� r� r' r' r( r� > s z6HTMLTokenizer.scriptDataDoubleEscapedLessThanSignStatec C s� | j �� }|ttd�B v rR| j�td |d�� | j�� dkrH| j | _ q�| j | _ nB|t v r�| j�td |d�� | j|7 _n| j � |� | j | _ dS r� )r rB r rF r- rC r r� r� r� r r� r rI r� r' r' r( r� I s z,HTMLTokenizer.scriptDataDoubleEscapeEndStatec C s0 | j �� }|tv r$| j �td� �n|tv rJ| jd �|dg� | j| _n�|dkr\| � � n�|dkrn| j | _n�|dv r�| j �t d dd �� | jd �|dg� | j| _n�|d kr�| j �t d d d �� | jd �d dg� | j| _nF|t u �r| j �t d d d �� | j| _n | jd �|dg� | j| _dS )NTr, r5 r� r� )�'�"rW rR r) �#invalid-character-in-attribute-namer* rr rs r: z#expected-attribute-name-but-got-eof)r rB r rw r r rC �attributeNameStater rq r� r- r r r r� r' r' r( r� Y s< � � � z&HTMLTokenizer.beforeAttributeNameStatec C s� | j �� }d}d}|dkr&| j| _�n.|tv r\| jd d d || j �td� 7 < d}n�|dkrjd}n�|tv r|| j| _n�|dkr�| j | _n�|d kr�| j � t d d d �� | jd d d d 7 < d}n�|dv �r | j � t d dd �� | jd d d |7 < d}nH|t u �r6| j � t d dd �� | j| _n | jd d d |7 < d}|�r�| jd d d �t�| jd d d< | jd d d� D ]>\}}| jd d d |k�r�| j � t d dd �� �qҐq�|�r�| �� dS )NTFrW r, rT r r� r� rr r) rs r* r: �r� r� rR r� zeof-in-attribute-namezduplicate-attribute)r rB �beforeAttributeValueStater r r rw r �afterAttributeNameStater� r- rC r r r rl r rq )r# r, �leavingThisState� emitTokenrh �_r' r' r( r� w s^ � � � � � � z HTMLTokenizer.attributeNameStatec C sD | j �� }|tv r$| j �td� �n |dkr8| j| _�n|dkrJ| �� n�|tv rp| jd � |dg� | j | _n�|dkr�| j | _n�|dkr�| j � t d d d �� | jd � d dg� | j | _n�|d v r�| j � t d d d �� | jd � |dg� | j | _nF|tu �r$| j � t d dd �� | j| _n | jd � |dg� | j | _dS )NTrW r� r, r5 r� rr r) rs r* r: r� z&invalid-character-after-attribute-namez expected-end-of-tag-but-got-eof)r rB r rw r� r rq r r rC r� r� r- r r r r� r' r' r( r� � s@ � � � z%HTMLTokenizer.afterAttributeNameStatec C sh | j �� }|tv r$| j �td� �n@|dkr8| j| _�n,|dkrX| j| _| j �|� �n |dkrj| j| _n�|dkr�| j � t d dd�� | � � n�|d kr�| j � t d d d�� | j d d d d7 < | j| _n�|dv �r| j � t d dd�� | j d d d |7 < | j| _nL|tu �rB| j � t d dd�� | j| _n"| j d d d |7 < | j| _dS )NTr� rQ r� r� r) z.expected-attribute-value-but-got-right-bracketr* rr rs r, rT r r: )rW rR �`z"equals-in-unquoted-attribute-valuez$expected-attribute-value-but-got-eof)r rB r rw � attributeValueDoubleQuotedStater �attributeValueUnQuotedStaterI � attributeValueSingleQuotedStater- rC r rq r r r r� r' r' r( r� � sF � � � � z'HTMLTokenizer.beforeAttributeValueStatec C s� | j �� }|dkr | j| _n�|dkr0| �d� n�|dkrj| j�td dd�� | jd d d d 7 < nN|t u r�| j�td d d�� | j | _n&| jd d d || j � d � 7 < d S )Nr� rQ rr r) rs r* r, rT r r: z#eof-in-attribute-value-double-quote)r� rQ rr T� r rB �afterAttributeValueStater rg r- rC r r r r rw r� r' r' r( r� � s& � � �z-HTMLTokenizer.attributeValueDoubleQuotedStatec C s� | j �� }|dkr | j| _n�|dkr0| �d� n�|dkrj| j�td dd�� | jd d d d 7 < nN|t u r�| j�td d d�� | j | _n&| jd d d || j � d � 7 < d S )Nr� rQ rr r) rs r* r, rT r r: z#eof-in-attribute-value-single-quote)r� rQ rr Tr� r� r' r' r( r� s& � � �z-HTMLTokenizer.attributeValueSingleQuotedStatec C s | j �� }|tv r | j| _n�|dkr0| �d� n�|dkrB| �� n�|dv r|| j�t d dd�� | j d d d |7 < n�|d kr�| j�t d d d�� | j d d d d 7 < nV|t u r�| j�t d d d�� | j | _n.| j d d d || j � td�tB � 7 < dS )NrQ r� )r� r� rW rR r� r) z0unexpected-character-in-unquoted-attribute-valuer* r, rT r rr rs r: z eof-in-attribute-value-no-quotes)rQ r� r� r� rW rR r� rr T)r rB r r� r rg rq r- rC r r r r rw rF r� r' r' r( r� s4 � � � � z)HTMLTokenizer.attributeValueUnQuotedStatec C s� | j �� }|tv r | j| _n�|dkr.| �� np|dkr@| j| _n^|tu rt| j� t d dd�� | j � |� | j | _n*| j� t d dd�� | j � |� | j| _dS )Nr� r� r) z$unexpected-EOF-after-attribute-valuer* z*unexpected-character-after-attribute-valueT) r rB r r� r rq r� r r- rC r rI r r� r' r' r( r� . s& � � z&HTMLTokenizer.afterAttributeValueStatec C s� | j �� }|dkr&d| jd< | �� n^|tu rZ| j�td dd�� | j �|� | j | _ n*| j�td dd�� | j �|� | j | _ dS )Nr� Trk r) z#unexpected-EOF-after-solidus-in-tagr* z)unexpected-character-after-solidus-in-tag) r rB r rq r r- rC r rI r r r� r� r' r' r( r� B s � � z&HTMLTokenizer.selfClosingStartTagStatec C sD | j �d�}|�dd�}| j�td |d�� | j �� | j| _dS )Nr� rr r: �Commentr* T) r rw �replacer- rC r rB r r r� r' r' r( r� T s � z HTMLTokenizer.bogusCommentStatec C s� | j �� g}|d dkrR|�| j �� � |d dkrPtd dd�| _| j| _dS n�|d dv r�d}dD ](}|�| j �� � |d |vrfd } q�qf|r�td dd d dd �| _| j| _dS n�|d d k�rD| jd u�rD| jj j �rD| jj j d j | jj j k�rDd}d D ].}|�| j �� � |d |k�rd } �q2�q|�rD| j | _dS | j�td dd�� |�rt| j �|�� � �qZ| j| _dS )NrT r� r� r5 r* T)�d�D))�o�O�rN �C��t�T��y�Y��p�P��e�EF�Doctype)r+ rh �publicId�systemId�correct�[)r� r� �Ar� r� r� r) zexpected-dashes-or-doctype)r rB rC r r �commentStartStater � doctypeStater �tree� openElements� namespace�defaultNamespace�cdataSectionStater- rI r/ r� )r# rM �matched�expectedr' r' r( r� c sZ ���� �z(HTMLTokenizer.markupDeclarationOpenStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� | j�| j� | j| _nP|t u r�| j�td d d�� | j�| j� | j| _n| jd |7 < | j | _d S ) Nr� rr r) rs r* r, r: r� �incorrect-comment�eof-in-commentT) r rB �commentStartDashStater r- rC r r r r � commentStater� r' r' r( r� � s. � � � z HTMLTokenizer.commentStartStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� | j�| j� | j| _nT|t u r�| j�td d d�� | j�| j� | j| _n | jd d| 7 < | j | _d S ) Nr� rr r) rs r* r, � -�r� r� r� T) r rB �commentEndStater r- rC r r r r r� r� r' r' r( r� � s. � � � z#HTMLTokenizer.commentStartDashStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < nT|tu r�| j�td dd�� | j�| j� | j | _n | jd || j � d � 7 < d S ) Nr� rr r) rs r* r, r: r� )r� rr T) r rB �commentEndDashStater r- rC r r r r rw r� r' r' r( r� � s$ � � �zHTMLTokenizer.commentStatec C s� | j �� }|dkr | j| _n�|dkrV| j�td dd�� | jd d7 < | j| _nT|t u r�| j�td dd�� | j�| j� | j | _n | jd d| 7 < | j| _d S ) Nr� rr r) rs r* r, r� zeof-in-comment-end-dashT) r rB r� r r- rC r r r� r r r� r' r' r( r� � s$ � � z!HTMLTokenizer.commentEndDashStatec C s, | j �� }|dkr*| j�| j� | j| _n�|dkrd| j�td dd�� | jd d7 < | j| _n�|dkr�| j�td d d�� | j | _n�|d kr�| j�td d d�� | jd |7 < nj|t u r�| j�td d d�� | j�| j� | j| _n4| j�td d d�� | jd d| 7 < | j| _dS )Nr� rr r) rs r* r, u --�r� z,unexpected-bang-after-double-dash-in-commentr� z,unexpected-dash-after-double-dash-in-commentzeof-in-comment-double-dashzunexpected-char-in-commentz--T) r rB r- rC r r r r r� �commentEndBangStater r� r' r' r( r� � s@ � � � � �z HTMLTokenizer.commentEndStatec C s� | j �� }|dkr*| j�| j� | j| _n�|dkrN| jd d7 < | j| _n�|dkr�| j�td dd�� | jd d 7 < | j | _nT|t u r�| j�td d d�� | j�| j� | j| _n | jd d| 7 < | j | _d S ) Nr� r� r, z--!rr r) rs r* u --!�z eof-in-comment-end-bang-stateT) r rB r- rC r r r r� r r� r r� r' r' r( r� s, � � z!HTMLTokenizer.commentEndBangStatec C s� | j �� }|tv r | j| _nj|tu r\| j�td dd�� d| j d< | j�| j � | j | _n*| j�td dd�� | j � |� | j| _dS )Nr) �!expected-doctype-name-but-got-eofr* Fr� zneed-space-after-doctypeT) r rB r �beforeDoctypeNameStater r r- rC r r r rI r� r' r' r( r� s � � zHTMLTokenizer.doctypeStatec C s� | j �� }|tv rn�|dkrT| j�td dd�� d| jd< | j�| j� | j| _n�|dkr�| j�td dd�� d | jd < | j | _nR|t u r�| j�td d d�� d| jd< | j�| j� | j| _n|| jd < | j | _d S ) Nr� r) z+expected-doctype-name-but-got-right-bracketr* Fr� rr rs r: rh r� T) r rB r r- rC r r r r �doctypeNameStater r� r' r' r( r� * s4 � � � z$HTMLTokenizer.beforeDoctypeNameStatec C s | j �� }|tv r2| jd �t�| jd< | j| _n�|dkrh| jd �t�| jd< | j� | j� | j | _n�|dkr�| j� t d dd�� | jd d7 < | j | _nh|t u r�| j� t d dd�� d | jd < | jd �t�| jd< | j� | j� | j | _n| jd |7 < d S ) Nrh r� rr r) rs r* r: zeof-in-doctype-nameFr� T)r rB r r rl r �afterDoctypeNameStater r- rC r r r� r r� r' r' r( r� D s0 � � z HTMLTokenizer.doctypeNameStatec C sH | j �� }|tv r�n.|dkr8| j�| j� | j| _�n |tu r�d| jd< | j � |� | j�t d dd�� | j�| j� | j| _n�|dv r�d}d D ] }| j �� }||vr�d} q�q�|r�| j | _dS nD|d v �r d}d D ] }| j �� }||vr�d} q�q�|�r | j | _dS | j � |� | j�t d d d |id�� d| jd< | j | _dS )Nr� Fr� r) �eof-in-doctyper* r� T))�u�U)�b�B)�l�L)�i�Ir� ��s�S)r� r� r� r� )�m�Mz*expected-space-or-right-bracket-in-doctyper, r7 )r rB r r- rC r r r r rI r � afterDoctypePublicKeywordState� afterDoctypeSystemKeywordState�bogusDoctypeState)r# r, r� r� r' r' r( r� ] sT � � z#HTMLTokenizer.afterDoctypeNameStatec C s� | j �� }|tv r | j| _n�|dv rP| j�td dd�� | j �|� | j| _nT|t u r�| j�td dd�� d| j d< | j�| j � | j | _n| j �|� | j| _dS � N)r� r� r) �unexpected-char-in-doctyper* r� Fr� T) r rB r �"beforeDoctypePublicIdentifierStater r- rC r rI r r r r� r' r' r( r � s& � � z,HTMLTokenizer.afterDoctypePublicKeywordStatec C s� | j �� }|tv rn�|dkr0d| jd< | j| _n�|dkrLd| jd< | j| _n�|dkr�| j�t d dd�� d | jd < | j�| j� | j | _nh|t u r�| j�t d d d�� d | jd < | j�| j� | j | _n(| j�t d d d�� d | jd < | j | _d S )Nr� r5 r� r� r� r) �unexpected-end-of-doctyper* Fr� r� r T) r rB r r �(doctypePublicIdentifierDoubleQuotedStater �(doctypePublicIdentifierSingleQuotedStater- rC r r r r r� r' r' r( r � s: � � � z0HTMLTokenizer.beforeDoctypePublicIdentifierStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d | jd < | j�| j� | j| _nR|t u r�| j�td d d�� d | jd < | j�| j� | j| _n| jd |7 < d S )Nr� rr r) rs r* r� r: r� r Fr� r� T� r rB �!afterDoctypePublicIdentifierStater r- rC r r r r r� r' r' r( r � s0 � � � z6HTMLTokenizer.doctypePublicIdentifierDoubleQuotedStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d | jd < | j�| j� | j| _nR|t u r�| j�td d d�� d | jd < | j�| j� | j| _n| jd |7 < d S )Nr� rr r) rs r* r� r: r� r Fr� r� Tr r� r' r' r( r � s0 � � � z6HTMLTokenizer.doctypePublicIdentifierSingleQuotedStatec C s | j �� }|tv r | j| _n�|dkr<| j�| j� | j| _n�|dkrn| j�t d dd�� d| jd< | j | _n�|dkr�| j�t d dd�� d| jd< | j | _nh|t u r�| j�t d d d�� d | jd < | j�| j� | j| _n(| j�t d dd�� d | jd < | j | _d S ) Nr� r� r) r r* r5 r� r� r� Fr� T)r rB r �-betweenDoctypePublicAndSystemIdentifiersStater r- rC r r r �(doctypeSystemIdentifierDoubleQuotedState�(doctypeSystemIdentifierSingleQuotedStater r r� r' r' r( r � s> � � � � z/HTMLTokenizer.afterDoctypePublicIdentifierStatec C s� | j �� }|tv rn�|dkr4| j�| j� | j| _n�|dkrPd| jd< | j| _n�|dkrld| jd< | j | _nh|t kr�| j�t d dd�� d | jd < | j�| j� | j| _n(| j�t d d d�� d | jd < | j | _d S ) Nr� r� r5 r� r� r) r� r* Fr� r T) r rB r r- rC r r r r r r r r r� r' r' r( r s2 � � z;HTMLTokenizer.betweenDoctypePublicAndSystemIdentifiersStatec C s� | j �� }|tv r | j| _n�|dv rP| j�td dd�� | j �|� | j| _nT|t u r�| j�td dd�� d| j d< | j�| j � | j | _n| j �|� | j| _dS r ) r rB r �"beforeDoctypeSystemIdentifierStater r- rC r rI r r r r� r' r' r( r ) s& � � z,HTMLTokenizer.afterDoctypeSystemKeywordStatec C s� | j �� }|tv rn�|dkr0d| jd< | j| _n�|dkrLd| jd< | j| _n�|dkr�| j�t d dd�� d | jd < | j�| j� | j | _nh|t u r�| j�t d d d�� d | jd < | j�| j� | j | _n(| j�t d dd�� d | jd < | j | _d S ) Nr� r5 r� r� r� r) r r* Fr� r� T) r rB r r r r r r- rC r r r r r� r' r' r( r = s: � � � z0HTMLTokenizer.beforeDoctypeSystemIdentifierStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d | jd < | j�| j� | j| _nR|t u r�| j�td d d�� d | jd < | j�| j� | j| _n| jd |7 < d S )Nr� rr r) rs r* r� r: r� r Fr� r� T� r rB �!afterDoctypeSystemIdentifierStater r- rC r r r r r� r' r' r( r Z s0 � � � z6HTMLTokenizer.doctypeSystemIdentifierDoubleQuotedStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d | jd < | j�| j� | j| _nR|t u r�| j�td d d�� d | jd < | j�| j� | j| _n| jd |7 < d S )Nr� rr r) rs r* r� r: r� r Fr� r� Tr r� r' r' r( r r s0 � � � z6HTMLTokenizer.doctypeSystemIdentifierSingleQuotedStatec C s� | j �� }|tv rn~|dkr4| j�| j� | j| _n^|tu rt| j�t d dd�� d| jd< | j�| j� | j| _n | j�t d dd�� | j | _dS ) Nr� r) r� r* Fr� r T) r rB r r- rC r r r r r r r� r' r' r( r � s$ � �z/HTMLTokenizer.afterDoctypeSystemIdentifierStatec C sZ | j �� }|dkr*| j�| j� | j| _n,|tu rV| j �|� | j�| j� | j| _n dS )Nr� T) r rB r- rC r r r r rI r� r' r' r( r � s z HTMLTokenizer.bogusDoctypeStatec C s� g }|� | j�d�� |� | j�d�� | j�� }|tkr>q�q|dksJJ �|d dd � dkrv|d d d� |d< q�q|� |� qd�|�}|�d�}|dkr�t|�D ]}| j� t d d d �� q�|� dd �}|r�| j� t d |d �� | j | _ dS )N�]r� rT ���z]]r5 rr r r) rs r* r: rY T) rC r rw rB r rE �count�ranger- r r� r r )r# r, rB � nullCountr� r' r' r( r� � s2 � �z HTMLTokenizer.cdataSectionState)N)NF)N�__name__� __module__� __qualname__�__doc__r" r2 rP rf rg rq r ru r} r{ r� r� r� rv r� r� r| r� r� r r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r r r r r r r r r r r r r� � __classcell__r' r' r% r( r s� H P # 6 "- 3 r N) � __future__r r r Zpip._vendor.sixr rG � collectionsr r �sysr � constantsr r r r r r r r r r � _inputstreamr �_trier rZ �dictrm �objectr r' r' r' r( �<module> s