관리-도구
편집 파일: charsetprober.cpython-38.pyc
U ʗRe� � @ s8 d dl Z d dlZddlmZ e�d�ZG dd� d�ZdS )� N� )� ProbingStates% [a-zA-Z]*[�-�]+[a-zA-Z]*[^a-zA-Z�-�]?c @ sn e Zd ZdZddd�Zdd� Zedd� �Zd d � Zed d � �Z d d� Z e dd� �Z e dd� �Z e dd� �ZdS )� CharSetProbergffffff�?Nc C s d | _ || _t�t�| _d S �N)�_state� lang_filter�logging� getLogger�__name__�logger)�selfr � r ��/builddir/build/BUILDROOT/alt-python38-pip-22.2.1-2.el8.x86_64/opt/alt/python38/lib/python3.8/site-packages/pip/_vendor/chardet/charsetprober.py�__init__+ s zCharSetProber.__init__c C s t j| _d S r )r � DETECTINGr �r r r r �reset0 s zCharSetProber.resetc C s d S r r r r r r � charset_name3 s zCharSetProber.charset_namec C s t �d S r )�NotImplementedError)r �byte_strr r r �feed7 s zCharSetProber.feedc C s | j S r )r r r r r �state: s zCharSetProber.statec C s dS )Ng r r r r r �get_confidence> s z CharSetProber.get_confidencec C s t �dd| �} | S )Ns ([ -])+� )�re�sub)�bufr r r �filter_high_byte_onlyA s z#CharSetProber.filter_high_byte_onlyc C sZ t � }t�| �}|D ]@}|�|dd� � |dd� }|�� sJ|dk rJd}|�|� q|S )u7 We define three types of bytes: alphabet: english alphabets [a-zA-Z] international: international characters [-ÿ] marker: everything else [^a-zA-Z-ÿ] The input buffer can be thought to contain a series of words delimited by markers. This function works to filter all words that contain at least one international character. All contiguous sequences of markers are replaced by a single space ascii character. This filter applies to all scripts which do not use English characters. N���� �r )� bytearray�INTERNATIONAL_WORDS_PATTERN�findall�extend�isalpha)r �filtered�words�word� last_charr r r �filter_international_wordsF s z(CharSetProber.filter_international_wordsc C s� t � }d}d}t| ��d�} t| �D ]R\}}|dkrB|d }d}q$|dkr$||krr|sr|�| ||� � |�d� d}q$|s�|�| |d � � |S ) a[ Returns a copy of ``buf`` that retains only the sequences of English alphabet and high byte characters that are not between <> characters. This filter can be applied to all scripts which contain both English characters and extended ASCII characters, but is currently only used by ``Latin1Prober``. Fr �c� >r � <r TN)r � memoryview�cast� enumerater# )r r% �in_tag�prev�curr�buf_charr r r �remove_xml_tagsh s z CharSetProber.remove_xml_tags)N)r � __module__� __qualname__�SHORTCUT_THRESHOLDr r �propertyr r r r � staticmethodr r) r4 r r r r r ' s !r )r r �enumsr �compiler! r r r r r �<module> s �