|
Apache/2.2.15 (CentOS) Linux obd60-6c49958d75-2q7cw 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 uid=48(apache) gid=48(apache) groups=48(apache) server ip : 104.21.65.202 | your ip : 10.244.126.0 safemode OFF > / usr / lib64 / python2.6 / |
| Filename | /usr/lib64/python2.6/robotparser.pyc |
| Size | 7.67 kb |
| Permission | rw-r--r-- |
| Owner | apache |
| Create time | 23-Dec-2025 17:41 |
| Last modified | 20-Jun-2019 19:45 |
| Last accessed | 22-Apr-2026 05:24 |
| Actions | edit | rename | delete | download (gzip) |
| View | text | code | image |
Ñò
§ÚêLc @ s} d Z d d k Z d d k Z d g Z d d d � � YZ d d d � � YZ d d
d � � YZ d e i f d
� � YZ d S( s< robotparser.py
Copyright (C) 2000 Bastian Kleineidam
You can choose between two licenses when using this package:
1) GNU GPLv2
2) PSF license for Python 2.2
The robots.txt Exclusion Protocol is implemented as specified in
http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
iÿÿÿÿNt RobotFileParserc B sb e Z d Z d d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z
d
� Z RS( ss This class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
t c C s> g | _ d | _ t | _ t | _ | i | � d | _ d S( Ni ( t entriest Nonet
default_entryt Falset disallow_allt allow_allt set_urlt last_checked( t selft url( ( s# /usr/lib64/python2.6/robotparser.pyt __init__ s
c C s | i S( s· Returns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to
check for new robots.txt files periodically.
( R ( R
( ( s# /usr/lib64/python2.6/robotparser.pyt mtime s c C s d d k } | i � | _ d S( sY Sets the time the robots.txt file was last fetched to the
current time.
iÿÿÿÿN( t timeR ( R
R ( ( s# /usr/lib64/python2.6/robotparser.pyt modified) s c C s/ | | _ t i | � d d !\ | _ | _ d S( s, Sets the URL referring to a robots.txt file.i i N( R t urlparset hostt path( R
R ( ( s# /usr/lib64/python2.6/robotparser.pyR 1 s c C s¾ t � } | i | i � } g } | D] } | | i � q& ~ } | i � | i | _ | i d j o
t | _ nF | i d j o
t | _ n) | i d j o | o | i | � n d S( s4 Reads the robots.txt URL and feeds it to the parser.i� i� i� iÈ N( i� i� (
t URLopenert openR t stript closet errcodet TrueR R t parse( R
t openert ft _[1]t linet lines( ( s# /usr/lib64/python2.6/robotparser.pyt read6 s '
c C sE d | i j o! | i d j o
| | _ qA n | i i | � d S( Nt *( t
useragentsR R R t append( R
t entry( ( s# /usr/lib64/python2.6/robotparser.pyt
_add_entryD s c C s6 d } d } t � } xü| D]ô} | d 7} | pQ | d j o t � } d } q� | d j o | i | � t � } d } q� n | i d � } | d j o | | } n | i � } | p q n | i d d � } t | � d j o#| d i � i � | d <t i | d i � � | d <| d d j oE | d j o | i | � t � } n | i i
| d � d } q| d d j o8 | d j o' | i i
t | d t
� � d } q q| d d j o8 | d j o' | i i
t | d t � � d } q qq q W| d j o | i | � n d S(
s� parse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by
one or more blank lines.i i i t #t :s
user-agentt disallowt allowN( t EntryR$ t findR t splitt lent lowert urllibt unquoteR! R" t rulelinest RuleLineR R ( R
R t statet
linenumberR# R t i( ( s# /usr/lib64/python2.6/robotparser.pyR M sP
c C s� | i o t S| i o t St i t i t i | � � d � p d } x/ | i D]$ } | i | � o | i
| � SqT W| i o | i i
| � St S( s= using the parsed robots.txt decide if useragent can fetch urli t /( R R R R R. t quoteR R/ R t
applies_tot allowanceR ( R
t useragentR R# ( ( s# /usr/lib64/python2.6/robotparser.pyt can_fetch� s
,
c C s5 d i g } | i D] } | t | � d q ~ � S( NR s
( t joinR t str( R
R R# ( ( s# /usr/lib64/python2.6/robotparser.pyt __str__� s ( t __name__t
__module__t __doc__R R
R R R R$ R R: R= ( ( ( s# /usr/lib64/python2.6/robotparser.pyR s 3 R1 c B s) e Z d Z d � Z d � Z d � Z RS( so A rule line is a single "Allow:" (allowance==True) or "Disallow:"
(allowance==False) followed by a path.c C s> | d j o | o
t } n t i | � | _ | | _ d S( NR ( R R. R6 R R8 ( R
R R8 ( ( s# /usr/lib64/python2.6/robotparser.pyR � s
c C s | i d j p | i | i � S( NR ( R t
startswith( R
t filename( ( s# /usr/lib64/python2.6/robotparser.pyR7 ¡ s c C s | i o d p d d | i S( Nt Allowt Disallows : ( R8 R ( R
( ( s# /usr/lib64/python2.6/robotparser.pyR= ¤ s ( R> R? R@ R R7 R= ( ( ( s# /usr/lib64/python2.6/robotparser.pyR1 � s R) c B s2 e Z d Z d � Z d � Z d � Z d � Z RS( s? An entry has one or more user-agents and zero or more rulelinesc C s g | _ g | _ d S( N( R! R0 ( R
( ( s# /usr/lib64/python2.6/robotparser.pyR ª s c C sj g } x' | i D] } | i d | d g � q Wx* | i D] } | i t | � d g � q: Wd i | � S( Ns User-agent: s
R ( R! t extendR0 R< R; ( R
t rett agentR ( ( s# /usr/lib64/python2.6/robotparser.pyR= ® s
c C sa | i d � d i � } xA | i D]6 } | d j o t S| i � } | | j o t Sq# Wt S( s2 check if this entry applies to the specified agentR5 i R ( R+ R- R! R R ( R
R9 RG ( ( s# /usr/lib64/python2.6/robotparser.pyR7 ¶ s
c C s0 x) | i D] } | i | � o | i Sq
Wt S( sZ Preconditions:
- our agent applies to this entry
- filename is URL decoded( R0 R7 R8 R ( R
RB R ( ( s# /usr/lib64/python2.6/robotparser.pyR8 Ã s
( R> R? R@ R R= R7 R8 ( ( ( s# /usr/lib64/python2.6/robotparser.pyR) ¨ s
R c B s# e Z d � Z d � Z d � Z RS( c G s t i i | | � d | _ d S( NiÈ ( R. t FancyURLopenerR R ( R
t args( ( s# /usr/lib64/python2.6/robotparser.pyR Í s c C s d S( N( NN( R ( R
R t realm( ( s# /usr/lib64/python2.6/robotparser.pyt prompt_user_passwdÑ s c C s( | | _ t i i | | | | | | � S( N( R R. RH t http_error_default( R
R t fpR t errmsgt headers( ( s# /usr/lib64/python2.6/robotparser.pyRL Ö s ( R> R? R RK RL ( ( ( s# /usr/lib64/python2.6/robotparser.pyR Ì s ( ( ( ( R@ R R. t __all__R R1 R) RH R ( ( ( s# /usr/lib64/python2.6/robotparser.pyt <module> s �$
§ÚêLc @ s} d Z d d k Z d d k Z d g Z d d d � � YZ d d d � � YZ d d
d � � YZ d e i f d
� � YZ d S( s< robotparser.py
Copyright (C) 2000 Bastian Kleineidam
You can choose between two licenses when using this package:
1) GNU GPLv2
2) PSF license for Python 2.2
The robots.txt Exclusion Protocol is implemented as specified in
http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
iÿÿÿÿNt RobotFileParserc B sb e Z d Z d d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z
d
� Z RS( ss This class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
t c C s> g | _ d | _ t | _ t | _ | i | � d | _ d S( Ni ( t entriest Nonet
default_entryt Falset disallow_allt allow_allt set_urlt last_checked( t selft url( ( s# /usr/lib64/python2.6/robotparser.pyt __init__ s
c C s | i S( s· Returns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to
check for new robots.txt files periodically.
( R ( R
( ( s# /usr/lib64/python2.6/robotparser.pyt mtime s c C s d d k } | i � | _ d S( sY Sets the time the robots.txt file was last fetched to the
current time.
iÿÿÿÿN( t timeR ( R
R ( ( s# /usr/lib64/python2.6/robotparser.pyt modified) s c C s/ | | _ t i | � d d !\ | _ | _ d S( s, Sets the URL referring to a robots.txt file.i i N( R t urlparset hostt path( R
R ( ( s# /usr/lib64/python2.6/robotparser.pyR 1 s c C s¾ t � } | i | i � } g } | D] } | | i � q& ~ } | i � | i | _ | i d j o
t | _ nF | i d j o
t | _ n) | i d j o | o | i | � n d S( s4 Reads the robots.txt URL and feeds it to the parser.i� i� i� iÈ N( i� i� (
t URLopenert openR t stript closet errcodet TrueR R t parse( R
t openert ft _[1]t linet lines( ( s# /usr/lib64/python2.6/robotparser.pyt read6 s '
c C sE d | i j o! | i d j o
| | _ qA n | i i | � d S( Nt *( t
useragentsR R R t append( R
t entry( ( s# /usr/lib64/python2.6/robotparser.pyt
_add_entryD s c C s6 d } d } t � } xü| D]ô} | d 7} | pQ | d j o t � } d } q� | d j o | i | � t � } d } q� n | i d � } | d j o | | } n | i � } | p q n | i d d � } t | � d j o#| d i � i � | d <t i | d i � � | d <| d d j oE | d j o | i | � t � } n | i i
| d � d } q| d d j o8 | d j o' | i i
t | d t
� � d } q q| d d j o8 | d j o' | i i
t | d t � � d } q qq q W| d j o | i | � n d S(
s� parse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by
one or more blank lines.i i i t #t :s
user-agentt disallowt allowN( t EntryR$ t findR t splitt lent lowert urllibt unquoteR! R" t rulelinest RuleLineR R ( R
R t statet
linenumberR# R t i( ( s# /usr/lib64/python2.6/robotparser.pyR M sP
c C s� | i o t S| i o t St i t i t i | � � d � p d } x/ | i D]$ } | i | � o | i
| � SqT W| i o | i i
| � St S( s= using the parsed robots.txt decide if useragent can fetch urli t /( R R R R R. t quoteR R/ R t
applies_tot allowanceR ( R
t useragentR R# ( ( s# /usr/lib64/python2.6/robotparser.pyt can_fetch� s
,
c C s5 d i g } | i D] } | t | � d q ~ � S( NR s
( t joinR t str( R
R R# ( ( s# /usr/lib64/python2.6/robotparser.pyt __str__� s ( t __name__t
__module__t __doc__R R
R R R R$ R R: R= ( ( ( s# /usr/lib64/python2.6/robotparser.pyR s 3 R1 c B s) e Z d Z d � Z d � Z d � Z RS( so A rule line is a single "Allow:" (allowance==True) or "Disallow:"
(allowance==False) followed by a path.c C s> | d j o | o
t } n t i | � | _ | | _ d S( NR ( R R. R6 R R8 ( R
R R8 ( ( s# /usr/lib64/python2.6/robotparser.pyR � s
c C s | i d j p | i | i � S( NR ( R t
startswith( R
t filename( ( s# /usr/lib64/python2.6/robotparser.pyR7 ¡ s c C s | i o d p d d | i S( Nt Allowt Disallows : ( R8 R ( R
( ( s# /usr/lib64/python2.6/robotparser.pyR= ¤ s ( R> R? R@ R R7 R= ( ( ( s# /usr/lib64/python2.6/robotparser.pyR1 � s R) c B s2 e Z d Z d � Z d � Z d � Z d � Z RS( s? An entry has one or more user-agents and zero or more rulelinesc C s g | _ g | _ d S( N( R! R0 ( R
( ( s# /usr/lib64/python2.6/robotparser.pyR ª s c C sj g } x' | i D] } | i d | d g � q Wx* | i D] } | i t | � d g � q: Wd i | � S( Ns User-agent: s
R ( R! t extendR0 R< R; ( R
t rett agentR ( ( s# /usr/lib64/python2.6/robotparser.pyR= ® s
c C sa | i d � d i � } xA | i D]6 } | d j o t S| i � } | | j o t Sq# Wt S( s2 check if this entry applies to the specified agentR5 i R ( R+ R- R! R R ( R
R9 RG ( ( s# /usr/lib64/python2.6/robotparser.pyR7 ¶ s
c C s0 x) | i D] } | i | � o | i Sq
Wt S( sZ Preconditions:
- our agent applies to this entry
- filename is URL decoded( R0 R7 R8 R ( R
RB R ( ( s# /usr/lib64/python2.6/robotparser.pyR8 Ã s
( R> R? R@ R R= R7 R8 ( ( ( s# /usr/lib64/python2.6/robotparser.pyR) ¨ s
R c B s# e Z d � Z d � Z d � Z RS( c G s t i i | | � d | _ d S( NiÈ ( R. t FancyURLopenerR R ( R
t args( ( s# /usr/lib64/python2.6/robotparser.pyR Í s c C s d S( N( NN( R ( R
R t realm( ( s# /usr/lib64/python2.6/robotparser.pyt prompt_user_passwdÑ s c C s( | | _ t i i | | | | | | � S( N( R R. RH t http_error_default( R
R t fpR t errmsgt headers( ( s# /usr/lib64/python2.6/robotparser.pyRL Ö s ( R> R? R RK RL ( ( ( s# /usr/lib64/python2.6/robotparser.pyR Ì s ( ( ( ( R@ R R. t __all__R R1 R) RH R ( ( ( s# /usr/lib64/python2.6/robotparser.pyt <module> s �$