Python Request get post proxy common examples

The following is the text of this article. The following cases are for reference.

Pip install requests

pip install requests -i https://pypi.tuna.tsinghua.edu.cn/simple

2. Common parameters carried in Requests requests

1. Parameter description

2、headers

def get_headers():
        user_gent = [
            'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/'
            '537.36',
            "MQQBrowser/25 (Linux; U; 2.3.3; zh-cn; HTC Desire S Build/GRI40;480*800)",
            "Mozilla/5.0 (Windows NT 5.2) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30",
            "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1",
            "Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; SAMSUNG; OMNIA7)",
            "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; XBLWP7; ZuneWP7)",
            "Mozilla/5.0 (Windows NT 5.2) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30",
            "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0",
            "Mozilla/4.0 (compatible; MSIE 60; Windows NT 5.1; SV1; .NET CLR 2.0.50727)",
            "Opera/9.80 (Windows NT 5.1; U; zh-cn) Presto/2.9.168 Version/11.50",
            "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)",
            "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; TheWorld)"
        ]
        headers = {
            'User-Agent': user_gent[random.randint(0, len(user_gent) - 1)],
        }
        return headers

3. Common parameters of requests: url, headers, proxies, verify, timeout

url = "https://www.baidu.com/" #Requested link
headers = {"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"}  # headers = get_headers()
proxies = { 'http': f'http {port}', 'https': f'http {port}'} # IP proxy, please note that both http and https must be written.
proxies = { 'http': 'http 10809','https': 'http 10809'} # Add system proxy VPN
verify = Flase # Set to False when using an IP proxy or VPN
timeout = 5 # Each time the connection is requested and does not respond for more than 5 seconds, an exception will be thrown.

Three, Requests Get Post

1、Get

params={"key":"value"} # Parameters passed in the url, the effect is like: http://www.baidu.com?key=value
response = requests.get(url=url, headers=headers, params=params, proxies=proxies, verify = Flase, timeout=5)

2、Post

parameter = {"key1": "value1", "key2": "value2"}
# requests.post() method json and data description
# In the chrom browser, the data format is Form Data, then data is used to send data, or it can be distinguished by {} (curly brackets)
# In the chrom browser, the data format is Request Payload, then json is used to send the data, or it can be distinguished by {} (curly brackets)
response = requests.post(url=url, headers=headers, data=parameter, proxies=proxies, verify = Flase, timeout=5)
files = {"file":open("xxx.txt","rb")} # Submit files
response = requests.post(url=url, headers=headers, files=files , proxies=proxies, verify = Flase, timeout=5)

Request Form Data Request form data,
The Request Payload uses json
Regarding file upload, please refer to this blog of mine:requests post file uploadhttps://blog.csdn.net/EXIxiaozhou/article/details/126975807

4. Common codes for Requests

1. Commonly used request codes

import requests
from requests import exceptions
request_count = 0
while request_count < 3:
    try:
        response = requests.get(url=url, headers=get_headers(), proxies=proxy,  verify = Flase, timeout=5)
        print(f"{xxxx} - Request successful {url} IP:{self.proxy}\n", end='')
        break
    except request_exceptions.ConnectTimeout:
        print(f'{xxxx} - Request failed ConnectTimeout! IP:{self.proxy}\n', end='')
    except request_exceptions.RequestException:
        print(f'{xxxx} - Request failed RequestException! IP:{self.proxy}\n', end='')
    proxy = proxy_obj.get_proxy() # Please follow my more blogs for details about IP proxy
    request_count += 1

2. requests file download

import requests
url = "https://xxxx.pdf"
file_path = 'xxxx.pdf'
response = requests.get(url=url, headers=get_headers(), proxies=proxy,  verify = Flase, timeout=5)
with open(file_path , 'wb') as fis:
    for chunk in response.iter_content(chunk_size=1000):
        fis.write(chunk)
        fis.flush()

3. Introduction to common attributes of response

response.statis_code #Return status, 200 indicates successful connection
response.text # String form of response content, content of url link
response.encoding # The encoding method of the response content. If there is no charset in the header, the encoding is ISO-8859-1.
response.apparent_encoding # Analyze the encoding method of the response content in the content
response.encoding = response.apparent_encoding # or =UTF-8
response.content #Binary form of response content
response.raise_for_status() # If it is not 200, an exception will be thrown
response.cookies() # Returns a dictionary type attribute
response.json() # If json is returned, the content can be extracted by parsing the dictionary.

5. Requests exception handling

1. Commonly used exception handling

from requests import exceptions
try:
    # Code to be executed
except exceptions.ConnectTimeout:
    print(f'{xxxx} - Request failed, connection timed out! IP:{self.proxy}\n', end='')
except exceptions.RequestException:
    print(f'{xxxx} - Request failed, request exception! IP:{self.proxy}\n', end='')
except exceptions.ReadTimeout:
	print(f'{xxxx} - Request failed, proxy read timeout! IP:{self.proxy}\n', end='')
except exceptions.ProxyError:
	print(f'{xxxx} - The request failed and the proxy cannot be connected! IP:{self.proxy}\n', end='')
except exceptions.ConnectionError:
	print(f'{xxxx} - The request failed to an unknown server or the network environment is abnormal! IP:{self.proxy}\n', end='')
except Exception:
	print(f'{xxxx} - Uncaught exception above request failed!}\n', end='')

Summarize

The above is what I will talk about today. This article only briefly introduces the use of requests, and requests provides a large number of functions and methods that allow us to quickly and conveniently request network resources. Subsequent common codes about requests will be continuously updated in this blog. ;
Requests Chinese document:https://www.w3cschool.cn/requests2