目标站点 有声小说-有声读物-电台广播在线听书-蜻蜓FM
python爬虫实战搞的是PC端,移动端可能有更加方便的接口,欢迎大家留意讨论反正是练手就随便抓一下pc端的包好了
主要内容
- 1.post请求登录
- 2.hmacMD5算法的简单使用
这个例子的登录非常简单没有任何加密直接post就行真的是一点加密和未知参数都没有
私信小编01即可获取大量Python学习资源
?
python实现,注意此处是类的一个方法不完整无法直接运行
def login(self,user_id,password): data = { account_type: 5, device_id: web, user_id: user_id, password: password } response = self.session.post(self.login_url,data=data) if response.status_code==200: temp = response.json() errorno = temp[errorno] errormsg = temp[errormsg] if errorno == 0: print(login successful!,登录成功!) data = temp[data] self.qingting_id = data[qingting_id] self.access_token = data[access_token] else: print(Login failed,登录失败) print(errormsg)
录成功后我们把access_token和qingting_id拿到,相当于一个登录后的标志如果账号是会员相当于一个会员标志
音频的真实地址请求了这样一个url:
https://audio.qingting.fm/audiostream/redirect/294280/11604885
其中294280是专辑id,
11604885是当前音频的id
?
其中还带了一些参数比如access_token,qingting_id(登录成功的response中有,上图没有登录所有是空的),另外还有一些比如t是时间戳,
device_id=MOBILESITE(不变)
关键就在于sign(尝试过不加sign会返回一个签名错误)
可以通过全局搜索试一下是哪个js生成的这个sign我全局搜索了一下
device_id
在mian.一大堆.js 找到了生成sign的函数(需要自己分辨一下是一个device_id: “MOBILESITE”的)
搜索其他关键字应该也是可以顺利找到的
?
这里的sign是u这个变量它是由c这个变量通过一堆加密处理得到的
我们可以控制台输出一下u和c
?
所以我们就知道了sign实际是加密了请求的其他参数
一开始我误以为是单纯的MD5所以卡了好久(还进入函数内部看他具体是怎么实现的看的一头雾水)
其实代码已经告诉用的是
createHmac(“md5”, “fpMn12&38f_2e”)
查了下Hmac发现就是一种现成的算法,还有不同的模式MD5是其中一种,需要一个秘钥
这里什么都告诉你了,用Hmac-md5秘钥是fpMn12&38f_2e
找个在线加密的网站试了下,果然和刚才控制台输出的一样
?
python的话需要import
hmac这个库
import hmacimport time base_url = "https://audio.qingting.fm"bookid = "294280"id = "11590788"access_token = ""qingting_id =""timestamp = str(round(time.time()*1000))data = f"/audiostream/redirect/{bookid}/{id}?access_token={access_token}&device_id=MOBILESITE&qingting_id={qingting_id}&t={timestamp}"message = data.encode(utf-8)key = "fpMn12&38f_2e".encode(utf-8)sign = hmac.new(key, message, digestmod=MD5).hexdigest()whole_url = base_url+data+"&sign="+signprint(whole_url)
得到一个音频可以做到了剩下的就是得到一堆了,其实我们得到每个音频的id就可以了
我请求的是这个接口
info_api = https://i.qingting.fm/capi/channel/{self.bookid}/programs/{self.version}?curpage={str(page)}&pagesize=30&order=asc
version在声书主页的源代码中,只要改curpage就可以翻页了 完整源码
import requestsimport reimport hmacimport timefrom tqdm import tqdmfrom bs4 import BeautifulSoupimport osimport jsonimport sysimport urllib3urllib3.disable_warnings() class QingTing(): def __init__(self,user_id,password,bookurl,ifLogin): self.ifLogin = ifLogin self.user_id = user_id self.password = password self.session = requests.session() self.session.headers.update({user-agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36}) self.login_url = "https://u2.qingting.fm/u2/api/v4/user/login" self.qingting_id = self.access_token = self.bookurl = bookurl self.bookurl = https://www.qingting.fm/channels/257790 self.bookid = self.bookurl.split(/)[-1] self.version = self.qingtinghost = https://audio.qingting.fm self.save_path = self.bookname = def login(self,user_id,password): data = { account_type: 5, device_id: web, user_id: user_id, password: password } response = self.session.post(self.login_url,data=data,verify=False) if response.status_code==200: temp = response.json() errorno = temp[errorno] errormsg = temp[errormsg] if errorno == 0: print(login successful!,登录成功!) data = temp[data] self.qingting_id = data[qingting_id] self.access_token = data[access_token] else: print(Login failed,登录失败) print(errormsg) time.sleep(10) sys.exit(0) def __get_version(self): response = self.session.get(url=self.bookurl,verify=False) if response.status_code==200: soup = BeautifulSoup(response.text,lxml) temp_bookname = soup.select(div.album-info-root > div.top > div.info.right > h1)[0].string replaced_pattern = [\\\/:\*\?\"<>|] self.bookname = re.sub(replaced_pattern, ,temp_bookname,flags=re.M +re.S) if not os.path.exists(self.bookname): os.makedirs(self.bookname) matched = re.search(\"version\":\"(\w+)",response.text,re.S) if matched: version = matched.group(1) self.version = version return version def __get_total_page(self): self.__get_version() page = 1 info_api = fhttps://i.qingting.fm/capi/channel/{self.bookid}/programs/{self.version}?curpage={str(page)}&pagesize=30&order=asc response = self.session.get(info_api,verify=False) if response.status_code==200: temp = response.json() total = temp[data][total] total_page = int(int(total)/30)+1 return total,total_page def get_book_info(self): total,total_page = self.__get_total_page() print(self.bookname,共{}集.format(total)) for page in range(1,total_page+1): info_api = fhttps://i.qingting.fm/capi/channel/{self.bookid}/programs/{self.version}?curpage={str(page)}&pagesize=30&order=asc response = self.session.get(info_api,verify=False) programs = response.json()[data][programs] for program in programs: print(program[id],program[title]) yield program def get_src(self,id): bookid = self.bookid access_token = self.access_token qingting_id =self.qingting_id timestamp = str(round(time.time()*1000)) data = f"/audiostream/redirect/{bookid}/{id}?access_token={access_token}&device_id=MOBILESITE&qingting_id={qingting_id}&t={timestamp}" message = data.encode(utf-8) key = "fpMn12&38f_2e".encode(utf-8) sign = hmac.new(key, message, digestmod=MD5).hexdigest() whole_url = self.qingtinghost+data+"&sign="+sign return whole_url def downloadFILE(self,url,name): resp = self.session.get(url=url,stream=True,verify=False) if resp.headers[Content-Type] ==audio/mpeg: content_size = int(int(resp.headers[Content-Length])/1024) with open(name, "wb") as f: print("Pkg total size is:",content_size,k,start...) for data in tqdm(iterable=resp.iter_content(1024),total=content_size,unit=k,desc=name): f.write(data) print(name , "download finished!") else: errorno = resp.json()[errorno] errormsg = resp.json()[errormsg] print(没有权限下载,请登录已购此音频的账号。) print(errorno:,errorno,errormsg) def run(self): if self.ifLogin: self.login(self.user_id,self.password) programs = self.get_book_info() count = 0 for program in programs: count+=1 try: id = program[id] title = str(count).zfill(4)+ +program[title]+.m4a if not self.bookname ==: title = os.path.join(self.bookname,title) whole_url = self.get_src(id) self.downloadFILE(whole_url,title) except Exception as e: print(e) with open(log.txt,a,encoding=utf-8) as f: f.write(str(count)+str(e)+\n) def get_config_info(): with open(config.json,r,encoding=utf-8) as f: config = json.loads(f.read()) return config if __name__ == "__main__": pyinstaller -F -i ico.ico QingTingFM.py config = get_config_info() if config["ifLogin"]: bookurl = input(请输入要下载音频的主页链接:(如[url=https://www.qingting.fm/channels/257790]https://www.qingting.fm/channels/257790[/url])) isvalid = re.search(https://www.qingting.fm/channels/\d+,bookurl) if isvalid: q = QingTing(config["user_id"],config["password"],bookurl,1) q.run() else: print("输入的主页格式错误") else: 不登录 bookurl = input(请输入要下载音频的主页链接:(如[url=https://www.qingting.fm/channels/257790]https://www.qingting.fm/channels/257790[/url])) isvalid = re.search(https://www.qingting.fm/channels/\d+,bookurl) if isvalid: q = QingTing(config["user_id"],config["password"],bookurl,0) q.run() else: print("输入的主页格式错误")
配置文件部分
{ "ifLogin":1, "user_id":"135", "password":"pwd"}