Я попытался создать программу для извлечения всех песен из моего Daily Mix 1 в Spotify. Я знаю логи c, которые мне нужно использовать, но я не могу получить весь исходный код.
Вот код, который я написал:
import requests
from bs4 import BeautifulSoup
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"}
result = requests.get("https://open.spotify.com/playlist/37i9dQZF1E38L6D2gtQHWw", headers=headers)
src = result.content
soup = BeautifulSoup(src, 'lxml')
print(soup.prettify())
Вот вывод, который я получаю:
<!DOCTYPE html>
<html class="" dir="ltr" lang="en">
<head>
<meta charset="utf-8"/>
<title>
Daily Mix 1 on Spotify
</title>
<meta content="Pritam, Arijit Singh, Rahat Fateh Ali Khan and more" name="description"/>
<meta content="notranslate" property="google"/>
<meta content="174829003346" property="fb:app_id"/>
<meta content="Daily Mix 1" property="og:title"/>
<meta content="Pritam, Arijit Singh, Rahat Fateh Ali Khan and more" property="og:description"/>
<meta content="https://open.spotify.com/playlist/37i9dQZF1E38L6D2gtQHWw" property="og:url"/>
<meta content="https://dailymix-images.scdn.co/v1/img/9b9643ab7444d6184d1b33e534cacc15c1e44186/1/en/default" property="og:image"/>
<meta content="music.playlist" property="og:type"/>
<meta content="Spotify" property="og:site_name"/>
<meta content="Spotify" property="al:android:app_name"/>
<meta content="com.spotify.music" property="al:android:package"/>
<meta content="spotify://playlist/37i9dQZF1E38L6D2gtQHWw" property="al:android:url"/>
<meta content="Spotify" property="al:ios:app_name"/>
<meta content="324684580" property="al:ios:app_store_id"/>
<meta content="spotify://playlist/37i9dQZF1E38L6D2gtQHWw" property="al:ios:url"/>
<meta content="@spotify" property="twitter:site"/>
<meta content="Daily Mix 1" property="twitter:title"/>
<meta content="Pritam, Arijit Singh, Rahat Fateh Ali Khan and more" property="twitter:description"/>
<meta content="https://dailymix-images.scdn.co/v1/img/9b9643ab7444d6184d1b33e534cacc15c1e44186/1/en/default" property="twitter:image"/>
<meta content="audio" property="twitter:card"/>
<meta content="324684580" property="twitter:app:id:iphone"/>
<meta content="com.spotify.music" property="twitter:app:id:googleplay"/>
<meta content="Spotify" property="twitter:audio:partner"/>
<meta content="https://open.spotify.com/twitter/vmap/playlist/37i9dQZF1E38L6D2gtQHWw" property="twitter:audio:source"/>
<meta content="" property="twitter:audio:artist_name"/>
<meta content="https://open.spotify.com/embed/playlist/37i9dQZF1E38L6D2gtQHWw" property="twitter:player"/>
<meta content="504" property="twitter:player:width"/>
<meta content="584" property="twitter:player:height"/>
<link href="https://open.spotify.com/playlist/37i9dQZF1E38L6D2gtQHWw" rel="canonical"/>
<link href="https://open.spotify.com/playlist/37i9dQZF1E38L6D2gtQHWw/metadata" rel="opengraph"/>
<link href="android-app://com.spotify.music/spotify/playlist/37i9dQZF1E38L6D2gtQHWw" rel="alternate"/>
<script type="application/ld+json">
{"@context":"http://schema.googleapis.com/","@type":"MusicGroup","@id":"https://open.spotify.com/playlist/37i9dQZF1E38L6D2gtQHWw","url":"https://open.spotify.com/playlist/37i9dQZF1E38L6D2gtQHWw","name":"Daily Mix 1","description":"Pritam, Arijit Singh, Rahat Fateh Ali Khan and more","datePublished":null,"potentialAction":{"@type":"ListenAction","target":[{"@type":"EntryPoint","urlTemplate":"https://open.spotify.com/playlist/37i9dQZF1E38L6D2gtQHWw","actionPlatform":["http://schema.org/DesktopWebPlatform","http://schema.org/IOSPlatform","http://schema.googleapis.com/GoogleAudioCast","http://schema.googleapis.com/GoogleVideoCast"]},"android-app://com.spotify.music/spotify/playlist/37i9dQZF1E38L6D2gtQHWw/play"],"expectsAcceptanceOf":{"@type":"Offer","category":"free","eligibleRegion":[]}}}
</script>
<link href="https://open.scdn.co/cdn/images/favicon32.a19b4f5b.png" rel="icon" sizes="32x32" type="image/png"/>
<link href="https://open.scdn.co/cdn/images/favicon16.19fc3918.png" rel="icon" sizes="16x16" type="image/png"/>
<link href="https://open.scdn.co/cdn/images/favicon.5cb2bd30.ico" rel="icon"/>
<link href="https://open.scdn.co/cdn/build/web-player/web-player.0d026902.css" rel="stylesheet"/>
<link href="https://open.scdn.co/cdn/generated/manifest-web-player.a3df468f.json" rel="manifest"/>
<script data-testid="config" id="config" type="application/json">
{"clientId":"0971d","accessToken":"BQAnNLIsV4fJD_TRpX1-Va1MZYIrBld9IcyKockZtMnMpLNkvooFpbhEj2HWfiJGioAcInNRtx9gRBU0BzA","accessTokenExpirationTimestampMs":1586190548984,"isAnonymous":true,"app_name":"web_player_prototype","recaptchaWebPlayerFraudSiteKey":"6LcntJ0UAAAAAFSb5F-Lfe5YeOjG5vexXYbUnSbT"}
</script>
<script id="baba" type="application/json">
[]
</script>
<script src="https://open.scdn.co/cdn/js/gtag.cfea467f.js"></script>
<script async="" src="https://www.googletagmanager.com/gtag/js?id=UA-5784146-31"></script>
<link crossorigin="anonymous" href="https://api.spotify.com" rel="preconnect"/>
<link crossorigin="anonymous" href="https://apresolve.spotify.com" rel="preconnect"/>
<link crossorigin="anonymous" href="https://daily-mix.scdn.co" rel="preconnect"/>
<link crossorigin="anonymous" href="https://exp.wg.spotify.com" rel="preconnect"/>
<link crossorigin="anonymous" href="https://gew-dealer.spotify.com" rel="preconnect"/>
<link crossorigin="anonymous" href="https://gew-spclient.spotify.com" rel="preconnect"/>
<link crossorigin="anonymous" href="https://i.scdn.co" rel="preconnect"/>
<link crossorigin="anonymous" href="https://lineup-images.scdn.co" rel="preconnect"/>
<link crossorigin="anonymous" href="https://mosaic.scdn.co" rel="preconnect"/>
<link crossorigin="anonymous" href="https://open.scdn.co" rel="preconnect"/>
<link crossorigin="anonymous" href="https://pixel-static.spotify.com" rel="preconnect"/>
<link crossorigin="anonymous" href="https://pixel.spotify.com" rel="preconnect"/>
<link crossorigin="anonymous" href="https://pl.scdn.co" rel="preconnect"/>
<link crossorigin="anonymous" href="https://spclient.wg.spotify.com" rel="preconnect"/>
<link as="font" crossorigin="anonymous" href="https://open.scdn.co/cdn/fonts/CircularSpUIv3T-Book.3466e0ec.woff2" rel="preload" type="font/woff2"/>
<link as="font" crossorigin="anonymous" href="https://open.scdn.co/cdn/fonts/CircularSpUIv3T-Bold.8d0a45cc.woff2" rel="preload" type="font/woff2"/>
<link as="font" crossorigin="anonymous" href="https://open.scdn.co/cdn/fonts/CircularSpUIv3T-Light.afd9ab26.woff2" rel="preload" type="font/woff2"/>
<link as="font" crossorigin="anonymous" href="https://open.scdn.co/cdn/fonts/spoticon_regular_2.d319d911.woff2" rel="preload" type="font/woff2"/>
<link href="https://api.spotify.com" rel="preconnect"/>
<link href="https://spclient.wg.spotify.com" rel="preconnect"/>
<link href="https://apresolve.spotify.com" rel="preconnect"/>
<style>
.grecaptcha-badge { display: none !important;}
</style>
<script async="" defer="" src="https://www.google.com/recaptcha/api.js?render=6LcntJ0UAAAAAFSb5F-Lfe5YeOjG5vexXYbUnSbT"></script>
<link href="https://open.scdn.co/cdn/generated/opensearch.7e787b90.xml" rel="search" title="Spotify" type="application/opensearchdescription+xml"/>
</head>
<body class="" data-locale="en" data-market="IN">
<div class="body-drag-top">
</div>
<script async="" id="cidsyncjs" src="https://pixel-static.spotify.com/sync.min.js"></script>
<noscript>
<img height="0" id="cidmgmt" src="https://pixel.spotify.com/v1/sync?nojs=1" style="display: none;" width="0"/>
</noscript>
<script data-locale="en" id="jsonTranslations" type="application/json">
["Something went wrong","Try reloading the page","RELOAD PAGE","New Folder","Learn More","Advertisement","Spotify no longer supports this version of {0}. Please update your browser for uninterrupted listening.","Equalizer","Settings","Hide advanced settings","Show advanced settings","So you've discovered the tab limit...","You've got too many tabs open. Close this one and continue listening.","Enable secure playback in your browser","Visit the Spotify support site to see how.","GO TO SUPPORT SITE","Manual","Flat","Acoustic","Bass booster","Bass reducer","Classical","Dance","Deep","Electronic","HipHop","Jazz","Latin","Loudness","Lounge","Piano","Pop","RnB","Rock","Small speakers","Spoken word","Treble booster","Treble reducer","Vocal booster","equalizer.playbackOnOtherDevice","equalizer.notSupported","Enable equalizer","Presets","Reset","Reload","Liked Songs","This is already in your '{0}' playlist.","These are already in your '{0}' playlist.","Some of these are already in
your '{0}' playlist.","Already added","Some already added","Add all","Add anyway","Add new ones","Don't add","Get Spotify Premium","Enjoy unlimited access to music,
personalized playlists, and more. Eligible members receive their first month on us.","Subscribe","Log out","%price%/month after. Terms and conditions apply. One month free not available for users who have already tried Premium.","Hide Advertisement","Full screen","Close","More","Pause","Play","Remove from Your Library","Save to
Your Library","Recently played","Employee only","Revert to the classic Desktop app - Changes will be applied immediately and you will land on Home.<br/>To opt back in, click the screen icon in the Now Playing Bar.<br/>To help us improve the experience, <a href={0}>send feedback</a>","Download","Unlock downloads and other features with Premium","Remove download","Cancel download","progress.downloading-tracks","Top bar and user menu","Go back","Go forward","Private session","You’re listening
anonymously in a private session.","You're offline","Make sure you're online. Spotify works best with an internet connection.","Sign up","Log in","See all","Downloading {0} tracks","Download complete","Profile","Genre","Explicit","{0} min","Episode","Podcast","Song","Album","By {0}","Playlist","Artist","Close","Account","Upgrade to Premium","Private session","Settings","Update Spotify now","Update available","Upgrade","Don't play this song","Don't play this artist","Copy Link","Unfollow","Follow","Copy Show Link","Copy Episode Link","Add to Queue","Add to Playlist","Remove from this Playlist","See Episode Description","We couldn't find the concert you're looking for.","Available on {0}","From {0} on {1}","Sold on {0}","Details","This concert already happened. Check out <a href=\"spotify:app:concerts\"> upcoming concerts</a> in the Browse section.","Start Radio","Copy Artist Link","Delete","Create Folder","Rename","Remove from Your Library","Make secret","Make public","Make non-collaborative","Make collaborative","Delete","Copy Playlist link","Save to Your Library","Copy Album Link","Display","Show announcements about new releases","Show
desktop notifications when the song changes","Language","Choose language - Changes will be applied after restarting the app","Restart App","Remove from your Liked Songs","Save to your Liked Songs","Copy Song Link","Explicit content","Explicit content can't be played on this Family account","Allow playback of explicit-rated content","Autoplay","Autoplay similar songs when your music ends","Automatic","Low","Normal","High","Very High","Loud","Normal","Quiet","Music quality","Streaming quality","Normalize volume - Set the same volume level for all songs and podcasts","Volume level - Adjust the volume for your environment","Social","Make my new playlists public","Start a private session to listen anonymously","Share my listening activity on Spotify","Show my recently played artists","An error occurred","Minimized","Yes","No","sec","Playback and automix","Crossfade songs","Allow smooth transitions between songs in a playlist","Startup and window behaviour","Open Spotify automatically after you log into the computer","Close button should minimize the Spotify window","Compatibility","Enable hardware acceleration","Privacy","Block all cookies for this installation of the Spotify desktop app; read more details in the <a class=\"settings__cookiesDiabled-link\" href=\"https://www.spotify.com/legal/privacy-policy/\">privacy policy</a>. Please note that enabling this setting may negatively impact your Spotify experience. Changes will be applied after restarting the app","Start listening with a free Spotify account","Get the most out of Spotify with a free account","No credit card, ever","Get unlimited podcasts","Play your favorite music, with ads","Following","We and %open_cookies_vendor_tag%our partners%close_cookies_vendor_tag% use cookies to personalize your experience, to show you ads based on your interests, and for measurement and analytics purposes. By using our website and our services, you agree to our use of cookies as described in our %open_cookies_policy_tag%Cookie Policy%close_cookies_policy_tag%.","Cookie policy","Confirm your age","Main","Home","Search","Your Library","Open in Desktop app","Playing from","Exit full screen","{0}","{0} with {1}","{0} with {1} and {2}","{0} with {1}, {2} and {3}","{0} with {1}, {2}, {3} and {4} more...","{0}","{0} & {1}","{0}, {1} & {2}","{0}, {1}, {2} & {3}","{0}, {1}, {2}, and {3} more...","Do you really want to delete this folder and all playlists inside?","CANCEL","DELETE","Do you really want to delete this playlist?","CANCEL","DELETE","New Playlist","Create new playlist","Playlist Name","CANCEL","CREATE","Add to playlist","NEW PLAYLIST","The Lineup","Recommended For You","Want to see more concerts in your area? Check out your concerts section.","See More Concerts","Find Tickets","Spotify – Play Queue","Spotify – Download for Desktop","%name% - %show% | Podcast on Spotify","Oooops, something went wrong while loading the episode.","You might also like","Sign up free","Already have
an account?","Log in","Spotify – Your Library","Third-party licenses","About Spotify","Copyright © {0} Spotify AB.<br/>Spotify® is a registered trademark of the Spotify Group.","Content provided by","{0}/{1}","Monthly Listeners","Headliner","Less","More","Add to Playlist","Various artists","Play Queue","There are no songs to play in the queue.","Now Playing","Next in Queue","Next Up","Get our free app","Seamlessly listen to music you love. Download the Spotify app for your computer.","DOWNLOAD DESKTOP APP","PAUSE","RESUME","PLAY","Finished","Played",{"one":"{0} min left","other":"{0} mins left"},"You are offline. Please go online and try again.","Oooops, something went wrong while loading your songs.","Find more of the songs you love in Browse and save to your Liked Songs. ","Songs you’ve liked live here","DISCOVER","SIGN UP FREE","DOWNLOAD APP","A new version of Spotify is available ({0}).","Click here to download.","Downloading a new version of Spotify...","Spotify has been updated to version {0}.","Please restart to install.","Read Less","Read More","Spotify – {0}","Playlist","Let's find some songs for your playlist","It's a bit
empty here...","NEW RELEASES","Oooops, something went wrong while loading the playlist.","Loading","Recommended Songs","Based on the title of this playlist.","Based
on the songs in this playlist.","Refresh",{"one":"{0} episode","other":"{0} episodes"},{"one":"{0} song","other":"{0} songs"},"By","various artists","PAUSE","PLAY","Presented By {0}",{"one":"{0} like","other":"{0} likes"},"PREMIUM","Add","PAUSE","PLAY","You need to be at least 19 years old to listen to explicit content marked with","Continue","Playlists","Podcasts","Artists","Albums","Oooops, something went wrong while loading your artists.","Follow artists you love to add them to Your Library.","Your artists will appear here.","Oooops, something went wrong while loading your podcasts.","Looking for something new?","Podcasts you’ve liked live here","Oooops, something went wrong while loading your albums.","Save albums you love to build the collection of your dreams.","Save your favourite albums","Oooops, something went wrong while loading your playlists.","Create your first playlist","We’ll help you make the perfect mixtape, minus the tape.","Start typing...","Playlists","Create Playlist","Collaborative Playlist","Install App","Enable repeat","Enable repeat one","Disable repeat","Disable shuffle","Enable shuffle","Skip back 15 seconds","Skip forward 15 seconds","Player controls","Previous","Pause","Play","Next","Now playing: {0} by {1}","Remove","Show video","Hide video","Expand Video","Collapse","Play Track","Skip this ad in:","Expand","Change progress","Download the free app","Queue","Cookies","Privacy","Connect to a device","Play and control Spotify on all your devices.","Start Spotify on another device and it will magically appear here.","Learn more","Connect to a device","What is Spotify Connect?","This Computer","This Web Browser","Listening On","Spotify Connect","Unavailable to control","Unmute","Mute","Change volume","You're listening on","Preview of Spotify","Sign up to get unlimited songs and podcasts with occasional ads. No credit card needed.","Spotify – {0}","Oooops, something went wrong while loading the artist.",{"one":"{0} monthly listener","other":"{0} monthly listeners"},"Overview","Related Artists","About","concerts","Verified Artist","%name% | Podcast on Spotify","Oooops, something went wrong while loading the podcast.","Spotify – Home","Oooops, something went wrong while loading the categories.","Hide Announcements","Sponsored","spotify exclusive","Presented By","Remove","Save","PAUSE","PLAY","Search by city","Change","This artist has no concert listings.","Near {0}","Other Locations","This artist has no upcoming concerts near {0}.","This artist has no upcoming concerts near you.","To see more concerts by artists you love, <a href=\"spotify:app:concerts\" data-section-id=\"footer\"> browse all concerts</a>.","Biography","Where people listen","Albums","Singles and EPs","Compilations","Appears On","SHOW MORE","SHOW LESS","Popular","We could not find the location you're looking for.","There was an error requesting data.","Cancel","Artists","Albums","Playlists","Podcasts","Episodes","Profiles","Genres","Spotify – Search","Spotify – {0}","Oooops, something went wrong while loading the album.","Album","Premium Only","This artist has asked us to release this album
on Premium only for a little while, but check back soon.","Search for Artists, Songs, or Podcasts","Clear search field","Songs","Showing {0} for \"{1}\"","No results found for \"{0}\"","Please make sure your words are spelled correctly or use less or different keywords.","Top result","Recent searches","Clear recent searches","Made For You","Charts","New Releases","PODCASTS","Play / Pause","Shuffle","Repeat","Skip to previous","Skip to next","Raise volume","Lower volume","Home","Search","Your playlists","Your podcasts","Your artists","Your albums","Liked songs","Made for you","Currently playing","Charts","Added to your Liked Songs","Saved to Your Library","Removed from your Liked Songs","Removed from Your Library","Track was added to your playlist","Album was added to your playlist","Episode was added to your playlist","Got it! Song saved to Your Library.","Got it! We won't play this in your Daily Mix.","Link copied to clipboard","Oooops, something went wrong while loading the album's tracks.","Oooops, something went wrong while loading the artist's albums.","Oooops, something went wrong while loading the user's playlists.","Oooops, something went wrong with the playback.","Oooops, something went wrong.","Welcome to your Spotify app","Can't play the current song.","Please turn off offline mode and try again.","Please turn off offline mode to download.","We can't play this until you allow this artist in the Spotify phone app.","We can't play this until you allow
this track in the Spotify phone app.","Spotify can't play this in {0}. If you have the file on your computer you can import it.","Spotify can't play this right now.
If you have the file on your computer you can import it.","Spotify can't play this right now.","The selected song will be played after the ads.","Please try that again after this ad.","Your track will play after the ads. Skip ads to get back to your music faster!","You'll be able to skip the ad and get back to your content after {0} seconds.","You've exceeded your {0} hour time limit. Upgrade to Spotify Premium to listen without restrictions.","We're not able to play this video in your current location.","Please upgrade Spotify to play this video.","This video cannot be played on your operating system version.","We're not able to play this video in your current location.","This video is unavailable. Try another?","Sorry, we're not able to play this video.","Sorry, we're not able to play this video.","Hmm... we can't seem to play this video. Try installing the latest version of Spotify.","Sorry! You can't play this content while connected to another device.","Spotify can't play this right now because it contains explicit content.","The selected content will play after the ads"]
</script>
<script id="features" type="application/json">
{"enableShows":true,"isFusionSearchInspiredTracksEnabled":false,"isImage2Enabled":false,"isShareEditorialContext":false,"isSocialSessionPopupEnabled":false,"isTracingEnabled":false,"upgradeButton":"control","useEnhancedChromeOSPWACallout":false,"mwpLibrary":false,"mwpServerSideProfile":false,"isCollaborativePlaylistsEnabled":false,"mwp":false,"isMWPErrorCodeEnabled":false,"mwpGenrePages":false,"mwpPreAuthTest":false,"mwpPreAuthVariant":null,"isMWPQualarooEnabled":false,"isMwpRadioEntity":true,"preauthRecaptcha":false,"isEqualizerABEnabled":false,"isPathfinderForArtistPageEnabled":false,"sharedPlaybackAbstraction":false,"accountlessPodcastPlayback":false}
</script>
<script src="https://open.scdn.co/cdn/build/web-player/web-player.0d026902.js"></script>
<script src="https://open.scdn.co/cdn/build/web-player/vendor~web-player.4c2d31a8.js"></script>
<script defer="" src="https://www.fastly-insights.com/insights.js?k=040e3997-282c-4275-ba9b-a406ce78b133&dnt=1"></script>
</body>
</html>
Используемые мной заголовки работали другие сайты, такие как Amazon и Wikipedia, так что я не думаю, что это проблема. Я также не думаю, что проблема связана с javascript, потому что в других программах для очистки веб-сайтов, таких как amazon (которые также содержат много тегов <script>), код отображался идеально.
Пожалуйста, расскажите, в чем проблема.
PS - Пожалуйста, не рекомендуйте селен или скрап в ваших решениях.