Как проанализировать набор данных JSON из определенного блока HTML Javascript, используя Python?

Question

Я пытаюсь очистить некоторые данные онлайн-продаж от https://www.timbuk2.com/collections/sale. Я могу сузить их, чтобы извлечь блок сценария, который содержит набор данных JSON, но я изо всех сил пытаюсь извлечь данные JSON.

Вот что у меня есть

import requests
from bs4 import BeautifulSoup
import re
import json
import pandas as pd

URL = 'https://www.timbuk2.com/collections/sale'
response = requests.get(URL)
soup = BeautifulSoup(response.content, 'html.parser')

scripts = soup(text=re.compile(r'products:'))[0].parent
print(scripts)

Моя цель - разобрать словарь BOLD.collection_filter.config

BOLD.collection_filter.config = {
      products: [{
    "order":0,
    "id":1431161864234,
    "title":"Command Messenger Bag",
    "vendor":"Timbuk2-Sale",
    "type": "Messenger","tags": ["category-Messenger","collections-bestsellers","collections-bike","collections-work","gender-Mens","gender-Womens","madetype-Finished Good","site-Sale","product_type-Messenger","occasions-Bike","occasions-Travel","occasions-Work"],
    "handle": "174-command-messenger-bag-sale","image": "1/2572/9458/products/timbuk2-messenger-command-messenger-bag-nautical-adobe_174-4-7480_front-sale-ffe38181818183ff-1989.jpg?v=1539383378","created_at": "2018-04-20 15:14:13 -0700",
    "published_at": "2018-04-20 15:14:13 -0700","image_aspect_ratio": "1.2006003001500751",
    "image_width":2400,
    "image_height":1999,
    "option_values":[["S","Midway"],["M","Jet Black"],["M","Midway"],["M","Surplus"],["M","Black\/Gunmetal"],["M","Carbon Full-Cycle Twill"],["M","Black\/Red Devil"],["M","Abyss"],["M","Carbon and Molasses"],["M","Oxide and Adobe"],["L","Midway"],["M","Nautical\/Adobe"]],
    "options":["Size","Color"],
    "insights":null,"variants": [

Спасибо.

Answer 1

Этот вопрос, похоже, задает вопрос, как извлечь конкретную переменную javascrip. Попробуйте это с другими примерами

scripts = soup(text=re.compile(r'products:'))[0].parent
variable_name = "BOLD.collection_filter.config"
re.search(r"(?<=%s).*?(\{.*)(?=\;|<\/script>)"%variable_name,str(scripts),re.S).group(1)

Answer 2

Сначала получите строку, которая является допустимой JSON. Поиграйтесь с онлайн-форматером JSON (например, https://jsonformatter.org/)

Действителен следующий JSON:

{
    "products": [{
    "order":0,
    "id":1431161864234,
    "title":"Command Messenger Bag",
    "vendor":"Timbuk2-Sale",
    "type": "Messenger","tags": ["category-Messenger","collections-bestsellers","collections-bike","collections-work","gender-Mens","gender-Womens","madetype-Finished Good","site-Sale","product_type-Messenger","occasions-Bike","occasions-Travel","occasions-Work"],
    "handle": "174-command-messenger-bag-sale","image": "1/2572/9458/products/timbuk2-messenger-command-messenger-bag-nautical-adobe_174-4-7480_front-sale-ffe38181818183ff-1989.jpg?v=1539383378","created_at": "2018-04-20 15:14:13 -0700",
    "published_at": "2018-04-20 15:14:13 -0700","image_aspect_ratio": "1.2006003001500751",
    "image_width":2400,
    "image_height":1999,
    "option_values":[["S","Midway"],["M","Jet Black"],["M","Midway"],["M","Surplus"],["M","Black\/Gunmetal"],["M","Carbon Full-Cycle Twill"],["M","Black\/Red Devil"],["M","Abyss"],["M","Carbon and Molasses"],["M","Oxide and Adobe"],["L","Midway"],["M","Nautical\/Adobe"]],
    "options":["Size","Color"]
}]
}

Так что, может быть, немного адаптируем ваше регулярное выражение.

Тогда используйте

import json
from pprint import pprint

data = json.loads(your_string)

pprint(data)