Очистка информации с сайта с помощью lxml - PullRequest
Новая
       28

Очистка информации с сайта с помощью lxml

0 голосов
/ 16 января 2011

Я пытаюсь получить список всех заголовков с сайта Reddit.com, используя lxml.Я использовал этот запрос: 

  reddit = etree.HTML( urllib.urlopen("http://www.reddit.com/r/all/top").read() )
  reddit.xpath("//div[contains(@class,'title')]//b/text()")

Однако при запуске выражения в оболочке Python ничего не появляется.XPath неверен?

Запуск с Python 2.7

Вот полный код: 

import urllib
import os, random, sys, math
from lxml import etree

def main():

    reddit = etree.HTML( urllib.urlopen("http://www.reddit.com/r/all/top").read() )
    reddit.xpath("//div[contains(@class,'title')]//b/text()")



if __name__ == "__main__":
    main()

Ответы [ 2 ]

6 голосов
/ 16 января 2011

Reddit имеет API . Вам не нужно это очищать. Просто добавьте '.json' в конце URL: 

#!/usr/bin/env python
import json
import urllib2

url = "http://www.reddit.com/r/all/top/.json"
data = json.load(urllib2.urlopen(url))
for child in data['data']['children']:
    print child['data']['title']

Пример вывода 

Dear America, I Saw You Naked: And yes, we were laughing. Confessions of an ex-TSA agent
My wife and I are expecting our son in June, so I installed a fiber-optic star ceiling :)
You wouldn't download a car: Honda releases concept car 3D printing files
So my liquor store I managed closed today, the VP came in to collect the liquor but told me "we're not going to resell the beer, we'll be here about an hour fill up your car."
Baby Olinguito (Recently Discovered Species!)
Bower Bird- in a desperate bid for attention from the opposite sex, Bower males build nests, then decorate with objects of a single color. (xpost- /r/everythingscience)
My friend works as a English teacher in Sweden.
My kid's homework, I think the page designer has had enough.
Man Washes up in Marshall Islands 'After 16 Months Adrift' at sea
Kitten plays the air harp
New roommate already started off on a bad note with us.
MRW a program crashes and asks to contact tech support... and I am tech support.
Jack Black just posted this to facebook. "This is fan art. But it's exactly how I remember it."
Looks like Colorado's legalization has caused problems after all. [4]
My new kitten likes to "hold hands." She does this for as long as you offer your finger.
Ahahaha he got you go-wahhhh
Shipwrecked man makes land 'after 16 months adrift'
As someone who's taken math at university
Footage released of Guardian editors destroying Snowden hard drives: GCHQ technicians watched as journalists took angle grinders and drills to computers after weeks of tense negotiations
TIL Mike Tyson offered a zoo attendant $10,000 to open the cage of a bullying gorilla so he could "smash that silverback's snotbox." His offer was declined.
Microsoft being helpful as always
President Barack Obama says in a new interview that he would support efforts to remove marijuana from the federal government’s list of the most serious narcotics, but that Congress must act to make the change.
advisory
Vila Franca's Islet, Azores Archipelago, Portugal [1440x900] - How can that be so spherical?
The dad on my Child Development book is putting the kids helmet on backwards.
2 голосов
/ 16 января 2011

Вы не были подключены к Интернету.Попробуйте еще раз.

И / ИЛИ

Ваша установка Python либо повреждена, либо вы смешали две трассировки стека ... обратите внимание, как пути внезапно меняются с 3.1 на 2.7 !!!!!!!

Обновление

В оболочке ничего не появляется, потому что вы ничего не печатаете.

По крайней мере, если вместо reddit.xpath("blahblah") вы делаете

result = reddit.xpath("blahblah")
print result

вы увидите, что ваша текущая версия "blahblah" производит [] и будет в хорошей форме, чтобы заметить, если игра с "blahblah" улучшит ситуацию.

...