Как избежать ошибки в DBpedia Spotlight в python?

Question

Когда я запускаю эту программу, она выдает ошибку в тексте, и я не знаю, почему, я пытался запустить ее также из файла, и это также дает мне ошибку, как это может работать?

import json
from SPARQLWrapper import SPARQLWrapper, JSON
import requests
import urllib.parse

## initial consts
BASE_URL = 'http://api.dbpedia-spotlight.org/en/annotate?text={text}&confidence={confidence}&support={support}'

TEXT = """TITULO: Albedo SUBTITULO Y PARRAFO: ===Trees===
Because forests generally have a low albedo, (the majority of the ultraviolet and visible spectrum is absorbed through photosynthesis), some scientists have suggested that greater heat absorption by trees could offset some of the carbon benefits of afforestation (or offset the negative climate impacts of deforestation). In the case of evergreen forests with seasonal snow cover albedo reduction may be great enough for deforestation to cause a net cooling effect.<ref>  cite journal   last1 = Betts   first1 = RA   date = 2000   title = Offset of the potential carbon sink from boreal forestation by decreases in surface albedo   journal = Nature   volume = 408   issue = 6809  pages = 187–190   doi = 10.1038/35041545   pmid=11089969 bibcode = 2000Natur.408..187B </ref> Trees also impact climate in extremely complicated ways through evapotranspiration. The water vapor causes cooling on the land surface, causes heating where it condenses, acts a strong greenhouse gas, and can increase albedo when it condenses into clouds.<ref>  cite journal   last1 = Boucher   date = 2004   title = Direct human influence of irrigation on atmospheric water vapour and climate   journal = Climate Dynamics   volume = 22   issue = 6–7  pages = 597–603   doi=10.1007/s00382-004-0402-4 display-authors=etal bibcode = 2004ClDy...22..597B   url = https://www.academia.edu/25329256</ref> Scientists generally treat evapotranspiration as a net cooling impact, and the net climate impact of albedo and evapotranspiration changes from deforestation depends greatly on local climate.<ref>  cite journal   last1 = Bonan   first1 = GB   date = 2008   title = Forests and Climate Change: Forcings, Feedbacks, and the Climate Benefits of Forests   journal = Science   volume = 320   issue = 5882  pages = 1444–1449   doi = 10.1126/science.1155121   pmid=18556546 bibcode = 2008Sci...320.1444B    url = https://zenodo.org/record/1230896 </ref>
TITULO: Albedo SUBTITULO Y PARRAFO: ===Human activities===
Human activities (e.g., deforestation, farming, and urbanization) change the albedo of various areas around the globe. However, quantification of this effect on the global scale is difficult, further study is required to determine anthropogenic effects.<ref>  Cite journal last=Sagan first=Carl last2=Toon first2=Owen B. last3=Pollack first3=James B. date=1979 title=Anthropogenic Albedo Changes and the Earths Climate journal=Science volume=206 issue=4425 pages=1363–1368 issn=0036-8075 jstor=1748990 doi=10.1126/science.206.4425.1363 pmid=17739279 bibcode=1979Sci...206.1363S</ref>
TITULO: Abraham Lincoln SUBTITULO Y PARRAFO: ==U.S. House of Representatives, 1847–1849==
File:Abraham Lincoln by Nicholas Shepherd, 1846-crop.jpg thumb upright alt=Middle aged clean shaven Lincoln from the hips up. Lincoln in his late 30s as a member of the U.S. House of Representatives. Photo taken by one of Lincolns law students around 1846.
True to his record, Lincoln professed to friends in 1861 to be an old line Whig, a disciple of Henry Clay.<ref name=Donald1996/>  rp 222  Their party favored economic modernization in banking, tariffs to fund internal improvements including railroads, and urbanization.<ref name=Boritt2002/>  rp 137–153
TITULO: Abraham Lincoln SUBTITULO Y PARRAFO: ===Re-election===
  Main 1864 United States presidential election
File:ElectoralCollege1864.svg thumb upright=1.3 alt=Map of the U.S. showing Lincoln winning all the Union states except for Kentucky, New Jersey, and Delaware. The Southern states are not included. An Electoral College (United States) electoral landslide for Lincoln (in red) in the 1864 election; southern states (brown) and territories (gray) not in play
TITULO: Algeria SUBTITULO Y PARRAFO: ===Research and alternative energy sources===
Algeria has invested an estimated 100 billion dinars towards developing research facilities and paying researchers. This development program is meant to advance alternative energy production, especially solar and wind power.<ref>  cite web url=http://portail.cder.dz/spip.php?article1571  title=Archived copy  accessdate=31 October 2016  url-status=dead  archiveurl=https://web.archive.org/web/20161101101749/http://portail.cder.dz/spip.php?article1571  archivedate=1 November 2016 </ref> Algeria is estimated to have the largest solar energy potential in the Mediterranean, so the government has funded the creation of a solar science park in Hassi RMel. Currently, Algeria has 20,000 research professors at various universities and over 780 research labs, with state-set goals to expand to 1,000. Besides solar energy, areas of research in Algeria include space and satellite telecommunications, nuclear power and medical research.
Algeria has invested an estimated 100 billion dinars towards developing research facilities and paying researchers. This development program is meant to advance alternative energy production, especially solar and wind power.<ref>  cite web url=http://portail.cder.dz/spip.php?article1571  title=Archived copy  accessdate=31 October 2016  url-status=dead  archiveurl=https://web.archive.org/web/20161101101749/http://portail.cder.dz/spip.php?article1571  archivedate=1 November 2016 </ref> Algeria is estimated to have the largest solar energy potential in the Mediterranean, so the government has funded the creation of a solar science park in Hassi RMel. Currently, Algeria has 20,000 research professors at various universities and over 780 research labs, with state-set goals to expand to 1,000. Besides solar energy, areas of research in Algeria include space and satellite telecommunications, nuclear power and medical research.
TITULO: Anthropology SUBTITULO Y PARRAFO: ==== Ecological ====
  main Ecological anthropology
Ecological anthropology is defined as the study of cultural adaptations to environments.<ref name=Kottak>  cite book last=Kottak first=Conrad Phillip title=Anthropology : appreciating human diversity year=2010 publisher=McGraw-Hill location=New York isbn=978-0-07-811699-5 pages=579–584 edition=14th</ref> The sub-field is also defined as, the study of relationships between a population of humans and their biophysical environment.<ref name=Townsend>  cite book last=Townsend first=Patricia K. title=Environmental anthropology : from pigs to policies year=2009 publisher=Waveland Press location=Prospect Heights, Ill. isbn=978-1-57766-581-6 page=https://archive.org/details/environmentalant0000town/page/104 104 edition=2nd url=https://archive.org/details/environmentalant0000town/page/104</ref> The focus of its research concerns how cultural beliefs and practices helped human populations adapt to their environments, and how their environment across space and time.<ref name=Kottak CP 1999>  cite journal last1=Kottak first1=Conrad P. jstor=683339 title=The New Ecological Anthropology journal=American Anthropologist volume=101 issue=1 pages=23–35 year=1999 doi=10.1525/aa.1999.101.1.23 url=https://www.researchgate.net/publication/227632647 hdl=2027.42/66329</ref> The contemporary perspective of environmental anthropology, and arguably at least the backdrop, if not the focus of most of the ethnographies and cultural fieldworks of today, is political ecology. Many characterize this new perspective as more informed with culture, politics and power, globalization, localized issues, century anthropology and more.<ref name=Pyke G 1984>  cite journal last1=Pyke first1=G H title=Optimal Foraging Theory: A Critical Review journal=Annual Review of Ecology and Systematics volume=15 page=523 year=1984 doi=10.1146/annurev.es.15.110184.002515 url=https://www.researchgate.net/publication/243636746</ref> The focus and data interpretation is often used for arguments for/against or creation of policy, and to prevent corporate exploitation and damage of land. Often, the observer has become an active part of the struggle either directly (organizing, participation) or indirectly (articles, documentaries, books, ethnographies). Such is the case with environmental justice advocate Melissa Checker and her relationship with the people of Hyde Park.<ref name=Checker M 2005>  cite book author=Melissa Checker title=Polluted promises: environmental racism and the search for justice in a southern town url=https://archive.org/details/pollutedpromises0000chec url-access=registration date=August 2005 publisher=NYU Press isbn=978-0-8147-1657-1</ref>
TITULO: Anthropology SUBTITULO Y PARRAFO: === Urban ===
  main Urban anthropology
Urban anthropology is concerned with issues of urbanization, poverty, and neoliberalism. Ulf Hannerz quotes a 1960s remark that traditional anthropologists were a notoriously agoraphobic lot, anti-urban by definition. Various social processes in the Western World as well as in the Third World (the latter being the habitual focus of attention of anthropologists) brought the attention of Anthropology#Focus on the other cultures specialists in other cultures closer to their homes.<ref name=Hannerz/> There are two main approaches to urban anthropology: examining the types of cities or examining the social issues within the cities. These two methods are overlapping and dependent of each other. By defining different types of cities, one would use social factors as well as economic and political factors to categorize the cities. By directly looking at the different social issues, one would also be studying how they affect the dynamic of the city.<ref>  cite journal doi=10.1177/1466138110370412  jstor=24047982 title=Chinese consumers: The Romantic reappraisal journal=Ethnography volume=11 issue=3 pages=331–357 year=2010 last1=Griffiths first1=M. B. last2=Chapman first2=M. last3=Christiansen first3=F. url=https://www.academia.edu/7368130</ref>
Ecological anthropology is defined as the study of cultural adaptations to environments.<ref name=Kottak>  cite book last=Kottak first=Conrad Phillip title=Anthropology : appreciating human diversity year=2010 publisher=McGraw-Hill location=New York isbn=978-0-07-811699-5 pages=579–584 edition=14th</ref> The sub-field is also defined as, the study of relationships between a population of humans and their biophysical environment.<ref name=Townsend>  cite book last=Townsend first=Patricia K. title=Environmental anthropology : from pigs to policies year=2009 publisher=Waveland Press location=Prospect Heights, Ill. isbn=978-1-57766-581-6 page=https://archive.org/details/environmentalant0000town/page/104 104 edition=2nd url=https://archive.org/details/environmentalant0000town/page/104</ref> The focus of its research concerns how cultural beliefs and practices helped human populations adapt to their environments, and how their environment across space and time.<ref name=Kottak CP 1999>  cite journal last1=Kottak first1=Conrad P. jstor=683339 title=The New Ecological Anthropology journal=American Anthropologist volume=101 issue=1 pages=23–35 year=1999 doi=10.1525/aa.1999.101.1.23 url=https://www.researchgate.net/publication/227632647 hdl=2027.42/66329</ref> The contemporary perspective of environmental anthropology, and arguably at least the backdrop, if not the focus of most of the ethnographies and cultural fieldworks of today, is political ecology. Many characterize this new perspective as more informed with culture, politics and power, globalization, localized issues, century anthropology and more.<ref name=Pyke G 1984>  cite journal last1=Pyke first1=G H title=Optimal Foraging Theory: A Critical Review journal=Annual Review of Ecology and Systematics volume=15 page=523 year=1984 doi=10.1146/annurev.es.15.110184.002515 url=https://www.researchgate.net/publication/243636746</ref> The focus and data interpretation is often used for arguments for/against or creation of policy, and to prevent corporate exploitation and damage of land. Often, the observer has become an active part of the struggle either directly (organizing, participation) or indirectly (articles, documentaries, books, ethnographies). Such is the case with environmental justice advocate Melissa Checker and her relationship with the people of Hyde Park.<ref name=Checker M 2005>  cite book author=Melissa Checker title=Polluted promises: environmental racism and the search for justice in a southern town url=https://archive.org/details/pollutedpromises0000chec url-access=registration date=August 2005 publisher=NYU Press isbn=978-0-8147-1657-1</ref>
TITULO: Agricultural science SUBTITULO Y PARRAFO: ==Fields or related disciplines==
TITULO: Andorra SUBTITULO Y PARRAFO: ==Foreign relations, defence, and security==
  Main Foreign relations of Andorra
File:Bruxelles rue de la Montagne 10.jpg thumb The embassy of Andorra in Brussels
Andorra does not have its own armed forces,<ref name=cia/> although there is a small ceremonial army. Responsibility for defending the nation rests primarily with France and Spain.<ref>  cite web url=http://www.boe.es/buscar/doc.php?id=BOE-A-1993-16868  title=Documento BOE-A-1993-16868  publisher=BOE.es  date=30 June 1993  accessdate=26 August 2012</ref> However, in case of emergencies or natural disasters, the Sometent (an alarm) is called and all able-bodied men between 21 and 60 of Andorran nationality must serve.<ref>  cite web url=http://turisme.andorralavella.ad/?q=en/node/281  title=El Sometent &#124; Tourism  publisher=Turisme.andorralavella.ad  date=17 May 2011  accessdate=26 August 2012</ref><ref name=recarrega.net>  cite web url=http://recarrega.net/arxius/Decret%20veguers%20Sometent,%20del%2023%20d%27octubre%20de%201984.pdf  title=Decret veguers Sometent, del 23 doctubre de 1984  date=  accessdate=26 August 2012</ref> This is why all Andorrans, and especially the head of each house (usually the eldest able-bodied man of a house) should, by law, keep a rifle, even though the law also states that the police will offer a firearm in case of need.<ref name=recarrega.net/> Andorra is a full member of the United Nations (UN), the Organization for Security and Co-operation in Europe (OSCE), and has a special agreement with the European Union (EU).
TITULO: Agriculture SUBTITULO Y PARRAFO: === Crop cultivation systems ===
File:An example of slash and burn agriculture practice Thailand.jpg thumb left Slash and burn shifting cultivation, Thailand
TITULO: Agriculture SUBTITULO Y PARRAFO: === Livestock production systems ===
  Main Livestock Animal husbandry
  See also List of domesticated animals
"""
#open("datos_01_noduplicado1.csv","r",encoding='utf-8').read() 

CONFIDENCE = '0.5'
SUPPORT = '20'
REQUEST = BASE_URL.format(
    text=urllib.parse.quote_plus(TEXT), 
    confidence=CONFIDENCE, 
    support=SUPPORT
)
HEADERS = {'Accept': 'application/json'}
sparql = SPARQLWrapper("http://dbpedia.org/sparql")
all_urls = []

r = requests.get(url=REQUEST, headers=HEADERS)
response = r.json()
resources = response['Resources']
for res in resources:
    all_urls.append(res['@URI'])
    print(all_urls)

Ваш вывод показывает следующую ошибку

~\Anaconda3\lib\json\decoder.py in raw_decode(self, s, idx)
    353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None
    356         return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Та же программа, но теперь читает из внешнего файла, но с тем же текстом из верхнего раздела ТЕКСТ.

import json
from SPARQLWrapper import SPARQLWrapper, JSON
import requests
import urllib.parse
import re

all_urls = []
## initial consts
BASE_URL = 'http://api.dbpedia-spotlight.org/en/annotate?text={text}&confidence={confidence}&support={support}'

i=0

lines_seen = set() # holds lines already seen
for TEXT in open("datos_01_noduplicado1.csv","r",encoding='utf-8'):
    i=i+1
    print("El número de la línea es : " + str(i))
    CONFIDENCE = '0.5'
    SUPPORT = '10'
    REQUEST = BASE_URL.format(
        text=urllib.parse.quote_plus(TEXT), 
        confidence=CONFIDENCE, 
        support=SUPPORT
    )
    HEADERS = {'Accept': 'application/json'}
    sparql = SPARQLWrapper("http://dbpedia.org/sparql")


    r = requests.get(url=REQUEST, headers=HEADERS)
    response = r.json()
    resources = response['Resources']
    for res in resources:

        all_urls.append(res['@URI'])
    print(all_urls)

Это также дает ошибку, хотя это другая.

KeyError                                  Traceback (most recent call last)
<ipython-input-39-b5e27ff2b93f> in <module>
     28     r = requests.get(url=REQUEST, headers=HEADERS)
     29     response = r.json()
---> 30     resources = response['Resources']
     31     for res in resources:
     32 

KeyError: 'Resources'

Я программирую в Jupyter Notebook. Спасибо

Answer 1

Проблема с вашим первым фрагментом кода заключается в том, что текст, который вы передаете в качестве параметра для HTTP-вызова, слишком длинный. Если вы распечатаете объект ответа, вы увидите:

<Response [414]>, что соответствует 414 URI Too Long Ссылка

Если вы передадите меньший текст, dbpedia-spotlight сможет аннотировать сущности за вас.

---

Для второго код, который вы поместили, у вас есть две проблемы, первая из них заключается в том, что dbpedia-spotlight может отвечать со статусом 403 после последовательных вызовов службы аннотирования, чтобы проверить, что я предлагаю вам сделать:

r = requests.get(url=REQUEST, headers=HEADERS)

if r.status_code == requests.codes['ok']:
    .
    .
    .

И другая проблема, которая может произойти, заключается в том, что если в запрошенном вами тексте ничего не было аннотировано, в ответе не будет ключа Resources, поэтому для решения этой проблемы я предлагаю вам использовать функцию get() вместо того, чтобы :

resources = response['Resources']

У вас будет:

resources = response.get('Resources', [])

Таким образом, окончательный код для второго вставленного вами фрагмента будет таким:

import json
from SPARQLWrapper import SPARQLWrapper, JSON
import requests
import urllib.parse
import re

all_urls = []
## initial consts
BASE_URL = 'http://api.dbpedia-spotlight.org/en/annotate?text={text}&confidence={confidence}&support={support}'

i=0

lines_seen = set() # holds lines already seen
for TEXT in open("datos_01_noduplicado1.csv","r",encoding='utf-8'):
    i=i+1
    print("El número de la línea es : " + str(i))
    CONFIDENCE = '0.5'
    SUPPORT = '10'
    REQUEST = BASE_URL.format(
        text=urllib.parse.quote_plus(TEXT), 
        confidence=CONFIDENCE, 
        support=SUPPORT
    )
    HEADERS = {'Accept': 'application/json'}
    sparql = SPARQLWrapper("http://dbpedia.org/sparql")

    r = requests.get(url=REQUEST, headers=HEADERS)
    if r.status_code == requests.codes['ok']:
        response = r.json()
        resources = response.get('Resources', [])
        for res in resources:
            all_urls.append(res['@URI'])

        print(all_urls)