Исключение при потоковой передаче твитов https://stream.twitter.com/1/statuses/sample.json - PullRequest
Новая
       34

Исключение при потоковой передаче твитов https://stream.twitter.com/1/statuses/sample.json

0 голосов
/ 10 марта 2020

Я новичок в apache flume и получаю следующее сообщение об ошибке, пытаясь захватить твиты в реальном времени с использованием flume 

2020-03-10 20:29:36,660 (Twitter Stream consumer-1[Waiting for 250 milliseconds]) [ERROR - org.apache.flume.source.twitter.TwitterSource.onException(TwitterSource.java:326)] Exception while streaming tweets
https://stream.twitter.com/1/statuses/sample.jsonRelevant discussions can be found on the Internet at:
        http://www.google.co.jp/search?q=ec814753 or
        http://www.google.co.jp/search?q=0a74ccb0
TwitterException{exceptionCode=[ec814753-0a74ccb0 4eaddaa2-4424c519 4eaddaa2-4424c517], statusCode=-1, retryAfter=-1, rateLimitStatus=null, featureSpecificRateLimitStatus=null, version=2.2.6}
        at twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:200)
        at twitter4j.internal.http.HttpClientWrapper.request(HttpClientWrapper.java:65)
        at twitter4j.internal.http.HttpClientWrapper.get(HttpClientWrapper.java:93)
        at twitter4j.TwitterStreamImpl.getSampleStream(TwitterStreamImpl.java:160)
        at twitter4j.TwitterStreamImpl$4.getStream(TwitterStreamImpl.java:149)
        at twitter4j.TwitterStreamImpl$4.getStream(TwitterStreamImpl.java:147)
        at twitter4j.TwitterStreamImpl$TwitterStreamConsumer.run(TwitterStreamImpl.java:426)
Caused by: java.io.FileNotFoundException: https://stream.twitter.com/1/statuses/sample.json
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1950)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1945)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1944)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1514)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
        at twitter4j.internal.http.HttpResponseImpl.<init>(HttpResponseImpl.java:37)
        at twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:164)
        ... 6 more
Caused by: java.io.FileNotFoundException: https://stream.twitter.com/1/statuses/sample.json
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
        at twitter4j.internal.http.HttpResponseImpl.<init>(HttpResponseImpl.java:35)
        ... 7 more

flume.conf file 

TwitterAgent.sources = Twitter 
TwitterAgent.channels = MemChannel 
TwitterAgent.sinks = HDFS

# Describing/Configuring the source 
TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.consumerKey=XXXXXXXXXXXXXXXXXX
TwitterAgent.sources.Twitter.consumerSecret=XXXXXXXXXXXXXXXXXXXXX
TwitterAgent.sources.Twitter.accessToken=XXXXXXXXXXXXXXX
TwitterAgent.sources.Twitter.accessTokenSecret=XXXXXXXXXXXXXXXXXXXXXXXXX
TwitterAgent.sources.Twitter.keywords=sports,cricket,tennis,football
# Describing/Configuring the sink 

TwitterAgent.sources.Twitter.keywords= hadoop,election,sports, cricket,Big data

TwitterAgent.sinks.HDFS.channel=MemChannel
TwitterAgent.sinks.HDFS.type=hdfs
TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:9000/user/Hadoop/twitter_data
TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream
TwitterAgent.sinks.HDFS.hdfs.writeformat=Text
TwitterAgent.sinks.HDFS.hdfs.batchSize=10
TwitterAgent.sinks.HDFS.hdfs.rollSize=0
TwitterAgent.sinks.HDFS.hdfs.rollCount=10000
TwitterAgent.sinks.HDFS.hdfs.rollInterval=600

TwitterAgent.sinks.HDFS.hdfs.userLocalTimeStamp = true

TwitterAgent.channels.MemChannel.type=memory
TwitterAgent.channels.MemChannel.capacity=10000
TwitterAgent.channels.MemChannel.transactionCapacity=1000

TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sinks.HDFS.channel = MemChannel

Jar-файлы, которые я использовал:

  1. flume-sources-1.0-SNAPSHOT.jar
  2. twitter4j-core-4.0.7.jar
  3. twitter4j-asyn c -4.0.7.jar
  4. twitter4j-stream-4.0.7.jar

Может ли кто-нибудь предоставить решение .. Спасибо.

...