Читать json построчно, используя генератор

Question

У меня есть довольно большой файл json и функция, которая читает файл json.

Мне нужно сохранять состояние между вызовами функций (в следующий раз, когда функция будет вызвана, мне не нужно читать файл json из первой строки, вместо этого мне нужна функция, откуда можно забрать он остался), поэтому первое, что мне пришло в голову, был протокол генератора.

Это моя первая попытка, но поведение функции не является желаемым:

def first_task(file_name):
    """"
    Read file_name line by line
    """
    import json
    data = []
    with open(file_name) as f: 
        for line in f:
            try:
                data.append(json.loads(line))
            except ValueError:  # includes simplejson.decoder.JSONDecodeError
                print ('Decoding JSON has failed')
    yield data

Я хотел бы, чтобы first_task("test.json") вызывался в первый раз , чтобы вернуть первую строку из json, затем, когда функция вызывается во второй раз для возврата второй строки с json и т. Д. До достижения EOF.

Образец из json файла:

{"venue":{"venue_name":"Datong High School","lon":0,"lat":0,"venue_id":23779799},"visibility":"public","response":"no","guests":0,"member":{"member_id":120119272,"photo":"http:\/\/photos3.meetupstatic.com\/photos\/member\/b\/2\/b\/c\/thumb_262125756.jpeg","member_name":"Allen Wang"},"rsvp_id":1658733801,"mtime":1489925470960,"event":{"event_name":"Play Intermediate Volleyball","event_id":"jkpwmlywgbmb","time":1491613200000,"event_url":"https:\/\/www.meetup.com\/Taipei-Sports-and-Social-Club\/events\/236786445\/"},"group":{"group_topics":[{"urlkey":"fitness","topic_name":"Fitness"},{"urlkey":"mountain-biking","topic_name":"Mountain Biking"},{"urlkey":"sports","topic_name":"Sports and Recreation"},{"urlkey":"outdoors","topic_name":"Outdoors"},{"urlkey":"fun-times","topic_name":"Fun Times"},{"urlkey":"winter-and-summer-sports","topic_name":"Winter and Summer Sports"},{"urlkey":"adventure","topic_name":"Adventure"},{"urlkey":"water-sports","topic_name":"Water Sports"},{"urlkey":"sports-and-socials","topic_name":"Sports and Socials"},{"urlkey":"hiking","topic_name":"Hiking"},{"urlkey":"excercise","topic_name":"Exercise"},{"urlkey":"recreational-sports","topic_name":"Recreational Sports"}],"group_city":"Taipei","group_country":"tw","group_id":16585312,"group_name":"Taipei Sports and Social Club","group_lon":121.45,"group_urlname":"Taipei-Sports-and-Social-Club","group_lat":25.02}}
{"venue":{"venue_name":"Cafe Vitus","lon":121.54731,"lat":25.052959,"venue_id":19712922},"visibility":"public","response":"no","guests":0,"member":{"member_id":221379606,"photo":"http:\/\/photos2.meetupstatic.com\/photos\/member\/8\/3\/c\/4\/thumb_263973732.jpeg","member_name":"Benita  Syu"},"rsvp_id":1658877353,"mtime":1489925471668,"event":{"event_name":"New Place! Every Saturday night multilingual café","event_id":"hvkmsmywfbhc","time":1490439600000,"event_url":"https:\/\/www.meetup.com\/polyglottw\/events\/238185973\/"},"group":{"group_topics":[{"urlkey":"language","topic_name":"Language & Culture"},{"urlkey":"language-exchange","topic_name":"Language Exchange"},{"urlkey":"chinese-language","topic_name":"Chinese Language"}],"group_city":"Taipei","group_country":"tw","group_id":18743595,"group_name":"Multilingual Cafe Language Exchange","group_lon":121.45,"group_urlname":"polyglottw","group_lat":25.02}}
{"venue":{"venue_name":"Panera Bread","lon":0,"lat":0,"venue_id":24945082},"visibility":"public","response":"yes","guests":0,"member":{"member_id":44748032,"photo":"http:\/\/photos4.meetupstatic.com\/photos\/member\/d\/7\/1\/6\/thumb_64255062.jpeg","member_name":"Valerie"},"rsvp_id":1658877355,"mtime":1489925472035,"event":{"event_name":"Meet & Greet Icebreaker conversations in Morris County","event_id":"236222256","time":1490389200000,"event_url":"https:\/\/www.meetup.com\/Mingle-Around-In-North-Jersey-Single-Events-Adventures\/events\/236222256\/"},"group":{"group_topics":[{"urlkey":"wine","topic_name":"Wine"},{"urlkey":"hiking","topic_name":"Hiking"},{"urlkey":"diningout","topic_name":"Dining Out"},{"urlkey":"marketing","topic_name":"Marketing"},{"urlkey":"newintown","topic_name":"New In Town"},{"urlkey":"socialnetwork","topic_name":"Social Networking"},{"urlkey":"women","topic_name":"Women's Social"},{"urlkey":"outdoors","topic_name":"Outdoors"},{"urlkey":"professional-networking","topic_name":"Professional Networking"},{"urlkey":"adventure","topic_name":"Adventure"},{"urlkey":"singles-30s-50s","topic_name":"Singles 30's-50's"},{"urlkey":"small-business-marketing-strategy","topic_name":"Small Business Marketing Strategy"},{"urlkey":"professional-singles","topic_name":"Single Professionals"},{"urlkey":"dating-and-relationships","topic_name":"Dating and Relationships"},{"urlkey":"singles-40s-50s","topic_name":"Singles 40's - 50's"}],"group_city":"Hackensack","group_country":"us","group_id":17370312,"group_name":"Mingle Around 30s 40s 50s (Single Events & Adventures)","group_lon":-74.05,"group_urlname":"Mingle-Around-In-North-Jersey-Single-Events-Adventures","group_state":"NJ","group_lat":40.89}}

Answer 1

Хотя не совсем понятно, что вам нужно, если вам просто нужны отдельные строки, вместо построения списка вы можете просто выдать каждую строку и затем вызвать генератор с next, как вам нужно, так как это будет только прочитайте файл один раз и предоставьте отдельные строки по мере необходимости, пока он не будет исчерпан:

## contents of example.txt

{"first": "line"}
{"second": "line"}
{"third": "line"}

Построение поставщика линии:

def supply_line(file_name):
    with open(file_name) as fh:
        for line in fh:
            yield json.loads(line)

Когда вы вызываете его с помощью next не забудьте указать значение по умолчанию, чтобы у вас не было обтекания try/except, чтобы поймать исключение StopIteration:

producer = supply_line('file_with_json_lines.txt')

In [7]: next(producer, '')
Out[7]: {'first': 'line'}

In [8]: next(producer, '')
Out[8]: {'second': 'line'}

In [9]: next(producer, '')
Out[9]: {'third': 'line'}

## when the file is done, it will produce a default, which in this case is an empty string
In [10]: next(producer, '')
Out[10]: ''

In [11]: next(producer, '')
Out[11]: ''

Если вам нужно чтобы начать сначала, вы можете вызвать генератор снова, если вы хотите сохранить все строки в списке, вы можете привести генератор к списку (, однако это будет загружать файл каждый раз ):

In [13]: all_lines = list(supply_line('file_with_json_lines.txt'))

In [14]: all_lines
Out[14]: [{'first': 'line'}, {'second': 'line'}, {'third': 'line'}]

И, очевидно, for l oop:

In [15]: for line in supply_line('file_with_json_lines.txt'):
    ...:     print(line)
    ...:
{'first': 'line'}
{'second': 'line'}
{'third': 'line'}

Answer 2

Это должно работать. Без проверки ошибок.

def gen(file_name):
    with open(file_name) as fh:
        line = fh.readline()
        while line:
            yield json.loads(line)
            line = fh.readline()

В python 3.8 вы можете сделать это так:

def gen(file_name):
    with open(file_name) as fh:
        while line := fh.readline():
            yield json.loads(line)