Как работать между двумя строками?

Question

У меня есть текстовый файл:

<span class="html-tag">&lt;script&gt;</span></td></tr><tr><td class="line-number" value="1431"></td><td class="line-content">            var awbManifests = {"requestId":"16d1-4451-9b12-f61a87e9cd11","errorMessage":null,"errorCode":null,"success":true,"content":[{"id":"5ec8-444e-9d5b-f7487ce592c2","storeId":"10001","createdDate":1541923869937,"createdBy":"asdf","updatedDate":1541968417296,"updatedBy":"dsa","type":"airwaybill","value":"5468468464568466","logisticTrackingID":"5468468464568466","senderName":"dasdf","senderAddress":"Batuceper","receiverName":"ATIK","receiverAddress":"JL. SRIKATON BARAT\n","manifestList":[{"logisticProviderCode":"asd","blibliAirwayBillNumber":"5468468464568466","status":"DEPARTED FROM TRANSIT [GATEWAY JAKARTA]","timestamp":1541976677000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"RECEIVED AT ORIGIN GATEWAY [GATEWAY JAKARTA]","timestamp":1541976343000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"PROCESSED AT SORTING CENTER [JAKARTA]","timestamp":1541968348000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"RECEIVED AT SORTING CENTER [JAKARTA]","timestamp":1541960930000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"SHIPMENT RECEIVED BY asdf COUNTER OFFICER AT [JAKARTA]","timestamp":1541926728000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]}]}],"pageMetaData":null};</td></tr><tr><td class="line-number" value="1432"></td><td class="line-content">            var ordersTracking = [{"orderItemId":"53000116530","product":null,"shipment":"asdf","airwaybillNumber":"5468468464568466","receiver":null,"receivedDate":null,"relation":null,"status":"Valid","productType":"Regular","eligibleForFeedback":false,"feedback":null,"invalidAWBJiraNumber":"","mismatchAWBJiraNumber":"","isAirwayBillValid":true,"mismatchAirwayBill":false}];

И я хочу получить результат от var awbManifests = до первого знака ;, поэтому вывод должен быть только в формате JSON, например:

{"requestId":"16d1-4451-9b12-f61a87e9cd11","errorMessage":null,"errorCode":null,"success":true,"content":[{"id":"5ec8-444e-9d5b-f7487ce592c2","storeId":"10001","createdDate":1541923869937,"createdBy":"asdf","updatedDate":1541968417296,"updatedBy":"dsa","type":"airwaybill","value":"5468468464568466","logisticTrackingID":"5468468464568466","senderName":"dasdf","senderAddress":"Batuceper","receiverName":"ATIK","receiverAddress":"JL. SRIKATON BARAT\n","manifestList":[{"logisticProviderCode":"asd","blibliAirwayBillNumber":"5468468464568466","status":"DEPARTED FROM TRANSIT [GATEWAY JAKARTA]","timestamp":1541976677000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"RECEIVED AT ORIGIN GATEWAY [GATEWAY JAKARTA]","timestamp":1541976343000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"PROCESSED AT SORTING CENTER [JAKARTA]","timestamp":1541968348000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"RECEIVED AT SORTING CENTER [JAKARTA]","timestamp":1541960930000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"SHIPMENT RECEIVED BY asdf COUNTER OFFICER AT [JAKARTA]","timestamp":1541926728000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]}]}],"pageMetaData":null}

Пока что я могу сделать только это, но эта команда не отображает все строки json:

grep -o -P '(?<=var awbManifests = ).*(?=pageMetaData)' test.html

Как мне это исправить?

Answer 1

С помощью Perl вы также можете извлечь их с помощью жадного квантификатора

> cat regex_jsoon.dat
<span class="html-tag">&lt;script&gt;</span></td></tr><tr><td class="line-number" value="1431"></td><td class="line-content">            var awbManifests = {"requestId":"16d1-4451-9b12-f61a87e9cd11","errorMessage":null,"errorCode":null,"success":true,"content":[{"id":"5ec8-444e-9d5b-f7487ce592c2","storeId":"10001","createdDate":1541923869937,"createdBy":"asdf","updatedDate":1541968417296,"updatedBy":"dsa","type":"airwaybill","value":"5468468464568466","logisticTrackingID":"5468468464568466","senderName":"dasdf","senderAddress":"Batuceper","receiverName":"ATIK","receiverAddress":"JL. SRIKATON BARAT\n","manifestList":[{"logisticProviderCode":"asd","blibliAirwayBillNumber":"5468468464568466","status":"DEPARTED FROM TRANSIT [GATEWAY JAKARTA]","timestamp":1541976677000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"RECEIVED AT ORIGIN GATEWAY [GATEWAY JAKARTA]","timestamp":1541976343000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"PROCESSED AT SORTING CENTER [JAKARTA]","timestamp":1541968348000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"RECEIVED AT SORTING CENTER [JAKARTA]","timestamp":1541960930000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"SHIPMENT RECEIVED BY asdf COUNTER OFFICER AT [JAKARTA]","timestamp":1541926728000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]}]}],"pageMetaData":null};</td></tr><tr><td class="line-number" value="1432"></td><td class="line-content">            var ordersTracking = [{"orderItemId":"53000116530","product":null,"shipment":"asdf","airwaybillNumber":"5468468464568466","receiver":null,"receivedDate":null,"relation":null,"status":"Valid","productType":"Regular","eligibleForFeedback":false,"feedback":null,"invalidAWBJiraNumber":"","mismatchAWBJiraNumber":"","isAirwayBillValid":true,"mismatchAirwayBill":false}];

> perl -ne ' { s/.*var awbManifests = (.*?);.*/\1/g; print } ' regex_jsoon.dat
{"requestId":"16d1-4451-9b12-f61a87e9cd11","errorMessage":null,"errorCode":null,"success":true,"content":[{"id":"5ec8-444e-9d5b-f7487ce592c2","storeId":"10001","createdDate":1541923869937,"createdBy":"asdf","updatedDate":1541968417296,"updatedBy":"dsa","type":"airwaybill","value":"5468468464568466","logisticTrackingID":"5468468464568466","senderName":"dasdf","senderAddress":"Batuceper","receiverName":"ATIK","receiverAddress":"JL. SRIKATON BARAT\n","manifestList":[{"logisticProviderCode":"asd","blibliAirwayBillNumber":"5468468464568466","status":"DEPARTED FROM TRANSIT [GATEWAY JAKARTA]","timestamp":1541976677000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"RECEIVED AT ORIGIN GATEWAY [GATEWAY JAKARTA]","timestamp":1541976343000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"PROCESSED AT SORTING CENTER [JAKARTA]","timestamp":1541968348000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"RECEIVED AT SORTING CENTER [JAKARTA]","timestamp":1541960930000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"SHIPMENT RECEIVED BY asdf COUNTER OFFICER AT [JAKARTA]","timestamp":1541926728000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]}]}],"pageMetaData":null}

>

или еще одна версия без подстановки.

> perl -ne ' { print "$x\n"  if /var awbManifests = (.*?);/osg  and $x=$1 } ' regex_jsoon.dat
{"requestId":"16d1-4451-9b12-f61a87e9cd11","errorMessage":null,"errorCode":null,"success":true,"content":[{"id":"5ec8-444e-9d5b-f7487ce592c2","storeId":"10001","createdDate":1541923869937,"createdBy":"asdf","updatedDate":1541968417296,"updatedBy":"dsa","type":"airwaybill","value":"5468468464568466","logisticTrackingID":"5468468464568466","senderName":"dasdf","senderAddress":"Batuceper","receiverName":"ATIK","receiverAddress":"JL. SRIKATON BARAT\n","manifestList":[{"logisticProviderCode":"asd","blibliAirwayBillNumber":"5468468464568466","status":"DEPARTED FROM TRANSIT [GATEWAY JAKARTA]","timestamp":1541976677000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"RECEIVED AT ORIGIN GATEWAY [GATEWAY JAKARTA]","timestamp":1541976343000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"PROCESSED AT SORTING CENTER [JAKARTA]","timestamp":1541968348000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"RECEIVED AT SORTING CENTER [JAKARTA]","timestamp":1541960930000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]},{"logisticProviderCode":"asdf","blibliAirwayBillNumber":"5468468464568466","status":"SHIPMENT RECEIVED BY asdf COUNTER OFFICER AT [JAKARTA]","timestamp":1541926728000,"additionalInfo":[{"label":"Third Party Tracking ID","value":null,"type":"STRING","description":"Third Party Tracking ID"}]}]}],"pageMetaData":null}
>

Answer 2

Мне понадобилось время, чтобы понять, что происходит. Это интересно, потому что вы используете lookbehinds ((?<=)) и lookaheads в своем регулярном выражении, и это очень полезные конструкции, которые я почти никогда не использую.

Принцип lookahead / lookbehind заключается в том, что строка внутри группы look * совпадает, но отсутствует в совпадающей строке. Это очень полезно с grep -o. Взгляд назад (?<=var awbManifests = ) используется правильно, но взгляд вперед (?=pageMetaData) должен быть (?=;). Но теперь проблема, с которой вы сталкиваетесь, состоит в том, что регулярное выражение соответствует слишком большому количеству текста.

По умолчанию квантификаторы регулярных выражений +, * и {n,m} являются жадными, это означает, что они пытаются сопоставить как можно больше текста. В режиме perl (-P) в grep доступны не жадные квантификаторы. Их синтаксис: +? для +, *? для * и {n,m}? для {n,m}.

Здесь применяется регулярное выражение:

grep -o -P '(?<=var awbManifests = ).*?(?=;)' test.html

Но есть еще одна проблема: если одна из строк JSON содержит ;, JSON не будет полностью согласован. Чтобы учесть строки, используйте вместо этого:

grep -o -P '(?<=var awbManifests = )[^"]*("([^"]|\\")*"[^"]*?)*?(?=;)' test.html

В приведенном выше регулярном выражении:

• [^"] соответствует чему угодно, кроме двойной кавычки.
• \\ обратные слэши необходимо экранировать, поскольку они имеют значение в регулярных выражениях.
• "([^"]|\\")*" соответствует одной строке с учетом экранированных кавычек.
• [^"]*("([^"]|\\")*"[^"]*)* соответствует нескольким строкам с текстом до и после него. Этот шаблон соответствует полному JSON, но является жадным.
• [^"]*("([^"]|\\")*"[^"]*?)*? соответствует JSON без жадности.