Извлечение вложенного объекта с помощью Logstash и Ruby - PullRequest
Новая
       6

Извлечение вложенного объекта с помощью Logstash и Ruby

0 голосов
/ 20 сентября 2018

У меня есть такая структура XML: 

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE us-patent-application SYSTEM "us-patent-application-v44-2014-04-03.dtd" [ ]>
<us-patent-application lang="EN" dtd-version="v4.4 2014-04-03" file="US20180000001A1-20180104.XML" status="PRODUCTION" id="us-patent-application" country="US" date-produced="20171219" date-publ="20180104">
    <us-bibliographic-data-application lang="EN" country="US">
        <us-parties>
            <inventors>
                <inventor sequence="00" designation="us-only">
                    <addressbook>
                        <last-name>Evans</last-name>
                        <first-name>Mike</first-name>
                        <address>
                            <city>Emerald Park</city>
                            <country>CA</country>
                        </address>
                    </addressbook>
                </inventor>
                <inventor sequence="01" designation="us-only">
                    <addressbook>
                        <last-name>Lucas</last-name>
                        <first-name>Lisa</first-name>
                        <address>
                            <city>Regina</city>
                            <country>CA</country>
                        </address>
                    </addressbook>
                </inventor>
                <inventor sequence="02" designation="us-only">
                    <addressbook>
                        <last-name>Smith</last-name>
                        <first-name>John R.</first-name>
                        <address>
                            <city>Regina</city>
                            <country>CA</country>
                        </address>
                    </addressbook>
                </inventor>
            </inventors>
        </us-parties>
    </us-bibliographic-data-application>
</us-patent-application>

Я бы хотел, чтобы Logstash вывел эту структуру: 

{
    "us-patent-application": {
        "us-bibliographic-data-application": {
            "us-parties": {
                "inventors": [
                    "Mike Evans",
                    "Lisa Lucas",
                    "John R. Smith"
                ]
            }
        }
    }
}

Я попытался решить эту «комбинацию» имен в одном массивев Logstash, но я не могу найти рабочее решение.

На данный момент я сосредоточен на использовании сценария ruby ​​в плагине фильтра Ruby Logstash.Я использую этот подход, потому что мне не удалось найти работающее решение с использованием Xpath в XML-фильтре Logstash.

Вот файл конфигурации Logstash 'main.conf': 

input {

    file {
        path => [
            "/opt/uspto/*.xml"
            ]
        start_position => "beginning"
        #use for testing
        sincedb_path => "/dev/null"
        # set this sincedb path when not testing
        #sincedb_path => "/opt/logstash/tmp/sincedb"
        exclude => "*.gz"
        type => "xml"
        codec => multiline {
             #pattern => "<wo-ocr-published-application"
             pattern => "<?xml version=\"1.0\" encoding=\"UTF-8\"\?>"
             negate => "true"
             what => "previous"
             max_lines => 300000
            }
    }
}

filter {

    if "multiline" in [tags] {

        xml {
            source => "message"
            #store_xml => false # this limits the data indexed to only xpath and grok created fields
            store_xml => true #saves ALL xml nodes if it can - can be VERY large
            target => "xmldata" # only used with store_xml => true
        }


        ruby {
            path => "/etc/logstash/rubyscripts/inventors.rb"
          }

    }
}

output {

    file {
        path => [ "/tmp/logstash_output_text_file" ]
        codec => rubydebug
    }
}

А вот сценарий изобретателей.rb: 

# the value of `params` is the value of the hash passed to `script_params`
# in the logstash configuration
def register(params)
        @drop_percentage = params["percentage"]
end

def filter(event)
        # get the number of inventors to loop over
        # convert the array key string number 0 to an integer
        n = event.get('[num_inventors][0]').to_i
        # set a loop number to start with
        i = 0
        #create empty arrays to fill
        firstname = []
        lastname = []
        # loop over inventors until n is reached
        while (i < n) do
                #get the inventors first name
                fname = event.get('[event][us-patent-application][us-bibliographic-data-application][us-parties][inventors][inventor][addressbook][last-name]')
                #puts"first name #{fname}"
                # push the first name into firstname array
                firstname.push(fname)
                #get the inventors last name
                lname = event.get('[event][us-patent-application][us-bibliographic-data-application][us-parties][inventors][inventor][addressbook][last-name]')
                #puts"last name #{lname}"
                # push the last name into firstname array
                lastname.push(lname)
                #increment n up 1
                i += 1
        end
        #merge firstname and lastname arrays
        names = firstname.zip(lastname)
        # push the names array to the event
        event.set('allnames', names)
        return [event]
end

Наконец, вотВывод Elasticsearch: 

{
          "host" => "localhost.localdomain",
      "allnames" => [],
          "type" => "xml",
    "@timestamp" => 2018-09-20T17:28:05.332Z,
       "message" => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<!DOCTYPE us-patent-application SYSTEM \"us-patent-application-v44-2014-04-03.dtd\" [ ]>\r\n<us-patent-application lang=\"EN\" dtd-version=\"v4.4 2014-04-03\" file=\"US20180000001A1-20180104.XML\" status=\"PRODUCTION\" id=\"us-patent-application\" country=\"US\" date-produced=\"20171219\" date-publ=\"20180104\">\r\n\t<us-bibliographic-data-application lang=\"EN\" country=\"US\">\r\n\t\t<us-parties>\r\n\t\t\t<inventors>\r\n\t\t\t\t<inventor sequence=\"00\" designation=\"us-only\">\r\n\t\t\t\t\t<addressbook>\r\n\t\t\t\t\t\t<last-name>Evans</last-name>\r\n\t\t\t\t\t\t<first-name>Mike</first-name>\r\n\t\t\t\t\t\t<address>\r\n\t\t\t\t\t\t\t<city>Emerald Park</city>\r\n\t\t\t\t\t\t\t<country>CA</country>\r\n\t\t\t\t\t\t</address>\r\n\t\t\t\t\t</addressbook>\r\n\t\t\t\t</inventor>\r\n\t\t\t\t<inventor sequence=\"01\" designation=\"us-only\">\r\n\t\t\t\t\t<addressbook>\r\n\t\t\t\t\t\t<last-name>Lucas</last-name>\r\n\t\t\t\t\t\t<first-name>Lisa</first-name>\r\n\t\t\t\t\t\t<address>\r\n\t\t\t\t\t\t\t<city>Regina</city>\r\n\t\t\t\t\t\t\t<country>CA</country>\r\n\t\t\t\t\t\t</address>\r\n\t\t\t\t\t</addressbook>\r\n\t\t\t\t</inventor>\r\n\t\t\t\t<inventor sequence=\"02\" designation=\"us-only\">\r\n\t\t\t\t\t<addressbook>\r\n\t\t\t\t\t\t<last-name>Smith</last-name>\r\n\t\t\t\t\t\t<first-name>Scott R.</first-name>\r\n\t\t\t\t\t\t<address>\r\n\t\t\t\t\t\t\t<city>Regina</city>\r\n\t\t\t\t\t\t\t<country>CA</country>\r\n\t\t\t\t\t\t</address>\r\n\t\t\t\t\t</addressbook>\r\n\t\t\t\t</inventor>\r\n\t\t\t</inventors>\r\n\t\t</us-parties>\r\n\t</us-bibliographic-data-application>\r",
      "@version" => "1",
          "tags" => [
        [0] "multiline",
        [1] "_xmlparsefailure"
    ],
          "path" => "/opt/uspto/test.xml"
}

Я ищу поведение массива "allnames" => []: 

"allnames" => ["Mike Evans",
               "Lisa Lucas",
               "John R. Smith"],

Не могу понять, как правильно захватитьузлы «имя» и «фамилия» в моем скрипте ruby.Я дергаю себя за волосы, пытаясь найти рабочий раствор.Любые идеи приветствуются!

1 Ответ

0 голосов
/ 21 сентября 2018

Пожалуйста, посмотрите, решает ли это ваше требование.Я очень надеялся, что это можно решить с помощью xml + xpath.Но я думаю, что все функции xpath не поддерживаются.:( 

input {

    file {
        path => [
            "/opt/uspto/*.xml"
            ]
        start_position => "beginning"
        #use for testing
        sincedb_path => "/dev/null"
        # set this sincedb path when not testing
        #sincedb_path => "/opt/logstash/tmp/sincedb"
        exclude => "*.gz"
        type => "xml"
        codec => multiline {
             #pattern => "<wo-ocr-published-application"
             pattern => "<?xml version=\"1.0\" encoding=\"UTF-8\"\?>"
             negate => "true"
             what => "previous"
             max_lines => 300000
            }
    }
}

filter {

    if "multiline" in [tags] {

        xml {
            source => "message"
            store_xml => false  
            target => "xmldata" # only used with store_xml => true
            force_array=> false
            xpath => [ "//us-bibliographic-data-application/us-parties/inventors/inventor/addressbook/first-name/text()" , "[xmldata][us-bibliographic-data-application][us-parties][inventors][first_name]" , 
                       "//us-bibliographic-data-application/us-parties/inventors/inventor/addressbook/last-name/text()", "[xmldata][us-bibliographic-data-application][us-parties][inventors][last_name]" ]

        }

        ruby {  
           code => ' first_name = event.get("[xmldata][us-bibliographic-data-application][us-parties][inventors][first_name]")
                    last_name = event.get("[xmldata][us-bibliographic-data-application][us-parties][inventors][last_name]")
                    event.set("[xmldata][us-bibliographic-data-application][us-parties][inventors][names]", first_name.zip(last_name).map{ |a| a.join(" ") })
           '
       }

       mutate {
            remove_field => ["message", "host", "path", "[xmldata][us-bibliographic-data-application][us-parties][inventors][first_name]", "[xmldata][us-bibliographic-data-application][us-parties][inventors][last_name]" ]
        }
    }
}

output {

    file {
        path => [ "/tmp/logstash_output_text_file" ]
        codec => rubydebug
    }
}

Вывод 

{
    "@timestamp": "2018-09-21T12:00:26.428Z",
    "@version": "1",
    "tags": [
        "multiline"
    ],
    "type": "xml",
    "xmldata": {
        "us-bibliographic-data-application": {
            "us-parties": {
                "inventors": {
                    "names": [
                        "Mike Evans",
                        "Lisa Lucas",
                        "John R. Smith"
                    ]
                }
            }
        }
    }
}
...