Нужна помощь в создании класса загрузчика хранилища данных GAE?

Question

Нужна помощь в создании класса загрузчика хранилища данных GAE для загрузки данных с помощью appcfg.py?Любой другой способ упростить этот процесс?есть ли подробный пример лучше, чем здесь

При попытке использовать bulkloader.yaml:

Uploading data records.
[INFO    ] Logging to bulkloader-log-20100701.041515
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
[INFO    ] Opening database: bulkloader-progress-20100701.041515.sql3
[INFO    ] Connecting to livelihoodproducer.appspot.com/remote_api
[INFO    ] Starting import; maximum 10 entities per post
[ERROR   ] [Thread-1] WorkerThread:
Traceback (most recent call last):
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/adaptive_thread_pool.py", line 150, in WorkOnItems
    status, instruction = item.PerformWork(self.__thread_pool)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 693, in PerformWork
    transfer_time = self._TransferItem(thread_pool)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 848, in _TransferItem
    self.content = self.request_manager.EncodeContent(self.rows)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1269, in EncodeContent
    entity = loader.create_entity(values, key_name=key, parent=parent)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 385, in create_entity
    return self.dict_to_entity(input_dict, self.bulkload_state)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 133, in dict_to_entity
    self.__run_import_transforms(input_dict, instance, bulkload_state_copy)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 233, in __run_import_transforms
    value = self.__dict_to_prop(transform, input_dict, bulkload_state)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 188, in __dict_to_prop
    value = transform.import_transform(value)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/bulkloader_parser.py", line 93, in __call__
    return self.method(*args, **kwargs)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/bulkload/transform.py", line 143, in generate_foreign_key_lambda
    return datastore.Key.from_path(kind, value)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore_types.py", line 387, in from_path
    'received %r (a %s).' % (i + 2, id_or_name, typename(id_or_name)))
BadArgumentError: Expected an integer id or string name as argument 2; received None (a NoneType).
[INFO    ] [Thread-3] Backing off due to errors: 1.0 seconds
[INFO    ] Unexpected thread death: Thread-1
[INFO    ] An error occurred. Shutting down...
[ERROR   ] Error in Thread-1: Expected an integer id or string name as argument 2; received None (a NoneType).

[INFO    ] 30 entites total, 0 previously transferred
[INFO    ] 0 entities (733 bytes) transferred in 2.8 seconds
[INFO    ] Some entities not successfully transferred

В процессе я загрузил данные CSV, вручную вставленные в appspot.com.Пока я пытаюсь загрузить свои собственные данные CSV, порядок столбцов должен быть таким же, как CSV, загруженный с appspot.com?как насчет пустого значения?

Answer 1

Я создал config.yaml с конфигурацией массового загрузчика, а также написал простую вспомогательную функцию для обработки None-ссылок. Я не знаю, почему это не сделано в оригинальном помощнике.

Помощник (файл helpers.py очень прост, просто поместите его в тот же каталог, куда вы поместили config.yaml):

from google.appengine.api import datastore
def create_foreign_key(kind, key_is_id=False):
  def generate_foreign_key_lambda(value):
    if value is None:
      return None

    if key_is_id:
      value = int(value)
    return datastore.Key.from_path(kind, value)

  return generate_foreign_key_lambda

И это вырезано из моего config.yaml:

python_preamble:
- import: helpers # this will import our helper
[other imports]
...
- kind: ArticleComment
  connector: simplexml
  connector_options:
    xpath_to_nodes: "/blog/Comments/Comment"
    style: element_centric

  property_map:
    - property: __key__
      external_name: key
      export_transform: transform.key_id_or_name_as_string

    - property: parent_comment
      external_name: parent-comment
      export_transform: transform.key_id_or_name_as_string
      import_transform: helpers.create_foreign_key('ArticleComment')
      #                 ^^^^^^^ here it is
      #                 use this instead of transform.create_foreign_key

Answer 2

Похоже, у вас есть справочные свойства со значениями None, такие значения неправильно обрабатываются помощниками массового загрузчика.