Я работаю в работе Glue (версия 2.0), используя функцию закладок, во второй раз, когда я запускаю работу без изменений в файле, я получаю следующее сообщение об ошибке:
AnalysisException: '\nDatasource does not support writing empty or nested empty schemas.\nPlease make sure the data schema has at least one or more column(s).\n ;'
Это сгенерированный скрипт консолью AWS без каких-либо модификаций, источником являются файлы S3, использующие каталог данных, а выводом является другое ведро.
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
## @type: DataSource
## @args: [database = "raw_dev_edocs", table_name = "esocial_s2200", transformation_ctx = "datasource0"]
## @return: datasource0
## @inputs: []
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "raw_dev_edocs", table_name = "esocial_s2200", transformation_ctx = "datasource0")
## @type: ApplyMapping
## @args: [mapping = [("esocial", "struct", "esocial", "struct"), ("tenant", "string", "tenant", "string"), ("year", "string", "year", "string"), ("month", "string", "month", "string"), ("day", "string", "day", "string")], transformation_ctx = "applymapping1"]
## @return: applymapping1
## @inputs: [frame = datasource0]
applymapping1 = ApplyMapping.apply(frame = datasource0, mappings = [("esocial", "struct", "esocial", "struct"), ("tenant", "string", "tenant", "string"), ("year", "string", "year", "string"), ("month", "string", "month", "string"), ("day", "string", "day", "string")], transformation_ctx = "applymapping1")
## @type: ResolveChoice
## @args: [choice = "make_struct", transformation_ctx = "resolvechoice2"]
## @return: resolvechoice2
## @inputs: [frame = applymapping1]
resolvechoice2 = ResolveChoice.apply(frame = applymapping1, choice = "make_struct", transformation_ctx = "resolvechoice2")
## @type: DropNullFields
## @args: [transformation_ctx = "dropnullfields3"]
## @return: dropnullfields3
## @inputs: [frame = resolvechoice2]
dropnullfields3 = DropNullFields.apply(frame = resolvechoice2, transformation_ctx = "dropnullfields3")
## @type: DataSink
## @args: [connection_type = "s3", connection_options = {"path": "s3://senior-datalake-stage/dev/edocs/S2200test"}, format = "parquet", transformation_ctx = "datasink4"]
## @return: datasink4
## @inputs: [frame = dropnullfields3]
datasink4 = glueContext.write_dynamic_frame.from_options(frame = dropnullfields3, connection_type = "s3", connection_options = {"path": "s3://senior-datalake-stage/dev/edocs/S2200test"}, format = "parquet", transformation_ctx = "datasink4")
job.commit()
Как упомянуто в этом do c (https://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html) закладка отслеживается с использованием метки времени изменения, мой сценарий такой же, как в примере документации.
Когда я сбрасываю закладку, задание выполняется успешно.
Схема таблицы, из которой Glue читает:
Column name
Data type
Key
Comment
1
tenant
string
Partition (0)
2
year
string
Partition (1)
3
month
string
Partition (2)
4
day
string
Partition (3)
5
esocial.xmlns
string
6
esocial.evtadmissao.id
string
7
esocial.evtadmissao.ideevento.indretif
string
8
esocial.evtadmissao.ideevento.tpamb
string
9
esocial.evtadmissao.ideevento.procemi
string
10
esocial.evtadmissao.ideevento.verproc
string
11
esocial.evtadmissao.ideevento.nrrecibo
string
12
esocial.evtadmissao.ideempregador.tpinsc
string
13
esocial.evtadmissao.ideempregador.nrinsc
string
14
esocial.evtadmissao.trabalhador.cpftrab
string
15
esocial.evtadmissao.trabalhador.nistrab
string
16
esocial.evtadmissao.trabalhador.nmtrab
string
17
esocial.evtadmissao.trabalhador.sexo
string
18
esocial.evtadmissao.trabalhador.racacor
string
19
esocial.evtadmissao.trabalhador.estciv
string
20
esocial.evtadmissao.trabalhador.grauinstr
string
21
esocial.evtadmissao.trabalhador.nascimento.dtnascto
string
22
esocial.evtadmissao.trabalhador.nascimento.codmunic
string
23
esocial.evtadmissao.trabalhador.nascimento.uf
string
24
esocial.evtadmissao.trabalhador.nascimento.paisnascto
string
25
esocial.evtadmissao.trabalhador.nascimento.paisnac
string
26
esocial.evtadmissao.trabalhador.nascimento.nmmae
string
27
esocial.evtadmissao.trabalhador.nascimento.nmpai
string
28
esocial.evtadmissao.trabalhador.documentos.ctps.nrctps
string
29
esocial.evtadmissao.trabalhador.documentos.ctps.seriectps
string
30
esocial.evtadmissao.trabalhador.documentos.ctps.ufctps
string
31
esocial.evtadmissao.trabalhador.documentos.rg.nrrg
string
32
esocial.evtadmissao.trabalhador.documentos.rg.orgaoemissor
string
33
esocial.evtadmissao.trabalhador.documentos.rg.dtexped
string
34
esocial.evtadmissao.trabalhador.documentos.rne.nrrne
string
35
esocial.evtadmissao.trabalhador.documentos.rne.orgaoemissor
string
36
esocial.evtadmissao.trabalhador.documentos.rne.dtexped
string
37
esocial.evtadmissao.trabalhador.endereco.brasil.tplograd
string
38
esocial.evtadmissao.trabalhador.endereco.brasil.dsclograd
string
39
esocial.evtadmissao.trabalhador.endereco.brasil.nrlograd
string
40
esocial.evtadmissao.trabalhador.endereco.brasil.bairro
string
41
esocial.evtadmissao.trabalhador.endereco.brasil.cep
string
42
esocial.evtadmissao.trabalhador.endereco.brasil.codmunic
string
43
esocial.evtadmissao.trabalhador.endereco.brasil.uf
string
44
esocial.evtadmissao.trabalhador.endereco.brasil.complemento
string
45
esocial.evtadmissao.trabalhador.infodeficiencia.deffisica
string
46
esocial.evtadmissao.trabalhador.infodeficiencia.defvisual
string
47
esocial.evtadmissao.trabalhador.infodeficiencia.defauditiva
string
48
esocial.evtadmissao.trabalhador.infodeficiencia.defmental
string
49
esocial.evtadmissao.trabalhador.infodeficiencia.defintelectual
string
50
esocial.evtadmissao.trabalhador.infodeficiencia.reabreadap
string
51
esocial.evtadmissao.trabalhador.infodeficiencia.infocota
string
52
esocial.evtadmissao.trabalhador.contato.foneprinc
string
53
esocial.evtadmissao.trabalhador.contato.fonealternat
string
54
esocial.evtadmissao.trabalhador.contato.emailprinc
string
55
esocial.evtadmissao.trabalhador.contato.emailalternat
string
56
esocial.evtadmissao.trabalhador.dependente.struct.tpdep
string
57
esocial.evtadmissao.trabalhador.dependente.struct.nmdep
string
58
esocial.evtadmissao.trabalhador.dependente.struct.dtnascto
string
59
esocial.evtadmissao.trabalhador.dependente.struct.cpfdep
string
60
esocial.evtadmissao.trabalhador.dependente.struct.depirrf
string
61
esocial.evtadmissao.trabalhador.dependente.struct.depsf
string
62
esocial.evtadmissao.trabalhador.dependente.struct.inctrab
string
63
esocial.evtadmissao.trabalhador.dependente.array
bigint
64
esocial.evtadmissao.trabalhador.indpriempr
string
65
esocial.evtadmissao.trabalhador.trabestrangeiro.dtchegada
string
66
esocial.evtadmissao.trabalhador.trabestrangeiro.classtrabestrang
string
67
esocial.evtadmissao.trabalhador.trabestrangeiro.casadobr
string
68
esocial.evtadmissao.trabalhador.trabestrangeiro.filhosbr
string
69
esocial.evtadmissao.vinculo.matricula
string
70
esocial.evtadmissao.vinculo.tpregtrab
string
71
esocial.evtadmissao.vinculo.tpregprev
string
72
esocial.evtadmissao.vinculo.cadini
string
73
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.dtadm
string
74
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.tpadmissao
string
75
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.indadmissao
string
76
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.tpregjor
string
77
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.natatividade
string
78
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.dtbase
string
79
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.cnpjsindcategprof
string
80
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.fgts.opcfgts
string
81
esocial.evtadmissao.vinculo.inforegimetrab.infoceletista.fgts.dtopcfgts
string
82
esocial.evtadmissao.vinculo.infocontrato.codcargo
string
83
esocial.evtadmissao.vinculo.infocontrato.codcateg
string
84
esocial.evtadmissao.vinculo.infocontrato.remuneracao.vrsalfx
string
85
esocial.evtadmissao.vinculo.infocontrato.remuneracao.undsalfixo
string
86
esocial.evtadmissao.vinculo.infocontrato.duracao.tpcontr
string
87
esocial.evtadmissao.vinculo.infocontrato.duracao.dtterm
string
88
esocial.evtadmissao.vinculo.infocontrato.duracao.clauassec
string
89
esocial.evtadmissao.vinculo.infocontrato.localtrabalho.localtrabgeral.tpinsc
string
90
esocial.evtadmissao.vinculo.infocontrato.localtrabalho.localtrabgeral.nrinsc
string
91
esocial.evtadmissao.vinculo.infocontrato.horcontratual.qtdhrssem
string
92
esocial.evtadmissao.vinculo.infocontrato.horcontratual.tpjornada
string
93
esocial.evtadmissao.vinculo.infocontrato.horcontratual.tmpparc
string
94
esocial.evtadmissao.vinculo.infocontrato.horcontratual.horario
bigint
95
esocial.evtadmissao.vinculo.infocontrato.filiacaosindical.cnpjsindtrab
string
96
esocial.evtadmissao.vinculo.infocontrato.observacoes.observacao
string
97
esocial.evtadmissao.vinculo.afastamento.dtiniafast
string
98
esocial.evtadmissao.vinculo.afastamento.codmotafast
string
Кто-нибудь знает, как я могу решить эту проблему?