Я пытался запустить кластер Cloudbreak Hortonworks (комбинированный с HDP 3.1, HDF 3.3.1 и Search 4.0).Я следовал примерам чертежей для HA-узлов Hive и HDFS, которые, кажется, работают нормально, но когда я пытаюсь подключить источник аутентификации LDAP (FreeIPA), я даже не могу заставить сервер Ambari установить.Ниже приведены, как мне кажется, соответствующие сообщения об ошибках, связанные с настройкой сервера LDAP и Ambari, но я не вижу, где на стороне сервера Cloudbreak я мог бы изменить эти конфигурации (предположительно, сценарии Salt).
Could not dispatch event: org.apache.ambari.server.events.listeners.hosts.HostUpdateListener@786ff1cb to public void org.apache.ambari.server.events.listeners.hosts.HostUpdateListener.onHostStateUpdate(org.apache.ambari.server.events.HostStateUpdateEvent) throws org.apache.ambari.server.AmbariException
2019-02-07 16:24:15,512 INFO [agent-register-processor-7] HostImpl:343 - Received host registration, host=[hostname=broker-1,fqdn=broker-1.littlemac.punchcyber.com,domain=littlemac.punchcyber.com,architecture=x86_64,processorcount=4,physicalprocessorcount=4,osname=redhat,osversion=7.6,osfamily=redhat,memory=15791384,uptime_hours=0,mounts=(available=47363536,mountpoint=/,used=5052956,percent=10%,size=52416492,device=/dev/nvme0n1p2,type=xfs)(available=15685408,mountpoint=/hadoopfs,used=32992,percent=1%,size=15718400,device=/dev/nvme1n1,type=xfs)]
, registrationTime=1549556655499, agentVersion=2.7.3.0
2019-02-07 16:24:15,513 INFO [agent-register-processor-7] TopologyManager:645 - TopologyManager.onHostRegistered: Entering
2019-02-07 16:24:15,513 INFO [agent-register-processor-7] TopologyManager:706 - TopologyManager: Queueing available host broker-1.littlemac.punchcyber.com
2019-02-07 16:24:15,545 INFO [clientInboundChannel-1] AlertDefinitionsHolder:95 - Loaded 0 alert definitions for 0 clusters for host broker-1.littlemac.punchcyber.com
2019-02-07 16:24:20,024 INFO [ambari-client-thread-38] AmbariLocalAuthenticationProvider:71 - User not found: cloudbreak
2019-02-07 16:24:20,024 INFO [ambari-client-thread-38] AbstractContextSource:415 - Property 'userDn' not set - anonymous context will be used for read-write operations
2019-02-07 16:24:20,024 INFO [ambari-client-thread-38] FilterBasedLdapUserSearch:96 - SearchBase not set. Searches will be performed from the root: dc=ambari,dc=apache,dc=org
2019-02-07 16:24:20,025 WARN [ambari-client-thread-38] AmbariLdapAuthenticationProvider:128 - Failed to communicate with the LDAP server: localhost:33389; nested exception is javax.naming.CommunicationException: localhost:33389 [Root exception is java.net.ConnectException: Connection refused (Connection refused)]
2019-02-07 16:24:20,026 INFO [ambari-client-thread-38] AmbariAuthenticationEventHandlerImpl:136 - Failed to authenticate cloudbreak: The user does not exist in the Ambari database
2019-02-07 16:24:20,028 INFO [ambari-client-thread-34] AmbariLocalAuthenticationProvider:71 - User not found: cloudbreak
2019-02-07 16:24:20,030 WARN [ambari-client-thread-34] AmbariLdapAuthenticationProvider:128 - Failed to communicate with the LDAP server: localhost:33389; nested exception is javax.naming.CommunicationException: localhost:33389 [Root exception is java.net.ConnectException: Connection refused (Connection refused)]
2019-02-07 16:24:20,030 INFO [ambari-client-thread-34] AmbariAuthenticationEventHandlerImpl:136 - Failed to authenticate cloudbreak: The user does not exist in the Ambari database
2019-02-07 16:24:20,032 INFO [ambari-client-thread-38] AmbariLocalAuthenticationProvider:71 - User not found: cloudbreak
2019-02-07 16:24:20,033 WARN [ambari-client-thread-38] AmbariLdapAuthenticationProvider:128 - Failed to communicate with the LDAP server: localhost:33389; nested exception is javax.naming.CommunicationException: localhost:33389 [Root exception is java.net.ConnectException: Connection refused (Connection refused)]
2019-02-07 16:24:20,034 INFO [ambari-client-thread-38] AmbariAuthenticationEventHandlerImpl:136 - Failed to authenticate cloudbreak: The user does not exist in the Ambari database
2019-02-07 16:24:24,048 INFO [ambari-client-thread-38] UserHookService:107 - Executing user hook for BatchUserHookContext{userGroups={cloudbreak=[]}}.
2019-02-07 16:24:24,049 WARN [ambari-client-thread-38] UserHookService:192 - Post user creation hook disabled.
2019-02-07 16:24:24,049 WARN [ambari-client-thread-38] UserHookService:112 - Prerequisites for user hook are not satisfied. Hook not triggered
2019-02-07 16:24:24,412 WARN [ambari-client-thread-38] Errors:173 - The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.ambari.server.api.services.RootServiceComponentConfigurationService.getConfigurations(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo), should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.ambari.server.api.services.RootServiceComponentConfigurationService.getConfiguration(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String), should not consume any entity.
2019-02-07 16:24:24,434 WARN [ambari-client-thread-38] AmbariServerConfigurationKey:159 - Invalid Ambari server configuration key: ldap-configuration:ambari.ldap.advance.collision_behavior
2019-02-07 16:24:30,077 INFO [ambari-client-thread-34] AnnotationSizeOfFilter:53 - Using regular expression provided through VM argument net.sf.ehcache.pool.sizeof.ignore.pattern for IgnoreSizeOf annotation : ^.*cache\..*IgnoreSizeOf$
2019-02-07 16:24:30,088 INFO [ambari-client-thread-34] AgentLoader:88 - Located valid 'tools.jar' at '/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/../lib/tools.jar'
2019-02-07 16:24:30,096 INFO [ambari-client-thread-34] JvmInformation:446 - Detected JVM data model settings of: 64-Bit OpenJDK JVM with Compressed OOPs and Concurrent Mark-and-Sweep GC
2019-02-07 16:24:30,343 INFO [ambari-client-thread-34] AgentLoader:198 - Extracted agent jar to temporary file /tmp/ehcache-sizeof-agent8929649546267510658.jar
2019-02-07 16:24:30,343 INFO [ambari-client-thread-34] AgentLoader:138 - Trying to load agent @ /tmp/ehcache-sizeof-agent8929649546267510658.jar
2019-02-07 16:24:30,352 INFO [ambari-client-thread-34] DefaultSizeOfEngine:111 - using Agent sizeof engine
2019-02-07 16:24:30,358 INFO [ambari-client-thread-34] TimelineMetricsCacheSizeOfEngine:70 - Creating custom sizeof engine for TimelineMetrics.
2019-02-07 16:24:30,374 INFO [ambari-client-thread-34] TimelineMetricCacheProvider:84 - Creating Metrics Cache with timeouts => ttl = 3600, idle = 1800
2019-02-07 16:24:30,435 INFO [ambari-client-thread-34] TimelineMetricCacheProvider:95 - Registering metrics cache with provider: name = timelineMetricCache, guid: ambari0.littlemac.punchcyber.com/10.101.0.62-ad5ce1d3-7aca-4070-b965-595de265f567
Кроме того, вот шаблон кластера, который я использую для запуска этого кластера из CLI облачного хранилища.Обратите внимание, что это Cloudbreak 2.8 TP, который, как я понимаю, является техническим предварительным просмотром, но мне нужно иметь возможность запустить кластер в GovCloud в какой-то момент, и поддержка этого впервые появляется в 2.8.
{
"name": "HDP and HDF analysis pipeline",
"description": "d",
"type": "ANALYSIS",
"cloudPlatform": "AWS",
"datalakeRequired": "NONE",
"inputs": {},
"general": {
"credentialName": "mycred",
"name": "mycluster"
},
"placement": {
"region": "myregion",
"availabilityZone": "myavailabilityzone"
},
"tags": {
"userDefinedTags": {}
},
"parameters": {},
"customDomain": {
"clusterNameAsSubdomain": false,
"hostgroupNameAsHostname": true
},
"cluster": {
"ambari": {
"blueprintName": "HDP 3.1",
"platformVersion": "HDP 3.1",
"ambariRepoDetailsJson": {
"version": "2.7.3.0",
"baseUrl": "http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0",
"gpgKeyUrl": "http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins"
},
"ambariStackDetails": {
"version": "3.1",
"verify": true,
"enableGplRepo": true,
"stack": "HDP",
"repositoryVersion": "3.1.0.0-78",
"versionDefinitionFileUrl": "http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.0.0/HDP-3.1.0.0-78.xml",
"stackOs": "redhat7",
"mpacks": [
{
"name": "hdf-ambari-mpack-3-3-1-0-10-rhel7"
},
{
"name": "solr-service-mpack-4-0-0"
}
]
},
"userName": "myadminusername",
"password": "mypassword",
"validateBlueprint": false,
"ambariSecurityMasterKey": "mysecretmasterkey",
"enableSecurity": false
},
"ldapConfigName": "myldapconfig",
"cloudStorage": {
"s3": {
"instanceProfile": "my::S3::profile"
}
},
"rdsConfigNames": [
"ambariprod",
"rangerprod",
"hiveprod",
"supersetprod"
]
},
"imageSettings": {
"imageCatalog": "cloudbreak-default",
"imageId": "2c5cd6ea-7d5b-4004-73f2-27ad2e1ce7a9"
},
"stackAuthentication": {
"publicKeyId": "mypublickey"
},
"imageType": "base",
"network": {
"parameters": {
"subnetId": "mysubnetid",
"vpcId": "myvpcid"
}
},
"instanceGroups": [
{
"parameters": {},
"template": {
"parameters": {},
"instanceType": "m5.xlarge",
"volumeType": "standard",
"volumeCount": 1,
"volumeSize": 100,
"rootVolumeSize": 50,
"awsParameters": {
"encryption": {
"type": "NONE"
}
}
},
"nodeCount": 1,
"group": "ambari",
"type": "GATEWAY",
"recoveryMode": "MANUAL",
"recipeNames": [
"ambari-setup"
],
"securityGroup": {
"securityRules": [
{
"subnet": "0.0.0.0/0",
"ports": "9443",
"protocol": "tcp"
},
{
"subnet": "0.0.0.0/0",
"ports": "22",
"protocol": "tcp"
},
{
"subnet": "0.0.0.0/0",
"protocol": "tcp",
"ports": "8443"
},
{
"subnet": "0.0.0.0/0",
"protocol": "tcp",
"ports": "6080"
}
]
}
},
{
"parameters": {},
"template": {
"parameters": {},
"instanceType": "m5.xlarge",
"volumeType": "standard",
"volumeCount": 1,
"volumeSize": 100,
"rootVolumeSize": 50,
"awsParameters": {
"encryption": {
"type": "NONE"
}
}
},
"nodeCount": 1,
"group": "nn1",
"type": "CORE",
"recoveryMode": "MANUAL",
"recipeNames": [
"nn1-setup"
],
"securityGroup": {
"securityRules": [
{
"subnet": "0.0.0.0/0",
"protocol": "tcp",
"ports": "22"
}
]
}
},
{
"parameters": {},
"template": {
"parameters": {},
"instanceType": "m5.xlarge",
"volumeType": "standard",
"volumeCount": 1,
"volumeSize": 100,
"rootVolumeSize": 50,
"awsParameters": {
"encryption": {
"type": "NONE"
}
}
},
"nodeCount": 1,
"group": "nn2",
"type": "CORE",
"recoveryMode": "MANUAL",
"recipeNames": [
"nn2-setup"
],
"securityGroup": {
"securityRules": [
{
"subnet": "0.0.0.0/0",
"protocol": "tcp",
"ports": "22"
}
]
}
},
{
"parameters": {},
"template": {
"parameters": {},
"instanceType": "m5.xlarge",
"volumeType": "standard",
"volumeCount": 2,
"volumeSize": 100,
"rootVolumeSize": 50,
"awsParameters": {
"encryption": {
"type": "NONE"
}
}
},
"nodeCount": 1,
"group": "edge",
"type": "CORE",
"recoveryMode": "MANUAL",
"recipeNames": [
"edge-node-setup"
],
"securityGroup": {
"securityRules": [
{
"subnet": "0.0.0.0/0",
"protocol": "tcp",
"ports": "22"
}
]
}
},
{
"parameters": {},
"template": {
"parameters": {},
"instanceType": "m5.xlarge",
"volumeType": "standard",
"volumeCount": 1,
"volumeSize": 100,
"rootVolumeSize": 50,
"awsParameters": {
"encryption": {
"type": "NONE"
}
}
},
"nodeCount": 0,
"group": "search",
"type": "CORE",
"recoveryMode": "MANUAL",
"recipeNames": [
"single-ebs-setup"
],
"securityGroup": {
"securityRules": [
{
"subnet": "0.0.0.0/0",
"protocol": "tcp",
"ports": "22"
}
]
}
},
{
"parameters": {},
"template": {
"parameters": {},
"instanceType": "m5.xlarge",
"volumeType": "standard",
"volumeCount": 1,
"volumeSize": 100,
"rootVolumeSize": 50,
"awsParameters": {
"encryption": {
"type": "NONE"
}
}
},
"nodeCount": 1,
"group": "hbase",
"type": "CORE",
"recoveryMode": "MANUAL",
"recipeNames": [
"single-ebs-setup"
],
"securityGroup": {
"securityRules": [
{
"subnet": "0.0.0.0/0",
"protocol": "tcp",
"ports": "22"
}
]
}
},
{
"parameters": {},
"template": {
"parameters": {},
"instanceType": "m5.xlarge",
"volumeType": "standard",
"volumeCount": 1,
"volumeSize": 100,
"rootVolumeSize": 50,
"awsParameters": {
"encryption": {
"type": "NONE"
}
}
},
"nodeCount": 1,
"group": "broker_1",
"type": "CORE",
"recoveryMode": "MANUAL",
"recipeNames": [
"named-kafka-setup"
],
"securityGroup": {
"securityRules": [
{
"subnet": "0.0.0.0/0",
"protocol": "tcp",
"ports": "22"
},
{
"subnet": "0.0.0.0/0",
"protocol": "tcp",
"ports": "6667"
}
]
}
},
{
"parameters": {},
"template": {
"parameters": {},
"instanceType": "m5.xlarge",
"volumeType": "standard",
"volumeCount": 1,
"volumeSize": 100,
"rootVolumeSize": 50,
"awsParameters": {
"encryption": {
"type": "NONE"
}
}
},
"nodeCount": 1,
"group": "broker",
"type": "CORE",
"recoveryMode": "MANUAL",
"recipeNames": [
"single-ebs-setup"
],
"securityGroup": {
"securityRules": [
{
"subnet": "0.0.0.0/0",
"protocol": "tcp",
"ports": "22"
},
{
"subnet": "0.0.0.0/0",
"protocol": "tcp",
"ports": "6667"
}
]
}
},
{
"parameters": {},
"template": {
"parameters": {},
"instanceType": "m5.xlarge",
"volumeType": "standard",
"volumeCount": 1,
"volumeSize": 100,
"rootVolumeSize": 50,
"awsParameters": {
"encryption": {
"type": "NONE"
}
}
},
"nodeCount": 1,
"group": "datanode",
"type": "CORE",
"recoveryMode": "MANUAL",
"recipeNames": [
"single-ebs-setup"
],
"securityGroup": {
"securityRules": [
{
"subnet": "0.0.0.0/0",
"protocol": "tcp",
"ports": "22"
}
]
}
},
{
"parameters": {},
"template": {
"parameters": {},
"instanceType": "m5.xlarge",
"volumeType": "standard",
"volumeCount": 1,
"volumeSize": 100,
"rootVolumeSize": 50,
"awsParameters": {
"encryption": {
"type": "NONE"
}
}
},
"nodeCount": 0,
"group": "worker",
"type": "CORE",
"recoveryMode": "MANUAL",
"recipeNames": [
"single-ebs-setup"
],
"securityGroup": {
"securityRules": [
{
"subnet": "0.0.0.0/0",
"protocol": "tcp",
"ports": "22"
}
]
}
},
{
"parameters": {},
"template": {
"parameters": {},
"instanceType": "m5.xlarge",
"volumeType": "standard",
"volumeCount": 1,
"volumeSize": 100,
"rootVolumeSize": 50,
"awsParameters": {
"encryption": {
"type": "NONE"
}
}
},
"nodeCount": 0,
"group": "gpu",
"type": "CORE",
"recoveryMode": "MANUAL",
"recipeNames": [
"single-ebs-setup"
],
"securityGroup": {
"securityRules": [
{
"subnet": "0.0.0.0/0",
"protocol": "tcp",
"ports": "22"
}
]
}
}
]
}
Мне удалось найти чертежи на странице Apache Ambari Github, в которых используется HDP 3.1 , поэтому я уверен, что это "должно" работать ... но я не знаю, где внести изменения всценарий установки Ambari для решения этих проблем.