Я следую инструкциям Реализация Terraform в Microsoft Azure к письму. Пока все хорошо, и я достиг модуля, в котором рабочие пространства terraform используются для создания сред из конвейера Azure DevOps CI / CD. Имя модуля: «Использование Azure DevOps»
Проект Azure DevOps опубликован c - https://dev.azure.com/MarkKharitonov0271/_git/Globomantics-testing
Запуск рабочих областей конвейер выпуска для меня постоянно терпит неудачу:
2020-01-19T03:23:20.5420275Z Error: authorization.RoleDefinitionsClient#CreateOrUpdate: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client '0e648d2d-a49f-407e-99de-9d6343876a8c' with object id '0e648d2d-a49f-407e-99de-9d6343876a8c' does not have authorization to perform action 'Microsoft.Authorization/roleDefinitions/write' over scope '/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3' or the scope is invalid. If access was recently granted, please refresh your credentials."
2020-01-19T03:23:20.5420512Z
2020-01-19T03:23:20.5420989Z on vnet-peering.tf line 52, in resource "azurerm_role_definition" "vnet-peering":
2020-01-19T03:23:20.5421339Z 52: resource "azurerm_role_definition" "vnet-peering" {
(Вы можете просмотреть vnet -peering.tf в репозитории проекта - https://dev.azure.com/MarkKharitonov0271/_git/Globomantics-testing?path=%2Fnetworking%2Fvnet-peering.tf)
В любом случае, если я правильно понимаю сообщение об ошибке, оно утверждает, что у участника службы 0e648d2d-a49f-407e-99de-9d6343876a8c нет прав на создание нового определения роли в подписке 2b38509c-a310-4c8f-bd78-9e400cc874e3
Итак, я проверил службу Принципал:
PS /home/mark> az ad sp show --id '0e648d2d-a49f-407e-99de-9d6343876a8c'
{
"accountEnabled": "True",
"addIns": [],
"alternativeNames": [],
"appDisplayName": "MarkKharitonov0271-Globomantics-testing-2b38509c-a310-4c8f-bd78-9e400cc874e3",
"appId": "0ae4ffc7-149d-45ac-ab15-c9f61e4591f8",
"appOwnerTenantId": "717e5a4d-529c-4ab2-a1c5-6a5f6345d8e4",
"appRoleAssignmentRequired": false,
"appRoles": [],
"applicationTemplateId": null,
"deletionTimestamp": null,
"displayName": "MarkKharitonov0271-Globomantics-testing-2b38509c-a310-4c8f-bd78-9e400cc874e3",
"errorUrl": null,
"homepage": "https://VisualStudio/SPN",
"informationalUrls": {
"marketing": null,
"privacy": null,
"support": null,
"termsOfService": null
},
"keyCredentials": [],
"logoutUrl": null,
"notificationEmailAddresses": [],
"oauth2Permissions": [
{
"adminConsentDescription": "Allow the application to access MarkKharitonov0271-Globomantics-testing-2b38509c-a310-4c8f-bd78-9e400cc874e3 on behalf of the signed-in user.",
"adminConsentDisplayName": "Access MarkKharitonov0271-Globomantics-testing-2b38509c-a310-4c8f-bd78-9e400cc874e3",
"id": "d0f141b9-fc6b-4f3c-9217-018d74712ee1",
"isEnabled": true,
"userConsentDescription": "Allow the application to access MarkKharitonov0271-Globomantics-testing-2b38509c-a310-4c8f-bd78-9e400cc874e3 on your behalf.",
"userConsentDisplayName": "Access MarkKharitonov0271-Globomantics-testing-2b38509c-a310-4c8f-bd78-9e400cc874e3",
}
],
"objectId": "0e648d2d-a49f-407e-99de-9d6343876a8c",
"objectType": "ServicePrincipal",
"odata.metadata": "https://graph.windows.net/717e5a4d-529c-4ab2-a1c5-6a5f6345d8e4/$metadata#directoryObjects/@Element",
"odata.type": "Microsoft.DirectoryServices.ServicePrincipal",
"passwordCredentials": [],
"preferredSingleSignOnMode": null,
"preferredTokenSigningKeyEndDateTime": null,
"preferredTokenSigningKeyThumbprint": null,
"publisherName": "Default Directory",
"replyUrls": [
"https://VisualStudio/SPN"
],
"samlMetadataUrl": null,
"samlSingleSignOnSettings": null,
"servicePrincipalNames": [
"https://VisualStudio/SPN136d4f76-7262-4ab0-8fbb-7be74dfc803b",
"0ae4ffc7-149d-45ac-ab15-c9f61e4591f8"
],
"servicePrincipalType": "Application",
"signInAudience": "AzureADMyOrg",
"tags": [],
"tokenEncryptionKeyId": null
}
Обратите внимание, что у субъекта службы appId равно 0ae4ffc7-149d-45ac-ab15-c9f61e4591f8.
И, похоже, он соответствует тому, который был создан Azure DevOps, когда я добавил Terraform задачи в конвейер, когда он хотел авторизовать доступ к подписке. Действительно:
Теперь шаг применения терраформы ссылается на тот же принцип обслуживания:
Наконец, у этого Принципала службы, похоже, есть доступ к подписке для участника:
PS /home/mark> az role assignment list --assignee '0e648d2d-a49f-407e-99de-9d6343876a8c'
[
{
"canDelegate": null,
"id": "/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/providers/Microsoft.Authorization/roleAssignments/346f1b92-0621-44c0-b88a-343c52637a0f",
"name": "346f1b92-0621-44c0-b88a-343c52637a0f",
"principalId": "0e648d2d-a49f-407e-99de-9d6343876a8c",
"principalName": "https://VisualStudio/SPN136d4f76-7262-4ab0-8fbb-7be74dfc803b",
"principalType": "ServicePrincipal",
"roleDefinitionId": "/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c",
"roleDefinitionName": "Contributor",
"scope": "/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3",
"type": "Microsoft.Authorization/roleAssignments"
}
]
PS /home/mark>
Итак, я не понимаю, что происходит.
РЕДАКТИРОВАТЬ 1
Итак, я смог запустить точно такую же конфигурацию terraform с точно таким же принципалом обслуживания из Azure Cloud Shell.
Сначала я сбросил учетные данные, вышел из системы и снова вошел в систему под учетной записью этого участника службы:
PS /home/mark/terraform> az ad sp credential reset --name https://VisualStudio/SPN136d4f76-7262-4ab0-8fbb-7be74dfc803b
{
"appId": "0ae4ffc7-149d-45ac-ab15-c9f61e4591f8",
"name": "https://VisualStudio/SPN136d4f76-7262-4ab0-8fbb-7be74dfc803b",
"password": "e...3",
"tenant": "717e5a4d-529c-4ab2-a1c5-6a5f6345d8e4"
}
PS /home/mark/terraform> az account clear
Logout successful. Re-login to your initial Cloud Shell identity with 'az login --identity'. Login with a new identity with 'az login'.
PS /home/mark/terraform> az login --service-principal -u https://VisualStudio/SPN136d4f76-7262-4ab0-8fbb-7be74dfc803b -p e...3 --tenant 717e5a4d-529c-4ab2-a1c5-6a5f6345d8e4
Cloud Shell is automatically authenticated under the initial account signed-in with. Run 'az login' only if you need to use a different account
{
"cloudName": "AzureCloud",
"id": "2b38509c-a310-4c8f-bd78-9e400cc874e3",
"isDefault": true,
"name": "Visual Studio Enterprise",
"state": "Enabled",
"type": "servicePrincipal"
}
}
]
Затем я клонировал репозиторий Git для получения кода Terraform и скопировал внутреннюю конфигурацию, сохраненную из предыдущих модулей:
PS /home/mark/terraform> git clone https://MarkKharitonov0271@dev.azure.com/MarkKharitonov0271/Globomantics-testing/_git/Globomantics-testing
Cloning into 'Globomantics-testing'...
remote: Azure Repos
remote: We noticed you're using an older version of Git. For the best experience, upgrade to a newer version.
remote: Found 9 objects to send. (2 ms)
Unpacking objects: 100% (9/9), done.
Checking connectivity... done.
PS /home/mark/terraform> cd ./Globomantics-testing/networking/
PS /home/mark/terraform/Globomantics-testing/networking> dir
Directory: /home/mark/terraform/Globomantics-testing/networking
Mode LastWriteTime Length Name
---- ------------- ------ ----
------ 1/19/20 4:41 AM 40 backend.tf
------ 1/19/20 4:41 AM 2256 main.tf
------ 1/19/20 4:41 AM 436 terraform.tfvars
------ 1/19/20 4:41 AM 2423 vnet-peering.tf
------ 1/19/20 4:41 AM 279 workspacetest.sh
PS /home/mark/terraform/Globomantics-testing/networking> copy ../../1-main-vnet/backend-config.txt .
Теперь я готов инициализировать Terraform и установить рабочее пространство для разработки:
PS /home/mark/terraform/Globomantics-testing/networking> terraform init -backend-config='backend-config.txt'
Initializing modules...
Downloading Azure/vnet/azurerm 1.2.0 for vnet-main...
- vnet-main in .terraform/modules/vnet-main
Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "azurerm" (hashicorp/azurerm) 1.41.0...
- Downloading plugin for provider "template" (hashicorp/template) 2.1.2...
...
PS /home/mark/terraform/Globomantics-testing/networking> terraform workspace select development
Switched to workspace "development".
Теперь план terraform:
PS /home/mark/terraform/Globomantics-testing/networking> terraform plan -var sec_client_secret='F...$' -out main.tfplan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.template_file.subnet_prefixes[0]: Refreshing state...
data.template_file.subnet_prefixes[1]: Refreshing state...
data.template_file.subnet_prefixes[2]: Refreshing state...
azurerm_resource_group.main: Refreshing state... [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet]
data.azurerm_subscription.current: Refreshing state...
module.vnet-main.azurerm_resource_group.vnet: Refreshing state... [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet]
module.vnet-main.azurerm_virtual_network.vnet: Refreshing state... [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet]
module.vnet-main.azurerm_subnet.subnet[0]: Refreshing state... [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet/subnets/web]
module.vnet-main.azurerm_subnet.subnet[2]: Refreshing state... [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet/subnets/app]
module.vnet-main.azurerm_subnet.subnet[1]: Refreshing state... [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet/subnets/database]
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# azurerm_role_assignment.vnet will be created
+ resource "azurerm_role_assignment" "vnet" {
+ id = (known after apply)
+ name = (known after apply)
+ principal_id = "63309b8d-908a-4dcf-b95b-25eae33aaceb"
+ principal_type = (known after apply)
+ role_definition_id = (known after apply)
+ role_definition_name = (known after apply)
+ scope = "/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet"
+ skip_service_principal_aad_check = (known after apply)
}
# azurerm_role_definition.vnet-peering will be created
+ resource "azurerm_role_definition" "vnet-peering" {
+ assignable_scopes = [
+ "/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3",
]
+ id = (known after apply)
+ name = "allow-vnet-peer-action-development"
+ role_definition_id = (known after apply)
+ scope = "/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3"
+ permissions {
+ actions = [
+ "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write",
+ "Microsoft.Network/virtualNetworks/peer/action",
+ "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read",
+ "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/delete",
]
+ not_actions = []
}
}
# azurerm_virtual_network_peering.main will be created
+ resource "azurerm_virtual_network_peering" "main" {
+ allow_forwarded_traffic = (known after apply)
+ allow_gateway_transit = (known after apply)
+ allow_virtual_network_access = (known after apply)
+ id = (known after apply)
+ name = "development_2_sec"
+ remote_virtual_network_id = "/subscriptions/2b1285d1-f4a7-4cc3-a7b1-0d0fc31d6192/resourceGroups/security/providers/Microsoft.Network/virtualNetworks/security"
+ resource_group_name = "development-vnet"
+ use_remote_gateways = (known after apply)
+ virtual_network_name = "development-vnet"
}
# azurerm_virtual_network_peering.sec will be created
+ resource "azurerm_virtual_network_peering" "sec" {
+ allow_forwarded_traffic = (known after apply)
+ allow_gateway_transit = (known after apply)
+ allow_virtual_network_access = (known after apply)
+ id = (known after apply)
+ name = "sec_2_development"
+ remote_virtual_network_id = "/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet"
+ resource_group_name = "security"
+ use_remote_gateways = (known after apply)
+ virtual_network_name = "security"
}
Plan: 4 to add, 0 to change, 0 to destroy.
...
------------------------------------------------------------------------
This plan was saved to: main.tfplan
To perform exactly these actions, run the following command to apply:
terraform apply "main.tfplan"
И, наконец, терраформ применяется:
PS /home/mark/terraform/Globomantics-testing/networking> terraform apply main.tfplan
azurerm_role_definition.vnet-peering: Creating...
azurerm_role_definition.vnet-peering: Creation complete after 1s [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/providers/Microsoft.Authorization/roleDefinitions/b9614597-de13-c2c6-a275-9186847642ed]
azurerm_role_assignment.vnet: Creating...
azurerm_role_assignment.vnet: Creation complete after 1s [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet/providers/Microsoft.Authorization/roleAssignments/7f6a642d-d43e-05b5-cf34-3d20d02cd255]
azurerm_virtual_network_peering.main: Creating...
azurerm_virtual_network_peering.sec: Creating...
azurerm_virtual_network_peering.main: Still creating... [10s elapsed]
azurerm_virtual_network_peering.sec: Still creating... [10s elapsed]
azurerm_virtual_network_peering.main: Creation complete after 11s [id=/subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet/virtualNetworkPeerings/development_2_sec]
azurerm_virtual_network_peering.sec: Creation complete after 11s [id=/subscriptions/2b1285d1-f4a7-4cc3-a7b1-0d0fc31d6192/resourceGroups/security/providers/Microsoft.Network/virtualNetworks/security/virtualNetworkPeerings/sec_2_development]
...
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
Outputs:
resource_group_name = development-vnet
vnet_id = /subscriptions/2b38509c-a310-4c8f-bd78-9e400cc874e3/resourceGroups/development-vnet/providers/Microsoft.Network/virtualNetworks/development-vnet
vnet_name = development-vnet
И все работает нормально, используя тот же сервис Прин cipal, созданный Azure DevOps.
Далее я попытаюсь удалить служебное соединение в проекте Azure DevOps, создать новое вручную и повторить выпуск.
РЕДАКТИРОВАТЬ 2
Та же ошибка с новым служебным подключением. Я не понимаю этого. Что я нахожу «любопытным», так это то, что все ресурсы в main.tf были подготовлены просто отлично, но нет в vnet-peering.tf.
Что здесь происходит?
РЕДАКТИРОВАТЬ 3
Заменил задачи Terraform в Microsoft DevLabs на задачи Чарльза Зиппа, потому что их использовал автор курса Pluralsight. Тот же результат.