Я новичок в стеке MS BI и пытаюсь создать раздел в поле таблицы SQL DW, то есть cust_id, который уникален для каждого клиента (tenant_id) и присутствует в каждой таблице, так что данные каждого клиента остаются в соответствующих разделах,не влияя на других.
Ниже приведена структура таблицы, данные и выходные данные разделов:
create table emp
(
cust_id integer not null,
emp_id varchar(5) not null,
emp_name varchar(10) not null
)
with
(
clustered columnstore index
,distribution = round_robin
,partition (cust_id range right for values (100,200,300) )
)
create table dept
(
cust_id integer not null,
dept_id varchar(5) not null,
emp_id varchar(5) not null,
dep_name varchar(10) not null
)
with
(
clustered columnstore index
,distribution = round_robin
,partition (cust_id range right for values (100,200,300) )
)
create statistics emp_stats on dbo.emp(cust_id)
create statistics dept_stats on dbo.dept(cust_id)
emp table:
101 EMP01 XYZ
101 EMP02 ABC
101 EMP03 DEF
201 EE001 JACK
201 EE002 MIKE
dept table:
cust_id dept_id emp_id dep_name
101 D0001 EMP01 IT
101 D0001 EMP02 IT
201 DEP01 EE001 ENG
201 DEP02 EE002 HR
SELECT sch.name AS [schema_name],
tbl.[name] AS [table_name],
ds.type_desc,
prt.[partition_number],
rng.[value] AS [current_partition_range_boundary_value],
prt.[rows] AS [partition_rows]
FROM sys.schemas sch
INNER JOIN sys.tables tbl ON sch.schema_id = tbl.schema_id
INNER JOIN sys.partitions prt ON prt.[object_id] = tbl.[object_id]
INNER JOIN sys.indexes idx ON prt.[object_id] = idx.[object_id] AND prt.[index_id] = idx.[index_id]
INNER JOIN sys.data_spaces ds ON idx.[data_space_id] = ds.[data_space_id]
INNER JOIN sys.partition_schemes ps ON ds.[data_space_id] = ps.[data_space_id]
INNER JOIN sys.partition_functions pf ON ps.[function_id] = pf.[function_id]
LEFT JOIN sys.partition_range_values rng ON pf.[function_id] = rng.[function_id] AND rng.[boundary_id] = prt.[partition_number]
WHERE tbl.name in ('emp','dept')
order by table_name, partition_number
schema_name table_name type_desc partition_number current_partition_range_boundary_value partition_rows
dbo dept PARTITION_SCHEME 1 100 1
dbo dept PARTITION_SCHEME 2 200 1
dbo dept PARTITION_SCHEME 3 300 1
dbo dept PARTITION_SCHEME 4 NULL 1
dbo emp PARTITION_SCHEME 1 100 1
dbo emp PARTITION_SCHEME 2 200 1
dbo emp PARTITION_SCHEME 3 300 1
dbo emp PARTITION_SCHEME 4 NULL 2
Вопросы / разъяснения:
1) Whether the partition created on cust_id (tenant_id) field along with round_robin distribution method correct? What is the right way to do it? Need to segregate the customer specific data for both performance (load + query) & security reasons.
2) How can we load specific customer data into their respective partition (cust_id) – syntax in SQL DW?
insert into emp (partition = <partition_name_number> ) ?
3) How do I verify that the data is getting loaded into correct respective partition as I am unable to understand the output from above query as to how it is showing 4 partitions and only 1 row for cust_id 101 in emp table when actually there are 3 ? Was expecting that since 101 is between 100 and 200, it should be in partition_number = 1 and 201 which is between 200 and 300 in partition_number = 2? Is this a wrong assumption on how partition range works? Can’t we simply have a List partition created in SQL DW for each cust_id ?
4) As per MS documentation, it by default divides each table into 60 distributed databases and that there should be at least 1M per distribution for a partition. When you don’t know how many customer’s data and their volume you may have in future, how do we approach towards it ?
5) When creating semantic layer (Analysis Services-SSAS) on top of DW, is it helpful to do further do partition on tenant_id or some other field?
Спасибо за вашу помощь, отзывы и предложения!!