вы можете использовать DataFrame.merge и fillna , используя метод ffill
:
df1 = pd.DataFrame({'old_index': [5, 9, 872],
'Value 1': [1.234, -0.233, 28.11]})
df2 = pd.DataFrame({'old_index': [6, 15, 23],
'Value 2': [-1.234, 0.433, 2.11]})
new_df = df1.merge(df2, how = 'outer').sort_values(by = 'old_index').fillna(method = 'ffill').reset_index(drop = True)
new_df.index+=1
new_df
old_index Value 1 Value 2
1 5 1.234 NaN
2 6 1.234 -1.234
3 9 -0.233 -1.234
4 15 -0.233 0.433
5 23 -0.233 2.110
6 872 28.110 2.110