Извините, но я не знаю RMagick или R. Но вот команды ImageMagick CLI для рисования эллиптических и круглых прямоугольников.
Ввод:

Овал:
Read the image
Clone it and use -vignette to draw an ellipse and extract the alpha channel, blur it, then apply -level
Then put the result into the alpha channel of the input image
convert thumbnail.gif \
\( -clone 0 -background none -virtual-pixel none -vignette 0x0+0+0 \
-alpha extract -blur 0x10 -level 50x100% \) \
-alpha off -compose copyopacity -composite \
result.png

Круглый прямоугольник:
Read the input image and get the bottom right corner coordinate
Read it again.
Clone it, fill it with black, draw a round rectangle, blur it, apply -level
Then put the result into the alpha channel of the input image
wm1=`convert thumbnail.gif -format "%[fx:w-1]" info:`
hm1=`convert thumbnail.gif -format "%[fx:h-1]" info:`
convert thumbnail.gif \
\( -clone 0 -fill black -colorize 100 \
-fill white -virtual-pixel black -draw "roundrectangle 0,0 $wm1,$hm1 15,15" \
-blur 0x10 -level 50x100% \) \
-alpha off -compose CopyOpacity -composite \
result2.png

Итак, вам нужны RMagick-эквиваленты -vignette, -composite, -draw, -blur и -level.