Самоссылочный habtm находится немного за пределами стандартного набора инструментов Rails. Вам нужно настроить finder_sql или иметь 2 отношения:
has_and_belongs_to_many :related_documents, join_table: :related_documents,
class_name: "Document",
foreign_key: :referent_id,
association_foreign_key: :reference_id
has_and_belongs_to_many :referenced_documents, join_table: :related_documents,
class_name: "Document",
foreign_key: :reference_id,
association_foreign_key: :referent_id
Если вам нужно объединить результаты в один SQL, вам нужно настроить Finder-SQL, см., Например, https://gist.github.com/srpouyet/4121517
has_and_belongs_to_many :related_documents,
class_name: 'Document',
join_table: :related_documents,
foreign_key: :reference_id,
association_foreign_key: :referent_id,
uniq: true,
finder_sql: proc {
%(SELECT DISTINCT "documents".* FROM "documents"
INNER JOIN "related_documents"
ON "documents"."id" = "related_documents"."referent_id"
WHERE "related_documents"."reference_id" = #{id}
UNION
SELECT DISTINCT "documents".* FROM "documents"
INNER JOIN "related_documents"
ON "documents"."id" = "related_documents"."reference_id"
WHERE "related_documents"."referent_id" = #{id}
)}