Ответ прост, когда вы знаете, что использовать.Вот пример ручного вызова, который можно использовать в тесте JUnit.
DocumentationTool systemDocumentationTool = ToolProvider.getSystemDocumentationTool();
String[] args = new String[] {
"-sourcepath",
"./src/test/java",
"-subpackages",
"<whatever-package-in-test-sources>",
"<whatever-package-in-test-sources>",
"-d",
"whatever not used just to show compatibility",
"-author",
"whatever not used just to show compatibility",
"-doctitle",
"whatever not used just to show compatibility",
"-windowtitle",
"whatever not used just to show compatibility",
"-classdir",
BUILD_PROPERTY_FILE_LOCATION
};
DocumentationTool.DocumentationTask task = systemDocumentationTool.getTask(null, null, null,
<MyDocletClass>.class, Arrays.asList(args), null);
task.call();
// Your checks after invocation