Чтобы добавить полнотекстовый поиск в мое приложение App Engine, я добавил в свою модель следующее поле:
private List<String> fullText;
Для проверки поиска я взял следующий текст:
Oxandrolone is a synthetic anabolic steroid derived from dihydrotestosterone by substituting 2nd carbon atom for oxygen (O). It is widely known for its exceptionally small level of androgenicity accompanied by moderate anabolic effect. Although oxandrolone is a 17-alpha alkylated steroid, its liver toxicity is very small as well. Studies have showed that a daily dose of 20 mg oxandrolone used in the course of 12 weeks had only a negligible impact on the increase of liver enzymes[1][2]. As a DHT derivative, oxandrolone does not aromatize (convert to estrogen, which causes gynecomastia or male breast tissue). It also does not significantly influence the body's normal testosterone production (HPTA axis) at low dosages (10 mg). When dosages are high, the human body reacts by reducing the production of LH (luteinizing hormone), thinking endogenous testosterone production is too high; this in turn eliminates further stimulation of Leydig cells in the testicles, causing testicular atrophy (shrinking). Oxandrolone used in a dose of 80 mg/day suppressed endogenous testosterone by 67% after 12 weeks of therapy[3].
И применил этот код Java к нему:
StringTokenizer st = new StringTokenizer(recordText);
List<String> fullTextSearchSupport = new ArrayList<String>();
while (st.hasMoreTokens())
{
String token = st.nextToken().trim();
if (token.length() > 3)
{
fullTextSearchSupport.add(token);
}
}
Я получил следующий ArrayList из String tokens:
[Oxandrolone, synthetic, anabolic, steroid, derived, from, dihydrotestosterone, substituting, carbon, atom, oxygen, (O)., widely, known, exceptionally, small, level, androgenicity, accompanied, moderate, anabolic, effect., Although, oxandrolone, 17-alpha, alkylated, steroid,, liver, toxicity, very, small, well., Studies, have, showed, that, daily, dose, oxandrolone, used, course, weeks, only, negligible, impact, increase, liver, enzymes[1][2]., derivative,, oxandrolone, does, aromatize, (convert, estrogen,, which, causes, gynecomastia, male, breast, tissue)., also, does, significantly, influence, body's, normal, testosterone, production, (HPTA, axis), dosages, mg)., When, dosages, high,, human, body, reacts, reducing, production, (luteinizing, hormone),, thinking, endogenous, testosterone, production, high;, this, turn, eliminates, further, stimulation, Leydig, cells, testicles,, causing, testicular, atrophy, (shrinking)., Oxandrolone, used, dose, mg/day, suppressed, endogenous, testosterone, after, weeks, therapy[3].]
Что меня удивило, так это то, что при разбиении строки на токены StringTokenizer оставляет такие знаки препинания, как запятые, точки, скобки и скобки.
Например, для текстового поиска токен:
derivative,
может быть просто
derivative
и
enzymes[1][2].
может быть просто:
enzymes
Каков наилучший способ вывода только английских слов, необходимых для текстового поиска, за исключением знаков препинания и специальных символов?
Я попытался уменьшить соединительные слова меньшего размера (a, by, for) с помощью этого условия:
token.length() > 3
но, очевидно, этого недостаточно.