В настоящее время я играю с Word2Vec в deeplearning4j (используя scala).
и я заметил, что с моей текущей конфигурацией это очень медленно при использовании графического сервера. Я проверил через 8 часов, что он все еще работает только с 4000 Words/sec и в большинстве случаев < 100 Seq/Sec. Я убил процесс и перезапустил его с серверным процессором, который работает при 150.000 Words/sec и около 2000-3000 Seq/Sec и достиг того же состояния менее чем за 20 минут.
Мне было бы интересно узнать, если это проблема конфигурации или что-то еще.
Мои зависимости выглядят так:
//CPU dependency (excluded in GPU run but enabled in CPU)
//libraryDependencies += "org.nd4j" % "nd4j-native-platform" % "1.0.0-beta3"
libraryDependencies += "org.deeplearning4j" % "deeplearning4j-core" % "1.0.0-beta3"
libraryDependencies += "org.deeplearning4j" % "deeplearning4j-ui_2.11" % "1.0.0-beta3"
libraryDependencies += "org.deeplearning4j" % "deeplearning4j-cuda-10.0" % "1.0.0-beta3"
libraryDependencies += "org.nd4j" % "nd4j-cuda-10.0-platform" % "1.0.0-beta3"
libraryDependencies += "org.bytedeco.javacpp-presets" % "cuda" % "10.0-7.4-1.4.4" classifier "windows-x86_64-redist"
Моя конфигурация Word2Vec выглядит так (как для CPU, так и для GPU)
val w2v = new Word2Vec.Builder()
.minWordFrequency(5)
.iterations(8)
.layerSize(100)
.seed(42)
.windowSize(5)
.iterate(iter)
.batchSize(128)
.tokenizerFactory(tokenizerFactory)
.useUnknown(true)
.build();
А вот и вывод журнала
[info] Running com.learning.Word2VecBuilder
08:38:36.464 [run-main-0] INFO org.nd4j.linalg.factory.Nd4jBackend - Loaded [JCublasBackend] backend
08:38:39.715 [run-main-0] INFO org.nd4j.nativeblas.NativeOpsHolder - Number of threads used for NativeOps: 32
08:38:40.929 [run-main-0] INFO org.nd4j.nativeblas.Nd4jBlas - Number of threads used for BLAS: 0
08:38:40.935 [run-main-0] INFO o.n.l.a.o.e.DefaultOpExecutioner - Backend used: [CUDA]; OS: [Windows 10]
08:38:40.935 [run-main-0] INFO o.n.l.a.o.e.DefaultOpExecutioner - Cores: [4]; Memory: [5,3GB];
08:38:40.936 [run-main-0] INFO o.n.l.a.o.e.DefaultOpExecutioner - Blas vendor: [CUBLAS]
08:38:40.937 [run-main-0] INFO o.n.l.j.o.e.CudaExecutioner - Device Name: [GeForce GTX 1060 6GB]; CC: [6.1]; Total/free memory: [6442450944]
08:38:41.318 [run-main-0] DEBUG o.n.j.handler.impl.CudaZeroHandler - Creating bucketID: 5
Maximum number of Memory is: 5726797824 bytes
08:38:41.782 [run-main-0] INFO o.d.m.s.SequenceVectors - Starting vocabulary building...
08:38:41.782 [run-main-0] DEBUG o.d.m.w.wordstore.VocabConstructor - Target vocab size before building: [0]
08:38:41.937 [run-main-0] DEBUG o.d.m.w.wordstore.VocabConstructor - Trying source iterator: [0]
08:38:41.937 [run-main-0] DEBUG o.d.m.w.wordstore.VocabConstructor - Target vocab size before building: [0]
08:39:02.679 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [100000]; Current vocabulary size: [166411]; Sequences/sec: 4785,38; Words/sec: 50201,03;
08:39:21.238 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [200000]; Current vocabulary size: [472153]; Sequences/sec: 5388,22; Words/sec: 310297,65;
08:39:40.490 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [300000]; Current vocabulary size: [600278]; Sequences/sec: 5194,27; Words/sec: 155047,27;
08:39:58.260 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [400000]; Current vocabulary size: [716479]; Sequences/sec: 5627,46; Words/sec: 199443,61;
08:40:15.360 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [500000]; Current vocabulary size: [869586]; Sequences/sec: 5847,95; Words/sec: 327225,85;
08:40:31.590 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [600000]; Current vocabulary size: [993343]; Sequences/sec: 6161,43; Words/sec: 280355,95;
08:40:50.260 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [700000]; Current vocabulary size: [1122310]; Sequences/sec: 5356,19; Words/sec: 269141,78;
08:41:05.564 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [800000]; Current vocabulary size: [1236145]; Sequences/sec: 6534,24; Words/sec: 338252,61;
08:41:18.590 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [900000]; Current vocabulary size: [1347346]; Sequences/sec: 7676,95; Words/sec: 424641,79;
08:41:31.288 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [1000000]; Current vocabulary size: [1477759]; Sequences/sec: 7875,26; Words/sec: 524719,64;
08:41:46.564 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [1100000]; Current vocabulary size: [1599449]; Sequences/sec: 6546,22; Words/sec: 451319,13;
08:41:58.174 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [1200000]; Current vocabulary size: [1714450]; Sequences/sec: 8613,26; Words/sec: 597122,74;
08:42:14.504 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [1300000]; Current vocabulary size: [1806469]; Sequences/sec: 6123,70; Words/sec: 275960,26;
08:42:30.468 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [1400000]; Current vocabulary size: [1891943]; Sequences/sec: 6264,09; Words/sec: 277645,01;
08:42:44.215 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [1500000]; Current vocabulary size: [1978590]; Sequences/sec: 7274,31; Words/sec: 399430,64;
08:43:00.936 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [1600000]; Current vocabulary size: [2061284]; Sequences/sec: 5980,50; Words/sec: 290828,00;
08:43:13.851 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [1700000]; Current vocabulary size: [2141019]; Sequences/sec: 7742,93; Words/sec: 356932,17;
08:43:32.731 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [1800000]; Current vocabulary size: [2200118]; Sequences/sec: 5296,61; Words/sec: 177499,68;
08:43:55.174 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [1900000]; Current vocabulary size: [2206803]; Sequences/sec: 4455,73; Words/sec: 28245,24;
08:44:16.881 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [2000000]; Current vocabulary size: [2266169]; Sequences/sec: 4606,81; Words/sec: 166374,03;
08:44:33.816 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [2100000]; Current vocabulary size: [2340684]; Sequences/sec: 5904,93; Words/sec: 281809,33;
08:44:49.214 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [2200000]; Current vocabulary size: [2419663]; Sequences/sec: 6494,35; Words/sec: 336157,81;
08:44:58.814 [run-main-0] DEBUG o.d.m.w.wordstore.VocabConstructor - Waiting till all processes stop...
08:44:58.814 [run-main-0] DEBUG o.d.m.w.wordstore.VocabConstructor - Vocab size before truncation: [2470993], NumWords: [104287573], sequences parsed: [2259529], counter: [104286594]
08:45:00.379 [run-main-0] DEBUG o.d.m.w.wordstore.VocabConstructor - Scavenger: Words before: 2470993; Words after: 448171;
08:45:00.380 [run-main-0] DEBUG o.d.m.w.wordstore.VocabConstructor - Vocab size after truncation: [448171], NumWords: [101414224], sequences parsed: [2259529], counter: [104286594]
08:45:01.664 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Adding UNK element to vocab...
08:45:03.976 [run-main-0] INFO o.d.m.w.wordstore.VocabConstructor - Sequences checked: [2259529], Current vocabulary size: [448172]; Sequences/sec: [5912,01];
08:45:03.990 [run-main-0] INFO o.d.m.e.loader.WordVectorSerializer - Projected memory use for model: [341,93 MB]
08:45:04.648 [run-main-0] DEBUG o.n.j.handler.impl.CudaZeroHandler - Creating bucketID: 3
08:45:04.662 [run-main-0] DEBUG o.n.j.handler.impl.CudaZeroHandler - Creating bucketID: 4
08:45:04.697 [run-main-0] DEBUG o.n.j.handler.impl.CudaZeroHandler - Creating bucketID: 0
08:45:04.702 [run-main-0] INFO o.d.m.e.inmemory.InMemoryLookupTable - Initializing syn1...
08:45:05.877 [run-main-0] INFO o.d.m.s.SequenceVectors - Building learning algorithms:
08:45:05.877 [run-main-0] INFO o.d.m.s.SequenceVectors - building ElementsLearningAlgorithm: [SkipGram]
08:45:05.882 [run-main-0] DEBUG o.n.j.handler.impl.CudaZeroHandler - Creating bucketID: 1
08:45:05.884 [run-main-0] DEBUG o.n.j.handler.impl.CudaZeroHandler - Creating bucketID: 2
08:45:05.900 [run-main-0] INFO o.d.m.s.SequenceVectors - Starting learning process...
08:49:39.953 [VectorCalculationsThread 2] INFO o.d.m.s.SequenceVectors - Epoch: [1]; Words vectorized so far: [1216051]; Lines vectorized so far: [100000]; Seq/sec: [364,90]; Words/sec: [4437,40]; learningRate: [0.024962528224616736]
08:53:35.063 [VectorCalculationsThread 1] INFO o.d.m.s.SequenceVectors - Epoch: [1]; Words vectorized so far: [2278941]; Lines vectorized so far: [200000]; Seq/sec: [425,33]; Words/sec: [4475,91]; learningRate: [0.02492977581547878]
08:57:14.696 [VectorCalculationsThread 1] INFO o.d.m.s.SequenceVectors - Epoch: [1]; Words vectorized so far: [3267747]; Lines vectorized so far: [300000]; Seq/sec: [455,30]; Words/sec: [4483,80]; learningRate: [0.024899306083990255]
...
15:20:17.762 [VectorCalculationsThread 0] INFO o.d.m.s.SequenceVectors - Epoch: [1]; Words vectorized so far: [101800897]; Lines vectorized so far: [3100000]; Seq/sec: [53,89]; Words/sec: [4293,25]; learningRate: [0.021863066154592867]
15:39:39.156 [VectorCalculationsThread 3] INFO o.d.m.s.SequenceVectors - Epoch: [1]; Words vectorized so far: [106697973]; Lines vectorized so far: [3200000]; Seq/sec: [86,10]; Words/sec: [4289,67]; learningRate: [0.02171216087985193]
15:59:06.222 [VectorCalculationsThread 3] INFO o.d.m.s.SequenceVectors - Epoch: [1]; Words vectorized so far: [111588408]; Lines vectorized so far: [3300000]; Seq/sec: [85,68]; Words/sec: [4285,22]; learningRate: [0.021561457378555856]
16:24:31.347 [VectorCalculationsThread 2] INFO o.d.m.s.SequenceVectors - Epoch: [1]; Words vectorized so far: [117830242]; Lines vectorized so far: [3400000]; Seq/sec: [65,57]; Words/sec: [4274,56]; learningRate: [0.02136912956863643]