Was not realizing, that training of DNN with GPU is so faster, than with CPU.
I have old GTX770 card which is reported to have 3,213 GFLOPS.
My main CPU is Core i7-4770K 3.5 GHz, which is reported to have 177 GFLOPS.
It looks like my GPU should be 18 times faster than CPU.
Actually, I don’t observe this kind of speed. I wrote simple convolutional model for CIFAR-10 database and observed that is works approximately 7 times faster on GPU than on CPU (by eye).