Differences

This shows you the differences between two versions of the page.

Link to this comparison view

blog:2019:0210_neural_style [2019/02/13 07:39] (current)
Line 1: Line 1:
 +====== Neural style simulation with tensorflow ======
 +
 +{{tag>​deep_learning}}
 +
 +I've been away from my blog a bit too long already, but well, the past is the past, and we can't change it. So let's focus on the present and future anyway! Today I wan't to give a try to the so called **"​neural style"​** concept, where we can input an image, and say a painting with a "given style",​ and get that style applied on the input image.
 +
 +====== ======
 +
 +===== References =====
 +
 +  * https://​www.youtube.com/​watch?​v=LoePx3QC5Js
 +  * https://​deepart.io
 +  * https://​github.com/​hwalsuklee/​tensorflow-style-transfer => this is the implementation we will use as reference (it provides links to the original paper)
 +  * https://​github.com/​lengstrom/​fast-style-transfer
 +
 +===== Pre-requisite steps =====
 +
 +  * The first ste we take is to download the reference weights from [[http://​www.vlfeat.org/​matconvnet/​models/​imagenet-vgg-verydeep-19.mat|imagenet-vgg-verydeep-19.mat]]
 +  * Also we need to ensure we have the python packages: **numpy**, **scipy**, **Pillow**, **matplotlib**
 +
 +===== Initial tests =====
 +
 +  * So I built a minimal test front end, which can be executed with my custom command: <​code>​nv_style_transfer --content images/​tubingen.jpg --style images/​starry-night.jpg --output result.jpg</​code>​
 +
 +  * Unfortunately,​ the first result I got was the exception: <​code> ​   import utils
 +  File "/​mnt/​array1/​dev/​projects/​NervSeed/​python/​apps/​deep_learning/​style_transfer/​utils.py",​ line 3, in <​module>​
 +    import matplotlib.pyplot as plt
 +  File "/​mnt/​array1/​dev/​projects/​NervSeed/​tools/​linux/​python-3.6/​lib/​python3.6/​site-packages/​matplotlib/​pyplot.py",​ line 113, in <​module>​
 +    _backend_mod,​ new_figure_manager,​ draw_if_interactive,​ _show = pylab_setup()
 +  File "/​mnt/​array1/​dev/​projects/​NervSeed/​tools/​linux/​python-3.6/​lib/​python3.6/​site-packages/​matplotlib/​backends/​__init__.py",​ line 60, in pylab_setup
 +    [backend_name],​ 0)
 +  File "/​mnt/​array1/​dev/​projects/​NervSeed/​tools/​linux/​python-3.6/​lib/​python3.6/​site-packages/​matplotlib/​backends/​backend_tkagg.py",​ line 6, in <​module>​
 +    from six.moves import tkinter as Tk
 +  File "/​mnt/​array1/​dev/​projects/​NervSeed/​tools/​linux/​python-3.6/​lib/​python3.6/​site-packages/​six.py",​ line 92, in __get__
 +    result = self._resolve()
 +  File "/​mnt/​array1/​dev/​projects/​NervSeed/​tools/​linux/​python-3.6/​lib/​python3.6/​site-packages/​six.py",​ line 115, in _resolve
 +    return _import_module(self.mod)
 +  File "/​mnt/​array1/​dev/​projects/​NervSeed/​tools/​linux/​python-3.6/​lib/​python3.6/​site-packages/​six.py",​ line 82, in _import_module
 +    __import__(name)
 +  File "/​mnt/​array1/​dev/​projects/​NervSeed/​tools/​linux/​python-3.6/​lib/​python3.6/​tkinter/​__init__.py",​ line 36, in <​module>​
 +    import _tkinter # If this fails your Python may not be configured for Tk
 +ModuleNotFoundError:​ No module named '​_tkinter'</​code>​
 +
 +  * Now trying to get this Tk dependency... But, unfortunately,​ it seems this is something we should build python with.
 +  * Now trying to see if we can remove this dependency usage... **OK** actually the matplotlib dependency is only used to display the images for debugging, so I simply commented this module import in the **utils.py** file and also the **plot_images** function in that same file. Then the app seems to run as expected and I can see the training process going on. And finally I got my first style transfer result image :-)
 +
 +{{ blog:​2019:​0211:​first_style_transfer.jpg?​512 }}
 +
 +=> I know I know, that one is pretty standard result ;-) So then I moved to more personal content (ie. family picture) with a custom painting style.
 +
 +===== Family picture styling =====
 +
 +  * I first tried with the command: <​code>​nv_style_transfer --max_size 1920 --content /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​content/​famille_cap_antibes.jpg --style /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​styles/​colorful_sunrise.jpg --output /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​family_colorful_sunrise.jpg</​code>​
 +
 +  * => unfortunately,​ this didn't work on my dual 1070 config: I run out of **GPU memory**... :-( So it seems generating 1080p images is still a bit out of reach for me... Maybe if I try on my GTX 1080 instead ?
 +
 +  * So now trying with 720p output images: <​code>​nv_style_transfer --max_size 1280 --content /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​content/​famille_cap_antibes.jpg --style /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​styles/​colorful_sunrise.jpg --output /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​family_colorful_sunrise.jpg</​code>​
 +
 +  * And the first "​custom result"​ I got didn't look so great:
 +
 +{{ blog:​2019:​0211:​family_colorful_sunrise_1000.jpg?​600 }}
 +
 +  * Note that the style for that experiment came from the painting below:
 +
 +{{ blog:​2019:​0211:​colorful_sunrise.jpg?​600 }}
 +
 +  * Not quite a wonderfull success, isn't it ?... Yet, the result above was achieved with only 1000 training iterations, and the loss was still going "​largely down" when the training was stopped. So I decided to give it another try, this time with 5000 iterations instead to see if this could help producing a more convincing style transfer image :-)
 +
 +  * The second result I got with those 5000 iterations was:
 +
 +{{ blog:​2019:​0211:​family_colorful_sunrise_5000.jpg?​600 }}
 +
 +=> Still not quite an amazing image, but at least some of the small artifacts from the previous rendering were removed...
 +
 +  * At that point I realized that the training was taking soooo long because... my GPUs were actually busy mining crypto currencies at the same time !!! OOooops... ;-) So, maybe I can in fact produce 1080p images afterall ? Anyway, I decided to stick with 720p resolution and instead try with another painting style, which was the following:
 +
 +{{ blog:​2019:​0211:​paysage_texture.jpg?​600 }}
 +
 +
 +  * And the result I got (this time with +6000 iterations) was:
 +
 +{{ blog:​2019:​0211:​family_paysage_texture.jpg?​600 }}
 +
 +=> Again, the result is "​nice"​ but still not really terrific... In fact I'm starting "to feel" that what we really need as a "​style"​ image is not really a "​colorfull"​ painting but instead something with very specific "​shapes"​ instead. So let's try to find that now.
 +
 +  * I found this abstract painting, which I decided to use:
 +
 +{{ blog:​2019:​0211:​abstract_painting.jpg?​300 }}
 +
 +  * And the result, with "​only"​ 3000 iterations seems much more convincing to me (even if we still have a few artifacts... But I'm sure this can be fixed with some more training):
 +
 +{{ blog:​2019:​0211:​family_abstract_painting.jpg?​600 }}
 +
 +/*
 +nv_style_transfer --max_size 1280 --num_iter 6000 --content /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​content/​famille_cap_antibes.jpg --style /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​styles/​paysage_texture.jpg --output /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​family_paysage_texture.jpg
 +
 +
 +nv_style_transfer --max_size 1280 --num_iter 3000 --content /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​content/​famille_cap_antibes.jpg --style /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​styles/​abstract_painting.jpg --output /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​family_abstract_painting.jpg
 +
 +nv_style_transfer --max_size 1280 --num_iter 6000 --content /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​content/​chatte.jpg --style /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​styles/​nude_grayscale.jpg --output /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​chatte_grayscale.jpg
 +
 +nv_style_transfer --max_size 1280 --num_iter 6000 --content /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​content/​chatte2.jpg --style /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​styles/​cubism.jpg --output /​mnt/​array1/​dev/​data/​deep_learning/​style_transfer/​chatte2_cubism.jpg --loss_ratio 0.0007
 +*/
 +
 +<​note>​By the way, I also tried once more to generate a 1080p image [with mining stopped this time!] but unfortunately I still get the same "out of GPU memory"​ error.</​note>​
 +
 +===== Future tests =====
 +
 +  * As mentioned in the reference section above, there is actually a **fast style transfer** implementation that can apparently produce similar results much faster. I should definitely give this a try one day... But not today ;-).
 +