| |
— | blog:2019:0210_neural_style [2020/07/10 12:11] (current) – created - external edit 127.0.0.1 |
---|
| ====== Neural style simulation with tensorflow ====== |
| |
| {{tag>deep_learning}} |
| |
| I've been away from my blog a bit too long already, but well, the past is the past, and we can't change it. So let's focus on the present and future anyway! Today I wan't to give a try to the so called **"neural style"** concept, where we can input an image, and say a painting with a "given style", and get that style applied on the input image. |
| |
| ====== ====== |
| |
| ===== References ===== |
| |
| * https://www.youtube.com/watch?v=LoePx3QC5Js |
| * https://deepart.io |
| * https://github.com/hwalsuklee/tensorflow-style-transfer => this is the implementation we will use as reference (it provides links to the original paper) |
| * https://github.com/lengstrom/fast-style-transfer |
| |
| ===== Pre-requisite steps ===== |
| |
| * The first ste we take is to download the reference weights from [[http://www.vlfeat.org/matconvnet/models/imagenet-vgg-verydeep-19.mat|imagenet-vgg-verydeep-19.mat]] |
| * Also we need to ensure we have the python packages: **numpy**, **scipy**, **Pillow**, **matplotlib** |
| |
| ===== Initial tests ===== |
| |
| * So I built a minimal test front end, which can be executed with my custom command: <code>nv_style_transfer --content images/tubingen.jpg --style images/starry-night.jpg --output result.jpg</code> |
| |
| * Unfortunately, the first result I got was the exception: <code> import utils |
| File "/mnt/array1/dev/projects/NervSeed/python/apps/deep_learning/style_transfer/utils.py", line 3, in <module> |
| import matplotlib.pyplot as plt |
| File "/mnt/array1/dev/projects/NervSeed/tools/linux/python-3.6/lib/python3.6/site-packages/matplotlib/pyplot.py", line 113, in <module> |
| _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup() |
| File "/mnt/array1/dev/projects/NervSeed/tools/linux/python-3.6/lib/python3.6/site-packages/matplotlib/backends/__init__.py", line 60, in pylab_setup |
| [backend_name], 0) |
| File "/mnt/array1/dev/projects/NervSeed/tools/linux/python-3.6/lib/python3.6/site-packages/matplotlib/backends/backend_tkagg.py", line 6, in <module> |
| from six.moves import tkinter as Tk |
| File "/mnt/array1/dev/projects/NervSeed/tools/linux/python-3.6/lib/python3.6/site-packages/six.py", line 92, in __get__ |
| result = self._resolve() |
| File "/mnt/array1/dev/projects/NervSeed/tools/linux/python-3.6/lib/python3.6/site-packages/six.py", line 115, in _resolve |
| return _import_module(self.mod) |
| File "/mnt/array1/dev/projects/NervSeed/tools/linux/python-3.6/lib/python3.6/site-packages/six.py", line 82, in _import_module |
| __import__(name) |
| File "/mnt/array1/dev/projects/NervSeed/tools/linux/python-3.6/lib/python3.6/tkinter/__init__.py", line 36, in <module> |
| import _tkinter # If this fails your Python may not be configured for Tk |
| ModuleNotFoundError: No module named '_tkinter'</code> |
| |
| * Now trying to get this Tk dependency... But, unfortunately, it seems this is something we should build python with. |
| * Now trying to see if we can remove this dependency usage... **OK** actually the matplotlib dependency is only used to display the images for debugging, so I simply commented this module import in the **utils.py** file and also the **plot_images** function in that same file. Then the app seems to run as expected and I can see the training process going on. And finally I got my first style transfer result image :-) |
| |
| {{ blog:2019:0211:first_style_transfer.jpg?512 }} |
| |
| => I know I know, that one is pretty standard result ;-) So then I moved to more personal content (ie. family picture) with a custom painting style. |
| |
| ===== Family picture styling ===== |
| |
| * I first tried with the command: <code>nv_style_transfer --max_size 1920 --content /mnt/array1/dev/data/deep_learning/style_transfer/content/famille_cap_antibes.jpg --style /mnt/array1/dev/data/deep_learning/style_transfer/styles/colorful_sunrise.jpg --output /mnt/array1/dev/data/deep_learning/style_transfer/family_colorful_sunrise.jpg</code> |
| |
| * => unfortunately, this didn't work on my dual 1070 config: I run out of **GPU memory**... :-( So it seems generating 1080p images is still a bit out of reach for me... Maybe if I try on my GTX 1080 instead ? |
| |
| * So now trying with 720p output images: <code>nv_style_transfer --max_size 1280 --content /mnt/array1/dev/data/deep_learning/style_transfer/content/famille_cap_antibes.jpg --style /mnt/array1/dev/data/deep_learning/style_transfer/styles/colorful_sunrise.jpg --output /mnt/array1/dev/data/deep_learning/style_transfer/family_colorful_sunrise.jpg</code> |
| |
| * And the first "custom result" I got didn't look so great: |
| |
| {{ blog:2019:0211:family_colorful_sunrise_1000.jpg?600 }} |
| |
| * Note that the style for that experiment came from the painting below: |
| |
| {{ blog:2019:0211:colorful_sunrise.jpg?600 }} |
| |
| * Not quite a wonderfull success, isn't it ?... Yet, the result above was achieved with only 1000 training iterations, and the loss was still going "largely down" when the training was stopped. So I decided to give it another try, this time with 5000 iterations instead to see if this could help producing a more convincing style transfer image :-) |
| |
| * The second result I got with those 5000 iterations was: |
| |
| {{ blog:2019:0211:family_colorful_sunrise_5000.jpg?600 }} |
| |
| => Still not quite an amazing image, but at least some of the small artifacts from the previous rendering were removed... |
| |
| * At that point I realized that the training was taking soooo long because... my GPUs were actually busy mining crypto currencies at the same time !!! OOooops... ;-) So, maybe I can in fact produce 1080p images afterall ? Anyway, I decided to stick with 720p resolution and instead try with another painting style, which was the following: |
| |
| {{ blog:2019:0211:paysage_texture.jpg?600 }} |
| |
| |
| * And the result I got (this time with +6000 iterations) was: |
| |
| {{ blog:2019:0211:family_paysage_texture.jpg?600 }} |
| |
| => Again, the result is "nice" but still not really terrific... In fact I'm starting "to feel" that what we really need as a "style" image is not really a "colorfull" painting but instead something with very specific "shapes" instead. So let's try to find that now. |
| |
| * I found this abstract painting, which I decided to use: |
| |
| {{ blog:2019:0211:abstract_painting.jpg?300 }} |
| |
| * And the result, with "only" 3000 iterations seems much more convincing to me (even if we still have a few artifacts... But I'm sure this can be fixed with some more training): |
| |
| {{ blog:2019:0211:family_abstract_painting.jpg?600 }} |
| |
| /* |
| nv_style_transfer --max_size 1280 --num_iter 6000 --content /mnt/array1/dev/data/deep_learning/style_transfer/content/famille_cap_antibes.jpg --style /mnt/array1/dev/data/deep_learning/style_transfer/styles/paysage_texture.jpg --output /mnt/array1/dev/data/deep_learning/style_transfer/family_paysage_texture.jpg |
| |
| |
| nv_style_transfer --max_size 1280 --num_iter 3000 --content /mnt/array1/dev/data/deep_learning/style_transfer/content/famille_cap_antibes.jpg --style /mnt/array1/dev/data/deep_learning/style_transfer/styles/abstract_painting.jpg --output /mnt/array1/dev/data/deep_learning/style_transfer/family_abstract_painting.jpg |
| |
| nv_style_transfer --max_size 1280 --num_iter 6000 --content /mnt/array1/dev/data/deep_learning/style_transfer/content/chatte.jpg --style /mnt/array1/dev/data/deep_learning/style_transfer/styles/nude_grayscale.jpg --output /mnt/array1/dev/data/deep_learning/style_transfer/chatte_grayscale.jpg |
| |
| nv_style_transfer --max_size 1280 --num_iter 6000 --content /mnt/array1/dev/data/deep_learning/style_transfer/content/chatte2.jpg --style /mnt/array1/dev/data/deep_learning/style_transfer/styles/cubism.jpg --output /mnt/array1/dev/data/deep_learning/style_transfer/chatte2_cubism.jpg --loss_ratio 0.0007 |
| */ |
| |
| <note>By the way, I also tried once more to generate a 1080p image [with mining stopped this time!] but unfortunately I still get the same "out of GPU memory" error.</note> |
| |
| ===== Future tests ===== |
| |
| * As mentioned in the reference section above, there is actually a **fast style transfer** implementation that can apparently produce similar results much faster. I should definitely give this a try one day... But not today ;-). |
| |
| |