1 Introduction

Background. Advancement in convolutional network architecture and training techniques in recent years has enabled the automation of high-level image processing tasks, such as image segmentation [1, 2], image reconstruction and manipulation [3, 4], and fashion-related applications [5, 6, 7].

Fashion editing. Fashion image editing aims to modify real fashion images with user-provided color sketches to produce realistic and pleasing results. As opposed to commercial image-editing software such as Adobe Photoshop which requires a substantial amount of human skill and labour to achieve similar results, a fashion editing network, which can gain a high-level semantic understanding of the images via training of network parameters, enables low-effort manipulation of fashion images without prior experience in fashion design.

Fashion editing network and application. In this project, the architecture of a fashion editing network Fashion Editing Generative Adversarial Network (FE-GAN) [5] is reviewed, and a fashion editing GUI application is designed to demonstrate ways to achieve a more feature-rich and convenient experience in interactive manipulation of fashion images. Various tools and controls features are proposed for the application, including mask morphing, edge detection, skin color detection, zoom-in and export facilities, predefined patterns, keyboard shortcuts and checkerboard fill background.

Experiment with segmentation network on embedded device. Furthermore, in light of the increasing adoption of convolutional networks in embedded devices for daily life applications, an experiment is conducted to quantize and convert convolutional networks for inference on a commercial small single-board computer equipped with specialized hardware for tensor operations. In addition, the limitation in computing capability and performance of such embedded systems motivates further improvements in network quantization and architecture optimization to increase inference performance. A modification in encoder-decoder is proposed for segmentation network SegNet [8] and a small training experiment is performed. The qualitative and quantitive performance of the modified network is then evaluated. The modified network is further quantized and converted for inference on the single-board computer hardware for evaluation.

2 Fashion Editing Network

Fashion dataset. The DeepFashion [9] dataset, which consists of high-resolution professionally shot fashion images with varying pose, viewpoint and background, is used in the training and testing of the fashion editing network.

Generative Adversarial Network. A simple description of Generative Adversarial Network (GAN), which forms the base architecture of Fashion Editing Generative Adversarial Network (FE-GAN) [5, 6], is given as follows with an illustration provided in Figure 2. The network consists of a generator, which is an encoder-decoder network with an hour-glass shape, and a discriminator. Note that a noise layer or channel is usually concatenated with the input image channel(s) along the channel dimension.

Layers of input to fashion editing network. In a fashion editing network, a total of 10 channels with the same dimension as the input image are concatenated along the channel dimension, including the original RGB image (3 channels), a binary mask representing the erased portions (1 channel), a binary outline sketch describing structure of parts to be reconstructed (1 channel), a RGB color stroke map (3 channels), a synthesized new segmentation map for the intended RGB image result (1 channel) and a Gaussian noise (1 channel). Note that the Gaussian noise is essential in the training of GANs for non-determinism and better convergence.

Synthesized segmentation mask. After user modification to the fashion image, the correct segmentation labels for the edited areas may differ from that in the segmentation for the unedited image. For example, a change in correct segmentation label may result from a sketch and mask combination intended to replace pixels depicting clothing with synthesized pixels depicting skin, or vice versa. Therefore, a new segmentation label is synthesized to assign category labels inferred from the segmentation label of the unchanged neighboring areas and user-supplied outline sketches.

Post-processing of network output. Unmasked parts of the fashion image are not guaranteed to be the preserved by the network, and therefore such parts are replaced with the corresponding parts from the original image by masking operations. However, such operations may produce a resulting image where apparently artificial modification or addition is applied. To improve the visual appearance of the resulting image, it is proposed that a seamless combination of the original image and network output image can be obtained by using a Gaussian blurred mask for combination to ensure a smooth transition at the boundary of masked areas.

Limitation of network. The fashion editing network FE-GAN is limited in producing desirable realistic results for fashion images with complex patterns and inaccurate sketches. For example, a sketch without gradient color paint for edited areas for a gradient colored fashion item would lead to a resulting image that looks unrealistic. The network works better on images of fashion items with simple designs.

3 Fashion Editing GUI Application Design

A Graphical User Interface (GUI) application design is proposed to achieve a more feature-rich and convenient experience in interactive manipulation of fashion images, compared to the pre-existing primitive demo application adapted from [10].

Figure 3 shows part of an edited input image seen by the user in the editing window and the corresponding output of the network. Note that the edited input image is a rendering of separate layers including image, mask, sketch and color stroke.

Input	Output

Figure 3: Fashion editing network input and output example.

3.1 Tools

The main tools of the demo application are to be restructured and renamed as Eraser, Pencil and Paint for intuitive understanding by users.

Eraser, which is intuitively presented to users as removing parts of an image, produces a mask layer to be used as input to the fashion editing network.

• Mask erosion and dilation. A new supplementary tool is proposed for mask fine-tuning with image morphological functions erosion and dilation [11], which can be implemented by calling functions in the OpenCV library. This tool can help users easily correct hand-drawn mask to cover a smaller and larger area respectively to achieve intended mask coverage area.

Pencil, which is intuitively presented to users as sketching black outlines on the image, produces a sketch layer to be used as input to the fashion editing network.

• Edge pre-generation. A new supplementary tool is proposed for filling a specified erased region with pre-generated of black outlines with Canny edge detector [12] as illustrated in Figure 4. This tool can save users from spending unnecessary effort in redrawing the outlines in the case that the same outlines as for the original image is intended for erased areas before coloring.

  This tool may be further improved by replacing Canny edge detector with a deep learned semantic edge detector such as holistically-nested edge detection (HED) [13], provided that the outline layers used in training are also generated with such detector or hand-drawn to match its outline structure composed of thick and thin lines.

Paint, which is intuitively presented to users as putting colors picked from color-wheel on the image, produces a stroke layer to be used as input to the fashion editing network.

• Skin area marking. A new supplementary tool is proposed for marking of specific parts as skin in the new synthesized segmentation for the resulting image.

3.2 Controls

Zoom-in facility, which is useful for fine-grained edits that operate on an area spanning only a few pixels, enables an enlarged view of specific parts of a fashion image in the working window by mouse scrolling.

Export facility, which is useful for storing and sharing of sketches and masks in a multi-layer format, enables preservation of the masks and sketches in separate layers for future editing sessions.

Keyboard shortcut support, which is useful for implementing a smooth editing workflow without altering the current cursor position, enables quick access of frequently used tools and controls while drawing.

Fast-rewind for undo and fast-forward redo, which is useful for undo or redoing the drawing of a complex item that is composed of multiple edits and strokes, enables quick undoing or redoing of multiple steps by dragging the handle of a slider. This control feature may be further improved to intelligently group edits and strokes for a semantic undo and redo function.

Predefined patterns, which is useful for quick edits of fashion images with minimal artistic skill and effort, enables one-click stamping application of patterns including (i) textures such as stripes, floral patterns, skin and hair, and (ii) accessory items such as handbags, shoes and hats.

Checkerboard background for erased mask regions as illustrated in Figure 5 that can be switched on from a white background in the edited image, which is useful for displaying of erased portions as different from white-colored pixels, provides intuition for further application of sketches and strokes in the masked regions.

3.3 Application

Doodling application. An interesting side application of the fashion editing network or a similar generative network is a free-drawing application with automatic conversion of user-supplied sketch and paint (preferably of a similar style as training images) on a completely masked canvas (effectively a blank background) to a completely filled realistic looking image. An example is shown in Figure 6¹.

Colorization application. In addition, a colorization application can be achieved by providing pre-generated edges of fashion images as templates for free drawing with color strokes.

Input	Output

Figure 6: Fashion editing network input and output example.

4 An Experiment in Adaptation of Network to Embedded Device

Motivation. Embedded systems equipped with special hardware for tensor operations enable low-cost, server-less and mobile applications of convolutional networks. To explore the possibility of porting the GUI application proposed to run locally on embedded device systems, and, in particular, the possibility of porting the convolutional networks as backend of the application to embedded device systems, an experiment is performed on a simple convolutional network for segmentation, with the aim to eventually convert a complex convolutional network such as the fashion editing network with more complicated layers for inference on embedded systems.

Embedded device. A commercial small single-board computer equipped with specialized hardware customized for tensor operations², which alleviates the limitation of such single-board computer in computing capability and performance, is selected for this experiment.

4.1 Convolutional Network

Segmentation network. The network chosen in this experiment is a single unified segmentation network SegNet [8], which can be used for on-device segmentation of user-provided photographic images without the need for the user to supply an accompanying segmentation mask before further applications such as fashion editing. SegNet (2016) has an encoder-decoder architecture improved with the introduction of Unpooling boolean mask and skip connections to combine semantic information of large and small receptive fields.

Fashion segmentation dataset. Fashion images and labels are obtained from the CIHP (Crowd Instance-level Human Parsing) dataset [19] for human parsing/segmentation tasks, which includes 28280 training, 5000 validation and 5000 test images cropped multi-person instances from Microsoft COCO dataset [20]. Note that in the labeling of training data, the 20 category definitions proposed for Look Into Person [21] and CIHP (Crowd Instance-level Human Parsing) dataset [19], as follows: __Background, __Hat, __Hair, __Glove, __Sunglasses, __Upper-clothes, __Dress, __Coat, __Socks, __Pants, __Jumpsuits, __Scarf, __Skirt, __Face, __Left-arm, __Right-arm, __Left-leg, __Right-leg, __Left-shoe, __Right-shoe.

Motivation for decoder modification. The network is trained with the fashion images and labels, integer-quantized³, and converted to run on the single-board computer. Due to limited support of instructions for the special hardware, including the lack of support for CAST for boolean-to-integer conversion and EQUAL for element-wise equality used in the Unpooling layer [24] in SegNet, the non-trainable Unpooling layer is replaced with a trainable decoder scheme for feature fusion proposed as illustrated in Figure 7.

Feature fusion in decoder. The link to low-dimensional features from a shallow layer effectively functions as a residual connection⁴, which can increase training speed as network converges faster. A 1 × 1 point-wise convolution layer is added on the two branches concatenated along the channel dimension, followed by a ReLU non-linearity to prevent collapse of the 1 × 1 convolution layer into the 3 × 3 convolution layer that follows.

Dimensions and number of parameters. With the modification by introduction of the decoder scheme, total parameter count of the network slightly increased from 29.5 million to 30.7 million for dimension choices: channels C = {64,128,256,512,512}⁵ and height and width H = W = {320,160,80,40,20}.

Comparison with DeepLab v3+ decoder. The proposed decoder scheme as a method for high/low-dimensional feature concatenation is found to be similar to that in DeepLab v3+ (2018) [25]. However, there are two major differences. Firstly, for upsampling, we use nearest neighbor instead of bilinear interpolation, which can perform slightly better but is more computationally expensive [26]. Secondly, after concatenation of high-dimensional features (downsampled in depth and then upsampled in spatial resolution) and low-dimensional features with the same spatial dimension, we use a 1 × 1 convolution to reduce the number of channels followed by ReLU non-linearity before fine-tuning with 3 × 3 convolutions instead of directly applying a 3 × 3 convolution on concatenated input features. Both of these choices reduce computational complexity, and also may be able to better preserve spatial correspondence between the high-dimensional features and low level features.

Potential modifications to be experimented. Note that identity mapping residual connections within each convolution block as in ResNet [27] are not used but may be added for potential further improvement in segmentation. Further experiments are yet to be carried out on human parsing for using the idea of dense connections for deep supervision in DenseNet [28] and its adaption for densely connected skip connections for segmentation tasks in UNet++ [29] with transposed convolution [30] for upsampling.

4.2 Performance Evaluation of Modified Network

Issue that remains for SegNet models. The results from both SegNet (2016) and the modified SegNet show confusion over position labels pairs: Left-arm and Right-arm; Left-leg and Right-leg; Left-shoe and Right-shoe. It is hypothesized that this is due to the lack of explicit modeling of pose information in both networks. New state-of-the-art models such as LIP-JPPNet [21] and Graphonomy [31], which explicitly model pose information or spatial proximity relationship and adopt other boosting techniques, produce much more promising results than SegNet, which is evident by the quantitative figures in Figure 8.

Improvement. The proposed modified SegNet performs qualitatively better than SegNet (2016) for the segmentation task, as observed from examples shown in Figure 9. The results from SegNet (2016) show confusion over the correct label for a pants segment in the first image and for a background segment where a musical instrument is present in the second image. This issue is not seen in results from modified SegNet.

Issue with quantization. The modified network is further integer-quantized and converted for inference on the single-board computer hardware. The qualitative result of inference with the quantized network as exemplified in Figure 10 is unsatisfactory. Large patches of non-background area in the ground truth label are classified as background. It is hypothesized that dead activations result from the accumulation of post-training integer-quantization precision loss over repeated subsampling operations, as the input image goes through a series of decoders in network. This loss of information is an inherent problem of the encoder-decoder architecture. It is suggested that FlatteNet [32], which produces dense pixel-wise predictions for objection detections, may be explored and adapted for pixel-wise segmentation.

5 Conclusion

In this project, the architecture of Fashion Editing Generative Adversarial Network (FE-GAN) is reviewed, the applications of the fashion editing network are explored, and a design for a GUI application of the network with enhanced interactive editing features is proposed. In addition, the possibility of adaptation of convolutional networks to embedded systems is explored through an experiment in converting a modified SegNet to run on a commercial single-board computer with specialized hardware for accelerated network inference, in which unsatisfactory results are obtained indicating rooms for improvement in network design and/or conversion.

6 Future Work

Network architecture improvements. Much work in network design, quantization and/or conversion need to be done to enable successful conversion of large encoder-decoder models for embedded device systems.

An augmented reality fashion application. Real-time applications with video streamed from a camera connected to the single-board computer can be developed to make use of the special hardware network accelerator’s high inference speed for high frame-rate applications of encoder-decoder networks, such as augmented reality applications in social media applications or fashion stores providing virtual clothes try-on and clothing customization, which often involve complicated fashion item designs and inaccurate user input.

An enhanced editing application with hardware for drawing. A graphic drawing tablet⁷ with physical buttons an be found useful in the building of a physical button-controlled user interface, and a high-resolution pen stylus that records cursor hover events and pressure levels can be used with software design updates to create a smooth user experience in fashion editing compared to editing with a mouse connected to a computer.

Acknowledgements

Sincere gratitude is expressed to supervisor Prof. Luo, Ping and his colleagues for their generous advice on the understanding of convolutional networks and help on brainstorming of ideas for this project.

References