Active Learning over DNN: Automated Engineering Design Optimization for Fluid Dynamics Based on Self-Simulated Dataset

2020·Arxiv

Abstract

Abstract

Optimizing fluid-dynamic performance is an important engineering task. Traditionally, experts design shapes based on empirical estimations and verify them through expensive experiments. This costly process, both in terms of time and space, may only explore a limited number of shapes and lead to sub-optimal designs. In this research, a test-proven deep learning architecture is applied to predict the performance under various restrictions and search for better shapes by optimizing the learned prediction function. The major challenge is the vast amount of data points Deep Neural Network (DNN) demands, which is improvident to simulate. To remedy this drawback, a Frequentist active learning is used to explore regions of the output space that DNN predicts promising. This operation reduces the number of data samples demanded from 8000 (Stephan Eis- mann, 2017) to 625. The final stage, a user interface, made the model capable of optimizing with given user input of minimum area and viscosity. Flood fill is used to define a boundary area function so that the optimal shape does not bypass the minimum area. Stochastic Gradient Langevin Dynamics (SGLD) is employed to make sure the ultimate shape is optimized while circumventing the required area. Jointly, shapes with extremely low drags are found explored by a practical user interface with no human domain knowledge and modest computation overhead.

1 Introduction

Fluid Mechanics is a resource consuming process. NavierStokes Equation does not have closed form solutions because of its chaotic nature. As for its numerical solutions, a general procedure is absent.

Traditionally, the method of fluid dynamic optimization is for the experts to estimate based on empirical accounts and test potential possibilities with Wind Tunnel Testing (WTT). This approach is both costly and uncertain, as the prototyping involves the excessive use of building materials and the final result may not be proved the most optimized due to human incompleteness.

New attempts to expedite the design include replacing WTT with computational simulations, using, for instance, MATLAB. But, still, the numerical solutions for NavierStokes are unstable and the drawbacks for empirical design remain. Automation attempts are represented by Bayesian optimization (Stephan Eismann, 2017). In this study, a Bayesian optimization model is applied to automate the design process. However, their model is sample-heavy, which leads to the undesirably exorbitant simulation cost.

To remedy these drawbacks, the presented study explores design automatic algorithms that design shapes with desired aerodynamic performance with reduced human input and computation consumption. Incorporating active learning ideas with DNN to map the shape and drag coefficient samples from input, in the form as latent dimensions, the model in this study is able to search for optimal shapes. A Frequentist active learning method facilitates the model fit-ting by focusing on only the potential output space regions, reducing the cost for producing excessive data points.

This research differs from and builds on top of the previous studies in the following ways:

1. MATLAB is utilized to form a new self-simulated dataset that includes various settings and object shapes, creating a template that can generate physical models that can reduce the costs of actually making desired objects and evaluating the attributes of them in reality. This approach is both less time-consuming and costly material-wise.

2. A new system of algorithm is raised to make the process of engineering optimization as automatic, and computational and sample economic as possible. DNN forms the core of the presented system with other self-determinant

conditions to decide where the search should focus on. In such way, we make sure that the best fitting of the correlation between s and drags can be found, while other problems, such as local minimum, out-of-the-reasonable-range dilemma and so on, shall be avoided.

3. The manner machine learning is applied in aerodynamic engineering optimization is new. The latest such work shows a different approach to the problem (Stephan Eis- mann, 2017). The simulation process is different in that a template to simulate the physical properties of certain fluid environments is targeted in thie research. We employ DNN that only gives moderate consideration to bias to find the correlation between and drag with a much smaller dataset, while still being able to predict on an comparably accurate basis. Other works are more engineering-based, not showing significant contributions to automate such industrial design.

2 RELATED WORKS

Fluid dynamics focuses on the study of moving air or fluid (Dunbar, 2015). It investigates the interactions between fluid environment and solid object that moves through (Timmermans, 2015). It is important because of its vital applications in fields such as aerospace engineering and vehicle production (NASA, 2016). By studying the effects of fluid moving past a solid object, engineers can optimize their designs of aerodynamic machines (Shieh, 2009). One core objective in this process is to minimize the drag of solid objects under a number of realistic restraints (Lee- ham, 2017). However, the key equation, Navier-Stokes Equation itself that fluid mechanic works are based on has problems. Not only does it lack a set of closed-form solutions because of its chaotic nature, but the general rules of solving the equation numerically also does not exist (Fef- ferman, 2017). Moreover, the equation remains one of the six mathematically unproven hypothesis listed by Clay Mathematics Institute (Jaffe, 2000). These controversies mean that experts have to design shapes based off of their experience and verify estimated shapes in a simulator or wind tunnel test (WTT) for multiple rounds.

Figure 1: Wind tunnel concept. A picture that illustrates a wind tunnel that tests the stream line design and many

In a WTT, the subject of the test is placed in an open chamber. Examining how air propelled by a specialized fan would flow through the subject (Smith, 2008) pulls together a complete picture of the aerodynamic forces and other physical conditions on the model (Rossiter, 1964). Nevertheless, such a expensive method still requires expert knowledge. Results of such empirical estimations may not even be accurate after repeated prototyping. In general, the overall relationships between the shapes and drags are costly to search for under realistically given restrictions (Jenkins et al., 2016).

A group of researchers from Stanford University uses Bayesian optimization to accomplish this goal. Compared with the usual human search (J. Snoek & Adams., 2012b), the Bayesian model requires much fewer samples (Stephan Eismann, 2017). This late research in engineering optimization utilizes a statistical model to ease the complex enumeration process of samples while applying Bayesian optimization to find desirable aerodynamic design.

Figure 2: Bayesian Optimization Fitting Process, Performed as a Form of Active Learning

Bayesian Optimization is also a symbolic representative of the application of active learning in Machine Learning (Martin Pelikan, 1999). It is by nature a Gausian method trained with parameter families based on

As is seen in Figure 2 (Kaul et al., 2017). The regression is narrowed down sequentially with the addition of every new data point in a specific region that would provide the most advancement of regression fitting. This method sig-nificantly reduces the required number of data points by actively searching for desirable new input data for further training. However, Eismann’s study is largely based on processing the simulated two dimensional images, which, because Bayesian model often shows an exponential increase of complexity with additional dimensions, requires a higher number of training data points to achieve complete training.

3 METHODOLOGY

The core of our research is a DNN incorporated into a self-determinant loop that checks conditions and produces the optimized shape of the object under any given circumstance, which is set in MATLAB.

Figure 3: Flow of the system, procedure outline. self-produced.

Figure 3 shows the flow of the process. Our research focuses on making a comprehensive system that automatically finds an optimized shape under any given fluid environment. The process of aerodynamic optimization is sim-plified and made more digital based.

3.1 Physical Simulation with MATLAB

Resistances of objects of certain shapes in fluid, such as air and water, are studied in order for this research to be generalizable. The performance of individual design is defined by its drag coefficients () (Landau & Lifshitz, October, 2013), which stand for the resistance an object encounters in fluid dynamic environment.

For reference, stands for the drag force created by the fluid environment in the direction contrast the movement of the object. serves as the drag coefficient that depends on velocity, viscosity, and other parameters of the reference area. and V are all environmental factors that reflect the reference area, density of the fluid and flow velocity relative to the object, respectively.

Incompressible flow in a volume satisfies the Navier-Stokes equation (Constantin & Foias, 1988).

In this equation:

1. u represents the velocity of the flow, and is a vector field for 3-dimensional flow problems, or for 2-dimensional flow problems.

2. p represents the pressure in the volume, and is a function .

3. v is the viscosity of the fluid (in this study, we set the viscosity to be v = 0.2 as fluid water under room temperature to generalize our study).

The Navier-Stokes equation is the equivalent of Newtons second law for fluids. To interpret the equation, we remark that the change in the flow with respect to time, and represents the convective acceleration of the fluid. The right hand side represents the forces acting on the fluid: is the difference between velocity of a point and the mean velocity of its neighborhood. This term encourages the vector field to become uniform in the absence of other influence factors. is the gradient of the pressure, and drives fluid motion.

MATLAB is sued to simulate the dynamics of an object traveling through a fluid environment. To simulate the object, a geometry that represents the object and relevant boundary conditions is created. Then we convert the geometry into a mesh that represents the state of each point in the system. QuickerSim simulates fluid dynamics according to Navier-Stokes equation, and read off end results including drag through the provided API. Based on this setup, MATLAB generates a dataset of shapes and their corresponding drags.

3.2 Fitting the Model

3.2.1 Linear Regression and Cross Validation

Linear regression is a linear combination of the features (Freedman, 2009). Linear regression expresses itself in the form below:

It is also seen in the form as a matrix:

The loss function of such regression method is calculated in terms of the residual sum of squares, which is written in the expression as following:

This model is used first to approximate the correlation between the actual drag values. We choose to use such model because of its relatively low computational complexity as well as its ability to straightforwardly graph the relationship between actual and predicted drag values, if there is any. Later, by fitting our four-dimensional data again with DNN, we can directly visualize if there is any veritable improvement in our new model compared to the traditional ones.

Figure 4: Linear regression result compared to real drag values, while object width is set to 0.18. self-produced.

Figure 4 represents how the linear regression result fits the real drag values. The closer all the blue dots are to the red curve, f(x) = x, the more accurate the linear regression prediction is. Some pattern is reflected on a primitive level as showns in Figure 4

Cross-validation is a statistic model, also known as rotation estimation, or out of sample testing (Geisser, 1993). The method tests the generalizability of certain predictions, which serves our purpose of the research well. In our research, the entire dataset of 625 -drag pairs all directly serve as training data. Every four-dimensional has a corresponding drag label.

3.2.2 Deep Neural Network (DNN)

Such a more expensive model as feed-forward DNN compared with other of the kind is suitable in our case, because our required computation is not as complicated as usual image processing tasks for two reasons. As is stated above in the MATLAB simulation explanation, we only require 625 samples to reach a fair prediction that is generalizable. This number is significantly smaller than that of digital image processing which usually desire around 10000 two-dimensional samples. Moreover, our input data consist of a list of arrays, which has a lower computational cost than two-dimensional images. Therefore, because of the remarkably lower computational cost as a nature of our dataset, DNN is feasible.

DNN is composed of Fully-Connected layer (FCL) which can be expressed in the following form:

K represents a specific layer of the DNN, while j represents the specific neuron the denoted variables are referring to. W serves as a parameter that tells the connection between the neuron on layer and the neuron on layer, alongside b as the bias adjuster. As the shape indicates, DNN can somewhat be regarded as a CNN in computation or thinking.

The formula can be extended to look something as the series of equations below:

While training, suitable activation functions is necessary. Rectified Linear Units (ReLU), represented by the expression below, is suitable in this model:

(Vinod Nair, 2010). ReLU is a fairly popular kind of activation function in neural network.

ReLU, a single-sided function, has a constant-valued slope when so that it does not have sigmoids predicament of vanishing gradient. In the case of ReLU, only multiplications and comparisons are processed so that we may achieve a faster and more accountable convergence of results.

3.3 Automated Search for Drag-Reduced Shapes

The active learning (AL) used in this research is a Frequentist approach. It differs from Gaussian AL with an exponential increase of complexity in accordance with dimensions. In this study, the applied model maps the input space in a manner that only explores the specific regions indicated by DNN so that bias is traded in for a lower computation cost in the equations:

(bias-significant), thus decreasing the required sample size to 625 data points.

3.3.1 Restrictions for Self-Adjusting Parameters

Generally speaking, two parameters determine the way our model trains itself: train step and variable initializations. In the case of following conditions, our model would automatically determine to retrain itself: 1). when the taken derivative is found to be zero; or 2). the model is found to be stuck in a local minimum.

3.3.2 Check for Decreasing Loss

After the first training round, there are two other possibilities, however, if not a single final optimized case is found, which are elaborated below:

Firstly, there may exist no reasonable loss convergence in our tested cases. If the loss does not decrease in a reasonable distribution or is not reduced at all, our system will start another round of training. In this round, the two trainstep cases are le-2 and le-6, and so on until there is a reasonable convergence of loss. Figure 7 shows an example of how our model adjusts on its own in search of optimization:

However, in some cases, it is also possible that there is no liable convergence until it is out of the range for logical train step sizes. At such time, the system would automatically refresh its initializer so that there is a new initialization to avoid zero derivatives.

Secondly, too many reasonable convergences may be produced. On the one hand, to collate the performance accuracies, we juxtapose each of the test and train accuracies in each plausible training, and find out which convergence has the best general performance among all. On the other hand, to compare the stabilities, we take the difference between the level of test and train accuracies of each reasonable convergence respectively. The one that maintains relatively better train and test performances would be selected. Accuracies would be the priority factor of judgement.

3.3.3 Check for Drag-Minimized Shape

Undesirable initialization, incomplete fit of model and other factors unlisted may also lead to sub-optimal shapes. To prevent these situations, the mechanism in Figure 8 illustrates the test if the found optimized shape actually have the minimized drag in the given environment. Firstly, the found shapes drag is the smaller than all existing shapes, it will be added to the original dataset and feedback to the training system for retraining. If the shape is actually also the global minimum in the new model, then it will be selected. Also, if the found optimized shapes drag is not in fact the smallest compared to the previous drag values, then the found value will also be added to the training dataset. Our system would determine to get another round of retrain until a veritable drag-reduced shape is ensured.

3.4 User Interface

The final stage of the research centers around the implementation of a user interface that provides industrial designs based on boundary restriction given as input by the user and produce the optimal shape that fits.

3.4.1 Boundary Function

Industrial users may put in restrictions to produce an ideal shape for their specific requirements, including viscosity of fluid environment, and minimum body room area.

Viscosity Viscosity is defined in MATLAB and the user input section is provided through Python interface. Without further ado, refer to 3.1 MATLAB simulation.

Minimum Area To ensure that the result produced by the model offers enough enclose area to contain user-provided shapes, we apply Flood Fill algorithm as is visualized below in Figure 6 (Torbert, 2016).

With these judgement being made in the process, as the function,

, that maps the vector direction of descent to either 0, meaning the shape is in the boundary, or 1, the shape is not in the boundary.

Figure 6: Flood Fill Visualization (JavaScript, 2019).

This algorithm assures the parameter surrounds a given inner shape. In reality, each square in the figure stands for a pixel: say, if the red dot is a user input, flood fill saturates the area around the dot so that there is no open section where the flood simply penetrates the boundary, which serves as a prerequisite for the minimum shape.

3.4.2 Searching for the Optimal Shape

Figure 7: Optimization with Restrictions, self-produced

A method called Stochastic Gradient Langevin Dynamics (SGLD) is used to find optimization while successfully avoiding bypassing the restricted areas given by the user (Welling & Yee W., 2011). The original gradient descent direction is given by . To avoid it to bypass the restricted area, we use a half Gaussian noise shown by

(it is based on a second order Gaussian/Normal Distribution: ) (Rasmussen, 2004). It is half because it is limited only to form the vectors that point to directions to make sure the vector is descending, which is achieved by

.

The resulting path travelled by these vectors, will travel dynamically with a general path down the gradient direction.

4 EXPERIMENT

4.1 MATLAB Simulation

We use MATLAB to create 3D environmental simulations and shadow the three-dimensional dynamic system into a two-dimensional graph and fit a surface.

Figure 8: MATLAB simulation examples top: Color-coded map represents relative velocity of environment against the object. bottom: Geometrical meshing of surrounding against moving objects. self-produced.

In this case, only 625 sample points is required for each case of our training. Dataset of such size can be simulated with MATLAB on a single local laptop in less than 3h (2.2 GHz Intel Core i7, 16 GB 2400 MHz DDR4).

As Figure 8 shows above, MATLAB template produces a simulation on a random, non-repetitive basis as such, along with their representing force if drag, for further training. B-SPLINE does well to visualize the boundary conditions of the 2D representation created.

4.2 Deep Neural Network Regression

Pre-training selection first takes place to eliminate simulations that is too close to singularity or off-scale to be for meaningful consideration.

At this point, more accurate predictions of drag values from s are desirable. So, we develop a FCDNN built upon six FCL’s which shows an optimized accuracy compared to the peers.

Figure 9: Optimized DNN architecture in the case of 0.18 width. self-produced.

Table 1 below shows examples of comparing performance between networks of different number of layers:

Table 1: Performance of DNN Architectures for Regression Loss

With two width cases four types of deep network architectures, eight different cases indicate where the optimized deep learning architecture can be achieved. As other different cases have been tested, there is a common trend shown by the two set of examples in Table 1. Six-layer DNN has significant improvement of performance in terms of MSE compared to other architectures with fewer layers, while another additional layer shows no perceptible reduction in MSE in the case of 0.18 width and actually increases MSE in the 0.15 width case. Therefore, six-layer DNN has superiority in its notably low MSE and effectiveness.

In both width cases and more, our AL over DNN model with its project-specific architecture is proven to make more accurate prediction in comparison with other traditional regression models, as is shown in Table 2:

Table 2: Performance of Different Models

Table 3 visualizes the comparisons of computation costs of our study and a previous work based on Bayesian optimization. We reduce both simulation and training time by extracting dimensional information from 2D objects through the fitted spline and train with DNN model.

Table 3: Computation Cost Comparison with Related Study

Stacking of multiple different machine learning methods have insignificant boost to the accuracy. Stacking and ensemble function to remove unshared shortcomings of each method, which this study is not a suitable case. As for the better accuracy caused by the intrinsic advantage of other methods themselves, the method shows better performance in most cases through direct comparisons.

So with many factors taken into account, the six-layer FCDNN in this study is concluded an accountable model with consistent performances and high level of accuracies in both train and test. This architecture improves the loss of our prediction to be generally under 0.0005.

4.3 Automated Engineering Optimization

The Frequentist AL boosts the automation of our system. When the search of a drag-minimized shape falls into a local minimum or when the initialization of the training leads to a zero-derivative, the prediction result is not in its best case. Our system succeeds in avoiding such conditions.

Figure 10: Active learning advancement process top: Simulation results of the predicted optimized shape after first round training; bottom: The predicted optimized result after the final round of training. self-produced.

As shown in Figure 10, in this first effort to optimize shape, drag is shown to be 0.394 in Figure 10(a), which apparently is not indeed optimized and is captured by our mechanism. So it automatically goes into the second round of training, that reports the result below. After few rounds of active searches, our system automatically comes to the shape in Figure 10(b), which has a drag of only 0.281, which is then proven to be the actual minimum. This reliability is similarly observed in other cases with widths of 0.18, 0.20, and 0.30, which serves to support the efficacy of this method

5 DISCUSSION

5.1 Summary and Expectations

The deep learning architecture in this research is new in its accuracy and stability performance. It perceptibly reduces the loss of prediction thus increase the efficiency. Visualized with the data above, this presented model is able to predict the relationship between s and drags accurately and precisely.

The application to use AI to find a fit for the correlation between s and drags is new to this research. We map a matrix of four-dimensional arrays to a column vector of one-dimensional drags, instead of finding the fit of the entire process of objects moving through given fluid environment. A relatively accurate result only requires a small data size.

This optimization system filters cases not yet fully trained and makes sure that the optimized shape is actually found with the simulated information. This also means that our system has the ability to avoid bugs in similar traditional algorithms.

5.2 Proposed Future Extensions

The following are potential aspects for our system to dig deeper into the field of work.

5.2.1 So, supplementary parts may be used to boost the engineering performance of the shape. Consequently, a method of finding such appropriate add-on’s may be studied to include into our system. Since we often need to set restrictions for shapes that we are optimizing to serve for specific tasks, the ultimate shape we get may not be close at all to the original global minimum of drag of our trained model.

5.2.2 Other features may also be desired. For example, ability of an object maintaining its current height, stability or agility of an object. With varying purposes of the optimization, these conditions can be set into the MATLAB environment to be run through and tested the same way as we do on the drags.

6 CONCLUSION

In the end, our research successfully demonstrates that a more systematic and automatic aerodynamic engineering optimization is feasible by getting a regression mean error at below 0.0005 for most preset width cases and achieving actually drag-reduced shapes with the fit model through our systematic model. Compared to the previous researches, we have the advantages of requiring less training samples, less computation costs and time while improving the automation of engineering design and avoiding training bugs. On top of what we have already done, detailed additions as described in the discussion section is able to further increase the comprehensiveness of our system. These future bonus shall be achieved in a pretty similar way as we do here with the drags, except with different simulation features.

Our research successfully innovates on two things: finding a trend that relates to drag values, and searching for a drag-minimized shape. This provides a insight into how drag is influenced by objects shape with merely our machine learning prediction. This process can be more straightforward and practical than other methods that attempt to find such correlation.

ACKNOWLEDGEMENTS

The author would like to offer his cordial gratitudes for the assistance of the research mentor, Shengjia Zhao, on this research. He bears constructive advices to the author when challenges and confusions are to be dealt with.

References

Peter Constantin and Ciprian Foias. Navier-stokes equa- tions. University of Chicago Press, 1988.

B. Dunbar. What is aerodynamics? 2015. URL https://www.nasa.gov/audience/ forstudents/k-4/stories/nasa-knows/ what-is-aerodynamics-k4.html.

Charles L. Fefferman. Existence and smoothness of the navierstokes equation. Clay Mathematics, Ins., 2017.

David A. Freedman. Statistical models: Theory and prac- tice. page 26, 2009.

Seymour Geisser. Predictive inference. 1993.

H. Larochelle J. Snoek and R. P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pp. 29512959, 2012b.

Arthur M. Jaffe. The millennium grand challenge in math- ematics. volume 53, pp. 652–660. Clay Mathematics, Ins., 2000.

Eloquent JavaScript. Flood fill. January 2019.

Bruce Jenkins, Principal Analyst, and Ora Research. Time for automotive oems to transition from wind tunnel testing to simulation-driven design. Ora Research, 2016.

Puneith Kaul, Daniel Golovin, and Greg Kochanski. Hy- perparameter tuning in cloud machine learning engine using bayesian optimization. Google Cloud Platform, August 2017.

L. D. Landau and E. M. Lifshitz. Fluid mechanics. Elsevier, October, 2013. ISBN 9781483140506.

L. Leeham. Bjorn’s corner: Aircraft drag reduction, part 2. 2017. URL https: //leehamnews.com/2017/10/27/ bjorns-corner-aircraft-drag-reduction-part-2.

Erick Cant-Paz Martin Pelikan, David E. Goldbergr. Boa: the bayesian optimization algorithm. volume 1, pp. 525– 532. GECCO’99 Proceedings of the 1st Annual Conference on Genetic and Evolutionary Computation, 1999. ISBN 1-55860-611-4.

NASA. Aerodynamics & aeroacoustics. (n.d.). 2016. URL https://www.dantecdynamics.com/ aerodynamics-aeroacoustics.

Carl Edward Rasmussen. Gaussian processes in machine learning. Advanced lectures on machine learning, pp. 63–71, 2004.

Karl Rosaen. K-fold cross-validation. 2018. URL http: //karlrosaen.com/ml/learning-log/ 2016-06-20/.

J.E. Rossiter. Wind tunnel experiments on the flow over rectangular cavities at subsonic and transonic speeds. RAE Technical Report No. 64037, 1964.

Jyh-Sherg Shieh. Fundamentals of fluid mechanics. 2009.

Richard Smith. Wind tunnels and cfd. 2008. URL https://www.symscape.com/blog/ wind-tunnels-and-cfd.

Stefano Ermon Stephan Eismann, Stefan Bartzsch. Shape optimization in laminar flow with a label-guided variational autoencoder. arXiv preprint arXiv:1712.03599v1, 2017.

M. Timmermans. Wind tunnel testing. SHOL, 2015. URL https://www.nlr. org/capabilities/wind-nuisance/ shol-wind-tunnel-testing/.

Shane Torbert. Applied computer science (2nd ed.), June 2016.

Geoffrey E. Hinton Vinod Nair. Rectified linear units im- prove restricted boltzmann machines. 2010.

Max Welling and Teh Yee W. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681–688, 2011.