Higher Order Orthogonal Polynomials as Activation Functions in Artificial Neural Networks

Abstract

Activation functions are used in Artificial Neural Networks to provide non-linearity to the system. Several different activation functions in use are very well known by almost any AI practitioner however this is not the case for polynomial activation functions. Increasing attention to these valuable mathematical functions can encourage more research and help to fill the gap. During this work, Chebyshev and Hermite orthogonal polynomials were used as activation functions. Calculations were conducted on 3 different datasets with different hyperparameters. According to the results, calculations done by Chebyshev activation functions take less time, but Chebyshev can be more fragile depending on the solved problem. On the other hand, Hermite shows a more robust and generalized behavior, it is less dependent on the problem type, and it improves by necessary adjustments.