Home         Authors   Papers   Year of conference   Themes   Organizations        To MES conference

Automatic Differentiation in a Dataflow Language on the Example of a Deep Learning Problem
Authors
	Klimov A.V.
Date of publication
	2021
DOI
	10.31114/2078-7707-2021-3-94-98
Abstract
	We describe a method for automatic converting a given program into a program for calculating derivatives with respect to designated parameters. The original program as well as converted one are represented in our dataflow parallel programming language UPL. A program in UPL is graphically represented as a graph of nodes connected with arcs. A node has several input ports and calculates a function, sending result to other node’s ports. A program node denotes a family of activation nodes (or node instances), each with its own index, or tag. An arc carries a result value expression and a target tag expression, which are calculated in the activation. Having almost arbitrary program graph in UPL that calculates a single scalar result value the hypothetical converter builds another program graph, which calculates the set of derivatives of the result of original program w.r.t. its parameters. The converter works on per node basis adding an adjoint node for each source node. Inverted adjoint arcs, each carrying a derivative of the result with respect to original value, connect these adjoint nodes. The arc’s tag expressions are rewritten to the other node’s context (it must be feasible). A special kind of arcs –– multicast fan-out arcs and summation fan-in arcs –– are transformed one into another. The constructed adjoint graph calculates the required derivatives in backward direction. As an example, a program of a multilayer neural network (perceptron) is considered (see upper part of the Figure 1 in the paper), for which a classical backpropagation algorithm is built (see lower part of Figure 1) according to the described rules. It calculates derivatives of the loss function with respect to weight parameters, which can be used further for weights correction in the frame of deep learning process.
Keywords
	automatic differentiation, back propagation, deep learning, dataflow language, graphical programming.
Library reference
	Klimov A.V. Automatic Differentiation in a Dataflow Language on the Example of a Deep Learning Problem // Problems of Perspective Micro- and Nanoelectronic Systems Development - 2021. Issue 3. P. 94-98. doi:10.31114/2078-7707-2021-3-94-98
URL of paper
	http://www.mes-conference.ru/data/year2021/pdf/D044.pdf