‎

1. Multilayer Perceptrons

1 Multilayer Perceptrons

(ns clj-d2l.multilayer-perceptrons
  (:require [clojure.java.io :as io]
            [clj-djl.ndarray :as nd]
            [clj-djl.engine :as engine]
            [clj-djl.model :as model]
            [clj-djl.nn :as nn]
            [clj-djl.training :as training]
            [clj-djl.training.loss :as loss]
            [clj-djl.training.tracker :as tracker]
            [clj-djl.training.optimizer :as optimizer]
            [clj-djl.training.listener :as listener]
            [com.hypirion.clj-xchart :as c]))

(defn plot [filename title x y]
  (-> (c/xy-chart
     {title
      {:x x
       :y y
       :style {:marker-type :none}}})
    (c/spit filename)))

1.1 Hidden Layers

1.1.1 Incorporating Hidden Layers

1.1.2 From Linear to Nonlinear

1.1.3 Vectorization and Minibatch

1.2 Activation Functions

1.2.1 ReLU Function

rectified linear unit:

(def manager (nd/base-manager))
(def x (nd/arange manager -8.0 8.0 0.1))
(nd/attach-gradient x)
(def y (nn/relu x))

(def X (nd/to-vec x))
(def Y (nd/to-vec y))

(plot "figure/relu.svg" "ReLU" X Y)

The derivative of the relu function is:

(with-open [gc (-> (engine/get-instance) (engine/new-gradient-collector))]
  (let [y (nn/relu x)]
    (training/backward gc y)))

(def res (nd/get-gradient x))
(def X (nd/to-vec x))
(def Y (nd/to-vec res))

(plot "figure/grad_relu.svg" "grad of ReLU" X Y)

1.3 Sigmoid Function

(def y (nn/sigmoid x))
(def Y (nd/to-vec y))
(plot "figure/sigmoid.svg" "Sigmoid" X Y)

The derivative of the sigmoid function is:

(with-open [gc (-> (engine/get-instance) (engine/new-gradient-collector))]
  (let [y (nn/sigmoid x)]
    (training/backward gc y)))

(def res (nd/get-gradient x))
(def Y (nd/to-vec res))
(plot "figure/grad_sigmoid.svg" "grad of Sigmoid" X Y)

1.4 Tanh Function

(def y (nn/tanh x))
(def Y (nd/to-vec y))
(plot "figure/tanh.svg" "Tanh" X Y)

The derivative of the Tanh function is:

(with-open [gc (-> (engine/get-instance) (engine/new-gradient-collector))]
  (let [y (nn/tanh x)]
    (training/backward gc y)))

(def res (nd/get-gradient x))
(def Y (nd/to-vec res))
(plot "figure/grad_tanh.svg" "grad of Tanh" X Y)

Table of Contents

1 Multilayer Perceptrons

1.1 Hidden Layers

1.1.1 Incorporating Hidden Layers

1.1.2 From Linear to Nonlinear

1.1.3 Vectorization and Minibatch

1.2 Activation Functions

1.2.1 ReLU Function

1.3 Sigmoid Function

1.4 Tanh Function