Pyspark тензор потока нейронной сети

Question

# Нужна помощь в пунктах 3–6 ниже: код обучения по частям

correct_prediction = tf.equal (pred, y) точность = tf.reduce_mean ( tf.cast (correct_prediction, tf.float32))

def train_nn (): с tf.Session () в качестве sess:

    sess.run(init)

    ## this is needed to print debug statements during training.
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(coord=coord)

    x_train, x_valid = features[:num_train_examples], features[num_train_examples:]
    y_train, y_valid = labels[:num_train_examples], labels[num_train_examples:]

    training_costs = []
    training_accs = []

    validation_costs = []
    validation_accs = []

    best_acc = 0.
    weights
    biases

    for epoch in range(training_epochs):

        '''
        We recommend you first think about how you will implement this on your own before proceeding to read any further.

        HINT: You should implement the following procedure here:

        An epoch is one pass through your training data
        1. Keep a counter of your epoch's total cost. 
           You will need this to average over the batches.
        2. Keep a counter of the number of correct predictions in your epoch. 
           You will need this to sum over the batches to calculate per epoch accuracy.

        ***

** Для каждого пакета (вы должны иметь num_batches количество партий в целом) - Обучение по партиям - 3. Подмножество своих функций и меток из x_train и y_train ex. для партии 1 вы должны выбрать все примеры в интервале [0, batch_size) для партии 2, она будет между [batch_size, 2 * batch_size). Убедитесь, что в качестве вашей последней партии учтена возможная дробная партия. 4. Помассируйте x_batch и y_batch в numpy массивах формы (size_of_batch, 28) и (size_of_batch, 1) соответственно

            5. Feed the x_batch and y_batch into your tensorflow graph and execute the optimizer, cost and pred 
                in order to train your model using the current batch and also get back the batch_cost and batch_predictions
            6. Count the number of correct predictions for this batch and add it to the counter for the 
                correct predictions in the epoch**

---

        7. Calculate your average_epoch_cost as the total_epoch_cost divided by the number of training examples
        8. Append the average_epoch_cost to `training_costs`
        9. Calculate your epoch_accuracy as the total number of correct predictions in your epoch
                divided by the number of training examples
        10. Append the epoch_accuracy to `training_accs`

        --Validation--

        11. Massage your validation labels (y_valid) into a numpy arrays of shape (validation_set_size, 1)
        12. With y_valid and x_valid as input to your graph, calculate the validation loss and validation predictions
                We are calculating validation accuracy at the end of each epoch
        13. Calculate the number of correct validation predictions by comparing against your validation labels
        14. Append validation costs and validation accuracy to their respective lists
        15. Avoid printing a lot of debug information when you submit the assignment. 
            This reduces the speed of execution.
            If you want to print some information every so often, you can use the following line 
            at the end of your epoch loop:
            if epoch%display_step==0:
                print("Epoch %d | Tr cost: %f | Tr accuracy %f | Va cost: %f | Va accuracy: 
                    %f"%(epoch + 1,avg_epoch_cost, this_epoch_accuracy, batch_valid_cost, valid_accuracy))
        '''
    ###
    ### YOUR CODE HERE
    ###
    ## Assuming the above part is completed, you should now use your trained model to make predictions on the test set.

        _, training_cost, training_acc = sess.run([train_op, cost, accuracy], feed_dict={x:x_train, y:y_train})

        validation_cost, validation_acc = sess.run([cost, accuracy], feed_dict={x:x_valid, y:y_valid})

        training_costs.append(training_cost)
        training_accs.append(training_acc)

        validation_costs.append(validation_cost)
        validation_accs.append(validation_acc)

        if validation_acc > best_acc:


            training_costs = [np.float64(x) for x in training_costs]
            training_accs = [np.float64(x) for x in training_accs]

            validation_costs = [np.float64(x) for x in validation_costs]
            validation_accs = [np.float64(x) for x in validation_accs]

    print("Optimization Finished!")
    #

    test_predictions = []


    '''
    HINT:
    Using the `test_features` as input to your trained graph,
        run the graph once more to calculate the predictions on the test set
    Cast the returned numpy array into the python list called `test_predictions`
        where each element has type np.float64
    '''

    ###
    ### YOUR CODE HERE
    ###
    test_predictions = sess.run(pred, feed_dict={x: test_features}).reshape(-1).astype(np.float64, copy=False)


    #coord.request_stop()
    #coord.join(threads)



## close TF session if open
if 'session' in locals() and sess is not None:
    print('Close interactive session')
    sess.close()

return training_costs, validation_costs, training_accs, validation_accs, test_predictions