Home>

I downloaded the following code from github and executed it, but the title error occurred and I could not execute it.
I have a Unicode error. Can you tell me how to solve it?
[Github] ArmRobot

I will describe the details of the error.
When the above code is executed on the terminal,
"ArmRobot-MLmaster/python/unityagents/environment.py"Line 138 in the file
"p = self._conn.recv (self._buffer_size) .decode ('utf-8')"results in the following error message:
It seems angry that I can't decode.
Can you tell me if you know how to deal with this error?

Error message
line 138, in __init__
    p = self._conn.recv (self._buffer_size) .decode ('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 63: invalid start byte
Applicable source code
import atexit
import io
import glob
import json
import logging
import numpy as np
import os
import socket
import subprocess
import struct
from .brain import BrainInfo, BrainParameters, AllBrainInfo
from .exception import UnityEnvironmentException, UnityActionException, UnityTimeOutException
from .curriculum import Curriculum
from PIL import Image
from sys import platform
logging.basicConfig (level = logging.INFO)
logger = logging.getLogger ("unityagents")

class UnityEnvironment (object):
    def __init __ (self, file_name, worker_id = 0,
                 base_port = 5005, curriculum = None,
                 seed = 0, docker_training = False):
        "" "
        Starts a new unity environment and establishes a connection with the environment.
        Notice: Currently communication between Unity and Python takes place over an open socket without authentication.
        Ensure that the network where training takes place is secure.
        : string file_name: Name of Unity environment binary.
        : int base_port: Baseline port number to connect to Unity environment over.worker_id increments over this.
        : int worker_id: Number to add to communication port (5005) [0]. Used for asynchronous agent scenarios.
        : param docker_training: Informs this class whether the process is being run within a container.
        "" "
        atexit.register (self.close)
        self.port = base_port + worker_id
        self._buffer_size = 12000
        self._version_ = "API-3"
        self._loaded = False
        self._open_socket = False
        try:
            # Establish communication socket
            self._socket = socket.socket (socket.AF_INET, socket.SOCK_STREAM)
            self._socket.setsockopt (socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
            self._socket.bind (("localhost", self.port))self._open_socket = True
        except socket.error:
            self._open_socket = True
            self.close ()
            raise socket.error ("Couldn't launch new environment because worker number {} is still in use."
                               "You may need to manually close a previously opened environment"
                               "or use a different worker number.". format (str (worker_id)))
        cwd = os.getcwd ()
        file_name = (file_name.strip ()
                     .replace ('. app', '') .replace ('. exe', '') .replace ('. x86_64', '') .replace ('. x86', ''))
        true_filename = os.path.basename (os.path.normpath (file_name))
        logger.debug ('The true file name is {}'. format (true_filename))
        launch_string = None
        if platform == "linux" or platform == "linux2":
            candidates = glob.glob (os.path.join (cwd, file_name) + '.x86_64')
            if len (candidates) == 0:
                candidates = glob.glob (os.path.join (cwd, file_name) + '.x86')
            if len (candidates) == 0:
                candidates = glob.glob (file_name + '.x86_64')
            if len (candidates) == 0:
                candidates = glob.glob (file_name + '.x86')
            if len (candidates)>0:
                launch_string = candidates [0]
        elif platform == 'darwin':
            candidates = glob.glob (os.path.join (cwd, file_name + '.app', 'Contents', 'MacOS', true_filename))
            if len (candidates) == 0:
                candidates = glob.glob (os.path.join (file_name + '.app', 'Contents', 'MacOS', true_filename))
            if len (candidates) == 0:
                candidates = glob.glob (os.path.join (cwd, file_name + '.app', 'Contents', 'MacOS', '*'))
            if len (candidates) == 0:
                candidates = glob.glob (os.path.join (file_name + '.app', 'Contents', 'MacOS', '*'))
            if len (candidates)>0:
                launch_string = candidates [0]
        elif platform == 'win32':
            candidates = glob.glob (os.path.join (cwd, file_name + '.exe'))
            if len (candidates) == 0:
                candidates = glob.glob (file_name + '.exe')
            if len (candidates)>0:
                launch_string = candidates [0]
        if launch_string is None:
            self.close ()
            raise UnityEnvironmentException ("Couldn't launch the {0} environment."
                                            "Provided filename does not match any environments."
                                            .format (true_filename))
        else:
            logger.debug ("This is the launch string {}". format (launch_string))
            # Launch Unity environment
            if docker_training == False:
                proc1 = subprocess.Popen (
                    [launch_string,
                     '--port', str (self.port),
                     '--seed', str (seed)])
            else:
                "" "
                Comments for future maintenance:
                    xvfb-run is a wrapper around Xvfb, a virtual xserver where all
                    rendering is done to virtual memory.It automatically creates anew virtual server automatically picking a server number `auto-servernum`.
                    The server is passed the arguments using `server-args`, we are telling
                    Xvfb to create Screen number 0 with width 640, height 480 and depth 24 bits.
                    Note that 640 X 480 are the default width and height.The main reason for
                    us to add this is because we'd like to change the depth from the default
                    of 8 bits to 24.
                    However, this means that we will need to pass the arguments through
                    a shell which is why we set `shell = True`. Now, this adds its own
                    complications. E.g SIGINT can bounce off the shell and not get propagated
                    to the child processes.This is why we add `exec`, so that the shell gets
                    launched, the arguments are passed to `xvfb-run`.`exec` replaces the shell
                    we created with `xvfb`.
                "" "
                docker_ls = ("exec xvfb-run --auto-servernum"
                             "--server-args = '-screen 0 640x480x24'"
                             "{0} --port {1} --seed {2}"). Format (launch_string,
                                                                  str (self.port),
                                                                  str (seed))
                proc1 = subprocess.Popen (docker_ls,
                                         stdout = subprocess.PIPE,
                                         stderr = subprocess.PIPE,
                                         shell = True)
        self._socket.settimeout (30)
        try:
            try:
                self._socket.listen (1)
                self._conn, _ = self._socket.accept ()
                self._conn.settimeout (30)
                import pdb;pdb.set_trace ()
#An error occurs at the following line
                p = self._conn.recv (self._buffer_size) .decode ('utf-8')
                p = json.loads (p)
            except socket.timeout as e:
                raise UnityTimeOutException (
                    "The Unity environment took too long to respond. Make sure {} does not need user interaction to"
                    "launch and that the Academy and the external Brain (s) are attached to objects in the Scene."
                    .format (str (file_name)))
The #StackOverflow specification doesn't allow you to post more than 10,000 characters. Please refer to the above link from github
Tried

-Check the character code of the executed shell (echo $LANG)
The terminal character code is ja_JP.UTF-8
The line of error occurrence due to mismatch between the terminal and code character code seems thin.

Debugging (pdb)
Nothing was stored in the variable p.
Confirming the contents of data sent via socket is not possible

-I haven't found any information on how to find a solution even after searching on the Internet
Specifying .decode ('utf-8') in bytes type in python3 results in an error
Technical notes
UnicodeDecodeError:'utf-8'codec can't decode byte 0xbc in position 0 : invalid start byte

Supplemental information (FW/tool version etc.)

macOS Sierra 10.12.6
Unity 2018.2.4.f1
Terminal 2.7.3 (Character code ja_JP.UTF-8)
Python 3.6.6 :: Anaconda, Inc.

  • Answer # 1

      

    UnicodeDecodeError:'utf-8'codec can't decode byte 0xfe

    It is speculation, but since it isdecode byte 0xfe, the data is sent withutf-16instead ofutf-8Looks like.

    def __init __ (self, file_name, worker_id = 0,
                     base_port = 5005, curriculum = None,
                     seed = 0, docker_training = False):


    and port number:5005is not aDockercontainer.

    Please check theLOCALEsetting of theDockercontainer again.

    ◆ Reference information
    In Docker, reconfirm container locale


    Unity-Technologies/ml-agents/Issuess/1026

    ArmRobot ProjectVersion.txt
    Unity version: 2018.1.0f2

  • Answer # 2

    Is such invalid data received because the received data cannot be decoded to UTF8?
    How about checking the communication data first?