Home>

I want to solve the problem when importing CSV.
Currently, I am trying to access the application placed on AWS and use the CSV import function, but to import a large CSV file,
"413 Request Entity Too Large"
Was thrown. So in nginx.config
I have set the value of "client_max_body_size".
One more point, "FILE_UPLOAD_PERMISSIONS = 0o644" was also added to settings.py.
However, I still got an error, and this time, "nginx error!" Was thrown and the nginx logo screen was displayed.

After checking some operations, I found that this "nginx error!" Was output after a certain period of time (about 30 seconds).
So, I set the timeout in nginx.config, but it doesn't change.

environment
Instance type: t2.micro
ap: django
web: nginx (gunicorn)
db: PostgreSQL

nginx.conf (The file size is set to 3M bytes and the timeout is set to 9999 seconds. * There is 40M in the comment, but please ignore it.)

# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
    worker_connections 1024;
}
http {
    log_format main'$remote_addr-$remote_user [$time_local] "$request"'
                      '$status $body_bytes_sent "$http_referer"'
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log /var/log/nginx/access.log main;
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    # keepalive_timeout 65;
    keepalive_timeout 9999;
    types_hash_max_size 4096;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    # Load modular configuration files from the /etc/nginx/conf.d directory.
    #See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;
    # When importing a file of about 40 Mbytes, an error (413 Request Entity Too Large) occurred (3 locations in total below)
    client_max_body_size 3M;
    # Time-out settings
    proxy_connect_timeout 9999;
    proxy_send_timeout 9999;
    proxy_read_timeout 9999;
    send_timeout 9999;
    client_body_timeout 9999;
    client_header_timeout 9999;server {
        # Setting to listen for requests on port 80 (HTTP)
        listen 80;
        listen [::]: 80;
        #Specify Elastic IP address
        server_name xxx.xxx.xxx.xxx;
        #Public directory settings
        root/usr/share/nginx/html;
        # Because an error (413 Request Entity Too Large) occurred when importing a file of about 40 Mbytes.
        client_max_body_size 3M;
        #The following may or may not be necessary
        client_body_buffer_size 3M;
        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;
       # Static file placement Directory and URL mapping
       location/static {
          alias/usr/share/nginx/html/static;
       }
       location/{
          #Host: Transfer host name
          proxy_set_header Host $http_host;
          # X-Forwarded-For: Forward source address
          proxy_set_header X-Fowarded-For $proxy_add_x_forwarded_for;
          # X-Forwarded-Proto: Forward URL schemes (such as http)
          proxy_set_header X-Fowrded-Proto $scheme;
          # When importing a file of about 40 Mbytes, an error (413 Request Entity Too Large) occurred (the above 3 locations in total)
          client_max_body_size 3M;
          # If the URL does not correspond to "/ static" (reverse proxy)
          proxy_pass http://127.0.0.1:8000;
        }
        error_page 404 /404.html;
            location = /40x.html {
        }
        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
# Settings for a TLS enabled server.
#
#server {
# listen 443 ssl http2;
# listen [::]: 443 ssl http2;
# server_name _;
# root/usr/share/nginx/html;
#
# ssl_certificate "/etc/pki/nginx/server.crt";
# ssl_certificate_key "/etc/pki/nginx/private/server.key";
# ssl_session_cache shared: SSL: 1m;
# ssl_session_timeout 10m;
# ssl_ciphers PROFILE = SYSTEM;
# ssl_prefer_server_ciphers on;
#
# # Load configuration files for the default server block.
#include /etc/nginx/default.d/*.conf;#
# error_page 404 /404.html;
#location = /40x.html {
#}
#
# error_page 500 502 503 504 /50x.html;
#location = /50x.html {
#}
#}
}


settings.py (I made permission settings on the last line)

from .settings_common import *
# SECURITY WARNING: keep the secret key used in production secret!
Common with # settings_common
#SECRET_KEY ='* hto) tc = a23-pbhy @ vmj-kpnr # of96n&jgyrqr @) e% 9% c (d * nj'
os.makedirs (os.path.join (BASE_DIR,'logs'), exist_ok = True)
# SECURITY WARNING: don't run with debug turned on in production!
# Whether to enable debug mode (must be False in production operation)
DEBUG = False
# List of allowed host names
ALLOWED_HOSTS = [os.environ.get ('ALLOWED_HOSTS')]
# Where to place static files
STATIC_ROOT ='/ usr/share/nginx/html/static'
# Logging
LOGGING = {
    'version': 1, # 1 fixed
    'disable_existing_loggers': False,
    #Logger settings
    'loggers': {
        Logger used by #Django
        'django': {
            'handlers': ['file'],


            'level':'INFO',
        },


        #common Logger used by applications
            'level':'INFO',
        },


        # setting_worktypemaster Logger used by the application
            'level':'INFO',
        },


        # setting_serialnomaster The logger used by the application
        'setting_serialnomaster': {
            'handlers': ['file'],'level':'INFO',
        },


    },


    # Handler settings
    'handlers': {
        'file': {
            'level':'INFO',
            'class':'logging.handlers.TimedRotatingFileHandler',
            'filename': os.path.join (BASE_DIR,'logs/django.log'),
            'formatter':'prod',
            'when':'D', # log rotation (switch to new file) interval
Unit (D = day)
            'interval': 1, # Log rotation interval (daily)
            'backupCount': 7, # Number of log files to save
        },


    },


    # Formatter settings
    'formatters': {
        'prod': {
            'format':'\ t'.join ([[
                '% (asctime) s',
                '[% (Level name) s]',
                '% (pathname) s (Line:% (lineno) d)',
                '% (message) s'
            ])
        },


    }
}
# To avoid error 413 (Request Entity Too Large)
FILE_UPLOAD_PERMISSIONS = 0o644


django.log

2020-11-17 19: 52: 06,922 [WARNING] /home/app_admin/venv_xxx_project/lib64/python3.7/site-packages/django/utils/log.py (Line: 228) Not Found: /nginx-logo. png
2020-11-17 19: 52: 06,924 [WARNING] /home/app_admin/venv_xxx_project/lib64/python3.7/site-packages/django/utils/log.py (Line: 228) Not Found: /poweredby.png
2020-11-17 19: 55: 16,211 [WARNING] /home/app_admin/venv_xxx_project/lib64/python3.7/site-packages/django/utils/log.py (Line: 228) Not Found: /nginx-logo. png
2020-11-17 19: 55: 16,213 [WARNING] /home/app_admin/venv_xxx_project/lib64/python3.7/site-packages/django/utils/log.py (Line: 228) Not Found: /poweredby.png
2020-11-17 19: 58: 58,605 [WARNING] /home/app_admin/venv_xxx_project/lib64/python3.7/site-packages/django/utils/log.py (Line: 228) Not Found: /nginx-logo. png
2020-11-17 19: 58: 58,606 [WARNING] /home/app_admin/venv_xxx_project/lib64/python3.7/site-packages/django/utils/log.py (Line: 228) Not Found: /poweredby.png
2020-11-17 20: 20: 07,443 [WARNING] /home/app_admin/venv_xxx_project/lib64/python3.7/site-packages/django/utils/log.py (Line: 228) Not Found: /nginx-logo. png
2020-11-17 20: 20: 07,464 [WARNING] /home/app_admin/venv_xxx_project/lib64/python3.7/site-packages/django/utils/log.py (Line: 228) Not Found: /poweredby.png

I would like to resolve it as soon as possible. Anything is fine. Can you comment if there are any points that you have overlooked? Thanks for your cooperation.

  • Answer # 1

    I've noticed it lightly since I asked a question last night, but it requires gunicorn settings.
    As a start command for gunicorn

    "$Gunicorn --bind 127.0.0.1:8000 project name.wsgi -D"

    I was running gunicorn with
    By adding "--timeoout" to the option, this problem seems to be solved (I will check the operation from now on).
    Therefore, as an actual command

    "$Gunicorn --bind 127.0.0.1:8000 project name.wsgi -D --timeout 9999"

    If so, I think that even a 40MB file can be read without any problem.
    The default is probably around 30 seconds.

    In addition, we will inform you if there is a problem.

    Postscript
    As mentioned above, the gunicorn timeout was the cause of this bug.
    gunicorn startup command options
    "--Log-level = DEBUG" was added to output the log, and about 3 CSV imports were performed.
    Below is what the terminal displays.
    You can see that the setting information is displayed in the first half and "WORKER TIME OUT" is displayed in the log information in the second half.

    (venv_xxx_project) [app_admin @ ip-172-31-36-217 xxx_project] $gunicorn --bind 127.0.0.1:8000 xxx_project.wsgi --timeout 33 --log-level = DEBUG
    [2020-11-18 10:55:10 +0900] [10999] [DEBUG] Current configuration:
      config: None
      bind: ['127 .0.0.1: 8000']
      backlog: 2048
      workers: 1
      worker_class: sync
      threads: 1
      worker_connections: 1000
      max_requests: 0
      max_requests_jitter: 0
      timeout: 33
      graceful_timeout: 30
      keepalive: 2
      limit_request_line: 4094
      limit_request_fields: 100
      limit_request_field_size: 8190
      reload: False
      reload_engine: auto
      reload_extra_files: []
      spew: False
      check_config: False
      preload_app: False
      sendfile: None
      reuse_port: False
      chdir:/home/app_admin/venv_xxx_project/xxx_nippo_20201102/venv_xxx_project/xxx_project
      daemon: Falseraw_env: []
      pidfile: None
      worker_tmp_dir: None
      user: 1001
      group: 1001
      umask: 0
      initgroups: False
      tmp_upload_dir: None
      secure_scheme_headers: {'X-FORWARDED-PROTOCOL':'ssl','X-FORWARDED-PROTO':'https','X-FORWARDED-SSL':'on'}
      forwarded_allow_ips: ['12 7.0.0.1']
      accesslog: None
      disable_redirect_access_to_syslog: False
      access_log_format:% (h) s% (l) s% (u) s% (t) s "% (r) s"% (s) s% (b) s "% (f) s" "% (a) ) s "
      errorlog:-
      loglevel: DEBUG
      capture_output: False
      logger_class: gunicorn.glogging.Logger
      logconfig: None
      logconfig_dict: {}
      syslog_addr: udp: // localhost: 514
      syslog: False
      syslog_prefix: None
      syslog_facility: user
      enable_stdio_inheritance: False
      statsd_host: None
      dogstatsd_tags:
      statsd_prefix:
      proc_name: None
      default_proc_name: xxx_project.wsgi
      pythonpath: None
      paste: None
      on_starting:<function OnStarting.on_starting at 0x7f6cc9a7cd40>on_reload:<function OnReload.on_reload at 0x7f6cc9a7ce60>when_ready:<function WhenReady.when_ready at 0x7f6cc9a7cf80>pre_fork:<function Prefork.pre_fork at 0x7f6cc9a8f0e0>post_fork:<function Postfork.post_fork at 0x7f6cc9a8f200>post_worker_init:<function PostWorkerInit.post_worker_init at 0x7f6cc9a8f320>worker_int:<function WorkerInt.worker_int at 0x7f6cc9a8f440>worker_abort:<function WorkerAbort.worker_abort at 0x7f6cc9a8f560>pre_exec:<function PreExec.pre_exec at 0x7f6cc9a8f680>pre_request:<function PreRequest.pre_request at 0x7f6cc9a8f7a0>post_request:<function PostRequest.post_request at 0x7f6cc9a8f830>child_exit:<function ChildExit.child_exit at 0x7f6cc9a8f950>worker_exit:<function WorkerExit.worker_exit at 0x7f6cc9a8fa70>nworkers_changed:<function NumWorkersChanged.nworkers_changed at 0x7f6cc9a8fb90>on_exit:<function OnExit.on_exit at 0x7f6cc9a8fcb0>proxy_protocol: False
      proxy_allow_ips: ['127.0.0.1']
      keyfile: None
      certfile: None
      ssl_version: 2
      cert_reqs: 0
      ca_certs: None
      suppress_ragged_eofs: True
      do_handshake_on_connect: False
      ciphers: None
      raw_paste_global_conf: []
      strip_header_spaces: False
    [2020-11-18 10:55:10 +0900] [10999] [INFO] Starting gunicorn 20.0.4
    [2020-11-18 10:55:10 +0900] [10999] [DEBUG] Arbiter booted
    [2020-11-18 10:55:10 +0900] [10999] [INFO] Listening at: http://127.0.0.1:8000 (10999)
    [2020-11-18 10:55:10 +0900] [10999] [INFO] Using worker: sync
    [2020-11-18 10:55:10 +0900] [11002] [INFO] Booting worker with pid: 11002
    [2020-11-18 10:55:10 +0900] [10999] [DEBUG] 1 workers
    [2020-11-18 10:55:33 +0900] [11002] [DEBUG] POST/working_listworkinglistimport /
    [2020-11-18 10:56:07 +0900] [10999] [CRITICAL] WORKER TIMEOUT (pid: 11002)
    [2020-11-18 10:56:07 +0900] [11002] [INFO] Worker exiting (pid: 11002)
    [2020-11-18 10:56:07 +0900] [11007] [INFO] Booting worker with pid: 11007
    [2020-11-18 10:56:07 +0900] [11007] [DEBUG] GET /nginx-logo.png
    [2020-11-18 10:56:07 +0900] [11007] [DEBUG] GET /poweredby.png
    [2020-11-18 10:56:35 +0900] [11007] [DEBUG] POST/working_listworkinglistimport /
    [2020-11-18 10:57:09 +0900] [11007] [DEBUG] GET/working_list
    [2020-11-18 10:57:35 +0900] [11007] [DEBUG] GET/boaform/admin/formLogin
    [2020-11-18 10:57:57 +0900] [11007] [DEBUG] GET/working_listworkinglistimport /
    [2020-11-18 10:58:38 +0900] [11007] [DEBUG] POST/working_listworkinglistimport /
    [2020-11-18 10:59:08 +0900] [11007] [DEBUG] GET/working_list
    [2020-11-18 10:59:14 +0900] [11007] [DEBUG] GET/working_listworkinglistimport /
    [2020-11-18 10:59:27 +0900] [11007] [DEBUG] POST/working_listworkinglistimport /
    [2020-11-18 11:00:00 +0900] [10999] [CRITICAL] WORKER TIMEOUT (pid: 11007)
    [2020-11-18 11:00:00 +0900] [11007] [INFO] Worker exiting (pid: 11007)
    [2020-11-18 11:00:00 +0900] [11025] [INFO] Booting worker with pid: 11025
    [2020-11-18 11:00:00 +0900] [11025] [DEBUG] GET /poweredby.png
    [2020-11-18 11:00:00 +0900] [11025] [DEBUG] GET /nginx-logo.png


    With the above, this case will be resolved.
    Thank you very much.