Home>

Last question
https://www.tutorialfor.com/go.php?id=304864
When I tried to import a 40MB CSV file with, an error occurred and at the time of the question I had set nginx.conf, but after the question I imported a 50MB CSV file by additionally setting gunicorn. Since I succeeded in doing so, I tried to solve it myself, but I got an error when importing a 100MB CSV file. As the title suggests, I am reviewing the settings because I want to import a 1GB file, but I'm not sure. What you see in your browser is "Server Error (500)" or "nginx error!".
The log information at the time of importing the configuration file and the 100 Mbyte file is described below.

nginx.con (I set 1G as client_max_body_size)

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
    worker_connections 1024;
}
http {
    log_format main'$remote_addr-$remote_user [$time_local] "$request"'
                      '$status $body_bytes_sent "$http_referer"'
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log /var/log/nginx/access.log main;
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    # keepalive_timeout 65;
    keepalive_timeout 9999;
    types_hash_max_size 4096;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    # Load modular configuration files from the /etc/nginx/conf.d directory.
    #See http://nginx.org/en/docs/ngx_core_module.html#include
    proxy_connect_timeout 9999;
    proxy_send_timeout 9999;
    proxy_read_timeout 9999;
    send_timeout 9999;
    client_body_timeout 9999;
    client_header_timeout 9999;server {
        # Setting to listen for requests on port 80 (HTTP)
        listen 80;
        listen [::]: 80;
        #Specify Elastic IP address
        server_name xxx.xxx.xxx.xxx;
        #Public directory settings
        root/usr/share/nginx/html;
        client_max_body_size 1G;
        #The following may or may not be necessary
        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;
       # Static file placement Directory and URL mapping
       location/static {
          alias/usr/share/nginx/html/static;
       }
       location/{
          #Host: Transfer host name
          proxy_set_header Host $http_host;
          # X-Forwarded-For: Forward source address
          proxy_set_header X-Fowarded-For $proxy_add_x_forwarded_for;
          # X-Forwarded-Proto: Forward URL schemes (such as http)
          proxy_set_header X-Fowrded-Proto $scheme;
         #client_max_body_size 1000M;
          client_max_body_size 1G;
          # If the URL does not correspond to "/ static" (reverse proxy)
          proxy_pass http://127.0.0.1:8000;
        }
        error_page 404 /404.html;
            location = /40x.html {
        }
        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }}


/var/log/nginx/error.log

(venv_xxx_project) [[email protected] ~] $sudo tail -f /var/log/nginx/error.log
2020/11/18 14:02:48 [error] 12701 # 0: * 1 client intended to send too large body: 102416786 bytes, client: yyy.yyy.yyy.yyy, server: xxx.xxx.xxx.xxx, request: "POST /working_listworkinglistimport/ HTTP/1.1", upstream: "http://127.0.0.1:8000/working_listworkinglistimport/", host: "xxx.xxx.xxx.xxx", referrer: "http: //xxx. xxx.xxx.xxx/working_listworkinglistimport/ "
2020/11/18 14:04:56 [emerg] 12773 # 0: "client_body_buffer_size" directive invalid value in /etc/nginx/nginx.conf:61
2020/11/18 14:05:46 [emerg] 12787 # 0: "client_body_buffer_size" directive invalid value in /etc/nginx/nginx.conf:61
2020/11/18 14:05:52 [emerg] 12801 # 0: "client_body_buffer_size" directive invalid value in /etc/nginx/nginx.conf:61
2020/11/18 14:08:35 [error] 12827 # 0: * 2 upstream prematurely closed connection while reading response header from upstream, client: yyy.yyy.yyy.yyy, server: xxx.xxx.xxx.xxx, request: "POST /working_listworkinglistimport/ HTTP/1.1", upstream: "http://127.0.0.1:8000/working_listworkinglistimport/", host: "xxx.xxx.xxx.xxx", referrer: "http: //xxx. xxx.xxx.xxx/working_listworkinglistimport/"
2020/11/18 14:27:08 [emerg] 13007 # 0: * 1 malloc (1024164151) failed (12: Cannot allocate memory), client: yyy.yyy.yyy.yyy, server: xxx.xxx.xxx. xxx, request: "POST /working_listworkinglistimport/ HTTP/1.1", host: "xxx.xxx.xxx.xxx", referrer: "http://xxx.xxx.xxx.xxx/working_listworkinglistimport/"
2020/11/18 14:27:38 [emerg] 13007 # 0: * 3 malloc (1024164151) failed (12: Cannot allocate memory), client: yyy.yyy.yyy.yyy, server: xxx.xxx.xxx. xxx, request: "POST /working_listworkinglistimport/ HTTP/1.1", host: "xxx.xxx.xxx.xxx", referrer: "http://xxx.xxx.xxx.xxx/working_listworkinglistimport/"
2020/11/18 14:28:08 [emerg] 13007 # 0: * 4 malloc (1024164151) failed (12: Cannot allocate memory), client: yyy.yyy.yyy.yyy, server: xxx.xxx.xxx. xxx, request: "POST /working_listworkinglistimport/ HTTP/1.1", host: "xxx.xxx.xxx.xxx", referrer: "http://xxx.xxx.xxx.xxx/working_listworkinglistimport/"
2020/11/18 14:32:53 [error] 13049 # 0: * 1 sendfile () failed (104: Connection reset by peer) while sending request to upstream, client: yyy.yyy.yyy.yyy, server: xxx .xxx.xxx.xxx, request: "POST /working_listworkinglistimport/ HTTP/1.1", upstream: "http://127.0.0.1:8000/working_listworkinglistimport/", host: "xxx.xxx.xxx.xxx", referrer: "http://xxx.xxx.xxx.xxx/working_listworkinglistimport/"
2020/11/18 14:49:35 [error] 13406 # 0: * 1 upstream prematurely closed connection while reading response header from upstream, client: yyy.yyy.yyy.yyy, server: xxx.xxx.xxx.xxx, request: "POST /working_listworkinglistimport/ HTTP/1.1", upstream: "http://127.0.0.1:8000/working_listworkinglistimport/", host: "xxx.xxx.xxx.xxx", referrer: "http: //xxx. xxx.xxx.xxx/working_listworkinglistimport/"
2020/11/18 15:13:46 [error] 13553 # 0: * 5 upstream prematurely closed connection while reading response header from upstream, client: yyy.yyy.yyy.yyy, server: xxx.xxx.xxx.xxx, request: "POST /working_listworkinglistimport/ HTTP/1.1", upstream: "http://127.0.0.1:8000/working_listworkinglistimport/", host: "xxx.xxx.xxx.xxx", referrer: "http: //xxx. xxx.xxx.xxx/working_listworkinglistimport/"

django.log

(venv_xxx_project) [[email protected] xxx_project] $tail -f logs/django.log
    self.full_clean ()
  File "/home/app_admin/venv_xxx_project/lib64/python3.7/site-packages/django/forms/forms.py", line 376, in full_clean
    self._clean_fields ()
  File "/home/app_admin/venv_xxx_project/lib64/python3.7/site-packages/django/forms/forms.py", line 397, in _clean_fields
    value = getattr (self,'clean_% s'% name) ()
  File "/home/app_admin/venv_xxx_project/xxx_nippo_20201102/venv_xxx_project/xxx_project/working_list/forms.py", line 26, in clean_file
    for row in reader:
MemoryError
2020-11-18 15: 08: 01,058 [WARNING] /home/app_admin/venv_xxx_project/lib64/python3.7/site-packages/django/utils/log.py (Line: 228) Not Found: /.env
2020-11-18 15: 08: 03,157 [WARNING] /home/app_admin/venv_xxx_project/lib64/python3.7/site-packages/django/utils/log.py (Line: 228) Not Found: /
2020-11-18 15: 13: 46,996 [WARNING] /home/app_admin/venv_xxx_project/lib64/python3.7/site-packages/django/utils/log.py (Line: 228) Not Found: /nginx-logo. png
2020-11-18 15: 13: 46,998 [WARNING] /home/app_admin/venv_xxx_project/lib64/python3.7/site-packages/django/utils/log.py (Line: 228) Not Found: /poweredby.png

environment
Instance type: t2.micro
ap: django
web: nginx (gunicorn)
db: PostgreSQL

I was in a hurry and it was too early for me to resolve myself, so it was a hassle twice, but it doesn't matter what I do, please let me know if you have overlooked something. Thank you. * Due to the number of characters, the gunicorn log will be added below.

  • Answer # 1

    Apparently it's down due to lack of memory, so it seems almost certain.

    As I wrote in the comment, I feel that the design of letting the WEB server receive 1G files via HTTP request is rather unreasonable, so I put the file in S3 or process it based on this csv. If you want to do this, you should do something about the design, such as having another server perform the processing.
    I think it's a little different to raise the specs because that is the only bottleneck.

  • Answer # 2

    I will add it to the above question column because the number of characters is insufficient.

    gunicorn: log (timeout set to 9999)

    (venv_xxx_project) [app_admin @ ip-172-31-36-217 xxx_project] $gunicorn --bind 127.0.0.1:8000 xxx_project.wsgi --log-level = DEBUG --timeout 9999
    [2020-11-18 15:12:29 +0900] [13563] [DEBUG] Current configuration:
      config: None
      bind: ['127 .0.0.1: 8000']
      backlog: 2048
      workers: 1
      worker_class: sync
      threads: 1
      worker_connections: 1000
      max_requests: 0
      max_requests_jitter: 0
      timeout: 9999
      graceful_timeout: 30
      keepalive: 2
      limit_request_line: 4094
      limit_request_fields: 100
      limit_request_field_size: 8190
      reload: False
      reload_engine: auto
      reload_extra_files: []
      spew: False
      check_config: False
      preload_app: False
      sendfile: None
      reuse_port: False
      chdir:/home/app_admin/venv_xxx_project/xxx_nippo_20201102/venv_xxx_project/xxx_project
      daemon: False
      raw_env: []
      pidfile: None
      worker_tmp_dir: None
      user: 1001
      group: 1001
      umask: 0
      initgroups: False
      tmp_upload_dir: None
      secure_scheme_headers: {'X-FORWARDED-PROTOCOL':'ssl','X-FORWARDED-PROTO':'https','X-FORWARDED-SSL':'on'}
      forwarded_allow_ips: ['12 7.0.0.1']
      accesslog: None
      disable_redirect_access_to_syslog: False
      access_log_format:% (h) s% (l) s% (u) s% (t) s "% (r) s"% (s) s% (b) s "% (f) s" "% (a) ) s "
      errorlog:-
      loglevel: DEBUG
      capture_output: False
      logger_class: gunicorn.glogging.Loggerlogconfig: None
      logconfig_dict: {}
      syslog_addr: udp: // localhost: 514
      syslog: False
      syslog_prefix: None
      syslog_facility: user
      enable_stdio_inheritance: False
      statsd_host: None
      dogstatsd_tags:
      statsd_prefix:
      proc_name: None
      default_proc_name: xxx_project.wsgi
      pythonpath: None
      paste: None
      on_starting:<function OnStarting.on_starting at 0x7f2d588e0d40>on_reload:<function OnReload.on_reload at 0x7f2d588e0e60>when_ready:<function WhenReady.when_ready at 0x7f2d588e0f80>pre_fork:<function Prefork.pre_fork at 0x7f2d588f30e0>post_fork:<function Postfork.post_fork at 0x7f2d588f3200>post_worker_init:<function PostWorkerInit.post_worker_init at 0x7f2d588f3320>worker_int:<function WorkerInt.worker_int at 0x7f2d588f3440>worker_abort:<function WorkerAbort.worker_abort at 0x7f2d588f3560>pre_exec:<function PreExec.pre_exec at 0x7f2d588f3680>pre_request:<function PreRequest.pre_request at 0x7f2d588f37a0>post_request:<function PostRequest.post_request at 0x7f2d588f3830>child_exit:<function ChildExit.child_exit at 0x7f2d588f3950>worker_exit:<function WorkerExit.worker_exit at 0x7f2d588f3a70>nworkers_changed:<function NumWorkersChanged.nworkers_changed at 0x7f2d588f3b90>on_exit:<function OnExit.on_exit at 0x7f2d588f3cb0>proxy_protocol: False
      proxy_allow_ips: ['127.0.0.1']
      keyfile: None
      certfile: None
      ssl_version: 2
      cert_reqs: 0
      ca_certs: None
      suppress_ragged_eofs: True
      do_handshake_on_connect: False
      ciphers: None
      raw_paste_global_conf: []
      strip_header_spaces: False
    [2020-11-18 15:12:29 +0900] [13563] [INFO] Starting gunicorn 20.0.4
    [2020-11-18 15:12:29 +0900] [13563] [DEBUG] Arbiter booted
    [2020-11-18 15:12:29 +0900] [13563] [INFO] Listening at: http://127.0.0.1:8000 (13563)
    [2020-11-18 15:12:29 +0900] [13563] [INFO] Using worker: sync
    [2020-11-18 15:12:29 +0900] [13566] [INFO] Booting worker with pid: 13566
    [2020-11-18 15:12:29 +0900] [13563] [DEBUG] 1 workers
    [2020-11-18 15:13:00 +0900] [13566] [DEBUG] GET/working_list[2020-11-18 15:13:02 +0900] [13566] [DEBUG] GET/working_listworkinglistimport /
    [2020-11-18 15:13:31 +0900] [13566] [DEBUG] POST/working_listworkinglistimport /
    [2020-11-18 15:13:46 +0900] [13573] [INFO] Booting worker with pid: 13573
    [2020-11-18 15:13:46 +0900] [13573] [DEBUG] GET /nginx-logo.png
    [2020-11-18 15:13:46 +0900] [13573] [DEBUG] GET /poweredby.png

    Addendum 11-29-11: 29
    If you paste the result of the free command in the answer to the question, the header information will be left-justified, so paste it here so that there is no misunderstanding.

                total used free shared buff/cache available
    Mem: 983 774 64 108 144 12
    Swap: 0 0 0
                  total used free shared buff/cache available
    Mem: 983 773 72 108 137 16
    Swap: 0 0 0
                  total used free shared buff/cache available
    Mem: 983 773 70 109 139 15
    Swap: 0 0 0
                  total used free shared buff/cache available
    Mem: 983 774 67 109 141 14
    Swap: 0 0 0
                  total used free shared buff/cache available
    Mem: 983 774 65 109 143 12
    Swap: 0 0 0
                  total used free shared buff/cache available
    Mem: 983 774 63 110 145 11
    Swap: 0 0 0
                  total used free shared buff/cache available
    Mem: 983 774 71 110 137 15
    Swap: 0 0 0
                  total used free shared buff/cache available
    Mem: 983 160 665 110 156 619
    Swap: 0 0 0
                  total used free shared buff/cache available
    Mem: 983 160 666 110 156 619
    Swap: 0 0 0
                  total used free shared buff/cache available
    Mem: 983 160 665 110 157 619
    Swap: 0 0 0
                  total used free shared buff/cache available
    Mem: 983 160 664 110 158 618
    Swap: 0 0 0
                  total used free shared buff/cache available
    Mem: 983 160 663 110 159 618
    Swap: 0 0 0
                  total used free shared buff/cache available
    Mem: 983 160 656 110 165 614
    Swap: 0 0 0