| # |
Sep 13th 2019, 14:23 |
 admad |
you should just upload file and use a shell / command to do the importing in background |
| # |
Sep 13th 2019, 14:23 |
admad |
ideally importing 500,000 lines of csv should be even done in a web request. |
| # |
Sep 13th 2019, 14:22 |
felipe.marinho |
Yes, his approach was to use the input type="hidden"... I'll try to upload the CSV, match the fields, and after use the uploaded CSV. |
| # |
Sep 13th 2019, 14:22 |
admad |
sorry but i am not going to bother reading that :slightly_smiling_face: |
| # |
Sep 13th 2019, 14:21 |
felipe.marinho |
I adapted this tutorial: https://quickadminpanel.com/blog/how-to-import-csv-in-laravel-and-choose-matching-fields/ |
| # |
Sep 13th 2019, 14:20 |
admad |
if not possible then query builder to do multi insert |
| # |
Sep 13th 2019, 14:20 |
admad |
next don't use ORM methods to insert. if server allows Best would be to `LOAD DATA INFILE` syntax to directly make mysql use the csv file to insert rows |
| # |
Sep 13th 2019, 14:18 |
admad |
also make sure post_max_size and max_file_size are set to large enough values in php.ini to allow large file uploads |
| # |
Sep 13th 2019, 14:18 |
admad |
using hidden input to pass csv data sounds like a very brittle approach. use a file input to upload |
| # |
Sep 13th 2019, 14:17 |
ckjksl |
Locally will be faster, but are you trying to save 500,000 entities all at once? |
| # |
Sep 13th 2019, 14:15 |
slackebot |
approach is the best option for now... locally it will be faster. |
| # |
Sep 13th 2019, 14:15 |
felipe.marinho |
I'm using MYSQL 5.5, I read about update the mysql version, I'll try to install the MariaDB updated and test it... and maybe latter try to create all this entities using an uploaded csv... I tried to use the file inside the TMP folder, but after it uploads and use it inside the action the server delete the file... I saved the tmp file path in the input type="hidden" i created first, but the file was deleted :( I think the upload |
| # |
Sep 13th 2019, 14:11 |
ckjksl |
Maybe instead of trying to create them all at once, you can stagger it so it saves 100 (or whatever number you want) at a time? 1000 seconds isn't very long |
| # |
Sep 13th 2019, 14:07 |
felipe.marinho |
@ckjksl The PHP Timeout in the php.ini is max_execution_time=1000, and it crashes instantaneously, not even a single second lol... Yes, I'm using saveMany to save all those data... |
| # |
Sep 13th 2019, 14:02 |
slackebot |
=> $this->toIso8601String(), 'timezone' => $this->getTimezone()->getName(), 'fixedNowTime' => static::hasTestNow() ? static::getTestNow()->toIso8601String() : false ]; ``` seems to trigger it. |
| # |
Sep 13th 2019, 14:02 |
sdevore |
I realize I asked this yesterday afternoon but any thoughts on how to find where xdebug is crashing in __debugInfo() after updating from 3.6 to 3.7, I updated chronos to 1.2.8 since that seemed to mention some issues with __debugInfo() the line in my code that seems to trigger it is $enrolled_course->set(‘course’, $overlap); and stepping though it is almost seems that something in DateFormat::__debugInfo(); ``` return [ 'ti |
| # |
Sep 13th 2019, 14:02 |
ckjksl |
Is it because of PHP timeout? |
| # |
Sep 13th 2019, 14:00 |
felipe.marinho |
Locally, using the WAMP the page loads for 2 minutes, but create the 500.000 entities.... but in the production the page crash (ERR_CONNECTION_RESET on chrome) when I start the processing... |
| # |
Sep 13th 2019, 13:56 |
ckjksl |
Would it be easier to save the file and read the lines you need rather than passing it through `type="hidden"`? |
| # |
Sep 13th 2019, 13:55 |
ckjksl |
@felipe.marinho Are you sure the 500,000 line CSV uploaded successfully? |
| # |
Sep 13th 2019, 13:27 |
info315 |
I will track it, looks like this is the best practice |
| # |
Sep 13th 2019, 13:15 |
dereuromark |
if you dont leverage it, you can also exclude for now :slightly_smiling_face: |
| # |
Sep 13th 2019, 13:11 |
info315 |
Ok |
| # |
Sep 13th 2019, 13:08 |
dereuromark |
usually people track it for project level. |
| # |
Sep 13th 2019, 13:00 |
info315 |
Should the file `Migrations/schema-dump-default.lock` be excluded from git or better get tracked? |
| # |
Sep 13th 2019, 12:59 |
felipe.marinho |
hey guys... I created an CSV uploader action, but I'm passing the CSV data to from 1 page to another using an input type="hidden". I tested with 30 lines, 100 lines, and it was OK BUT now the client is testing with a 500.000 lines CSV, and it's not working. What is another approach to pass this content and not screw up with the user's browser? |
| # |
Sep 13th 2019, 11:12 |
spriz |
Ah wait, it actually does in `PhpConfig` |
| # |
Sep 13th 2019, 11:11 |
spriz |
I guess the Configure::load does not fail even though there is no file? Or no? |
| # |
Sep 13th 2019, 10:57 |
lilhermit |
Thanks anyway guys |
| # |
Sep 13th 2019, 10:57 |
lilhermit |
maybe, I just like to keep my build/deployment as simple as possible |
| # |
Sep 13th 2019, 10:56 |
dereuromark |
Maybe. Writing out an empty file on deploy was always easier ^^ |
| # |
Sep 13th 2019, 10:51 |
lilhermit |
@dereuromark might be worth adding the loading of `app_local.php` in a file_exists on your blog? |
| # |
Sep 13th 2019, 10:50 |
lilhermit |
I thought I must have been missing something |
| # |
Sep 13th 2019, 10:50 |
spriz |
:tada: |
| # |
Sep 13th 2019, 10:50 |
lilhermit |
Yes just tested it and your correct ;) |
| # |
Sep 13th 2019, 10:49 |
lilhermit |
perfect then ignore everything I've said it won't break on deployment ;) |
| # |
Sep 13th 2019, 10:49 |
spriz |
unless you removed it, I guess it is :ok_hand: |
| # |
Sep 13th 2019, 10:49 |
spriz |
https://github.com/cakephp/app/blob/master/src/Console/Installer.php#L59 @lilhermit |
| # |
Sep 13th 2019, 10:49 |
lilhermit |
I did not know the `cp` is one on `composer install` ? |
| # |
Sep 13th 2019, 10:47 |
spriz |
actually when you do `composer install`, exactly that is done by default :slightly_smiling_face: |
| # |
Sep 13th 2019, 10:46 |
spriz |
Put all the sensitive stuff in env vars and move on :ok_hand: |