| # |
Apr 18th 2019, 17:52 |
 ricksaccous |
which is lengthy for many records* |
| # |
Apr 18th 2019, 17:52 |
ricksaccous |
which may be lengthy |
| # |
Apr 18th 2019, 17:52 |
ricksaccous |
saveMany will always expect hydrated entities in |
| # |
Apr 18th 2019, 17:51 |
ricksaccous |
raw queries always best option |
| # |
Apr 18th 2019, 17:51 |
pedrorocha |
As curiosity, if I were using the most recent version of cakephp, should I do it with saveMany() or raw queries is still my best option? |
| # |
Apr 18th 2019, 17:48 |
pedrorocha |
Inner loop memory usage run: 1 -> 2.89 MB Inner loop memory usage run: 2 -> 37.19 MB Inner loop memory usage run: 3 -> 37.61 MB Inner loop memory usage run: 4 -> 38.04 MB Total loop memory usage -> 31.85 MB Awesome! Thank you guys |
| # |
Apr 18th 2019, 17:37 |
pedrorocha |
Thank you! |
| # |
Apr 18th 2019, 17:36 |
ricksaccous |
what* |
| # |
Apr 18th 2019, 17:36 |
ricksaccous |
.... |
| # |
Apr 18th 2019, 17:36 |
ricksaccous |
i think that he linked up there is sufficient |
| # |
Apr 18th 2019, 17:35 |
pedrorocha |
Just to be sure $db = ConnectionManager::getDataSource('default'); $db->rawQuery("INSERT ...."); is what I should be using in case of many records? |
| # |
Apr 18th 2019, 17:34 |
ricksaccous |
thanks |
| # |
Apr 18th 2019, 17:33 |
ricksaccous |
interesting, i'll consider this if i run into bottlenecks |
| # |
Apr 18th 2019, 17:33 |
admad |
update or insert depending on what you need |
| # |
Apr 18th 2019, 17:33 |
admad |
use what i linked above |
| # |
Apr 18th 2019, 17:32 |
ricksaccous |
@admad speaking of i have a point where i might be patching in 400 records.... do you recommend saving that instead with raw sql if possible? |
| # |
Apr 18th 2019, 17:31 |
admad |
https://book.cakephp.org/3.0/en/orm/query-builder.html#inserting-data |
| # |
Apr 18th 2019, 17:31 |
ricksaccous |
yeah ;) |
| # |
Apr 18th 2019, 17:31 |
ricksaccous |
using the ORM to just build the raw save query |
| # |
Apr 18th 2019, 17:31 |
pedrorocha |
Unfortunately I'm using cakephp 3.1 and I'm afraid I can upgrade it in production.. Can it be done in cakephp 3.1? |
| # |
Apr 18th 2019, 17:31 |
ricksaccous |
and probably skipping to raw sql saves is faster |
| # |
Apr 18th 2019, 17:30 |
ricksaccous |
i'd just be worried about the patchEntities anyway |
| # |
Apr 18th 2019, 17:30 |
admad |
ORM methods are meant for bulk saves anyway. Use lower level db methods |
| # |
Apr 18th 2019, 17:30 |
ricksaccous |
no you'd have to do a saveMany |
| # |
Apr 18th 2019, 17:30 |
pedrorocha |
->save() accepts my whole array of data? |
| # |
Apr 18th 2019, 17:29 |
ricksaccous |
in one query |
| # |
Apr 18th 2019, 17:29 |
ricksaccous |
@pedrorocha why not save 15k per loop pass |
| # |
Apr 18th 2019, 17:27 |
slackebot |
loop memory usage run: 10 -> 2.69 MB Inner loop memory usage run: 11 -> 2.69 MB Inner loop memory usage run: 12 -> 2.48 MB Total loop memory usage -> 2.41 MB Why is this happening? How can I free the memory used by the inner loop on each run? |
| # |
Apr 18th 2019, 17:27 |
slackebot |
I comment the `->save()` line, it looks like this (as it should be): Inner loop memory usage run: 1 -> 2.66 MB Inner loop memory usage run: 2 -> 2.68 MB Inner loop memory usage run: 3 -> 2.69 MB Inner loop memory usage run: 4 -> 2.69 MB Inner loop memory usage run: 5 -> 2.69 MB Inner loop memory usage run: 6 -> 2.69 MB Inner loop memory usage run: 7 -> 2.69 MB Inner loop memory usage run: 8 -> 2.69 MB Inner loop memory usage run: 9 -> 2.69 MB Inner |
| # |
Apr 18th 2019, 17:27 |
slackebot |
-> 23.52 MB Inner loop memory usage run: 5 -> 29.82 MB Inner loop memory usage run: 6 -> 38.11 MB Inner loop memory usage run: 7 -> 44.41 MB Inner loop memory usage run: 8 -> 50.7 MB Inner loop memory usage run: 9 -> 56.99 MB Inner loop memory usage run: 10 -> 67.29 MB Inner loop memory usage run: 11 -> 73.58 MB Inner loop memory usage run: 12 -> 79.67 MB Total loop memory usage -> 82.13 MB It should not be wasting this much memory on each loop, when |
| # |
Apr 18th 2019, 17:27 |
pedrorocha |
Hello guys, I'm trying to save 320k records on my db using a foreach loop to iterate through an array of data. I loaded the model outside the loop and split the data in batches of 15k, the inner loop looks like this: ```foreach ($users as $user) { $dataToSave = array('user_id' => $user->id, 'notification_id' => $notificationId); $userNotification = $this->UserNotification->newEntity(); $userNotification = |
| # |
Apr 18th 2019, 17:27 |
slackebot |
$this->UserNotification->patchEntity($userNotification, $dataToSave); $this->UserNotification->save($userNotification); }``` It's wrapped around a while loop counting the number of runs and managing the batches. When I run this code and track the memory usage of it, this is what I'm getting: Inner loop memory usage run: 1 -> 2.61 MB Inner loop memory usage run: 2 -> 9.43 MB Inner loop memory usage run: 3 -> 16.23 MB Inner loop memory usage run: 4 |
| # |
Apr 18th 2019, 16:32 |
admad |
@mrfeedback `<?=` is short echo, different than short open tag. It's unaffected by php config and always available |
| # |
Apr 18th 2019, 16:20 |
mrfeedback |
`Note that this directive does not control the <?= shorthand tag, which can be ; used regardless of this directive.` |
| # |
Apr 18th 2019, 16:20 |
mrfeedback |
oh. okay forget what i said |
| # |
Apr 18th 2019, 16:18 |
mrfeedback |
yep i mean `<?=` as `short_tag_open` is also not recommened as `On` for production environments according to the official php.ini https://github.com/php/php-src/blob/master/php.ini-production |
| # |
Apr 18th 2019, 16:14 |
ricksaccous |
the second is never used |
| # |
Apr 18th 2019, 16:13 |
ricksaccous |
@mrfeedback <?= is very distinct from <? ?> |
| # |
Apr 18th 2019, 16:13 |
waspinator |
never heard of it being a problem before |
| # |
Apr 18th 2019, 16:13 |
waspinator |
you mean this `<?=` |
| # |
Apr 18th 2019, 16:11 |
waspinator |
https://github.com/cakephp/cakephp.org/blob/master/src/Template/Element/Layout/default/footer/social_links.ctp |