Log message #4183943

# At Username Text
# Apr 18th 2019, 17:52 devito in cake 2 is there a databaseengine for caching?
# Apr 18th 2019, 17:52 ricksaccous which is lengthy for many records*
# Apr 18th 2019, 17:52 ricksaccous which may be lengthy
# Apr 18th 2019, 17:52 ricksaccous saveMany will always expect hydrated entities in
# Apr 18th 2019, 17:51 ricksaccous raw queries always best option
# Apr 18th 2019, 17:51 pedrorocha As curiosity, if I were using the most recent version of cakephp, should I do it with saveMany() or raw queries is still my best option?
# Apr 18th 2019, 17:48 pedrorocha Inner loop memory usage run: 1 -> 2.89 MB Inner loop memory usage run: 2 -> 37.19 MB Inner loop memory usage run: 3 -> 37.61 MB Inner loop memory usage run: 4 -> 38.04 MB Total loop memory usage -> 31.85 MB Awesome! Thank you guys
# Apr 18th 2019, 17:37 pedrorocha Thank you!
# Apr 18th 2019, 17:36 ricksaccous what*
# Apr 18th 2019, 17:36 ricksaccous ....
# Apr 18th 2019, 17:36 ricksaccous i think that he linked up there is sufficient
# Apr 18th 2019, 17:35 pedrorocha Just to be sure $db = ConnectionManager::getDataSource('default'); $db->rawQuery("INSERT ...."); is what I should be using in case of many records?
# Apr 18th 2019, 17:34 ricksaccous thanks
# Apr 18th 2019, 17:33 ricksaccous interesting, i'll consider this if i run into bottlenecks
# Apr 18th 2019, 17:33 admad update or insert depending on what you need
# Apr 18th 2019, 17:33 admad use what i linked above
# Apr 18th 2019, 17:32 ricksaccous @admad speaking of i have a point where i might be patching in 400 records.... do you recommend saving that instead with raw sql if possible?
# Apr 18th 2019, 17:31 admad https://book.cakephp.org/3.0/en/orm/query-builder.html#inserting-data
# Apr 18th 2019, 17:31 ricksaccous yeah ;)
# Apr 18th 2019, 17:31 ricksaccous using the ORM to just build the raw save query
# Apr 18th 2019, 17:31 pedrorocha Unfortunately I'm using cakephp 3.1 and I'm afraid I can upgrade it in production.. Can it be done in cakephp 3.1?
# Apr 18th 2019, 17:31 ricksaccous and probably skipping to raw sql saves is faster
# Apr 18th 2019, 17:30 ricksaccous i'd just be worried about the patchEntities anyway
# Apr 18th 2019, 17:30 admad ORM methods are meant for bulk saves anyway. Use lower level db methods
# Apr 18th 2019, 17:30 ricksaccous no you'd have to do a saveMany
# Apr 18th 2019, 17:30 pedrorocha ->save() accepts my whole array of data?
# Apr 18th 2019, 17:29 ricksaccous in one query
# Apr 18th 2019, 17:29 ricksaccous @pedrorocha why not save 15k per loop pass
# Apr 18th 2019, 17:27 slackebot loop memory usage run: 10 -> 2.69 MB Inner loop memory usage run: 11 -> 2.69 MB Inner loop memory usage run: 12 -> 2.48 MB Total loop memory usage -> 2.41 MB Why is this happening? How can I free the memory used by the inner loop on each run?
# Apr 18th 2019, 17:27 slackebot I comment the `->save()` line, it looks like this (as it should be): Inner loop memory usage run: 1 -> 2.66 MB Inner loop memory usage run: 2 -> 2.68 MB Inner loop memory usage run: 3 -> 2.69 MB Inner loop memory usage run: 4 -> 2.69 MB Inner loop memory usage run: 5 -> 2.69 MB Inner loop memory usage run: 6 -> 2.69 MB Inner loop memory usage run: 7 -> 2.69 MB Inner loop memory usage run: 8 -> 2.69 MB Inner loop memory usage run: 9 -> 2.69 MB Inner
# Apr 18th 2019, 17:27 slackebot -> 23.52 MB Inner loop memory usage run: 5 -> 29.82 MB Inner loop memory usage run: 6 -> 38.11 MB Inner loop memory usage run: 7 -> 44.41 MB Inner loop memory usage run: 8 -> 50.7 MB Inner loop memory usage run: 9 -> 56.99 MB Inner loop memory usage run: 10 -> 67.29 MB Inner loop memory usage run: 11 -> 73.58 MB Inner loop memory usage run: 12 -> 79.67 MB Total loop memory usage -> 82.13 MB It should not be wasting this much memory on each loop, when
# Apr 18th 2019, 17:27 pedrorocha Hello guys, I'm trying to save 320k records on my db using a foreach loop to iterate through an array of data. I loaded the model outside the loop and split the data in batches of 15k, the inner loop looks like this: ```foreach ($users as $user) { $dataToSave = array('user_id' => $user->id, 'notification_id' => $notificationId); $userNotification = $this->UserNotification->newEntity(); $userNotification =
# Apr 18th 2019, 17:27 slackebot $this->UserNotification->patchEntity($userNotification, $dataToSave); $this->UserNotification->save($userNotification); }``` It's wrapped around a while loop counting the number of runs and managing the batches. When I run this code and track the memory usage of it, this is what I'm getting: Inner loop memory usage run: 1 -> 2.61 MB Inner loop memory usage run: 2 -> 9.43 MB Inner loop memory usage run: 3 -> 16.23 MB Inner loop memory usage run: 4
# Apr 18th 2019, 16:32 admad @mrfeedback `<?=` is short echo, different than short open tag. It's unaffected by php config and always available
# Apr 18th 2019, 16:20 mrfeedback `Note that this directive does not control the <?= shorthand tag, which can be ; used regardless of this directive.`
# Apr 18th 2019, 16:20 mrfeedback oh. okay forget what i said
# Apr 18th 2019, 16:18 mrfeedback yep i mean `<?=` as `short_tag_open` is also not recommened as `On` for production environments according to the official php.ini https://github.com/php/php-src/blob/master/php.ini-production
# Apr 18th 2019, 16:14 ricksaccous the second is never used
# Apr 18th 2019, 16:13 ricksaccous @mrfeedback <?= is very distinct from <? ?>
# Apr 18th 2019, 16:13 waspinator never heard of it being a problem before
# Apr 18th 2019, 16:13 waspinator you mean this `<?=`