How to set up DIY batching in Salesforce Flows (to overcome the error “Too many retries of batch save in the presence of Apex triggers with failures” or for whatever reasons YOU may need)

If you google “how to batch jobs in Salesforce flows” you’ll get some results, and they may work for you. None of them worked for me in the situation I was experiencing as far as I could tell. But I was 99% sure that I could figure out a DIY solution - and that part can be really fun! It took me a few days, but it works, and I want to share it so that anyone else who might need it can use it. Well, something like what I did, because my solution is slightly particular to my situation. But I hope you can interpret and adapt it for your situation - or at least be inspired to try!

The Situation / Use Case: We use Salesforce to keep track of the students we serve - in particular we deliver extra-curricular instructional classes to students. We create Class records, then connect students to those classes with a junction object called Class Registrations. The Class Registration record has two linked records - the Class record and the student’s Contact record.

At first this was a simple bulkification job (ok, not simple really. Bulkification can be tough to learn, and I’m also using screens and datatables and loops and blah blah blah - it only seems simple after you’ve done it a ton of times). Maybe the better word is straightforward - it was a straightforward bulkification job. We used a screen component to show the list of students, let users pick which ones should be registered into a new Class, then we loop through each of the students that was chosen, assign them to what I like to call a fake or temporary or invisible Class Registration record, add that record to a Record Collection, and go through the loop again until we’ve created these fake records for all the chosen students, filled up the Record Collection, and then we exit the loop and use a Create Records component to create all the records in the Record Collection.

The Problem: After a Class Registration record is created in our system, someone else’s funky Apex code is triggered to create “Attendance” records for each student that was registered. This is HIGHLY valuable and we need this to happen. However, the Apex code is part of a managed package, so the code is hidden and I’m nervous to mess with it or try to get rid of it and replace it with our own process (at least right now). What happened when we had a user select a high volume of students for whom to create Class Registration records was that when the Create Records component ran, it spit back an error that read:

So if we tried to create too many Class Registration records at one time, the Apex trigger that then creates Attendance records was going to bring us down.

BUT CAN’T NO ONE BRING US DOWN.

Now this is going to get a bit long and convoluted, but who cares because it works! Let’s see how we solved it:

It looks so long, right!?!? I’ll break it down, piece by piece.

This first part is your classic bulkification solution. The first part is a screen that shows the user the list of students they chose to create Class Registration records for (they did this in previous steps), shows them the COUNT of students they have chosen, and asks them to confirm they want to go forward. That COUNT of records comes into play later, even though it is in a step that comes before the image shown here.

Typically after this loop finishes we can just add the CREATE RECORDS component and go on with our days happily and merrily. But if you hit the “Too many retries of batch save in the presence of Apex triggers with failures” error, then you’ll need to add a few (9 by my count) steps/components to overcome it.

In English, here is what I wanted to happen next: Let’s say the user chose 35 students for whom to create Class Registration records. I wanted to isolate the first 10 of those 35, create them, then go back for the next 10, create those, then the next 10, then the last 5, and then be finished. This is called Batching. You probably use it in Salesforce all the time in different ways than this. Here is how I found the way to batch, in a few parts:

Remember in our Bulkification loop we created a RECORD COLLECTION that held all the temporary records that I want to create. It contains temporary (or fake, or invisible) records that have just 2 data points on them - the Class record ID and the Contact record ID. That is all we need to create a Class Registration record. The RECORD COLLECTION is a variable that I get to name, and I chose to name it “TempRegistrationCollection” (aren’t I clever). Step 1 above is that I take all the records from TempRegistrationCollection and just add them all to another RECORD COLLECTION variable that I call “BatchPriorClass” (that name comes from the fact in this flow users can go down two paths at the beginning - one of which is to pick a ton of students from a prior class they were in and register all or some of them into a future class. This path I’m showing you is when the user goes down this route). So in our scenario the user chose 35 students to make Class Registration records for. So now the TempRegistrationCollection variable and the BatchPriorClass record collection variable are both holding all 35 of those temporary records to create. WHY in the world did I do this? Oh, you’ll find out down below in a few steps, but just trust me for now, we want this separate record collection.

Next I REMOVE all records from another record collection called “ToRemove” from the BatchPriorClass record collection. The ToRemove record collection is populated in a later step. So it seems counterintuitive to have it here in the process. The FIRST time this process runs, there is nothing in the ToRemove record collection, so nothing is removed from the BatchPriorClass record collection. But LATER in the flow, we’ll take the 10 records we’ve already created (we’ll go 10 at a time below, because 10 at a time does not trigger that error message) and add them to the ToRemove collection, so we can remove them from the BatchPriorClass collection on the second, third, fourth, or however many batches we need to go through.

Next step, we SORT the BatchPriorClass record collection. A great benefit of the SORT component in Flows is that you can limit the number of records that stay in the record collection after you sort it. A great limitation of this feature is that you have to type in a static number, like 10. You can’t have it be a variable or a formula. 

Now my BatchPriorClass record collection has ONLY 10 records in it - the first 10 I sorted from the total 35 that were originally in that collection. Believe it or not, I’m going to take these 10 and add them to YET ANOTHER record collection, this time called the “CreateBatch” collection:

So at this point, the TempRegistration collection still has 35 records in it, the BatchPriorClass collection has only the first 10 records in it, and now the CreateBatch collection has those same 10 records in it. So next we create those 10 records:

Something new I learned based on this: BEFORE the records are created, the CreateBatch collection still held those “temporary” (or fake or invisible) records that had 2 data points - the Class ID and the Contact ID. But AFTER the records are created, the CreateBatch collection now contains the REAL records that were just created, and you can see this because the collection is now filled with 10 records but only 1 data point about them - the new record ID that was created for them!! For example, before the create, the CreateBatch collection had an item that looked like: “ContactID = ‘0030W00003srcfaQAA’, ClassID = ‘a2MAJ00000011V72AI’”

But after the create step, that item in the CreateBatch collection looks like this: “ID = ‘a2KAJ000000096T2AQ’” - which is the ID of the Class Registration record. And this change to the CreateBatch collection is why we needed a few extra steps of adding the same records to different collections in the flow.

Now some fun tricks and acrobatics: 

First, we want to empty that CreateBatch record collection. If we don’t do this and a few other steps, we’ll hit another error when the Flow would accidentally try to create Duplicate Class Registration records (thank goodness we already had a validation rule against that!!!).

Next, we take all the records that we had created in the BatchPriorClass (which is still the 10 records and they did NOT change when we did the Create Records step - very important!) and add those to the previously referenced ToRemove record collection. So the NEXT time around, there will be 10 records to remove from the process, so that we don’t try to create those same ones again.

Another clean-up step is next - we empty out the TempRegistrationsCollection from all the way back at the beginning. We do this because we are going to head back into that initial loop soon, and if we don’t empty the collection now, then the loop will ADD the same 35 temporary (fake, invisible) records to the collection, making it 70 records long. NO. THANK. YOU.

Next we count how many records are in the ToRemove collection and add that number to a variable named “CoutnOfToRemove” - clever again, right?. Right now it is 10. On the next go around it will add another 10 to it (the next batch of 10 records) and the ToRemove collection will be 20, then 30, etc. So right now the Count of records in the ToRemove collection has gone from 0 to 10.

We did this count so that we don’t get stuck in an infinite loop trying to always go back and get more records to create. Our next step is a decision. If the CountOfSelected records (that count of how many records the user chose at the beginning of our process - in our example it was 35) is NOT EQUAL to the count of the records in the ToRemove record collection and the CountOfToRemove variable. Right now on this first batch, the CountOfToRemove (10) will NOT equal the count of records in CountOfSelected (35).

Now we have to do a funny little thing. IF you are using the Auto-Layout in the Flow designer, you’ll need to switch to the FreeForm layout. This will allow you to go BACK to the loop with one of these outcomes. IF the Count of the CountOfToRemove does equal the CountOfSelected, that means we’ve created all the records we needed to and we don’t need to go back and do another batch, so we direct users to an ending Thank You screen. If we need more batches to create, we go back to the loop and basically start the whole process over again.

After you make that connection, you can go back to the Auto-Layout format and it will look like this:

On the next go around, we fill up TempRegistrationRecords collection again with the same 35 that were originally chosen and add those 35 again to the BatchPriorClass collection, but THIS time when we Remove the ToRemove from BatchPriorClass, 10 temporary (fake, invisible) records will be removed from that collection - the same ones we just created. **Remember, if we had just tried to remove the ones from the CreateBatch collection, then it would not have worked, because the TempRegistrationRecords and BatchPriorClass collections are filled with these temporary items made up of ClassID and ContactID - but the CreateBatch collection got changed from that format into just the RecordID of the new Class Registration records that were created. The two of these lists won’t quite communicate with each other the way we want them to.

And there you have it - batch jobs in a Salesforce flow that overcome a “Too many retries of batch save in the presence of Apex triggers with failures” error, and serve as a template for any other batch jobs you may need to DIY in your flow.

Let me know if this helps you at all!!