Before I get in to the solution of this article, let me tell you how it started and fill you in on the problem that arose. I wrote a procedure to get daily deltas of users - those of which who had created/updated their account on the given day (and including the day before for good measure on the GMT timestamp). The simple search criteria was just the following:
updated_at:[yyyy-mm-dd TO yyyy-mm-dd]
Simple, right? the []'s being the dates are inclusive, while using {} would mean exclusively. Auth0 lets you mix these on either side depending on your use. While this is all well and good, Auth0 will limit the number of results (even with paging) to 1000 only.
So, your first option is that you could have your procedure create a user export job, and then parsing through the results and eliminating those which do not meet your updated_at search criteria. I can tell you first hand that eventually the amount of users will just get to be too much and cumbersome to process through. However, I've been told by the good people at Auth0 that currently in their development pipelines, they are making an update to this procedure where you can use search parameters. More on that later.
Otherwise, I have come up with the following:
* One thing to note on the above script is that on line 9, set page to 0, not 1.
This creates a recursive promise, and running through the pagination for as much as it gives it. For every 1000th record it comes to, it takes the created_at date from the last user, and concatenates on to the end of the query:
created_at:{yyyy-mm-ddThh:mm:ss.nnnZ TO *]
This makes it go from this date/time (exclusively) to the max it can find. Important to note here as well that results from every query are being sorted from the created_at field in ascending order. Which means it's going to start from the earliest first.
As always, hope this helps. I sure wish I knew this in its entirety before I started, frustrating having to make amends and re-query past days in order to catch up.
J.
Comments
Post a Comment