How to efficiently import ~16M users with custom fields?
-
Hi everyone,
I’m working on a project where I need to import approximately 16 million users into a NodeBB forum.
In addition to the standard fields, I also need to set some custom fields for each user during the import.I’ve already tried a few approaches:
- Direct writes to MongoDB (objects collection), but it seems there’s a lot happening under the hood when creating a user (hooks, hashes, plugins, etc.), and it’s almost impossible to replicate everything manually.
- Using the Write API (POST /api/v3/users) works correctly, but it’s painfully slow at scale.
- The CSV Import plugin doesn’t allow me to set the user’s creation date (timestamp), which is essential for my use case, so it’s not an option either.
Has anyone dealt with importing a very large number of users into NodeBB?
Do you have any recommendations or best practices for this kind of task?Thanks in advance for any advice or ideas!
-
Hi everyone,
I’m working on a project where I need to import approximately 16 million users into a NodeBB forum.
In addition to the standard fields, I also need to set some custom fields for each user during the import.I’ve already tried a few approaches:
- Direct writes to MongoDB (objects collection), but it seems there’s a lot happening under the hood when creating a user (hooks, hashes, plugins, etc.), and it’s almost impossible to replicate everything manually.
- Using the Write API (POST /api/v3/users) works correctly, but it’s painfully slow at scale.
- The CSV Import plugin doesn’t allow me to set the user’s creation date (timestamp), which is essential for my use case, so it’s not an option either.
Has anyone dealt with importing a very large number of users into NodeBB?
Do you have any recommendations or best practices for this kind of task?Thanks in advance for any advice or ideas!
Somethings you can try:
- Bypass the write api and call
User.createdirectly in a migration script. - One of the expensive parts of
User.createis hashing the password, you can disable this part and force everyone to reset their password. - Disable sending a welcome notification everytime a user is created. [[code](https://github.com/NodeBB/NodeBB/blob/master/src/user/create.js#L101)]
- Disable welcome email. [[code](https://github.com/NodeBB/NodeBB/blob/master/src/user/create.js#L152-L157)]
- Disable registration analytics. [[code](https://github.com/NodeBB/NodeBB/blob/master/src/user/create.js#L98)]
- If all the usernames are unique in your data you can disable the unique name generator. [[code](https://github.com/NodeBB/NodeBB/blob/master/src/user/create.js#L65-L70)]
- Same as above but for emails, if all the emails are already unique you can disable the email available check. [[code](https://github.com/NodeBB/NodeBB/blob/master/src/user/create.js#L152-L157)]
These are just some of the low effort ones.
