mongodb

This story was originally published on HackerNoon at: https://hackernoon.com/lessons-i-learned-from-managing-hundreds-of-millions-of-data-in-mongodb.
In this post, I will share r.eal experience that I gained while working with hundred's of millions of pieces of data in MongoDB
Check more stories related to programming at: https://hackernoon.com/c/programming. You can also check exclusive content about #mongodb, #scaling, #database, #best-mongodb-practices, #mongodb-lessons, #bulk-operations, #aggregation-pipeline, #mastering-mongodb, and more.

This story was written by: @thedevtimeline. Learn more about this writer by checking @thedevtimeline's about page, and for more stories, please visit hackernoon.com.

In this post, I will share real experience that I gained while working with hundreds of millions of pieces of data in MongoDB. Don't store all data in a single Mongo collection. Use Bulk Operations to execute multiple write operations (inserts, updates, deletes) efficiently.

This Week in Startups is brought to you by…

OpenPhone. Create business phone numbers for you and your team that work through an app on your smartphone or desktop. TWiST listeners can get an extra 20% off any plan for your first 6 months at openphone.com/twist

Embroker. The Embroker Startup Insurance Program helps startups secure the most important types of insurance at a lower cost and with less hassle. Save up to 20% off of traditional insurance today at Embroker.com/twist. While you’re there, get an extra 10% off using offer code TWIST.

Roots is a real estate investment platform for all investors. There are no entry fees, and you can start with as little as $100. Head to investwithroots.com/TWIST to sign up and start investing today!

*

Today’s show:

MongoDB CEO Dev Ittycheria joins Jason to discuss recent tech trends (10:58), the move towards remote work (17:58), his strategy for building winning teams (37:17), and much more!

*

Time stamps:

(00:00) MongoDB CEO Dev Ittycheria joins Jason

(2:32) Dev’s background and how MongoDB operates

(9:28) OpenPhone - Get 20% off your first six months at https://openphone.com/twist

(10:58) The last 9 months in tech and the pace of AI

(13:57) RIFs and staffing a modern-day tech company

(17:58) Remote, Hybrid, or back to the office

(22:25) Embroker - Use code TWIST to get an extra 10% off insurance at https://Embroker.com/twist

(25:33) Hiring and inspiring the next generation

(28:47) The trust but verify operating philosophy

(31:32) Entitlement in the tech industry

(35:52) Roots - Head to investwithroots.com/TWIST to sign up and start investing today!

(37:17) Great leadership and managing a team

(41:14) Bad management and the feedback cycle

(53:30) Recruiting talent in the U.S.

*

Check out MongoDB: https://www.mongodb.com

Follow Dev: https://twitter.com/dittycheria

*

Read LAUNCH Fund 4 Deal Memo: https://www.launch.co/four

Apply for Funding: https://www.launch.co/apply

Buy ANGEL: https://www.angelthebook.com

Great recent interviews: Steve Huffman, Brian Chesky, Aaron Levie, Sophia Amoruso, Reid Hoffman, Frank Slootman, Billy McFarland, PrayingForExits, Jenny Lefcourt

Check out Jason’s suite of newsletters: https://substack.com/@calacanis

*

Follow Jason:

Twitter: https://twitter.com/jason

Instagram: https://www.instagram.com/jason

LinkedIn: https://www.linkedin.com/in/jasoncalacanis

*

Follow TWiST:

Substack: https://twistartups.substack.com

Twitter: https://twitter.com/TWiStartups

YouTube: https://www.youtube.com/thisweekin

*

Subscribe to the Founder University Podcast: https://www.founder.university/podcast

In this episode of Syntax, Wes and Scott talk about their respective processes for converting JavaScript to TypeScript in their projects, why you might want to, and tooling for moving to TypeScript.

Sentry - Sponsor

If you want to know what’s happening with your code, track errors and monitor performance with Sentry. Sentry’s Application Monitoring platform helps developers see performance issues, fix errors faster, and optimize their code health. Cut your time on error resolution from hours to minutes. It works with any language and integrates with dozens of other services. Syntax listeners new to Sentry can get two months for free by visiting Sentry.io and using the coupon code TASTYTREAT during sign up.

Show Notes

• 00:10 Welcome
• 00:36 Sponsor: Sentry
• 02:23 Electrical updates
• 08:16 Moving to TypeScript
• 09:34 What are you doing with GraphQL?
• GraphQL Codegen
• 11:35 Why move to TypeScript?
• 12:00 Refactoring is easier
• 14:58 Better DX
• 18:42 Adding a tsconfig.json file
• 22:43 Figure out your tooling
• 23:50 Type Checker
• 25:13 MongoDB to TypScript Generator
• 26:44 Eslint-Typescript
• 28:03 Deprank
• Deprank
• 29:18 Refactor and rename to .ts
• 32:40 Typing your code
• 43:04 Utilities and unions
• 46:11 Where do you put your types?
• 49:29 Typing Dependencies
• Definitely Typed
• Raycast Search npm
• 54:26 Global types
• 01:08 Now what?
• 03:58 SIIIIICK ××× PIIIICKS ×××

××× SIIIIICK ××× PIIIICKS ×××

• Scott: Elegrp USB C Walloutlet
• Wes: Barrina Shop Lights

Shameless Plugs

• Scott: LevelUp Tutorials
• Wes: Wes Bos Tutorials

Tweet us your tasty treats

• Scott’s Instagram
• LevelUpTutorials Instagram
• Wes’ Instagram
• Wes’ Twitter
• Wes’ Facebook
• Scott’s Twitter
• Make sure to include @SyntaxFM in your tweets

In this potluck episode of Syntax, Wes and Scott answer your questions about peer dependencies, Vitest, NVM and PNPM, using sprites for images, common MongoDB operations, and more!

Prismic - Sponsor

Prismic is a Headless CMS that makes it easy to build website pages as a set of components. Break pages into sections of components using React, Vue, or whatever you like. Make corresponding Slices in Prismic. Start building pages dynamically in minutes. Get started at prismic.io/syntax.

Sanity - Sponsor

Sanity.io is a real-time headless CMS with a fully customizable Content Studio built in React. Get a Sanity powered site up and running in minutes at sanity.io/create. Get an awesome supercharged free developer plan on sanity.io/syntax.

LogRocket - Sponsor

LogRocket lets you replay what users do on your site, helping you reproduce bugs and fix issues faster. It’s an exception tracker, a session re-player and a performance monitor. Get 14 days free at logrocket.com/syntax.

Show Notes

• 00:03 Welcome
• 01:53 Configuring home internet routers
• 04:42 Scott’s Home Assistant update
• Mushroom Theme
• 07:52 Could you explain to me peer-dependencies and how does it work?
• 13:24 Using Vitest do you still have to transpile code?
• 16:14 Can you talk about helpful and common MongoDB operations, beyond just CRUD.
• 18:49 How can I update the “updatedAt” field of the document on every save automatically?
• 20:40 What is aggregation, and when do you use it?
• 25:33 Sponsor: Prismic
• 27:27 How does NVM relate to PNPM?
• pnpm
• nvm
• 30:45 I’m looking to upskill from front-end JavaScript?
• 33:53 Is it possible to have a private NPM repo I can “npm install” from, or do I put my components up on NPM publicly?
• Creating a private npm package
• 37:51 Sponsor: LogRocket
• 39:14 Should a majority of lodash functions be considered deprecated?
• angus c just
• 42:36 Please do an episode on programming/learning with ADHD.
• 44:04 Should I still be putting images in sprites?
• 49:20 Does Mux have a simple mechanism for adding auth to each video or group of videos?
• Mux Create playback restriction
• 53:48 Sponsor: Sanity.io
• 55:02 Is there copyright issues with using public APIs?
• Moneypuck
• 59:38 ××× SIIIIICK ××× PIIIICKS ×××

××× SIIIIICK ××× PIIIICKS ×××

• Scott: Toto Bidet
• Wes: Sodastream

Shameless Plugs

• Scott: LevelUp Tutorials
• Wes: Wes Bos Tutorials

Tweet us your tasty treats

• Scott’s Instagram
• LevelUpTutorials Instagram
• Wes’ Instagram
• Wes’ Twitter
• Wes’ Facebook
• Scott’s Twitter
• Make sure to include @SyntaxFM in your tweets

In this episode of Syntax, Scott and Wes do a part 2 about Serverless — databases, files, secrets, auth, and more!

Sanity - Sponsor

Sanity.io is a real-time headless CMS with a fully customizable Content Studio built in React. Get a Sanity powered site up and running in minutes at sanity.io/create. Get an awesome supercharged free developer plan on sanity.io/syntax.

Freshbooks - Sponsor

Get a 30 day free trial of Freshbooks at freshbooks.com/syntax and put SYNTAX in the “How did you hear about us?” section.

Show Notes

2:47 - Wes tried Cloudflare Workers

• Also this is so cool:

Hey Wes, just listened to the latest Syntax episode on the serverless setup. Not sure if it’s an episode idea or not, but if you wanna do a bit of a dive on Cloudflare’s service workers, I’m currently leading an “invisible infrastructure migration” right now from a legacy WordPress setup to a new Storyblok/Netlify setup. We’re using Cloudflare’s service workers to basically “stitch” the headers/menus/footers from the old WordPress site into our new Netlify pages, but serving the page back as if it was part of the normal domain. This means we can migrate from the old to the new slowly without massively disrupting SEO, doing a lengthy/costly rebuild, etc.

• A word on Digital Ocean
• Kubernetes + FAAS allows you to scale up/down

13:54 - Secret management

• Some have a great UI
• Some have a CLI
• Some only have production
• Some have dev/staging/prod

16:24 - Vendor lock-in

• Two kinds of vendor lock-in
  • Lock into a low-level provider (Like AWS, or MongoDB)
  • Lock into a framework
• Questions to ask:
  • Can I go, take my app as-is, and host it on another provider?
  • Can I refactor the config and run my code as-is?
  • Do I need to refactor my code for it to run on other platforms?
• Next.js will only run on Now
• There is a community package
• Begin all runs on Arc.codes
• Firebase is locked in?

25:12 - Sharing dependencies

• Each function will have its own package.json, which can be a pain
• Publish utils a private module
• AWS Layers
• Import/export
• Bundle and tree shake

30:26 - Local development

• Now dev
• NPX sandbox
• Wrangler for Cloudflare workers

36:40 - Existing applications

• Difficult to move with many routes, but easy to move a Graphql API that has one single route
• Maybe do piece by piece instead of all at once
• Begin has http express method

45:21 - Data

• Any DB you want
• Dynamo DB integrated into many
• Firebase
• KV Storage for Cloudflare workers
• Fauna

48:14 - File storage

• Generally files go in the associated file place like Amazon S3, Backblaze B2, Cloudinary
• Many also have this integrated as well

52:18 - Auth

• Serverless is ephemeral and stateless
• JWT likely as sessions will work, but doesn’t really make sense

Links

• Cloudflare Workers
• Akamai
• MongoDB Stitch
• Hitler uses Kubernetes
• Digital Ocean
• Kubernetes
• Firebase
• Google Cloud
• Architect
• Next.js
• Now.sh
• Begin
• Netlify
• Now
• Wrangler
• Apollo Federation
• Monaco
• Postman
• Codesandbox
• DynamoDB
• Amazon S3
• Backblaze B2
• Cloudinary

××× SIIIIICK ××× PIIIICKS ×××

• Scott: The Power of Bad by John Tierney
• Wes: Socket Organizer

Shameless Plugs

• Scott: Animating React with Framer Motion - Sign up for the year and save 25%!
• Wes: All Courses - Use the coupon code ‘Syntax’ for $10 off!

Tweet us your tasty treats!

• Scott’s Instagram
• LevelUpTutorials Instagram
• Wes’ Instagram
• Wes’ Twitter
• Wes’ Facebook
• Scott’s Twitter
• Make sure to include @SyntaxFM in your tweets

It’s another Potluck! In this episode, Scott and Wes answer your questions about CSS, Angular, Svelte, dev job preparation and more!

Sentry - Sponsor

If you want to know what’s happening with your errors, track them with Sentry. Sentry is open-source error tracking that helps developers monitor and fix crashes in real time. Cut your time on error resolution from five hours to five minutes. It works with any language and integrates with dozens of other services. Syntax listeners can get two months for free by visiting Sentry and using the coupon code “tastytreat”.

Sanity - Sponsor

Sanity.io is a real-time headless CMS with a fully customizable Content Studio built in React. Get a Sanity powered site up and running in minutes at Sanity. Get an awesome supercharged free developer plan on Sanity.

Show Notes

3:23 - Q: How big is too big for state? I have an object of around 1000 items which will be used in a search box on the frontend of my site, but the list of items won’t be updated very often. What is the most efficient way to store these items and deliver them to the browser? Would you use state, a JSON object, a MongoDB document or some other way I haven’t thought of?

7:17 - Q: How much do I need to know before I start applying for dev jobs?

9:26 - Q: Do you folks have any opinions on Skeleton loaders, and any thoughts on how best to implement them? At my day job we’ve recently implemented skeleton loaders in a way that inherits the parent styles. Would love to know if there’s any best practices for this or ways we can improve our setup.

13:01 - Q: Do you think we still need to use CSS resets like normalize and sanitizer?

14:56 - Q: Scott, can you tell the story of how you landed an interview for a job in Japan? I am moving to Japan next year and wondering how that opportunity arose for you.

20:04 - Q: Do you guys have an opinion on angular good or bad? Or is it that you’re more comfortable and have used React and Vue more often, so they’re largely at the center of any framework discussions?

22:54 - Q: Could you talk about naming and structuring files/folders in your applications? How do you name components and folders? Do you follow some sort of path of how to structure files in a web app?

29:01 - Q: Do you think that Node is almost too powerful to handle some of the simpler tasks (a site that just needs a contact form for instance), or am I just missing some very simple configuration? It seems like other languages, such as PHP, might be better suited for some of these simple tasks and Node is really just meant for larger scale application? I guess what my question really boils down to is, for a freelancer, I understand that I need to have my hands in a lot of different tech stacks, but should I really dive in on PHP (or something else) to handle some of these simpler features a client might need?

33:24 - Q: What do you guys think about Svelte? Before everything goes WASM will Svelte be the future of web programming?

37:24 - Q: What smart phones and laptops do you use for work? (specific model and why)

42:38 - Q: Should I apply for other jobs even if my contract might get renewed? Should I feel loyal to a company?

××× SIIIIICK ××× PIIIICKS ×××

• Scott: DaVinci Resolve 16
• Wes: Affinity Photo

Shameless Plugs

• Wes: Wes’ Courses — use coupon code “syntax” at checkout and get and extra $10 off.
• Scott: Scott’s Crouses

Tweet us your tasty treats!

• Scott’s Instagram
• LevelUpTutorials Instagram
• Wes’ Instagram
• Wes’ Twitter
• Wes’ Facebook
• Scott’s Twitter
• Make sure to include @SyntaxFM in your tweets