Something that repeatedly comes up about free software projects is how much it's possible to achieve. Can you write an operating system from scratch? That's been done, several times. A desktop environment? Been done. A complete toolchain (compiler, linker, essential libraries)? Yeah, done.
These are all examples of large, impressive systems that have been made by one or a small number of people, even just in their free time. How is that possible? Here's a couple of ideas.
Have fun. If you're not enjoying it, and you're not getting paid (and maybe even then), you should perhaps do something else. Or something differently. Is it not fun because memory management in C is tedious? Maybe use a different language or some helper libraries that take the tediousness away. Is it not challenging enough? Make N bigger.
Do not believe the people who say your idea isn't plausible to do. They might be right, and if you idea is ambitious, it might take a long time to get it done. Believing everyone who says you can't is just demoralising, especially online where there are always at least 12765 people awake and happy to denigrate anyone else's ideas.
Do it. If you fail, you've had fun for a while and you probably learnt a thing or two. If you don't even try, those who don't believe in you were right.
Start small and simple, then grow organically. No, smaller and simpler than that. There is much truth in Antoine de Saint Exupéry's saying, "It seems that perfection is attained not when there is nothing more to add, but when there is nothing more to remove.". Your first version should be as small and simple as you can make it. If you can think of anything to remove that doesn't render the thing unusable for all use cases, you can simplify further.
When you have the smallest, simplest first version that is possible, you can start growing. Get a user (even if it is just you), and whenever they want a new feature, and that feature fits in your vision of your program, implement that next. Repeat a thousand times. Keep doing it. Find others to help. Don't give up. Persevere. It might take a year or ten, but you'll eventually have something that makes people wonder how you managed to make something so impressive.