Meteor Scaling - Redis Oplog [Status: Prod ready]

announce
406

Future of RedisOplog

After several discussions with people from the community, we realised that RedisOplog needs to become a standard. It saved several companies from collapsing due to high-traffic, not only making them scalable, but ultra-fast.

Currently it’s being backed and supported by my company, but if it’s not generating any revenue, there’s no guarantee (for some people) that it will continue. I am not saying that if it’s not generating revenue it will stop, I am saying that if other companies don’t see RedisOplog as profitable for the creators, they won’t think it will be supported in the future.

However I’m a believer in MIT license. So I’m a bit torn in how to proceed. We have the options:

  1. Create an opencollective account - Experience from different people shows that it does not generate too much

  2. Asking a yearly fee for companies (excluding charities, non-profit ones) that make over X profit / month. Or something like that.

  3. Ask ridiculous prices for consulting for an implementation strategy.

Currently I’m not convinced that this is the best way to go. As long as Cult of Coders is alive, RedisOplog will be maintained. We were in a tough period, we were having a toxic relationship with Meteor, while we loved it, it wasn’t fit for enterprise or high-traffic projects.

But it seems love always finds a way, that’s how Grapher and this package came alive. And ofcourse we had to make them open-source, it belonged to the community not to the company.

That being said I have some exciting news:

Next steps of Redis Oplog

  • Create a database request aggregator so if you have multiple listeners, you will query the database in a minimal way (This is super hard-core)
  • Modify the way it extends Mongo’s internals for reactivity to allow interoperability with other packages that do this.
  • Clear all the bugs
  • Study integration with Notifications API from MongoDB 3.6
  • Study moving to NPM, and offering MongoDB/Minimongo as a driver, and Meteor integration as a bridge. Allowing reactivity with any database, as long as you have the driver.
12 Likes
407

Hi @diaconutheodor ,

I continue to admire your efforts!
We are one of the companies saved by your products.

Right off the bat: Add a “pay what you want” option asap. See how it goes. I will for sure participate. (Apologies if this is what you meant by “opencollective”).

Here are some things I would pay for:

  • Stats. What are my subscriptions doing? (Could setup Kadira, but it’s a hassle).

  • Being able to install a package “that just works”, and not worry about my own Redis server.

RedisOplog is such a must have for high traffic sites, that MDG should offer it as an upgrade trough galaxy! Then you could get a cut of that. If they dont, perhaps NodeChef or somebody else would like to partner up.

Just some quick thoughts, hopefully a little helpful :slight_smile:

4 Likes
408

Thank you @diaconutheodor for your effort in the Redis Oplog and the other great contributions you made to the community.

Option two sounds reasonable to me. I’m fan of MIT license but I also think the creator of Open Source technologies need to be supported by the community directly or indirectly.

I’m curios about this, given your experience with Meteor and state of the technology and ecosystem today, do you still have those caveats when using the platform? are you folks (Cult of Coders) still using Meteor and how is your business going overall?

409

Yep - everyone just wants to make sure you guys stay alive :smiley: the market is constantly changing… maybe in 1-2 years freelance Meteor gigs will die out but having 1000 customers paying a trivial $100 per year might make it reasonable for you to keep the project going.

Indeed - I think when Galaxy launched many people expected something like that to be built into it.

Indeed - even with software that is MIT licensed there’s always a “catch” somewhere. However, charging money upfront may be the most honest way to do business in some cases.

To this point - I wonder if this could be implemented with some kind of simple JS utility that lets you play with in-memory data. It might be a bit crazy - but maybe not?

2 Likes
410

I’ve given this some thought and I just realised that this software wasn’t built by me, or my company alone, it was built together with the community. So the license won’t change now or EVER in the future, it will remain MIT. Case closed.

I will open some way to donate, and offer bounty hunts to fix bugs or implement features. My plan is to invest everything it’s making inside it. If people like it, they will invest. And maybe add a message when Meteor starts and isDevelopment about RedisOplog donation options. :smiley:

We are using only Meteor & React currently. Nothing else. We’re laser focused on these 2 technologies and we are thinking about React Native whether it’s worth it to start a new project with it or not. When we discuss with clients they do not care about which technology we use, really, they care about getting the job done, fast, cost-effective and very well done. We know no other tool better suited for this than Meteor.

Yep - everyone just wants to make sure you guys stay alive :smiley: the market is constantly changing… maybe in 1-2 years freelance Meteor gigs will die out but having 1000 customers paying a trivial $100 per year might make it reasonable for you to keep the project going.

@msavin it’s tempting but no I cannot force this. Our projects wouldn’t be anywhere if we would have to pay for lots of great tools that we have in production right now.

@nadeemjq very glad your app is doing well. Didn’t quite understand the install a package that just works part. We need Redis!

5 Likes
411

I really respect and share your attitude toward the Open Source MIT license, after all I’ll go as far as saying that the Open Source technologies along with the community around it are the ingredients that allowed me among millions of others to have a decent shot at starting our own businesses and chasing our bold ideas, so far that I’m forever grateful to each contributes in this ecosystem and I’m willing to help as much as I can including the payment of commercial tools built around it.

With that said, it’s very disheartening to see great Open Source tech that solves real community pain points get abandoned because the contributors get burned out or failed to get the financial or personal reward from it. That would be a big loss for the community both technically and personally, in addition to being real business risk, so I’d rather pay then see that happen. The reality of life is that we all have limited time, and for those who took the extra step to solve our problems and make our life easier, they should also open the door for us to give back to them, when we later benefit from their their work, so we all prosper :slight_smile:

All that to say I respect your work and I hope to see you succeed.

3 Likes
412

Yes, of course redis is needed. What I meant was that it would be nice if there was a plug and play package that connected to a redis-server that your company had already setup specifically for meteor, and that I could connect to if I paid a fee.

Right now I pay $7/month for Redis @ NodeChef. I have to set it up, I have to google all the different settings, I have to think about the next version, I have to share the keys with my team. etc, etc… What if I could instead just install redisOplog and put a licence key in my settings.json file? Then the package would automatically connect to your servers. No devOps for me, money for you. :slight_smile:

4 Likes
413

This is the last time I’ll bring it up: the people who would be interested in this package care more about sustainability than cost. In fact, it’s starting to look like many people are not respecting this as a free open source tool.

It’s great that you want to make it available to everyone, and like we discussed, you can keep discount it to $0/year for non-profits and new companies. Everyone else should and wants to pay, so that you can do things like:

  • develop a specification and documentation
  • at least offer notice if the package will be shut down
  • make new updates to keep up with Meteor releases
  • solves bugs sooner than later
  • offer long term support
  • etc

By being MIT licensed, you absolved yourself of these duties, and because of that, its hard for people to embrace this package.

When Does Open Source Makes Sense?

Open source makes a lot of sense for horizontal solutions like Linux, Meteor, OpenStack, so on, because many corporations have an interest in it, and so it makes sense for them to pool their resources and efforts. In other words, its a strategy for collaboration, attracting and growing talent, saving costs, etc.

Open source makes less sense for vertical solutions - things that address specific problems in specific markets. For example, Android’s apps and services, Meteor APM and Galaxy, Enterprise Linux distros, etc, costs money and could even be proprietary.

Redis Oplog is a vertical solution - it solves a problem that some companies experience when using a specific tool - and thus, it needs to be treated as such.

If you want to take a darker perspective into it: people do not want their businesses to be at the mercy of a third party package maintainer. So, they want to pay you not just out of reciprocity, but also to create a healthy relationship that ensures that the package will be properly maintained.

It’s sort of like when you do freelance work: some people may want an agreement that says you will ensure their app runs for a specific period of time.

And actually, they are trying to prevent the current situation: the package has outstanding bugs and issues, and it has not been updated for 6 months and counting. If someone would have made a bet on redis-oplog 6 months ago, there’s a chance they would have ended up in pain.

Anyways - this might read much colder over text than I intend to - but what I’m trying to say is that you made something that many people are really excited about, and they are calling on you to step it up.

It sounds like you like open source software a lot - and that you are not comfortable going a different way. However - why not take a risk and try something new? That’s what all this is really about it. In the worst case scenario, you’ll learn something, give everyone a refund, and revert to open source. In the best, well, you can figure :smiley:

10 Likes
414

Very well said with a lot of truth.

415

Right! So @diaconutheodor how about a subscription model (not a license)? Maybe it gives us an SLA for support based on tiers (which is a non-issue since you are super fast).

Remember, companies only want to see that it is being invested in (ever wondered why people still use Microsoft?). Not paying sends the wrong message. That money can go into the things @msavin listed above.

416

@msavin your points are very correct. I am already thinking of some ways to keep it MIT but also make it sustainable.

@ramez
I think for now the ways to go are:

  • Open the possibility of donation
  • Remind about donation on Meteor.startup() with a console.log
  • Create a Premium Package Offer where companies pay monthly and they get integration support and their issues have priority in getting solved. So basically having a contract for an MIT product. I think this can solve the issue of trust.
417

Hello guys, I did my homework and I thought a lot about this:

Conclusion:
RedisOplog will eventually die, being replaced by ChangeNotifications. The reasoning is found in my comment.

This is not bad news. It’s actually exciting news and I’m super happy about this.

5 Likes
418

Wouldn’t there still be some level of performance improvement above Mongo’s change notifications if using redis-oplog channels?

419

Me and my business would pay for a solution like that (redis oplog).

420

This is what I initially thought. But it does not, because Mongo can efficiently communicate the changes. Notifications API supports Aggregation Framework meaning you can do something like:

Posts.watch([{
   $match: { _id: postId }
}], function (changes) { ... })

^ Not sure that will be the API but close to that.

2 Likes
421

RedisOplog just got resuscitated. It seems that we can only watch changes for documents on immutable fields and would still have issues with FLS queries (read the GitHub thread to understand).

Ok, we’re back in business. RedisOplog will not die just yet. I’ll be thinking in depth about what it needs in the next days.

11 Likes
422

I’ve been following this package since the start last year (happy 1 year birthday?)

However I’m still not clear on when redis-oplog becomes beneficial for performance. Reading the thread it seems that the package performs slightly worse for apps with only a small number of users or low complexity queries therefore should only be used for large/complex applications

The docs/Github page doesn’t say either…

423

Actually I’ve been on Meteor forms for some time now, and it’s not clear to me when Meteor hits performance limits, it seems there many optimizations and workaround prior to hitting those limits, furthermore only few subset of apps do reach those scales…so really not much concrete general data to reason with as far as I can tell… all I read about is people worrying what happens when they reach that scale, they build the new Facebook and the 2 billion users all rush to signup at the same time :sweat_smile:

1 Like
424

I think big part of why we have not been hearing about this is because most people that hit these limits just ditch Meteor. Mixmax is a great example - they jumped on Meteor in the early days and then switched off it once the oplog began crashing their app. It’s a shame because this limitation is destroying Meteor’s success stories.

425

That’s unfortunate indeed.

I was curious about Mixmax case since you mentioned it and I found this blog from 2015. It seems the reason that they moved their font-end off Meteor is the initial page load time and not the oplog scaling (unless they stated that somewhere else). In fact all the reasons they’ve mentioned has been solved by the recent Meteor versions (SSR, dynamic imports and ability to switch the view layer).

But I wouldn’t be surprised if companies switch framework as soon as they get traction if they face performance issues which is very unfortunate for Meteor as you mentioned since it was a major enabler for those companies to succeed in the first place.

1 Like