Meteor and the Internet of Things (IoT) - whats best to use? Collection, Meteor-Streams, Streamy

Use upsert on the local collection, with a key of the topic. Then you will have a collection with one record for each distinct topic, including wildcards, with the last message in it. That’s what I use.

Tom, have you looked at our project ThingStudio http://www.thingstud.io? This is, of course, a Meteor app.

1 Like

Hi Mike, yes I already looked weeks ago on the ThingStudio but have to do it again to see your improvements. Btw. looks overall very nice!

Tom

Hi @arunoda, thanks for your feedback.

Do you mean it is dead in case that you do not support it any more or is it dead in your eyes while you wouldn’t build it again (that way) with your todays knowledge?

Cheers, Tom

Hi Mike ( @mkarliner )

do you use a Mongo.Collection with upsert() or do you access (upsert) the local collection only within the client app?

I gave an example how I do it by now. This still works fine but I am not sure if as nice as yours.

In the next post you find just small examples for a server based ticker updating a client local (volatile) collection

For those interested in, I have created some meteorpads for this

  1. As simple as possible:
    http://meteorpad.com/pad/jYDG3wjryGc6DHXa6/Sample_Publish_to_Local-Collection

  2. Using Cursors .observe to get broadcasted information
    http://meteorpad.com/pad/i7ec92p45kswPF89E/Sample_Publish_to_Local-Collection_via_observe

  3. Do it the reactive way as described by DiscoverMeteor template level descriptions
    http://meteorpad.com/pad/Zq4QdMW84rKGersFH/Sample_Publish_to_Local-Collection_Reactive_way

1 Like

Hi Tom,

What are some advantages and/or disadvantages of each approach you used for the meteorpad examples?

Updated meteorpads at post 11

Hi Donald,

to be honest I don’t know this by now. I just created some tests to get suitable and working options.

To know what appears to be best practise, I created this thread on Stack Exchange

Hopefully someone will help to point out.

Hi tom,

i’m working on something similar, with a lot connected devices. My suggestions:

  1. Websockets are your friends. Also webrtc is a good tool. Operational transformations are another excellent tool. Each cover a slightly different scenario, with websockets focusing on compatibility and performance, webrtc on bandwidth efficiency and OT on consistency. Especially the OT are useful in the industrial sector, I suggest you give them a look before finding down the line that a race condition is breaking your software. Trust. Me. On. This.

  2. If you have problem with performance I strongly suggest aggregating the data.

  3. I suggest you drop the polling pattern, for a pub/sub pattern.

Hi @muaddib

thanks for your reply and your suggestions.

R1.) We do not have practical experiences with WebRTC but Sockets and OT are already used on our projects. My Cons to them in conjuction with meteor is just that we have a “different/another” channel between client and server and we have to control and audit that in addition. So I am trying to check if there is a (best) “meteor” friendly / aligned way.

R2.) Performance is not an issue yet

R3.) What do you mean by “drop the polling pattern” in my case? Did I wrote something about that? We are running the current project on a UDOO board which combines an ARM Cortex and an Arduino CPU on one board. There is no “networking” just serial line communication between the two devices. The Arduino part runs some self written real time software which handles the hardware independly and send ready records to the server process. On that we just have a SerialLine listener. For other “remote” devices we are very happy with MQTT (RabbitMQ) as transport mechanism also for latency and on/off situations.

I am assured by your note “each cover a slight different scenario” that for our use-case the local meteor collection shown in the meteorpads might a good and meteor friendly solution.

If you are interested Muaddib, I would really like to get more from your project and to exchange some knowledge with you.

Cheers
Tom

1 Like

first of all I’m dictating this to my phone on the train. I apologize in advance for all the mistakes I will make. But this topic really interest me.

second, i warn you that my advice are really general and might not be suited for your use case. take my advice with a grain of salt.

at first I didn’t click on the app link. now I think I got it better. basically you have several clients feeding data To the server, and a few clients consuming data from the server.

why you not use DDP client libraries? It’s native, redundant and fast.

IoT is really cool, but hardware adds a LOT of complexity. Why not use raspberry pi? It makes much more sense industrially speaking. It’s more stable, and libraries are much more optimized (which means that they are faster despite your weaker CPU). Plus in 10 years what do you think you will still be able to buy?

Pro tip: hardware producer are bound by the GPS License Agreement not to increase the precision of the system. you are not bound by such agreement. you can use 2 receivers to greatly increase the precision of the system

Feel free to drop me any question you have.

first of all I’m dictating this to my phone on the train. I apologize in advance for all the mistakes I will make. But this topic really interest me.

Great voice or voice recognition or both :slight_smile: not much mistakes at your post.

second, i warn you that my advice are really general and might not be suited for your use case. take my advice with a grain of salt.

Yeah, for sure but thanks to remind that

why you not use DDP client libraries

At least that is my question, If you are looking at YukanOO Streamy package this is DDP. I am asking for pros and cons to use this instead of pub/sub with a local colllection.

One cons (as implemented by Yukan and MDG by now) are nasty debugging messages from meteor: see issue-4 and issue-5

IoT is really cool, but hardware adds a LOT of complexity.

Fully agree, especially in germany. We have much work to fullfill norms like CE, TÜV and so on to be able to bring our product to market. This is something totally unknown for us in case that we just delivered software previously without those long terms.

But in the end you could create great complex but easy to handle products.

Why not use raspberry pi?

Why should I rely on Raspberry today? The UDOO and ODROID pieces are much more performant than a RasPi. At least we do not depend to any of them all. Wherever we got running our Linux Core image - we are on the scene.

you can use 2 receivers to greatly increase

Nice idea and tip, but this will raise our puchase price too much because we already have choosing a very good but expensive GPS module.

The solutions I proposed you are not DDP based. It boils down to how much data are you sending. If performance is not an issue, a collection should be fine. On the polar opposite you have websockets. You could write a C++ wrapper (many already exists) that simply pipe output to websocket, allowing you to use much less CPU. If you format the output correctly it shouldn’t be hard to debug either.

About TuV, since I’m Italian I know what you mean. Pro tip:
If you use already certified hardware and USB connections, then simply print a 3d case to encolose everything, you can get away just with a self-declaration of conformity, saving HUGE bucks.

About raspberry pi vs Udoo:

Let’s you need to get a full low voltage directive certification. It should cost AT LEAST 40k $ (but I bet it will easily double) if you use a german company.

Now you get a platform built on top of UDOO certfied. UDOO changes 2 years down the road. You just wasted a lot of money.

Instead the raspberry pi guys promise you that they will ship the SAME PCB 10 years down the road. It’s a nice plus.

About being faster: It’s not always better, it’s not always true. Optimization means a lot.

About Gps accuracy: If you have 4 cheap gps receivers on your boat you can use Desargues’ theorem and greatly increase the precision of your location.

Check this paper

http://2012.sensorapps.org/sites/default/files/uploads/1569519559.pdf

with 3 receiver you get an order of magnitude more precision (6 m to 0.6m). I think that if you put 4 receiver on a big boat, plus you use differential equations you can get down to 1 cm.

I think I found a new toy project :stuck_out_tongue:

This is pretty cool setup! I am thinking about decisions and will test it on next sailing trip.
Will keep you informed :wink:

1 Like

Hi,

I noticed this thread and wanted to point out that we too are looking for a solution that permits us to send updates from a server to one/many clients. For our MMO (google ironbane) we really need a performant socket solution but I have serious doubts about Meteor Streams which we’re using right now. I haven’t tried Streamy yet so we might switch if you guys think that would be a better/more efficient solution.

Having a local collection and subscribing/publishing it does not seem to be an option as Meteor writes to the database on each change which would cause a huge lag in our usecase. We need to do network updates every 0.2 seconds, like any realtime multiplayer game but so far I’m not sure which path we should proceed with.

If I’m not mistaken Streamy is built on top of meteor streams, but should look at the source to confirm. Either way Meteor Streams will probably be more performant as it has less overhead calculations. In other words with Meteor streams you’ll have to write all the code yourself but it’ll be adjusted to your exact needs, so you’ll be able to optimize.

Also if you need very fast database performance you should look into redis as a database.

If these solutions don’t fit your needs you’ll probably have to write your own solution, which wouldn’t be a surprise if you’re building a realtime mmo.

1 Like

Hi @nlammertyn, @nikke

First of Streamy and MeteorStreams are independently solutions with different architectures.

We are working very successful with local meteor collection and own publisher methods. For our solutions it is also absolutely nice feature, that we have access to previous documents on local collection as well.

@nikke what you are looking for should be realizable by observe methods as well. Have a look at this timer demo which will update every 0.1 second.

http://meteorpad.com/pad/gnoYXFtKk9ByEWwsm/Copy%20of%20Sample_Publish_to_Local-Collection_via_observe

Maybe you get a hint out of that.

Cheers,
Tom

2 Likes

@nlammertyn

Meteor Streams works fine for now, it’s just an outdated and dead project. It’s bound to cause problems later on as Meteor improves over time. We actually did build our own solution before Meteor came out using socket.io, but now we’d like to stick with Meteor as that’s a managed solution and has caused us far less headaches than in the days of node.js, grunt, etc. Integrating a solution like socket.io that runs simultaneously with Meteor is not an option as a socket.io server would require an extra port, breaking SSL. Also having two websocket servers is just absurd I guess.

@tomfreudenberg

Thanks for the example. I was not aware that one could publish on the server like that. Is the server still creating a mongodb collection behind the scenes? E.g. is the Ticker counter being persisted somewhere? I guess it doesn’t matter as long as it doesn’t impact performance, but I’m still curious to know.

The main problem we’re facing is that we mostly need to send single events to users. Consider a player that has attacked a monster. We’d need to send an attack event using the monster ID and the weapon ID they’re carrying.

player.stream.emit('combat:attackEntity', {
    victimEntityUuid: victimEntity.uuid,
    itemUuid: item.uuid
});

Note that we don’t care about saving this event, or having access to previous documents. It’s just fire and forget. Using a local collection for this becomes hard unless we use an ‘events’ collection and publish it to each user, that acts as a message bus. I believe this is exactly what Meteor Streams is doing, except that it is outdated so we might take a closer look at your example. Thanks again!