Multiple users using one Twitter Streaming API


#1

Hi,
I’m creating app using Twitter Streaming API (this node package to me more specific: https://www.npmjs.com/package/twitter).

Everything works flawlessly until 2 people start using app. If one person is using app, I can just push all the stream messages to DB and pub/sub to client. If two or more people are using the app, and they want to filter different tweets I need to put more filters into stream (because I can’t maintain more than one streams) and filter publications. And there comes my problem, how can I recognize which stream messages should go to desired client? Messages coming from stream are “unpossessed” and I have to match these with my clients.

I came up with idea of another collection with this schema:

TrackedStatuses.insert({
                searched_by: id,
                track: query
});

Everytime a new tweet appears in the stream I iterate through TrackedStatuses.tracks (TrackedStatuses.distinct(‘track’)) and check if this tweet text contains one of these TrackedStatuses.tracks.

Is there any simpler way to do this? For now, I don’t know how to simplify that process because:
-I cannot maintain 2 or more streams
-Twitter API cannot be run on client side because it’s node dependant

If that description isn’t clear, here’s my flow and core parts of my code:

  1. User submits search form

  2. New TrackedStatuses document is added:

    Meteor.methods({
    newSearch: function(id, query) {
    TrackedStatuses.remove({searched_by : id});
    TrackedStatuses.insert({
    searched_by: id,
    track: query
    });

         StartStream();
     },
    
  3. Stream restarts with a new ‘track’ parameter:

    twit.stream(‘statuses/filter’, {
    ‘track’: TrackedStatuses.distinct(‘track’).toString();
    },

  4. Once message appears in stream, I try to match it with TrackedStatuses.track (if message contains one of those) which leads me to TrackedStatuses.searched_by which is Meteor.userId()

  5. At this point, I can pub/sub it with filter like this:

    Meteor.publish(“messages”, function (filter) {
    var self = this;
    var subHandle = Messages.find({ searched_by : filter}).observeChanges({
    added: function (id, fields) {
    self.added(“messages”, id, fields);
    }
    });
    self.ready();
    self.onStop(function () {
    subHandle.stop();
    });
    });

I know my way of explaining thing maybe not clear so I’ll prepare flowchart explaining process asap.

Thanks.