How do we access the Package Server API with DDP?

trusktr · June 30, 2015, 3:47am

I’m not sure how to use DDP to access the Package Server API.

I tried:

var p = DDP.connect('https://packages.meteor.com')
p.call('syncNewPackageData', 'foo') // what token?

and get

Error invoking Method 'syncNewPackageData': Match failed [400]

And the same error with p.call('syncNewData'). (This is my first time trying to use DDP. )

Mind sharing a small example of how to call those methods?

robfallows · June 30, 2015, 12:48pm

Start with an empty syncToken object and make the call:

var syncToken = {};
var chunk = p.call('syncNewPackageData', syncToken);

You will now have a chunk object with a bunch of stuff, including:

chunk.syncToken - the next syncToken to use for the next chunk (pass this into the call).
chunk.collections - the package data as described in the API reference

HTH

trusktr · June 30, 2015, 6:32pm

@robfallows Thanks for the suggestion, but no luck yet! Here’s what I just tried:

> var p = DDP.connect('https://packages.meteor.com')
> p.status()
  <- Object {status: "connected", connected: true, retryCount: 0}
> var chunk = p.call('syncNewPackageData', {})
> chunk
  <- undefined

trusktr · June 30, 2015, 6:37pm

@robertpitt I tried your suggestion mentioned on GitHub but it didn’t work (note it requires the https for me):

DDP.connect("https://packages.meteor.com", {
    onConnected: function() {
      Packages.connection.call("syncNewPackageData", {format: "1.1"}, function(err, data) {
        console.dir(data);
      })
    }
})

That results in

Uncaught ReferenceError: Packages is not defined

when the onConnected handler fires.

trusktr · June 30, 2015, 6:41pm

Aha, the following works!

var p = DDP.connect("https://packages.meteor.com", {
    onConnected: function() {
      p.call("syncNewPackageData", {format: "1.1"}, function(err, data) {
        console.dir(data);
      })
    }
})

I’m not yet sure how I’d get the source code of a package though. Do we use syncNewData for that? And how? The Package Server API docs lacks detail. Oh, is that a typo where it says

Package information comes in three collections, which we sync to the client with the DDP method syncNewData.

? Did it mean to say syncNewPackageData? How are we supposed to know what we can pass inside the syncToken (I mean, aside from laboriously digging through source code)?

trusktr · June 30, 2015, 7:07pm

@sashko How would I find the info for a specific package? Do I need to loop through the 500-item-long array looking at each packageName/version until I find what I want? Or is there some API for this?

ccorcos · June 30, 2015, 9:16pm

I just figured out how to do all of this. It looks like you’ll have to download everything to be able to fetch a single one…

@Packages = new Mongo.Collection('packages')
@SyncToken = new Mongo.Collection('token')

@PackageServer = DDP.connect('https://packages.meteor.com/')

@getSyncToken = ->
  token = SyncToken.findOne()
  if token
    return R.omit(['_id'], token)
  else
    return undefined

@setSyncToken = (syncToken) ->
  SyncToken.remove({})
  SyncToken.insert(syncToken)

processVersion = (doc) ->
  obj = {}
  obj._id = doc._id
  obj.dependencies = R.keys(doc.dependencies)
  obj.version = doc.version
  obj.name = doc.packageName
  obj.description = doc.description
  return obj

@updatePackageData = (collections) ->
  packages = R.map(processVersion, collections.versions)
  for p in packages
    Packages.upsert(p._id, p)

@update = ->
  console.log "updating..."
  updateLoop()

@updateLoop = ->
  upToDate = false
  while not upToDate
    token = getSyncToken()
    console.log "...fetch, #{token?.versions}"
    {syncToken, collections, upToDate, resetData} = PackageServer.call("syncNewPackageData", token or  {format: "1.1"})

    if resetData
      console.log "...reset"
      SyncToken.remove({})
      Packages.remove({})
    
    setSyncToken(syncToken)
    
    if collections?.versions
      console.log "...upsert #{collections?.versions?.length}"
      updatePackageData(collections)
    
    if upToDate
      console.log("...done")
      return
    else
      console.log("...not done yet")

centiqdev · July 1, 2015, 12:12am

Hey, it’s robert here, it’s pretty simple to rotate the package meta, example:

if (Meteor.isServer) {
  // Sync Token
  var syncToken = {format: "1.1"};

  // Connection
  var conn = DDP.connect("https://packages.meteor.com");

  var result;
  while(!(result = conn.call("syncNewPackageData", syncToken)).upToDate ) {
    syncToken = result.syncToken;

    console.log(
      "packages(%d) - versions(%d) - builds(%d) next(%s)",
      syncToken.packages,
      syncToken.versions,
      syncToken.builds,
      result.upToDate ? "false" : "true"
    );

    // Use the following to access data
    result.collections.packages;
    result.collections.versions;
    result.collections.builds;
    result.collections.releaseTracks;
    result.collections.releaseVersions;
  }
}

The tar.gz location for the source code of each release should be in result.collections.versions and looks like:

{ _id: '6ThA4YurAd7MMKG2N',
  compilerVersion: 'meteor/12',
  containsPlugins: false,
  dependencies: 
   { meteor: { constraint: null, references: [Object] },
     webapp: { constraint: null, references: [Object] },
     livedata: { constraint: null, references: [Object] },
     'mongo-livedata': { constraint: null, references: [Object] },
     underscore: { constraint: null, references: [Object] },
     deps: { constraint: null, references: [Object] },
     retry: { constraint: null, references: [Object] },
     reload: { constraint: null, references: [Object] } },
  description: 'Update the client when new client code is available',
  earliestCompatibleVersion: '1.0.0',
  ecRecordFormat: '1.0',
  git: null,
  lastUpdated: Thu Sep 04 2014 01:38:03 GMT+0100 (BST),
  packageName: 'autoupdate',
  published: Sat Aug 02 2014 22:57:09 GMT+0100 (BST),
  publishedBy: { username: 'ekate', id: 'Nz3s5L6cdmS5Kt9Eq' },
  source: 
   { tarballHash: 'CUERMjgZpHvE5ORCvOdMNfRFYp9/DVqgjSmS9OULzx8=',
     treeHash: 'yN24hFTEhrGV1UC1S6zeTXIV7/gV+zX8tVSYqXi2G6k=',
     url: 'https://warehouse.meteor.com/sources/Nz3s5L6cdmS5Kt9Eq/1407016629326/CAnpNQWKSL/autoupdate-1.0.0-source.tgz' },
  version: '1.0.0' }

centiqdev · July 1, 2015, 12:17am

Im not sure if there is an api for searching for an individual package, this is primarily used to populate a local sqlite database on your machine, so you have a fast index for sarching packages and dependencies.

The sync token is also stored in sql so each time you run meteor it will update the difference since the last run…

centiqdev · July 1, 2015, 12:19am

Also Joe, this API isn’t really intended for public use, not discouraged but not encouraged if you know what I mean.

ccorcos · July 1, 2015, 12:55am

@trusktr, you can take this code and deploy a server with the API you desire

robfallows · July 1, 2015, 9:55am

Weird. My code is just:

Meteor.startup(function() {
  var p = DDP.connect('https://packages.meteor.com');
  var result = p.call('syncNewPackageData', {});
  console.log(result);
});

and I get the data. I can only surmise that due to the additional network latency (I’m in Nottingham, UK), the “onConnected” has had enough time to be there before the request for data is made.

Lesson learned: Just because it works doesn’t mean it works!

Glad you got it all together in the end .

As a final thought, you could also take a look at fastosphere - an open source atmosphere.

trusktr · July 6, 2015, 12:36am

@robfallows Nice! Fastosphere’s source code has some nice example code. By the way, I was trying it on the client side, in Chrome console, so that’s why I had to use the callback method. Makes sense that it’s sync on the server.