Webtrends Data Collection API

In creating our Data Collection API, we chose to use POST rather than the previous GET method that our javascript uses.  Why?

This is a great question and worth a quick discussion.


Why a new collection method?

It's worth mentioning that the GET request, that the WT javascript uses, works great but the very practice of using javascript has a specific use case:  web sites, portals, rich internet applications and such can/do use javascript.  But the use cases for an open Data Collection API are much more broad than that and includes offline devices, embedded operating systems, mobile applications, desktop applications in a wide variety of code languages.  Also, the hardware running these applications vary from set top boxes to televisions and support for image formats may or may not be able to be assumed.


That said, here are the reasons I remember being discussed in making the POST decision (I may have missed several others too):


A few reasons why to use POST

First, there are no URL length limits to deal with.  That is a very real issue when creating a GET request in all browsers.  All you need is a long page title and URL with custom data and you could easily exceed browser limits (it cuts off the rest). Browsers naturally need to deal with buffer overflow attacks and a host of other issues so they (wisely) put request limits in place for URLs. 



Second, POST doesn’t require an image request. Many devices don’t support certain image types and applications, offline devices, online/offline set top boxes etc. aren’t using a browser anyway so an image request is pointless.



Third, POST avoids “beacon filtering” that ad blockers and such do.  It's not overly common but some browser plugins block any 1x1 image requests.  In fact, we found out this year that at least one major manufacturer of mobile devices always filters 1x1 image requests on all their devices.  Ostensibly this was in order to reduce device bandwidth and increase performance but it's worth noting 



Fourth, code support for highly secure but very high performance POSTs exists in virtually all development platforms.  This is because POST is used for purchase transactions and a host of other generally secure transactions.  It makes sense when creating a library for various platforms to use POST to take advantage of this support in case customers want to send visitor IDs or other important data.  Similarly, the body of a POST is not visible in a browser window address bar 



Fifth, we’re currently using our DC API for use in our Mobile SDK libraries and since a “browser” is not assumed in these libraries, a GET request for an image seems silly in the application context. Instead of sending a 1x1 image with a 200 success return, we send back a more valuable response for debugging including, optionally, an echo of the exact request we received so you can debug your applications.



Sixth, it’s easy to add support for additional formats for the parameters in the POST body should people demand it.  I.e. we’ll add XML, JSON and other formats if customers want/request it (tell us what you are doing, we're interested!).  Right the POST body we accept is a series of simple text "parameter=value" pairs but that doesn’t need to be the case.  Incidentally, this may increase performance for large batch upload (sending a log queue of saved up events because a device was offline would be more efficient in JSON than any other format for example).



Seventh, and most importantly, our Data Collection API uses POST... and if you want to use DC API and all the awesome debug features it gives you, make the switch now!  Yes, this is my advertisement for the DC API   Maintaining code you create to support “No script” GET requests in code can become a hassle unless your requests are simple.  And DC API is great if I do say so myself.  The "verbose mode", return status, etc. is very handy.  Check it out at developer.webtrends.com






For those of you using "No Script" GET requests

None of this is to suggest that GET isn’t valid.  It's well understood and the simplest to drop into an application for sure.  Customers have been using this for years now and it's quick and effective but it would be worthwhile to make the switch to DC API.  It's easy to do and you'll be really happy about the flexibility it affords.